Reset lost admin password of Grafana server (CentOS 7)

It could happen to lose your Grafana admin password and luckily there is a pretty easy way to the system administrator of the Grafana server to reset the admin password.
There is really good and simple page of how you can reset the admin password with grafana-cli in the documentation of Grafana website. Probably you can check it, too.

The problem is that most distributions like the one we use (CentOS 7) it rearrange the home directory of the Grafana software and if you try to use grafana-cli you’ll end up with the error mentioned in the section below the solution we offer you here! And you really could mess up the thing and hurry to say this software is buggy!
First, “grafana-cli admin reset-admin-password ” needs to know where is your configuration file and Grafana home directory, because from the configuration it will extract what kind of back-end server you use like sqlite3 or MySQL (or other) and the path or login data if needed to the back-end, then it will reset the password with the given one. At the bottom of this article there is another method of resetting the Grafana admin password without grafana-cli – manual reset in the database.
So first you should check with what command line arguments was started your instance of Grafana server:
Keep on reading!

Upgrade self-hosted atlassian bitbucket 4.x to 5.x in CentOS 7

Here we are going to show you a real example of how we upgraded out Atlassian Bitbucket server from 4.14.4 (around April 2017 installation) with the latest version of Atlassian Bitbucket 5.14.0. We use

  1. using self-hosted instance of Bitbucket 4.14.4
  2. Linux distro – CentOS 7.
  3. MySQL server for back-end. So there is a jdbc mysql driver (which should be installed after the upgrade).
  4. NGINX is used as proxy for our main HTTPS url. So we have changed our default configuration in server (in sever.xml).
  5. Bitbucket is loaded from a URL/bitbucket – “https://dev.example.com/bitbucket/”. So we have changed our default configuration (in sever.xml).

and you’ll see there are some pitfalls you can avoid if you follow our article. The latest git program in CentOS 7 is 1.8 and is not compatible with the new Atlassian Bitbucket 5.x, so we need to solve this problem before updating the server. Check out the official upgrade page here
Keep on reading!

Delay loading all pages in the Chromium and Google Chrome browsers tabs during start up and save a lot of memory

Recently we noticed our Chromium began to behavior really strange (we noticed it from Chromium version 69). When starting the Chromium browser the system tended to lag for 4-8 seconds and it was apparently connected with the browsers’ tabs opened in the previous session.
These days many people open multiple Internet pages in the browsers’ tabs and even more use the feature of

quitting the browser and then starting it back and the browser restores the previous session – all tabs with the pages loaded before.

. And time passes and we get used to open and leave more and more tabs, so not it is normal to have more than 30 opened page tabs in your browser, but last versions of the Chromium something has changed, because when you start the browser it completely loads all the pages, we do not think this was the case before so we dig it into the problem.
We use Chromium under (Gentoo) Linux and here is the problem:

top - 16:20:09 up 16 days, 16:36, 14 users,  load average: 4.77, 0.55, 0.43
Tasks: 288 total,  10 running, 201 sleeping,   0 stopped,   0 zombie
%Cpu(s): 82.6 us, 16.2 sy,  0.0 ni,  1.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem :  8143960 total,  3012776 free,  3828732 used,  1302452 buff/cache
KiB Swap:  7606268 total,  6855752 free,   750516 used.  2554612 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                 
16082 myuser     20   0 3975632 371280 191940 S 106.9  4.6   0:05.86 /usr/lib64/chromium-browser/chrome --extra-plugin-dir=/usr/lib64/nsbrowser/plugins --ppapi-flash-path=+ 
16380 myuser     20   0 1779900 116276  76624 S  28.4  1.4   0:00.41 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16420 myuser     20   0 1777396 117096  85464 S  18.6  1.4   0:00.30 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16540 myuser     20   0 1850308 109576  76616 R  17.6  1.3   0:00.24 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16565 myuser     20   0 1769716 108588  81336 S  17.6  1.3   0:00.23 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16400 myuser     20   0 1775876 105220  76932 R  12.7  1.3   0:00.21 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16334 myuser     20   0 1770128 111588  83760 S  11.8  1.4   0:00.26 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
 5644 myuser      9 -11 3474604  11112   8368 S   9.8  0.1  14:50.60 /usr/bin/pulseaudio --start --log-target=syslog                                                         
16346 myuser     20   0 1772160 103924  76920 R   9.8  1.3   0:00.17 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16369 myuser     20   0 1772060 101696  74928 R   9.8  1.2   0:00.16 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16409 myuser     20   0 1783340 115752  77116 R   9.8  1.4   0:00.18 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16435 myuser     20   0 1770664 105088  81656 S   9.8  1.3   0:00.19 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16483 myuser     20   0 1772228 112008  83176 S   9.8  1.4   0:00.17 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16509 myuser     20   0 1772116 108544  81732 S   9.8  1.3   0:00.20 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16448 myuser     20   0 1771368 101300  74956 R   8.8  1.2   0:00.14 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16459 myuser     20   0 1692864 102668  80676 R   8.8  1.3   0:00.15 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16506 myuser     20   0 1766364  99588  78384 S   7.8  1.2   0:00.15 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16577 myuser     20   0 1767296 102268  80272 S   7.8  1.3   0:00.14 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
21462 myuser     20   0 2599480 493592  95284 S   7.8  6.1  17:00.39 /usr/lib64/firefox/firefox -contentproc -childID 13 -isForBrowser -prefsLen 51222 -schedulerPrefs 0001+ 
16417 myuser     20   0 1767568  99584  75992 R   5.9  1.2   0:00.12 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16530 myuser     20   0 1764332  99036  78768 S   5.9  1.2   0:00.11 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16341 myuser     20   0 1688088  96856  77776 S   4.9  1.2   0:00.09 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16395 myuser     20   0 1765648  96284  75132 S   4.9  1.2   0:00.10 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16464 myuser     20   0 1773864 104360  76260 S   4.9  1.3   0:00.12 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16467 myuser     20   0 1765740  96184  75520 S   4.9  1.2   0:00.10 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16491 myuser     20   0 1691968  96864  75744 S   3.9  1.2   0:00.14 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16496 myuser     20   0 1691964  97048  75852 S   3.9  1.2   0:00.18 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16501 myuser     20   0 1691492  92592  72604 S   3.9  1.1   0:00.10 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16578 myuser     20   0 1765432  93580  73868 S   3.9  1.1   0:00.10 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16587 myuser     20   0 1765416  93624  73856 S   3.9  1.1   0:00.10 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
28861 myuser     20   0 3503208 405676  91716 S   3.9  5.0  11:41.21 /usr/bin/firefox                                                                                        
16454 myuser     20   0 1617744  87852  70072 R   2.9  1.1   0:00.09 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
16548 myuser     20   0 1692100  97336  76440 S   2.9  1.2   0:00.13 /usr/lib64/chromium-browser/chrome --type=renderer --file-url-path-alias=/gen=/usr/lib64/chromium-brow+ 
 5430 root      20   0  510884 133980 116036 S   2.0  1.6  39:33.10 /usr/bin/X -nolisten tcp -auth /var/run/sddm/{0c17f929-befd-402b-a80b-207e92f61272} -background none -+ 
16011 myuser     20   0  140316   3192   2632 R   2.0  0.0   0:00.44 top -d1                                                                                                 
 5193 root     -51   0       0      0      0 S   1.0  0.0  97:33.55 [irq/46-iwlwifi]                                                                                        
 5601 myuser     20   0 3422104  59664  23276 S   1.0  0.7  41:55.20 /usr/bin/kwin_x11 -session 10147150e9da000152673938100000083010144_1534452204_442575

As you can see starting Chromium it just flooded the system with hungry for CPU processes, so many that even a decent machine with 4 cores will get into troubles with 9 running Chromium process for a while! The load at the first seconds goes to 5-9 (as you can see here is 4.77 from below 1 – actually 0.5 ).
Keep on reading!

Use Intel QuickSync with ffmpeg to have hardware accelerated video decoding and encoding – CentOS 7.4.1708

This howto shows how you can use ffmpeg to hardware accelerate the video decoding and encoding if you have the proper Intel processor. Some of the Intel processors have GPU unit in them, which can be used to decode and encode video streams and it is pretty fast! High end Intel CPU could encode around 20 h264 (using faster preset) video streams on a single processor mini server as SUPERMICRO system using X11SSV-M4F.
Here are the steps how to enable the hardware options and what and how to install the software to hardware encode a video stream successful under linux – CentOS 7 distro.

For the first four steps just follow the howto “Install Intel Media Studio SDK and GPU driver from Intel Media Server Studio 2018 R1 – CentOS 7 (7.4.1708)” Here we include only the headers of the steps, because they are important part of this howto.
Installing Media Server Studio your system will get validated stack of software by Intel to build application against Media SDK interface:

  • libmfx.a dispatcher – the API for your application
  • libmfxhw64.so hardware implementation
  • related libraries (including OpenCL)
  • specific versions for libva and libdrm
  • user mode driver
  • modified i915 graphics driver module
  • kernel patches

Here are the steps:

STEP 1) Enable internal graphics in your system.

STEP 2) Install the right CentOS 7 – 7.4.1708

STEP 3) Download Intel Media Server Studio 2018 R1

STEP 4) Install Intel Media Studio SDK from Intel Media Server Studio 2018 R1

“Install Intel Media Studio SDK and GPU driver from Intel Media Server Studio 2018 R1 – CentOS 7 (7.4.1708)”

STEP 5) Tune pkgconfig libmfx (Intel library, installed with the Media Server Studio)

Create a file:

/usr/lib64/pkgconfig/libmfx.pc

with the following content:

prefix=/opt/intel/mediasdk
exec_prefix=${prefix}
libdir=${prefix}/lib/lin_x64
includedir=${prefix}/include
 
Name: libmfx
Description: Intel Media SDK
Version: 16.8
Libs: -L${libdir} -lmfx -lva -lstdc++ -ldl -lva-drm -ldrm
Cflags: -I${includedir} -I/usr/include/libdrm

STEP 6) Move the Media SDK include files in a mfx sub-directory

This is required because ffmpeg expect these files to be in a sub-directory

[root@srv ~]# mkdir /opt/intel/mediasdk/include/mfx
[root@srv ~]# cp /opt/intel/mediasdk/include/*.h /opt/intel/mediasdk/include/mfx/

STEP 7) Install yasm – a dependency for the ffmpeg software.

[root@srv ~]# wget http://www.tortall.net/projects/yasm/releases/yasm-1.3.0.tar.gz
[root@srv ~]# tar xzf yasm-1.3.0.tar.gz
[root@srv ~]# cd yasm-1.3.0
[root@srv yasm-1.3.0]# export CFLAGS="-O2 -march=native -msse3 -fomit-frame-pointer -pipe"
[root@srv yasm-1.3.0]# export CXXFLAGS="${CFLAGS}"
[root@srv yasm-1.3.0]# export CHOST="x86_64-pc-linux-gnu"
[root@srv yasm-1.3.0]# export LDFLAGS="-Wl,-O1"
[root@srv yasm-1.3.0]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
....
....
....
[root@srv yasm-1.3.0]# make -j8
gcc -std=gnu99  -I.  \
  -c -o genperf.o `test -f tools/genperf/genperf.c || echo './'`tools/genperf/genperf.c
gcc -std=gnu99  -I.  \
  -c -o gp-perfect.o `test -f tools/genperf/perfect.c || echo './'`tools/genperf/perfect.c
gcc -std=gnu99  -I.  \
  -c -o gp-phash.o `test -f libyasm/phash.c || echo './'`libyasm/phash.c
gcc -std=gnu99  -I.  \
  -c -o gp-xmalloc.o `test -f libyasm/xmalloc.c || echo './'`libyasm/xmalloc.c
gcc -std=gnu99  -I.  \
  -c -o gp-xstrdup.o `test -f libyasm/xstrdup.c || echo './'`libyasm/xstrdup.c
gcc -std=gnu99  -I.  \
  -c -o re2c-main.o `test -f tools/re2c/main.c || echo './'`tools/re2c/main.c
gcc -std=gnu99  -I.  \
  -c -o re2c-code.o `test -f tools/re2c/code.c || echo './'`tools/re2c/code.c
....
....
....
[root@srv yasm-1.3.0]# make install
make  install-recursive
make[1]: Entering directory `/root/yasm-1.3.0'
....
....
....
[root@srv yasm-1.3.0]#
[root@srv yasm-1.3.0]# ldconfig
[root@srv yasm-1.3.0]# cd ../
[root@srv ~]#

STEP 8) Download ffmpeg version 2.8.X and compile it

it is important to use this branch 2.8.X because we’ve tested with the others 3.2.X, 3.3.X, 3.4.X and there were errors and we unable to use the the GPU unit to accelerate our video decoding and encoding. Here is the different ffmpeg release https://ffmpeg.org/download.html#releases or you can use the github to download https://github.com/FFmpeg/FFmpeg/releases

[root@srv ~]# wget https://www.ffmpeg.org/releases/ffmpeg-2.8.14.tar.bz2
[root@srv ~]# tar xjf ffmpeg-2.8.14.tar.bz2
[root@srv ~]# cd ffmpeg-2.8.14
[root@srv ffmpeg-2.8.14]# export CFLAGS="-O2 -march=native -msse3 -fomit-frame-pointer -pipe"
[root@srv ffmpeg-2.8.14]# export CXXFLAGS="${CFLAGS}"
[root@srv ffmpeg-2.8.14]# export CHOST="x86_64-pc-linux-gnu"
[root@srv ffmpeg-2.8.14]# export LDFLAGS="-Wl,-O1"
[root@srv ffmpeg-2.8.14]# ./configure --enable-libmfx --prefix=/usr/local/ffmpeg-qsv
install prefix            /usr/local/ffmpeg-qsv
source path               .
C compiler                gcc
C library                 glibc
ARCH                      x86 (generic)
big-endian                no
runtime cpu detection     yes
yasm                      yes
MMX enabled               yes
MMXEXT enabled            yes
3DNow! enabled            yes
3DNow! extended enabled   yes
SSE enabled               yes
SSSE3 enabled             yes
AVX enabled               yes
XOP enabled               yes
FMA3 enabled              yes
FMA4 enabled              yes
i686 features enabled     yes
CMOV is fast              yes
EBX available             yes
EBP available             yes
debug symbols             yes
strip symbols             yes
optimize for size         no
optimizations             yes
static                    yes
shared                    no
postprocessing support    no
new filter support        yes
network support           yes
threading support         pthreads
safe bitstream reader     yes
SDL support               no
opencl enabled            no
texi2html enabled         no
perl enabled              yes
pod2man enabled           yes
makeinfo enabled          no
makeinfo supports HTML    no

Enabled programs:
ffmpeg                  ffprobe                 ffserver

External libraries:
iconv                   lzma                    zlib
libmfx

Enabled decoders:
aac                     ayuv                    frwu
aac_fixed               bethsoftvid             g2m
aac_latm                bfi                     g723_1
aasc                    bink                    g729
ac3                     binkaudio_dct           gif
ac3_fixed               binkaudio_rdft          gsm
adpcm_4xm               bintext                 gsm_ms
adpcm_adx               bmp                     h261
adpcm_afc               bmv_audio               h263
adpcm_ct                bmv_video               h263i
adpcm_dtk               brender_pix             h263p
adpcm_ea                c93                     h264
adpcm_ea_maxis_xa       cavs                    h264_qsv
adpcm_ea_r1             ccaption                hap
adpcm_ea_r2             cdgraphics              hevc
adpcm_ea_r3             cdxl                    hevc_qsv
adpcm_ea_xas            cinepak                 hnm4_video
adpcm_g722              cljr                    hq_hqa
adpcm_g726              cllc                    hqx
adpcm_g726le            comfortnoise            huffyuv
adpcm_ima_amv           cook                    iac
adpcm_ima_apc           cpia                    idcin
adpcm_ima_dk3           cscd                    idf
adpcm_ima_dk4           cyuv                    iff_byterun1
adpcm_ima_ea_eacs       dca                     iff_ilbm
adpcm_ima_ea_sead       dds                     imc
adpcm_ima_iss           dfa                     indeo2
adpcm_ima_oki           dirac                   indeo3
adpcm_ima_qt            dnxhd                   indeo4
adpcm_ima_rad           dpx                     indeo5
adpcm_ima_smjpeg        dsd_lsbf                interplay_dpcm
adpcm_ima_wav           dsd_lsbf_planar         interplay_video
adpcm_ima_ws            dsd_msbf                jacosub
adpcm_ms                dsd_msbf_planar         jpeg2000
adpcm_sbpro_2           dsicinaudio             jpegls
adpcm_sbpro_3           dsicinvideo             jv
adpcm_sbpro_4           dss_sp                  kgv1
adpcm_swf               dvbsub                  kmvc
adpcm_thp               dvdsub                  lagarith
adpcm_thp_le            dvvideo                 loco
adpcm_vima              dxa                     mace3
adpcm_xa                dxtory                  mace6
adpcm_yamaha            eac3                    mdec
aic                     eacmv                   metasound
alac                    eamad                   microdvd
alias_pix               eatgq                   mimic
als                     eatgv                   mjpeg
amrnb                   eatqi                   mjpegb
amrwb                   eightbps                mlp
amv                     eightsvx_exp            mmvideo
anm                     eightsvx_fib            motionpixels
ansi                    escape124               movtext
ape                     escape130               mp1
apng                    evrc                    mp1float
ass                     exr                     mp2
asv1                    ffv1                    mp2float
asv2                    ffvhuff                 mp3
atrac1                  ffwavesynth             mp3adu
atrac3                  fic                     mp3adufloat
atrac3p                 flac                    mp3float
aura                    flashsv                 mp3on4
aura2                   flashsv2                mp3on4float
avrn                    flic                    mpc7
avrp                    flv                     mpc8
avs                     fourxm                  mpeg1video
avui                    fraps                   mpeg2_qsv
mpeg2video              prores                  tscc2
mpeg4                   prores_lgpl             tta
mpegvideo               ptx                     twinvq
mpl2                    qcelp                   txd
msa1                    qdm2                    ulti
msmpeg4v1               qdraw                   utvideo
msmpeg4v2               qpeg                    v210
msmpeg4v3               qtrle                   v210x
msrle                   r10k                    v308
mss1                    r210                    v408
mss2                    ra_144                  v410
msvideo1                ra_288                  vb
mszh                    ralf                    vble
mts2                    rawvideo                vc1
mvc1                    realtext                vc1_qsv
mvc2                    rl2                     vc1image
mxpeg                   roq                     vcr1
nellymoser              roq_dpcm                vima
nuv                     rpza                    vmdaudio
on2avc                  rv10                    vmdvideo
opus                    rv20                    vmnc
paf_audio               rv30                    vorbis
paf_video               rv40                    vp3
pam                     s302m                   vp5
pbm                     sami                    vp6
pcm_alaw                sanm                    vp6a
pcm_bluray              sgi                     vp6f
pcm_dvd                 sgirle                  vp7
pcm_f32be               shorten                 vp8
pcm_f32le               sipr                    vp9
pcm_f64be               smackaud                vplayer
pcm_f64le               smacker                 vqa
pcm_lxf                 smc                     wavpack
pcm_mulaw               smvjpeg                 webp
pcm_s16be               snow                    webvtt
pcm_s16be_planar        sol_dpcm                wmalossless
pcm_s16le               sonic                   wmapro
pcm_s16le_planar        sp5x                    wmav1
pcm_s24be               srt                     wmav2
pcm_s24daud             ssa                     wmavoice
pcm_s24le               stl                     wmv1
pcm_s24le_planar        subrip                  wmv2
pcm_s32be               subviewer               wmv3
pcm_s32le               subviewer1              wmv3image
pcm_s32le_planar        sunrast                 wnv1
pcm_s8                  svq1                    ws_snd1
pcm_s8_planar           svq3                    xan_dpcm
pcm_u16be               tak                     xan_wc3
pcm_u16le               targa                   xan_wc4
pcm_u24be               targa_y216              xbin
pcm_u24le               tdsc                    xbm
pcm_u32be               text                    xface
pcm_u32le               theora                  xl
pcm_u8                  thp                     xsub
pcm_zork                tiertexseqvideo         xwd
pcx                     tiff                    y41p
pgm                     tmv                     yop
pgmyuv                  truehd                  yuv4
pgssub                  truemotion1             zero12v
pictor                  truemotion2             zerocodec
pjs                     truespeech              zlib
png                     tscc                    zmbv
ppm

Enabled encoders:
a64multi                huffyuv                 png
a64multi5               jpeg2000                ppm
aac                     jpegls                  prores
ac3                     ljpeg                   prores_aw
ac3_fixed               mjpeg                   prores_ks
adpcm_adx               movtext                 qtrle
adpcm_g722              mp2                     r10k
adpcm_g726              mp2fixed                r210
adpcm_ima_qt            mpeg1video              ra_144
adpcm_ima_wav           mpeg2_qsv               rawvideo
adpcm_ms                mpeg2video              roq
adpcm_swf               mpeg4                   roq_dpcm
adpcm_yamaha            msmpeg4v2               rv10
alac                    msmpeg4v3               rv20
alias_pix               msvideo1                s302m
amv                     nellymoser              sgi
apng                    pam                     snow
ass                     pbm                     sonic
asv1                    pcm_alaw                sonic_ls
asv2                    pcm_f32be               srt
avrp                    pcm_f32le               ssa
avui                    pcm_f64be               subrip
ayuv                    pcm_f64le               sunrast
bmp                     pcm_mulaw               svq1
cinepak                 pcm_s16be               targa
cljr                    pcm_s16be_planar        tiff
comfortnoise            pcm_s16le               tta
dca                     pcm_s16le_planar        utvideo
dnxhd                   pcm_s24be               v210
dpx                     pcm_s24daud             v308
dvbsub                  pcm_s24le               v408
dvdsub                  pcm_s24le_planar        v410
dvvideo                 pcm_s32be               vorbis
eac3                    pcm_s32le               wavpack
ffv1                    pcm_s32le_planar        webvtt
ffvhuff                 pcm_s8                  wmav1
flac                    pcm_s8_planar           wmav2
flashsv                 pcm_u16be               wmv1
flashsv2                pcm_u16le               wmv2
flv                     pcm_u24be               xbm
g723_1                  pcm_u24le               xface
gif                     pcm_u32be               xsub
h261                    pcm_u32le               xwd
h263                    pcm_u8                  y41p
h263p                   pcx                     yuv4
h264_qsv                pgm                     zlib
hevc_qsv                pgmyuv                  zmbv

Enabled hwaccels:
h263_vaapi              hevc_vaapi              vc1_qsv
h264_qsv                mpeg2_qsv               vc1_vaapi
h264_vaapi              mpeg2_vaapi             wmv3_vaapi
hevc_qsv                mpeg4_vaapi

Enabled parsers:
aac                     dvdsub                  mpegvideo
aac_latm                flac                    opus
ac3                     g729                    png
adx                     gsm                     pnm
bmp                     h261                    rv30
cavsvideo               h263                    rv40
cook                    h264                    tak
dca                     hevc                    vc1
dirac                   mjpeg                   vorbis
dnxhd                   mlp                     vp3
dpx                     mpeg4video              vp8
dvbsub                  mpegaudio               vp9
dvd_nav

Enabled demuxers:
aa                      g723_1                  mpjpeg
aac                     g729                    mpl2
ac3                     gif                     mpsub
act                     gsm                     msnwc_tcp
adf                     gxf                     mtv
adp                     h261                    mv
adx                     h263                    mvi
aea                     h264                    mxf
afc                     hevc                    mxg
aiff                    hls                     nc
amr                     hnm                     nistsphere
anm                     ico                     nsv
apc                     idcin                   nut
ape                     idf                     nuv
apng                    iff                     ogg
aqtitle                 ilbc                    oma
asf                     image2                  paf
asf_o                   image2_alias_pix        pcm_alaw
ass                     image2_brender_pix      pcm_f32be
ast                     image2pipe              pcm_f32le
au                      image_bmp_pipe          pcm_f64be
avi                     image_dds_pipe          pcm_f64le
avr                     image_dpx_pipe          pcm_mulaw
avs                     image_exr_pipe          pcm_s16be
bethsoftvid             image_j2k_pipe          pcm_s16le
bfi                     image_jpeg_pipe         pcm_s24be
bfstm                   image_jpegls_pipe       pcm_s24le
bink                    image_pictor_pipe       pcm_s32be
bintext                 image_png_pipe          pcm_s32le
bit                     image_qdraw_pipe        pcm_s8
bmv                     image_sgi_pipe          pcm_u16be
boa                     image_sunrast_pipe      pcm_u16le
brstm                   image_tiff_pipe         pcm_u24be
c93                     image_webp_pipe         pcm_u24le
caf                     ingenient               pcm_u32be
cavsvideo               ipmovie                 pcm_u32le
cdg                     ircam                   pcm_u8
cdxl                    iss                     pjs
cine                    iv8                     pmp
concat                  ivf                     pva
data                    jacosub                 pvf
daud                    jv                      qcp
dfa                     latm                    r3d
dirac                   live_flv                rawvideo
dnxhd                   lmlm4                   realtext
dsf                     loas                    redspark
dsicin                  lrc                     rl2
dss                     lvf                     rm
dts                     lxf                     roq
dtshd                   m4v                     rpl
dv                      matroska                rsd
dvbsub                  mgsts                   rso
dxa                     microdvd                rtp
ea                      mjpeg                   rtsp
ea_cdata                mlp                     sami
eac3                    mlv                     sap
epaf                    mm                      sbg
ffm                     mmf                     sdp
ffmetadata              mov                     sdr2
filmstrip               mp3                     segafilm
flac                    mpc                     shorten
flic                    mpc8                    siff
flv                     mpegps                  sln
fourxm                  mpegts                  smacker
frm                     mpegtsraw               smjpeg
g722                    mpegvideo               smush
sol                     tmv                     wav
sox                     truehd                  wc3
spdif                   tta                     webm_dash_manifest
srt                     tty                     webvtt
stl                     txd                     wsaud
str                     vc1                     wsvqa
subviewer               vc1t                    wtv
subviewer1              vivo                    wv
sup                     vmd                     xa
swf                     vobsub                  xbin
tak                     voc                     xmv
tedcaptions             vplayer                 xwma
thp                     vqf                     yop
tiertexseq              w64                     yuv4mpegpipe

Enabled muxers:
a64                     ipod                    pcm_s32be
ac3                     ircam                   pcm_s32le
adts                    ismv                    pcm_s8
adx                     ivf                     pcm_u16be
aiff                    jacosub                 pcm_u16le
amr                     latm                    pcm_u24be
apng                    lrc                     pcm_u24le
asf                     m4v                     pcm_u32be
asf_stream              matroska                pcm_u32le
ass                     matroska_audio          pcm_u8
ast                     md5                     psp
au                      microdvd                rawvideo
avi                     mjpeg                   rm
avm2                    mkvtimestamp_v2         roq
bit                     mlp                     rso
caf                     mmf                     rtp
cavsvideo               mov                     rtp_mpegts
crc                     mp2                     rtsp
dash                    mp3                     sap
data                    mp4                     segment
daud                    mpeg1system             singlejpeg
dirac                   mpeg1vcd                smjpeg
dnxhd                   mpeg1video              smoothstreaming
dts                     mpeg2dvd                sox
dv                      mpeg2svcd               spdif
eac3                    mpeg2video              spx
f4v                     mpeg2vob                srt
ffm                     mpegts                  stream_segment
ffmetadata              mpjpeg                  swf
filmstrip               mxf                     tee
flac                    mxf_d10                 tg2
flv                     mxf_opatom              tgp
framecrc                null                    truehd
framemd5                nut                     uncodedframecrc
g722                    oga                     vc1
g723_1                  ogg                     vc1t
gif                     oma                     voc
gxf                     opus                    w64
h261                    pcm_alaw                wav
h263                    pcm_f32be               webm
h264                    pcm_f32le               webm_chunk
hds                     pcm_f64be               webm_dash_manifest
hevc                    pcm_f64le               webp
hls                     pcm_mulaw               webvtt
ico                     pcm_s16be               wtv
ilbc                    pcm_s16le               wv
image2                  pcm_s24be               yuv4mpegpipe
image2pipe              pcm_s24le

Enabled protocols:
async                   hls                     rtmpt
cache                   http                    rtp
concat                  httpproxy               srtp
crypto                  icecast                 subfile
data                    md5                     tcp
ffrtmphttp              mmsh                    udp
file                    mmst                    udplite
ftp                     pipe                    unix
gopher                  rtmp

Enabled filters:
acrossfade              dctdnoiz                paletteuse
adelay                  deband                  pan
adrawgraph              decimate                perms
aecho                   deflate                 pixdesctest
aeval                   dejudder                psnr
aevalsrc                deshake                 qp
afade                   detelecine              random
aformat                 dilation                removegrain
ainterleave             drawbox                 removelogo
allpass                 drawgraph               replaygain
allrgb                  drawgrid                reverse
allyuv                  dynaudnorm              rgbtestsrc
alphaextract            earwax                  rotate
alphamerge              edgedetect              scale
amerge                  elbg                    scale2ref
amix                    equalizer               select
amovie                  erosion                 sendcmd
anull                   extractplanes           separatefields
anullsink               fade                    setdar
anullsrc                fftfilt                 setfield
apad                    field                   setpts
aperms                  fieldmatch              setsar
aphasemeter             fieldorder              settb
aphaser                 flanger                 showcqt
aresample               format                  showfreqs
areverse                fps                     showinfo
aselect                 framepack               showpalette
asendcmd                framerate               showspectrum
asetnsamples            framestep               showvolume
asetpts                 gradfun                 showwaves
asetrate                haldclut                showwavespic
asettb                  haldclutsrc             shuffleplanes
ashowinfo               hflip                   sidechaincompress
asplit                  highpass                signalstats
astats                  histogram               silencedetect
astreamsync             hqx                     silenceremove
atadenoise              hstack                  sine
atempo                  hue                     smptebars
atrim                   idet                    smptehdbars
avectorscope            il                      split
bandpass                inflate                 ssim
bandreject              interleave              swapuv
bass                    join                    tblend
bbox                    lenscorrection          telecine
biquad                  life                    testsrc
blackdetect             lowpass                 thumbnail
blend                   lut                     tile
cellauto                lut3d                   transpose
channelmap              lutrgb                  treble
channelsplit            lutyuv                  trim
chorus                  mandelbrot              unsharp
codecview               mergeplanes             vectorscope
color                   movie                   vflip
colorbalance            negate                  vignette
colorchannelmixer       noformat                volume
colorkey                noise                   volumedetect
colorlevels             null                    vstack
compand                 nullsink                w3fdif
concat                  nullsrc                 waveform
copy                    overlay                 xbr
crop                    pad                     yadif
curves                  palettegen              zoompan
dcshift

Enabled bsfs:
aac_adtstoasc           imx_dump_header         mpeg4_unpack_bframes
chomp                   mjpeg2jpeg              noise
dump_extradata          mjpega_dump_header      remove_extradata
h264_mp4toannexb        mov2textsub             text2movsub
hevc_mp4toannexb        mp3_header_decompress

Enabled indevs:
dv1394                  lavfi                   v4l2
fbdev                   oss

Enabled outdevs:
fbdev                   oss                     v4l2

License: LGPL version 2.1 or later
Creating config.mak, config.h, and doc/config.texi...
[root@conv11 ffmpeg-2.8.14]# make -j 8
CC      libavdevice/oss.o
CC      libavdevice/alldevices.o
CC      libavdevice/avdevice.o
CC      libavdevice/lavfi.o
CC      libavdevice/dv1394.o
....
....
....
LD      ffmpeg_g
LD      ffprobe_g
LD      ffserver_g
CP      ffserver
STRIP   ffserver
CP      ffprobe
CP      ffmpeg
STRIP   ffprobe
STRIP   ffmpeg
[root@conv11 ffmpeg-2.8.14]# make install
INSTALL doc/ffmpeg.1
INSTALL doc/ffprobe.1
INSTALL doc/ffserver.1
INSTALL doc/ffmpeg-all.1
....
....
....
INSTALL libavutil/tea.h
INSTALL libavutil/lzo.h
INSTALL libavutil/avconfig.h
INSTALL libavutil/ffversion.h
INSTALL libavutil/libavutil.pc
[root@conv11 ffmpeg-2.8.14]#

And now you have a ffmpeg installation build against Intel API libmfx dispatcher, which could benefit from the hardware acceleration in decoding and encoding video streams. Thus using a Intel QuickSync technology.

STEP 9) ffmpeg example – how to use the GPU hardware decoding and encoding

[root@conv11 ffmpeg-2.8.14]# cd
[root@conv11 ~]# wget http://distribution.bbb3d.renderfarming.net/video/mp4/bbb_sunflower_1080p_60fps_stereo_abl.mp4
[root@conv11 ~]# /usr/local/ffmpeg-qsv/bin/ffmpeg -vcodec h264_qsv -i bbb_sunflower_1080p_60fps_stereo_abl.mp4 -vcodec h264_qsv -b:v 600k -maxrate 1200K -s 1280x720 -aspect 16:9 -preset medium -maxrate 500K -g 50 -strict -2 -c:a aac -b:a 96k -ar 44100 -f mpegts -threads 0 output.mp4
ffmpeg version 2.8.14 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-16)
  configuration: --enable-libmfx --prefix=/usr/local/ffmpeg-qsv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'bbb_sunflower_1080p_60fps_stereo_abl.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2013-12-16 18:03:30
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    composer        : Sacha Goedegebure
  Duration: 00:10:34.53, start: 0.000000, bitrate: 6490 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), nv12, 1920x2160, 6005 kb/s, 60 fps, 60 tbr, 60k tbn, 60k tbc (default)
    Metadata:
      creation_time   : 2013-12-16 18:03:30
      handler_name    : GPAC ISO Video Handler
    Stream #0:1(und): Audio: mp3 (mp4a / 0x6134706D), 48000 Hz, stereo, s16p, 160 kb/s (default)
    Metadata:
      creation_time   : 2013-12-16 18:03:37
      handler_name    : GPAC ISO Audio Handler
    Stream #0:2(und): Audio: ac3 (ac-3 / 0x332D6361), 48000 Hz, 5.1(side), fltp, 320 kb/s (default)
    Metadata:
      creation_time   : 2013-12-16 18:03:37
      handler_name    : GPAC ISO Audio Handler
    Side data:
      audio service type: main
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
Output #0, mpegts, to 'output.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    composer        : Sacha Goedegebure
    title           : Big Buck Bunny, Sunflower version
    artist          : Blender Foundation 2008, Janus Bager Kristensen 2013
    comment         : Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
    genre           : Animation
    encoder         : Lavf56.40.101
    Stream #0:0(und): Video: h264 (h264_qsv), nv12, 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 600 kb/s, 60 fps, 90k tbn, 60 tbc (default)
    Metadata:
      creation_time   : 2013-12-16 18:03:30
      handler_name    : GPAC ISO Video Handler
      encoder         : Lavc56.60.100 h264_qsv
    Stream #0:1(und): Audio: aac, 44100 Hz, 5.1(side), fltp, 96 kb/s (default)
    Metadata:
      creation_time   : 2013-12-16 18:03:37
      handler_name    : GPAC ISO Audio Handler
      encoder         : Lavc56.60.100 aac
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_qsv) -> h264 (h264_qsv))
  Stream #0:2 -> #0:1 (ac3 (native) -> aac (native))
Press [q] to stop, [?] for help
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[h264_qsv @ 0x3587360] A decode call did not consume any dataitrate=1162.5kbits/s    
    Last message repeated 14 times
frame=38072 fps=126 q=-0.0 Lsize=   90023kB time=00:10:34.48 bitrate=1162.3kbits/s    
video:71572kB audio:7436kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 13.942102%
[root@conv11 ~]# /usr/local/ffmpeg-qsv/bin/ffmpeg -i output.mp4
ffmpeg version 2.8.14 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-16)
  configuration: --enable-libmfx --prefix=/usr/local/ffmpeg-qsv
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
Input #0, mpegts, from 'output.mp4':
  Duration: 00:10:34.57, start: 1.400000, bitrate: 1162 kb/s
  Program 1 
    Metadata:
      service_name    : Big Buck Bunny, Sunflower version
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 90k tbn, 120 tbc
    Stream #0:1[0x101](und): Audio: aac (LC) ([15][0][0][0] / 0x000F), 44100 Hz, 5.1, fltp, 100 kb/s

You can see ffmpeg is using VA-API and /opt/intel/mediasdk/lib64/iHD_drv_video.so for video acceleration encoding.

* Live stream decoding and encoding with hardware acceleration and outputting multiple output

The input stream is from a multicast source and the output is a wowza server

/usr/local/ffmpeg-qsv/bin/ffmpeg -vcodec h264_qsv -i 'udp://239.10.10.1:5000?fifo_size=1000000&overrun_nonfatal=1' -filter_complex '[0:v]yadif=1:0:0,split=3[v1][v2][v3];[v1]scale=1280:720,framerate=50[out1];[v2]scale=960:540,framerate=25[out2];[v3]scale=640:360,framerate=25[out3]' -map '[out1]' -map '[out2]' -map '[out3]' -map 0:a -streamid 0:0x151 -streamid 1:0x150 -streamid 2:0x149 -streamid 3:0x140 -vcodec h264_qsv -preset medium -maxrate:0 1900K -bufsize:0 2400K -b:v:0 1500k -maxrate:1 1200K -bufsize:1 1000K -b:v:1 1000k -maxrate:2 900K -bufsize:2 1000K -b:v:2 600k -g 50 -strict -2 -c:a aac -b:a 96k -ar 44100 -reorder_queue_size 32 -threads 8 -fflags +igndts -copyts -f mpegts 'udp://10.10.10.10:30000?pkt_size=2632&buffer_size=20000000&fifo_size=2000000'
ffmpeg version 2.8.14 Copyright (c) 2000-2018 the FFmpeg developers
  built with gcc 4.8.5 (GCC) 20150623 (Red Hat 4.8.5-16)
  configuration: --enable-libmfx --prefix=/usr/local/ffmpeg-qsv3
  libavutil      54. 31.100 / 54. 31.100
  libavcodec     56. 60.100 / 56. 60.100
  libavformat    56. 40.101 / 56. 40.101
  libavdevice    56.  4.100 / 56.  4.100
  libavfilter     5. 40.101 /  5. 40.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  2.101 /  1.  2.101
[h264_qsv @ 0x21e6960] non-existing PPS 0 referenced
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
    Last message repeated 1 times
Input #0, mpegts, from 'udp://239.10.10.33:5000?fifo_size=1000000&overrun_nonfatal=1':
  Duration: N/A, start: 1375.213422, bitrate: N/A
  Program 74 
    Metadata:
      service_name    : Test 1 HD
      service_provider: Test
    Stream #0:0[0x734]: Video: h264 (High) ([27][0][0][0] / 0x001B), nv12, 1920x1080, 25 fps, 25 tbr, 90k tbn, 90k tbc
    Stream #0:1[0x744]: Audio: mp2 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 256 kb/s
[Parsed_framerate_7 @ 0x2413620] time base:1/180000 -> 1/180000 exact:1
[Parsed_framerate_7 @ 0x2413620] fps -> fps:25/1 scene score:7.000000 interpolate start:15 end:240
[Parsed_framerate_5 @ 0x2438d20] time base:1/180000 -> 1/180000 exact:1
[Parsed_framerate_5 @ 0x2438d20] fps -> fps:25/1 scene score:7.000000 interpolate start:15 end:240
[Parsed_framerate_3 @ 0x2934640] time base:1/180000 -> 1/180000 exact:1
[Parsed_framerate_3 @ 0x2934640] fps -> fps:50/1 scene score:7.000000 interpolate start:15 end:240
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
Output #0, mpegts, to 'udp://10.10.10.10:30000?pkt_size=2632&buffer_size=20000000&fifo_size=2000000':
  Metadata:
    encoder         : Lavf56.40.101
    Stream #0:0: Video: h264 (h264_qsv), nv12, 1280x720, q=2-31, 1500 kb/s, 50 fps, 90k tbn, 50 tbc
    Metadata:
      encoder         : Lavc56.60.100 h264_qsv
    Stream #0:1: Video: h264 (h264_qsv), nv12, 960x540, q=2-31, 1000 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.60.100 h264_qsv
    Stream #0:2: Video: h264 (h264_qsv), nv12, 640x360, q=2-31, 600 kb/s, 25 fps, 90k tbn, 25 tbc
    Metadata:
      encoder         : Lavc56.60.100 h264_qsv
    Stream #0:3: Audio: aac, 44100 Hz, stereo, fltp, 96 kb/s
    Metadata:
      encoder         : Lavc56.60.100 aac
Stream mapping:
  Stream #0:0 (h264_qsv) -> yadif (graph 0)
  framerate (graph 0) -> Stream #0:0 (h264_qsv)
  framerate (graph 0) -> Stream #0:1 (h264_qsv)
  framerate (graph 0) -> Stream #0:2 (h264_qsv)
  Stream #0:1 -> #0:3 (mp2 (native) -> aac (native))
Press [q] to stop, [?] for help
libva info: VA-API version 1.0.0
libva info: va_getDriverName() returns 0
libva info: User requested driver 'iHD'
libva info: Trying to open /opt/intel/mediasdk/lib64/iHD_drv_video.so
libva info: Found init function __vaDriverInit_1_0
libva info: va_openDriver() returns 0
[h264_qsv @ 0x21ea740] A decode call did not consume any data=00:23:14.04 bitrate=  46.6kbits/s    
frame=  937 fps= 59 q=-0.0 Lq=-0.0 q=-0.0 size=    8267kB time=00:23:14.57 bitrate=  48.6kbits/s    
video:7225kB audio:228kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 10.923792%

* Troubleshooting

If you get a build error like this, you skipped the (STEP 6) Move the Media SDK include files in a mfx sub-directory. This is required because ffmpeg expect these files to be in a sub-directory. Just move the files in a sub-directory mfx as it is shown in (STEP 6)

gcc -Wl,-O1 -Wl,--as-needed -Wl,-z,noexecstack -o /tmp/ffconf.7IhNLjmP/test /tmp/ffconf.7IhNLjmP/test.o -lm -ldl -llzma -lz -pthread -pthread
use_pkg_config libmfx libmfx mfx/mfxvideo.h MFXInit
check_pkg_config libmfx libmfx mfx/mfxvideo.h MFXInit
pkg-config --exists --print-errors libmfx
check_func_headers mfx/mfxvideo.h MFXInit -I/opt/intel/mediasdk/include -I/usr/include/libdrm -L/opt/intel/mediasdk/lib/lin_x64 -lmfx -lva -lstdc++ -ldl -lva-drm -ldrm
check_ld cc -I/opt/intel/mediasdk/include -I/usr/include/libdrm -L/opt/intel/mediasdk/lib/lin_x64 -lmfx -lva -lstdc++ -ldl -lva-drm -ldrm
check_cc -I/opt/intel/mediasdk/include -I/usr/include/libdrm -L/opt/intel/mediasdk/lib/lin_x64
BEGIN /tmp/ffconf.7IhNLjmP/test.c
    1   #include <mfx/mfxvideo.h>
    2   #include <stdint.h>
    3   long check_MFXInit(void) { return (long) MFXInit; }
    4   int main(void) { int ret = 0;
    5    ret |= ((intptr_t)check_MFXInit) & 0xFFFF;
    6   return ret; }
END /tmp/ffconf.7IhNLjmP/test.c
gcc -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -O2 -march=native -msse3 -fomit-frame-pointer -pipe -std=c11 -fomit-frame-pointer -pthread -I/opt/intel/mediasdk/include -I/usr/include/libdrm -L/opt/intel/mediasdk/lib/lin_x64 -c -o /tmp/ffconf.7IhNLjmP/test.o /tmp/ffconf.7IhNLjmP/test.c
/tmp/ffconf.7IhNLjmP/test.c:1:26: fatal error: mfx/mfxvideo.h: No such file or directory
 #include <mfx/mfxvideo.h>
                          ^
compilation terminated.
require libmfx mfx/mfxvideo.h MFXInit -llibmfx
check_lib libmfx mfx/mfxvideo.h MFXInit -llibmfx
check_func_headers mfx/mfxvideo.h MFXInit -llibmfx
check_ld cc -llibmfx
check_cc
BEGIN /tmp/ffconf.7IhNLjmP/test.c
    1   #include <mfx/mfxvideo.h>
    2   #include <stdint.h>
    3   long check_MFXInit(void) { return (long) MFXInit; }
    4   int main(void) { int ret = 0;
    5    ret |= ((intptr_t)check_MFXInit) & 0xFFFF;
    6   return ret; }
END /tmp/ffconf.7IhNLjmP/test.c
gcc -D_ISOC99_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_POSIX_C_SOURCE=200112 -D_XOPEN_SOURCE=600 -O2 -march=native -msse3 -fomit-frame-pointer -pipe -std=c11 -fomit-frame-pointer -pthread -c -o /tmp/ffconf.7IhNLjmP/test.o /tmp/ffconf.7IhNLjmP/test.c
/tmp/ffconf.7IhNLjmP/test.c:1:26: fatal error: mfx/mfxvideo.h: No such file or directory
 #include <mfx/mfxvideo.h>
                          ^
compilation terminated.
ERROR: libmfx not found

bacula – Device not in SD Device resources or no matching Media Type

If you happen to have such error in your bacula backup server:

20-Jun 00:19 bkp-srv-dir JobId 23741: No prior Full backup Job record found.
20-Jun 00:19 bkp-srv-dir JobId 23741: No prior or suitable Full backup found in catalog. Doing FULL backup.
20-Jun 00:19 bkp-srv-dir JobId 23741: Start Backup JobId 23741, Job=srv1-img.2018-06-20_00.19.09_56
20-Jun 00:19 bkp-srv-sd JobId 23741: Fatal error: Device reservation failed for JobId=23741: 
20-Jun 00:19 bkp-srv-dir JobId 23741: Fatal error: 
     Storage daemon didn't accept Device "srv1-img" because:
     3924 Device "srv1-img" not in SD Device resources or no matching Media Type.
20-Jun 00:19 bkp-srv-dir JobId 23741: Error: Bacula bkp-srv-dir 7.0.5 (28Jul14):
  Build OS:               x86_64-pc-linux-gnu ubuntu 16.04
  JobId:                  23741
  Job:                    srv1-img.2018-06-20_00.19.09_56
  Backup Level:           Full (upgraded from Incremental)
  Client:                 "srv1-img" 7.4.0 (16Jan16) x86_64-pc-linux-gnu,ubuntu,12.04
  FileSet:                "srv1-img" 2018-04-13 23:05:00
  Pool:                   "srv1-img-full" (From Job FullPool override)
  Catalog:                "DefaultCatalog" (From Client resource)
  Storage:                "srv1-img" (From Job resource)
  Scheduled time:         20-Jun-2018 00:19:06
  Start time:             20-Jun-2018 00:19:11
  End time:               20-Jun-2018 00:19:11
  Elapsed time:           0 secs
  Priority:               10
  FD Files Written:       0
  SD Files Written:       0
  FD Bytes Written:       0 (0 B)
  SD Bytes Written:       0 (0 B)
  Rate:                   0.0 KB/s
  Software Compression:   None
  VSS:                    no
  Encryption:             no
  Accurate:               no
  Volume name(s):         
  Volume Session Id:      3105
  Volume Session Time:    1521814321
  Last Volume Bytes:      0 (0 B)
  Non-fatal FD errors:    1
  SD Errors:              0
  FD termination status:  
  SD termination status:  
  Termination:            *** Backup Error ***

First thing to check is the status of Storage Devices you have with bconsole:

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*status storage
Automatically selected Storage: bkp-srv
Connecting to Storage daemon bkp-srv at 78.140.166.26:9103

bkp-srv-sd Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
Daemon started 20-Jun-18 00:43. Jobs: run=1, running=0.
 Heap: heap=135,168 smbytes=302,033 max_bytes=478,209 bufs=243 max_bufs=262
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
===================================================================
 23736  Incr          3    383.9 K  OK       19-Jun-18 14:05 srv1-dns
 23737  Incr          3    859.9 K  OK       19-Jun-18 14:05 srv2-dns
 23738  Incr        485    301.8 M  OK       19-Jun-18 14:05 srv3-dns
 23739  Incr         12    91.44 M  OK       19-Jun-18 14:05 srv4-dns
 23740  Full          1    1.358 G  OK       19-Jun-18 14:14 BackupCatalog
 23741  Full          0         0   Error    20-Jun-18 14:19 srv1-img
====

Device status:

Device "localstorage" (/mnt/storage/backups/localstorage) is not open.
==

Device "srv1-dns" (/mnt/storage/backups/srv1-dns) is not open.
==

Device "srv2-dns" (/mnt/storage/backups/srv2-dns) is not open.
==

Device "srv3-dns" (/mnt/storage/backups/srv3-dns) is not open.
==

Device "srv4-dns" (/mnt/storage/backups/srv4-dns) is not open.
==

Device "bkp-srv" (/mnt/storage/backups/bkp-srv) is not open.
==
====

Used Volume status:
====

====

As you can see there is no storage Device with name “srv1-img” as it expected from the error above.
You should check two files:

bacula-dir.conf

and

bacula-sd.conf

In file “/etc/bacula/bacula-dir.conf” you should have section like this:

Storage {
  Name = srv1-img
  Address = 10.10.10.10
  SDPort = 9103
  Password = mysecretpass
  Device = srv1-img
  Media Type = File
  Maximum Concurrent Jobs = 10
}

And in file “/etc/bacula/bacula-sd.conf”:

Device {
   Name = srv1-img
   Media Type = File
   Archive Device = /mnt/storage/backups/srv1-img
   LabelMedia = yes;
   Random Access = Yes;
   AutomaticMount = yes;
   RemovableMedia = no;
   AlwaysOpen = no;
 }

The important parts for this particular error (“3924 Device “srv1-img” not in SD Device resources or no matching Media Type.”) are highlighted:

  1. “Name” – the name of the storage
  2. “Media Type” – the type of the storage

You may also check the path of the storage – /mnt/storage/backups/srv1-img.

The two parameters must be the same! And if you verify these two configurations exist in the files and the above two parameters are the same and still get the error you MUST restart your SD (storage device) daemon, a “reload” command from the bconsole won’t work!

bkp-srv bacula # systemctl restart bacula-sd.service
bkp-srv bacula # systemctl status bacula-sd.service
* bacula-sd.service - LSB: Start Bacula Storage Daemon at boot time
   Loaded: loaded (/etc/init.d/bacula-sd; bad; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-20 00:43:22 UTC; 4s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 25249 ExecStop=/etc/init.d/bacula-sd stop (code=exited, status=0/SUCCESS)
  Process: 25259 ExecStart=/etc/init.d/bacula-sd start (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/bacula-sd.service
           `-25269 /usr/sbin/bacula-sd -c /etc/bacula/bacula-sd.conf -u bacula -g tape

Jun 20 00:43:22 bkp-srv systemd[1]: Starting LSB: Start Bacula Storage Daemon at boot time...
Jun 20 00:43:22 bkp-srv bacula-sd[25259]:  * Starting Bacula Storage daemon... bacula-sd
Jun 20 00:43:22 bkp-srv bacula-sd[25259]:    ...done.
Jun 20 00:43:22 bkp-srv systemd[1]: Started LSB: Start Bacula Storage Daemon at boot time.

And verify it restarted successfully.

Then you the storage device named “srv1-img” should appear in bconsole status storage command:
First thing to check is the status of Storage Devices you have with bconsole:

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*status storage
Automatically selected Storage: bkp-srv
Connecting to Storage daemon bkp-srv at 78.140.166.26:9103

bkp-srv-sd Version: 7.0.5 (28 July 2014) x86_64-pc-linux-gnu ubuntu 16.04
Daemon started 20-Jun-18 00:43. Jobs: run=1, running=0.
 Heap: heap=135,168 smbytes=302,033 max_bytes=478,209 bufs=243 max_bufs=262
 Sizes: boffset_t=8 size_t=8 int32_t=4 int64_t=8 mode=0,0

Running Jobs:
No Jobs running.
====

Jobs waiting to reserve a drive:
====

Terminated Jobs:
 JobId  Level    Files      Bytes   Status   Finished        Name 
===================================================================
 23736  Incr          3    383.9 K  OK       19-Jun-18 14:05 srv1-dns
 23737  Incr          3    859.9 K  OK       19-Jun-18 14:05 srv2-dns
 23738  Incr        485    301.8 M  OK       19-Jun-18 14:05 srv3-dns
 23739  Incr         12    91.44 M  OK       19-Jun-18 14:05 srv4-dns
 23740  Full          1    1.358 G  OK       19-Jun-18 14:14 BackupCatalog
 23741  Full          0         0   Error    20-Jun-18 14:19 srv1-img
====

Device status:

Device "localstorage" (/mnt/storage/backups/localstorage) is not open.
==

Device "srv1-dns" (/mnt/storage/backups/srv1-dns) is not open.
==

Device "srv2-dns" (/mnt/storage/backups/srv2-dns) is not open.
==

Device "srv3-dns" (/mnt/storage/backups/srv3-dns) is not open.
==

Device "srv4-dns" (/mnt/storage/backups/srv4-dns) is not open.
==

Device "bkp-srv" (/mnt/storage/backups/bkp-srv) is not open.
==

Device "srv1-img" (/mnt/storage/backups/srv1-img) is not open.
==
====

Used Volume status:
====

====

Extras – rerun the job and verify the backup was executed.

bkp-srv bacula # bconsole 
Connecting to Director localhost:9101
1000 OK: 1 bkp-srv-dir Version: 7.0.5 (28 July 2014)
Enter a period to cancel a command.
*run
Automatically selected Catalog: DefaultCatalog
Using Catalog "DefaultCatalog"
A job name must be specified.
The defined Job resources are:
     1: BackupCatalog
     2: srv1-dns
     3: srv2-dns
     4: srv3-dns
     5: srv4-dns
     6: srv1-img
Select Job resource (1-33): 6
Run Backup job
JobName:  srv1-img
Level:    Incremental
Client:   srv1-img
FileSet:  srv1-img
Pool:     Default (From Job resource)
Storage:  srv1-img (From Job resource)
When:     2018-06-20 00:43:59
Priority: 10
OK to run? (yes/mod/no): yes
Job queued. JobId=23742
You have messages.
*messages 
20-Jun 00:44 bkp-srv-dir JobId 23742: No prior Full backup Job record found.
20-Jun 00:44 bkp-srv-dir JobId 23742: No prior or suitable Full backup found in catalog. Doing FULL backup.
20-Jun 00:44 bkp-srv-dir JobId 23742: Start Backup JobId 23742, Job=srv1-img.2018-06-20_00.44.01_00
20-Jun 00:44 bkp-srv-dir JobId 23742: Created new Volume="srv1-img-full0394", Pool="srv1-img-full", MediaType="File" in catalog.
20-Jun 00:44 bkp-srv-dir JobId 23742: Using Device "srv1-img" to write.
20-Jun 00:44 bkp-srv-sd JobId 23742: Labeled new Volume "srv1-img-full0394" on file device "srv1-img" (/mnt/storage/backups/srv1-img).
20-Jun 00:44 bkp-srv-sd JobId 23742: Wrote label to prelabeled Volume "srv1-img-full0394" on file device "srv1-img" (/mnt/storage/backups/srv1-img)
*messages
20-Jun 00:44 bkp-srv-dir JobId 23742: Volume used once. Marking Volume "srv1-img-full0394" as Used.
*messages
You have no messages.

sysbench – FATAL: Too large position discovered in request

Here is a quick fix, which could take you a long time solving it even with Internet, because not so much information on the topic!
Let’s you want to run sysbench to check the IO of an SSD or NVME or your old hard drive? You run something like:

sysbench --test=fileio --file-total-size=4G --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd prepare
sysbench --test=fileio --file-total-size=4G --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd run
sysbench --test=fileio --file-total-size=4G --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd cleanup

Total of 4G files split in 128000 files and each file should be 32K seems OK at first, but you got:

FATAL: Too large position discovered in request

And the solution is simple enough:

file-total-size = file-num * file-block-size

And put the file-total-size in the units you used for file-block-size to be sure there are no rounding!
To put in simple words:

the size of file data should evenly divide by the number of files

So the above will transform to:

ssysbench --test=fileio --file-total-size=4096000K --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd prepare
ssysbench --test=fileio --file-total-size=4096000K --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd run
ssysbench --test=fileio --file-total-size=4096000K --file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 --max-time=300 --file-extra-flags=direct --file-test-mode=rndrd cleanup

Here are the computes:

–file-total-size=4096000K = –file-num=128000 * –file-block-size=32K

Simple equation could save you a lot of time!

* Good run – no errors!

srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096000K \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

128000 files, 32Kb each, 4000Mb total
Creating files for the test...
srv@local:/disks/1/test# echo 3 > /proc/sys/vm/drop_caches
srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096000K \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 100

Extra file open flags: 16384
128000 files, 32Kb each
3.9062Gb total file size
Block size 32Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
Time limit exceeded, exiting...
(last message repeated 99 times)
Done.

Operations performed:  4605591 Read, 0 Write, 0 Other = 4605591 Total
Read 140.55Gb  Written 0b  Total transferred 140.55Gb  (479.74Mb/sec)
15351.63 Requests/sec executed

Test execution summary:
    total time:                          300.0067s
    total number of events:              4605591
    total time taken by event execution: 29893.5378
    per-request statistics:
         min:                                  0.07ms
         avg:                                  6.49ms
         max:                                 40.43ms
         approx.  95 percentile:               7.06ms

Threads fairness:
    events (avg/stddev):           46055.9100/51.84
    execution time (avg/stddev):   298.9354/0.00


sysbench 0.4.12:  multi-threaded system evaluation benchmark

srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096000K \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd run
Removing test files...

* Bad run – with the error: FATAL: Too large position discovered in request

srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096M \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd prepare
sysbench 0.4.12:  multi-threaded system evaluation benchmark

128000 files, 32Kb each, 4096M total
Creating files for the test...
srv@local:/disks/1/test# echo 3 > /proc/sys/vm/drop_caches
srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096M \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 100

Extra file open flags: 16384
128000 files, 32.768Kb each
3.9999Gb total file size
Block size 32Kb
Number of random requests for random IO: 200000
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random read test
Threads started!
FATAL: Too large position discovered in request!
(last message repeated 30 times)
FATAL: Failed to read file! file: 80714 pos: 2 errno = 22 ()
FATAL: Too large position discovered in request!
(last message repeated 67 times)
Done.

Operations performed:  1 Read, 0 Write, 0 Other = 1 Total
Read 32Kb  Written 0b  Total transferred 32Kb  (114.71Kb/sec)
    3.58 Requests/sec executed

Test execution summary:
    total time:                          0.2790s
    total number of events:              1
    total time taken by event execution: 0.0038
    per-request statistics:
         min:                                  3.79ms
         avg:                                  3.79ms
         max:                                  3.79ms
         approx.  95 percentile:         10000000.00ms

Threads fairness:
    events (avg/stddev):           0.0100/0.10
    execution time (avg/stddev):   0.0000/0.00
srv@local:/disks/1/test# sysbench --test=fileio --file-total-size=4096M \
--file-block-size=32K --file-num=128000 --num-threads=100 --max-requests=0 \
--max-time=300 --file-extra-flags=direct --file-test-mode=rndrd cleanup
Removing test files...

mysql slave requested master to start replication from position greater than the file size

If you happen to reset your master mysql server without shutting down the mysql process (probably because of your super collocation cut the power!!!) you could have a slave server, which have the following error:

Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from position > file size; the first event 'mysql-bin.005173' at 644642725, the last event read from './mysql-bin.005173' at 4, the last byte read from './mysql-bin.005173' at 4.'

The slave server wants to read a position in the master, which does not exist probably because it was not committed to the file.

It is fairly easy (and in most cases safe) to just correct the position and restart the replication! Take the Master_Log_File and Read_Master_Log_Pos on the slave with:

[root@mysql1 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 50497
Server version: 5.6.38-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: 
                  Master_Host: 10.10.10.10
                  Master_User: replusr
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.005173
          Read_Master_Log_Pos: 644642725
               Relay_Log_File: mysqld-relay-bin.000479
                Relay_Log_Pos: 644642888
        Relay_Master_Log_File: mysql-bin.005173
             Slave_IO_Running: No
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 644642725
              Relay_Log_Space: 644643109
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Client requested master to start replication from position > file size; the first event 'mysql-bin.005173' at 644642725, the last event read from './mysql-bin.005173' at 4, the last byte read from './mysql-bin.005173' at 4.'
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: ce8a6c29-cf8e-11e5-9d39-000000000001
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for the slave I/O thread to update it
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 180525 16:27:22
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
1 row in set (0.00 sec)

As you can see the highlighted variables are important, that is the file and the position you should check in your master mysql server. So go to your master datadir (or the binlog directory, which could be changed with “binlog-dir”) and check whether there is Master_Log_File: “mysql-bin.005173” with position Read_Master_Log_Pos: “644642725”

srv@local-master ~ # cd /var/lib/mysql/
srv@local-master mysql # ls -altr mysql-bin.005173
-rw-rw---- 1 mysql mysql 644639026 24 may 12,22 mysql-bin.005173
srv@local-master mysql # mysqlbinlog mysql-bin.005173|tail -n 15
# at 644638722
#180524 12:03:31 server id 2  end_log_pos 644638930 CRC32 0xfeabe1ab    Query   thread_id=88263388      exec_time=0     error_code=0
SET TIMESTAMP=1527152611/*!*/;
UPDATE `group_desc` SET `id` = '153357',`name` = 'Test Group Name',`tags` = '|Test Group |' WHERE  `id` = '153357'
/*!*/;
# at 644638930
#180524 12:03:31 server id 2  end_log_pos 644639026 CRC32 0x5ca1b693    Query   thread_id=88263388      exec_time=0     error_code=0
SET TIMESTAMP=1527152611/*!*/;
COMMIT
/*!*/;
DELIMITER ;
# End of log file
ROLLBACK /* added by mysqlbinlog */;
/*!50003 SET COMPLETION_TYPE=@OLD_COMPLETION_TYPE*/;
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=0*/;
srv@local-master ~ # 

And yes, there is no position Read_Master_Log_Pos: “644642725” as the slave requested! The position number starts with

# at

for example

# at 644638930

As you can see from the bash commands above we got the last 15 lines of our binlog mysql file and the last position was 644638930. Here is what is going on: slave requested to continue from master’s position at 644642725, but master has last position 644638930:

644642725 > 644638930

To fix it just use the next binlog file and position 1 and your slave will continue normally. Let’s say there is a possibility your master missed to write the last commands to the master’s binlog because of the reset and in this situation your slave could be out of sync and in this case you should recover your slave from a full mysql dump and import in the slave. But in most cases it is fairly safe to continue.

[root@mysql1 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 50506
Server version: 5.6.38-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> STOP SLAVE;
Query OK, 0 rows affected (0.12 sec)

mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.005174',MASTER_LOG_POS=1;
Query OK, 0 rows affected (0.47 sec)

mysql> START SLAVE;
Query OK, 0 rows affected (0.04 sec)

And if everything is OK your slave will continue with no errors (just with a big delay – behind your master):

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.10.10.10
                  Master_User: replusr
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.005175
          Read_Master_Log_Pos: 328685690
               Relay_Log_File: mysqld-relay-bin.000003
                Relay_Log_Pos: 36999
        Relay_Master_Log_File: mysql-bin.005175
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 36836
              Relay_Log_Space: 334719008
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 103216
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 2
                  Master_UUID: ce8a6c29-cf8e-11e5-9d39-000000000001
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Repair by sorting
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
1 row in set (0.00 sec)

But if you get:

Duplicate entry

Last_SQL_Errno: 1062
Last_SQL_Error: Error 'Duplicate entry '422987' for key 'PRIMARY'' on query. Default database: 'mydb'. Query: 'INSERT INTO `spec_cookie` (`userid`, `userip`, `cookie`, `added`) VALUES ('96201', '2591115382', 'f3b81be45a484c652d38a2c70f8c44c30d4d04d1293918c9071e052ffd9c76f7', NOW())'

You might get into troubles if you continue, be careful!!! Examine the query, select the data in the slave and in the master, if they are equal you can skip it the error with:

[srv@local-slave ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 50506
Server version: 5.6.38-log MySQL Community Server (GPL)

Copyright (c) 2000, 2017, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> STOP SLAVE;
Query OK, 0 rows affected (0.09 sec)

mysql> SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1;
Query OK, 0 rows affected (0.00 sec)

mysql> START SLAVE;
Query OK, 0 rows affected (0.12 sec)

mysql>

If they are not equal you can change them manually and continue again. More about Duplicate entry you can check here (similar mysql binlog problems): mysql slave reset and fixing relay log read failure

Add a raw disk to a virtualbox virtual machine

This is strange there is no way to add a physical disk to your virtual machine under virtualbox! Still it is possible and it is simple, but you need to execute few commands under console so you need to open a terminal and to have a root privileges!
To use a raw disk in our virtual machine we must create a special VMDK file and then to use it when adding a hard drive to our virtualbox virtual machine with “Use an existing virtual hard disk dile”.

Here are the right steps to add a raw disk:

STEP 1) Permissions – your user must have (write) permissions to access raw disks

In most cases your user do not have write permission to the disk – it is the same under MS Windows and any Linux distribution (and probably MAC)! So here is the right way how to give permissions under Linux and Windows:

  • Under Linux (Ubuntu, Centos, Fedora, Gentoo and probably all other). There are at least two ways to give WRITE permission to the raw disk (use one of them, the first one is preferred). We want to add our third disk the “/dev/sdc” so the examples are with this device:
    1. add your user to “disk” group and log out. After you log in you’ll have WRITE permissions to the disks. Probably you must log off your GUI (Gnome, KDE and so on), too!
      srv@local ~ $ sudo sudo usermod -a -G disk myuser
      

      “myuser” is the username of the user I am logged in. The log out and log in (if you are using a GUI – gnome, kde or something else, you must log out from the GUI, too and then log in again).

    2. execute virtualbox with root user
    3. change the permission of the physical device you want to use (this is temporary, because next time you reboot you must change it again)
      srv@local ~ $ sudo chmod o+rw /dev/sdc 
      
  • Under Windows 10 (or 7) – you must start the command prompt and Virtualbox with “run as Administrator” – look at the next step (STEP 2).

Keep on reading!

mariadb mysql slave cannot stop and then crashes on start up with segmentation fault or abort

Let’s show you what means

“That’s specifics of InnoDB data storage.”

Here is the what this specific case will teach you:

  1. basic debug with strace
  2. MySQL could hangs in a infinite loop during a shutdown (could be seen with strace)
  3. InnoDB files could get corrupted without a hardware issue, but with a kill -KILL or probably it was corrupted before the shutdown and the kill?
  4. list the opened files (and the IDs = file descriptors) of a mysql process
  5. start a mysql slave without a starting the replication on start with “skip-slave-start”
  6. Remove a corrupted innodb file, which causes database (mysql process) crash leaving your with no database at all – replace a innodb file, start the server, then drop the innodb table with sql command and recreate it to continue using healthy table
  7. 2 rows (4 INTEGER columns) could eat up a lot of space – 12G and probably infinite! === “That’s specifics of InnoDB data storage.”
    This table file is 12 Gigabytes in size!!!

    MariaDB [mysql]> select * from gtid_slave_pos;
    +-----------+----------+-----------+-------------+
    | domain_id | sub_id   | server_id | seq_no      |
    +-----------+----------+-----------+-------------+
    |         0 | 16983943 |       101 | 45790450502 |
    |         0 | 16983944 |       101 | 45790450503 |
    +-----------+----------+-----------+-------------+
    2 rows in set (0.00 sec)
    

Here is presented a specific case with replication, but in your case you may not use replication, your problem table could be another (and your mariadb/mysql server crashes on start up or selecting from a specific table or on shutdown?), so find the problem table and remove it, here we show you how to do it! BACKUP the mysql datadir BEFORE any intervention!

Here we have a situation: a mariadb (mysql) server running as slave to a really busy master server, so you could expect 10 000 update/insert/delete queries. Everything was working till the time we wanted to shutdown the mysql process, which occurred to be impossible.

STEP 1) We tried everything from “systemctl stop mysql” to kill -TERM multiple times

5 hour the mysql process was running with 2000 opened file descriptors to multiple table files. The strace showed this:

[pid 14824] <... io_getevents resumed> []{0, 500000000}) = 0
[pid 14815] <... io_getevents resumed> []{0, 500000000}) = 0
[pid 14824] io_getevents(139723876253696, 1, 256,  <unfinished ...>
[pid 14815] io_getevents(139723876356096, 1, 256,  <unfinished ...>
[pid 14825] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 14825] futex(0x55d16885deb0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 14825] futex(0x55d16885dedc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3576261, {1525268239, 312230000}, ffffffff <unfinished ...>
[pid 14852] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 14852] futex(0x55d16885dd30, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 14852] futex(0x55d16885dd5c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 2775717, {1525268240, 308225000}, ffffffff <unfinished ...>
[pid 14825] <... futex resumed> )       = -1 ETIMEDOUT (Connection timed out)
[pid 14825] futex(0x55d16885deb0, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 14825] futex(0x55d16885dedc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 3576263, {1525268240, 304889000}, ffffffff <unfinished ...>
[pid 14862] <... nanosleep resumed> NULL) = 0
[pid 14862] nanosleep({1, 0},  <unfinished ...>
[pid 14821] <... io_getevents resumed> []{0, 500000000}) = 0
[pid 14821] io_getevents(139723876315136, 1, 256,  <unfinished ...>
[pid 14820] <... io_getevents resumed> []{0, 500000000}) = 0

And this has been repeating many times for hours without any disk activity on flushing any IO…so no use to wait for something, which apparently won’t finish at all.

STEP 2) So a

kill -9

was used to stop the mysql process. What could go wrong??? InnoDB is awesome and cannot corrupt if the hardware is OK, right? Yeahhh right…
When the start command was executed, the mysql process started, the innodb engine recovery completed successfully and after 5 minutes without listening socket and heavy IO reading there is the segmentation fault crash and YOU have no database….
So here is one of the crashes taken from the log:

2018-05-02 19:51:54 139990018914496 [ERROR] Plugin 'innodb' already installed
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Uses event mutexes
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Compressed tables use zlib 1.2.8
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Using Linux native AIO
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Number of pools: 1
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Using SSE2 crc32 instructions
2018-05-02 19:51:54 139990018914496 [Note] InnoDB: Initializing buffer pool, total size = 64G, instances = 8, chunk size = 128M
2018-05-02 19:51:56 139990018914496 [Note] InnoDB: Completed initialization of buffer pool
2018-05-02 19:51:56 139913709942528 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2018-05-02 19:51:56 139990018914496 [Note] InnoDB: Highest supported file format is Barracuda.
2018-05-02 19:51:56 139990018914496 [Note] InnoDB: Starting crash recovery from checkpoint LSN=311205983427362
2018-05-02 19:51:57 139990018914496 [Note] InnoDB: Last binlog file '/var/log/mysql-binlog/mysql-bin.227009', position 38590879
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: 128 out of 128 rollback segments are active.
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: Creating shared tablespace for temporary tables
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: Waiting for purge to start
2018-05-02 19:52:12 139990018914496 [Note] InnoDB: 5.7.21 started; log sequence number 311205983427371
2018-05-02 19:52:12 139915446597376 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2018-05-02 19:52:12 139990018914496 [Note] Plugin 'FEEDBACK' is disabled.
2018-05-02 19:52:12 139990018914496 [Note] Recovering after a crash using tc.log
2018-05-02 19:52:12 139990018914496 [Note] Starting crash recovery...
2018-05-02 19:52:12 139990018914496 [Note] Crash recovery finished.
2018-05-02 19:52:12 139990018914496 [Note] Server socket created on IP: '0.0.0.0'.
2018-05-02 19:52:12 139990018914496 [Warning] 'user' entry 'root@srvdns2' ignored in --skip-name-resolve mode.
2018-05-02 19:52:12 139990018914496 [Warning] 'proxies_priv' entry '@% root@srvdns1' ignored in --skip-name-resolve mode.
2018-05-02 19:53:52 139915446597376 [Note] InnoDB: Buffer pool(s) load completed at 180502 19:53:52
180502 19:57:12 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.2.13-MariaDB-10.2.13+maria~xenial
key_buffer_size=2147483648
read_buffer_size=262144
max_used_connections=0
max_threads=10002
thread_count=6
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 6871600 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7f40700009a8
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x7f515c4fecf8 thread_stack 0x49000
*** buffer overflow detected ***: /usr/sbin/mysqld terminated
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x777e5)[0x7f51f54eb7e5]
/lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x5c)[0x7f51f558d15c]
/lib/x86_64-linux-gnu/libc.so.6(+0x117160)[0x7f51f558b160]
/lib/x86_64-linux-gnu/libc.so.6(+0x1190a7)[0x7f51f558d0a7]
/usr/sbin/mysqld(my_addr_resolve+0xde)[0x55a9e9f1832e]
/usr/sbin/mysqld(my_print_stacktrace+0x1e2)[0x55a9e9eff2e2]
/usr/sbin/mysqld(handle_fatal_signal+0x345)[0x55a9e9999b95]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f51f5eda390]
/lib/x86_64-linux-gnu/libc.so.6(+0x9f849)[0x7f51f5513849]
/usr/sbin/mysqld(insert_dynamic+0x2a)[0x55a9e9ed3a4a]
/usr/sbin/mysqld(_Z25rpl_load_gtid_slave_stateP3THD+0x424)[0x55a9e98c43a4]
/usr/sbin/mysqld(handle_slave_background+0xe7)[0x55a9e97834e7]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f51f5ed06ba]
/lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f51f557b41d]
======= Memory map: ========
55a9e933e000-55a9ea423000 r-xp 00000000 08:03 148565                     /usr/sbin/mysqld
55a9ea622000-55a9ea6f4000 r--p 010e4000 08:03 148565                     /usr/sbin/mysqld
55a9ea6f4000-55a9ea7aa000 rw-p 011b6000 08:03 148565                     /usr/sbin/mysqld
55a9ea7aa000-55a9eb03d000 rw-p 00000000 00:00 0 
55a9eba46000-55aa4e67b000 rw-p 00000000 00:00 0                          [heap]

STEP 3) So the default way of repairing the InnoDB is to use

innodb_force_recovery

in your my.cnf configuration file:

[mysqld]
innodb_force_recovery=1

But again and again crashes even enabling all the options of innodb_force_recovery=1,2,3,4,5 and last 6. But when using “innodb_force_recovery=4” and 5 and 6 we have some strange additional error:

2018-05-02 21:43:34 139667439187712 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`gtid_slave_pos` in the cache. Attempting to load the tablespace with space id 63584
2018-05-02 21:43:34 139667439187712 [Warning] InnoDB: Allocated tablespace ID 63584 for mysql/gtid_slave_pos, old maximum was 0

The innodb_force_recovery did not help we still cannot start our MySQL database, but there were two things:

  1. No errors were reported by the InnoDB Engine – “InnoDB: Buffer pool(s) load completed” and Crash recovery always finished without errors
  2. The MySQL starts successfully, but not listening socket and then after 3~5 minutes of extensive IO reading from the disk by the mysql process it crashes

It was like something big was loading just on the start and in fact needed on the very beginning, so what start immediately after successful load of the mysql process?

REPLICATION

Remember this is a slave!

STEP 4) So trying to prevent the start of the replication on start of the mysql process with:

[mysqld]
skip-slave-start

STEP 5) strace and file descritor to find the offender table file.

The skip-slave-start option DID the trick – NO Segmentation fault

the mysql processes immediately after successful engines load began to listen on sockets. So the big offender was probably something connected with the replication! A big table used in the replication? How to find it, ok remove the “skip-slave-start”, then start the mysql process and wait for the time the IO read kicks in, then strace the mysql process (find the ID of the process with ps – the first number of the following command is the ID of the mysql process):

srv@local ~ # ps axuf|grep mysql|grep -v grep
mysql    6969  239  8.3 78561320 10983848 ?   Ssl  22:53 108:59 /usr/sbin/mysqld
srv@local ~ # strace -f -p 6969
strace -f -p 6969
strace: Process 6969 attached with 27 threads
[pid  6996] rt_sigtimedwait([HUP QUIT ALRM TERM TSTP], NULL, NULL, 8 <unfinished ...>
[pid  6994] futex(0x55941cb5d1ec, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid  6995] futex(0x7f93c6de1d24, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid  6993] restart_syscall(<... resuming interrupted nanosleep ...> <unfinished ...>
[pid  6992] futex(0x55941cb5d0ec, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid  6991] futex(0x55941cb5cd6c, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid  6990] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6997] mremap(0x7f9298b76000, 2210828288, 2210836480, MREMAP_MAYMOVE <unfinished ...>
[pid  6989] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6988] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6986] futex(0x55947cfe941c, FUTEX_WAIT_PRIVATE, 33, NULL <unfinished ...>
[pid  6985] futex(0x55947cfe941c, FUTEX_WAIT_PRIVATE, 31, NULL <unfinished ...>
[pid  6984] futex(0x55947cfe941c, FUTEX_WAIT_PRIVATE, 32, NULL <unfinished ...>
[pid  6997] <... mremap resumed> )      = 0x7f9298b76000
[pid  6984] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid  6983] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6984] futex(0x55947cfe941c, FUTEX_WAIT_PRIVATE, 33, NULL <unfinished ...>
[pid  6982] io_getevents(140347217702912, 1, 256,  <unfinished ...>
[pid  6981] io_getevents(140347217723392, 1, 256,  <unfinished ...>
[pid  6985] <... futex resumed> )       = -1 EAGAIN (Resource temporarily unavailable)
[pid  6980] io_getevents(140347217743872, 1, 256,  <unfinished ...>
[pid  6979] io_getevents(140347217764352, 1, 256,  <unfinished ...>
[pid  6978] io_getevents(140347217825792, 1, 256,  <unfinished ...>
[pid  6977] io_getevents(140347217846272, 1, 256,  <unfinished ...>
[pid  6985] futex(0x55947cfe941c, FUTEX_WAIT_PRIVATE, 33, NULL <unfinished ...>
[pid  6976] io_getevents(140347217866752, 1, 256,  <unfinished ...>
[pid  6975] io_getevents(140347219357696, 1, 256,  <unfinished ...>
[pid  6974] io_getevents(140347217784832, 1, 256,  <unfinished ...>
[pid  6973] io_getevents(140347217805312, 1, 256,  <unfinished ...>
[pid  6971] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6970] restart_syscall(<... resuming interrupted futex ...> <unfinished ...>
[pid  6969] futex(0x55941a4be944, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished ...>
[pid  6997] pread64(38, "\201L\321\365\0\5\315\3\0\5\315\0\0\5\315\6\0\1\30\307o\303\365]E\277\0\0\0\0\0\0"..., 16384, 6228590592) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210836480, 2210844672, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\210\271\277\343\0\5\315\6\0\5\315\3\0\5\315\7\0\1\30\307o\343\362~E\277\0\0\0\0\0\0"..., 16384, 6228639744) = 16384
[pid  6997] pread64(38, "\305\17\303k\0\5\315\7\0\5\315\6\0\5\315\10\0\1\30\307o\377\224gE\277\0\0\0\0\0\0"..., 16384, 6228656128) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210844672, 2210852864, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\217\351V~\0\5\315\10\0\5\315\7\0\5\315\n\0\1\30\307p\34\31\363E\277\0\0\0\0\0\0"..., 16384, 6228672512) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210852864, 2210861056, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\367\321\330\232\0\5\315\n\0\5\315\10\0\5\315\v\0\1\30\307p>\316\360E\277\0\0\0\0\0\0"..., 16384, 6228705280) = 16384
[pid  6997] pread64(38, "\374v\202\177\0\5\315\v\0\5\315\n\0\5\315\r\0\1\30\307p\\alE\277\0\0\0\0\0\0"..., 16384, 6228721664) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210861056, 2210869248, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "s\261\0A\0\5\315\r\0\5\315\v\0\5\315\16\0\1\30\307p\212\1AE\277\0\0\0\0\0\0"..., 16384, 6228754432) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210869248, 2210877440, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\214\t\7\244\0\5\315\16\0\5\315\r\0\5\315\21\0\1\30\307p\242H\37E\277\0\0\0\0\0\0"..., 16384, 6228770816) = 16384
[pid  6997] pread64(38, "<\n\272\"\0\5\315\21\0\5\315\16\0\5\315\24\0\1\30\307p\311i\313E\277\0\0\0\0\0\0"..., 16384, 6228819968) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210877440, 2210885632, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] mremap(0x7f9298b76000, 2210885632, 2210893824, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "n|I\n\0\5\315\24\0\5\315\21\0\5\315\25\0\1\30\307p\360EqE\277\0\0\0\0\0\0"..., 16384, 6228869120) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210893824, 2210902016, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\270\206\326\207\0\5\315\25\0\5\315\24\0\5\315\27\0\1\30\307q\1a\251E\277\0\0\0\0\0\0"..., 16384, 6228885504) = 16384
[pid  6997] pread64(38, "{l\226S\0\5\315\27\0\5\315\25\0\5\315\31\0\1\30\307q&\205!E\277\0\0\0\0\0\0"..., 16384, 6228918272) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210902016, 2210910208, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\201\265]&\0\5\315\31\0\5\315\27\0\5\315\32\0\1\30\307qK\365\212E\277\0\0\0\0\0\0"..., 16384, 6228951040) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210910208, 2210918400, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\352?\236\327\0\5\315\32\0\5\315\31\0\5\315\33\0\1\30\307qob:E\277\0\0\0\0\0\0"..., 16384, 6228967424) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210918400, 2210926592, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] mremap(0x7f9298b76000, 2210926592, 2210934784, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "3c\33\301\0\5\315\33\0\5\315\32\0\5\315\36\0\1\30\307q\2236%E\277\0\0\0\0\0\0"..., 16384, 6228983808) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210934784, 2210942976, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "Zi\343\177\0\5\315\36\0\5\315\33\0\5\315\37\0\1\30\307q\325\27LE\277\0\0\0\0\0\0"..., 16384, 6229032960) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210942976, 2210951168, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "v5h%\0\5\315\37\0\5\315\36\0\5\315!\0\1\30\307r\f\325\364E\277\0\0\0\0\0\0"..., 16384, 6229049344) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210951168, 2210959360, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "0\6Yn\0\5\315!\0\5\315\37\0\5\315\"\0\1\30\307sCX@E\277\0\0\0\0\0\0"..., 16384, 6229082112) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210959360, 2210967552, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\203\243K\203\0\5\315\"\0\5\315!\0\5\315$\0\1\30\307t;\234\302E\277\0\0\0\0\0\0"..., 16384, 6229098496) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210967552, 2210975744, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "9\264\332j\0\5\315$\0\5\315\"\0\5\315%\0\1\30\307\177\370#\371E\277\0\0\0\0\0\0"..., 16384, 6229131264) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210975744, 2210983936, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, ":\200+\337\0\5\315%\0\5\315$\0\5\315&\0\1\30\307\200\20U-E\277\0\0\0\0\0\0"..., 16384, 6229147648) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210983936, 2210992128, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\35-th\0\5\315&\0\5\315%\0\5\315(\0\1\30\307\200+\333CE\277\0\0\0\0\0\0"..., 16384, 6229164032) = 16384
[pid  6997] pread64(38, "\246Y\305\351\0\5\315(\0\5\315&\0\5\315)\0\1\30\307\200[\324\305E\277\0\0\0\0\0\0"..., 16384, 6229196800) = 16384
[pid  6997] mremap(0x7f9298b76000, 2210992128, 2211000320, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\340(\2\350\0\5\315)\0\5\315(\0\5\315+\0\1\30\307\200\214L\365E\277\0\0\0\0\0\0"..., 16384, 6229213184) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211000320, 2211008512, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "A\374\37\23\0\5\315+\0\5\315)\0\5\315,\0\1\30\307\200\252\30\373E\277\0\0\0\0\0\0"..., 16384, 6229245952) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211008512, 2211016704, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "iW\274\365\0\5\315,\0\5\315+\0\5\315.\0\1\30\307\200\332\266\256E\277\0\0\0\0\0\0"..., 16384, 6229262336) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211016704, 2211024896, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "\343c\340\347\0\5\315.\0\5\315,\0\5\315/\0\1\30\307\200\356\272YE\277\0\0\0\0\0\0"..., 16384, 6229295104) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211024896, 2211033088, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "}\35\32i\0\5\315/\0\5\315.\0\5\3150\0\1\30\307\201\5\226\256E\277\0\0\0\0\0\0"..., 16384, 6229311488) = 16384
[pid  6997] pread64(38, "\322\237\206\377\0\5\3150\0\5\315/\0\5\3151\0\1\30\307\201\257\241\272E\277\0\0\0\0\0\0"..., 16384, 6229327872) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211033088, 2211041280, MREMAP_MAYMOVE) = 0x7f9298b76000
[pid  6997] pread64(38, "/\250\21!\0\5\3151\0\5\3150\0\5\3152\0\1\30\307\203\3\237\2E\277\0\0\0\0\0\0"..., 16384, 6229344256) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211041280, 2211049472, MREMAP_MAYMOVE) = 0x7f9298b76000

You see multiple lines (thousands) with:

[pid  6997] pread64(38, "/\250\21!\0\5\3151\0\5\3150\0\5\3152\0\1\30\307\203\3\237\2E\277\0\0\0\0\0\0"..., 16384, 6229344256) = 16384
[pid  6997] mremap(0x7f9298b76000, 2211041280, 2211049472, MREMAP_MAYMOVE) = 0x7f9298b76000

The pread64 – reads from file descriptor with ID=38, then you can list all file descriptors for the mysql process ID with:

srv@local mysql # ls -altr /proc/6969/fd
total 0
dr-xr-xr-x 9 mysql mysql  0 May  2 21:32 ..
l-wx------ 1 root  root  64 May  2 21:35 2 -> /var/log/mysql/error.log
dr-x------ 2 root  root   0 May  2 21:35 .
lrwx------ 1 root  root  64 May  2 21:35 9 -> /tmp/ibgyw4AI (deleted)
lrwx------ 1 root  root  64 May  2 21:35 8 -> /tmp/ibuaprWP (deleted)
lrwx------ 1 root  root  64 May  2 21:35 7 -> /tmp/ibfXwsBW (deleted)
lrwx------ 1 root  root  64 May  2 21:35 6 -> /var/lib/mysql/ibdata1
lrwx------ 1 root  root  64 May  2 21:35 5 -> /var/lib/mysql/aria_log.00000001
lr-x------ 1 root  root  64 May  2 21:35 4 -> /var/lib/mysql
lrwx------ 1 root  root  64 May  2 21:35 38 -> /var/lib/mysql/mysql/gtid_slave_pos.ibd
lrwx------ 1 root  root  64 May  2 21:35 37 -> /var/lib/mysql/mysql/event.MYD
lrwx------ 1 root  root  64 May  2 21:35 36 -> /var/lib/mysql/mysql/event.MYI
lrwx------ 1 root  root  64 May  2 21:35 35 -> /var/lib/mysql/mysql/procs_priv.MYD
lrwx------ 1 root  root  64 May  2 21:35 34 -> /var/lib/mysql/mysql/procs_priv.MYI
lrwx------ 1 root  root  64 May  2 21:35 33 -> /var/lib/mysql/mysql/columns_priv.MYD
lrwx------ 1 root  root  64 May  2 21:35 32 -> /var/lib/mysql/mysql/columns_priv.MYI
lrwx------ 1 root  root  64 May  2 21:35 31 -> /var/lib/mysql/mysql/tables_priv.MYD
lrwx------ 1 root  root  64 May  2 21:35 30 -> /var/lib/mysql/mysql/tables_priv.MYI
lrwx------ 1 root  root  64 May  2 21:35 3 -> /var/lib/mysql/aria_log_control
lrwx------ 1 root  root  64 May  2 21:35 29 -> /var/lib/mysql/mysql/roles_mapping.MYD
lrwx------ 1 root  root  64 May  2 21:35 28 -> /var/lib/mysql/mysql/roles_mapping.MYI
lrwx------ 1 root  root  64 May  2 21:35 27 -> /var/lib/mysql/mysql/proxies_priv.MYD
lrwx------ 1 root  root  64 May  2 21:35 26 -> /var/lib/mysql/mysql/proxies_priv.MYI
lrwx------ 1 root  root  64 May  2 21:35 25 -> /var/lib/mysql/mysql/host.MYD
lrwx------ 1 root  root  64 May  2 21:35 24 -> /var/lib/mysql/mysql/host.MYI
lrwx------ 1 root  root  64 May  2 21:35 23 -> /var/lib/mysql/mysql/db.MYD
lrwx------ 1 root  root  64 May  2 21:35 22 -> /var/lib/mysql/mysql/db.MYI
lrwx------ 1 root  root  64 May  2 21:35 21 -> /var/lib/mysql/mysql/user.MYD
lrwx------ 1 root  root  64 May  2 21:35 20 -> /var/lib/mysql/mysql/user.MYI
lrwx------ 1 root  root  64 May  2 21:35 19 -> socket:[49519]
lrwx------ 1 root  root  64 May  2 21:35 18 -> socket:[49518]
lrwx------ 1 root  root  64 May  2 21:35 17 -> /var/lib/mysql/mysql/servers.MYD
lrwx------ 1 root  root  64 May  2 21:35 16 -> /var/lib/mysql/mysql/servers.MYI
lrwx------ 1 root  root  64 May  2 21:35 15 -> /var/lib/mysql/tc.log
lrwx------ 1 root  root  64 May  2 21:35 13 -> /tmp/ib281FZH (deleted)
lrwx------ 1 root  root  64 May  2 21:35 12 -> /var/lib/mysql/ibtmp1
lrwx------ 1 root  root  64 May  2 21:35 11 -> /var/lib/mysql/ib_logfile1
lrwx------ 1 root  root  64 May  2 21:35 10 -> /var/lib/mysql/ib_logfile0
l-wx------ 1 root  root  64 May  2 21:35 1 -> /var/log/mysql/error.log
lrwx------ 1 root  root  64 May  2 21:35 0 -> /dev/pts/3

And again file descriptor with ID=38 is:

/var/lib/mysql/mysql/gtid_slave_pos.ibd

And when you check the size – 12G…..12G for this table? Why? What is used for?

STEP 6) And from the manual:

The mysql.gtid_slave_pos table is used in replication by slave servers to keep track of their current position (the global transaction ID of the last transaction applied). Using the table allows the slave to maintain a consistent value for the gtid_slave_pos system variable across server restarts. See Global Transaction ID.

You should never attempt to modify the table directly. If you do need to change the global gtid_slave_pos value, use SET GLOBAL gtid_slave_pos = ... instead.

The table is updated with the new position as part of each transaction committed during replication. This makes it preferable that the table is using the same storage engine as the tables otherwise being modified in the transaction, since otherwise a multi-engine transaction is needed that can reduce performance.

Hmm “The table is updated with the new position as part of each transaction committed during replication” and 12G table – could it track minillions of transactions? Probably not, in fact this table has only 2-3-4 rows at a given time!!! But the size is 12G, well as we said this is:
“That’s specifics of InnoDB data storage. Did you try to run OPTIMIZE TABLE mysql.gtid_slave_pos? It should allow to reclaim the disk space.” (taken from: https://jira.mariadb.org/browse/MDEV-12318).

STEP 7) Remove a corrupted innodb file, which causes database (mysql process) crash leaving your with no database at all and then recreate the table

Restart your mysql server with “skip-slave-start” again to be able to start it (look above).
So a table probably with couple of rows takes 12G and MariaDB is using it for the replication on start up, but the replication position is kept on another place “master.info”, can we delete this offender file “gtid_slave_pos.ibd”? Yes we can, move the file out of its place (mysql datadir) and then create a second table with:

MariaDB [mysql]> use mysql
MariaDB [mysql]> CREATE TABLE `gtid_slave_pos1` (
    ->   `domain_id` int(10) unsigned NOT NULL,
    ->   `sub_id` bigint(20) unsigned NOT NULL,
    ->   `server_id` int(10) unsigned NOT NULL,
    ->   `seq_no` bigint(20) unsigned NOT NULL,
    ->   PRIMARY KEY (`domain_id`,`sub_id`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Replication slave GTID position';
Query OK, 0 rows affected (0.01 sec)

Copy gtid_slave_pos1.ibd to gtid_slave_pos.ibd and restart the mysql process, it will report an error for a table mysql.gtid_slave_pos but you will be able to drop the table and then create it with the same name: “gtid_slave_pos” (you could drop the temporary one “gtid_slave_pos1”)

MariaDB [mysql]> use mysql
MariaDB [mysql]> DROP TABLE `gtid_slave_pos`;
Query OK, 0 rows affected (0.01 sec)
MariaDB [mysql]> CREATE TABLE `gtid_slave_pos` (
    ->   `domain_id` int(10) unsigned NOT NULL,
    ->   `sub_id` bigint(20) unsigned NOT NULL,
    ->   `server_id` int(10) unsigned NOT NULL,
    ->   `seq_no` bigint(20) unsigned NOT NULL,
    ->   PRIMARY KEY (`domain_id`,`sub_id`)
    -> ) ENGINE=InnoDB DEFAULT CHARSET=latin1 COMMENT='Replication slave GTID position';
Query OK, 0 rows affected (0.01 sec)
MariaDB [mysql]> DROP TABLE `gtid_slave_pos1`;
Query OK, 0 rows affected (0.01 sec)

The create table statement is taken from another server probably it is a good idea to do it yourself, login on a healthy mysql server and issue: “show create table mysql.gtid_slave_pos;”.
So now you have a healthy mysql.gtid_slave_posto be used for your replication. Restart your mysql server removing “skip-slave-start” from your configuration file and here it is the replication is OK and running:

MariaDB [(none)]> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 10.10.10.10
                  Master_User: replusr
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.071301
          Read_Master_Log_Pos: 64980976
               Relay_Log_File: mysqld-relay-bin.230012
                Relay_Log_Pos: 80704376
        Relay_Master_Log_File: mysql-bin.090129
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: test
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 80704077
              Relay_Log_Space: 113209377078
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 30944
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 101
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
                   Using_Gtid: No
                  Gtid_IO_Pos: 
      Replicate_Do_Domain_Ids: 
  Replicate_Ignore_Domain_Ids: 
                Parallel_Mode: conservative
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Waiting for room in worker thread event queue
1 row in set (0.00 sec)

Review of netdata graphs – nginx, php-fpm, mysql, memcached, redis, mail (postfix)

Here we show what to expect from the netdata graphics when using it in a web server. So we included here only the specific graphs for a web server:

  1. nginx – the web server
  2. php-fpm – the application, fastcgi php
  3. mysql – the database server
  4. memcached – memory cache
  5. redis – more sophisticated memory/disk cache
  6. mail – postfix mail server to send and receive mails

You can also visit our review of the generic graphs like system overview, cpu, memory and disks here: Review of netdata graphs – system overview, cpu, memory, disks and nfs

So here are the graphs netdata 1.10 offers to us:

CHART 1) Nginx Graphs

1) all active connections; 2) requests per second to nginx

main menu
Nginx Graphs

CHART 2) Nginx Graphs 2

1) nginx active connections by their status – reading (from client), writing (from client), idle (doing nothing, but opened to the client); 2) connections rate – accepted and handled

main menu
Nginx Graphs 2

CHART 3) PHP-FPM – FastCGI PHP performance metrics

1) active connections – active (executing PHP code on the CPU right now – “php running”), max active, idle; 2) requests; 3) performance – max children reached or slow requests (it depends on your version of netdata).

main menu
PHP-FPM – FastCGI PHP performance metrics

CHART 4) PHP-FPM – request information

1) reuqest duration – minimum, maximum, avarage – how much time do a request take time – very useful to see how fast is your backend application. 2) request CPU in procentages; 3) request memory – reuested memory by your php fpm processes.

main menu
PHP-FPM – request information

CHART 5) MySQL – performance metrics

1) bandwidth – The amount of data sent to mysql clients (out) and received from mysql clients (in); 2) queries – The number of statements executed by the server. To see a slow queries the slow query log should be enabled.

main menu
MySQL – performance metrics

CHART 6) MySQL – handlers and locks

1) handlers – netdata Quotation: “Usage of the internal handlers of mysql. This chart provides very good insights of what the mysql server is actually doing. – commit, the number of internal COMMIT statements; delete, the number of times that rows have been deleted from tables; prepare, a counter for the prepare phase of two-phase commit operations; read first, the number of times the first entry in an index was read. A high value suggests that the server is doing a lot of full index scans; e.g. SELECT col1 FROM foo, with col1 indexed; read key, the number of requests to read a row based on a key. If this value is high, it is a good indication that your tables are properly indexed for your queries; read next, the number of requests to read the next row in key order. This value is incremented if you are querying an index column with a range constraint or if you are doing an index scan; read prev, the number of requests to read the previous row in key order. This read method is mainly used to optimize ORDER BY … DESC; read rnd, the number of requests to read a row based on a fixed position. A high value indicates you are doing a lot of queries that require sorting of the result. You probably have a lot of queries that require MySQL to scan entire tables or you have joins that do not use keys properly; read rnd next, the number of requests to read the next row in the data file. This value is high if you are doing a lot of table scans. Generally this suggests that your tables are not properly indexed or that your queries are not written to take advantage of the indexes you have; rollback, the number of requests for a storage engine to perform a rollback operation; savepoint, the number of requests for a storage engine to place a savepoint; savepoint rollback, the number of requests for a storage engine to roll back to a savepoint; update, the number of requests to update a row in a table; write, the number of requests to insert a row in a table.” 2) MySQL table locks counters, netdata Quotation: ” immediate, the number of times that a request for a table lock could be granted immediately – waited, the number of times that a request for a table lock could not be granted immediately and a wait was needed. If this is high and you have performance problems, you should first optimize your queries, and then either split your table or tables or use replication.”

main menu
MySQL – handlers and locks

CHART 7) MySQL – sorts, selects and temporaries

1) mysql SELECT JOIN – full range, range, scan; 2) mysql sorts – range and scan; 3) temporaries – disk tables (writing to the disk is slow and should be avoided!!!) and tables.

main menu
MySQL – sorts, selects and temporaries

CHART 8) MySQL – connections and binlog

1) connections in seconds – all and aborted – if you are using persistent connections to MySQL you can see a busy MySQL server could have 2-3 new connections in a minute, because all the application backend uses the pool of already opened connections to the server. 2) connection errors – accepted, internal, max, peer_addr, select, tcpwrap; 3) binlog transactions per second

main menu
MySQL – connections and binlog

CHART 9) MySQL – binlog and threads

1) binlog statement cache; 2) MySQL threads – connected, cached, running; 3) threads cache misses

main menu
MySQL – binlog and threads

CHART 10) MySQL – Innodb engine infromation

1) Innodb I/O bandwidth – reads and writes; 2) Innodb I/O Operations – reads, writes and fsyncs; 3) Innodb Pending I/O Operations – reads and fsyncs; 4) Innodb Log Operations – write requests and writes.

main menu
MySQL – Innodb engine infromation

CHART 11) MySQL – Innodb engine infromation 2

1) Innodb OS Log Operations – fsyncs; 2) Innodb OS Log bandwidth – write (megabytes/s); 3) Innodb current row locks – current_waits; 4) Innodb row operations – inserted, read, updated and deleted.

main menu
MySQL – Innodb engine infromation 2

CHART 12) MySQL – Innodb engine infromation 3

1) Innodb buffer pool pages – data, dirty, free, flushed, misc, total; 2) Innodb buffer pool bytes – data and dirty; 3) Innodb buffer pool read ahaed – all, evicted, random; 4) Innodb buffer pool requests – reads and writes per second.

main menu
MySQL – Innodb engine infromation 3

CHART 13) MySQL – Innodb engine infromation 4

1) Innodb buffer pool operations – disk reads – operations per second.

main menu
MySQL – Innodb engine infromation 4

CHART 14) MySQL – query cache (qcache)

1) query cache operations – hits, low memory prunes, inserts, not cached; 2) queries in the cache; 3) query cache free memory; 4) query cache memory blocks – free and total.

main menu
MySQL – query cache (qcache)

CHART 15) MySQL – myisam engine information

This server does not uses MyISAM engine, so you can see almost everything is zero – 1) MyISAM key cache blocks – unused and used; 2) MyISAM key cache requests – reads and writes; 3) MyISAM key cache disk operation – reads and writes.

main menu
MySQL – myisam engine information

CHART 16) MySQL – files

1) open files – how many files are opened at the moment; 2) opened file rate – files per second.

main menu
MySQL – files

CHART 17) Memcached – distributed memory caching system. A key-value memory storage.

1) cache size – available and used; 2) network – in and out megabytes per second.

main menu
Memcached – distributed memory caching system. A key-value memory storage.

CHART 18) Memcached – connections and items

1) connections – current and total. Persistent connections are used, so no new connections often; 2) items cached – current and total. 3) items – evicted (forced removed – be careful here, this means your cached items are forcedly removed by the server because of lack of memory?) and reclaims (expired items).

main menu
Memcached – connections and items

CHART 19) Memcached – get and set operations

1) get operation requests – hits and misses; 2) get operations rate – requests per second; 3) set operation requests – requests per second.

main menu
Memcached – get and set operations

CHART 20) Memcached – check and set ops, delete ops, increment ops

1) check and set operation requests – hits, misses, bad value; 2) delete operation requests – hits and misses; increment operation requests – hits and misses

main menu
Memcached – check and set ops, delete ops, increment ops

CHART 21) Memcached – decrement ops, touch ops

1) decrement operation request – hits and misses; 2) touch operation requests – hits and misses; 3) touch operation requests rate – requests per second.

main menu
Memcached – decrement ops, touch ops

CHART 22) Postfix – mail service

1) Postfix Queue Emails – the emails in the queue of the mail transfer agent, these mails are in transfer state; 2) Postfix Queue Emails size – size.

main menu
Postfix – mail service

CHART 23) Redis – performance metrics for in-memory data structure store, used as a database, cache and message broker.

1) operations – commands and operations per second; 2) hit rate – persentage, the effectiveness of the cache.

main menu
Redis – performance metrics for in-memory data structure store, used as a database, cache and message broker.

CHART 24) Redis – memory, keys, network

1) Redis memory utilization – total and lua; 2) keys – how many keys does each database have – keys per database name; 3) network – Redis network bandwidth – in and out in megabytes per second.

main menu
Redis – memory, keys, network

CHART 25) Redis – connections and replication

1) Redis connections – received per second – it’s like new connections and if you use persistent connections no new connections are opened often; 2) Redis clients – connected processes to the redis server; 3) replication – connected slave servers.

main menu
Redis – connections and replication

CHART 26) Redis – persistence (save the databases to the disks)

1) Persistence changes since last save – changes – how many changed items have been there since last save of the databases to the disks. 2) Duration of the RDB Save operation – rdb save in time; 3) Status of the last RDB Save Operation – rdb status.

main menu
Redis – persistence (save the databases to the disks)

CHART 27) Web server access logs information

Live parsing of the access logs – be careful here, because this could take a good deal of CPU and I/O of your busy server. Here we included only the default nginx log, which does not save many records. netdata Quotation: “Information extracted from a server log file. web_log plugin incrementally parses the server log file to provide, in real-time, a break down of key server performance metrics. For web servers, an extended log file format may optionally be used (for nginx and apache) offering timing information and bandwidth for both requests and responses. web_log plugin may also be configured to provide a break down of requests per URL pattern (check /etc/netdata/python.d/web_log.conf).” – 1) responses – success and bad requests per second; 2) Response codes – 1xx and 4xx and more if any in the logs.

main menu
Web server access logs information

CHART 28) Web server access logs information – detailed response code, bandwidth, http methods

1) detailed response code – requests per second; 2) bandwidth of the requests and reponses; 3) Requests per HTTP Method – GET, POST, PUT, DELETE and so on if they present in the logs.

main menu
Web server access logs information – detailed response code, bandwidth, http methods

CHART 29) Web server access logs information – http versions, ip protocols, clients

1) Requests per HTTP Version – 1.0, 1.1 and 2.0 if any in the logs; 2) Requests per IP protocol – IPv4 and IPv6 (if used); 3) clients – unique client IPs per data collection.

main menu
Web server access logs information – http versions, ip protocols, clients

CHART 30) Web server access logs information – unique client IPs

Unique client IPs since last restart of netdata

main menu
Web server access logs information – unique client IPs