mariadb (MySQL) stopped flushing relay-bin log after mysqld-relay-bin.999999

We have a pretty big MariaDB master-slave setup, which is really under heavy load and suddenly one of our slaves stopped purging relay logs! And soon the free space got less and it was clear were were going to run out of space for the data partition.

Here is the problem we have only 49G left free space for our mysql datadir “/var/lib/mysql-datadir”. You can see the replication is OK and running (it’s not behind – “Seconds_Behind_Master: 0”), the

Relay_Log_File says mysqld-relay-bin.999999

but it is not true! You can see there are tones of more relay logs after “mysqld-relay-bin.999999” – in fact the current MySQL relay binlog file is “mysqld-relay-bin.1071413” (look below we included a listing of the datadir).

And “FLUSH RELAY LOGS” does not help, at all!

It just not freed any byte on the partition and no files were removed after “mysqld-relay-bin.999999” as before.
Keep on reading!

apt-mark – upgrade with the exception of certain packages

If you are in a situation when you want to upgrade your system, but do not want to upgrade a certain software in it you can just instruct apt not to upgrade these packages with:

apt-mark hold <package name(s)>

Here is how you can block updating 4 packages – ca-certificates, firefox, ghostscript, linux-firmware. First we update and upgrade and you can see there is no packages to keep back, and then we use apt-mark to “hold” package “linux-firmware” and ca-certificates, firefox, ghostscript at once. Initiating apt upgrade again will give you “The following packages have been kept back:” and it will include all packages, which will not be upgraded (it will include dependencies, which require some of the blocked packages).
Keep on reading!

ssh remote command escape special characters when using variable

You should always enclose the command given to the ssh client for remote executing!

myuser@srv-local:~$ CMD="cat /etc/*release";ssh root@1.1.1.1 "$CMD"
Gentoo Base System release 2.0.3
myuser@srv-local:~$ CMD="cat /etc/*release";ssh root@1.1.1.1 $CMD
cat: /etc/lsb-release: No such file or directory
cat: /etc/os-release: No such file or directory
myuser@srv-remote:~$

You see the difference! The second line the special character “*” asteriks will be expanded by the shell locally and then the result will be send to the remote server for execution. In the second case the remote server will receive a command “cat /etc/lsb-release /etc/os-release” (because our local system has there two files) and not what you want “cat /etc/*release” on the remote.
We use variables above, because we want to point out

the problem, which often occurs when you use ssh remote command execution in a script.

Keep on reading!

Selinux permission denied and no log in audit.log

So you execute a script and get a “Permission denied” and you know you have enabled SELinux. OK to disable the selinux is not an option (and never will be), so the first thing to check is the audit log to see what is the error and what the selinux tools will offer to solve it.

But there are no entries in the audit log when you execute your script!

So you decide to temporarily disable the selinux to check if this permission denied issues is still caused by it with:

setenforce 0

And the script just executes fine no error! Then again you put back the Enforcing with:

setenforce 1
./myscript
Permission Denied

And NO added lines in audit.log (/var/log/audit/audit.log in our system!). Apparently the logging is just fine, because it got sometime entries, but when executing our script, which is just a simple:

 
find /mnt/storA/servers/webroots/

After some research it appeared that

not all AVC denials may be logged when SELinux denies access.

Too many applications and system libraries check for permissions, which might not use or even need after that and the logging could grow exponentially or be less informative for the real cause of a problem!
Keep on reading!

Install the new storcli to manage (LSI/AVAGO/Broadcom) MegaRAID controller under CentOS 7

After the acquisition of LSI there was a major change with the management console utility for the MegaRAID controllers. The utility was renamed from MegaCli (MegaCli64, megacli) to

storcli (storcli64)

We have new controllers like AVAGO MegaRAID SAS-9361-4i and really old ones like LSI 2108 MegaRAID (in fact Supermicro AOC-USAS2LP-H8iR) and the two controllers could be manage with the new cli. even the old controller, which is on more than 8 years could be manage by the new cli.
Interesting fact is that the storcli output and argument syntax and is almost identical to the one really old cli – tw_cli – the 3Ware management utility. As you know LSI bought 3ware RAID adapter business in 2009.
Keep on reading!

Access Violation error when compiling packets in Gentoo – symlink

Here is another example of Access violation error when building packages in Gentoo. This time the build process could not make a symbolic link in “/usr/bin” and the build process of the package failed with:

 * ACCESS DENIED:  symlink:      /usr/bin/stransmit
CMake Error: failed to create symbolic link '/usr/bin/stransmit': permission denied

A detail explanation is available in our first article on the subject here – Access Violation error, when compiling packets in Gentoo.
All packages are built in a sandbox and there is a sandbox configuration in

/etc/sandbox.d/00default

, which instruct the build process where could write. If you get such error in 99.99% there is a bug in the package and if you do not want to wait for fixing it (report it!) you can manually edit the SANDBOX_WRITE variable and add the path, which causes the build failure. Build the package and remove the added path!!! Or you risk to make your system less secure!

We have problem with building the package “net-libs/srt-1.3.1”

srv1 src # emerge -v net-libs/srt

These are the packages that would be merged, in order:

[ebuild  N     ] net-libs/srt-1.3.1::gentoo  USE="-doc -gnutls -libressl" ABI_X86="32 (64) (-x32)" 0 KiB

Total: 1 package (1 new), Size of downloads: 0 KiB

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) net-libs/srt-1.3.1::gentoo
 * srt-1.3.1.tar.gz BLAKE2B SHA512 size 😉 ...                                                                                                        [ ok ]
>>> Unpacking source...
.....
.....
>>> Install srt-1.3.1 into /var/tmp/portage/net-libs/srt-1.3.1/image/ category net-libs
 * abi_x86_32.x86: running multilib-minimal_abi_src_install
>>> Working in BUILD_DIR: "/var/tmp/portage/net-libs/srt-1.3.1/work/srt-1.3.1-abi_x86_32.x86"
make -j6 -l10 install 
[ 28%] Built target haicrypt_virtual
[ 40%] Built target srtsupport_virtual
[ 80%] Built target srt_virtual
[ 83%] Built target srt_static
[ 85%] Built target srt_shared
[ 90%] Built target srt-file-transmit
[ 95%] Built target srt-live-transmit
[100%] Built target srt-multiplex
Install the project...
-- Install configuration: "Gentoo"
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/libsrt.so.1.3.1
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/libsrt.so.1
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/libsrt.so
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/libsrt.a
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/version.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/srt.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/logging_api.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/platform_sys.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/udt.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/srt4udt.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/pkgconfig/haisrt.pc
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib32/pkgconfig/srt.pc
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-live-transmit
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-live-transmit
 * ACCESS DENIED:  symlink:      /usr/bin/stransmit
CMake Error: failed to create symbolic link '/usr/bin/stransmit': permission denied
-- Created symlink: /usr/bin/stransmit -> srt-live-transmit
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-file-transmit
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-file-transmit
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-multiplex
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-multiplex
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-ffplay
 * abi_x86_64.amd64: running multilib-minimal_abi_src_install
>>> Working in BUILD_DIR: "/var/tmp/portage/net-libs/srt-1.3.1/work/srt-1.3.1-abi_x86_64.amd64"
make -j6 -l10 install 
[ 11%] Built target srtsupport_virtual
[ 52%] Built target srt_virtual
[ 80%] Built target haicrypt_virtual
[ 83%] Built target srt_static
[ 85%] Built target srt_shared
[ 90%] Built target srt-multiplex
[ 95%] Built target srt-file-transmit
[100%] Built target srt-live-transmit
Install the project...
-- Install configuration: "Gentoo"
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/libsrt.so.1.3.1
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/libsrt.so.1
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/libsrt.so
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/libsrt.a
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/version.h
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/srt.h
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/logging_api.h
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/platform_sys.h
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/udt.h
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/include/srt/srt4udt.h
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/pkgconfig/haisrt.pc
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/lib64/pkgconfig/srt.pc
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-live-transmit
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-live-transmit
 * ACCESS DENIED:  symlink:      /usr/bin/stransmit
CMake Error: failed to create symbolic link '/usr/bin/stransmit': permission denied
-- Created symlink: /usr/bin/stransmit -> srt-live-transmit
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-file-transmit
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-file-transmit
-- Installing: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-multiplex
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-multiplex
-- Up-to-date: /var/tmp/portage/net-libs/srt-1.3.1/image/usr/bin/srt-ffplay
>>> Completed installing srt-1.3.1 into /var/tmp/portage/net-libs/srt-1.3.1/image/

 * Final size of build directory: 14632 KiB (14.2 MiB)
 * Final size of installed tree:   5324 KiB ( 5.1 MiB)

 * --------------------------- ACCESS VIOLATION SUMMARY ---------------------------
 * LOG FILE: "/var/log/sandbox/sandbox-25570.log"
 * 
VERSION 1.0
FORMAT: F - Function called
FORMAT: S - Access Status
FORMAT: P - Path as passed to function
FORMAT: A - Absolute Path (not canonical)
FORMAT: R - Canonical Path
FORMAT: C - Command Line

F: symlink
S: deny
P: /usr/bin/stransmit
A: /usr/bin/stransmit
R: /usr/bin/stransmit
C: /usr/bin/cmake -E create_symlink srt-live-transmit /usr/bin/stransmit 

F: symlink
S: deny
P: /usr/bin/stransmit
A: /usr/bin/stransmit
R: /usr/bin/stransmit
C: /usr/bin/cmake -E create_symlink srt-live-transmit /usr/bin/stransmit 
 * --------------------------------------------------------------------------------

>>> Failed to emerge net-libs/srt-1.3.1, Log file:

>>>  '/var/tmp/portage/net-libs/srt-1.3.1/temp/build.log'

In the installation phase occurred the package failure leaving half installed package. So we edited the “/etc/sandbox.d/00default” and added “:/usr/bin” at the end of SANDBOX_WRITE:

SANDBOX_WRITE="/usr/tmp/conftest:/usr/lib/conftest:/usr/lib32/conftest:/usr/lib64/conftest:/usr/tmp/cf:/usr/lib/cf:/usr/lib32/cf:/usr/lib64/cf:/usr/bin"

Then rebuild the package with emerge and remove the added path “:/usr/bin”. This is a dirty workaround, but it will allow you to use srt (and as a dependency to another packages’ installation).

AVAGO MegaRAID SAS-9361-4i with CacheCade – create a new virtual drive RAID5 with SSD caching

Here is howto article for creating a RAID5 device in MegaRAID SAS-9361-4i with SSD caching. First and really important thing is to have the CacheCade ability to the controller, which should be purchased because it is a software add-on.
To have an SSD caching your virtual raid drive (probably hard disk drives) with a MegaRAID controller one setup is the following:

  1. LSI LSI00415 MegaRAID 9361-4i SGL
  2. LSI LSI00293

And it is advisable to have Cache Protection to protect your setup (it is extra protection to the battery kit – it is not the same) – LSI LSI00418 LSICVM02.

You can check also our AVAGO MegaRaid SAS 9361-4i with CacheCade and CacheVault BIOS configuration utilities review.

Here are the steps to create a RAID5 device with SSD caching using the BIOS Configuration Utility:

STEP 1) Supermicro device initialization

main menu
Start up your server

Keep on reading!

AVAGO MegaRaid SAS 9361-4i with CacheCade and CacheVault BIOS configuration utilities review

Here is the screenshots’ review of LSI LSI00415 MegaRAID 9361-4i SGL controller with CacheVault Accessory kit LSI LSI00293. The controller has 3 hard disk and an SSD for the cache vault.
As you can see we use LSI, but in deed this is the AVAGO controller. First it was LSI, which was acquired by AVAGO and then AVAGO acquired Broadcom and rebranded to Broadcom. It is good to know it how these names are connected, because any controllers still use them in BIOS, management console cli, manuals and so on. You are going to see that in the controller’s BIOS and controller initialization boot up you can read only AVAGO – no LSI or Broadcom. Probably the most famous and familiar name of the three company names is LSI (hardware controller).
The screenshots are from the BIOS of “LSI LSI00415 MegaRAID 9361-4i SGL” with LSI LSI00293, CacheCade and LSI LSI00418 LSICVM02 – CacheVault Accessory kit.

So as you see you need two additional features to have a SSD as a cache device in your LSI (AVAGO/Broadcom) controller!

Check out our article AVAGO MegaRAID SAS-9361-4i with CacheCade – create a new virtual drive RAID5 with SSD caching.

SCREENSHOT 1) Booting the controller and checking the devices.

main menu
LSI controller Booting

Keep on reading!

ffmpeg chroot jail from Gentoo linux (ffmpeg 4.1 with gcc 8.2)

Here we build a Gentoo ssh chroot jail. You may need this because

  • you want the bleeding edge versions of the video libraries like x264, x265, ffmpeg the latest version and so on.
  • latest version of GNU C library and the compiler GNU GCC (version 8.2) at the moment
  • use all the optimizations for compiling the ffmpeg and the libraries for you current processor. Use “-march=native”
  • you want just to be able to update your system and not to break your carefully compiled latest version ffmpeg
  • experiment with GCC compile flags
  • update and test in another directory by just copying the old chroot jail directory
  • no need of any additional software on the host (like virtualization lxc, docker, qemu, etc)
  • or you might want a 32bit version (why? really? but it is possible…)

Of course, you can use for the base any Linux system, but it is easy with Gentoo – the latest versions of almost all important libraries and a packet system, which builds everything for you after a little bit of first tuning…

So here is how to do it:
Keep on reading!

Compilation failure in configure with syntax error near unexpected token -Wall,WFLAGS and AX_CHECK_COMPILE_FLAG

If you are trying to build program and in the configure stage you got something like this:

./configure: line 6849: syntax error near unexpected token `-Wall,WFLAGS="$WFLAGS -Wall"'
./configure: line 6849: `AX_CHECK_COMPILE_FLAG(-Wall,WFLAGS="$WFLAGS -Wall")'

You are missing

autoconf-archive

Check your system for the package including this software (a collection of freely re-usable Autoconf macros): https://www.gnu.org/software/autoconf-archive/

In our case we were experiencing a missing dependency for “app-admin/metalog” in Gentoo Linux system. Probably there is a bug, because autoconf-archive is not in the dependency graph of “app-admin/metalog” (app-admin/metalog-20181125), but it should be included.
Such kind of error could occur in all Linux systems. Here is what to install in

  • CentOS 7
    yum install autoconf-archive
    
  • Ubuntu 16/17/18+
  • apt install autoconf-archive
    
  • Gentoo Linux
  • emerge -va autoconf-archive