Linux | Any IT here? Help Me!

Gentoo updating KDE – package blocks

In continuing our tips for updating big or multiple packages in Gentoo this time we show how to update KDE Platform, Plasma and Apps packages. In Gentoo KDE Desktop GUI is split into three major categories:

kde-frameworks
kde-plasma
kde-apps

How we managed to update in our current situation:

Sometimes when updating a group or a package with big dependency graph it is much easier to drop the -“u” update argument and to rebuild some packages with the updates.
use qlist to update/re(build) to pull currently installed packages with some name or category (categories)
Remove obsolete packages – you do not need them, they can just make problems when updating, because emerge will take into consideration their requirements and dependencies and your update could be impossible!

More on the subject of update tips here: Gentoo update tips when updating packages with blocks and masked files
Keep on reading!

Ubuntu 16/18 LTS – load a new kernel without rebooting the server

Here are the commands needed to load a new kernel without rebooting your server or desktop computer. Why you need this? As said in our first article for CentOS 7 – sometime rebooting a server could take 5 to 10 minutes and loading a new kernel is just up to a minute. In fact in most cases loading the new kernel and starting the system then is just under 20-30 seconds, so upgrading your server even with new kernel is super easy lately. We tested it on Ubuntu 16 and Ubuntu 18 servers and it was successful. The system uses systemd and the process is really easy and safe for the systems.

When the processes is initiated the system shutdowns normally (shutting down all running service with systemd) and then load the system immediately with the new kernel and starts the services as usual!

So no need to worry about unflushed data or not proper shutdown of a service! It’s like a normal reboot but without a hardware reboot and is a lot faster!
Here is what is required to load a kernel without hardware rebooting your computer box:

kexec-tools
Load the new kernel, initram file and the command line arguments with “kexec”
Start a systemd target – kexec.target

Ubuntu 16/18 LTS using kexec to load a new kernel

The real commands only for Ubuntu 16/18 LTS:

sudo apt -y install kexec-tools
sudo kexec -l /boot/vmlinuz-4.15.0-33-generic --initrd=/boot/initrd.img-4.15.0-33-generic --command-line="root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro  quiet splash"
sudo systemctl start kexec.target

Here is a real world example with all the output:
And again update your system to see if there is a new kernel and install “kexec-tools”. In our case indeed there is a new kernel – vmlinuz-4.15.0-33-generic

myuser@srv:~$ sudo apt -y upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
  libllvm5.0
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  amd64-microcode autotools-dev dblatex debhelper dh-strip-nondeterminism docbook-dsssl docbook-utils docbook-xml docbook-xsl fonts-lato fonts-lmodern fonts-texgyre
  intel-microcode iucode-tool jadetex javascript-common kernel-common kernel-package kernel-patch-scripts kernel-wedge kerneloops kerneloops-applet kernelshark kerneltop
  libfile-homedir-perl libfile-stripnondeterminism-perl libfile-which-perl libjs-jquery libllvm6.0 libmail-sendmail-perl libosp5 libostyle1c2 libpotrace0 libptexenc1
  libqpdf21 libruby2.3 libsgmls-perl libsp1c2 libsynctex1 libsys-hostname-long-perl libtexlua52 libtexluajit2 libwebpdemux1 libxml2-utils libzzip-0-13
  linux-headers-4.15.0-33 linux-headers-4.15.0-33-generic linux-image-4.15.0-33-generic linux-modules-4.15.0-33-generic linux-modules-extra-4.15.0-33-generic lmodern lynx
  lynx-common openjade po-debconf preview-latex-style prosper ps2eps python-apt rake ruby ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit
  ruby2.3 rubygems-integration sgml-data sgmlspl sp tex-common tex-gyre texlive texlive-base texlive-bibtex-extra texlive-binaries texlive-extra-utils texlive-font-utils
  texlive-fonts-recommended texlive-fonts-recommended-doc texlive-generic-recommended texlive-latex-base texlive-latex-base-doc texlive-latex-extra
  texlive-latex-extra-doc texlive-latex-recommended texlive-latex-recommended-doc texlive-luatex texlive-math-extra texlive-pictures texlive-pictures-doc texlive-pstricks
  texlive-pstricks-doc tipa trace-cmd xmlto xsltproc
The following packages will be upgraded:
.....
.....
Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
        This may take some time... done.
Processing triggers for linux-image-4.15.0-33-generic (4.15.0-33.36~16.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-33-generic
/etc/kernel/postinst.d/vboxadd:
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.15.0-33-generic
Found initrd image: /boot/initrd.img-4.15.0-33-generic
Found linux image: /boot/vmlinuz-4.13.0-36-generic
Found initrd image: /boot/initrd.img-4.13.0-36-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done

myuser@srv:~$ sudo apt -y install kexec-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
  libllvm5.0
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
  kexec-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 77,4 kB of archives.
After this operation, 276 kB of additional disk space will be used.
Get:1 http://bg.archive.ubuntu.com/ubuntu xenial-updates/main amd64 kexec-tools amd64 1:2.0.16-1ubuntu1~16.04.1 [77,4 kB]
Fetched 77,4 kB in 0s (707 kB/s)      
Preconfiguring packages ...
Selecting previously unselected package kexec-tools.
(Reading database ... 253895 files and directories currently installed.)
Preparing to unpack .../kexec-tools_1%3a2.0.16-1ubuntu1~16.04.1_amd64.deb ...
Unpacking kexec-tools (1:2.0.16-1ubuntu1~16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up kexec-tools (1:2.0.16-1ubuntu1~16.04.1) ...
Generating /etc/default/kexec...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for ureadahead (0.100.0-19) ...

     ┌──────────────────────────────────────────────────────────────────┤ Configuring kexec-tools ├───────────────────────────────────────────────────────────────────┐
     │                                                                                                                                                                │ 
     │ If you choose this option, a system reboot will trigger a restart into a kernel loaded by kexec instead of going through the full system boot loader process.  │ 
     │                                                                                                                                                                │ 
     │ Should kexec-tools handle reboots (sysvinit only)?                                                                                                             │ 
     │                                                                                                                                                                │ 
     │                                                 <Yes>                                                    <No>                                                  │ 
     │                                                                                                                                                                │ 
     └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

On the above configuration question mark “” and press Enter.

So we performed an update and there was a new kernel vmlinuz-4.15.0-33-generic, which we would like to load without hardware reboot.
Here is our new kernel in “/boot”

myuser@srv:~$ ls -altr /boot/
total 130420
-rw-r--r--  1 root root   184840 Jan 28  2016 memtest86+_multiboot.bin
-rw-r--r--  1 root root   184380 Jan 28  2016 memtest86+.elf
-rw-r--r--  1 root root   182704 Jan 28  2016 memtest86+.bin
-rw-------  1 root root  3879946 Feb 17  2018 System.map-4.13.0-36-generic
-rw-r--r--  1 root root     2850 Feb 17  2018 retpoline-4.13.0-36-generic
-rw-r--r--  1 root root   213220 Feb 17  2018 config-4.13.0-36-generic
-rw-r--r--  1 root root  1501359 Feb 17  2018 abi-4.13.0-36-generic
-rw-r--r--  1 root root  7710912 May 17 16:50 vmlinuz-4.13.0-36-generic
-rw-------  1 root root  4041375 Aug 16 00:00 System.map-4.15.0-33-generic
-rw-r--r--  1 root root        0 Aug 16 00:00 retpoline-4.15.0-33-generic
-rw-r--r--  1 root root   216913 Aug 16 00:00 config-4.15.0-33-generic
-rw-r--r--  1 root root  1537455 Aug 16 00:00 abi-4.15.0-33-generic
-rw-------  1 root root  8108600 Aug 16 21:58 vmlinuz-4.15.0-33-generic
drwxr-xr-x 24 root root     4096 Sep  7 14:15 ..
-rw-r--r--  1 root root 51290506 Sep  7 14:15 initrd.img-4.13.0-36-generic
-rw-r--r--  1 root root 54451680 Sep  7 14:15 initrd.img-4.15.0-33-generic
drwxr-xr-x  3 root root     4096 Sep  7 14:15 .
drwxr-xr-x  5 root root     4096 Sep  7 14:16 grub

Now we know the kernel and initram file names we just check the kernel arguments in the kernel, load them with kexec and start an systemd target to load the new kernel:

myuser@srv:~$ grep vmlinuz-4.15.0-33-generic /boot/grub/grub.cfg 
        linux   /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro  quiet splash $vt_handoff
                linux   /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro  quiet splash $vt_handoff
                linux   /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro  quiet splash $vt_handoff init=/sbin/upstart
                linux   /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro recovery nomodeset 
myuser@srv:~$ sudo kexec -l /boot/vmlinuz-4.15.0-33-generic --initrd=/boot/initrd.img-4.15.0-33-generic --command-line="root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro  quiet splash"
myuser@srv:~$ sudo systemctl start kexec.target
Connection to srv closed by remote host.
Connection to srv closed.

Use the first line of the grep output above (or you can cat the file and see what is in it if you have any doubts) to take the proper kernel boot arguments and do not include anything starting with “$”.

As you can see systemd performs a normal shutdown of all services and targets.

The ssh connection is immediately closed because the reboot is initiated.
After 10-15 seconds our host is alive and the new kernel is loaded successfully:

root@test ~ $ ssh root@srv
root@srv's password: 
Last login: Wed Sep  5 17:15:08 2018 from test
[root@srv ~]# uname -a
Linux srv.local 4.15.0-33-generic #36~16.04.1-Ubuntu SMP Wed Aug 15 17:21:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]#

Because we do not wanted to mess up the two output in one article we decided to split it in two separate ones, so here is the previous one for CentOS 7 – “CentOS 7 – load a new kernel without rebooting the CentOS 7 server”

CentOS 7 – load a new kernel without rebooting the server

Here are the commands needed to load a new kernel without rebooting your server or desktop computer. Why you need this? Sometime rebooting a server could take 5 to 10 minutes and loading a new kernel is just up to a minute. In fact in most cases loading the new kernel and starting the system then is just under 20-30 seconds, so upgrading your server even with new kernel is super easy lately. We tested it on CentOS 7 server and it was successful. The system uses systemd and the process is really easy and safe for the systems.

When the processes is initiated the system shutdowns normally (shutting down all running service with systemd) and then load the system immediately with the new kernel and starts the services as usual!

kexec-tools
Load the new kernel, initram file and the command line arguments with “kexec”
Start a systemd target – kexec.target

CentOS 7 using kexec to load a new kernel

The real commands only for CentOS 7:

yum install -y kexec-tools
kexec -l /boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 --initrd=/boot/initramfs-3.10.0-862.11.6.el7.x86_64.img --command-line="root=/dev/mapper/centos_srv-root ro crashkernel=auto rd.lvm.lv=centos_srv/root rd.lvm.lv=centos_srv/swap rhgb quiet LANG=en_US.UTF-8"
systemctl start kexec.target

Here is a real world example with all the output:
As you can see it is important to load the initram file and the exact arguments to the kernel. You should take them from the grub 2 configuration – /boot/grub2/grub.cfg
Here is the real output:

[root@srv ~]# yum -y update
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
base                                                                                                                                                 | 3.6 kB  00:00:00     
centos-sclo-rh                                                                                                                                       | 3.0 kB  00:00:00     
centos-sclo-sclo                                                                                                                                     | 2.9 kB  00:00:00     
extras                                                                                                                                               | 3.4 kB  00:00:00     
updates                                                                                                                                              | 3.4 kB  00:00:00     
(1/4): extras/7/x86_64/primary_db                                                                                                                    | 187 kB  00:00:00     
(2/4): updates/7/x86_64/primary_db                                                                                                                   | 5.2 MB  00:00:01     
(3/4): centos-sclo-sclo/x86_64/primary_db                                                                                                            | 292 kB  00:00:01     
(4/4): centos-sclo-rh/x86_64/primary_db                                                                                                              | 3.7 MB  00:00:02     
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-862.11.6.el7 will be installed
---> Package kernel-headers.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-headers.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package kernel-tools.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.11.6.el7 will be an update
....
....
[root@srv ~]# yum install -y kexec-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
Package kexec-tools-2.0.15-13.el7.x86_64 already installed and latest version
Nothing to do

So we performed an update and there was a new kernel kernel.x86_64 0:3.10.0-862.11.6.el7, which we would like to load without hardware reboot.
Here is our new kernel in “/boot”

[root@srv ~]# ls -altr /boot/
total 171812
-rw-------.  1 root root  3228420 22 Aug  2017 System.map-3.10.0-693.el7.x86_64
-rw-r--r--.  1 root root   140894 22 Aug  2017 config-3.10.0-693.el7.x86_64
-rw-r--r--.  1 root root      166 22 Aug  2017 .vmlinuz-3.10.0-693.el7.x86_64.hmac
-rwxr-xr-x.  1 root root  5877760 22 Aug  2017 vmlinuz-3.10.0-693.el7.x86_64
-rw-r--r--.  1 root root   293027 22 Aug  2017 symvers-3.10.0-693.el7.x86_64.gz
drwxr-xr-x.  3 root root       17 20 Feb  2018 efi
drwxr-xr-x.  2 root root       27 20 Feb  2018 grub
-rw-r--r--.  1 root root   611343 20 Feb  2018 initrd-plymouth.img
-rw-------.  1 root root 51315067 20 Feb  2018 initramfs-0-rescue-cc7889764e86441b8d1eb54e29e81a91.img
-rwxr-xr-x.  1 root root  5877760 20 Feb  2018 vmlinuz-0-rescue-cc7889764e86441b8d1eb54e29e81a91
-rw-------.  1 root root  3409912 21 May 23,50 System.map-3.10.0-862.3.2.el7.x86_64
-rw-r--r--.  1 root root   147823 21 May 23,50 config-3.10.0-862.3.2.el7.x86_64
-rw-r--r--.  1 root root      170 21 May 23,50 .vmlinuz-3.10.0-862.3.2.el7.x86_64.hmac
-rwxr-xr-x.  1 root root  6228832 21 May 23,50 vmlinuz-3.10.0-862.3.2.el7.x86_64
-rw-r--r--.  1 root root   304943 21 May 23,52 symvers-3.10.0-862.3.2.el7.x86_64.gz
dr-xr-xr-x. 17 root root      224 14 Jun  7,18 ..
-rw-------.  1 root root 12997841 14 Jun  7,19 initramfs-3.10.0-693.el7.x86_64kdump.img
-rw-------.  1 root root 20771492 14 Jun  7,20 initramfs-3.10.0-693.el7.x86_64.img
-rw-------.  1 root root 13007444 14 Jun  7,23 initramfs-3.10.0-862.3.2.el7.x86_64kdump.img
-rw-r--r--.  1 root root   147859 14 Aug 22,02 config-3.10.0-862.11.6.el7.x86_64
-rw-------.  1 root root  3414344 14 Aug 22,02 System.map-3.10.0-862.11.6.el7.x86_64
-rw-r--r--.  1 root root      171 14 Aug 22,02 .vmlinuz-3.10.0-862.11.6.el7.x86_64.hmac
-rwxr-xr-x.  1 root root  6242208 14 Aug 22,02 vmlinuz-3.10.0-862.11.6.el7.x86_64
-rw-r--r--.  1 root root   305158 14 Aug 22,05 symvers-3.10.0-862.11.6.el7.x86_64.gz
-rw-------.  1 root root 20774393  5 Sep 15,20 initramfs-3.10.0-862.11.6.el7.x86_64.img
drwx------.  5 root root       97  5 Sep 15,20 grub2
dr-xr-xr-x.  5 root root     4096  5 Sep 15,21 .
-rw-------.  1 root root 20782404  5 Sep 15,21 initramfs-3.10.0-862.3.2.el7.x86_64.img

Now we know the kernel and initram file names we just check the kernel arguments in the kernel, load them with kexec and start an systemd target to load the new kernel:

[root@srv ~]# grep vmlinuz-3.10.0-862.11.6.el7.x86_64 /boot/grub2/grub.cfg 
        linux16 /vmlinuz-3.10.0-862.11.6.el7.x86_64 root=/dev/mapper/centos_srv-root ro crashkernel=auto rd.lvm.lv=centos_srv/root rd.lvm.lv=centos_srv/swap rhgb quiet LANG=en_US.UTF-8
[root@srv ~]# kexec -l /boot/vmlinuz-3.10.0-862.11.6.el7.x86_64 --initrd=/boot/initramfs-3.10.0-862.11.6.el7.x86_64.img --command-line="root=/dev/mapper/centos_srv-root ro crashkernel=auto rd.lvm.lv=centos_srv/root rd.lvm.lv=centos_srv/swap rhgb quiet LANG=en_US.UTF-8"
[root@srv ~]# systemctl start kexec.target
Connection to srv closed by remote host.
Connection to srv closed.

As you can see systemd performs a normal shutdown of all services and targets.

The ssh connection is immediately closed because the reboot is initiated.
After 10-15 seconds our host is alive and the new kernel is loaded successfully:

root@test ~ $ ssh root@srv
root@srv's password: 
Last login: Wed Sep  5 15:16:08 2018 from test
[root@srv ~]# uname -a
Linux srv.local 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]#

Because we do not wanted to mess up the two output from different linux distros in one article we decided to split it in two separate ones, so here is the one for Ubuntu 16/18 LTS – “Ubuntu 16/18 LTS – load a new kernel without rebooting the server“

Gentoo updating perl with many masked and blocked packages

After our previous howto Gentoo – update dev-libs/icu on a desktop box with KDE GUI and many masked packages we want to show you another example of how to update perl, which could have hundreds dependencies installed as perl modules. You can check for more details above howto, details about what to do when there are masked and blocked packages, because here we use only some of the options we have the proper ones for this case.
Just to note when updating perl you must reinstall (recompile, rebuild) all the external modules (additional packages) and all perl modules installed with CPAN module (if you do not know what is this, you probably have not used it so no worries).

How we managed to update in our current situation:

use qlist to update/re(build) to pull currently installed packages with some name or category (categories)
Add or remove USE flags if needed – emerge will show you information about it. Use package.use, package.mask, package.unmask and so on.
Use verbose,verbose-conflicts and backtrack with emerge
Include explicitly packages, which block our updates in the emerge line! You could specify packages with the versions.

More on the subject of Perl update tips here: Gentoo update tips when updating packages with blocks and masked files
And a new article on the subject – Gentoo – updating perl and problems like perl-core/ is blocking virtual/perl-
Keep on reading!

Build your own kernel under Ubuntu using mainline (latest) kernel

Here we will show how to build your own kernel under Ubuntu 16/17/18. We are going to use

mainline kernel

because most of the cases when we need to build our own is when we need some new feature presented in the new kernels. So Ubuntu mainline kernels are selected new kernels, which are packed in deb packets for simplicity and can be used to test new features, but they are not supported by Ubuntu teams and you should test them really carefully if you want to use them in production!
The mainline kernels are here: http://kernel.ubuntu.com/~kernel-ppa/mainline/. This is the main address, where you could download the latest kernels and some other legacy one if you have old system.
This howto is based on https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel and https://wiki.ubuntu.com/Kernel/MainlineBuilds – we summarize the commands needed to build your new and latest kernel and show you some typical mistakes and problems during the process.
To show you more information our purpose will be to build the one of the latest kernels with low latency features enabled (Ubuntu has such type of kernel configuration)! And here is one of the most important dependency, which is not explicitly included or even mentioned in most of the tutorials for building your own kernels –

you MUST include new or even latest Linux firmware package

All Linux distros have a package or packages containing files with different firmware for multiple devices supported by the kernel modules. Basically firmware is used by a kernel module and it can be a micro code (program), which is uploaded in the device or just complete the kernel module to function properly with a device. Recent days firmware is like a black box the manufacturer of the devices pack a proprietary code in a firmware blob and the source code is unknown to the community.

The first thing you should do is to choose the kernel version you want to build. We want version 4.17.x as more stable of the latest 4.18, which was released a few days weeks ago (let’s wait for a month after the initial release before using it!).
So the main address is here http://kernel.ubuntu.com/~kernel-ppa/mainline/ and the kernel files, which we are going to use to rebuild are here http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.17.19/
Keep on reading!

Gentoo – update dev-libs/icu on a desktop box with KDE GUI and many masked packages

No, we are not going to answer why someone will use Gentoo for Desktop, but well such human beings still exist and we have one piece of snippet the updating old dev-libs/icu package, because KDE Platform and new version of Chromium depend on a new version >=dev-libs/icu-59.
The main reason to include this update here is show how to deal with the dependency hell in Gentoo – multiple blocked packages and some old and deprecated packages, but still installed.

To summarize it up at the beginning how we did it

and then you’ll see what are the steps we took to resolve the conflicts and masked packages:

Use verbose,verbose-conflicts and backtrack with emerge
Remove only big GUI packages, which have really big dependency graph like office suites or development IDEs
Remove obsolete packages – you do not need them, they can just make problems when updating, because emerge will take into consideration their requirements and dependencies and your update could be impossible!
Include explicitly packages, which block our updates in the emerge line!
use tools like “equery” (part of app-portage/gentoolkit) for checking dependencies and/or which packages depend on the queried package. You can use it with specific version for the package. “qlist” (part of app-portage/portage-utils) also is a handful tool.
Sometimes when updating a group or a package with big dependency graph it is much easier to drop the -“u” update argument and to rebuild some packages with the updates.

More on the subject of update tips here: Gentoo update tips when updating packages with blocks and masked files
Keep on reading!

Enable BBR tcp congestion control under CentOS 7

It’s been a while since a new tcp congestion has been developed (by Google), which has the potential to replace the default cubic one! The TCP BBR tries to keep being fast and improves the network connectivity when there is packet loss. Enabling it on the server does not require it to enable it on the client to function BBR algorithm properly. So probably we’ll see the switch to this new tcp congestion control in the next major linux distros releases.

Here we present how to enable TCP BBR on your CentOS 7 box. As you know the CentOS 7 uses kernel 3.10.X and the TCP BBR was introduced in version 4.9+ kernels and in version 4.13 and on it has a tcp-level pacing so you do not need the qdisc “fq” to be enabled, when using TCP BBR. So we need a new kernel version 4.9 and on and the easiest and safest way to do it to use the mainline kernel from https://www.elrepo.org/. We have a detailed howto for this if you need more information on using mainline kernel with your CentOS 7.

Here are the steps, you need to be root to execute the following commands:

STEP 1) Update your system and reboot

yum clean all
yum -y update
reboot

Keep on reading!

Close socket as if the remote closed the connection

If you have a hung process and it happened to be in this state because of the network, for example your client or server program is in read timeout state, you can use

lsof and gbg

to close the network socket simulating the other (remote) end closed it and the process will continue operating normally.

In our case there is a couple of nrpe process hung in read from a network socket:

[root@srv ~]# lsof -n -p 9948
COMMAND  PID USER   FD   TYPE             DEVICE SIZE/OFF    NODE NAME
nrpe    9948 nrpe  cwd    DIR                9,2     4096       2 /
nrpe    9948 nrpe  rtd    DIR                9,2     4096       2 /
nrpe    9948 nrpe  txt    REG                9,2    69960 1053396 /usr/sbin/nrpe
nrpe    9948 nrpe  mem    REG                9,2    62184 1053312 /usr/lib64/libnss_files-2.17.so
nrpe    9948 nrpe  mem    REG                9,2   402384 1051943 /usr/lib64/libpcre.so.1.2.0
nrpe    9948 nrpe  mem    REG                9,2   155784 1057231 /usr/lib64/libselinux.so.1
nrpe    9948 nrpe  mem    REG                9,2   144792 1051919 /usr/lib64/libpthread-2.17.so
nrpe    9948 nrpe  mem    REG                9,2   106848 1053314 /usr/lib64/libresolv-2.17.so
nrpe    9948 nrpe  mem    REG                9,2    15688 1051678 /usr/lib64/libkeyutils.so.1.5
nrpe    9948 nrpe  mem    REG                9,2    58728 1051843 /usr/lib64/libkrb5support.so.0.1
nrpe    9948 nrpe  mem    REG                9,2    90664 1051808 /usr/lib64/libz.so.1.2.7
nrpe    9948 nrpe  mem    REG                9,2    19776 1053308 /usr/lib64/libdl-2.17.so
nrpe    9948 nrpe  mem    REG                9,2   210840 1051701 /usr/lib64/libk5crypto.so.3.1
nrpe    9948 nrpe  mem    REG                9,2    15920 1051682 /usr/lib64/libcom_err.so.2.1
nrpe    9948 nrpe  mem    REG                9,2   963576 1051755 /usr/lib64/libkrb5.so.3.3
nrpe    9948 nrpe  mem    REG                9,2   320408 1051956 /usr/lib64/libgssapi_krb5.so.2.2
nrpe    9948 nrpe  mem    REG                9,2  2173512 1051792 /usr/lib64/libc-2.17.so
nrpe    9948 nrpe  mem    REG                9,2    42520 1051997 /usr/lib64/libwrap.so.0.7.6
nrpe    9948 nrpe  mem    REG                9,2   117680 1053310 /usr/lib64/libnsl-2.17.so
nrpe    9948 nrpe  mem    REG                9,2  2512832 1051648 /usr/lib64/libcrypto.so.1.0.2k
nrpe    9948 nrpe  mem    REG                9,2   470360 1051690 /usr/lib64/libssl.so.1.0.2k
nrpe    9948 nrpe  mem    REG                9,2   164240 1049135 /usr/lib64/ld-2.17.so
nrpe    9948 nrpe    0r   CHR                1,3      0t0    1028 /dev/null
nrpe    9948 nrpe    1w   CHR                1,3      0t0    1028 /dev/null
nrpe    9948 nrpe    2w   CHR                1,3      0t0    1028 /dev/null
nrpe    9948 nrpe    3u  unix 0xffff961d48d37000      0t0   19091 socket
nrpe    9948 nrpe    6u  IPv4          261850576      0t0     TCP 10.10.10.10:5666->10.10.10.254:39056 (ESTABLISHED)

As you can see the FD column shows the File Descriptor number of the opened file (network resource here) and you can use it with

gdb

to simulate closing the network socket as if the remote close it but from the same machine.

[root@srv ~]# gdb -p 9948
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Attaching to process 9948
Reading symbols from /usr/sbin/nrpe...Reading symbols from /usr/sbin/nrpe...(no debugging symbols found)...done.
(no debugging symbols found)...done.
....
....
Loaded symbols for /lib64/libpcre.so.1
Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libnss_files.so.2
0x00007f91e8295c70 in __read_nocancel () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install nrpe-3.2.0-6.el7.x86_64
(gdb) call shutdown(6, 0)
$1 = 0
(gdb) quit
A debugging session is active.

        Inferior 1 [process 9948] will be detached.

Quit anyway? (y or n) Y
Detaching from program: /usr/sbin/nrpe, process 9948

Just call

call shutdown(FileDescriptorID, 0)

and quit the gdb. In our case the FileDescriptorID is 6, so we executed

call shutdown(6, 0)

And the network socket between this machine and the remote one will be terminated, so the process nrpe could continue its execution.
Of course, in your cases you can look for a specific network connection among many other, but lsof is the tool you can use to identify the connection and the right file descriptor number to use in gdb.

Delete files in a directory with checking for the free space with find and stat command – effective and fast approach

If you have a big storage for let’s say your cache proxy and you may want to delete some files fast you could use

find

linux command to delete files. There are plenty examples in the Internet how to do it but many of them use pipes and sorts or other manipulation of the output, which could require the whole output before running some of the piped commands and if you have millions of files you could wait for an hour or two before the command to run and then to see it is not efficient for you case!
So we need a command (or commands), which begins to delete files immediately with some sort of simple selection. We can use linux command “find” with the most used options like

“-type f” – only files
“-mtime +/-N” – only files older or newer than N days. “-mtime +5” – files or directory older than 5 days and “-mtime -5” files or directory newer than 5 days
“-name ‘*.jpg'” – files or directories with patterns in the name *.jpg, for example “sample.jpg”

So we’ll use

“find” to delete files
and a while cycle periodically to check for the free space
and if the free space gets greater than we would like we will kill the find command.

This approach is probably the most effective one, because we run only once the “find” command – if you have multiple runs of the find command in directory with many sub-directories it will check them on every execution in the same order – you may imagine what is like to have hundreds of thousands sub-directories full of files!

The goal is to execute only one find command and to stop it when we reach the desired free space!

So here an example:

echo "STARTING WITH /mnt/cache/"
find /mnt/cache/ -type f -name '*.jpg' -mtime +60 -delete &>/dev/null &
PID=$!
stime=60

while kill -0 "$PID" >/dev/null 2>&1; do
    FREESPACE=$(($(stat -f --format="%a*%S/1024" "/mnt/cache/")))
    if [[ $FREESPACE -gt 50000000 ]]
    then 
        kill "$PID"
        break
    fi
    echo "SLEEPING FOR $stime"
    sleep ${stime}s
done
echo "TERMINATING"
exit 0

The above piece of bash code will find in /mnt/cache/ only files with names ‘*.jpg’ and older than 60 days and will delete them. In parallel we check if the find command is still executing (because it could return finding nothing or just a small amount of files) and sleep for a 60 seconds. If there is 50000000 kylobites (50Gbytes) it will kill the find command!
Someone would argue we could have used

timeout command,

but this will kill “find” every time the check time passes and on every execution of find there will be the same files to check from the previous run!!! And if the first “find” command passed 10000 files and deleted 100, the second time it will check again this 9900 not deleted files and then will continue with the new one and if you execute it again and again you could enter in a endless loop, because the time is used only for checking the files previously checked and not deleted.

Here is the bash script with two parameters for the command line:

#!/bin/bash

if [ "$1" == "" ]
then 
        echo "USAGE: <script> <path> <time=60s>"
        exit 0
fi
if [ "$2" == "" ] || [ "$2" == 0 ]
then
        stime=60
else
        stime=$2
fi

sleep ${stime}s

echo "STARTING WITH $1"
find "$1" -type f -delete &>/dev/null &
PID=$!

while kill -0 "$PID" >/dev/null 2>&1; do
    FREESPACE=$(($(stat -f --format="%a*%S/1024" "$1")))
    if [[ $FREESPACE -gt 50000000 ]]
    then 
        kill "$PID"
        break
    fi
    echo "SLEEPING FOR $2"
    sleep ${stime}s
done
echo "TERMINATING"
exit 0

Check for the missing two parameters. The second parameter is not mandatory and the default value is 60s. Probably you can tune the script for your needs:

the find commands to look for specific files with name/mtime/type and so on.
third parameter in the command line to set minimum free space to check.
the time of the second parameter to be passed with the argument for s=seconds, m=minutes, h=hours and so on.

Install xmr-stak 2.4 in a machine with NVIDIA video card under Ubuntu 18.04 LTS

This is simple “how to” to get started mining crypto currency with XMR-STAK using cryptonight algorithm like monero, electroneum, intensecoin and more. This time we are going to use PPA repository for the NVIDIA drivers and the CUDA toolkit. Installing the official NVIDIA driver we can use our ASUS ROG POSEIDON GTX 1080Ti Platinum Edition to mine along with our processor AMD Ryzen Threadripper 1950X (16 cores with 32 threads with 3.4GHz base frequency and boost to 4GHz).
XMR-STAK could be used only for CPU mining, but now we installed an NVIDIA card and Ubuntu 18.04 LTS. Ubuntu is installed with the default open source NVIDIA driver (nouveau), which is OK for every day and office use, but if you like to game or to mine you must install the proprietary driver shipped by NVIDIA. The following howto will show you how to do it!

Just follow the simple steps and you’ll have an optimized GPU and CPU miner for cryptonight crypto currencies:

STEP 1) Install Ubuntu 18.04 LTS.

If you do not have Ubuntu installed yet it is time to do it. You can see how easy is to do it here even if you have multiple systems installed – Install Ubuntu Desktop 18.04 LTS on a PC with existing windows 10 and linux If your machine is only for mining and you want to have only one system it’s even simpler and you can still follow the above howto!

STEP 2) Install NVIDIA proprietary driver and CUDA Toolkit

Here we show you a brief command set to execute, but if you want a detailed explanations you could check the two howtos on the subject: Install NVIDIA proprietary drivers on Ubuntu 18.0 LTS (x86_64) and Install NVIDIA Cuda on Ubuntu 18.04 LTS (x86_64)

myuser@srv:~$ sudo apt -y update
myuser@srv:~$ sudo add-apt-repository -y ppa:graphics-drivers/ppa
myuser@srv:~$ sudo apt -y update
myuser@srv:~$ sudo ubuntu-drivers autoinstall
myuser@srv:~$ sudo reboot
myuser@srv:~$ sudo apt install -y nvidia-cuda-toolkit gcc-6

Check if your driver is loaded successfully before continuing:

myuser@srv:~$ dmesg|grep -i nvidia
[    1.414268] nvidia: loading out-of-tree module taints kernel.
[    1.414273] nvidia: module license 'NVIDIA' taints kernel.
[    1.632334] nvidia: module verification failed: signature and/or required key missing - tainting kernel
[    1.639698] nvidia-nvlink: Nvlink Core is being initialized, major device number 236
[    1.640031] nvidia 0000:0a:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=none:owns=io+mem
[    1.640146] NVRM: loading NVIDIA UNIX x86_64 Kernel Module  396.24.10  Tue Jul 10 10:00:18 PDT 2018 (using threaded interrupts)
[    1.647029] nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms  396.24.10  Tue Jul 10 08:53:56 PDT 2018
[    1.648372] [drm] [nvidia-drm] [GPU ID 0x00000a00] Loading driver
[    1.648373] [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:0a:00.0 on minor 0
[   37.086767] nvidia-uvm: Loaded the UVM driver in 8 mode, major device number 511
[   37.761570] caller os_map_kernel_space.part.7+0xda/0x120 [nvidia] mapping multiple BARs
[   38.532918] input: HDA NVidia HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.1/0000:0a:00.1/sound/card0/input16
[   38.532960] input: HDA NVidia HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.1/0000:0a:00.1/sound/card0/input17
[   38.532996] input: HDA NVidia HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.1/0000:0a:00.1/sound/card0/input18
[   38.533028] input: HDA NVidia HDMI/DP,pcm=9 as /devices/pci0000:00/0000:00:03.1/0000:0a:00.1/sound/card0/input19
[   40.851562] caller os_map_kernel_space.part.7+0xda/0x120 [nvidia] mapping multiple BARs
[   41.205592] nvidia-modeset: Allocated GPU:0 (GPU-3bad60bf-8ff7-4cba-8b51-a931299a56d8) @ PCI:0000:0a:00.0

STEP 3) Install dependencies for XMR-STAK

Around 380Mbytes of libraries and tools to be installed in addition.

myuser@srv:~$ sudo apt -y install libboost-all-dev libleveldb-dev libcurl4-openssl-dev libmicrohttpd-dev libminiupnpc-dev libgmp-dev libmicrohttpd-dev libssl-dev cmake build-essential screen git
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.4ubuntu1).
build-essential set to manually installed.
The following additional packages will be installed:
  autotools-dev cmake-data dh-python gir1.2-harfbuzz-0.0 ibverbs-providers icu-devtools libboost-atomic-dev libboost-atomic1.65-dev libboost-atomic1.65.1
  libboost-chrono-dev libboost-chrono1.65-dev libboost-chrono1.65.1 libboost-container-dev libboost-container1.65-dev libboost-container1.65.1 libboost-context-dev
  libboost-context1.65-dev libboost-context1.65.1 libboost-coroutine-dev libboost-coroutine1.65-dev libboost-coroutine1.65.1 libboost-date-time-dev
  libboost-date-time1.65-dev libboost-dev libboost-exception-dev libboost-exception1.65-dev libboost-fiber-dev libboost-fiber1.65-dev libboost-fiber1.65.1
  libboost-filesystem-dev libboost-filesystem1.65-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.65-dev libboost-graph-parallel1.65.1
  libboost-graph1.65-dev libboost-graph1.65.1 libboost-iostreams-dev libboost-iostreams1.65-dev libboost-locale-dev libboost-locale1.65-dev libboost-log-dev
  libboost-log1.65-dev libboost-log1.65.1 libboost-math-dev libboost-math1.65-dev libboost-math1.65.1 libboost-mpi-dev libboost-mpi-python-dev libboost-mpi-python1.65-dev
  libboost-mpi-python1.65.1 libboost-mpi1.65-dev libboost-mpi1.65.1 libboost-numpy-dev libboost-numpy1.65-dev libboost-numpy1.65.1 libboost-program-options-dev
  libboost-program-options1.65-dev libboost-program-options1.65.1 libboost-python-dev libboost-python1.65-dev libboost-python1.65.1 libboost-random-dev
  libboost-random1.65-dev libboost-random1.65.1 libboost-regex-dev libboost-regex1.65-dev libboost-regex1.65.1 libboost-serialization-dev libboost-serialization1.65-dev
  libboost-serialization1.65.1 libboost-signals-dev libboost-signals1.65-dev libboost-signals1.65.1 libboost-stacktrace-dev libboost-stacktrace1.65-dev
  libboost-stacktrace1.65.1 libboost-system-dev libboost-system1.65-dev libboost-test-dev libboost-test1.65-dev libboost-test1.65.1 libboost-thread-dev
  libboost-thread1.65-dev libboost-timer-dev libboost-timer1.65-dev libboost-timer1.65.1 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.65-dev
  libboost-type-erasure1.65.1 libboost-wave-dev libboost-wave1.65-dev libboost-wave1.65.1 libboost1.65-dev libboost1.65-tools-dev libcurl4 libexpat1-dev libfabric1
  libgcrypt20-dev libglib2.0-dev libglib2.0-dev-bin libgmpxx4ldbl libgnutls-dane0 libgnutls-openssl27 libgnutls28-dev libgnutlsxx28 libgpg-error-dev libgraphite2-dev
  libharfbuzz-dev libharfbuzz-gobject0 libhwloc-dev libhwloc-plugins libhwloc5 libibverbs-dev libibverbs1 libicu-dev libicu-le-hb-dev libicu-le-hb0 libiculx60
  libidn2-0-dev libidn2-dev libjsoncpp1 libleveldb1v5 libltdl-dev libmicrohttpd12 libnl-route-3-200 libnuma-dev libopenmpi-dev libopenmpi2 libp11-kit-dev libpcre16-3
  libpcre3-dev libpcre32-3 libpcrecpp0v5 libpsm-infinipath1 libpython-dev libpython-stdlib libpython2.7-dev libpython3-dev libpython3.6-dev librdmacm1 librhash0
  libsnappy1v5 libssl-doc libtasn1-6-dev libtasn1-doc libtool libunbound2 libuv1 mpi-default-bin mpi-default-dev nettle-dev openmpi-bin openmpi-common python python-dev
  python-minimal python2.7 python2.7-dev python2.7-minimal python3-dev python3-distutils python3-lib2to3 python3.6-dev zlib1g-dev git-man liberror-perl libutempter0
Suggested packages:
  cmake-doc ninja-build libboost-doc graphviz libboost1.65-doc gccxml libmpfrc++-dev libntl-dev xsltproc doxygen docbook-xml docbook-xsl default-jdk fop libcurl4-doc
  libidn11-dev libkrb5-dev libldap2-dev librtmp-dev libssh2-1-dev libgcrypt20-doc libglib2.0-doc gmp-doc libgmp10-doc libmpfr-dev gnutls-doc gnutls-bin libgraphite2-utils
  libhwloc-contrib-plugins icu-doc leveldb-doc libtool-doc minissdpd openmpi-doc autoconf automaken gfortran | fortran95-compiler gcj-jdk gfortran python-doc python-tk git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk gitweb git-cvs git-mediawiki git-svn byobu | screenie | iselect
  python2.7-doc binfmt-support
The following NEW packages will be installed:
  autotools-dev cmake cmake-data dh-python gir1.2-harfbuzz-0.0 ibverbs-providers icu-devtools libboost-all-dev libboost-atomic-dev libboost-atomic1.65-dev
  libboost-atomic1.65.1 libboost-chrono-dev libboost-chrono1.65-dev libboost-chrono1.65.1 libboost-container-dev libboost-container1.65-dev libboost-container1.65.1
  libboost-context-dev libboost-context1.65-dev libboost-context1.65.1 libboost-coroutine-dev libboost-coroutine1.65-dev libboost-coroutine1.65.1 libboost-date-time-dev
  libboost-date-time1.65-dev libboost-dev libboost-exception-dev libboost-exception1.65-dev libboost-fiber-dev libboost-fiber1.65-dev libboost-fiber1.65.1
  libboost-filesystem-dev libboost-filesystem1.65-dev libboost-graph-dev libboost-graph-parallel-dev libboost-graph-parallel1.65-dev libboost-graph-parallel1.65.1
  libboost-graph1.65-dev libboost-graph1.65.1 libboost-iostreams-dev libboost-iostreams1.65-dev libboost-locale-dev libboost-locale1.65-dev libboost-log-dev
  libboost-log1.65-dev libboost-log1.65.1 libboost-math-dev libboost-math1.65-dev libboost-math1.65.1 libboost-mpi-dev libboost-mpi-python-dev libboost-mpi-python1.65-dev
  libboost-mpi-python1.65.1 libboost-mpi1.65-dev libboost-mpi1.65.1 libboost-numpy-dev libboost-numpy1.65-dev libboost-numpy1.65.1 libboost-program-options-dev
  libboost-program-options1.65-dev libboost-program-options1.65.1 libboost-python-dev libboost-python1.65-dev libboost-python1.65.1 libboost-random-dev
  libboost-random1.65-dev libboost-random1.65.1 libboost-regex-dev libboost-regex1.65-dev libboost-regex1.65.1 libboost-serialization-dev libboost-serialization1.65-dev
  libboost-serialization1.65.1 libboost-signals-dev libboost-signals1.65-dev libboost-signals1.65.1 libboost-stacktrace-dev libboost-stacktrace1.65-dev
  libboost-stacktrace1.65.1 libboost-system-dev libboost-system1.65-dev libboost-test-dev libboost-test1.65-dev libboost-test1.65.1 libboost-thread-dev
  libboost-thread1.65-dev libboost-timer-dev libboost-timer1.65-dev libboost-timer1.65.1 libboost-tools-dev libboost-type-erasure-dev libboost-type-erasure1.65-dev
  libboost-type-erasure1.65.1 libboost-wave-dev libboost-wave1.65-dev libboost-wave1.65.1 libboost1.65-dev libboost1.65-tools-dev libcurl4 libcurl4-openssl-dev
  libexpat1-dev libfabric1 libgcrypt20-dev libglib2.0-dev libglib2.0-dev-bin libgmp-dev libgmpxx4ldbl libgnutls-dane0 libgnutls-openssl27 libgnutls28-dev libgnutlsxx28
  libgpg-error-dev libgraphite2-dev libharfbuzz-dev libharfbuzz-gobject0 libhwloc-dev libhwloc-plugins libhwloc5 libibverbs-dev libibverbs1 libicu-dev libicu-le-hb-dev
  libicu-le-hb0 libiculx60 libidn2-0-dev libidn2-dev libjsoncpp1 libleveldb-dev libleveldb1v5 libltdl-dev libmicrohttpd-dev libmicrohttpd12 libminiupnpc-dev
  libnl-route-3-200 libnuma-dev libopenmpi-dev libopenmpi2 libp11-kit-dev libpcre16-3 libpcre3-dev libpcre32-3 libpcrecpp0v5 libpsm-infinipath1 libpython-dev
  libpython-stdlib libpython2.7-dev libpython3-dev libpython3.6-dev librdmacm1 librhash0 libsnappy1v5 libssl-dev libssl-doc libtasn1-6-dev libtasn1-doc libtool
  libunbound2 libuv1 mpi-default-bin mpi-default-dev nettle-dev openmpi-bin openmpi-common python python-dev python-minimal python2.7 python2.7-dev python2.7-minimal
  python3-dev python3-distutils python3-lib2to3 python3.6-dev zlib1g-dev git git-man liberror-perl libutempter0 screen
0 upgraded, 179 newly installed, 0 to remove and 26 not upgraded.
Need to get 76.9 MB of archives.
After this operation, 380 MB of additional disk space will be used.
Get:1 http://gb.archive.ubuntu.com/ubuntu bionic/main amd64 python2.7-minimal amd64 2.7.15~rc1-1 [1292 kB]
....
....

Tune the screen program (this is not mandatory, but is important to have good scrolling) and start screen.

myuser@srv:~$ echo "termcapinfo xterm* ti@:te@" >> ~/.screenrc
myuser@srv:~$ screen -R xmr-stak

STEP 4) Compile XMR-STAK

First get the sources from the official github repository. The master is now with tag 2.4.7. If you do not know how to work with git and want the exact tag version it is easy, just open in the browser “https://github.com/fireice-uk/xmr-stak” change “Branch:master” (click on it and a dropdown will appear, click the tab “Tag” and select your tag version) to “Tag:2.4.7” then click on “Clone or download” on the right and click on “Download ZIP” and it will offer you to download a zip file with the name 2.4.7.zip (here is the URL: https://codeload.github.com/fireice-uk/xmr-stak/zip/2.4.7).
This step it is important to note that we MUST build XMR-STAK with GNU GCC 6 (6.4.0) and the default GNU GCC in Ubuntu 18.04 LTS is 7 (7.3.0), so we export environment variables “CC” and “CXX” to use gcc-6. Exporting CC and CXX is enough to compile succeessfully no need to mess with “update-alternatives”.

myuser@srv:~$ git clone https://github.com/fireice-uk/xmr-stak
Cloning into 'xmr-stak'...
remote: Counting objects: 5078, done.
remote: Total 5078 (delta 0), reused 0 (delta 0), pack-reused 5078
Receiving objects: 100% (5078/5078), 1.48 MiB | 1.86 MiB/s, done.
Resolving deltas: 100% (3628/3628), done.
myuser@srv:~$ #or just download the tag version 2.4.7 from this url: https://codeload.github.com/fireice-uk/xmr-stak/zip/2.4.7 or as it was described above. This 3 lines are comments, if you want tag 2.4.7 just uncomment the second and the third lines
myuser@srv:~$ #wget https://codeload.github.com/fireice-uk/xmr-stak/zip/2.4.7
myuser@srv:~$ #unzip 2.4.7.zip
myuser@srv:~$ mkdir -p ./xmr-stak/build
myuser@srv:~$ cd ./xmr-stak/build
myuser@srv:~/xmr-stak/build$ export CFLAGS="-O2 -march=native -msse3 -fomit-frame-pointer -pipe"
myuser@srv:~/xmr-stak/build$ export CHOST="x86_64-pc-linux-gnu"
myuser@srv:~/xmr-stak/build$ export CXXFLAGS="${CFLAGS}"
myuser@srv:~/xmr-stak/build$ export LDFLAGS="-Wl,-O1"
myuser@srv:~/xmr-stak/build$ export CC=/usr/bin/gcc-6
myuser@srv:~/xmr-stak/build$ export CXX=/usr/bin/g++-6
myuser@srv:~/xmr-stak/build$ cmake ../
-- The C compiler identification is GNU 6.4.0
-- The CXX compiler identification is GNU 6.4.0
-- Check for working C compiler: /usr/bin/gcc-6
-- Check for working C compiler: /usr/bin/gcc-6 -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/g++-6
-- Check for working CXX compiler: /usr/bin/g++-6 -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found CUDA: /usr (found suitable version "9.1", minimum required is "7.5") 
-- Looking for CL_VERSION_2_2
-- Looking for CL_VERSION_2_2 - found
-- Found OpenCL: /usr/lib/x86_64-linux-gnu/libOpenCL.so (found version "2.2") 
-- Found OpenSSL: /usr/lib/x86_64-linux-gnu/libcrypto.so (found version "1.1.0g") 
-- Configuring done
-- Generating done
-- Build files have been written to: /home/myuser/xmr-stak/build
myuser@srv:~/xmr-stak/build$ make -j 16
Scanning dependencies of target xmr-stak-c
[  8%] Building C object CMakeFiles/xmr-stak-c.dir/xmrstak/backend/cpu/crypto/c_jh.c.o
[  8%] Building C object CMakeFiles/xmr-stak-c.dir/xmrstak/backend/cpu/crypto/c_blake256.c.o
[  8%] Building C object CMakeFiles/xmr-stak-c.dir/xmrstak/backend/cpu/crypto/c_skein.c.o
[ 11%] Building C object CMakeFiles/xmr-stak-c.dir/xmrstak/backend/cpu/crypto/c_groestl.c.o
[ 14%] Building C object CMakeFiles/xmr-stak-c.dir/xmrstak/backend/cpu/crypto/c_keccak.c.o
[ 17%] Linking C static library bin/libxmr-stak-c.a
[ 17%] Built target xmr-stak-c
Scanning dependencies of target xmr-stak-backend
[ 20%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/version.cpp.o
[ 22%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/jconf.cpp.o
[ 25%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/backendConnector.cpp.o
[ 28%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/misc/telemetry.cpp.o
[ 34%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/net/jpsock.cpp.o
[ 34%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/misc/executor.cpp.o
[ 37%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/http/httpd.cpp.o
[ 42%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/cpu/hwlocMemory.cpp.o
[ 42%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/cpu/jconf.cpp.o
[ 45%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/cpu/crypto/cryptonight_common.cpp.o
[ 48%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/cpu/minethd.cpp.o
[ 51%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/misc/console.cpp.o
[ 54%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/backend/globalStates.cpp.o
[ 57%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/http/webdesign.cpp.o
[ 60%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/misc/uac.cpp.o
[ 62%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/misc/utility.cpp.o
[ 65%] Building CXX object CMakeFiles/xmr-stak-backend.dir/xmrstak/net/socket.cpp.o
[ 68%] Linking CXX static library bin/libxmr-stak-backend.a
[ 68%] Built target xmr-stak-backend
[ 71%] Building NVCC (Device) object CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_extra.cu.o
[ 74%] Building NVCC (Device) object CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_core.cu.o
Scanning dependencies of target xmr-stak
Scanning dependencies of target xmrstak_opencl_backend
[ 77%] Building CXX object CMakeFiles/xmr-stak.dir/xmrstak/cli/cli-miner.cpp.o
[ 85%] Building CXX object CMakeFiles/xmrstak_opencl_backend.dir/xmrstak/backend/amd/jconf.cpp.o
[ 85%] Building CXX object CMakeFiles/xmrstak_opencl_backend.dir/xmrstak/backend/amd/amd_gpu/gpu.cpp.o
[ 85%] Building CXX object CMakeFiles/xmrstak_opencl_backend.dir/xmrstak/backend/amd/minethd.cpp.o
[ 88%] Linking CXX executable bin/xmr-stak
[ 88%] Built target xmr-stak
[ 91%] Linking CXX shared library bin/libxmrstak_opencl_backend.so
[ 91%] Built target xmrstak_opencl_backend
Scanning dependencies of target xmrstak_cuda_backend
[ 94%] Building CXX object CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/jconf.cpp.o
[ 97%] Building CXX object CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/minethd.cpp.o
[100%] Linking CXX shared library bin/libxmrstak_cuda_backend.so
[100%] Built target xmrstak_cuda_backend
myuser@srv:~/xmr-stak/build$

STEP 5) Execute XMR-STAK, but before you must execute some tuning exports for the GPU and the linux kernel

Now is the time to use XMR-STAK.

myuser@srv:~/xmr-stak/build$ sudo sysctl -w vm.nr_hugepages=128
myuser@srv:~/xmr-stak/build$ export GPU_FORCE_64BIT_PTR=0
myuser@srv:~/xmr-stak/build$ export GPU_MAX_HEAP_SIZE=100
myuser@srv:~/xmr-stak/build$ export GPU_USE_SYNC_OBJECTS=1
myuser@srv:~/xmr-stak/build$ export GPU_MAX_ALLOC_PERCENT=100
myuser@srv:~/xmr-stak/build$ export GPU_SINGLE_ALLOC_PERCENT=100
myuser@srv:~/xmr-stak/build$ cd ./bin
myuser@srv:~/xmr-stak/build/bin$ ls -la
total 35804
drwxrwxr-x 2 myuser myuser     4096 Jul 19 16:13 .
drwxrwxr-x 4 myuser myuser     4096 Jul 19 16:05 ..
-rw-rw-r-- 1 myuser myuser  1614230 Jul 19 16:05 libxmr-stak-backend.a
-rw-rw-r-- 1 myuser myuser    75182 Jul 19 16:05 libxmr-stak-c.a
-rwxrwxr-x 1 myuser myuser 32570440 Jul 19 16:13 libxmrstak_cuda_backend.so
-rwxrwxr-x 1 myuser myuser  1307456 Jul 19 16:06 libxmrstak_opencl_backend.so
-rwxrwxr-x 1 myuser myuser  1075784 Jul 19 16:06 xmr-stak
myuser@srv:~/xmr-stak/build/bin$ ./xmr-stak

Always execute the exports before running the xmr-stak binary.
As you can see the XMR-STAK is in

./bin/xmr-stak

under the “~/xmr-stak/build”.

STEP 6) Mining with XMR-STAK.

Execute the program and follow the first run questions. Here you can see a MONERO configuration and the first run – Mining Monero with xmr-stak 2.4 in a machine with NVIDIA video card under Ubuntu 18.04 LTS.

* Options of the XMR-STAK program

As you can see all the algo you can use with this program – new and classic (the old) criptonight coins to mine like Monero, Electroneum, IntenseCoin and many more!

myuser@srv:~/xmr-stak/build/bin$ ./xmr-stak --help
Usage: xmr-stak [OPTION]...
 
  -h, --help                 show this help
  -v, --version              show version number
  -V, --version-long         show long version number
  -c, --config FILE          common miner configuration file
  -C, --poolconf FILE        pool configuration file
  --benchmark BLOCKVERSION   ONLY do a benchmark and exit
  --benchwait WAIT_SEC             ... benchmark wait time
  --benchwork WORK_SEC             ... benchmark work time
  --noCPU                    disable the CPU miner backend
  --cpu FILE                 CPU backend miner config file
  --noAMD                    disable the AMD miner backend
  --noAMDCache               disable the AMD(OpenCL) cache for precompiled binaries
  --openCLVendor VENDOR      use OpenCL driver of VENDOR and devices [AMD,NVIDIA]
                             default: AMD
  --amd FILE                 AMD backend miner config file
  --noNVIDIA                 disable the NVIDIA miner backend
  --nvidia FILE              NVIDIA backend miner config file
  -i --httpd HTTP_PORT       HTTP interface port
 
The following options can be used for automatic start without a guided config,
If config exists then this pool will be top priority.
  -o, --url URL              pool url and port, e.g. pool.usxmrpool.com:3333
  -O, --tls-url URL          TLS pool url and port, e.g. pool.usxmrpool.com:10443
  -u, --user USERNAME        pool user name or wallet address
  -r, --rigid RIGID          rig identifier for pool-side statistics (needs pool support)
  -p, --pass PASSWD          pool password, in the most cases x or empty ""
  --use-nicehash             the pool should run in nicehash mode
  --currency NAME            currency to mine

Supported coin options: 
        - aeon7
        - bbscoin
        - bittube
        - cryptonight
        - cryptonight_bittube2
        - cryptonight_masari
        - cryptonight_haven
        - cryptonight_heavy
        - cryptonight_lite
        - cryptonight_lite_v7
        - cryptonight_lite_v7_xor
        - cryptonight_v7
        - cryptonight_v7_stellite
        - graft
        - haven
        - intense
        - masari
        - monero7
        - ryo
        - stellite
        - turtlecoin

Version: xmr-stak 2.4.7 c5f0505
Brought to by fireice_uk and psychocrypt under GPLv3.

* Troubleshooting – the right version of GNU GCC

You must use GNU GCC 6, because version above 6 (7 and 8) are not supported by the NVIDIA 9.1 Toolkit. So if you end up with this error you must change to the version to 6 from 7 (the default in Ubuntu 18.04 LTS is GNU 7.3.0). GNU GCC 6 was installed as a dependency to NVIDIA 9.1 Toolkit in STEP 2) and we exported two environments “CC” and “CCX” in STEP 4)

[ 85%] Building CXX object CMakeFiles/xmrstak_opencl_backend.dir/xmrstak/backend/amd/jconf.cpp.o
In file included from /usr/include/host_config.h:50:0,
                 from /usr/include/cuda_runtime.h:78,
                 from <command-line>:0:
/usr/include/crt/host_config.h:121:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
 #error -- unsupported GNU version! gcc versions later than 6 are not supported!
  ^~~~~
In file included from /usr/include/host_config.h:50:0,
                 from /usr/include/cuda_runtime.h:78,
                 from <command-line>:0:
/usr/include/crt/host_config.h:121:2: error: #error -- unsupported GNU version! gcc versions later than 6 are not supported!
 #error -- unsupported GNU version! gcc versions later than 6 are not supported!
  ^~~~~
CMake Error at xmrstak_cuda_backend_generated_cuda_core.cu.o.Release.cmake:219 (message):
  Error generating
  /home/myuser/xmr-stak/build/CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/./xmrstak_cuda_backend_generated_cuda_core.cu.o


CMakeFiles/xmrstak_cuda_backend.dir/build.make:63: recipe for target 'CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_core.cu.o' failed
make[2]: *** [CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_core.cu.o] Error 1
make[2]: *** Waiting for unfinished jobs....
CMake Error at xmrstak_cuda_backend_generated_cuda_extra.cu.o.Release.cmake:219 (message):
  Error generating
  /home/myuser/xmr-stak/build/CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/./xmrstak_cuda_backend_generated_cuda_extra.cu.o


CMakeFiles/xmrstak_cuda_backend.dir/build.make:70: recipe for target 'CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_extra.cu.o' failed
make[2]: *** [CMakeFiles/xmrstak_cuda_backend.dir/xmrstak/backend/nvidia/nvcc_code/xmrstak_cuda_backend_generated_cuda_extra.cu.o] Error 1
CMakeFiles/Makefile2:105: recipe for target 'CMakeFiles/xmrstak_cuda_backend.dir/all' failed
make[1]: *** [CMakeFiles/xmrstak_cuda_backend.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 88%] Linking CXX executable bin/xmr-stak
[ 88%] Built target xmr-stak
[ 91%] Linking CXX shared library bin/libxmrstak_opencl_backend.so
[ 91%] Built target xmrstak_opencl_backend
Makefile:129: recipe for target 'all' failed
make: *** [all] Error 2

The solution is simple as said, just export CC and CXX before compilation in the same terminal console:

myuser@srv:~$ export CC=/usr/bin/gcc-6
myuser@srv:~$ export CXX=/usr/bin/g++-6