Here we offer you a relatively new way of keeping your server’s time (or your computer and laptop) synchronized with a reliable time service on the Internet.
systemd has a built-in feature – a small daemon (systemd-timesyncd) to periodically to contact NTP servers and keep the server’s clock synchronized with them!
Of course, you must use systemd in your Linux distribution. This article is for those Linux systems using systemd, not for upstart (sysvinit, openrc, upstart, runit and so on). Most of the modern Linux distributions use the systemd like Fedora, Ubuntu, CentOS, RedHat, Gentoo, SuSe and many more.
Once there were not many options to keep your server’s clock synced with NTP servers. Now we have simpler programs (some of which by the way could act as clients only!!!) – chrony, openntpd, systemd-timesyncd and more.
This time synchronization service is not going to open server port 123, it does not have the server capabilities of an NTP server. So you won’t need any firewall rules (like for ntpd). It is a simple client service to sync your time and keep it synchronized all the time with accuracy not more than 100ms.
Do not expect complex clock discipline like training or compensating. It just sets the time according to a selected time server from the configuration file in “/etc/systemd/timesyncd.conf”. The polling interval is automatically adjusted in minimal and maximal values from the configuration file and the daemon decides which is the actual interval based on the near-term drift it thinks. Possible back running clock if it needs to set in the past. The quality of the clock source could not be checked, so
in any case, you may not expect more than 100ms accuracy.
Of course, this service is actively developed and it has already many changes from the base client once it was!
Here is how you can enable it. Here are the steps: Keep on reading!
This article is to show how simple is to use an SSD cache device to a hard disk drive. We also included statistics and graphs for several days of usage in one of our streaming servers.
Our setup:
1 SSD disk Samsung 480G. It will be used for writeback cache device!
1 Hard disk drive 1T
We included several graphs of this setup from one of our static media servers serving HLS video streaming.
The effectiveness of the cache is around 2-4 times at least!
If you encounter this error trying to install a pip module or compile a program under the console you surely miss OpenSSL development packages! pip also may build a packages in your system and it could depend on generic library headers like in this case OpenSSL, which the installer (pip) won’t bring them and it will output an error as you can see
myuser@srv # sudo pip install pycurl pygeoip psutil
Collecting pycurl
Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pygeoip in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): psutil in /usr/lib/python2.7/dist-packages
Building wheels for collected packages: pycurl
Running setup.py bdist_wheel for pycurl ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpqVNq1upip-wheel- --python-tag cp27:
Using curl-config (libcurl 7.47.0)
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/curl
copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
running build_ext
building 'pycurl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
In file included from src/docstrings.c:4:0:
src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Failed building wheel for pycurl
Running setup.py clean for pycurl
Failed to build pycurl
Installing collected packages: pycurl
Running setup.py install for pycurl ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile:
Using curl-config (libcurl 7.47.0)
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/curl
copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
running build_ext
building 'pycurl' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
In file included from src/docstrings.c:4:0:
src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AbCshS/pycurl/
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command
If you are in a situation when you want to upgrade your system, but do not want to upgrade a certain software in it you can just instruct apt not to upgrade these packages with:
apt-mark hold <package name(s)>
Here is how you can block updating 4 packages – ca-certificates, firefox, ghostscript, linux-firmware. First we update and upgrade and you can see there is no packages to keep back, and then we use apt-mark to “hold” package “linux-firmware” and ca-certificates, firefox, ghostscript at once. Initiating apt upgrade again will give you “The following packages have been kept back:” and it will include all packages, which will not be upgraded (it will include dependencies, which require some of the blocked packages). Keep on reading!
There are multiple reasons to rebuild the official kernel of a Linux distro but this is not the purpose of this article
just cannot miss the chance to write that all the kernels are built therefore optimized for the very first 64bit Intel/AMD processor! But come on who wants the most important piece of software to be optimized not for his new and expensive processor but for one released 15 years ago???
. Here we are going to show you how to recompile the latest official Ubuntu kernel of Ubuntu 16.04 LTS – the one, which comes with the apt packages system, because this kernel comes with the latest and greatest patches of the Ubuntu team. You should not confuse this howto with the one, which compile a vanilla kernel or the mainline kernel from http://kernel.ubuntu.com/~kernel-ppa/mainline/. So if you want a new kernel or the latest from kernel.org this is the right tutorial for Ubuntu – Build your own kernel under Ubuntu using mainline (latest) kernel. The official latest kernel in the Ubuntu repository is not always the latest one from kernel.org, but you can be sure it is probably most secure one, because there are additional modifications, configurations and tests by the team. Here you can see what versions of the kernel are the officials in Ubuntu: https://wiki.ubuntu.com/Kernel/Support
Here are the commands needed to load a new kernel without rebooting your server or desktop computer. Why you need this? As said in our first article for CentOS 7 – sometime rebooting a server could take 5 to 10 minutes and loading a new kernel is just up to a minute. In fact in most cases loading the new kernel and starting the system then is just under 20-30 seconds, so upgrading your server even with new kernel is super easy lately. We tested it on Ubuntu 16 and Ubuntu 18 servers and it was successful. The system uses systemd and the process is really easy and safe for the systems.
When the processes is initiated the system shutdowns normally (shutting down all running service with systemd) and then load the system immediately with the new kernel and starts the services as usual!
So no need to worry about unflushed data or not proper shutdown of a service! It’s like a normal reboot but without a hardware reboot and is a lot faster!
Here is what is required to load a kernel without hardware rebooting your computer box:
kexec-tools
Load the new kernel, initram file and the command line arguments with “kexec”
Here is a real world example with all the output:
And again update your system to see if there is a new kernel and install “kexec-tools”. In our case indeed there is a new kernel – vmlinuz-4.15.0-33-generic
myuser@srv:~$ sudo apt -y upgrade
Reading package lists... Done
Building dependency tree
Reading state information... Done
Calculating upgrade... Done
The following package was automatically installed and is no longer required:
libllvm5.0
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
amd64-microcode autotools-dev dblatex debhelper dh-strip-nondeterminism docbook-dsssl docbook-utils docbook-xml docbook-xsl fonts-lato fonts-lmodern fonts-texgyre
intel-microcode iucode-tool jadetex javascript-common kernel-common kernel-package kernel-patch-scripts kernel-wedge kerneloops kerneloops-applet kernelshark kerneltop
libfile-homedir-perl libfile-stripnondeterminism-perl libfile-which-perl libjs-jquery libllvm6.0 libmail-sendmail-perl libosp5 libostyle1c2 libpotrace0 libptexenc1
libqpdf21 libruby2.3 libsgmls-perl libsp1c2 libsynctex1 libsys-hostname-long-perl libtexlua52 libtexluajit2 libwebpdemux1 libxml2-utils libzzip-0-13
linux-headers-4.15.0-33 linux-headers-4.15.0-33-generic linux-image-4.15.0-33-generic linux-modules-4.15.0-33-generic linux-modules-extra-4.15.0-33-generic lmodern lynx
lynx-common openjade po-debconf preview-latex-style prosper ps2eps python-apt rake ruby ruby-did-you-mean ruby-minitest ruby-net-telnet ruby-power-assert ruby-test-unit
ruby2.3 rubygems-integration sgml-data sgmlspl sp tex-common tex-gyre texlive texlive-base texlive-bibtex-extra texlive-binaries texlive-extra-utils texlive-font-utils
texlive-fonts-recommended texlive-fonts-recommended-doc texlive-generic-recommended texlive-latex-base texlive-latex-base-doc texlive-latex-extra
texlive-latex-extra-doc texlive-latex-recommended texlive-latex-recommended-doc texlive-luatex texlive-math-extra texlive-pictures texlive-pictures-doc texlive-pstricks
texlive-pstricks-doc tipa trace-cmd xmlto xsltproc
The following packages will be upgraded:
.....
.....
Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
This may take some time... done.
Processing triggers for linux-image-4.15.0-33-generic (4.15.0-33.36~16.04.1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-4.15.0-33-generic
/etc/kernel/postinst.d/vboxadd:
VirtualBox Guest Additions: Building the VirtualBox Guest Additions kernel modules.
/etc/kernel/postinst.d/zz-update-grub:
Generating grub configuration file ...
Warning: Setting GRUB_TIMEOUT to a non-zero value when GRUB_HIDDEN_TIMEOUT is set is no longer supported.
Found linux image: /boot/vmlinuz-4.15.0-33-generic
Found initrd image: /boot/initrd.img-4.15.0-33-generic
Found linux image: /boot/vmlinuz-4.13.0-36-generic
Found initrd image: /boot/initrd.img-4.13.0-36-generic
Found memtest86+ image: /boot/memtest86+.elf
Found memtest86+ image: /boot/memtest86+.bin
done
myuser@srv:~$ sudo apt -y install kexec-tools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following package was automatically installed and is no longer required:
libllvm5.0
Use 'sudo apt autoremove' to remove it.
The following NEW packages will be installed:
kexec-tools
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 77,4 kB of archives.
After this operation, 276 kB of additional disk space will be used.
Get:1 http://bg.archive.ubuntu.com/ubuntu xenial-updates/main amd64 kexec-tools amd64 1:2.0.16-1ubuntu1~16.04.1 [77,4 kB]
Fetched 77,4 kB in 0s (707 kB/s)
Preconfiguring packages ...
Selecting previously unselected package kexec-tools.
(Reading database ... 253895 files and directories currently installed.)
Preparing to unpack .../kexec-tools_1%3a2.0.16-1ubuntu1~16.04.1_amd64.deb ...
Unpacking kexec-tools (1:2.0.16-1ubuntu1~16.04.1) ...
Processing triggers for man-db (2.7.5-1) ...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for ureadahead (0.100.0-19) ...
Setting up kexec-tools (1:2.0.16-1ubuntu1~16.04.1) ...
Generating /etc/default/kexec...
Processing triggers for systemd (229-4ubuntu21.4) ...
Processing triggers for ureadahead (0.100.0-19) ...
┌──────────────────────────────────────────────────────────────────┤ Configuring kexec-tools ├───────────────────────────────────────────────────────────────────┐
│ │
│ If you choose this option, a system reboot will trigger a restart into a kernel loaded by kexec instead of going through the full system boot loader process. │
│ │
│ Should kexec-tools handle reboots (sysvinit only)? │
│ │
│ <Yes> <No> │
│ │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
On the above configuration question mark “” and press Enter.
So we performed an update and there was a new kernel vmlinuz-4.15.0-33-generic, which we would like to load without hardware reboot.
Here is our new kernel in “/boot”
Now we know the kernel and initram file names we just check the kernel arguments in the kernel, load them with kexec and start an systemd target to load the new kernel:
myuser@srv:~$ grep vmlinuz-4.15.0-33-generic /boot/grub/grub.cfg
linux /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro quiet splash $vt_handoff
linux /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro quiet splash $vt_handoff
linux /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro quiet splash $vt_handoff init=/sbin/upstart
linux /boot/vmlinuz-4.15.0-33-generic root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro recovery nomodeset
myuser@srv:~$ sudo kexec -l /boot/vmlinuz-4.15.0-33-generic --initrd=/boot/initrd.img-4.15.0-33-generic --command-line="root=UUID=061b2936-34bf-4da3-b7d2-b8bde0899f03 ro quiet splash"
myuser@srv:~$ sudo systemctl start kexec.target
Connection to srv closed by remote host.
Connection to srv closed.
Use the first line of the grep output above (or you can cat the file and see what is in it if you have any doubts) to take the proper kernel boot arguments and do not include anything starting with “$”.
As you can see systemd performs a normal shutdown of all services and targets. Normal shutdown
The ssh connection is immediately closed because the reboot is initiated.
After 10-15 seconds our host is alive and the new kernel is loaded successfully:
root@test ~ $ ssh root@srv
root@srv's password:
Last login: Wed Sep 5 17:15:08 2018 from test
[root@srv ~]# uname -a
Linux srv.local 4.15.0-33-generic #36~16.04.1-Ubuntu SMP Wed Aug 15 17:21:05 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]#
Here we will show how to build your own kernel under Ubuntu 16/17/18. We are going to use
mainline kernel
because most of the cases when we need to build our own is when we need some new feature presented in the new kernels. So Ubuntu mainline kernels are selected new kernels, which are packed in deb packets for simplicity and can be used to test new features, but they are not supported by Ubuntu teams and you should test them really carefully if you want to use them in production!
The mainline kernels are here: http://kernel.ubuntu.com/~kernel-ppa/mainline/. This is the main address, where you could download the latest kernels and some other legacy one if you have old system.
This howto is based on https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel and https://wiki.ubuntu.com/Kernel/MainlineBuilds – we summarize the commands needed to build your new and latest kernel and show you some typical mistakes and problems during the process.
To show you more information our purpose will be to build the one of the latest kernels with low latency features enabled (Ubuntu has such type of kernel configuration)! And here is one of the most important dependency, which is not explicitly included or even mentioned in most of the tutorials for building your own kernels –
you MUST include new or even latest Linux firmware package
All Linux distros have a package or packages containing files with different firmware for multiple devices supported by the kernel modules. Basically firmware is used by a kernel module and it can be a micro code (program), which is uploaded in the device or just complete the kernel module to function properly with a device. Recent days firmware is like a black box the manufacturer of the devices pack a proprietary code in a firmware blob and the source code is unknown to the community.
to install NVIDIA Cuda we need the NVIDIA proprietary driver working (and the libraries and files it provides). Ubuntu and most linux distributions come with the open source driver nouveau, but it is with limited performance and capabilities. For gaming and development you will need the official NVIDIA driver, the SDK and many additional proprietary libraries.
This howto is a sort of continuing the Install NVIDIA proprietary drivers on Ubuntu 18.0 LTS (x86_64) where we install the proprietary driver from the unofficial community driven repository “Graphics Drivers” team. This is Personal Package Archives (PPA) repository! So you should check it, too and be careful about this repository as the maintainers say. For completeness will include all the steps to install the NVIDIA proprietary driver (and for more details for this part just read the specific howto we offer above) and Cuda.
Here are the steps to install NVIDIA CUDA on Ubuntu 18.0 LTS (x86_64) with the help of ppa:graphics-drivers/ppa
STEP 1) Update your current repositories, add the new repository, update again and install the NVIDIA driver with the tool ubuntu-drivers
As you can see the nvidia 396 driver is going to be installed in our system. You can see all the additional (some of which are dependencies) packages to be installed.
After the reboot you should see something similar in your dmesg:
The new Ubuntu release is here and as usual it comes with nouveau kernel module for your NVIDIA GPU. Probably this driver is good for everyday office use, but if you like to play games you will need the proprietary driver from NVIDIA for sure! Here are the steps to install the official drivers from NVIDIA with the help of the unofficial community driven repository “Graphics Drivers” team. This is Personal Package Archives (PPA) repository! DKMS is used for building the NVIDIA kernel module when a new kernel is installed. So you do not have to worry if you update your kernel.
Be careful as it is stated in the repository:
Important: The contents of Personal Package Archives are not checked or monitored. You install software from them at your own risk.
and
You can update your system with unsupported packages from this untrusted PPA by adding …
Still till now no major problems in this repository from its creation and if you just have a home computer and you are a novice in Ubuntu (linux) world you can use this approach of installing NVIDIA proprietary drivers. There is another way of installing the NVIDIA driver from an official source and we are going to cover it soon (coming soon).
STEP 1) Update your system.
Always do this step before installing new software or repositories!
myuser@srv:~$ sudo apt update -y
STEP 2) Add the new repository, which has the nvidia proprietary driver – ppa:graphics-drivers/ppa
As you can see we use a tool for installing drivers under Ubuntu:
ubuntu-drivers
which is part of this package “ubuntu-drivers-common” and it should be installed by default if missing just install it with apt (look below for more information).
As you can see the nvidia 396 driver is going to be installed in our system. We included part of the output – the packages, which will be installed. When the setup is ready reboot your system to load the newly installed driver.
We use technologies like cookies to store and/or access device information. We do this to improve browsing experience and to show (non-) personalized ads. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
