Tag: gentoo

emerge – ERROR: 17.1 migration has not been performed – upgrade from 13 to 17.1

When upgrading to 17.1 profile you need to perform additional steps to get to a healthy Gentoo portage system using profile 17.1.
Here what we stumbled when we tried to upgrade from 13 to 17.1:

>>> Emerging (1 of 3) dev-libs/elfutils-0.177::gentoo
 * Fetching files in the background.
 * To view fetch progress, run in another terminal:
 * tail -f /var/log/emerge-fetch.log
 * elfutils-0.177.tar.bz2 BLAKE2B SHA512 size ;-) ...                                                                                                                [ ok ]
 * Please follow the instructions in the news item:
 * 2019-06-05-amd64-17-1-profiles-are-now-stable
 * or choose the 17.0 profile.
 * ERROR: dev-libs/elfutils-0.177::gentoo failed (setup phase):
 *   ERROR: 17.1 migration has not been performed!!
 * 
 * Call stack:
 *        ebuild.sh, line 591:  Called __source_all_bashrcs
 *        ebuild.sh, line 410:  Called __try_source '/usr/portage/profiles/default/linux/amd64/17.1/profile.bashrc'
 *        ebuild.sh, line 467:  Called __qa_source '/usr/portage/profiles/default/linux/amd64/17.1/profile.bashrc'
 *        ebuild.sh, line 112:  Called source '/usr/portage/profiles/default/linux/amd64/17.1/profile.bashrc'
 *   profile.bashrc, line   6:  Called die
 * The specific snippet of code:
 *              die "ERROR: 17.1 migration has not been performed!!"
 * 
 * If you need support, post the output of `emerge --info '=dev-libs/elfutils-0.177::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-libs/elfutils-0.177::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-libs/elfutils-0.177/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-libs/elfutils-0.177/temp/die.env'.
 * Working directory: '/var/tmp/portage/dev-libs/elfutils-0.177/homedir'
 * S: '/var/tmp/portage/dev-libs/elfutils-0.177/work/elfutils-0.177'

As you can see there is news, which consists of 12 steps to perform a successful upgrade to profile 17.1.
You can read the news with (the news output is included at the end of this article, just for completeness):

eselect news read 28

Specific note for our upgrade – in fact, our Gentoo installation was a chroot and it had not been updated for three years. The profile was 13.0 and portage supported only EAPI 5, which is a real problem. The new portage tree (we installed the latest) has no profile 13.0 and our emerge was half broken. We have preliminary steps described in Failed to install sys-apps/portage – Called pkg_preinst – portage._compat_upgrade.default_locations

Because we upgrade from 13.0 to 17.1, first, we need to perform the 17.0 upgrade (check the original steps here – https://gentoo.org/support/news-items/2017-11-30-new-17-profiles.html)

STEP 1) Select the previous profile 17.0 temporarily.

srv ~ # eselect profile set default/linux/amd64/17.0

You won’t be able to build any GNU GCC and you cannot continue with the next step.

STEP 2) Upgrade to GNU GCC 6.4.0 or later.

Our current GNU GCC is really old – 4.9.3. The latest version is gcc-9.2.0-r1 and we are going to use it.
Keep on reading!

Failed to install sys-apps/portage – Called pkg_preinst – portage._compat_upgrade.default_locations

Recently there was a problem with one compute node servers, which Gentoo (in fact, the chroot environment had not been updated for 3 years) could not emerge the new portage because it kept failing in the pkg_preinst with:

srv ~ # USE="-rsync-verify" emerge -va '=sys-apps/portage-2.3.76::gentoo'
.....
.....
>>> Installing (1 of 1) sys-apps/portage-2.3.76::gentoo
 * checking 1999 files for package collisions
1000 files checked ...
^[[B>>> Merging sys-apps/portage-2.3.76 to /
/usr/bin/python2.7: No module named _compat_upgrade
 * ERROR: sys-apps/portage-2.3.76::gentoo failed (preinst phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 133:  Called pkg_preinst
 *   environment, line 3238:  Called die
 * The specific snippet of code:
 *       env -u DISTDIR -u PORTAGE_OVERRIDE_EPREFIX -u PORTAGE_REPOSITORIES -u PORTDIR -u PORTDIR_OVERLAY PYTHONPATH="${D%/}${PYTHON_SITEDIR}${PYTHONPATH:+:${PYTHONPATH}}" "${PYTHON}" -m portage._compat_upgrade.default_locations || die;
 * 
 * If you need support, post the output of `emerge --info '=sys-apps/portage-2.3.76::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=sys-apps/portage-2.3.76::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/sys-apps/portage-2.3.76/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/sys-apps/portage-2.3.76/temp/environment'.
 * Working directory: '/var/tmp/portage/._portage_reinstall_.RKkxLW/pym'
 * S: '/var/tmp/portage/sys-apps/portage-2.3.76/work/portage-2.3.76'
!!! FAILED preinst: 1

>>> Failed to install sys-apps/portage-2.3.76, Log file:

>>>  '/var/tmp/portage/sys-apps/portage-2.3.76/temp/build.log'

Note this is the installation phase, where everything from the build process passes smoothly, but it fails to install it with some vague error of a dying line with a couple of python and portage environments. The problem is the current portage does not support EAPI 6 and 7 and some packages were unable to be installed!

And we fixed the issue by just using the method described in our previous article – Fix your broken or old portage in Gentoo, fix the emerge command
Cloning a new portage software and using it to emerge (install) a newer (or reinstall the current) portage in the system to fix the system’s portage and the emerge command:

cd /root
git clone https://anongit.gentoo.org/git/proj/portage.git
/root/portage/bin/emerge --sync
eselect profile set default/linux/amd64/17.1
/root/portage/bin/emerge -v portage

Now the latest and stable Gentoo profile is 17.1, so we set it to be sure we use it (in future this may change).
Keep on reading!

genkernel – ERROR: Unable to generate splash, ‘splash_geninitramfs’ was not found!

If you tried recently to build your kernel with genkernel (4.0.0_beta17, but you will have the same problem with 3.x) you may end up with the following error:

* ERROR: Unable to generate splash, 'splash_geninitramfs' was not found!

The splash_geninitramfs was part of the removed Gentoo ebuild package – “media-gfx/splashutils“. The software in the package splashutils had not been maintained for a really long time and eventually, it was removed. At present, there is no package offering the splash_geninitramfs. Still, the same functionality could be archived with “sys-boot/plymouth“, but the genkernel does not have the option to use it.
The genkernel command option, which triggers the error is “–splash” you must remove it. And check whether the following option in “/etc/genkernel.conf” is set to “no” (at least, in genkernel version 3.x is the default set to “yes”):

SPLASH="no"

When you instruct the genkernel not to add boot splash using splashutils the error will be gone and the kernel build will be successful.

The output error

We include the output error when using “–splash” with genkernel and the last part of genkernel.log.

livecd ~ # genkernel --splash --install --clean --no-mrproper --menuconfig all
* Gentoo Linux Genkernel; Version 4.0.0_beta17
* Using genkernel configuration from '/etc/genkernel.conf' ...
* Running with options: --splash --install --clean --no-mrproper --menuconfig all

* Working with Linux kernel 5.3.1-gentoo for x86_64
* Using kernel config file '/usr/share/genkernel/arch/x86_64/generated-config' ...
* 
* Note: The version above is subject to change (depends on config and status of kernel sources).

* kernel: >> Initializing ...
*         >> Running 'make clean' ...
*         >> --no-mrproper is set; Skipping 'make mrproper' ...
*         >> Running 'make oldconfig' ...
*         >> Invoking menuconfig ...
*         >> Re-running 'make oldconfig' due to changed kernel options ...
*         >> Kernel version has changed (probably due to config change) since genkernel start:
*            We are now building Linux kernel 5.3.1-gentoo-x86_64 for x86_64 ...
*         >> Compiling 5.3.1-gentoo-x86_64 bzImage ...
*         >> Compiling 5.3.1-gentoo-x86_64 modules ...
*         >> Installing 5.3.1-gentoo-x86_64 modules (and stripping) ...
*         >> Generating module dependency data ...
*         >> Saving config of successful build to '/etc/kernels/kernel-config-5.3.1-gentoo-x86_64' ...

* initramfs: >> Initializing ...
*         >> Appending devices cpio data ...
*         >> Appending base_layout cpio data ...
*         >> Appending auxilary cpio data ...
*         >> Appending blkid cpio data ...
*         >> Appending busybox cpio data ...
*         >> Appending mdadm cpio data ...
*         >> Appending modprobed cpio data ...
*         >> Appending splash cpio data ...
* ERROR: Unable to generate splash, 'splash_geninitramfs' was not found!
* Please consult '/var/log/genkernel.log' for more information and any
* errors that were reported above.
* 
* Report any genkernel bugs to bugs.gentoo.org and
* assign your bug to genkernel@gentoo.org. Please include
* as much information as you can in your bug report; attaching
* '/var/log/genkernel.log' so that your issue can be dealt with effectively.
* 
* Please do *not* report kernel compilation failures as genkernel bugs!

livecd ~ # cat /var/log/genkernel.log
.....
.....
*         >> Appending splash cpio data ...

* ERROR: Unable to generate splash, 'splash_geninitramfs' was not found!
* Please consult '/var/log/genkernel.log' for more information and any
* errors that were reported above.
* 
* Report any genkernel bugs to bugs.gentoo.org and
* assign your bug to genkernel@gentoo.org. Please include
* as much information as you can in your bug report; attaching
* '/var/log/genkernel.log' so that your issue can be dealt with effectively.
* 
* Please do *not* report kernel compilation failures as genkernel bugs!
* 

* mount: >> '/boot' is not a mountpoint; Nothing to restore ...
>>> Ended on: 2019-09-30 18:11:20 (after 0 days 0 hours 08 minutes 06 seconds)

emerge – cannot sync with gpg: keyserver refresh failed: General error because of wrong date

Trying to sync one of our virtual servers we got a sync error not able to refresh the OpenPGP keys. The virtual server was just resumed and it was OK before the pause. In addition, there were no errors in dmesg or some kind of kernel panics. All seemed to be working even the server was in the distributed compiling node and no problems there. But still, the emerge syncing the portage tree wasn’t possible!
And the problem was the date of our virtual server, which was 4 months behind the real date!

Check the time and date of the server – if it is behind or in the future with a big interval this is the root of the problems with the inability to refresh the GPG keys.

Just synchronize the clock of the server and be careful when you resume pause virtual servers! When you resume them you should synchronize the clock because in multiple environments the clock might be wrong!
We have multiple articles on the time syncronization topic – openntpd – immediately sync the clock on startup, simple time synchronization of a server (laptop, desktop) using built-in systemd-timesyncd service and more.

compile-local ~ # emerge --sync
>>> Syncing repository 'gentoo' into '/usr/portage'...
 * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
 * Refreshing keys from keyserver ...OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://hkps.pool.sks-keyservers.net
gpg: keyserver refresh failed: General error

^C

Exiting on signal Signals.SIGINT
compile-local ~ # date
Sat 08 Apr 2019 15:11:39 PM -00
compile-local ~ # /etc/init.d/ntpd restart
 * Starting OpenNTPD ...                                                                                                                                              [ ok ]
compile-local ~ # date
Sun 01 Sep 2019 08:02:34 AM -00
compile-local ~ #

Keep on reading!

openntpd – immediately sync the clock on startup

Here is our simple tip for your healthy server’s date and time:

Immediately synchronize the clock of your computer when using the openntpd (a lightweight version of ntpd with client-only mode).

Use the “-s” (lower “s” letter) to instruct the daemon ntpd to synchronize the clock immediately after it discovers a healthy time server! 

-s          Try to set the time immediately at startup, as opposed to slowly adjusting the clock.  ntpd will stay in the foreground for up to 15 seconds waiting
                 for one of the configured NTP servers to reply.

Find the start-up configuration file in your “/etc” (for your Linux distribution, its name is probably ntpd, for Gentoo it is “/etc/conf.d/ntpd”, the thing is to find the start-up confiuration script, not the ntpd.conf, which is the ntpd configuration file for the daemon) and include “-s” in the NTPD_OPTS:

cat /etc/conf.d/ntpd 
# /etc/conf.d/ntpd: config file for openntpd's ntpd

NTPD_OPTS="-s"

Restart the service.

If you use it in a virtualized environment like containers (docker, lxc, lxd and so on) and qemu, virtualbox, vmware and so on and you often suspend the machine to synchronize the clock when you resume it you must manually restart the openntpd service!!! Or you are going to wait for slowly adjusting the time as usual.

Information status

There is a utility to check what’s going on with the openntpd – ntpctl. It has only three read-only commands:

usage: ntpctl -s all | peers | Sensors | status

Keep on reading!

Install OpenStack swift client only

To manage files in the OpenStack cloud you need the swift client. This is not so tiny python tool (a lot of dependencies), which offers by means of files (as files are objects for the OpenStack):

  • delete – Delete files, directories and sub-directories. Be careful with a simple command you can delete all your files at once. “Delete a container or objects within a container.”
  • download – Download files to your local storage. “Download objects from containers.”
  • list – List all files in the main directory (i.e. the container), cannot be listed files in sub-directories. The output will be a file with path per line or you can enable extended output to show file size, time modified, the type of the file and the full file path. “Lists the containers for the account or the objects for a container.”
  • post – Creates containers and updates metadata, could create seurity keys for temporary urls – “Updates meta information for the account, container, or object; creates containers if not present.”
  • copy – Copy files to a new place within a container or to a different container. “Copies object, optionally adds meta”
  • stat – Display information for files, container and your account. No information is available per directory. Most typically you will read the file length, the type and modifcation date. “Displays information for the account, container, or object”
  • upload – uploads files and whole directories in a container – “Uploads files or directories to the given container.” You can read our article here – Upload files and directories with swift in OpenStack
  • capabilities – List the configuration options of your account like file size limits, file amount limits , limit requests per seconds and so on and which additional middleware your instance supports like bulk_delete, bulk_upload (the name are self-explainatory) – “List cluster capabilities.”
  • tempurl – Generate a temporary url for a file protected with time validity and a key – “Create a temporary URL.”
  • auth – Show your authentication token and storage url – “Display auth related environment variables.”

The text in the hyphens is from the swift help command. If you do not know what is a container with simple words these are the main sub-directories in your account if you use the list command without any argument (not adding a name after “list”):

myuser@myserver:~$ swift --os-username myusr --os-tenant-name myusr --os-password mypass --os-auth-url https://auth01.example.com:5000/v2.0 list
mycontainer1
anyname

The best way is to install the swift client from the package system of your Linux Distribution. The package system ensures the programs you install and upgrade are consistent within dependencies upgrades.

When you install this package “python-swiftclient” it depends on multiple python packages, which might be upgraded later, too, but the package system of a Linux distribution ensure the programs of python-swiftclient will work with them

. On the contrary, if you install it manually even with “pip” as it is offered in OpenStack site it is unsure what will happen and even your client program from “python-swiftclient” could stop working because of an upgraded dependancy library. Of course, the packages from the official package system could be a little bit older (but probably more stable) than the manual installs from the OpenStack official site. Still if you would like to install it with “pip” here you can check how you can do it: https://www.swiftstack.com/docs/integration/python-swiftclient.html
Keep on reading!

using portage eix for the first time – cannot open database file

Installing “app-portage/eix” in Gentoo to manage your portage updates you might encounter this error, when trying to use “eix” for the first time:

Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')

The chances are missing directory “/var/cache/eix/” or the user:group of the “/var/cache/eix/” is root:root, which is NOT right.

The user:group must be “portage:portage”.

So the solution is really simple:

mkdir -p /var/cache/eix
chown portage:portage /var/cache/eix

Output – the errors you might get

Using the eix-sync failed with:

root@srv1 ~ # eix-sync 
 * eix-cache does not exist
 * Running eix-update
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')
 * eix-update failed
 * Time statistics:
     6 seconds for initial eix-update
     6 seconds total

Using the “eix-update” failed, too.

root@srv ~ # eix-update 
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')

Output 2 – Successful update with eix

root@srv ~ # eix-update 
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
Database contains 19544 packages in 167 categories

Gentoo kde-frameworks/kdewebkit failed compilation with Qt5WebKit could not be found because dependency is required

Updating the KDE Plasma Desktop in our Gentoo workstations this time failed with 

CMake Error at /usr/share/cmake/Modules/CMakeFindDependencyMacro.cmake:48 (find_package):
  Found package configuration file:

    /usr/lib64/cmake/Qt5WebKit/Qt5WebKitConfig.cmake

  but it set Qt5WebKit_FOUND to FALSE so package "Qt5WebKit" is considered to
  be NOT FOUND.  Reason given by package:

  Qt5WebKit could not be found because dependency is required to have exact
  version 5.11.x.

It was strange because the previous emerge included the QT upgrade from old 5.11.2 to 5.12.1 and this dependency should have been resolved properly before:

emerge -vau $(qlist -IC|grep dev-qt|sort|uniq)

But apparently despite that the emerge built all QT libraries in dependency order the “dev-qt/qtwebkit” was built against the old QT libraries. And this is what is saying the above error!

The solution is really simple just rebuild the dev-qt/qtwebkit

root@srv ~ # emerge -va dev-qt/qtwebkit

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] dev-qt/qtwebkit-5.212.0_pre20180120:5/5.212::gentoo  USE="X geolocation hyphen jit multimedia opengl printsupport qml -gles2 -gstreamer -nsplugin 
-orientation -webp" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

Would you like to merge these packages? [Yes/No] yes

Keep on reading!

SSD cache device to a hard disk drive using LVM

This article is to show how simple is to use an SSD cache device to a hard disk drive. We also included statistics and graphs for several days of usage in one of our streaming servers.
Our setup:

  • 1 SSD disk Samsung 480G. It will be used for writeback cache device!
  • 1 Hard disk drive 1T

We included several graphs of this setup from one of our static media servers serving HLS video streaming.

The effectiveness of the cache is around 2-4 times at least!

Keep on reading!

Gentoo init – boot from linux software raid with superblock (metadata) version 1.2 and no autodetect

If you want to use software raid device in your Gentoo Linux system (init boot, not systemd) with boot partition resided in the raid for better redundancy but using metadata 1.2 version you won’t have autodetection feature on boot and there are some additional steps you may consider!

Linux kernel’s autodetection feature of the software raid is only for superblock (metadata) version 0.90!

If you create a software raid with superblock version 1, 1.1 or 1.2 you want to boot from it you must use:

  1. Grub 2, lilo won’t load the kernel from such paritions.
  2. include mdadm in initramfs
  3. initramfs booted with aditional parameter “domdadm” (in fact, it is not a “native” kernel option, read below)

Keep on reading!