Patch and resume compilation of a failed package in Gentoo – ebuild, local repository or ctrl+Z

A dependency package failed to compile throwing error and existing the emerge of a queue with a hundred and more packages. Or worse you installed a new version of a package and multiple rebuilds are pulled, but one of the dependencies fails and you may end up with a broken system? What can you do? There is no new version of the failed package and yes, there is a bug in the Gentoo’s Bugzilla – https://bugs.gentoo.org/. And there is a solution with a patch, which has not made its way to the production and in Gentoo portage yet.
The package in the portage is broken, no new fixed package is released, but there is a patch to fix your issue. Here is what you can do:

  • Make your own package with the fixed version of the original package and put it in your local repository (not the official one, because on every emerge –sync it will be deleted). You should make a local repository and put the ebuild and all necessary files.
  • Or just download the patch and patch the source in the directory, which still holds the source of the failed package and resume the compilation manually. Then install it. Using this tutorial – Resume installation after a package build error, when emerging firefox under Gentoo
  • Just after the uncompress operation of the emerge press CTRL+Z to put the operation in the background and download and patch. Then bring back the emerge from the background with “fg” command.

The second and third options are not permanent solutions, but they are fast enough to be used in some situations.
Here are steps for the first and second option you may have:

OPTION 1) Make your own package.

Create a local repository (for details Simple steps to create Gentoo custom repository and add a package):

root@srv ~ # mkdir -p /var/db/repos/my-local-portage/{metadata,profiles}
root@srv ~ # cat  << 'EOF' > /var/db/repos/my-local-portage/metadata/layout.conf
masters = gentoo
auto-sync = false
EOF
root@srv ~ # cat  << 'EOF' > /etc/portage/repos.conf/my-local-portage.conf
[my-local-portage]
location = /var/db/repos/my-local-portage 
EOF
root@srv ~ # cat  << 'EOF' > /var/db/repos/my-local-portage/profiles/repo_name
my-local-portage
EOF

Copy the ebuild file of the package you want to modify in the custom repository directory created above (it’s a good idea to copy all the sub-directories, too):
Keep on reading!

Simple steps to create Gentoo custom repository and add a package

Creating a custom repository would give you a chance to fast edit (ebuild) files of existing packages and drop better versions to the custom repository, which then will be used to install in the system. Here is the simplest way to create a Gentoo custom repository without installing any mandatory software. You may check the two Gentoo articles on the subject – https://wiki.gentoo.org/wiki/Custom_repository, which uses repoman (and additional software to install) and https://wiki.gentoo.org/wiki/Handbook:AMD64/Portage/CustomTree#Defining_a_custom_repository, which is part of a bigger article and without a clear example with a package as we are going to show.
Our custom repository name is “my-local-portage”.

STEP 1) Create the directories and basic configuration files for the new custom repository

Just two mandatory directories.

mkdir -p /var/db/repos/my-local-portage/{metadata,profiles}

The minimal configuration in two files:

cat  << 'EOF' > /var/db/repos/my-local-portage/metadata/layout.conf
masters = gentoo
auto-sync = false
EOF
cat  << 'EOF' > /var/db/repos/my-local-portage/profiles/repo_name
my-local-portage
EOF

Fix the permissions

chown -R portage:portage /var/db/repos/my-local-portage

The custom repository is set up. Now only the emerge should get the configuration to check for it in the next step.

STEP 2) Portage global configuration.

Add a file pointing for your custom repository in the Portage global configuration directory “/etc/portage”:

cat  << 'EOF' > /etc/portage/repos.conf/my-local-portage.conf
[my-local-portage]
location = /var/db/repos/my-local-portage
EOF

STEP 3) Add a package in the new custom repository.

The package version may be the same version as in the official Gentoo repository, but the package form the custom repository will be used if no repository is included in the “emerge” command.
For simplicity, we are going not to modify the ebuild file of a copied official package, but the idea is to copy an existing ebuild file and then change it for the user’s needs and the steps are the same as follow.
Just copy the file (and edit it). The package “app-text/calibre” was randomly selected for the example.

mkdir /var/db/repos/my-local-portage/app-text/calibre
cp /usr/portage/app-text/calibre/calibre-4.9.1-r1.ebuild /var/db/repos/my-local-portage/app-text/calibre/

Create the manifest files and you are ready:

cd /var/db/repos/my-local-portage/app-text/calibre/
ebuild calibre-4.9.1-r1.ebuild manifest
>>> Downloading 'ftp://ftp.free.fr/mirrors/ftp.gentoo.org/distfiles/2e/calibre-4.9.1.tar.xz'
--2020-02-06 18:04:59--  ftp://ftp.free.fr/mirrors/ftp.gentoo.org/distfiles/2e/calibre-4.9.1.tar.xz
           => '/usr/portage/distfiles/calibre-4.9.1.tar.xz.__download__'
Resolving ftp.free.fr... 212.27.60.27, 2a01:e0c:1:1598::1
Connecting to ftp.free.fr|212.27.60.27|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /mirrors/ftp.gentoo.org/distfiles/2e ... done.
==> SIZE calibre-4.9.1.tar.xz ... 37529656
==> PASV ... done.    ==> RETR calibre-4.9.1.tar.xz ... done.
Length: 37529656 (36M) (unauthoritative)

calibre-4.9.1.tar.xz                       100%[========================================================================================>]  35.79M  5.53MB/s    in 8.0s    

2020-02-06 18:05:08 (4.48 MB/s) - '/usr/portage/distfiles/calibre-4.9.1.tar.xz.__download__' saved [37529656]

>>> Creating Manifest for /var/db/repos/my-local-portage/app-text/calibre

Fix the permissions with

chown -R portage:portage /var/db/repos/my-local-portage

The manifest file contains the hash of the ebuild file and all the additional files if any (for this package there are no additional files). All file needed for the operation will be downloaded so they must be network accessed in the time of executing the command (except in the cases when they have already existed in the distfiles directory and/or the subdirectories of the /var/db/repos/my-local-portage/app-text/calibre)
Keep on reading!

more than the default 4 parallel processes using distributed compiling with distcc

Distributed compilation could greatly speed the build process of Gentoo packages (and not only Gentoo, of course). If you tend to use Gentoo on a laptop or a relatively old CPU you may want to build packages distributively across multiple hosts.
Different (Linux) distributions use different configurations and environment scheme and sometimes it is difficult to sift the configuration, which could be applied to your setup. This is not a tutorial on how to enable parallel processing in Gentoo but it is just our client-site setup.

By default, there is a limit of 4 parallel processes, which is utterly insufficient, because nowadays most servers have more than 8 cores/logical compute units (not to mention that probably most would have 16 and above cores compute units).

The environment variable DISTCC_HOSTS controls, which hosts will receive files for the compilation of what they support and what is the limit of parallel processes.

In Gentoo we set this variable in the /etc/portage/make.conf. Here what you may include in make.conf to have 16 parallel remote processes and up to maximum 4 local (if the remote fails):

MAKEOPTS="-j16 -l4"
FEATURES="distcc"
DISTCC_HOSTS="192.168.0.101/16"

We use the environment DISTCC_HOSTS (here in Gentoo put in the make.conf, but in another Linux distribution an environment variable with this name should be set) because it is easy to set up and control globally for the Gentoo emerge system.
According to the documents:

In order, distcc looks in the $DISTCC_HOSTS environment variable, the user’s $DISTCC_DIR/hosts file, and the system-wide host file.

So when using emerge to build the packages, the emerge will rely on $DISTCC_HOSTS in make.conf (/etc/portage/make.conf or /etc/make.conf if you still use the old path), “/var/tmp/portage/.distcc/” (the build process uses “portage” user and group, not root!) and “/etc/distcc/hosts”. The first option used in the order above will be set the hosts and the limitation for the distributed processing. So if you use $DISTCC_HOSTS in make.conf (or environment) you wouldn’t need to set the “hosts” file.
Separate the different hosts with white space if you have more than one and always use the notation “/LIMIT” for each host. The default value is only 4 parallel processes (i.e it is implicitly added /4 to each hosts in the configuration!)
Keep on reading!

Q_WEBENGINECORE_EXPORT QWebEngineFindTextResult has initializer but incomplete type

In addition to one of the cstdlib fix here – Gentoo building qtgui error – g++-v8/cstdlib:75:15: fatal error: stdlib.h: No such file or directory we were unable to to build “dev-qt/qtwebengine-5.14.1“.

It appeared the problem was a wrong order of header includes just like the cstdlib and if you have used the fix with QMAKE_CFLAGS_ISYSTEM in /usr/lib64/qt5/mkspecs/common/gcc-base.conf you would encounter this error. Revert back the original value in /usr/lib64/qt5/mkspecs/common/gcc-base.conf (and probably it is a good idea to rebuild the entire system with “emerge -e”):

QMAKE_CFLAGS_ISYSTEM = -isystem

and continue the build from the point you’ve stopped with something like:

ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild compile
ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild install
ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild qmerge
rm -Rf /var/tmp/portage/dev-qt/qtwebengine-5.14.1/

Keep on reading!

env python3.5: no such file or directory

The dependency conflicts may require a lot of time to resolve. On the contrary, uninstalling old software is easy, but it might lead to multiple broken programs and strange errors. Here is one of them, after removing the old python 3.4 and 3.5 (or whatever version it prints in your case) under Gentoo despite the rebuilding all dependencies as required by the emerge command for the new python 3.6 an error occurred trying to emerge the “app-misc/geoclue-2.5.3-r2

[20/75] /usr/lib/python-exec/python3.6/meson --internal exe --unpickle /var/tmp/portage/app-misc/geoclue-2.5.3-r2/work/geoclue-2.5.3-build/meson-private/meson_exe_glib-mkenums_af4feac41c42ec7289410f8b20070c0528a3a7c0.dat
FAILED: public-api/gclue-enum-types.c 
/usr/lib/python-exec/python3.6/meson --internal exe --unpickle /var/tmp/portage/app-misc/geoclue-2.5.3-r2/work/geoclue-2.5.3-build/meson-private/meson_exe_glib-mkenums_af4feac41c42ec7289410f8b20070c0528a3a7c0.dat
/usr/bin/env: âpython3.5â: No such file or directory
ninja: build stopped: subcommand failed.
 * ERROR: app-misc/geoclue-2.5.3-r2::gentoo failed (compile phase):
 *   ninja -v -j15 -l14 -C /var/tmp/portage/app-misc/geoclue-2.5.3-r2/work/geoclue-2.5.3-build failed
 * 
 * Call stack:
 *     ebuild.sh, line  125:  Called src_compile
 *   environment, line 2847:  Called meson_src_compile
 *   environment, line 1904:  Called eninja '-C' '/var/tmp/portage/app-misc/geoclue-2.5.3-r2/work/geoclue-2.5.3-build'
 *   environment, line 1274:  Called die
 * The specific snippet of code:
 *       "$@" || die "${nonfatal_args[@]}" "${*} failed"
 * 

Despite the lines starting with “/usr/lib/python-exec/python3.6/meson”, there is an error for missing “python 3.5“.

Here is another example, just trying to execute the distcc-config and it immediately throws the error for missing “python 3.5“.

root@srv ~ # distcc-config 
/usr/bin/env: ‘python3.5’: No such file or directory

The python script distcc-config starts with:

root@srv ~ $ head /usr/bin/distcc-config
#!/usr/bin/env python3.5
# Copyright 1999-2018 Gentoo Authors
# Distributed under the terms of the GNU General Public License v2

import os, re, signal, subprocess, sys

options=[
        '--get-hosts',
        '--set-hosts',
        '--get-verbose',

Apparently, multiple programs use this technic to explicitly put a specific version of python in the header of the python script to be used. Thus grantees the version in the header will be used even you change the system or user-preferred Python interpreter.

But uninstalling this python version all these programs will stop working till you rebuild them with emerge.
Keep on reading!

display packages from the Gentoo emerge resume list

A quick tip and an easy hack to see the emerge resume list, which will be used by the “emerge –resume” command. There are two resume lists – resume and resume_backup in the binary file /var/cache/edb/mtimedb.
The first two commands are python 3+.
The resume and resume_backup lists

python -c 'import portage; print(portage.mtimedb.get("resume", {}).get("mergelist"))'
python -c 'import portage; print(portage.mtimedb.get("resume_backup", {}).get("mergelist"))'

If you need the command for the python 2 you should use the following:

python -c 'import portage; print portage.mtimedb.get("resume", {}).get("mergelist")'
python -c 'import portage; print portage.mtimedb.get("resume_backup", {}).get("mergelist")'

Real world example

Resume list for

desktop ~ # emerge -v --nodeps $(qlist -IC|grep dev-qt)

These are the packages that would be merged, in order:

[ebuild     U  ] dev-qt/designer-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="declarative webkit -debug -test" 8,493 KiB
[ebuild     U  ] dev-qt/linguist-tools-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="qml -debug -test" 0 KiB
[ebuild     U  ] dev-qt/qdbus-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qtbluetooth-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -qml -test" 2,727 KiB
[ebuild   R    ] dev-qt/qtchooser-66::gentoo  USE="-test" 32 KiB
[ebuild     U  ] dev-qt/qtconcurrent-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 48,549 KiB
[ebuild     U  ] dev-qt/qtcore-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="icu -debug (-systemd) -test" 0 KiB
[ebuild     U  ] dev-qt/qtdbus-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qtdeclarative-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="jit widgets -debug -gles2 -localstorage -test" 20,753 KiB
[ebuild     U  ] dev-qt/qtgraphicaleffects-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 13,713 KiB
[ebuild     U  ] dev-qt/qtgui-5.14.0-r1:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="accessibility dbus egl evdev gif ibus jpeg libinput png udev vnc wayland%* xcb -debug -eglfs -gles2 -test -tslib -tuio" 0 KiB
[ebuild     U  ] dev-qt/qthelp-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qtimageformats-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="mng -debug -test (-jpeg2k%)" 1,768 KiB
[ebuild     U  ] dev-qt/qtlocation-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 5,974 KiB
[ebuild   R    ] dev-qt/qtlockedfile-2.4.1_p20171024::gentoo  USE="-doc" 694 KiB
[ebuild     U  ] dev-qt/qtmultimedia-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="alsa gstreamer openal pulseaudio qml widgets -debug -gles2 -test" 3,671 KiB
[ebuild     U  ] dev-qt/qtnetwork-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="libproxy networkmanager ssl -bindist -connman -debug -sctp -test" 0 KiB
[ebuild     U  ] dev-qt/qtopengl-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -gles2 -test" 0 KiB
[ebuild     U  ] dev-qt/qtpaths-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qtpositioning-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="qml -debug -geoclue -test" 0 KiB
[ebuild     U  ] dev-qt/qtprintsupport-5.14.0-r1:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="cups -debug -gles2 -test" 0 KiB
[ebuild     U  ] dev-qt/qtquickcontrols2-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test -widgets" 7,947 KiB
[ebuild     U  ] dev-qt/qtquickcontrols-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="widgets -debug -test" 5,843 KiB
[ebuild     U  ] dev-qt/qtscript-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="jit scripttools -debug -test" 2,584 KiB
[ebuild     U  ] dev-qt/qtsensors-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -qml -test" 1,996 KiB
[ebuild   R    ] dev-qt/qtsingleapplication-2.6.1_p20171024::gentoo  USE="X -doc" 0 KiB
[ebuild     U  ] dev-qt/qtspeech-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 99 KiB
[ebuild     U  ] dev-qt/qtsql-5.14.0:5/5.14.0::gentoo [5.12.2:5/5.12.2::gentoo] USE="mysql sqlite -debug -freetds -oci8 -odbc -postgres -test" 0 KiB
[ebuild     U  ] dev-qt/qtsvg-5.14.0-r1:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 1,822 KiB
[ebuild     U  ] dev-qt/qttest-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qttranslations-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 1,302 KiB
[ebuild     U  ] dev-qt/qtvirtualkeyboard-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="spell xcb -debug -handwriting -test" 10,705 KiB
[ebuild     U  ] dev-qt/qtwayland-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="libinput xcomposite -debug -test" 532 KiB
[ebuild     U  ] dev-qt/qtwebchannel-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="qml -debug -test" 192 KiB
[ebuild     U  ] dev-qt/qtwebengine-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="alsa pulseaudio system-icu widgets -bindist -debug -designer -jumbo-build -pax_kernel -system-ffmpeg -test (-geolocation%*)" 235,904 KiB
[ebuild     U  ] dev-qt/qtwebkit-5.212.0_pre20190629:5/5.212::gentoo [5.212.0_pre20180120:5/5.212::gentoo] USE="X geolocation hyphen jit multimedia opengl printsupport qml -gles2 -gstreamer -nsplugin -orientation -webp" 12,166 KiB
[ebuild     U  ] dev-qt/qtwidgets-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="gtk png xcb -debug -gles2 -test" 0 KiB
[ebuild     U  ] dev-qt/qtx11extras-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 125 KiB
[ebuild     U  ] dev-qt/qtxml-5.14.0:5/5.14::gentoo [5.12.2:5/5.12::gentoo] USE="-debug -test" 0 KiB
[ebuild     U  ] dev-qt/qtxmlpatterns-5.14.0:5/5.14::gentoo [5.11.1:5/5.11::gentoo] USE="-debug -qml% -test" 1,362 KiB

Total: 40 packages (37 upgrades, 3 reinstalls), Size of downloads: 388,941 KiB


>>> Verifying ebuild manifests
>>> Running pre-merge checks for dev-qt/qtwebkit-5.212.0_pre20190629

>>> Emerging (1 of 40) dev-qt/designer-5.14.0::gentoo
 * Fetching files in the background.
 * To view fetch progress, run in another terminal:
 * tail -f /var/log/emerge-fetch.log
 * qttools-everywhere-src-5.14.0.tar.xz BLAKE2B SHA512 size ;-) ...                                                                                                  [ ok ]
!!! Failed to set new SELinux execution context. Is your current SELinux context allowed to run Portage?
>>> Unpacking source...
>>> Unpacking qttools-everywhere-src-5.14.0.tar.xz to /var/tmp/portage/dev-qt/designer-5.14.0/work
^C

Exiting on signal 2
sandbox:stop  caught signal 2 in pid 22886
sandbox:stop  Send signal 4 more times to force SIGKILL
Sandboxed process killed by signal: Interrupt
^C * The ebuild phase 'die_hooks' has been killed by signal 2.

 * Messages for package dev-qt/designer-5.14.0:

We interrupted it on purpose with Ctrl+C.

The output of the above python command:

desktop ~ # python -c 'import portage; print(portage.mtimedb.get("resume", {}).get("mergelist"))'
[['ebuild', '/', 'dev-qt/designer-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/linguist-tools-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qdbus-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtbluetooth-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtchooser-66', 'merge'], ['ebuild', '/', 'dev-qt/qtconcurrent-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtcore-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtdbus-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtdeclarative-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtgraphicaleffects-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtgui-5.14.0-r1', 'merge'], ['ebuild', '/', 'dev-qt/qthelp-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtimageformats-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtlocation-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtlockedfile-2.4.1_p20171024', 'merge'], ['ebuild', '/', 'dev-qt/qtmultimedia-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtnetwork-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtopengl-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtpaths-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtpositioning-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtprintsupport-5.14.0-r1', 'merge'], ['ebuild', '/', 'dev-qt/qtquickcontrols2-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtquickcontrols-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtscript-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtsensors-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtsingleapplication-2.6.1_p20171024', 'merge'], ['ebuild', '/', 'dev-qt/qtspeech-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtsql-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtsvg-5.14.0-r1', 'merge'], ['ebuild', '/', 'dev-qt/qttest-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qttranslations-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtvirtualkeyboard-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtwayland-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtwebchannel-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtwebengine-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtwebkit-5.212.0_pre20190629', 'merge'], ['ebuild', '/', 'dev-qt/qtwidgets-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtx11extras-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtxml-5.14.0', 'merge'], ['ebuild', '/', 'dev-qt/qtxmlpatterns-5.14.0', 'merge']]
desktop ~ # python -c 'import portage; print(portage.mtimedb.get("resume_backup", {}).get("mergelist"))'
None

gentoo emerge resume – Invalid resume list and clean the list

Under Gentoo and emerge it will eventually end up using “–resume” paired with “–skipfirst” emerge command, which should resume last list of packages to build without the first one. Each execution of emerge command should reset the list but apparently it might not be the case (in fact two lists are kept – resume and resume_backup, read below).
The command just reads a list of packages and emerges them – it’s fast and it does not go through the whole process of dependancy check! But sometimes the list could be old or get corrupted or it just throws errors for “Invalid resume list“:

dekstop ~ # emerge --resume --skipfirst

These are the packages that would be merged, in order:

Calculating dependencies... done!
 * Invalid resume list:
 * 
 *   ('ebuild', '/', 'kde-frameworks/kpty-5.66.0', 'merge')
 *   ('ebuild', '/', 'kde-plasma/libkscreen-5.17.5', 'merge')
 *   ('ebuild', '/', 'kde-apps/kmbox-19.12.1', 'merge')
 *   ('ebuild', '/', 'kde-frameworks/kdesignerplugin-5.66.0', 'merge')
 *   ('ebuild', '/', 'kde-plasma/kwayland-integration-5.17.5', 'merge')
 *   ('ebuild', '/', 'kde-plasma/xembed-sni-proxy-5.17.5', 'merge')
 *   ('ebuild', '/', 'net-libs/accounts-qml-0.7-r1', 'merge')
 *   ('ebuild', '/', 'kde-apps/akonadi-notes-19.12.1', 'merge')
....
....
 * One or more packages are either masked or have missing dependencies:
 * 
 *   dev-lang/python:3.4[xml(+)] pulled in by:
 *     (dev-python/setuptools-40.6.2:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.5[xml(+)] pulled in by:
 *     (dev-python/setuptools-40.6.2:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.4 pulled in by:
 *     (dev-python/certifi-2018.8.24:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.5 pulled in by:
 *     (dev-python/certifi-2018.8.24:0/0::gentoo, installed)
 * 
 *   dev-libs/openssl:0/0= pulled in by:
 *     (dev-lang/python-2.7.17-r1:2.7/2.7::gentoo, installed)
 * 
 *   dev-lang/python:3.5 pulled in by:
 *     (dev-util/glib-utils-2.56.2:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.4 pulled in by:
 *     (sys-apps/file-5.35:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.5 pulled in by:
 *     (sys-apps/file-5.35:0/0::gentoo, installed)
 * 
 *   >=dev-libs/icu-58.1:0/60.2= pulled in by:
 *     (dev-lang/spidermonkey-52.9.1_pre1:52/52::gentoo, installed)
....
....
 *   dev-lang/python:3.4 pulled in by:
 *     (virtual/python-enum34-1:0/0::gentoo, installed)
 * 
 *   dev-lang/python:3.5 pulled in by:
 *     (virtual/python-enum34-1:0/0::gentoo, installed)
 * 
 *   dev-libs/openssl:0/0= pulled in by:
 *     (dev-lang/python-3.7.6:3.7/3.7m::gentoo, installed)
 * 
 * The resume list contains packages that are either masked or have
 * unsatisfied dependencies. Please restart/continue the operation
 * manually, or use --skipfirst to skip the first package in the list and
 * any other packages that may be masked or have missing dependencies.

Not all packages are included above, but it is clear that the list includes packages not only from the last emerge command. The list includes multiple old packages and dependencies about python:3.4, python:3.5, openssl:0/0.
The list is located in the binary file: /var/cache/edb/mtimedb and to clean the list you should execute:

desktop ~ # emaint --fix cleanresume
Emaint: fix cleanresume    100% [============================================>]

It is safe to the directory, too:

rm -Rf /var/cache/edb/

emaint is part of the sys-apps/portage package, so the program should always be available in the system.

*Bonus 1 – old packages

There is one more problem in the above example! In fact, you may clear the resume list with emaint or remove the whole directory “/var/cache/edb”, but still to receive such errors let’s say resuming log lists like KDE, gnome, qt packages (packages with many dependencies).

This problem persists not only with “–resume”, but also with “–keep-going” option!

When using “–keep-going” option you’ll get the same errors after the line “One or more packages are either masked or have missing dependencies”. So it is not an option, which may replace the “–resume” option.

Even all dependencies were resolved when the emerge list was generated when resuming a list of packages may throw errors for packages, which pull old packages because they had not been rebuilt against the updated of some of their dependencies. For example, simple enough from the above emerge resume command:
Some packages of our KDE resume list depends on the “dev-python/setuptools”, which has not been rebuilt against the new python (or after a removed old version python 3.4 and 3.5 or even another [old] package depends on the old python 3.5 and dev-python/setuptools, which has not been rebuilt after the python 3.4/3.5 removal!) and now this is a problem to resume the emerge list.

The solution is to take all those packages (after the line “One or more packages are either masked or have missing dependencies”) and rebuild or remove them.

Keep on reading!

meson.build – ERROR: Dependency “x11” not found, tried pkgconfig and cmake

Another day and another major update exposed an error of missing the “x11” library. Apparently, the pkg-config got corrupted for X11 library. Most of the time the error is self-explanatory, but sometimes it is too generic like:

Run-time dependency x11 found: NO (tried pkgconfig and cmake)

meson.build:412:2: ERROR: Dependency "x11" not found, tried pkgconfig and cmake

The name “x11” is a part of multiple libraries:

srv ~ # pkg-config --list-all |grep x11

gdkglext-1.0                               GdkGLExt - OpenGL Extension to GDK (x11 target)
gdk-2.0                                    GDK - GTK+ Drawing Kit (x11 target)
gtk+-2.0                                   GTK+ - GTK+ Graphical UI Library (x11 target)
gdkglext-x11-1.0                           GdkGLExt - OpenGL Extension to GDK (x11 target)
x11-xcb                                    X11 XCB - X Library XCB interface
gdk-x11-3.0                                GDK - GTK+ Drawing Kit
gdk-x11-2.0                                GDK - GTK+ Drawing Kit (x11 target)
gtkglext-1.0                               GtkGLExt - OpenGL Extension to GTK+ (x11 target)
x11                                        X11 - X Library
xkbcommon-x11                              xkbcommon-x11 - XKB API common to servers and clients - X11 support
scim-x11utils                              scim-x11utils - X11 Utilities for Smart Common Input Method platform
clutter-x11-1.0                            Clutter - Clutter Core Library
gtk+-x11-3.0                               GTK+ - GTK+ Graphical UI Library
libva-x11                                  libva-x11 - Userspace Video Acceleration (VA) x11 interface
gtkglext-x11-1.0                           GtkGLExt - OpenGL Extension to GTK+ (x11 target)
gtk+-x11-2.0                               GTK+ - GTK+ Graphical UI Library (x11 target)

The above example is of working x11 library pkg-config. In our broken system the 10th line was missing.
Keep on reading!

Gentoo – UnicodeEncodeError: ‘ascii’ codec can’t encode character ‘\xc4’ in position 83: ordinal not in range(128)

Recently emerging package =dev-lang/go-1.13.4 under Gentoo failed on one of our virtual servers with:

--- /usr/lib/go/test/fixedbugs/issue27836.dir/
Traceback (most recent call last):
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/_MergeProcess.py", line 234, in _spawn
    prev_mtimes=self.prev_mtimes, counter=counter)
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 1788, in wrapper
    return f(self, *args, **kwargs)
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 5385, in merge
    counter=counter)
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 4548, in treewalk
    rval = self._merge_contents(srcroot, destroot, cfgfiledict)
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 4828, in _merge_contents
    self.settings["EPREFIX"].lstrip(os.sep), cfgfiledict, mymtime):
  File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 5225, in mergeme
    encoding=_encodings['merge'])
  File "/usr/lib64/python3.6/site-packages/portage/util/movefile.py", line 256, in movefile
    selinux.rename(src, dest)
  File "/usr/lib64/python3.6/site-packages/portage/__init__.py", line 246, in __call__
    rval = self._func(*wrapped_args, **wrapped_kwargs)
  File "/usr/lib64/python3.6/site-packages/portage/_selinux.py", line 71, in rename
    os.rename(src, dest)
UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 83: ordinal not in range(128)

>>> Failed to install dev-lang/go-1.13.4, Log file:

>>>  '/var/tmp/portage/dev-lang/go-1.13.4/temp/build.log'

The above log shows that the emerge fails in the installation phase when moving the files to the proper path in the system. The problem there is a non-ASCII character in the file name or path, but the environment is set to use ASCII as language.

The solution is to check if the environment LANG is set and what it contains. In this case, we should set the LANG environment to utf8.

export LANG=en_US.UTF-8

Most of the cases this kind of error could occur with virtual servers, docker (or the other kind of containers like lxc, podman and so on) containers and chroot jails or screens with changed user with su or sudo! In our case, the LANG just got missed because of a switch user procedure in a container and the emerge failed with the above error. When the LANG is missing probably the default value is “C”. In fact, check not only LANG but also the “LC_ALL” environment variable (it may have different value, which is wrong!), which also should be “en_US.UTF-8”:

export LC_ALL=en_US.UTF-8

Or try removing it at all with

unset LC_ALL

Keep on reading!

Gentoo – updating perl and problems like perl-core/ is blocking virtual/perl-

When upgrading Perl under Gentoo it’s almost typical to have blocks of the kind:

[blocks B      ] <perl-core/File-Path-2.160.0 ("<perl-core/File-Path-2.160.0" is blocking virtual/perl-File-Path-2.160.0)
[blocks B      ] <perl-core/Module-CoreList-5.201.905.220 ("<perl-core/Module-CoreList-5.201.905.220" is blocking virtual/perl-Module-CoreList-5.201.905.220)
[blocks B      ] <perl-core/Archive-Tar-2.320.0 ("<perl-core/Archive-Tar-2.320.0" is blocking virtual/perl-Archive-Tar-2.320.0)

These blocks appeared when we tried updaring the Perl 5.26 to 5.30 (dev-lang/perl-5.30.0:0/5.30::gentoo from dev-lang/perl-5.26.2:0). This update is a major update.
So what does it mean?
The virtual package and the perl-core package should be the same version, but it appears there are no corresponding same versions to the virtual package versions (). The problem is that the update depends on a newer version of File-Path, Module-CoreList, and Archive-Tar but there are only virtual packages with the required versions. Virtual packages are just meta-packages and they do not install any module. First, what are the virtual/[perl-module] and perl-core/[perl-module]:

  • perl-core/[perl-module] – perl independent modules. In fact, the same module may be independent and it may be part of the build-in packages from the Perl main package – dev-lang/perl. Sometimes you may choose the independent package, because of a newer version than the version included in the dev-lang/perl package.
  • virtual/[perl-module] – perl meta-package, which ensures the installation of the module. The module could be an independent package from above or module, which is included in the main Perl package (in dev-lang/perl)

To solve the blocks of this kind “perl-core/ is blocking virtual/perl-” you must check for a newer version of perl-core/* if there is no newer version, just to remove the package and let the setup (emerge) use the module included in dev-lang/perl.

In our case, there are no newer versions of the three packages and we remove them with:

emerge -Cv perl-core/File-Path perl-core/Module-CoreList perl-core/Archive-Tar

And then emerge with no problems the new version of Perl with:

emerge --verbose --verbose-conflicts --backtrack=200000 $(qlist -IC|grep perl|sort|uniq) media-gfx/inkscape media-gfx/imagemagick dev-php/pecl-imagick

For detail explaination of why we use verbose-conflicts, backtrack, qlist and three excplicitly added packages you may read our article on similar subject – Gentoo updating perl with many masked and blocked packages
You may read the offcial documentation for more information – https://wiki.gentoo.org/wiki/Perl#Introduction

Updating our Perl from dev-lang/perl-5.26.2:0 to dev-lang/perl-5.30.0:0/5.30

Keep on reading!