Tag: ebuild

Patch and resume compilation of a failed package in Gentoo – ebuild, local repository or ctrl+Z

A dependency package failed to compile throwing error and existing the emerge of a queue with a hundred and more packages. Or worse you installed a new version of a package and multiple rebuilds are pulled, but one of the dependencies fails and you may end up with a broken system? What can you do? There is no new version of the failed package and yes, there is a bug in the Gentoo’s Bugzilla – https://bugs.gentoo.org/. And there is a solution with a patch, which has not made its way to the production and in Gentoo portage yet.
The package in the portage is broken, no new fixed package is released, but there is a patch to fix your issue. Here is what you can do:

  • Make your own package with the fixed version of the original package and put it in your local repository (not the official one, because on every emerge –sync it will be deleted). You should make a local repository and put the ebuild and all necessary files.
  • Or just download the patch and patch the source in the directory, which still holds the source of the failed package and resume the compilation manually. Then install it. Using this tutorial – Resume installation after a package build error, when emerging firefox under Gentoo
  • Just after the uncompress operation of the emerge press CTRL+Z to put the operation in the background and download and patch. Then bring back the emerge from the background with “fg” command.

The second and third options are not permanent solutions, but they are fast enough to be used in some situations.
Here are steps for the first and second option you may have:

OPTION 1) Make your own package.

Create a local repository (for details Simple steps to create Gentoo custom repository and add a package): 

root@srv ~ # mkdir -p /var/db/repos/my-local-portage/{metadata,profiles}
root@srv ~ # cat  << 'EOF' > /var/db/repos/my-local-portage/metadata/layout.conf
masters = gentoo
auto-sync = false
EOF
root@srv ~ # cat  << 'EOF' > /etc/portage/repos.conf/my-local-portage.conf
[my-local-portage]
location = /var/db/repos/my-local-portage 
EOF
root@srv ~ # cat  << 'EOF' > /var/db/repos/my-local-portage/profiles/repo_name
my-local-portage
EOF

Copy the ebuild file of the package you want to modify in the custom repository directory created above (it’s a good idea to copy all the sub-directories, too):
Keep on reading!

Simple steps to create Gentoo custom repository and add a package

Creating a custom repository would give you a chance to fast edit (ebuild) files of existing packages and drop better versions to the custom repository, which then will be used to install in the system. Here is the simplest way to create a Gentoo custom repository without installing any mandatory software. You may check the two Gentoo articles on the subject – https://wiki.gentoo.org/wiki/Custom_repository, which uses repoman (and additional software to install) and https://wiki.gentoo.org/wiki/Handbook:AMD64/Portage/CustomTree#Defining_a_custom_repository, which is part of a bigger article and without a clear example with a package as we are going to show.
Our custom repository name is “my-local-portage”.

STEP 1) Create the directories and basic configuration files for the new custom repository

Just two mandatory directories. 

mkdir -p /var/db/repos/my-local-portage/{metadata,profiles}

The minimal configuration in two files:

cat  << 'EOF' > /var/db/repos/my-local-portage/metadata/layout.conf
masters = gentoo
auto-sync = false
EOF
cat  << 'EOF' > /var/db/repos/my-local-portage/profiles/repo_name
my-local-portage
EOF

Fix the permissions

chown -R portage:portage /var/db/repos/my-local-portage

The custom repository is set up. Now only the emerge should get the configuration to check for it in the next step.

STEP 2) Portage global configuration.

Add a file pointing for your custom repository in the Portage global configuration directory “/etc/portage”:

cat  << 'EOF' > /etc/portage/repos.conf/my-local-portage.conf
[my-local-portage]
location = /var/db/repos/my-local-portage
EOF

STEP 3) Add a package in the new custom repository.

The package version may be the same version as in the official Gentoo repository, but the package form the custom repository will be used if no repository is included in the “emerge” command.
For simplicity, we are going not to modify the ebuild file of a copied official package, but the idea is to copy an existing ebuild file and then change it for the user’s needs and the steps are the same as follow.
Just copy the file (and edit it). The package “app-text/calibre” was randomly selected for the example.

mkdir /var/db/repos/my-local-portage/app-text/calibre
cp /usr/portage/app-text/calibre/calibre-4.9.1-r1.ebuild /var/db/repos/my-local-portage/app-text/calibre/

Create the manifest files and you are ready:

cd /var/db/repos/my-local-portage/app-text/calibre/
ebuild calibre-4.9.1-r1.ebuild manifest
>>> Downloading 'ftp://ftp.free.fr/mirrors/ftp.gentoo.org/distfiles/2e/calibre-4.9.1.tar.xz'
--2020-02-06 18:04:59--  ftp://ftp.free.fr/mirrors/ftp.gentoo.org/distfiles/2e/calibre-4.9.1.tar.xz
           => '/usr/portage/distfiles/calibre-4.9.1.tar.xz.__download__'
Resolving ftp.free.fr... 212.27.60.27, 2a01:e0c:1:1598::1
Connecting to ftp.free.fr|212.27.60.27|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /mirrors/ftp.gentoo.org/distfiles/2e ... done.
==> SIZE calibre-4.9.1.tar.xz ... 37529656
==> PASV ... done.    ==> RETR calibre-4.9.1.tar.xz ... done.
Length: 37529656 (36M) (unauthoritative)

calibre-4.9.1.tar.xz                       100%[========================================================================================>]  35.79M  5.53MB/s    in 8.0s    

2020-02-06 18:05:08 (4.48 MB/s) - '/usr/portage/distfiles/calibre-4.9.1.tar.xz.__download__' saved [37529656]

>>> Creating Manifest for /var/db/repos/my-local-portage/app-text/calibre

Fix the permissions with

chown -R portage:portage /var/db/repos/my-local-portage

The manifest file contains the hash of the ebuild file and all the additional files if any (for this package there are no additional files). All file needed for the operation will be downloaded so they must be network accessed in the time of executing the command (except in the cases when they have already existed in the distfiles directory and/or the subdirectories of the /var/db/repos/my-local-portage/app-text/calibre)
Keep on reading!

Q_WEBENGINECORE_EXPORT QWebEngineFindTextResult has initializer but incomplete type

In addition to one of the cstdlib fix here – Gentoo building qtgui error – g++-v8/cstdlib:75:15: fatal error: stdlib.h: No such file or directory we were unable to to build “dev-qt/qtwebengine-5.14.1“.

It appeared the problem was a wrong order of header includes just like the cstdlib and if you have used the fix with QMAKE_CFLAGS_ISYSTEM in /usr/lib64/qt5/mkspecs/common/gcc-base.conf you would encounter this error. Revert back the original value in /usr/lib64/qt5/mkspecs/common/gcc-base.conf (and probably it is a good idea to rebuild the entire system with “emerge -e”):

QMAKE_CFLAGS_ISYSTEM = -isystem

and continue the build from the point you’ve stopped with something like:

ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild compile
ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild install
ebuild /usr/portage/dev-qt/qtwebengine/qtwebengine-5.14.1.ebuild qmerge
rm -Rf /var/tmp/portage/dev-qt/qtwebengine-5.14.1/

Keep on reading!

Resume compilation of a package from where it failed under Gentoo

Sometimes problems are too specific, but they can show us a path to look for a more general problem and its solution. There was a nasty bug in emerging Firefox package in Gentoo basically it compiled all the source and then it got an error from the build script, but all the source was compiled successfully! So the solution was just to manually install with

ebuild

the package and not to wait for the fix 😉 A good workaround.

But what if we have a big package, which failed during compilation

because of “out of ram” or “out of space” or a missing library, which the maintainer did not included in the dependencies. So tens of minutes or even hours of compilation (yes, there are still such packages like chromium) is wasted and we must start up from the beginning? No you can continue the current failed compilation from the exact point of failure using “ebuild”.
In our example we have a failed compilation of chromium with “out of memory”. We need the exact version of the package, scroll to your emerge command and copy the version, in our case it was: chromium-67.0.3377.1.ebuild
Here are the commands:

STEP 1) Continue compilation

[root@local ]# ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild compile
>>> Existing ${T}/environment for 'chromium-67.0.3377.1' will be sourced.
>>> Run 'clean' to start with a fresh environment.
>>> Checking chromium-67.0.3377.1.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
 * checking ebuild checksums ;-) ...                                                                                                    [ ok ]
 * checking auxfile checksums ;-) ...                                                                                                   [ ok ]
 * checking miscfile checksums ;-) ...                                                                                                  [ ok ]
 * Checking for at least 3 GiB RAM ...                                                                                                  [ ok ]
 * Checking for at least 5 GiB disk space at "/var/tmp/portage/www-client/chromium-67.0.3377.1/temp" ...                                [ ok ]
>>> It appears that 'setup' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.prepared' to force prepare.
>>> It appears that 'configure' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.configured' to force configure.
>>> Compiling source in /var/tmp/portage/www-client/chromium-67.0.3377.1/work/chromium-67.0.3377.1 ...
ninja -v -j6 -l6 -C out/Release mksnapshot
ninja: Entering directory `out/Release'

As you can see in the output above an existing environment for ‘chromium-67.0.3377.1’ will be sourced.
Here the compilation continue from the last failed compilation script, it skipped multiple source dependencies.

STEP 2) Install the package

In fact two commands:

ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild install
ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild qmerge

The first command “install” will install the package in the working directory of the emerge process and then the second “qmerge” will install all the files of the package in the install directory to the live filesystem and will do some additional checks and modifications in your systems package database to install the package properly as if the emerge was used.

Resume installation after a package build error, when emerging firefox under Gentoo

If you recently tried to emerge the latest Firefox in Gentoo (firefox 58.0/58.0.1) and you ended up with a compilation error of this kind:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

The thing is that the build reports successful completion as you can see below from the report, but the emerge process is terminated with error, so in this situation, when the build process is finished successfully we can manually install the package with:

[root@local ]# cd /usr/portage/www-client/firefox
[root@local firefox]# ebuild firefox-58.0.1.ebuild install && ebuild firefox-58.0.1.ebuild qmerge

* The whole output of the error

Exception in thread ProcessReader:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 986, in _read
    callback(line.rstrip())
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 904, in __call__
    e(*args, **kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 720, in on_line
    self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
  File "/usr/lib64/python2.7/logging/__init__.py", line 1231, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1286, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1296, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1336, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 759, in handle
    self.emit(record)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 548, in emit
    self.fh.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/mach_commands.py", line 170, in build
    mach_context=self._mach_context)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 1232, in build
    telemetry_data = monitor.get_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 433, in get_resource_usage
    o['tiers'] = self.tiers.tiered_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 150, in tiered_resource_usage
    self.add_resources_to_dict(t_entry, phase=tier)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 159, in add_resources_to_dict
    end=end, phase=phase, per_cpu=False)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 468, in aggregate_cpu_percent
    data = self.phase_usage(phase)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 427, in phase_usage
    time_start, time_end = self.phases[phase]
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'

>>> Failed to emerge www-client/firefox-58.0.1, Log file:

>>>  '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'

 * Messages for package www-client/firefox-58.0.1:

 * You are enabling official branding. You may not redistribute this build
 * to any users on your network or the internet. Doing so puts yourself into
 * a legal problem with Mozilla Foundation
 * You can disable it by emerging firefox _with_ the bindist USE-flag
 * LINGUAS value bg is not enabled using L10N use flags
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'