distccd failed to exec with “No such file or directory”

And you think your compile box using distccd is ready you start emerge in your server/desktop/laptop and the first package is OK, the your emerge is using the distccd properly and the compilation is distributed to the compile box get compiled.
But just then another package gets a warning during build time:

distcc[9356] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

So everything is back in your machine not in the compile box! And you find in the logs of the distccd compile box:

Apr 23 00:36:17 compile distccd[6177]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:17 compile distccd[12965]: (dcc_job_summary) client: 10.10.10.10:54946 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:3492ms x86_64-pc-linux-gnu-clang++ ../../v8/src/accessors.cc
Apr 23 00:36:17 compile distccd[13490]: (dcc_job_summary) client: 10.10.10.10:54954 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1677ms x86_64-pc-linux-gnu-clang++ ../../v8/src/asmjs/asm-types.cc
Apr 23 00:36:18 compile distccd[6178]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:18 compile distccd[6097]: (dcc_job_summary) client: 10.10.10.10:54956 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2340ms x86_64-pc-linux-gnu-clang++ ../../v8/src/assembler.cc
Apr 23 00:37:22 compile distccd[6180]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:22 compile distccd[13307]: (dcc_job_summary) client: 10.10.10.10:54990 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1878ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking-job.cc
Apr 23 00:37:23 compile distccd[6184]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:23 compile distccd[13719]: (dcc_job_summary) client: 10.10.10.10:54992 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2139ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking.cc

Ahh you missed a package, then you emerge it fast with (assumed you used Gentoo, but the solution is valid for all distros)

[root@local ]# emerge -v sys-devel/clang sys-devel/clang-runtime

And start up the build process again (if Gentoo with emerge) and again the same situation? Again the same error, but you have the “x86_64-pc-linux-gnu-clang++” command and when you type x86_64-pc-linux-gnu-clang++ it executes properly! So what is the problem? The problem is that x86_64-pc-linux-gnu-clang++ is not in the current environment PATH:

compile ~ # whereis x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-clang++: /usr/lib64/llvm/5/bin/x86_64-pc-linux-gnu-clang++

The solution is very simple just restart “distccd”. A trivial one, but could save you time next time! If you install a package, which is expected to be used with distccd restart distccd!!!

Under Getnoo:

compile ~ # /etc/init.d/distccd restart
 * Caching service dependencies ...
 * Stopping distccd ...  [ ok ]
 * Starting distccd ...  [ ok ]
compile ~ #

* You might get error for another file, check if it exists if not install it and then restart the distccd daemon, for example you could get error for any of these:

x86_64-pc-linux-gnu-addr2line
x86_64-pc-linux-gnu-elfedit
x86_64-pc-linux-gnu-gprof
x86_64-pc-linux-gnu-ar
x86_64-pc-linux-gnu-g++
x86_64-pc-linux-gnu-ld
x86_64-pc-linux-gnu-as
x86_64-pc-linux-gnu-g++-6.4.0
x86_64-pc-linux-gnu-ld.bfd
x86_64-pc-linux-gnu-c++
x86_64-pc-linux-gnu-g++-7.2.0
x86_64-pc-linux-gnu-ld.gold
x86_64-pc-linux-gnu-c++-6.4.0
x86_64-pc-linux-gnu-gcc
x86_64-pc-linux-gnu-libgcrypt-config
x86_64-pc-linux-gnu-c++-7.2.0
x86_64-pc-linux-gnu-gcc-6.4.0
x86_64-pc-linux-gnu-llvm-config
x86_64-pc-linux-gnu-c++filt
x86_64-pc-linux-gnu-gcc-7.2.0
x86_64-pc-linux-gnu-nm
x86_64-pc-linux-gnu-clang
x86_64-pc-linux-gnu-gcc-ar
x86_64-pc-linux-gnu-objcopy
x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-gcc-nm
x86_64-pc-linux-gnu-objdump
x86_64-pc-linux-gnu-clang-5.0
x86_64-pc-linux-gnu-gcc-ranlib
x86_64-pc-linux-gnu-pcre-config
x86_64-pc-linux-gnu-clang++-5.0
x86_64-pc-linux-gnu-gcov
x86_64-pc-linux-gnu-pkg-config
x86_64-pc-linux-gnu-clang-cl
x86_64-pc-linux-gnu-gcov-6.4.0
x86_64-pc-linux-gnu-ranlib
x86_64-pc-linux-gnu-clang-cl-5.0
x86_64-pc-linux-gnu-gcov-7.2.0
x86_64-pc-linux-gnu-readelf
x86_64-pc-linux-gnu-clang-cpp
x86_64-pc-linux-gnu-gcov-dump
x86_64-pc-linux-gnu-size
x86_64-pc-linux-gnu-clang-cpp-5.0
x86_64-pc-linux-gnu-gcov-tool
x86_64-pc-linux-gnu-strings
x86_64-pc-linux-gnu-cpp
x86_64-pc-linux-gnu-gfortran
x86_64-pc-linux-gnu-strip
x86_64-pc-linux-gnu-cpp-6.4.0
x86_64-pc-linux-gnu-gfortran-6.4.0
x86_64-pc-linux-gnu-xml2-config
x86_64-pc-linux-gnu-cpp-7.2.0
x86_64-pc-linux-gnu-gfortran-7.2.0
x86_64-pc-linux-gnu-xslt-config
x86_64-pc-linux-gnu-curl-config
x86_64-pc-linux-gnu-gio-querymodules

Resume compilation of a package from where it failed under Gentoo

Sometimes problems are too specific, but they can show us a path to look for a more general problem and its solution. There was a nasty bug in emerging Firefox package in Gentoo basically it compiled all the source and then it got an error from the build script, but all the source was compiled successfully! So the solution was just to manually install with

ebuild

the package and not to wait for the fix 😉 A good workaround.

But what if we have a big package, which failed during compilation

because of “out of ram” or “out of space” or a missing library, which the maintainer did not included in the dependencies. So tens of minutes or even hours of compilation (yes, there are still such packages like chromium) is wasted and we must start up from the beginning? No you can continue the current failed compilation from the exact point of failure using “ebuild”.
In our example we have a failed compilation of chromium with “out of memory”. We need the exact version of the package, scroll to your emerge command and copy the version, in our case it was: chromium-67.0.3377.1.ebuild
Here are the commands:

STEP 1) Continue compilation

[root@local ]# ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild compile
>>> Existing ${T}/environment for 'chromium-67.0.3377.1' will be sourced.
>>> Run 'clean' to start with a fresh environment.
>>> Checking chromium-67.0.3377.1.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
 * checking ebuild checksums ;-) ...                                                                                                    [ ok ]
 * checking auxfile checksums ;-) ...                                                                                                   [ ok ]
 * checking miscfile checksums ;-) ...                                                                                                  [ ok ]
 * Checking for at least 3 GiB RAM ...                                                                                                  [ ok ]
 * Checking for at least 5 GiB disk space at "/var/tmp/portage/www-client/chromium-67.0.3377.1/temp" ...                                [ ok ]
>>> It appears that 'setup' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.prepared' to force prepare.
>>> It appears that 'configure' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.configured' to force configure.
>>> Compiling source in /var/tmp/portage/www-client/chromium-67.0.3377.1/work/chromium-67.0.3377.1 ...
ninja -v -j6 -l6 -C out/Release mksnapshot
ninja: Entering directory `out/Release'

As you can see in the output above an existing environment for ‘chromium-67.0.3377.1’ will be sourced.
Here the compilation continue from the last failed compilation script, it skipped multiple source dependencies.

STEP 2) Install the package

In fact two commands:

ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild install
ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild qmerge

The first command “install” will install the package in the working directory of the emerge process and then the second “qmerge” will install all the files of the package in the install directory to the live filesystem and will do some additional checks and modifications in your systems package database to install the package properly as if the emerge was used.

Resume installation after a package build error, when emerging firefox under Gentoo

If you recently tried to emerge the latest Firefox in Gentoo (firefox 58.0/58.0.1) and you ended up with a compilation error of this kind:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

The thing is that the build reports successful completion as you can see below from the report, but the emerge process is terminated with error, so in this situation, when the build process is finished successfully we can manually install the package with:

[root@local ]# cd /usr/portage/www-client/firefox
[root@local firefox]# ebuild firefox-58.0.1.ebuild install && ebuild firefox-58.0.1.ebuild qmerge

* The whole output of the error

Exception in thread ProcessReader:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 986, in _read
    callback(line.rstrip())
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 904, in __call__
    e(*args, **kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 720, in on_line
    self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
  File "/usr/lib64/python2.7/logging/__init__.py", line 1231, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1286, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1296, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1336, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 759, in handle
    self.emit(record)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 548, in emit
    self.fh.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/mach_commands.py", line 170, in build
    mach_context=self._mach_context)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 1232, in build
    telemetry_data = monitor.get_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 433, in get_resource_usage
    o['tiers'] = self.tiers.tiered_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 150, in tiered_resource_usage
    self.add_resources_to_dict(t_entry, phase=tier)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 159, in add_resources_to_dict
    end=end, phase=phase, per_cpu=False)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 468, in aggregate_cpu_percent
    data = self.phase_usage(phase)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 427, in phase_usage
    time_start, time_end = self.phases[phase]
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'

>>> Failed to emerge www-client/firefox-58.0.1, Log file:

>>>  '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'

 * Messages for package www-client/firefox-58.0.1:

 * You are enabling official branding. You may not redistribute this build
 * to any users on your network or the internet. Doing so puts yourself into
 * a legal problem with Mozilla Foundation
 * You can disable it by emerging firefox _with_ the bindist USE-flag
 * LINGUAS value bg is not enabled using L10N use flags
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'