Tag: distccd

more than the default 4 parallel processes using distributed compiling with distcc

Distributed compilation could greatly speed the build process of Gentoo packages (and not only Gentoo, of course). If you tend to use Gentoo on a laptop or a relatively old CPU you may want to build packages distributively across multiple hosts.
Different (Linux) distributions use different configurations and environment scheme and sometimes it is difficult to sift the configuration, which could be applied to your setup. This is not a tutorial on how to enable parallel processing in Gentoo but it is just our client-site setup.

By default, there is a limit of 4 parallel processes, which is utterly insufficient, because nowadays most servers have more than 8 cores/logical compute units (not to mention that probably most would have 16 and above cores compute units).

The environment variable DISTCC_HOSTS controls, which hosts will receive files for the compilation of what they support and what is the limit of parallel processes.

In Gentoo we set this variable in the /etc/portage/make.conf. Here what you may include in make.conf to have 16 parallel remote processes and up to maximum 4 local (if the remote fails):

MAKEOPTS="-j16 -l4"
FEATURES="distcc"
DISTCC_HOSTS="192.168.0.101/16"

We use the environment DISTCC_HOSTS (here in Gentoo put in the make.conf, but in another Linux distribution an environment variable with this name should be set) because it is easy to set up and control globally for the Gentoo emerge system.
According to the documents:

In order, distcc looks in the $DISTCC_HOSTS environment variable, the user’s $DISTCC_DIR/hosts file, and the system-wide host file.

So when using emerge to build the packages, the emerge will rely on $DISTCC_HOSTS in make.conf (/etc/portage/make.conf or /etc/make.conf if you still use the old path), “/var/tmp/portage/.distcc/” (the build process uses “portage” user and group, not root!) and “/etc/distcc/hosts”. The first option used in the order above will be set the hosts and the limitation for the distributed processing. So if you use $DISTCC_HOSTS in make.conf (or environment) you wouldn’t need to set the “hosts” file.
Separate the different hosts with white space if you have more than one and always use the notation “/LIMIT” for each host. The default value is only 4 parallel processes (i.e it is implicitly added /4 to each hosts in the configuration!)
Keep on reading!

distccd failed to exec with “No such file or directory”

And you think your compile box using distccd is ready you start emerge in your server/desktop/laptop and the first package is OK, the your emerge is using the distccd properly and the compilation is distributed to the compile box get compiled.
But just then another package gets a warning during build time:

distcc[9356] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

So everything is back in your machine not in the compile box! And you find in the logs of the distccd compile box:

Apr 23 00:36:17 compile distccd[6177]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:17 compile distccd[12965]: (dcc_job_summary) client: 10.10.10.10:54946 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:3492ms x86_64-pc-linux-gnu-clang++ ../../v8/src/accessors.cc
Apr 23 00:36:17 compile distccd[13490]: (dcc_job_summary) client: 10.10.10.10:54954 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1677ms x86_64-pc-linux-gnu-clang++ ../../v8/src/asmjs/asm-types.cc
Apr 23 00:36:18 compile distccd[6178]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:18 compile distccd[6097]: (dcc_job_summary) client: 10.10.10.10:54956 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2340ms x86_64-pc-linux-gnu-clang++ ../../v8/src/assembler.cc
Apr 23 00:37:22 compile distccd[6180]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:22 compile distccd[13307]: (dcc_job_summary) client: 10.10.10.10:54990 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1878ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking-job.cc
Apr 23 00:37:23 compile distccd[6184]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:23 compile distccd[13719]: (dcc_job_summary) client: 10.10.10.10:54992 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2139ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking.cc

Ahh you missed a package, then you emerge it fast with (assumed you used Gentoo, but the solution is valid for all distros)

[root@local ]# emerge -v sys-devel/clang sys-devel/clang-runtime

And start up the build process again (if Gentoo with emerge) and again the same situation? Again the same error, but you have the “x86_64-pc-linux-gnu-clang++” command and when you type x86_64-pc-linux-gnu-clang++ it executes properly! So what is the problem? The problem is that x86_64-pc-linux-gnu-clang++ is not in the current environment PATH:

compile ~ # whereis x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-clang++: /usr/lib64/llvm/5/bin/x86_64-pc-linux-gnu-clang++

The solution is very simple just restart “distccd”. A trivial one, but could save you time next time! If you install a package, which is expected to be used with distccd restart distccd!!!

Under Getnoo:

compile ~ # /etc/init.d/distccd restart
 * Caching service dependencies ...
 * Stopping distccd ...  [ ok ]
 * Starting distccd ...  [ ok ]
compile ~ #

* You might get error for another file, check if it exists if not install it and then restart the distccd daemon, for example you could get error for any of these:

x86_64-pc-linux-gnu-addr2line
x86_64-pc-linux-gnu-elfedit
x86_64-pc-linux-gnu-gprof
x86_64-pc-linux-gnu-ar
x86_64-pc-linux-gnu-g++
x86_64-pc-linux-gnu-ld
x86_64-pc-linux-gnu-as
x86_64-pc-linux-gnu-g++-6.4.0
x86_64-pc-linux-gnu-ld.bfd
x86_64-pc-linux-gnu-c++
x86_64-pc-linux-gnu-g++-7.2.0
x86_64-pc-linux-gnu-ld.gold
x86_64-pc-linux-gnu-c++-6.4.0
x86_64-pc-linux-gnu-gcc
x86_64-pc-linux-gnu-libgcrypt-config
x86_64-pc-linux-gnu-c++-7.2.0
x86_64-pc-linux-gnu-gcc-6.4.0
x86_64-pc-linux-gnu-llvm-config
x86_64-pc-linux-gnu-c++filt
x86_64-pc-linux-gnu-gcc-7.2.0
x86_64-pc-linux-gnu-nm
x86_64-pc-linux-gnu-clang
x86_64-pc-linux-gnu-gcc-ar
x86_64-pc-linux-gnu-objcopy
x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-gcc-nm
x86_64-pc-linux-gnu-objdump
x86_64-pc-linux-gnu-clang-5.0
x86_64-pc-linux-gnu-gcc-ranlib
x86_64-pc-linux-gnu-pcre-config
x86_64-pc-linux-gnu-clang++-5.0
x86_64-pc-linux-gnu-gcov
x86_64-pc-linux-gnu-pkg-config
x86_64-pc-linux-gnu-clang-cl
x86_64-pc-linux-gnu-gcov-6.4.0
x86_64-pc-linux-gnu-ranlib
x86_64-pc-linux-gnu-clang-cl-5.0
x86_64-pc-linux-gnu-gcov-7.2.0
x86_64-pc-linux-gnu-readelf
x86_64-pc-linux-gnu-clang-cpp
x86_64-pc-linux-gnu-gcov-dump
x86_64-pc-linux-gnu-size
x86_64-pc-linux-gnu-clang-cpp-5.0
x86_64-pc-linux-gnu-gcov-tool
x86_64-pc-linux-gnu-strings
x86_64-pc-linux-gnu-cpp
x86_64-pc-linux-gnu-gfortran
x86_64-pc-linux-gnu-strip
x86_64-pc-linux-gnu-cpp-6.4.0
x86_64-pc-linux-gnu-gfortran-6.4.0
x86_64-pc-linux-gnu-xml2-config
x86_64-pc-linux-gnu-cpp-7.2.0
x86_64-pc-linux-gnu-gfortran-7.2.0
x86_64-pc-linux-gnu-xslt-config
x86_64-pc-linux-gnu-curl-config
x86_64-pc-linux-gnu-gio-querymodules