Tag: distcc

more than the default 4 parallel processes using distributed compiling with distcc

Distributed compilation could greatly speed the build process of Gentoo packages (and not only Gentoo, of course). If you tend to use Gentoo on a laptop or a relatively old CPU you may want to build packages distributively across multiple hosts.
Different (Linux) distributions use different configurations and environment scheme and sometimes it is difficult to sift the configuration, which could be applied to your setup. This is not a tutorial on how to enable parallel processing in Gentoo but it is just our client-site setup.

By default, there is a limit of 4 parallel processes, which is utterly insufficient, because nowadays most servers have more than 8 cores/logical compute units (not to mention that probably most would have 16 and above cores compute units).

The environment variable DISTCC_HOSTS controls, which hosts will receive files for the compilation of what they support and what is the limit of parallel processes.

In Gentoo we set this variable in the /etc/portage/make.conf. Here what you may include in make.conf to have 16 parallel remote processes and up to maximum 4 local (if the remote fails):

MAKEOPTS="-j16 -l4"
FEATURES="distcc"
DISTCC_HOSTS="192.168.0.101/16"

We use the environment DISTCC_HOSTS (here in Gentoo put in the make.conf, but in another Linux distribution an environment variable with this name should be set) because it is easy to set up and control globally for the Gentoo emerge system.
According to the documents:

In order, distcc looks in the $DISTCC_HOSTS environment variable, the user’s $DISTCC_DIR/hosts file, and the system-wide host file.

So when using emerge to build the packages, the emerge will rely on $DISTCC_HOSTS in make.conf (/etc/portage/make.conf or /etc/make.conf if you still use the old path), “/var/tmp/portage/.distcc/” (the build process uses “portage” user and group, not root!) and “/etc/distcc/hosts”. The first option used in the order above will be set the hosts and the limitation for the distributed processing. So if you use $DISTCC_HOSTS in make.conf (or environment) you wouldn’t need to set the “hosts” file.
Separate the different hosts with white space if you have more than one and always use the notation “/LIMIT” for each host. The default value is only 4 parallel processes (i.e it is implicitly added /4 to each hosts in the configuration!)
Keep on reading!

Gentoo distcc compilation error – relocation R_X86_64_32 against .rodata.str1.1 can not be used when making a shared object

In our Gentoo systems we use distcc (distributed builds for C, C++ and Objective C) to build the packages fast! But after we upgraded to GNU GCC 8.2 some of the packages we tried to build failed with:

 0:02.86 /usr/lib64/distcc/bin/x86_64-pc-linux-gnu-gcc -std=gnu99 -o nsinstall_real -DXP_UNIX -O2  host_nsinstall.o host_pathsub.o
 0:02.88 /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/../../../../x86_64-pc-linux-gnu/bin/ld: host_nsinstall.o: relocation R_X86_64_32 against `.rodata.str1.1' can not be used when making a shared object; recompile with -fPIC
 0:02.88 host_nsinstall.o: error adding symbols: Bad value
 0:02.88 collect2: error: ld returned 1 exit status
 0:02.88 distcc[528] ERROR: compile (null) on localhost failed

As you know the GNU GCC must be (same versions and) compiled with the same flags on the two systems you use it – the local machine and the distcc build server. So we were pretty sure the GNU GCC is the same version in our local machine and the build distcc server, because the USE flags were the same in /etc/portage/make.conf but executing

gcc -v

showed there is a slight difference.
The local machine

srv.local ~ # gcc -v
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/gcc-bin/8.2.0/gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/8.2.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/g++-v8 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 8.2.0-r3 p1.4' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libmudflap --disable-libssp --disable-libmpx --disable-systemtap --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp
Thread model: posix
gcc version 8.2.0 (Gentoo 8.2.0-r3 p1.4)

The distcc server:

compile ~ # gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/8.2.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/g++-v8 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 8.2.0-r3 p1.4' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libmudflap --disable-libssp --disable-libmpx --disable-systemtap --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --disable-default-pie --enable-default-ssp
Thread model: posix
gcc version 8.2.0 (Gentoo 8.2.0-r3 p1.4)

The difference is in local has –enable-default-pie and distcc server has –disable-default-pie

Investigating farther we discovered that the USE make.conf flag is “pie” – https://packages.gentoo.org/useflags/pie and this flag is for “Build programs as Position Independent Executables (a security hardening technique)”. And the problem was the default value on the local machine it was enabled by default and for the remote it was disabled by default –

SO the real problem was the different profiles set on the two servers!

The remote distcc server no one checked and switched to the new portage profile 17 so the server still used the old portage profile 13, where the flag “pie” was excluded.
So change the profile to the same as the local machine and recompile GNU GCC package:

compile ~ # eselect profile set 12
compile ~ # emerge -va gcc

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild   R    ] sys-devel/gcc-8.2.0-r3:8.2.0::gentoo  USE="cxx fortran (multilib) nls nptl openmp pch (pie*) sanitize ssp vtv (-altivec) -debug -doc (-fixed-point) -go -graphite (-hardened) (-jit) (-libssp) -mpx -objc -objc++ -objc-gc -pgo -regression-test -systemtap -vanilla" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

Would you like to merge these packages? [Yes/No] Yes

As you can see the new flag enabled by “(pie*)”.

So be sure when using distcc the Gentoo portage profiles should be the same on the local and the remote distcc systems or you can end up with fatal differences and compilation errors!