Category: tips

distccd failed to exec with “No such file or directory”

And you think your compile box using distccd is ready you start emerge in your server/desktop/laptop and the first package is OK, the your emerge is using the distccd properly and the compilation is distributed to the compile box get compiled.
But just then another package gets a warning during build time:

distcc[9356] (dcc_build_somewhere) Warning: failed to distribute, running locally instead

So everything is back in your machine not in the compile box! And you find in the logs of the distccd compile box:

Apr 23 00:36:17 compile distccd[6177]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:17 compile distccd[12965]: (dcc_job_summary) client: 10.10.10.10:54946 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:3492ms x86_64-pc-linux-gnu-clang++ ../../v8/src/accessors.cc
Apr 23 00:36:17 compile distccd[13490]: (dcc_job_summary) client: 10.10.10.10:54954 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1677ms x86_64-pc-linux-gnu-clang++ ../../v8/src/asmjs/asm-types.cc
Apr 23 00:36:18 compile distccd[6178]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:36:18 compile distccd[6097]: (dcc_job_summary) client: 10.10.10.10:54956 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2340ms x86_64-pc-linux-gnu-clang++ ../../v8/src/assembler.cc
Apr 23 00:37:22 compile distccd[6180]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:22 compile distccd[13307]: (dcc_job_summary) client: 10.10.10.10:54990 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:1878ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking-job.cc
Apr 23 00:37:23 compile distccd[6184]: (dcc_execvp) ERROR: failed to exec x86_64-pc-linux-gnu-clang++: No such file or directory
Apr 23 00:37:23 compile distccd[13719]: (dcc_job_summary) client: 10.10.10.10:54992 COMPILE_ERROR exit:110 sig:0 core:0 ret:0 time:2139ms x86_64-pc-linux-gnu-clang++ ../../v8/src/heap/incremental-marking.cc

Ahh you missed a package, then you emerge it fast with (assumed you used Gentoo, but the solution is valid for all distros)

[root@local ]# emerge -v sys-devel/clang sys-devel/clang-runtime

And start up the build process again (if Gentoo with emerge) and again the same situation? Again the same error, but you have the “x86_64-pc-linux-gnu-clang++” command and when you type x86_64-pc-linux-gnu-clang++ it executes properly! So what is the problem? The problem is that x86_64-pc-linux-gnu-clang++ is not in the current environment PATH:

compile ~ # whereis x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-clang++: /usr/lib64/llvm/5/bin/x86_64-pc-linux-gnu-clang++

The solution is very simple just restart “distccd”. A trivial one, but could save you time next time! If you install a package, which is expected to be used with distccd restart distccd!!!

Under Getnoo:

compile ~ # /etc/init.d/distccd restart
 * Caching service dependencies ...
 * Stopping distccd ...  [ ok ]
 * Starting distccd ...  [ ok ]
compile ~ #

* You might get error for another file, check if it exists if not install it and then restart the distccd daemon, for example you could get error for any of these:

x86_64-pc-linux-gnu-addr2line
x86_64-pc-linux-gnu-elfedit
x86_64-pc-linux-gnu-gprof
x86_64-pc-linux-gnu-ar
x86_64-pc-linux-gnu-g++
x86_64-pc-linux-gnu-ld
x86_64-pc-linux-gnu-as
x86_64-pc-linux-gnu-g++-6.4.0
x86_64-pc-linux-gnu-ld.bfd
x86_64-pc-linux-gnu-c++
x86_64-pc-linux-gnu-g++-7.2.0
x86_64-pc-linux-gnu-ld.gold
x86_64-pc-linux-gnu-c++-6.4.0
x86_64-pc-linux-gnu-gcc
x86_64-pc-linux-gnu-libgcrypt-config
x86_64-pc-linux-gnu-c++-7.2.0
x86_64-pc-linux-gnu-gcc-6.4.0
x86_64-pc-linux-gnu-llvm-config
x86_64-pc-linux-gnu-c++filt
x86_64-pc-linux-gnu-gcc-7.2.0
x86_64-pc-linux-gnu-nm
x86_64-pc-linux-gnu-clang
x86_64-pc-linux-gnu-gcc-ar
x86_64-pc-linux-gnu-objcopy
x86_64-pc-linux-gnu-clang++
x86_64-pc-linux-gnu-gcc-nm
x86_64-pc-linux-gnu-objdump
x86_64-pc-linux-gnu-clang-5.0
x86_64-pc-linux-gnu-gcc-ranlib
x86_64-pc-linux-gnu-pcre-config
x86_64-pc-linux-gnu-clang++-5.0
x86_64-pc-linux-gnu-gcov
x86_64-pc-linux-gnu-pkg-config
x86_64-pc-linux-gnu-clang-cl
x86_64-pc-linux-gnu-gcov-6.4.0
x86_64-pc-linux-gnu-ranlib
x86_64-pc-linux-gnu-clang-cl-5.0
x86_64-pc-linux-gnu-gcov-7.2.0
x86_64-pc-linux-gnu-readelf
x86_64-pc-linux-gnu-clang-cpp
x86_64-pc-linux-gnu-gcov-dump
x86_64-pc-linux-gnu-size
x86_64-pc-linux-gnu-clang-cpp-5.0
x86_64-pc-linux-gnu-gcov-tool
x86_64-pc-linux-gnu-strings
x86_64-pc-linux-gnu-cpp
x86_64-pc-linux-gnu-gfortran
x86_64-pc-linux-gnu-strip
x86_64-pc-linux-gnu-cpp-6.4.0
x86_64-pc-linux-gnu-gfortran-6.4.0
x86_64-pc-linux-gnu-xml2-config
x86_64-pc-linux-gnu-cpp-7.2.0
x86_64-pc-linux-gnu-gfortran-7.2.0
x86_64-pc-linux-gnu-xslt-config
x86_64-pc-linux-gnu-curl-config
x86_64-pc-linux-gnu-gio-querymodules

Resume compilation of a package from where it failed under Gentoo

Sometimes problems are too specific, but they can show us a path to look for a more general problem and its solution. There was a nasty bug in emerging Firefox package in Gentoo basically it compiled all the source and then it got an error from the build script, but all the source was compiled successfully! So the solution was just to manually install with

ebuild

the package and not to wait for the fix 😉 A good workaround.

But what if we have a big package, which failed during compilation

because of “out of ram” or “out of space” or a missing library, which the maintainer did not included in the dependencies. So tens of minutes or even hours of compilation (yes, there are still such packages like chromium) is wasted and we must start up from the beginning? No you can continue the current failed compilation from the exact point of failure using “ebuild”.
In our example we have a failed compilation of chromium with “out of memory”. We need the exact version of the package, scroll to your emerge command and copy the version, in our case it was: chromium-67.0.3377.1.ebuild
Here are the commands:

STEP 1) Continue compilation

[root@local ]# ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild compile
>>> Existing ${T}/environment for 'chromium-67.0.3377.1' will be sourced.
>>> Run 'clean' to start with a fresh environment.
>>> Checking chromium-67.0.3377.1.tar.xz's mtime...
>>> WORKDIR is up-to-date, keeping...
 * checking ebuild checksums ;-) ...                                                                                                    [ ok ]
 * checking auxfile checksums ;-) ...                                                                                                   [ ok ]
 * checking miscfile checksums ;-) ...                                                                                                  [ ok ]
 * Checking for at least 3 GiB RAM ...                                                                                                  [ ok ]
 * Checking for at least 5 GiB disk space at "/var/tmp/portage/www-client/chromium-67.0.3377.1/temp" ...                                [ ok ]
>>> It appears that 'setup' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.setuped' to force setup.
>>> It appears that 'unpack' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.unpacked' to force unpack.
>>> It appears that 'prepare' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.prepared' to force prepare.
>>> It appears that 'configure' has already executed for 'chromium-67.0.3377.1'; skipping.
>>> Remove '/var/tmp/portage/www-client/chromium-67.0.3377.1/.configured' to force configure.
>>> Compiling source in /var/tmp/portage/www-client/chromium-67.0.3377.1/work/chromium-67.0.3377.1 ...
ninja -v -j6 -l6 -C out/Release mksnapshot
ninja: Entering directory `out/Release'

As you can see in the output above an existing environment for ‘chromium-67.0.3377.1’ will be sourced.
Here the compilation continue from the last failed compilation script, it skipped multiple source dependencies.

STEP 2) Install the package

In fact two commands:

ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild install
ebuild /usr/portage/www-client/chromium/chromium-67.0.3377.1.ebuild qmerge

The first command “install” will install the package in the working directory of the emerge process and then the second “qmerge” will install all the files of the package in the install directory to the live filesystem and will do some additional checks and modifications in your systems package database to install the package properly as if the emerge was used.

Delete millions of files slowly without loading the server

There a situations when we need to delete a great deal of files from our filesystem and if we just execute

rm -Rf

the server will surely get loaded and the service it provides will degrade! What if you cannot reformat the filesystem, because the server use it extensively, but you need to delete let’s say a couple of millions file from it? We can use find and usleep (in most linux distro this program is installed by an additional package). The idea is to delete files one by one tuning the pause between every delete. Here you can execute this command in the background or a screen:

find /mnt/storage/old/ -type f -exec echo {} \; -exec rm {} \; -exec usleep 200000 \;

usleep accepts microseconds, so 200000 microseconds are 0.2 seconds. You can tune it precisely with a step of just a microsecond. In the real world under the bash console we probably will use values of max 1/10 of a second around above 100000 microseconds. Execute the command and then watch your server load and tune.

  • usleep in CentOS 7 is installed with package “initscripts”, which is installed by default
  • usleep in Ubuntu is missing and probably won’t find any safe place to download a package to install, but it can be sort of replace with “sleep <floating_point_number>s”, GNU sleep could accept floating point number for the delay and when added “s” at the end it could sleep for a fractions of a seconds. So the command for the Ubuntu is slightly changed: 
    find /mnt/storage/old/ -type f -exec echo {} \; -exec rm {} \; -exec sleep 0.2s \;
  • not GNU version of sleep require NUMBER, so the smallest sleep is only 1 second, which is too big for the purpose. Check your man manual to see if your system has GNU sleep command.

Resume installation after a package build error, when emerging firefox under Gentoo

If you recently tried to emerge the latest Firefox in Gentoo (firefox 58.0/58.0.1) and you ended up with a compilation error of this kind:

UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

The thing is that the build reports successful completion as you can see below from the report, but the emerge process is terminated with error, so in this situation, when the build process is finished successfully we can manually install the package with:

[root@local ]# cd /usr/portage/www-client/firefox
[root@local firefox]# ebuild firefox-58.0.1.ebuild install && ebuild firefox-58.0.1.ebuild qmerge

* The whole output of the error

Exception in thread ProcessReader:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 986, in _read
    callback(line.rstrip())
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 904, in __call__
    e(*args, **kwargs)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/process.py", line 86, in handleLine
    line_handler(line)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 720, in on_line
    self.log(logging.INFO, 'build_output', {'line': line}, '{line}')
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
  File "/usr/lib64/python2.7/logging/__init__.py", line 1231, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1286, in _log
    self.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1296, in handle
    self.callHandlers(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 1336, in callHandlers
    hdlr.handle(record)
  File "/usr/lib64/python2.7/logging/__init__.py", line 759, in handle
    self.emit(record)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 548, in emit
    self.fh.write(msg)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 142-144: ordinal not in range(128)

35:47.22 0 compiler warnings present.
35:47.31 Exception when writing resource usage file: u'compile'
35:47.32 We know it took a while, but your build finally finished successfully!
To view resource usage of the build, run |mach resource-usage|.
Error running mach:

    ['build', '-v']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

KeyError: u'compile'

  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/mach_commands.py", line 170, in build
    mach_context=self._mach_context)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 1232, in build
    telemetry_data = monitor.get_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 433, in get_resource_usage
    o['tiers'] = self.tiers.tiered_resource_usage()
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 150, in tiered_resource_usage
    self.add_resources_to_dict(t_entry, phase=tier)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/python/mozbuild/mozbuild/controller/building.py", line 159, in add_resources_to_dict
    end=end, phase=phase, per_cpu=False)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 468, in aggregate_cpu_percent
    data = self.phase_usage(phase)
  File "/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1/testing/mozbase/mozsystemmonitor/mozsystemmonitor/resourcemonitor.py", line 427, in phase_usage
    time_start, time_end = self.phases[phase]
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'

>>> Failed to emerge www-client/firefox-58.0.1, Log file:

>>>  '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'

 * Messages for package www-client/firefox-58.0.1:

 * You are enabling official branding. You may not redistribute this build
 * to any users on your network or the internet. Doing so puts yourself into
 * a legal problem with Mozilla Foundation
 * You can disable it by emerging firefox _with_ the bindist USE-flag
 * LINGUAS value bg is not enabled using L10N use flags
 * ERROR: www-client/firefox-58.0.1::gentoo failed (compile phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_compile
 *   environment, line 5006:  Called die
 * The specific snippet of code:
 *       MOZ_MAKE_FLAGS="${MAKEOPTS}" SHELL="${SHELL:-${EPREFIX}/bin/bash}" MOZ_NOSPAM=1 ./mach build -v || die
 * 
 * If you need support, post the output of `emerge --info '=www-client/firefox-58.0.1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=www-client/firefox-58.0.1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/www-client/firefox-58.0.1/temp/environment'.
 * Working directory: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'
 * S: '/var/tmp/portage/www-client/firefox-58.0.1/work/firefox-58.0.1'

Busybox ash, Debian dash and simulating bash arrays

Busybox ash (Almquist shell) shell and Debian dash (Debian Almquist shell) are lightweight Unix shell and they are a variant of System V.4 variant of the Bourne shell. Ash/dash shell is known to be very small and is used mainly in embedded (ash) devices and installation scripts (Debian/Ubuntu setup).
Unfortunately they do not support arrays, which could be really a problem in many cases. But we can simulate the arrays with eval function.
So if you need to write a ash/dash script let’s say for an installation script of Ubuntu or Debian or a script for an embedded device, which uses busybox or even you do not want to use arrays in bash, you can follow the consepts below – create variable with a “name” concatenated with a number.

  • 1) Set a variable

    It can be done with two ways:

    1. for myi in 0 1 2 ; do
    setvar mvar$myi "Payload: $myi"
done

    2. for myi in 0 1 2 ; do
    eval mvar$myi=\"Payload: $myi\"
done

    This will create variables with names:

    mvar1, mvar2, mvar3

    and they can be used in any place of your script after the creation of the variables using “eval” or accessing them with the names.

    * bash shell do not support the command “setvar”, so for bash scripts use only eval version.

  • 2) Use a variable

    1. using “eval” 
      for myi in 0 1 2 ; do
    eval echo \$mvar$myi
done

myi=1
eval newvar="\$mvar$myi"
echo $newvar
    2. direct access 
      echo $mvar2
$mvar2="Payload 20"
echo $mvar2

SUPERMICRO IPMI/KVM module tips – reset the unit and the admin password

After the previous howto “SUPERMICRO IPMI to use one of the one interfaces or dedicated LAN port” (in the howto is showed how to install the needed tool for managing the IPMI/KVM unit under console) of setting the network configuration there are a couple of interesting and important tips when working with the IPMI/KVM module. Here are they are:

  1. Reset IPMI/KVM module – sometimes it happen the keyboard or mouse not to work when the Console Redirection is loaded, it is easy to reset the unit from the web interface, but there are case when the web interface is not working – so ssh to your server and try one of the following commands:
    * warm reset – it’s like a reboot, inform the IPMI/KVM to reboot itself.

    ipmitool -I open bmc reset warm

    It does not work in all situations! So try a cold reset
    * cold reset – resets the IPMI/KVM, it’s like unplug and plug the power to the unit.

    ipmitool -I open bmc reset cold
  2. Reset the configuration of an IPMI/KVM module to factory defaults. It is useful when something goes wrong when upgrading the firmware of the unit and the old configuration is not supported or it says it is, but at the end the unit does not work properly. In rare cases it might help when the KVM (Keyboard, Video, Monitor part aka Console redirection does not work)
    Here is the command for resetting to factory defaults:

    ipmitool -I open raw 0x3c 0x40
  3. Reset admin password – reset the password for the administrator login of the IPMI/KVM unit. It’s trivial losing the password so with the help of the local console to the server you can reset the password to a simple one and then change it from the web interface. 
    ipmitool -I open user set password 2 ADMIN

    The number “2” is the ID of the user, check it with:

    [root@srv0 ~]# ipmitool -I open user list
ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
1                    true    false      false      Unknown (0x00)
2   ADMIN            true    false      false      Unknown (0x00)
3                    true    false      false      Unknown (0x00)
4                    true    false      false      Unknown (0x00)
5                    true    false      false      Unknown (0x00)
6                    true    false      false      Unknown (0x00)
7                    true    false      false      Unknown (0x00)
8                    true    false      false      Unknown (0x00)
9                    true    false      false      Unknown (0x00)
10                   true    false      false      Unknown (0x00)

    Sometimes if a hacker got to your IPMI/KVM you could see the user table with the above command. There was a serious bug aka backdoor in some of these units, the ID of the ADMIN user or even the username could be changed, so you should use the list command to list the current user table.
    Use set name to set the username of the user.

    ipmitool -I open user set name 2 ADMIN
  4. Set a new network configuration. It’s worth mentioning again the howto for this purpose – “SUPERMICRO IPMI to use one of the one interfaces or dedicated LAN port”

All commands using the network option of the ipmitool

ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN bmc reset warm
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN bmc reset cold
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN raw 0x3c 0x40
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN user set password 2 ADMIN
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN user list

The IP 192.168.7.150 is the IP of your IPMI/KVM module, which you want to change with the above commands.

Tunneling the IPMI/KVM ports over ssh (supermicro ipmi ports)

The best security for the remote management unit in your server such as IPMI/KVM is to have local IP. All IPMI/KVM IP should be switched to a separated switch and a local sub-network used for the LAN Settings. So to be able to connect to the IPMI/KVM module you need a VPN connection to gain access to the local sub-network used for your servers’ management modules. However, sometimes the VPN cannot be used or it just happened the server is down, or you are at a place restricting unknown ports (or ports above 1024), which your VPN uses (that’s why the VPN server should use only one port from the most popular – 80, 443, but that’s a thing for another howto…) and so on. So you end with no ability to connect to the VPN server or you think you do not need at all a VPN server, because you always could use

openssh

to do the trick of tunneling ports from your computer to the IPMI/KVM module of your server through a server, which has an access to the local sub-network of the IPMI/KVM modules.

So here is what you need to get to the remote management of your server just using ssh for tunneling:

STEP 1) A server, which has access to the IP network of the IPMI/KVM modules.

Let’s say you set to all your servers’ IPMI/KVM modules IPs from network 192.168.7.0/24, so your server must have an IP from 192.168.7.0/24, for example 192.168.7.1, add it as an alias or to a dedicated LAN connected to the switch, in which of all your IPMI/KVM modules are plugged in. This server will be used as a transfer point to a selected IPMI/KVM IP.

STEP 2) Tunnel local selected ports using ssh to the server from STEP 1)

Use this command:

ssh -N -L 127.0.0.1:80:[IPMI-IP]:80 -L 127.0.0.1:443:[IPMI-IP]:443 -L 127.0.0.1:5900:[IPMI-IP]:5900 -L 127.0.0.1:623:[IPMI-IP]:623 root@[SERVER-IP]

For example using 192.168.7.150 for an IPMI/KVM IP:

[root@srv0 ~]# ssh -N -L 127.0.0.1:80:192.168.7.150:80 -L 127.0.0.1:443:192.168.7.150:443 -L 127.0.0.1:5900:192.168.7.150:5900 -L 127.0.0.1:623:192.168.7.150:623 root@example-server.com

With the above command you can use the web interface (https://127.0.0.1/, you could replace 127.0.0.1 with a local IP or a local IP alias of your machine), the java web start “Console Redirection” (the KVM – Keyboard, Video and Mouse) and you can mount Virtual Media from your computer to your server’s virtual CD/DVD device. Unfortunately to use properly the Virtual CD/DVD you must tunnel the UDP on port 623 (not only TCP 623), which is a little bit tricky. To tunnel the UDP packets

socat – Multipurpose relay (SOcket CAT)

program must be used.

STEP 3) Tunnel local selected ports using ssh to the server from STEP 1) and UDP port using socat

[root@srv0 ~]# socat -T15 udp4-recvfrom:623,reuseaddr,fork tcp:localhost:8000
[root@srv0 ~]# ssh -L8000:localhost:8000 -L 127.0.0.1:80:192.168.7.150:80 -L 127.0.0.1:443:192.168.7.150:443 -L 127.0.0.1:5900:192.168.7.150:5900 -L 127.0.0.1:623:192.168.7.150:623 root@example-server.com socat tcp4-listen:8000,reuseaddr,fork UDP:192.168.7.150:623

This will start a UDP listening socket on localhost port 8000. Every packet will be relayed using TCP to localhost 8000, which will be tunneled using ssh command to the remote server, where there is a started another socat listening TCP socket on port 8000, which will relay every packet to the UDP port 623 of IP 192.168.7.150. Replace the IP 192.168.7.150 with your IPMI/KVM IP.

* Here are the required ports for SUPERMICRO IPMI functionality in X9 and X10 motherboards

  • X9-motherboards, the ports are

    TCP Ports
    HTTP: 80
    HTTPS: 443
    SSH: 22
    WSMAN: 5985
    Video: 5901
    KVM: 5900
    CD/USB: 5120
    Floppy: 5123
    Virtual Media: 623
    SNMP: 161

    UDP ports:
    IPMI: 623

  • For X10-motherboards, the ports are

    TCP Ports
    HTTP: 80
    HTTPS: 443
    SSH: 22
    WSMAN: 5985
    Video: 5901
    KVM: 5900 , 3520
    CD/USB: 5120
    Floppy: 5123
    Virtual Media: 623
    SNMP: 161

    UDP ports:
    IPMI: 623

You could add the required port to the ssh command above if you need it!

Virtual Device mounted successfully

Successful mount in Console Redirection with Virtual Media:

main menu
Virtual Storage

if you are logged in the server and mount an ISO with the Virtual Device you’ll probably have this in “dmesg”:

[46683751.661063] usb 2-1.3.2: new high-speed USB device number 8 using ehci-pci
[46683751.795048] usb 2-1.3.2: New USB device found, idVendor=0ea0, idProduct=1111
[46683751.795051] usb 2-1.3.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[46683751.795365] usb-storage 2-1.3.2:1.0: USB Mass Storage device detected
[46683751.795553] scsi6 : usb-storage 2-1.3.2:1.0
[46683752.795730] scsi 6:0:0:0: CD-ROM            ATEN     Virtual CDROM    YS0J PQ: 0 ANSI: 0 CCS
[46683752.806839] sr0: scsi3-mmc drive: 40x/40x cd/rw xa/form2 cdda tray
[46683752.806842] cdrom: Uniform CD-ROM driver Revision: 3.20
[46683752.806933] sr 6:0:0:0: Attached scsi CD-ROM sr0
[46683752.806971] sr 6:0:0:0: Attached scsi generic sg1 type 5

Bring up network interface with an IP address using “ip” command

Lately many linux distributions do not ship by default with

ifconfig

which is considered as old style of setting the network when we need to do it manually.
The command is simple and self-explanatory but there is a catch! Just adding the IP won’t help you to bring up the network interface of your server. In fact we need two commands to instruct the network interface to bring up with an IP and then a third command to add a default gateway.
So here are the steps and commands to bring up an interface, set IP and gateway:

STEP 1) Add the IP to the network interface with

ip addr add 192.168.0.100/24 dev eth0

Change the IP with your IP address.

STEP 2) Bring up the interface link

ip link set eth0 up

If you omit this step a network interface, which is down won’t start and the next command (in step 3) will output an error! If your interface has been up already and you just add an additional IP to it you can skip this step (and probably the one below with the default gateway, but we do not describe this case here).

STEP 3) Bring up the interface link

ip route add default via 192.168.0.1

* The all three in one place for the right way of bringing up a network interface under linux with “ip” command:

ip addr add 192.168.0.100/24 dev eth0
ip link set eth0 up
ip route add default via 192.168.0.1

* Troubleshooting

as it was said: just adding an IP to a network interface, which is in down state, would not help to set an IP, but you would not understand it and when you tried to add the default route your would see not so informative error:

srv@local ~# ip addr add 192.168.0.100/24 dev eth0
srv@local ~# ip route add default via 192.168.0.1
ip: RTNETLINK answers: Network is unreachable

Network unreachable, but why I just added an IP. It is not enough just to add the IP, the link must also be set up, it’s like the

ifconfig eth0 up

.

Replace default program to open text files in Linux console

Ever wondered how to change your text editor when editing text files in Linux? Here is a newbie tip!
For example if you when you what to edit cron jobs you execute

[srv@local ~]# crontab -e

And you get in a text editor? Probably you like vim or nano or pico or vim or some other text editor? And you want to use it whenever the system needs a text editor?
There is an environment variable EDITOR, which could be set to one of the text editors mentioned above.
Temporary you could do it from the command line for the current session to open text files with “nano”

[srv@local ~]# export EDITOR="nano"

And when you open to edit cron jobs or edit a text file in “mc” or whenever the system needs a text editor it will use “nano”. If you replace “nano” with other editor it will be used.
To make it permanent you must put it in your current .bashrc file – “/home//.bashrc” (or more accurate “~/.bashrc”). Just add the same line as above at the end of your .bashrc file:

export EDITOR="nano"

And if you check your current environment you’ll see there is a variable named EDITOR:

[srv@local ~]# env|grep EDITOR
EDITOR=nano

More Linux tips on tips and Linux tips.