Author: neoX

Update supermicro X10SLM-F firmware BIOS under Linux with the SUM cli

Here is how we updated our Supermicro server X10SLM-F with the latest firmware at the moment.

Our current BIOS firmware version is 2.0

[root@srv ~]# lshw|grep -A 14 "core$"
  *-core
       description: Motherboard
       product: X10SLM-F
       vendor: Supermicro
       physical id: 0
       version: 1.02
       serial: 11111111111
       slot: To be filled by O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 2.0
          date: 04/24/2014
          size: 64KiB

Keep on reading!

portage is blocked by “the current version of portage supports EAPI ‘6’. You must upgrade”

We’ve synced the portage tree before upgrading our old portage package (big mistake! always upgrade the portage package before sync) and then do the sync. After the sync the portage upgrade was impossible, because a dependency package supported only a new portage API (probably a new package) in our case EAPI 7 and the offensive package was “app-eselect/eselect-pinentry“.
So there are two options:

  1. Find the last version of the portage,which does not depend on the package – app-eselect/eselect-pinentry
  2. Find if some of the USE flags disable the inclusion of this dependency – app-eselect/eselect-pinentry

We chose the second option and found that if we compiled the portage package with

-rsync-verify

the portage did not pull the dependency “app-eselect/eselect-pinentry” and then after a successful upgrade we had the portage supported EAPI 7 and reinstalled it with activated “-rsync-verify”.
Keep on reading!

pycurl.h: fatal error: openssl/ssl.h: No such file or directory

If you encounter this error trying to install a pip module or compile a program under the console you surely miss OpenSSL development packages!
pip also may build a packages in your system and it could depend on generic library headers like in this case OpenSSL, which the installer (pip) won’t bring them and it will output an error as you can see

myuser@srv # sudo pip install pycurl pygeoip psutil
Collecting pycurl
  Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pygeoip in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): psutil in /usr/lib/python2.7/dist-packages
Building wheels for collected packages: pycurl
  Running setup.py bdist_wheel for pycurl ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpqVNq1upip-wheel- --python-tag cp27:
  Using curl-config (libcurl 7.47.0)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/curl
  copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
  running build_ext
  building 'pycurl' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
  In file included from src/docstrings.c:4:0:
  src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pycurl
  Running setup.py clean for pycurl
Failed to build pycurl
Installing collected packages: pycurl
  Running setup.py install for pycurl ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile:
    Using curl-config (libcurl 7.47.0)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/curl
    copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
    running build_ext
    building 'pycurl' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
    In file included from src/docstrings.c:4:0:
    src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AbCshS/pycurl/
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command

Keep on reading!

Ubuntu AttributeError: ‘module’ object has no attribute ‘SSL_ST_INIT’

If you install libraries with

pip

command you might find yourself in the following situation:

root@srv:~# pip
Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
  File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 13, in <module>
    from pip.exceptions import InstallationError, CommandError, PipError
  File "/usr/lib/python2.7/dist-packages/pip/exceptions.py", line 6, in <module>
    from pip._vendor.six import iteritems
  File "/usr/lib/python2.7/dist-packages/pip/_vendor/__init__.py", line 64, in <module>
    vendored("cachecontrol")
  File "/usr/lib/python2.7/dist-packages/pip/_vendor/__init__.py", line 36, in vendored
    __import__(modulename, globals(), locals(), level=0)
  File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/__init__.py", line 9, in <module>
  File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/wrapper.py", line 1, in <module>
  File "/usr/share/python-wheels/CacheControl-0.11.5-py2.py3-none-any.whl/cachecontrol/adapter.py", line 3, in <module>
  File "/usr/share/python-wheels/requests-2.9.1-py2.py3-none-any.whl/requests/__init__.py", line 53, in <module>
  File "/usr/share/python-wheels/urllib3-1.13.1-py2.py3-none-any.whl/urllib3/contrib/pyopenssl.py", line 54, in <module>
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import rand, crypto, SSL
  File "/usr/local/lib/python2.7/dist-packages/OpenSSL/SSL.py", line 124, in <module>
    SSL_ST_INIT = _lib.SSL_ST_INIT
AttributeError: 'module' object has no attribute 'SSL_ST_INIT'

Keep on reading!

New Redis server (4.0.9) in Ubuntu 16.04 LTS

One of our critical service was under Ubuntu 16.04 LTS (and is scheduled for update, but this a another story!) and how it always happens other parts of our systems use new versions of Ubuntu. But the latest version in Ubuntu 16.04 is 3.0.6 https://packages.ubuntu.com/xenial-updates/redis-server.
Here you can find all the redis server versions available in the supported Ubuntu distributions: https://packages.ubuntu.com/search?keywords=redis-server&searchon=names
Keep on reading!

Replication hangs with Relay_Master_Log_File mysql-bin.999999 -wrong variables in mariadb (MySQL) show slave status

Several days after another .999999 hang out – mariadb (MySQL) stopped flushing relay-bin log after mysqld-relay-bin.999999, the monitoring of one of the slaves got critical with replication delayed. Then several hours it kept delaying without any apparent reason.
The slave status was weird, the

  • “Relay_Master_Log_File: mysql-bin.999999” and
  • “Exec_Master_Log_Pos: 104858214” were not changing,

which is essential for the slave server! In fact without these two values we do not know the real position, which is executed in the slave! And we cannot recover the slave if anything happened.
Keep on reading!

Redis server with Error registering fd event for the new client: Numerical result out of range

We have master slave setup with redis servers and after some time the master server began to refuse connections with

Error: Connection reset by peer

Looking in the redis server’s log in “/var/log/redis/redis-server.log” (Ubuntu way):

redis-server.log-13447:M 17 Jan 15:28:58.719 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:28:58.729 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:28:58.779 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:28:59.723 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:28:59.731 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:28:59.782 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:29:00.725 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:29:00.732 # Error registering fd event for the new client: Numerical result out of range (fd=24099)
redis-server.log-13447:M 17 Jan 15:29:00.784 # Error registering fd event for the new client: Numerical result out of range (fd=24099)

Keep on reading!

mariadb (MySQL) stopped flushing relay-bin log after mysqld-relay-bin.999999

We have a pretty big MariaDB master-slave setup, which is really under heavy load and suddenly one of our slaves stopped purging relay logs! And soon the free space got less and it was clear we were going to run out of space for the data partition.

Here is the problem we have only 49G left free space for our mysql datadir “/var/lib/mysql-datadir”. You can see the replication is OK and running (it’s not behind – “Seconds_Behind_Master: 0”), the

Relay_Log_File says mysqld-relay-bin.999999

but it is not true! You can see there are tones of more relay logs after “mysqld-relay-bin.999999” – in fact the current MySQL relay binlog file is “mysqld-relay-bin.1071413” (look below we included a listing of the datadir).

And “FLUSH RELAY LOGS” does not help, at all!

It just not freed any byte on the partition and no files were removed after “mysqld-relay-bin.999999” as before.
Keep on reading!

apt-mark – upgrade with the exception of certain packages

If you are in a situation when you want to upgrade your system, but do not want to upgrade a certain software in it you can just instruct apt not to upgrade these packages with:

apt-mark hold <package name(s)>

Here is how you can block updating 4 packages – ca-certificates, firefox, ghostscript, linux-firmware. First we update and upgrade and you can see there is no packages to keep back, and then we use apt-mark to “hold” package “linux-firmware” and ca-certificates, firefox, ghostscript at once. Initiating apt upgrade again will give you “The following packages have been kept back:” and it will include all packages, which will not be upgraded (it will include dependencies, which require some of the blocked packages).
Keep on reading!

ssh remote command escape special characters when using variable

You should always enclose the command given to the ssh client for remote executing!

myuser@srv-local:~$ CMD="cat /etc/*release";ssh root@1.1.1.1 "$CMD"
Gentoo Base System release 2.0.3
myuser@srv-local:~$ CMD="cat /etc/*release";ssh root@1.1.1.1 $CMD
cat: /etc/lsb-release: No such file or directory
cat: /etc/os-release: No such file or directory
myuser@srv-remote:~$

You see the difference! The second line the special character “*” asteriks will be expanded by the shell locally and then the result will be send to the remote server for execution. In the second case the remote server will receive a command “cat /etc/lsb-release /etc/os-release” (because our local system has there two files) and not what you want “cat /etc/*release” on the remote.
We use variables above, because we want to point out

the problem, which often occurs when you use ssh remote command execution in a script.

Keep on reading!