Update the conda installed by miniconda3 is simple:
conda update -n base -c defaults conda
And here is the update process.
Keep on reading!
Admin? Anyone? Let's say Anyhelp is appreciated!
Update the conda installed by miniconda3 is simple:
conda update -n base -c defaults conda
And here is the update process.
Keep on reading!
Recently emerging package =dev-lang/go-1.13.4 under Gentoo failed on one of our virtual servers with:
--- /usr/lib/go/test/fixedbugs/issue27836.dir/ Traceback (most recent call last): File "/usr/lib64/python3.6/site-packages/portage/dbapi/_MergeProcess.py", line 234, in _spawn prev_mtimes=self.prev_mtimes, counter=counter) File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 1788, in wrapper return f(self, *args, **kwargs) File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 5385, in merge counter=counter) File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 4548, in treewalk rval = self._merge_contents(srcroot, destroot, cfgfiledict) File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 4828, in _merge_contents self.settings["EPREFIX"].lstrip(os.sep), cfgfiledict, mymtime): File "/usr/lib64/python3.6/site-packages/portage/dbapi/vartree.py", line 5225, in mergeme encoding=_encodings['merge']) File "/usr/lib64/python3.6/site-packages/portage/util/movefile.py", line 256, in movefile selinux.rename(src, dest) File "/usr/lib64/python3.6/site-packages/portage/__init__.py", line 246, in __call__ rval = self._func(*wrapped_args, **wrapped_kwargs) File "/usr/lib64/python3.6/site-packages/portage/_selinux.py", line 71, in rename os.rename(src, dest) UnicodeEncodeError: 'ascii' codec can't encode character '\xc4' in position 83: ordinal not in range(128) >>> Failed to install dev-lang/go-1.13.4, Log file: >>> '/var/tmp/portage/dev-lang/go-1.13.4/temp/build.log'
The above log shows that the emerge fails in the installation phase when moving the files to the proper path in the system. The problem there is a non-ASCII character in the file name or path, but the environment is set to use ASCII as language.
The solution is to check if the environment LANG is set and what it contains. In this case, we should set the LANG environment to utf8.
export LANG=en_US.UTF-8
Most of the cases this kind of error could occur with virtual servers, docker (or the other kind of containers like lxc, podman and so on) containers and chroot jails or screens with changed user with su or sudo! In our case, the LANG just got missed because of a switch user procedure in a container and the emerge failed with the above error. When the LANG is missing probably the default value is “C”. In fact, check not only LANG but also the “LC_ALL” environment variable (it may have different value, which is wrong!), which also should be “en_US.UTF-8”:
export LC_ALL=en_US.UTF-8
Or try removing it at all with
unset LC_ALL
First, you need to install
swift command-line utility
and second, install the command-line tool to manage your account: Install OpenStack swift client only
With the capabilities command you may discover the following important policy and limits of your account like:
and many more.
In general, you will need:
All of the above information should be available from your OpenStack administrator.
Here an example output of the capabalities command:
myuser@myserver:~$ swift --os-username myusr --os-tenant-name myusr --os-password mypass --os-auth-url https://auth01.example.com:5000/v2.0 capabilities Core: swift Options: account_autocreate: True account_listing_limit: 20000 allow_account_management: False container_listing_limit: 20000 extra_header_count: 0 max_account_name_length: 256 max_container_name_length: 256 max_file_size: 5368709122 max_header_size: 8192 max_meta_count: 90 max_meta_name_length: 128 max_meta_overall_size: 4096 max_meta_value_length: 256 max_object_name_length: 1024 policies: [{'name': 'Policy-0', 'default': True}] strict_cors_mode: True Additional middleware: bulk_delete Options: max_deletes_per_request: 20000 Additional middleware: bulk_upload Options: max_containers_per_extraction: 20000 max_failed_extractions: 1000 Additional middleware: container_sync Options: realms: {} Additional middleware: crossdomain Additional middleware: formpost Additional middleware: keystoneauth Additional middleware: slo Options: max_manifest_segments: 1000 max_manifest_size: 2097152 min_segment_size: 1048576 Additional middleware: staticweb
You can see various middleware are activated with specific options – bulk_upload – to upload multiple files with one request (a list with files) and bulk_delete – to delete multiple files per one request and so on.
Here is an interesting tip for all who what to protect the sensitive information with ansible. Our example is simple enough – we want to protect our private key and we want to decrypt it when installing on the server. The copy ansible module has a decrypt feature and it can decrypt the file on-the-fly when the task is executed.
Here is how to use ansible vault to encrypt the file with the private key and the ansible playbook file to copy the file.
If you are a newbie in ansible you can check this article – First ansible use – install and execute a single command or multiple tasks in a playbook There you can see how to create your inventory file (and configure sudo if you remotely log in with unprivileged user) used herein the example.
myuser@srv ~ $ ansible-vault encrypt server.key New Vault password: Confirm New Vault password: Encryption successful
You can see the file now is changed and starts with:
myuser@srv ~ $ cat server.key $ANSIBLE_VAULT;1.1;AES256 62363263663865646361643461663531373637386631646262366333663831643435633263363336 3735326665326363356566303566626638316662376432640a326362326230353966353431383164 35353531653331306430656562616165353632643330393662313535326438363964303436306639 .... ....
--- - hosts: all tasks: - name: Copy server private key copy: src: server.key dest: /etc/env/server.key decrypt: yes owner: root group: root mode: 400 backup: no
myuser@srv ~ $ ansible-playbook --ask-vault-pass -l srv3 -i ./inventory.ini ./playbook-example.yml -b Vault password: PLAY [all] ***************************************************************************************************************************************************************** TASK [Gathering Facts] ***************************************************************************************************************************************************** ok: [srv3] TASK [Copy server private key] ********************************************************************************************************************************************* changed: [srv3] PLAY RECAP ***************************************************************************************************************************************************************** srv3 : ok=2 changed=1 unreachable=0 failed=0
And the file in the remote server (srv3 in the example) is unencrypted in /etc/env/server.key!
Another ansible quick tip showing how to restart a program properly. We want to restart the program or the service only if it is running (because some system on executing restart may start the service even it is in the stopped state).
Here is what the ansible playbook do:
If you are a newbie in ansible you can check this article – First ansible use – install and execute a single command or multiple tasks in a playbook There you can see how to create your inventory file (and configure sudo if you remotely log in with unprivileged user) used herein the example.
For our example we use the nginx webserver in the ansible playbook. Put the following code in a file and then execute ansible-playbook:
--- - hosts: all tasks: - name: Test for running nginx shell: ps axuf|grep 'nginx'|grep -v "grep" | tr -d "\n" | cat register: test_running_nginx changed_when: False tags: restart-nginx - name: First check the configuration shell: /usr/sbin/nginx -t register: test_nginx_config when: test_running_nginx.stdout != "" changed_when: False ignore_errors: True tags: restart-nginx - name: Restart nginx service: name=nginx state=restarted when: test_running_nginx.stdout != "" and test_nginx_config.rc == 0 tags: restart-nginx
myuser@srv ~ $ ansible-playbook -l srv2 -i ./inventory.ini ./playbook-example.yml -b PLAY [all] ***************************************************************************************************************************************************************** TASK [Gathering Facts] ***************************************************************************************************************************************************** ok: [srv2] TASK [Test for running nginx] ********************************************************************************************************************************************** ok: [srv2] TASK [First check the configuration] *************************************************************************************************************************************** ok: [srv2] TASK [Restart nginx] ******************************************************************************************************************************************************* changed: [srv2] PLAY RECAP ***************************************************************************************************************************************************************** srv2 : ok=4 changed=1 unreachable=0 failed=0
Here we add to the command line “-b”, which will escalate to root if it is needed (using sudo) because the remote connection is done with unprivileged user “myuser”. You can skip this option if you described the remote connection with the root user in the inventory file (or a system user, which has permissions to restart services).
Keep on reading!
Here is a quick ansible tip for system administrators for the ansible lineinfile. Imagine you want to insert a line after a word (or a predefined marker in your configuration file), but you want to insert the line ONLY if the word exists!
It could be done with lineinfile module but there is a limitation. The module will insert after the first occurrence of your marker or at the end of the file. Here is what the manual says: “If specified regular expression has no matches, EOF will be used instead.” And what if you what to insert some additional line to your structured configuration file? It will corrupt your configuration file, so we need something else!
Not only this! Imagine you have already inserted the line in a previous playbook run? It will be unwanted to add the line, again and again, each time the playbook is run. So here we propose the following solution:
Here we use three ansible modules – stat, shell, lineinfile and variables and conditional checks.
If you are a newbie in ansible you can check this article – First ansible use – install and execute a single command or multiple tasks in a playbook There you can see how to create your inventory file (and configure sudo if you remotely log in with unprivileged user) used herein the example:
--- - hosts: all tasks: - name: Test for nginx-config stat: path: /etc/nginx/nginx.conf register: test_exist_nginx_config tags: cors-insert-include - name: Test for \#FIRST-SRV-LOCATION tag shell: grep '#FIRST-SRV-LOCATION' /etc/nginx/nginx.conf | tr -d "\n" | cat register: test_first_srv_location when: test_exist_nginx_config.stat.exists changed_when: False tags: cors-insert-include - name: Test for cors-locations.loc inserted already shell: grep "cors-locations.loc" /etc/nginx/nginx.conf | tr -d "\n" | cat register: test_cors_locations_loc when: test_exist_nginx_config.stat.exists changed_when: False tags: cors-insert-include - name: Insert the includes after \#FIRST-SRV-LOCATION lineinfile: path: /etc/nginx/nginx.conf insertafter: '#FIRST-SRV-LOCATION' line: ' include /etc/nginx/conf.d/cors-locations.loc;' state: present when: test_exist_nginx_config.stat.exists and test_first_srv_location.stdout != "" and test_cors_locations_loc.stdout == "" tags: cors-insert-include
We want to insert a new include line after our predefined tag “#FIRST-SRV-LOCATION” in the nginx webserver’s main configuration file.
We’ve encountered the following error when issuing a publish command:
aptly@aptly-server:~$ aptly --config=/mnt/storage/aptly/.aptly.conf publish snapshot xenial-myrepo-initial ubuntu Loading packages... Generating metadata files and linking package files... ERROR: unable to publish: unable to process packages: error linking file to /mnt/storage/aptly/.aptly/public/ubuntu/pool/main/s/sftpcloudfs/sftpcloudfs_0.12.2-2_all.deb: file already exists and is different
And the snapshot had failed to publish. Check if the file is “aptly:aptly” (or the user and group your installation uses) because if someone has executed commands from the user root it may create some files with the user root (or other) and after that, some commands could fail. In our case, the file was with the right user for aptly and the solution was to remove the file manually (i.e. it is safe to remove it!) it was created again by the setup in the right time. Then execute the publish command again:
aptly@aptly-server:~$ rm /mnt/storage/aptly/.aptly/public/ubuntu/pool/main/s/sftpcloudfs/sftpcloudfs_0.12.2-2_all.deb aptly@aptly-server:~$ aptly --config=/mnt/storage/aptly/.aptly.conf publish snapshot xenial-myrepo-initial ubuntu Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshot xenial-myrepo-initial has been successfully published. Please setup your webserver to serve directory '/mnt/storage/aptly/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ubuntu/ xenial-myrepo main deb-src http://your-server/ubuntu/ xenial-myrepo main Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
The solution is simple, just remove the offensive file(s) and execute the command again. It is safe to remove the file manually.
This is also a common error in a typical aptly installation. The other two common errors related to the GPG keys are: aptly publish: ERROR: unable to initialize GPG signer. Missing pubring.gpg keys and aptly mirror – gpgv: Can’t check signature: public key not found. This secret key is used when you try to publish a repository (snapshot or mirror).
root@srv-aptly ~ # aptly publish snapshot xenial-myrepo-initial Loading packages... Generating metadata files and linking package files... 15683 / 107250 [====================>--------------------------------------------------------------------------------------------------------------------] 14.62% 2h53m50s 17025 / 107250 [=====================>--------------------------------------------------------------------------------------------------------------------] 15.87% 3h5m15sFinalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: gpg: no default secret key: secret key not available gpg: signing failed: secret key not available ERROR: unable to publish: unable to detached sign file: exit status 2
You are unable to sign the Release file because the keyring secring.gpg is missing a GPG key. Just create or import from your current servers the GPG key from keyring secring.gpg (for the root user it is /root/.gnupg/secring.gpg and in general this is the default path /[my-aptly-home-directory]/.gnupg/secring.gpg).
Here is the example with the two servers, exporting from your current and importing the key in your new (the second) server:
root@srv-aptly-1:~ # gpg --list-keys --keyring secring.gpg /root/.gnupg/secring.gpg ------------------------ pub 2048D/FDC7A25E 2017-09-16 uid My-aptly (aptly key no passphrase) <my-aptly@example.com> root@srv-aptly-1:~ # gpg --keyring secring.gpg --export --armor FDC7A25E > FDC7A25E.key root@srv-aptly-1:~ # gpg --list-secret-keys --keyring secring.gpg /root/.gnupg/secring.gpg ------------------------ sec 2048D/FDC7A25E 2017-09-16 uid My-aptly (aptly key no passphrase) <my-aptly@example.com> root@srv-aptly-1:~ # gpg --keyring secring.gpg --export-secret-key --armor FDC7A25E > FDC7A25E.sec
First is the public key (FDC7A25E.key) and second is the private key (FDC7A25E.sec). You must export them both and import them in your new server (or look below how to generate them in your server).
root@srv-aptly-2:~ # cat ./FDC7A25E.key| gpg --keyring secring.gpg --import gpg: key FDC7A25E: public key "My-aptly (aptly key no passphrase) <my-aptly@example.com>" imported gpg: Total number processed: 1 gpg: imported: 1 root@srv-aptly-2:~ # gpg --keyring secring.gpg --allow-secret-key-import --armor --import FDC7A25E.sec gpg: key FDC7A25E: secret key imported gpg: key FDC7A25E: "My-aptly (aptly key no passphrase) <my-aptly@example.com>" not changed gpg: Total number processed: 1 gpg: unchanged: 1 gpg: secret keys read: 1 gpg: secret keys imported: 1
And now you can publish your repository with:
root@srv-aptly-2: ~ # aptly publish snapshot xenial-myrepo-initial ubuntu Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshot xenial-myrepo-initial has been successfully published. Please setup your webserver to serve directory '/mnt/storage/aptly/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ubuntu/ xenial-myrepo main deb-src http://your-server/ubuntu/ xenial-myrepo main Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
The operation publish passed successfully.
If you just came here installing a new aptly server and getting this error as mentioned above you miss a GPG key in keyring secring.gpg.
root@srv-aptly: ~# gpg --default-new-key-algo rsa4096 --gen-key --keyring secring.gpg gpg (GnuPG) 2.2.11; Copyright (C) 2018 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Note: Use "gpg --full-generate-key" for a full featured key generation dialog. GnuPG needs to construct a user ID to identify your key. Real name: My-aptly Email address: my-aptly@example.com You selected this USER-ID: "MyName <my-aptly@example.com>" Change (N)ame, (E)mail, or (O)kay/(Q)uit? O We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: key B14B67D0CF27191B marked as ultimately trusted gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/77EC42A1F16127C83509292BB14B67D0CF27191B.rev' public and secret key created and signed. Note that this key cannot be used for encryption. You may want to use the command "--edit-key" to generate a subkey for this purpose. pub rsa4096 2019-07-08 [SC] [expires: 2021-07-07] 77EC42A1F16127C83509292BB14B67D0CF27191B uid MyName <my-aptly@example.com>
Just to note here we give you all the examples with the root user and the GPG keys are for the root user. You may use a different user for the aptly process and you must ensure the GPG keys to present for this user (the directories and files are the same, just home directory is different – the home directory of the aptly user i.e. “/[my-aptly-home-directory]/.gnupg/secring.gpg” and for all other GPG files “/[my-aptly-home-directory]/.gnupg/”).
In continuation of our aptly common mistakes here one more when making a second mirror aptly server to your master (you may encounter this error in many other situations, not only building a mirror aptly server). Again the problem is the GPG key like this one – aptly mirror – gpgv: Can’t check signature: public key not found this time the problem occurs when you try getting snapshot of your mirror repository.
By default Aptly uses the GNU key in keyring pubring.gpg (/root/.gnupg/pubring.gpg for the root user)
And even you may have the same key in other keyrings like trustedkeys.gpg you won’t be able to use them for signing process with the aptly snapshot.
root@srv-aptly-2:~ # aptly publish snapshot myrepo-initial ERROR: unable to initialize GPG signer: looks like there are no keys in gpg, please create one (official manual: http://www.gnupg.org/gph/en/manual.html)
The solution is to export the key from pubring.gpg keyring and then import the GPG key in keyring pubring.gpg in the new server. And then you won’t receive the error when making a snapshot with aptly. Or if your case is not making a second server, but your first aptly server you must generate the GPG key in pubring.gpg (look at the end how to do it and skip the lines below for GPU key export and import).
root@srv-aptly-1:~ # gpg --list-keys --keyring pubring.gpg /root/.gnupg/pubring.gpg ------------------------ pub 2048D/FDC7A25E 2017-09-16 uid My-aptly (aptly key no passphrase) <my-aptly@example.com> root@srv-aptly-1:~ # gpg --keyring pubring.gpg --export --armor FDC7A25E > FDC7A25E.key
root@srv-aptly-2:~ # cat ./FDC7A25E.key| gpg --keyring pubring.gpg --import gpg: key FDC7A25E: public key "My-aptly (aptly key no passphrase) <my-aptly@example.com>" imported gpg: Total number processed: 1 gpg: imported: 1 root@srv-aptly-2:~ # aptly publish snapshot myrepo-initial Loading packages... Generating metadata files and linking package files... 15683 / 107250 [====================>--------------------------------------------------------------------------------------------------------------------] 14.62% 2h53m50s 17025 / 107250 [=====================>--------------------------------------------------------------------------------------------------------------------] 15.87% 3h5m15sFinalizing metadata files..
If you just came here installing a new aptly server and getting this error as mentioned above you miss a GPG key in keyring pubring.gpg.
root@srv-aptly: ~# gpg --default-new-key-algo rsa4096 --gen-key --keyring pubring.gpg gpg (GnuPG) 2.2.11; Copyright (C) 2018 Free Software Foundation, Inc. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Note: Use "gpg --full-generate-key" for a full featured key generation dialog. GnuPG needs to construct a user ID to identify your key. Real name: My-aptly Email address: my-aptly@example.com You selected this USER-ID: "MyName <my-aptly@example.com>" Change (N)ame, (E)mail, or (O)kay/(Q)uit? O We need to generate a lot of random bytes. It is a good idea to perform some other action (type on the keyboard, move the mouse, utilize the disks) during the prime generation; this gives the random number generator a better chance to gain enough entropy. gpg: key B14B67D0CF27191B marked as ultimately trusted gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/77EC42A1F16127C83509292BB14B67D0CF27191B.rev' public and secret key created and signed. Note that this key cannot be used for encryption. You may want to use the command "--edit-key" to generate a subkey for this purpose. pub rsa4096 2019-07-08 [SC] [expires: 2021-07-07] 77EC42A1F16127C83509292BB14B67D0CF27191B uid MyName <my-aptly@example.com>
Just to note here we give you all the examples with the root user and the GPG keys are for the root user. You may use a different user for the aptly process and you must ensure the GPG keys to present for this user (the directories and files are the same, just home directory is different – the home directory of the aptly user i.e. “/[my-aptly-home-directory]/.gnupg/pubring.gpg” and for all other GPG files “/[my-aptly-home-directory]/.gnupg/”).
Always check the source what supports when trying to mirror! We have lost some time before discovering that our source repository does not support udeb and source packages! If you create a mirror with “-with-sources=true -with-udebs=true” the update process will require files, which may not exists in the source repository if it does not offer udeb or source files and you’ll end up with broken mirror and error for missing file!
Downloading & parsing package files... Downloading http://aptly.example.com/ubuntu/dists/xenial-myrepos/main/binary-amd64/Packages.bz2... ERROR: unable to update: no candidates for http://aptly-master.example.com/ubuntu/dists/xenial-myrepo/main/debian-installer/binary-amd64/Packages found
If you get error for “debian-installer/binary-amd64/Packages” not found, check the source repository if it offers udeb and/or source packages – probably not, so drop your mirror and recreate it including one or the two options
-with-sources=false -with-udebs=false