This article is to show mainly how to work with aptly by mirroring an official Ubuntu mirror. If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps
What we are going to do – this is what you need to have a mirror of an external application repository:
- Install aptly in Ubuntu 18 LTS
- Create a mirror in aptly
- Create a snapshot of the mirror created before
- Publish the snapshot to be used in other servers.
and at the last step there is an example how to use the mirror in your local machines.
STEP 1) Install aptly in Ubuntu 18.04 LTS.
As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.
STEP 2) Create a mirror in aptly.
Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):
aptly@srv:~$ gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import gpg: key 3B4FE6ACC0B21F32: 3 signatures not checked due to missing keys gpg: key 3B4FE6ACC0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" imported gpg: key D94AA3F0EFE21092: 3 signatures not checked due to missing keys gpg: key D94AA3F0EFE21092: public key "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" imported gpg: key 871920D1991BC93C: 1 signature not checked due to a missing key gpg: key 871920D1991BC93C: public key "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" imported gpg: Total number processed: 3 gpg: imported: 3 gpg: public key of ultimately trusted key 212A3D20E4D3351D not found gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
Probably you would like to have “main” and “universe” for the three bionic, bionic updates and bionic security.
First, main and universe for bionic repository. main is ~16G and universe is ~136, these numbers will vary in future when more packages are added.
Two commands are need for the aptly mirror:
- create – create the mirror.
- update – download the repository contents locally.
aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-main http://nl.archive.ubuntu.com/ubuntu/ bionic main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/InRelease... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] successfully added. You can run 'aptly mirror update bionic-main' to download repository contents. aptly@srv:~$ aptly mirror update bionic-main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/InRelease... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/main/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/main/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/main/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/main/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/main/source/Sources.gz... Building download queue... Download queue: 17044 items (16.13 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/b/binutils/binutils-s390x-linux-gnu_2.30-15ubuntu1_amd64.deb... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/p/pymongo/python3-pymongo_3.6.1+dfsg1-1_amd64.deb... ..... ..... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/l/language-pack-gnome-ml/language-pack-gnome-ml_18.04+20180423.dsc... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/l/language-pack-gnome-ml/language-pack-gnome-ml_18.04+20180423.tar.xz... Mirror `bionic-main` has been successfully updated. aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-universe http://nl.archive.ubuntu.com/ubuntu/ bionic universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/InRelease... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] successfully added. You can run 'aptly mirror update bionic-universe' to download repository contents. aptly@my-server-pc:~$ aptly mirror update bionic-universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/InRelease... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/universe/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/universe/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/universe/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/universe/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/universe/source/Sources.gz... Building download queue... Download queue: 157819 items (136.58 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/p/python-libcharmstore/python3-libcharmstore_0.0.3-0ubuntu1_all.deb... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/g/genometools/python-genometools_1.5.10+ds-2_all.deb... ..... ..... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/h/haskell-microlens-th/haskell-microlens-th_0.4.1.1.orig.tar.gz... Mirror `bionic-universe` has been successfully updated.
And we continue with the same commands for the boinic updates. Now bionic updates is 50.92 GiB and bionic universe is 26.73 GiB.
aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-updates-main http://nl.archive.ubuntu.com/ubuntu/ bionic-updates main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease... gpgv: Signature made Mon 06 Jan 2020 07:26:06 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] successfully added. You can run 'aptly mirror update bionic-updates-main' to download repository contents. aptly@srv:~$ aptly mirror update bionic-updates-main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease... gpgv: Signature made Mon 06 Jan 2020 07:26:06 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/main/source/Sources.gz... Building download queue... Download queue: 16286 items (50.92 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/p/pam/libpam0g-dev_1.1.8-3.6ubuntu2.18.04.1_i386.deb... ...... ...... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/l/linux-azure/linux-cloud-tools-4.18.0-1020-azure_4.18.0-1020.20~18.04.1_amd64.deb... Mirror `bionic-updates-main` has been successfully updated.
Then the bionic updates “universe”:
aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-updates-universe http://nl.archive.ubuntu.com/ubuntu/ bionic-updates universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease... gpgv: Signature made Tue 07 Jan 2020 03:12:11 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] successfully added. You can run 'aptly mirror update bionic-updates-universe' to download repository contents. aptly@srv:~$ aptly mirror update bionic-updates-universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/InRelease... gpgv: Signature made Tue 07 Jan 2020 03:12:11 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-updates/universe/source/Sources.gz... Building download queue... Download queue: 12123 items (26.73 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-6/libx32go9-dbg_6.5.0-2ubuntu1~18.04_i386.deb... ...... ...... Mirror `bionic-updates-universe` has been successfully updated.
And the last one is boinc updates “main” (1.02G) and “universe” (0.5G).
aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-security-main http://nl.archive.ubuntu.com/ubuntu/ bionic-security main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/InRelease... gpgv: Signature made Tue 07 Jan 2020 07:06:11 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-security-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] successfully added. You can run 'aptly mirror update bionic-security-main' to download repository contents. aptly@srv:~$ aptly mirror update bionic-security-main Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/InRelease... gpgv: Signature made Tue 07 Jan 2020 07:06:11 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/main/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/main/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/main/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/main/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/main/source/Sources.gz... Building download queue... Download queue: 740 items (1.02 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/l/linux-meta/linux-image-virtual-hwe-16.04_4.15.0.72.74_i386.deb... Downloading http://nl.archive.ubuntu.com/ubuntu/pool/main/c/ceph/python-rados_12.2.12-0ubuntu0.18.04.2_i386.deb.. ..... ..... Mirror `bionic-security-main` has been successfully updated. aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-security-universe http://nl.archive.ubuntu.com/ubuntu/ bionic-security universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/InRelease... gpgv: Signature made Wed 08 Jan 2020 03:12:37 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Mirror [bionic-security-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] successfully added. You can run 'aptly mirror update bionic-security-universe' to download repository contents. aptly@srv:~$ aptly mirror update bionic-security-universe Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/InRelease... gpgv: Signature made Wed 08 Jan 2020 03:12:37 AM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" Downloading & parsing package files... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/universe/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/universe/debian-installer/binary-i386/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/universe/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/universe/debian-installer/binary-amd64/Packages.gz... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic-security/universe/source/Sources.gz... Building download queue... Download queue: 278 items (0.50 GiB) Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/u/unbound/unbound-host_1.6.7-1ubuntu2.1_amd64.deb... ...... ...... Mirror `bionic-security-universe` has been successfully updated.
Here are listing of our mirrors:
aptly@srv:~$ aptly mirror list List of mirrors: * [bionic-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] * [bionic-security-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] * [bionic-security-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] * [bionic-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] * [bionic-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] * [bionic-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] To get more information about mirror, run `aptly mirror show <name>`.
STEP 3) Create snapshots of the mirrors
Now create the snapshots of the synchronized mirrors. These snapshots will be published in the next step. We choose the name of the snapshot as the name of the mirror but you may want to create the snapshots with a date suffix.
aptly@srv:~$ aptly snapshot create bionic-main from mirror bionic-main Snapshot bionic-main successfully created. You can run 'aptly publish snapshot bionic-main' to publish snapshot as Debian repository. aptly@srv:~$ aptly snapshot create bionic-universe from mirror bionic-universe Snapshot bionic-universe successfully created. You can run 'aptly publish snapshot bionic-universe' to publish snapshot as Debian repository. aptly@srv:~$ aptly snapshot create bionic-updates-main from mirror bionic-updates-main Snapshot bionic-updates-main successfully created. You can run 'aptly publish snapshot bionic-updates-main' to publish snapshot as Debian repository. aptly@srv:~$ aptly snapshot create bionic-updates-universe from mirror bionic-updates-universe Snapshot bionic-updates-universe successfully created. You can run 'aptly publish snapshot bionic-updates-universe' to publish snapshot as Debian repository. aptly@srv:~$ aptly snapshot create bionic-security-main from mirror bionic-security-main Snapshot bionic-security-main successfully created. You can run 'aptly publish snapshot bionic-security-main' to publish snapshot as Debian repository. aptly@srv:~$ aptly snapshot create bionic-security-universe from mirror bionic-security-universe Snapshot bionic-security-universe successfully created. You can run 'aptly publish snapshot bionic-security-universe' to publish snapshot as Debian repository.
The snapshot listing shows:
aptly@srv:~$ aptly snapshot list List of snapshots: * [bionic-main]: Snapshot from mirror [bionic-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] * [bionic-security-main]: Snapshot from mirror [bionic-security-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] * [bionic-security-universe]: Snapshot from mirror [bionic-security-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-security [src] [udeb] * [bionic-universe]: Snapshot from mirror [bionic-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic [src] [udeb] * [bionic-updates-main]: Snapshot from mirror [bionic-updates-main]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] * [bionic-updates-universe]: Snapshot from mirror [bionic-updates-universe]: http://nl.archive.ubuntu.com/ubuntu/ bionic-updates [src] [udeb] To get more information about snapshot, run `aptly snapshot show <name>`.
STEP 4) Publish the snapshots of the official mirrors.
The last step to be able to use “the official mirrors” (in fact, you are going to use their snapshots and if you update them you will continue to use the repositories as they look before the update).
Bionic main and universe:
aptly@srv:~$ aptly publish snapshot -component=main,universe bionic-main bionic-universe ubuntu Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshots bionic-main, bionic-universe have been successfully published. Please setup your webserver to serve directory '$HOME/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ubuntu/ bionic main universe deb-src http://your-server/ubuntu/ bionic main universe Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
Bionic-updates main and universe:
aptly@srv:~$ aptly publish snapshot -component=main,universe bionic-updates-main bionic-updates-universe ubuntu Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshots bionic-updates-main, bionic-updates-universe have been successfully published. Please setup your webserver to serve directory '$HOME/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ubuntu/ bionic-updates main universe deb-src http://your-server/ubuntu/ bionic-updates main universe Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
Bionic-security main and universe:
aptly@srv:~$ aptly publish snapshot -component=main,universe bionic-security-main bionic-security-universe ubuntu Loading packages... Generating metadata files and linking package files... Finalizing metadata files... Signing file 'Release' with gpg, please enter your passphrase when prompted: Clearsigning file 'Release' with gpg, please enter your passphrase when prompted: Snapshots bionic-security-main, bionic-security-universe have been successfully published. Please setup your webserver to serve directory '$HOME/.aptly/public' with autoindexing. Now you can add following line to apt sources: deb http://your-server/ubuntu/ bionic-security main universe deb-src http://your-server/ubuntu/ bionic-security main universe Don't forget to add your GPG key to apt with apt-key. You can also use `aptly serve` to publish your repositories over HTTP quickly.
STEP 5) Use the your repositories instead the official ones
To use them you first should import the key of your new repository and then add them to the Ubuntu’s configuration files. As mentioned in the beginning, we installed the aptly software following our previous article – Install aptly under Ubuntu 18 LTS with nginx serving the packages and the first steps, so the public key of our aptly server is located in the /srv/aptly/.aptly/public/key.pub accessed from the web from “https://aptly.example.com/key.pub“.
Login in your client machine and import the key:
root@srv2:~# wget http://apt.example.com/key.pub root@srv2:~# apt-key add ./key.pub OK
Then replace the official repositories in /etc/apt/sources.list with the new ones:
deb https://aptly.exmaple.com/ubuntu bionic main deb https://aptly.exmaple.com/ubuntu bionic universe deb https://aptly.exmaple.com/ubuntu bionic-updates main deb https://aptly.exmaple.com/ubuntu bionic-updates universe deb https://aptly.exmaple.com/ubuntu bionic-security main deb https://aptly.exmaple.com/ubuntu bionic-security universe
And update to download the metafiles and upgrade or install packages. apt program will access only your local repositories.
root@srv2:~# apt update Get:1 http://aptly.example.com/ubuntu bionic InRelease [22,4 kB] Get:2 http://aptly.example.com/ubuntu bionic-updates InRelease [22,5 kB] Get:3 http://aptly.example.com/ubuntu bionic-security InRelease [22,5 kB] Get:4 http://aptly.example.com/ubuntu bionic/main i386 Packages [1472 kB] Get:5 http://aptly.example.com/ubuntu bionic/main amd64 Packages [1494 kB] Get:6 http://aptly.example.com/ubuntu bionic/universe i386 Packages [12,5 MB] Get:7 http://aptly.example.com/ubuntu bionic/universe amd64 Packages [12,5 MB] Get:8 http://aptly.example.com/ubuntu bionic-updates/main amd64 Packages [1301 kB] Get:9 http://aptly.example.com/ubuntu bionic-updates/main i386 Packages [957 kB] Get:10 http://aptly.example.com/ubuntu bionic-updates/universe i386 Packages [1505 kB] Get:11 http://aptly.example.com/ubuntu bionic-updates/universe amd64 Packages [1568 kB] Get:12 http://aptly.example.com/ubuntu bionic-security/main amd64 Packages [941 kB] Get:13 http://aptly.example.com/ubuntu bionic-security/main i386 Packages [633 kB] Get:14 http://aptly.example.com/ubuntu bionic-security/universe i386 Packages [903 kB] Get:15 http://aptly.example.com/ubuntu bionic-security/universe amd64 Packages [946 kB] Fetched 36,8 MB in 32s (1160 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done 384 packages can be upgraded. Run 'apt list --upgradable' to see them. root@srv:~# apt upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following NEW packages will be installed: libllvm8 libwayland-egl1 linux-headers-4.15.0-74 linux-headers-4.15.0-74-generic linux-image-4.15.0-74-generic linux-modules-4.15.0-74-generic linux-modules-extra-4.15.0-74-generic python3-dateutil xdg-desktop-portal xdg-desktop-portal-gtk The following packages will be upgraded: appstream apt apt-config-icons apt-utils aptdaemon aptdaemon-data apturl apturl-common base-files bash bluez bluez-cups bluez-obexd bolt brltty bsdutils console-setup console-setup-linux debconf debconf-i18n deja-dup desktop-file-utils dmsetup dpkg fdisk fonts-liberation fonts-noto-cjk fonts-noto-color-emoji friendly-recovery fwupd ..... ..... xserver-xorg xserver-xorg-core xserver-xorg-input-all xserver-xorg-legacy xserver-xorg-video-all xwayland 384 upgraded, 10 newly installed, 0 to remove and 0 not upgraded. Need to get 385 MB of archives. After this operation, 467 MB of additional disk space will be used. Do you want to continue? [Y/n] Y Get:1 http://aptly.example.com/ubuntu bionic-updates/main amd64 base-files amd64 10.1ubuntu2.7 [60,3 kB] Get:2 http://aptly.example.com/ubuntu bionic-updates/main amd64 bash amd64 4.4.18-2ubuntu1.2 [614 kB] Get:3 http://aptly.example.com/ubuntu bionic-updates/main amd64 bsdutils amd64 1:2.31.1-0.4ubuntu3.4 [60,3 kB] Get:4 http://aptly.example.com/ubuntu bionic-updates/main amd64 tar amd64 1.29b-2ubuntu0.1 [234 kB] Get:5 http://aptly.example.com/ubuntu bionic-updates/main amd64 dpkg amd64 1.19.0.5ubuntu2.3 [1136 kB] Get:6 http://aptly.example.com/ubuntu bionic-updates/main amd64 grep amd64 3.1-2build1 [159 kB] ...... ...... Adding boot menu entry for EFI firmware configuration done Processing triggers for initramfs-tools (0.130ubuntu3.9) ... update-initramfs: Generating /boot/initrd.img-4.15.0-74-generic Processing triggers for libc-bin (2.27-3ubuntu1) ... root@srv2:~#
You see the accessed URLs for the metafiles and then the downloads are from our local repositories.
Throubleshooting – downloading files
You may encounter some problems, especially during the downloading phase. Some packages might get download error or to be unavailable at the moment, but the setup (mirror update command) will not fail till the end. It just reports an error at the end, printing all the problem files. The solution is simple just execute the command again (aka execute mirror update) and it will try downloading ONLY the problem files and it will successfully download them (in rear situations you may have to change the synchronization mirror):
Downloading http://nl.archive.ubuntu.com/ubuntu/pool/universe/h/haskell-microlens-th/haskell-microlens-th_0.4.1.1.orig.tar.gz... ERROR: unable to update: download errors: http://nl.archive.ubuntu.com/ubuntu/pool/universe/libp/libprotocol-acme-perl/libprotocol-acme-perl_1.01-2_all.deb: Get http:/ubuntu/pool/universe/libp/libprotocol-acme-perl/libprotocol-acme-perl_1.01-2_all.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/p/pyfai/python-pyfai-doc_0.15.0+dfsg1-1_all.deb: Get http:/ubuntu/pool/universe/p/pyfai/python-pyfai-doc_0.15.0%2bdfsg1-1_all.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/b/biniou/libbiniou-ocaml_1.0.12-2build2_amd64.deb: Get http:/ubuntu/pool/universe/b/biniou/libbiniou-ocaml_1.0.12-2build2_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/h/haskell-hsemail/libghc-hsemail-dev_2-1_amd64.deb: Get http:/ubuntu/pool/universe/h/haskell-hsemail/libghc-hsemail-dev_2-1_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/s/sfst/libsfst1-1.4-dev_1.4.7b-1build1_amd64.deb: Get http:/ubuntu/pool/universe/s/sfst/libsfst1-1.4-dev_1.4.7b-1build1_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/j/jmxetric/jmxetric_1.0.6-1.debian.tar.gz: Get http:/ubuntu/pool/universe/j/jmxetric/jmxetric_1.0.6-1.debian.tar.gz: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/j/jmxetric/jmxetric_1.0.6-1.dsc: Get http:/ubuntu/pool/universe/j/jmxetric/jmxetric_1.0.6-1.dsc: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/b/between/between_6+dfsg1-3_amd64.deb: Get http:/ubuntu/pool/universe/b/between/between_6%2bdfsg1-3_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/s/spice-gtk/gir1.2-spiceclientgtk-3.0_0.34-1.1build1_amd64.deb: Get http:/ubuntu/pool/universe/s/spice-gtk/gir1.2-spiceclientgtk-3.0_0.34-1.1build1_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/d/dataquay/libdataquay-dev_0.9.1-1_amd64.deb: Get http:/ubuntu/pool/universe/d/dataquay/libdataquay-dev_0.9.1-1_amd64.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/n/node-repeat-element/node-repeat-element_1.1.2+github.orig.tar.gz: Get http:/ubuntu/pool/universe/n/node-repeat-element/node-repeat-element_1.1.2%2bgithub.orig.tar.gz: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/e/emboss-explorer/libemboss-acd-perl_2.2.0-9_all.deb: Get http:/ubuntu/pool/universe/e/emboss-explorer/libemboss-acd-perl_2.2.0-9_all.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/n/node-prop-types/node-prop-types_15.6.0+ds-1.dsc: Get http:/ubuntu/pool/universe/n/node-prop-types/node-prop-types_15.6.0%2bds-1.dsc: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/r/ruby-cocoon/ruby-cocoon_1.2.6.orig.tar.gz: Get http:/ubuntu/pool/universe/r/ruby-cocoon/ruby-cocoon_1.2.6.orig.tar.gz: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/g/gcc-7-cross-ports/lib32go11-dbg-ppc64-cross_7.3.0-16ubuntu3cross1_all.deb: Get http:/ubuntu/pool/universe/g/gcc-7-cross-ports/lib32go11-dbg-ppc64-cross_7.3.0-16ubuntu3cross1_all.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/w/webkitgtk/libjavascriptcoregtk-3.0-dev_2.4.11-3ubuntu3_i386.deb: Get http:/ubuntu/pool/universe/w/webkitgtk/libjavascriptcoregtk-3.0-dev_2.4.11-3ubuntu3_i386.deb: dial tcp 213.136.12.213:80: i/o timeout http://nl.archive.ubuntu.com/ubuntu/pool/universe/x/xmltooling/libxmltooling-dev_1.6.4-1ubuntu2_i386.deb: Get http:/ubuntu/pool/universe/x/xmltooling/libxmltooling-dev_1.6.4-1ubuntu2_i386.deb: dial tcp 213.136.12.213:80: i/o timeout
The download of multiple file failed with timeout. Just execute mirror update again (with the name of the mirror you encounter the errors before). For example, update mirror with name bionic-main
aptly mirror update bionic-main
It will download only these files, NOT all other (not the other 130G+ files)!
Throubleshooting – create mirror
Another typical mistake is to not import the Ubuntu (or debian if you create debian mirrors) keys, the fillowing error will occur:
aptly@srv:~$ aptly mirror create -architectures=i386,amd64 -with-sources=true -with-udebs=true bionic-main http://nl.archive.ubuntu.com/ubuntu/ bionic main Looks like your keyring with trusted keys is empty. You might consider importing some keys. If you're running Debian or Ubuntu, it's a good idea to import current archive keys by running: gpg --no-default-keyring --keyring /usr/share/keyrings/debian-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import (for Ubuntu, use /usr/share/keyrings/ubuntu-archive-keyring.gpg) Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/InRelease... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Can't check signature: No public key Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver: gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 3B4FE6ACC0B21F32 Sometimes keys are stored in repository root in file named Release.key, to import such key: wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/Release... Downloading http://nl.archive.ubuntu.com/ubuntu/dists/bionic/Release.gpg... gpgv: Signature made Thu 26 Apr 2018 11:38:40 PM UTC gpgv: using RSA key 3B4FE6ACC0B21F32 gpgv: Can't check signature: No public key Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver: gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver keys.gnupg.net --recv-keys 3B4FE6ACC0B21F32 Sometimes keys are stored in repository root in file named Release.key, to import such key: wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2
You should have the Ubuntu repository public key in your trustedkeys keyring of the user, which executes the commands. In our article, we use “aptly” Unix user, so you keys must be imported in its keyring with (the first thing to do in the “STEP 2)” above):
gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import
Note the file is: “/usr/share/keyrings/ubuntu-archive-keyring.gpg” despite what is printed by the error output, it just prints an example, which is for Debian distribution.
Hi,
In company I’m working we have many VM-s. To reduce traffic to public repo I create mirror (currently using apt-mirror).
In DNS server (dnsmasq) used by VM-s I replace IP address of repos.influxdata.com to point to our mirror.
So there is no need to change /etc/apt/sources.list (to replace entry: deb https://repos.influxdata.com/debian buster stable)
However I have problem with apt-mirror and decide to replace it with aptly – it does not download i18n/Translation-en.bz2
and other files that cause that ‘apt update’ failes.
But if I download missing files by hand this solution – with apt-mirror – is working.
So my steps to create mirror (for this example I use repos.influxdata.com, however I also mirror other like deb.debian.org and security.debian.org)
# curl -sL https://repos.influxdata.com/influxdb.key | gpg –import
gpg: key 684A14CF2582E0C5: “InfluxDB Packaging Service ” not changed
gpg: Total number processed: 1
gpg: unchanged: 1
# gpg –no-default-keyring –list-keys –keyring pubring.gpg
/root/.gnupg/pubring.gpg
————————
pub rsa4096 2015-09-28 [SC]
05CE15085FC09D18E99EFB22684A14CF2582E0C5
uid [ unknown] InfluxDB Packaging Service
sub rsa4096 2015-09-28 [E]
# gpg –no-default-keyring –list-keys –keyring trustedkeys.gpg
/root/.gnupg/trustedkeys.gpg
—————————-
[part removed]
pub rsa4096 2015-09-28 [SC]
05CE15085FC09D18E99EFB22684A14CF2582E0C5
uid [ unknown] InfluxDB Packaging Service
sub rsa4096 2015-09-28 [E]
# gpg –no-default-keyring –list-keys –keyring secring.gpg
[empty – secring.pgp size is 0 ]
# aptly mirror create -architectures=amd64 influxdata https://repos.influxdata.com/debian buster stable
# aptly mirror update influxdata
# aptly snapshot create influxdata-2020-04-08 from mirror influxdata
# aptly publish snapshot influxdata-2020-04-08
Loading packages…
Generating metadata files and linking package files…
Finalizing metadata files…
Signing file ‘Release’ with gpg, please enter your passphrase when prompted:
gpg: no default secret key: secret key not available
gpg: signing failed: secret key not available
ERROR: unable to publish: unable to detached sign file: exit status 2
So I try
# curl -sL https://repos.influxdata.com/influxdb.key | gpg –import –no-default-keyring –keyring secring.gpg
gpg: key 684A14CF2582E0C5: public key “InfluxDB Packaging Service ” imported
gpg: Total number processed: 1
gpg: imported: 1
and then again:
# aptly publish snapshot influxdata-2020-04-08
Loading packages…
Generating metadata files and linking package files…
Finalizing metadata files…
Signing file ‘Release’ with gpg, please enter your passphrase when prompted:
gpg: Ohhhh jeeee: … this is a bug (../../g10/getkey.c:2757:lookup)
secmem usage: 1408/1408 bytes in 2/2 blocks of pool 1408/65536
ERROR: unable to publish: unable to detached sign file: signal: aborted
# dpkg -l | egrep ‘aptly|gpg’
ii aptly 1.3.0-5~bpo9+1 amd64 Swiss army knife for Debian repository management – main package
ii gpgv 2.1.18-8~deb9u4 amd64 GNU privacy guard – signature verification tool
ii gpgv1 1.4.21-4+deb9u1 amd64 GNU privacy guard – signature verification tool (deprecated “classic” version)
ii libgpg-error0:amd64 1.26-2 amd64 library for common error values and messages in GnuPG components
ii libgpgme11:amd64 1.8.0-3+b2 amd64 GPGME – GnuPG Made Easy (library)
How could I fix it ?
Thank You very much for any help.
—
Best regards
Lukasz Czyzewski
Fixed.
Unfortunately I don’t know what was the reason.
Just delete and create new VM. Install aptly, gnupg and gpg and ‘aptly publish …’ work then.
I look that in working VM a have more packages:
# dpkg -l | egrep ‘aptly|gpg’
ii gpg 2.2.12-1+deb10u1 amd64 GNU Privacy Guard — minimalist public key operations
ii gpg-agent 2.2.12-1+deb10u1 amd64 GNU privacy guard – cryptographic agent
ii gpg-wks-client 2.2.12-1+deb10u1 amd64 GNU privacy guard – Web Key Service client
ii gpg-wks-server 2.2.12-1+deb10u1 amd64 GNU privacy guard – Web Key Service server
ii gpgconf 2.2.12-1+deb10u1 amd64 GNU privacy guard – core configuration utilities
ii gpgsm 2.2.12-1+deb10u1 amd64 GNU privacy guard – S/MIME version
ii gpgv 2.2.12-1+deb10u1 amd64 GNU privacy guard – signature verification tool
ii libgpg-error0:amd64 1.35-1 amd64 GnuPG development runtime library
—
Best regards
Lukasz Czyzewski