aptly delete a mirror and remove all files

Executing drop command on a mirror will only remove the meta information for the mirror and it will not remove the package files occupying space on the file system.

Dropping mirror in aptly supposes to execute a clean command with aplty

aptly db cleanup

The newly created Bionic mirrors in the prevoius article on the aptly subject – Mirror the official Ubuntu repositories using aptly will be deleted here and removing all files with:

aptly@srv:~$ aptly mirror drop bionic-main
Mirror `bionic-main` has been removed.
aptly@srv:~$ aptly mirror drop bionic-security-main
Mirror `binonic-security-main` has been removed.
aptly@srv:~$ aptly mirror drop bionic-universe     
Mirror `bionic-universe` has been removed.
aptly@srv:~$ aptly mirror drop bionic-updates-main
Mirror `binonic-updates-main` has been removed.
aptly@srv:~$ aptly mirror drop bionic-updates-universe
Mirror `bionic-updates-universe` has been removed.
aptly@srv:~$ aptly mirror list
No mirrors found, create one with `aptly mirror create ...`.

The occupied space on the disk mounted in /srv is 270G:

aptly@srv:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  3.5M  391M   1% /run
/dev/sda3        19G  4.6G   13G  27% /
tmpfs           2.0G  204K  2.0G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda4       470G  270G  176G  61% /srv
tmpfs           395M     0  395M   0% /run/user/0
tmpfs           395M     0  395M   0% /run/user/1001

Actually freeing the space on the disk with the clean aptly command:

aptly@srv:~$ aptly db cleanup
Loading mirrors, local repos, snapshots and published repos...
Loading list of all packages...
Deleting unreferenced packages (143121)...
Building list of files referenced by packages...
Building list of files in package pool...
Deleting unreferenced files (194097)...
Disk space freed: 268.80 GiB...
Compacting database...

The occupied space on the disk mounted in /srv is below 2G after the cleaning command:

aptly@srv:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           395M  3.5M  391M   1% /run
/dev/sda3        19G  4.6G   13G  27% /
tmpfs           2.0G  204K  2.0G   1% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda4       470G    1G  176G   1% /srv
tmpfs           395M     0  395M   0% /run/user/0
tmpfs           395M     0  395M   0% /run/user/1001

aptly remove a package from a repository using the cli

Here is a fast tip – how to remove a package from our local aptly repository:

  1. Remove the package from the local repository.
  2. Create a new snapshot form the local repository.
  3. Publish the snapshot by switching to the newly created snapshot from the above step.

The commands executing over repository with name xenial-apps to remove package with name example-app and version 10.5.1.22-ubuntu20. The snapshot name xenial-apps1588149526 is just a temporary name used for the snapshot (the ID is unix timestamp of the current time).

aptly repo remove  xenial-apps 'example-app (= 10.5.1.22-ubuntu20)'
aptly snapshot create xenial-apps1588149526 from repo xenial-apps
aptly publish switch xenial-apps ubuntu xenial-apps1588149526

Real world example.

This is the log from our system with just changed names:
Keep on reading!

Mirror a PPA repositories using aptly – PHP (ppa:ondrej/php)

This is a simple example of how to mirror a PPA repository to a local server. The Ubuntu PPA to mirror is ppa:ondrej/php, which offers the user different PHP version generally not available in the Ubuntu installation. Of course, the user should be very careful about adding PPA repositories, because they are exactly what the abbreviation stands for Personal Package Archives.

If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps

What we are going to do – this is what you need to have a mirror of an external application repository:

  1. Install aptly in Ubuntu 18 LTS
  2. Create a mirror in aptly
  3. Create a snapshot of the mirror created before
  4. Publish the snapshot to be used in other servers.

and at the last step there is an example how to use the mirror in your local machines.

STEP 1) Install aptly in Ubuntu 18.04 LTS.

As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.
Change the user to aptly, because under this user the mirror process will happen.

root@srv ~ # su - aptly
aptly@srv:~$

STEP 2) Create a mirror in aptly.

Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):

aptly@srv:~$ gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver pool.sks-keyservers.net --recv-keys 4F4EA0AAE5267A6C
gpg: requesting key E5267A6C from hkp server pool.sks-keyservers.net
gpg: key E5267A6C: public key "Launchpad PPA for Ond\xc5\x99ej Sur�" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Here we’ve used the method to obtain the key from a GPG KEY server, but the key can be downloaded directrly from the original repository as suggested in the error message below.
If you are not sure where to download the key you could always just try to create the mirror ( in fact, this is in STEP 3) ) and get the error for missing key and how to obtain the key:
Keep on reading!

Mirror the official Ubuntu repositories using aptly

This article is to show mainly how to work with aptly by mirroring an official Ubuntu mirror. If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps

What we are going to do – this is what you need to have a mirror of an external application repository:

  1. Install aptly in Ubuntu 18 LTS
  2. Create a mirror in aptly
  3. Create a snapshot of the mirror created before
  4. Publish the snapshot to be used in other servers.

and at the last step there is an example how to use the mirror in your local machines.

STEP 1) Install aptly in Ubuntu 18.04 LTS.

As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.

STEP 2) Create a mirror in aptly.

Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):

aptly@srv:~$ gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import
gpg: key 3B4FE6ACC0B21F32: 3 signatures not checked due to missing keys
gpg: key 3B4FE6ACC0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" imported
gpg: key D94AA3F0EFE21092: 3 signatures not checked due to missing keys
gpg: key D94AA3F0EFE21092: public key "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" imported
gpg: key 871920D1991BC93C: 1 signature not checked due to a missing key
gpg: key 871920D1991BC93C: public key "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" imported
gpg: Total number processed: 3
gpg:               imported: 3
gpg: public key of ultimately trusted key 212A3D20E4D3351D not found
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

Probably you would like to have “main” and “universe” for the three bionic, bionic updates and bionic security.
First, main and universe for bionic repository. main is ~16G and universe is ~136, these numbers will vary in future when more packages are added.
Two commands are need for the aptly mirror:

  1. create – create the mirror.
  2. update – download the repository contents locally.

Keep on reading!

aplty – unable to find control.tar.gz part in package – change deb package compression from xz to gzip

We upgraded to a new version of Ubuntu and our CI (continuous integration) scripts began to throw errors when uploading packages to out aptly repository:

"Report":{"Warnings":["Unable to read file /srv/aptly/.aptly/upload/mysoft/mysoft-6.15-pk19.deb: unable to find control.tar.gz part in package /srv/aptly/.aptly/upload/mysoft/mysoft-6.15-pk19.deb"],"Added":[],"Removed":[]}}

But if the same command:

dpkg-deb --build $PKGNAME

is executed on our older Ubuntu 16 everyhting is perfect and no error when uploading the package in the repository.

It turns out the new version of dpkg 1.19.0 the dpkg-deb will compress the deb file with XZ by default. Before version 1.19 the default compression is gzip.

You may upgrade your aptly installation to 1.20 and above or just fix your script to use “-Zgzip” with dpkg-deb

dpkg-deb -Zgzip --build $PKGNAME

This command will force the dpkg-deb to use gzip to compress the debian package.

Change the compression of existing deb package

Thanks to the aptly bug report – https://github.com/aptly-dev/aptly/issues/655 (hopes this link stays forever) you may have a workarround to decompress and comrpess an existing package with antoher algorythm

dpkg-deb -R package.deb tmp
rm package.deb
fakeroot dpkg-deb -Zgzip -b tmp package.deb
rm -rf tmp

Install aptly under Ubuntu 18 LTS with nginx serving the packages and the first steps

This article is how to install aptly software, which offers easy Debian repository management.
First, few words for aptly and what tasks are really simple to do:

  • Mirror an existing (remote) repository. Make a local copy of Debian or Ubuntu repostories for all your internal infrastructure.
  • Create your own repositories
  • Create snapshots of repositories and mirrors.
  • Merge repositories
  • Make diff between repositories (in fact snapshots of repositories, but you may make a mirror of an repository and then make a snapshot and then make a diff with some other snapshot to see the changes between the different repositories or the time the snapshots are made).
  • Remove or add individual packages from official mirrored repositories.
  • Use api calls to manage the repositories. HTTP REST API is still in development, but a big part of it works.

For more information you may visit the official documentation page – https://www.aptly.info/doc/overview/

We are going to install the aptly and despite it could be used to serve the repository files we will use the Nginx web server for this work. Nginx is a more fast and reliable web server with easy installation of SSL certificates for our repositories.
The aptly is included in official Ubuntu repositories in the component universe, but at present, it is 2 to 3 versions behind the stable one from the aptly site, so we are going to use their repository to install aptly. Still, if you do not want to use
Keep on reading!

aptly publish ERROR: unable to publish: unable to process packages: error linking file to

We’ve encountered the following error when issuing a publish command:

aptly@aptly-server:~$ aptly --config=/mnt/storage/aptly/.aptly.conf publish snapshot xenial-myrepo-initial ubuntu
Loading packages...
Generating metadata files and linking package files...
ERROR: unable to publish: unable to process packages: error linking file to /mnt/storage/aptly/.aptly/public/ubuntu/pool/main/s/sftpcloudfs/sftpcloudfs_0.12.2-2_all.deb: file already exists and is different

And the snapshot had failed to publish. Check if the file is “aptly:aptly” (or the user and group your installation uses) because if someone has executed commands from the user root it may create some files with the user root (or other) and after that, some commands could fail. In our case, the file was with the right user for aptly and the solution was to remove the file manually (i.e. it is safe to remove it!) it was created again by the setup in the right time. Then execute the publish command again:

aptly@aptly-server:~$ rm /mnt/storage/aptly/.aptly/public/ubuntu/pool/main/s/sftpcloudfs/sftpcloudfs_0.12.2-2_all.deb 
aptly@aptly-server:~$ aptly --config=/mnt/storage/aptly/.aptly.conf publish snapshot xenial-myrepo-initial ubuntu
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:

Snapshot xenial-myrepo-initial has been successfully published.
Please setup your webserver to serve directory '/mnt/storage/aptly/.aptly/public' with autoindexing.
Now you can add following line to apt sources:
  deb http://your-server/ubuntu/ xenial-myrepo main
  deb-src http://your-server/ubuntu/ xenial-myrepo main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.

Common mistakes to appear this error are

  • File permissions
  • File ownership. As mentioned above aptly command executed by other user (like root). Probably it is a good idea to chown recursively the whole aptly root directory
  • Inerrupting the publish command execution
  • Inerrupting the drop command execution

The solution is simple, just remove the offensive file(s) and execute the command again. It is safe to remove the file manually.

aptly publish: gpg: no default secret key: secret key not available

This is also a common error in a typical aptly installation. The other two common errors related to the GPG keys are: aptly publish: ERROR: unable to initialize GPG signer. Missing pubring.gpg keys and aptly mirror – gpgv: Can’t check signature: public key not found. This secret key is used when you try to publish a repository (snapshot or mirror).

root@srv-aptly ~ # aptly publish snapshot xenial-myrepo-initial
Loading packages...
Generating metadata files and linking package files...
 15683 / 107250 [====================>--------------------------------------------------------------------------------------------------------------------]  14.62% 2h53m50s 
17025 / 107250 [=====================>--------------------------------------------------------------------------------------------------------------------]  15.87% 3h5m15sFinalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
gpg: no default secret key: secret key not available
gpg: signing failed: secret key not available
ERROR: unable to publish: unable to detached sign file: exit status 2

You are unable to sign the Release file because the keyring secring.gpg is missing a GPG key. Just create or import from your current servers the GPG key from keyring secring.gpg (for the root user it is /root/.gnupg/secring.gpg and in general this is the default path /[my-aptly-home-directory]/.gnupg/secring.gpg).

Here is the example with the two servers, exporting from your current and importing the key in your new (the second) server:

Export the secring.gpg GPG key from your server
root@srv-aptly-1:~ # gpg --list-keys --keyring secring.gpg
/root/.gnupg/secring.gpg
------------------------
pub   2048D/FDC7A25E 2017-09-16
uid                  My-aptly (aptly key no passphrase) <my-aptly@example.com>

root@srv-aptly-1:~ # gpg --keyring secring.gpg --export --armor FDC7A25E > FDC7A25E.key
root@srv-aptly-1:~ # gpg --list-secret-keys --keyring secring.gpg
/root/.gnupg/secring.gpg
------------------------
sec   2048D/FDC7A25E 2017-09-16
uid                  My-aptly (aptly key no passphrase) <my-aptly@example.com>

root@srv-aptly-1:~ # gpg --keyring secring.gpg --export-secret-key --armor FDC7A25E > FDC7A25E.sec

First is the public key (FDC7A25E.key) and second is the private key (FDC7A25E.sec). You must export them both and import them in your new server (or look below how to generate them in your server).

Copy the file to the second server (FDC7A25E.key) and then import it in keyring secring.gpg
root@srv-aptly-2:~ # cat ./FDC7A25E.key| gpg --keyring secring.gpg --import
gpg: key FDC7A25E: public key "My-aptly (aptly key no passphrase) <my-aptly@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@srv-aptly-2:~ # gpg --keyring secring.gpg --allow-secret-key-import --armor --import FDC7A25E.sec 
gpg: key FDC7A25E: secret key imported
gpg: key FDC7A25E: "My-aptly (aptly key no passphrase) <my-aptly@example.com>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1
gpg:       secret keys read: 1
gpg:   secret keys imported: 1

And now you can publish your repository with:

root@srv-aptly-2: ~ # aptly publish snapshot xenial-myrepo-initial ubuntu
Loading packages...
Generating metadata files and linking package files...
Finalizing metadata files...
Signing file 'Release' with gpg, please enter your passphrase when prompted:
Clearsigning file 'Release' with gpg, please enter your passphrase when prompted:

Snapshot xenial-myrepo-initial has been successfully published.
Please setup your webserver to serve directory '/mnt/storage/aptly/.aptly/public' with autoindexing.
Now you can add following line to apt sources:
  deb http://your-server/ubuntu/ xenial-myrepo main
  deb-src http://your-server/ubuntu/ xenial-myrepo main
Don't forget to add your GPG key to apt with apt-key.

You can also use `aptly serve` to publish your repositories over HTTP quickly.

The operation publish passed successfully.

Generate GPG Key

If you just came here installing a new aptly server and getting this error as mentioned above you miss a GPG key in keyring secring.gpg.

root@srv-aptly: ~# gpg --default-new-key-algo rsa4096 --gen-key --keyring secring.gpg
gpg (GnuPG) 2.2.11; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: My-aptly
Email address: my-aptly@example.com
You selected this USER-ID:
    "MyName <my-aptly@example.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key B14B67D0CF27191B marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/77EC42A1F16127C83509292BB14B67D0CF27191B.rev'
public and secret key created and signed.

Note that this key cannot be used for encryption.  You may want to use
the command "--edit-key" to generate a subkey for this purpose.
pub   rsa4096 2019-07-08 [SC] [expires: 2021-07-07]
      77EC42A1F16127C83509292BB14B67D0CF27191B
uid                      MyName <my-aptly@example.com>

NOTE

Just to note here we give you all the examples with the root user and the GPG keys are for the root user. You may use a different user for the aptly process and you must ensure the GPG keys to present for this user (the directories and files are the same, just home directory is different – the home directory of the aptly user i.e. “/[my-aptly-home-directory]/.gnupg/secring.gpg” and for all other GPG files “/[my-aptly-home-directory]/.gnupg/”).

aptly publish: ERROR: unable to initialize GPG signer. Missing pubring.gpg keys

In continuation of our aptly common mistakes here one more when making a second mirror aptly server to your master (you may encounter this error in many other situations, not only building a mirror aptly server). Again the problem is the GPG key like this one – aptly mirror – gpgv: Can’t check signature: public key not found this time the problem occurs when you try getting snapshot of your mirror repository.

By default Aptly uses the GNU key in keyring pubring.gpg (/root/.gnupg/pubring.gpg for the root user)

And even you may have the same key in other keyrings like trustedkeys.gpg you won’t be able to use them for signing process with the aptly snapshot.

Here is the error:

root@srv-aptly-2:~ # aptly publish snapshot myrepo-initial
ERROR: unable to initialize GPG signer: looks like there are no keys in gpg, please create one (official manual: http://www.gnupg.org/gph/en/manual.html)

The solution is to export the key from pubring.gpg keyring and then import the GPG key in keyring pubring.gpg in the new server. And then you won’t receive the error when making a snapshot with aptly. Or if your case is not making a second server, but your first aptly server you must generate the GPG key in pubring.gpg (look at the end how to do it and skip the lines below for GPU key export and import).

Export the pubring.gpg GPG key from your server
root@srv-aptly-1:~ # gpg --list-keys --keyring pubring.gpg
/root/.gnupg/pubring.gpg
------------------------
pub   2048D/FDC7A25E 2017-09-16
uid                  My-aptly (aptly key no passphrase) <my-aptly@example.com>

root@srv-aptly-1:~ # gpg --keyring pubring.gpg --export --armor FDC7A25E > FDC7A25E.key
Copy the file to the second server (FDC7A25E.key) and then import it in keyring pubring.gpg
root@srv-aptly-2:~ # cat ./FDC7A25E.key| gpg --keyring pubring.gpg --import
gpg: key FDC7A25E: public key "My-aptly (aptly key no passphrase) <my-aptly@example.com>" imported
gpg: Total number processed: 1
gpg:               imported: 1
root@srv-aptly-2:~ # aptly publish snapshot myrepo-initial
Loading packages...
Generating metadata files and linking package files...
 15683 / 107250 [====================>--------------------------------------------------------------------------------------------------------------------]  14.62% 2h53m50s 17025 / 107250 [=====================>--------------------------------------------------------------------------------------------------------------------]  15.87% 3h5m15sFinalizing metadata files..

Generate GPG Key

If you just came here installing a new aptly server and getting this error as mentioned above you miss a GPG key in keyring pubring.gpg.

root@srv-aptly: ~# gpg --default-new-key-algo rsa4096 --gen-key --keyring pubring.gpg
gpg (GnuPG) 2.2.11; Copyright (C) 2018 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Note: Use "gpg --full-generate-key" for a full featured key generation dialog.

GnuPG needs to construct a user ID to identify your key.

Real name: My-aptly
Email address: my-aptly@example.com
You selected this USER-ID:
    "MyName <my-aptly@example.com>"

Change (N)ame, (E)mail, or (O)kay/(Q)uit? O
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key B14B67D0CF27191B marked as ultimately trusted
gpg: revocation certificate stored as '/root/.gnupg/openpgp-revocs.d/77EC42A1F16127C83509292BB14B67D0CF27191B.rev'
public and secret key created and signed.

Note that this key cannot be used for encryption.  You may want to use
the command "--edit-key" to generate a subkey for this purpose.
pub   rsa4096 2019-07-08 [SC] [expires: 2021-07-07]
      77EC42A1F16127C83509292BB14B67D0CF27191B
uid                      MyName <my-aptly@example.com>

NOTE

Just to note here we give you all the examples with the root user and the GPG keys are for the root user. You may use a different user for the aptly process and you must ensure the GPG keys to present for this user (the directories and files are the same, just home directory is different – the home directory of the aptly user i.e. “/[my-aptly-home-directory]/.gnupg/pubring.gpg” and for all other GPG files “/[my-aptly-home-directory]/.gnupg/”).

aptly mirror: ERROR: unable to update: no candidates for debian-installer/binary-amd64/Packages found

Always check the source what supports when trying to mirror! We have lost some time before discovering that our source repository does not support udeb and source packages! If you create a mirror with “-with-sources=true -with-udebs=true” the update process will require files, which may not exists in the source repository if it does not offer udeb or source files and you’ll end up with broken mirror and error for missing file!

Downloading & parsing package files...
Downloading http://aptly.example.com/ubuntu/dists/xenial-myrepos/main/binary-amd64/Packages.bz2...
ERROR: unable to update: no candidates for http://aptly-master.example.com/ubuntu/dists/xenial-myrepo/main/debian-installer/binary-amd64/Packages found

If you get error for “debian-installer/binary-amd64/Packages” not found, check the source repository if it offers udeb and/or source packages – probably not, so drop your mirror and recreate it including one or the two options

-with-sources=false -with-udebs=false

Keep on reading!