Install aptly under Ubuntu 18 LTS with nginx serving the packages and the first steps

This article is how to install aptly software, which offers easy Debian repository management.
First, few words for aptly and what tasks are really simple to do:

  • Mirror an existing (remote) repository. Make a local copy of Debian or Ubuntu repostories for all your internal infrastructure.
  • Create your own repositories
  • Create snapshots of repositories and mirrors.
  • Merge repositories
  • Make diff between repositories (in fact snapshots of repositories, but you may make a mirror of an repository and then make a snapshot and then make a diff with some other snapshot to see the changes between the different repositories or the time the snapshots are made).
  • Remove or add individual packages from official mirrored repositories.
  • Use api calls to manage the repositories. HTTP REST API is still in development, but a big part of it works.

For more information you may visit the official documentation page – https://www.aptly.info/doc/overview/

We are going to install the aptly and despite it could be used to serve the repository files we will use the Nginx web server for this work. Nginx is a more fast and reliable web server with easy installation of SSL certificates for our repositories.
The aptly is included in official Ubuntu repositories in the component universe, but at present, it is 2 to 3 versions behind the stable one from the aptly site, so we are going to use their repository to install aptly. Still, if you do not want to use

STEP 1) Add the official aptly repository with its GPG key.

Add to your apt repository list file “/etc/apt/sources.list” the following line:

deb http://repo.aptly.info/ squeeze main

And add the GPG key of the aptly repository:

root@srv:~# sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-keys ED75B5A4483DA07C
Executing: /tmp/apt-key-gpghome.b3QjhQc6TS/gpg.1.sh --keyserver pool.sks-keyservers.net --recv-keys ED75B5A4483DA07C
gpg: key ED75B5A4483DA07C: public key "Andrey Smirnov <me@smira.ru>" imported
gpg: Total number processed: 1
gpg:               imported: 1

The GPG key is imported successfully.

STEP 2) Update your system, install aptly and dependencies and Nginx.

Update and install aptly and nginx.

myuser@srv:~# sudo apt update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease                                                                 
Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease                                                               
Get:4 http://repo.aptly.info squeeze InRelease [7,494 B]                 
Hit:5 http://archive.ubuntu.com/ubuntu bionic-security InRelease                                                   
Get:6 http://repo.aptly.info squeeze/main amd64 Packages [3,492 B]              
Fetched 11.0 kB in 1s (12.3 kB/s)     
Reading package lists... Done
Building dependency tree       
Reading state information... Done
All packages are up to date.
myuser@srv:~# sudo apt install -y aptly nginx screen rng-tools
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following additional packages will be installed:
  fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter
  libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core
Suggested packages:
  libgd-tools fcgiwrap nginx-doc ssl-cert
The following NEW packages will be installed:
  aptly fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter
  libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core screen rng-tools
0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded.
Need to get 10.7 MB/13.1 MB of archives.
After this operation, 32.4 MB of additional disk space will be used.
Get:1 http://repo.aptly.info squeeze/main amd64 aptly amd64 1.4.0 [10.7 MB]
.....
.....

STEP 3) Nginx configuration

Because we want to use Nginx despite aptly could be used as a webserver to offer the packages we need a simple Nginx configuration.
Here is the HTTP only configuration:

server {
    server_name aptly.example.com;

    root /srv/aptly/.aptly/public;

    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
}

And HTTP/HTTPS configuration (the let’s encrypt certificate is used):

server {
    listen 80;
    listen 443 ssl http2;
    server_name aptly.example.com;

    ssl_certificate /etc/letsencrypt/live/aptly.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/aptly.example.com/privkey.pem;

    root /srv/aptly/.aptly/public;

    location / {
        autoindex on;
        try_files $uri $uri/ =404;
    }
}

Put one of the texts above in the “/etc/nginx/site-enabled/aptly.exmaple.com.conf”

STEP 4*) Make user for aptly daemon.

This step not mandatory, you may use aptly as root or the user you log in, but it is a good practice to use an additional user for aptly, because all files (cache files for the mirrors or the repository files) will use this user and the aptly configuration would be in the home directory.

myuser@srv:~# sudo groupadd -g 1010 aptly
myuser@srv:~# sudo mkdir /srv/aptly
myuser@srv:~# sudo useradd -u 1010 -g 1010 aptly -s /bin/bash -d /srv/aptly
myuser@srv:~# sudo chown aptly:aptly -R /srv/aptly/

The last line will make you aptly user, which must be used to execute commands when managing the aptly repositories. As you can see we changed the home directory because we want to use a different home directory for the packages’ cache.

STEP 5) Generate repository key, which will be used to sign the published repositories.

You must import the public key to the server (and the key should be publicly available in your repository and out Nginx could do the job), which will use the repository, but first let’s create the public and private key (the private key is used for signing).
First, start the kernel random generator feeder and then switch the user to the “aptly”.

myuser@srv:~# sudo systemctl start rng-tools
myuser@srv:~# sudo su - aptly
aptly@srv:~#

Create a GPG2 batch file in the “aptly” user home directory (in our case – /srv/aptly) /srv/aptly/gpg2_generate_batch_file.txt

aptly@srv:~# cd
aptly@srv:~# cat >/srv/aptly/gpg2_generate_batch_file.txt <<EOF
%echo Generating a default key
Key-Type: RSA
Key-Length: 4096
Name-Real: MyCompanyName
Name-Comment: aptly key no passphrase
Name-Email: info@mycompanyname.com
Expire-Date: 0
%no-protection
# Do a commit here, so that we can later print "done" :-)
%commit
%echo done
EOF

Generate the keys

aptly@srv:~# screen
aptly@srv:~# gpg --batch --gen-key /srv/aptly/gpg2_generate_batch_file.txt
gpg: directory '/srv/aptly/.gnupg' created
gpg: keybox '/srv/aptly/.gnupg/pubring.kbx' created
gpg: Generating a default key
gpg: /srv/aptly/.gnupg/trustdb.gpg: trustdb created
gpg: key B0E10740DD603265 marked as ultimately trusted
gpg: directory '/srv/aptly/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/srv/aptly/.gnupg/openpgp-revocs.d/779CA0186E6747FC7BBCE153B0E10740DD603265.rev'
gpg: done

We need “screen” (execute screen before the generation of the keys) because you will get GPG generation error. So just execute screen and generate the key in the screen console. For more information check the troubleshooting section. In short, the generation of GPG keys fails because we switch the user with “su”.

aptly@srv:~# ls -altr .gnupg/
total 28
drwxr-xr-x 4 aptly aptly 4096 Oct 27 06:18 ..
-rw------- 1 aptly aptly   32 Oct 27 06:18 pubring.kbx~
srwx------ 1 aptly aptly    0 Oct 27 06:18 S.gpg-agent.ssh
srwx------ 1 aptly aptly    0 Oct 27 06:18 S.gpg-agent.extra
srwx------ 1 aptly aptly    0 Oct 27 06:18 S.gpg-agent.browser
srwx------ 1 aptly aptly    0 Oct 27 06:18 S.gpg-agent
drwx------ 2 aptly aptly 4096 Oct 27 06:18 private-keys-v1.d
-rw------- 1 aptly aptly 1240 Oct 27 06:18 trustdb.gpg
-rw-rw-r-- 1 aptly aptly 1369 Oct 27 06:18 pubring.kbx
drwx------ 2 aptly aptly 4096 Oct 27 06:18 openpgp-revocs.d
drwx------ 4 aptly aptly 4096 Oct 27 06:18 .
aptly@srv:~# gpg --list-secret-keys
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
/srv/aptly/.gnupg/pubring.kbx
-----------------------------
sec   rsa4096 2019-10-27 [SCEA]
      779CA0186E6747FC7BBCE153B0E10740DD603265
uid           [ultimate] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>

aptly@srv:~# gpg --list-keys
/srv/aptly/.gnupg/pubring.kbx
-----------------------------
pub   rsa4096 2019-10-27 [SCEA]
      779CA0186E6747FC7BBCE153B0E10740DD603265
uid           [ultimate] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>

With this key we are going to sign our repository, so we need a way to import the public key in the machines where we are going to use our local repository. Export the public key and put it in the Nginx’s document root directory (of the aptly virtual host) so we can access the key downloading it from our repository URL.
We’ve showed above how to list the public (–list-keys) and the private (–list-secret-keys). Just use the ID of the show command to export the key.

aptly@srv:~# mkdir -p /srv/aptly/.aptly/public
aptly@srv:~# gpg --output /srv/aptly/.aptly/public/key.pub --armor --export 779CA0186E6747FC7BBCE153B0E10740DD603265
aptly@srv:~# cat /srv/aptly/.aptly/public/key.pub 
-----BEGIN PGP PUBLIC KEY BLOCK-----

mQINBF21NqQBEAC1ldbeZw1i7Wc6X6e5tYGn7vnj1Pp1UCBkUHCnt6pBBIs1ZukP
......
......
N/t0tnT7HEaeGy52eg5ttnOh+j7GRlAsJCEaOi1OLoJp2Bv9MDmGZAj0D3FTlSpZ
ckiUMZwhbAa6ekOxGIH133E4LKGBcvLBeQR9bohN2NWyARs=
=i1Y2
-----END PGP PUBLIC KEY BLOCK-----

STEP 6*) Aptly configuration

This step is also not mandatory, because the default values are OK for most cases. Especially, if you make an additional aptly user and its home directory resides on the desired big (caching) partition because mirroring of repositories may require 200-400Gbytes.
The configuration file uses JSON format and you can check the default values here – https://www.aptly.info/doc/configuration/. Worth mentioning the following default values:

{
  "rootDir": "$HOME/.aptly",
  "downloadConcurrency": 4,
  "downloadSpeedLimit": 0,
  "architectures": [],
  "dependencyFollowSuggests": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowAllVariants": false,
  "dependencyFollowSource": false,
  "dependencyVerboseResolve": false,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "gpg",
  "downloadSourcePackages": false,
  "ppaDistributorID": "ubuntu",
  "ppaCodename": "",
  "FileSystemPublishEndpoints": {
    "test": {
      "rootDir": "/opt/aptly-publish",
      "linkMethod": "copy",
      "verifyMethod": "md5"
    }
}

Aptly supports amazon S3 and OpenStack Swift endpoints, too.
All files (packages) will be under “$HOME/.aptly” and in our user aptly – “/srv/aptly/.aptly/” and the configuration will be “/srv/aptly/.aptly.conf”. Change the default values accordingly.

Using aptly for the first time

Change to the aptly user (if you have followed the STEP (4)) and execute the “config show” command to see the currently loaded configuration:

aptly@srv:~# sudo su - aptly
aptly@srv:~# aptly config show
{
    "rootDir": "/srv/aptly/.aptly",
    "downloadConcurrency": 4,
    "downloadSpeedLimit": 0,
    "architectures": [],
    "dependencyFollowSuggests": false,
    "dependencyFollowRecommends": false,
    "dependencyFollowAllVariants": false,
    "dependencyFollowSource": false,
    "dependencyVerboseResolve": false,
    "gpgDisableSign": false,
    "gpgDisableVerify": false,
    "gpgProvider": "gpg",
    "downloadSourcePackages": false,
    "skipLegacyPool": true,
    "ppaDistributorID": "ubuntu",
    "ppaCodename": "",
    "skipContentsPublishing": false,
    "FileSystemPublishEndpoints": {},
    "S3PublishEndpoints": {},
    "SwiftPublishEndpoints": {}
}
aptly@srv:~#

Aptly cli commands:

aptly@srv:~# aptly 
aptly - Debian repository management tool

Commands:

    api         start API server/issue requests
    config      manage aptly configuration
    db          manage aptly's internal database and package pool
    graph       render graph of relationships
    mirror      manage mirrors of remote repositories
    package     operations on packages
    publish     manage published repositories
    repo        manage local package repositories
    serve       HTTP serve published repositories
    snapshot    manage snapshots of repositories
    task        manage aptly tasks
    version     display version

Use "aptly help <command>" for more information about a command.


Options:
  -architectures="": list of architectures to consider during (comma-separated), default to all available
  -config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
  -db-open-attempts=10: number of attempts to open DB if it's locked by other instance
  -dep-follow-all-variants: when processing dependencies, follow a & b if dependency is 'a|b'
  -dep-follow-recommends: when processing dependencies, follow Recommends
  -dep-follow-source: when processing dependencies, follow from binary to Source packages
  -dep-follow-suggests: when processing dependencies, follow Suggests
  -dep-verbose-resolve: when processing dependencies, print detailed logs
  -gpg-provider="": PGP implementation ("gpg", "gpg1", "gpg2" for external gpg or "internal" for Go internal implementation)
ERROR: unable to parse command

Import the public key

To use your local repository you must import the public key with apt in the machine, where you want to use the repository. Donwload the key and import it with:

myuser@my-server-pc:~$ wget http://apt.example.com/key.pub
--2019-10-27 06:48:32--  http://apt.example.com/key.pub
Resolving apt.example.com (apt.example.com)... 127.0.0.1
Connecting to apt.example.com (apt.example.com)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1688 (1.6K) [application/octet-stream]
Saving to: ‘key.pub’

key.pub                                    100%[========================================================================================>]   1.65K  --.-KB/s    in 0s      

2019-10-27 06:48:32 (253 MB/s) - ‘key.pub’ saved [1688/1688]
myuser@my-server-pc:~$ sudo apt-key add key.pub 
OK
myuser@my-server-pc:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub   rsa4096 2018-03-15 [SC] [expires: 2020-03-14]
      26DA 9D86 3030 2E0B 86A7  A2CB ED75 B5A4 483D A07C
uid           [ unknown] Andrey Smirnov <me@smira.ru>
sub   rsa4096 2018-03-15 [E] [expires: 2020-03-14]

pub   rsa4096 2019-10-27 [SCEA]
      779C A018 6E67 47FC 7BBC  E153 B0E1 0740 DD60 3265
uid           [ unknown] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      790B C727 7767 219C 42C8  6F93 3B4F E6AC C0B2 1F32
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub   rsa4096 2012-05-11 [SC]
      8439 38DF 228D 22F7 B374  2BC0 D94A A3F0 EFE2 1092
uid           [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>

/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub   rsa4096 2018-09-17 [SC]
      F6EC B376 2474 EDA9 D21B  7022 8719 20D1 991B C93C
uid           [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>

Troubleshooting

Getting this error when generating the key you missed to execute “screen” before the command.

aptly@srv:~$ gpg --batch --gen-key /srv/aptly/gpg2_generate_batch_file.txt
gpg: directory '/srv/aptly/.gnupg' created
gpg: keybox '/srv/aptly/.gnupg/pubring.kbx' created
gpg: Generating a default key
gpg: agent_genkey failed: Permission denied
gpg: key generation failed: Permission denied
gpg: done

This is probably because we switch the user with su and the tty is still to our old user:

aptly@srv:~# ls -al $(tty)
crw--w---- 1 myuser tty 136, 0 Oct 27 07:02 /dev/pts/0

It’s “aptly” user, but the owner is the “myuser”.

Just execute screen to enter a new shell and generate the keys, then you can leave with exit.

We’ve included the screen package in the install line in the first step.

Leave a Reply

Your email address will not be published. Required fields are marked *