This article is how to install aptly software, which offers easy Debian repository management.
First, few words for aptly and what tasks are really simple to do:
- Mirror an existing (remote) repository. Make a local copy of Debian or Ubuntu repostories for all your internal infrastructure.
- Create your own repositories
- Create snapshots of repositories and mirrors.
- Merge repositories
- Make diff between repositories (in fact snapshots of repositories, but you may make a mirror of an repository and then make a snapshot and then make a diff with some other snapshot to see the changes between the different repositories or the time the snapshots are made).
- Remove or add individual packages from official mirrored repositories.
- Use api calls to manage the repositories. HTTP REST API is still in development, but a big part of it works.
For more information you may visit the official documentation page – https://www.aptly.info/doc/overview/
We are going to install the aptly and despite it could be used to serve the repository files we will use the Nginx web server for this work. Nginx is a more fast and reliable web server with easy installation of SSL certificates for our repositories.
The aptly is included in official Ubuntu repositories in the component universe, but at present, it is 2 to 3 versions behind the stable one from the aptly site, so we are going to use their repository to install aptly. Still, if you do not want to use
STEP 1) Add the official aptly repository with its GPG key.
Add to your apt repository list file “/etc/apt/sources.list” the following line:
deb http://repo.aptly.info/ squeeze main
And add the GPG key of the aptly repository:
root@srv:~# sudo apt-key adv --keyserver pool.sks-keyservers.net --recv-keys ED75B5A4483DA07C Executing: /tmp/apt-key-gpghome.b3QjhQc6TS/gpg.1.sh --keyserver pool.sks-keyservers.net --recv-keys ED75B5A4483DA07C gpg: key ED75B5A4483DA07C: public key "Andrey Smirnov <me@smira.ru>" imported gpg: Total number processed: 1 gpg: imported: 1
The GPG key is imported successfully.
STEP 2) Update your system, install aptly and dependencies and Nginx.
Update and install aptly and nginx.
myuser@srv:~# sudo apt update Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease Hit:3 http://archive.ubuntu.com/ubuntu bionic-backports InRelease Get:4 http://repo.aptly.info squeeze InRelease [7,494 B] Hit:5 http://archive.ubuntu.com/ubuntu bionic-security InRelease Get:6 http://repo.aptly.info squeeze/main amd64 Packages [3,492 B] Fetched 11.0 kB in 1s (12.3 kB/s) Reading package lists... Done Building dependency tree Reading state information... Done All packages are up to date. myuser@srv:~# sudo apt install -y aptly nginx screen rng-tools Reading package lists... Done Building dependency tree Reading state information... Done The following additional packages will be installed: fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx-common nginx-core Suggested packages: libgd-tools fcgiwrap nginx-doc ssl-cert The following NEW packages will be installed: aptly fontconfig-config fonts-dejavu-core libfontconfig1 libgd3 libjbig0 libjpeg-turbo8 libjpeg8 libnginx-mod-http-geoip libnginx-mod-http-image-filter libnginx-mod-http-xslt-filter libnginx-mod-mail libnginx-mod-stream libtiff5 libwebp6 libxpm4 nginx nginx-common nginx-core screen rng-tools 0 upgraded, 19 newly installed, 0 to remove and 0 not upgraded. Need to get 10.7 MB/13.1 MB of archives. After this operation, 32.4 MB of additional disk space will be used. Get:1 http://repo.aptly.info squeeze/main amd64 aptly amd64 1.4.0 [10.7 MB] ..... .....
STEP 3) Nginx configuration
Because we want to use Nginx despite aptly could be used as a webserver to offer the packages we need a simple Nginx configuration.
Here is the HTTP only configuration:
server {
server_name aptly.example.com;
root /srv/aptly/.aptly/public;
location / {
autoindex on;
try_files $uri $uri/ =404;
}
}
And HTTP/HTTPS configuration (the let’s encrypt certificate is used):
server {
listen 80;
listen 443 ssl http2;
server_name aptly.example.com;
ssl_certificate /etc/letsencrypt/live/aptly.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/aptly.example.com/privkey.pem;
root /srv/aptly/.aptly/public;
location / {
autoindex on;
try_files $uri $uri/ =404;
}
}
Put one of the texts above in the “/etc/nginx/site-enabled/aptly.exmaple.com.conf”
STEP 4*) Make user for aptly daemon.
This step not mandatory, you may use aptly as root or the user you log in, but it is a good practice to use an additional user for aptly, because all files (cache files for the mirrors or the repository files) will use this user and the aptly configuration would be in the home directory.
myuser@srv:~# sudo groupadd -g 1010 aptly myuser@srv:~# sudo mkdir /srv/aptly myuser@srv:~# sudo useradd -u 1010 -g 1010 aptly -s /bin/bash -d /srv/aptly myuser@srv:~# sudo chown aptly:aptly -R /srv/aptly/
The last line will make you aptly user, which must be used to execute commands when managing the aptly repositories. As you can see we changed the home directory because we want to use a different home directory for the packages’ cache.
STEP 5) Generate repository key, which will be used to sign the published repositories.
You must import the public key to the server (and the key should be publicly available in your repository and out Nginx could do the job), which will use the repository, but first let’s create the public and private key (the private key is used for signing).
First, start the kernel random generator feeder and then switch the user to the “aptly”.
myuser@srv:~# sudo systemctl start rng-tools myuser@srv:~# sudo su - aptly aptly@srv:~#
Create a GPG2 batch file in the “aptly” user home directory (in our case – /srv/aptly) /srv/aptly/gpg2_generate_batch_file.txt
aptly@srv:~# cd aptly@srv:~# cat >/srv/aptly/gpg2_generate_batch_file.txt <<EOF %echo Generating a default key Key-Type: RSA Key-Length: 4096 Name-Real: MyCompanyName Name-Comment: aptly key no passphrase Name-Email: info@mycompanyname.com Expire-Date: 0 %no-protection # Do a commit here, so that we can later print "done" :-) %commit %echo done EOF
Generate the keys
aptly@srv:~# screen aptly@srv:~# gpg --batch --gen-key /srv/aptly/gpg2_generate_batch_file.txt gpg: directory '/srv/aptly/.gnupg' created gpg: keybox '/srv/aptly/.gnupg/pubring.kbx' created gpg: Generating a default key gpg: /srv/aptly/.gnupg/trustdb.gpg: trustdb created gpg: key B0E10740DD603265 marked as ultimately trusted gpg: directory '/srv/aptly/.gnupg/openpgp-revocs.d' created gpg: revocation certificate stored as '/srv/aptly/.gnupg/openpgp-revocs.d/779CA0186E6747FC7BBCE153B0E10740DD603265.rev' gpg: done
We need “screen” (execute screen before the generation of the keys) because you will get GPG generation error. So just execute screen and generate the key in the screen console. For more information check the troubleshooting section. In short, the generation of GPG keys fails because we switch the user with “su”.
aptly@srv:~# ls -altr .gnupg/
total 28
drwxr-xr-x 4 aptly aptly 4096 Oct 27 06:18 ..
-rw------- 1 aptly aptly 32 Oct 27 06:18 pubring.kbx~
srwx------ 1 aptly aptly 0 Oct 27 06:18 S.gpg-agent.ssh
srwx------ 1 aptly aptly 0 Oct 27 06:18 S.gpg-agent.extra
srwx------ 1 aptly aptly 0 Oct 27 06:18 S.gpg-agent.browser
srwx------ 1 aptly aptly 0 Oct 27 06:18 S.gpg-agent
drwx------ 2 aptly aptly 4096 Oct 27 06:18 private-keys-v1.d
-rw------- 1 aptly aptly 1240 Oct 27 06:18 trustdb.gpg
-rw-rw-r-- 1 aptly aptly 1369 Oct 27 06:18 pubring.kbx
drwx------ 2 aptly aptly 4096 Oct 27 06:18 openpgp-revocs.d
drwx------ 4 aptly aptly 4096 Oct 27 06:18 .
aptly@srv:~# gpg --list-secret-keys
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
/srv/aptly/.gnupg/pubring.kbx
-----------------------------
sec rsa4096 2019-10-27 [SCEA]
779CA0186E6747FC7BBCE153B0E10740DD603265
uid [ultimate] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>
aptly@srv:~# gpg --list-keys
/srv/aptly/.gnupg/pubring.kbx
-----------------------------
pub rsa4096 2019-10-27 [SCEA]
779CA0186E6747FC7BBCE153B0E10740DD603265
uid [ultimate] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>
With this key we are going to sign our repository, so we need a way to import the public key in the machines where we are going to use our local repository. Export the public key and put it in the Nginx’s document root directory (of the aptly virtual host) so we can access the key downloading it from our repository URL.
We’ve showed above how to list the public (–list-keys) and the private (–list-secret-keys). Just use the ID of the show command to export the key.
aptly@srv:~# mkdir -p /srv/aptly/.aptly/public aptly@srv:~# gpg --output /srv/aptly/.aptly/public/key.pub --armor --export 779CA0186E6747FC7BBCE153B0E10740DD603265 aptly@srv:~# cat /srv/aptly/.aptly/public/key.pub -----BEGIN PGP PUBLIC KEY BLOCK----- mQINBF21NqQBEAC1ldbeZw1i7Wc6X6e5tYGn7vnj1Pp1UCBkUHCnt6pBBIs1ZukP ...... ...... N/t0tnT7HEaeGy52eg5ttnOh+j7GRlAsJCEaOi1OLoJp2Bv9MDmGZAj0D3FTlSpZ ckiUMZwhbAa6ekOxGIH133E4LKGBcvLBeQR9bohN2NWyARs= =i1Y2 -----END PGP PUBLIC KEY BLOCK-----
STEP 6*) Aptly configuration
This step is also not mandatory, because the default values are OK for most cases. Especially, if you make an additional aptly user and its home directory resides on the desired big (caching) partition because mirroring of repositories may require 200-400Gbytes.
The configuration file uses JSON format and you can check the default values here – https://www.aptly.info/doc/configuration/. Worth mentioning the following default values:
{
"rootDir": "$HOME/.aptly",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": [],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": false,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"FileSystemPublishEndpoints": {
"test": {
"rootDir": "/opt/aptly-publish",
"linkMethod": "copy",
"verifyMethod": "md5"
}
}
}
Aptly supports amazon S3 and OpenStack Swift endpoints, too.
All files (packages) will be under “$HOME/.aptly” and in our user aptly – “/srv/aptly/.aptly/” and the configuration will be “/srv/aptly/.aptly.conf”. Change the default values accordingly.
Using aptly for the first time
Change to the aptly user (if you have followed the STEP (4)) and execute the “config show” command to see the currently loaded configuration:
aptly@srv:~# sudo su - aptly
aptly@srv:~# aptly config show
{
"rootDir": "/srv/aptly/.aptly",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": [],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": false,
"skipLegacyPool": true,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"skipContentsPublishing": false,
"FileSystemPublishEndpoints": {},
"S3PublishEndpoints": {},
"SwiftPublishEndpoints": {}
}
aptly@srv:~#
Aptly cli commands:
aptly@srv:~# aptly
aptly - Debian repository management tool
Commands:
api start API server/issue requests
config manage aptly configuration
db manage aptly's internal database and package pool
graph render graph of relationships
mirror manage mirrors of remote repositories
package operations on packages
publish manage published repositories
repo manage local package repositories
serve HTTP serve published repositories
snapshot manage snapshots of repositories
task manage aptly tasks
version display version
Use "aptly help <command>" for more information about a command.
Options:
-architectures="": list of architectures to consider during (comma-separated), default to all available
-config="": location of configuration file (default locations are /etc/aptly.conf, ~/.aptly.conf)
-db-open-attempts=10: number of attempts to open DB if it's locked by other instance
-dep-follow-all-variants: when processing dependencies, follow a & b if dependency is 'a|b'
-dep-follow-recommends: when processing dependencies, follow Recommends
-dep-follow-source: when processing dependencies, follow from binary to Source packages
-dep-follow-suggests: when processing dependencies, follow Suggests
-dep-verbose-resolve: when processing dependencies, print detailed logs
-gpg-provider="": PGP implementation ("gpg", "gpg1", "gpg2" for external gpg or "internal" for Go internal implementation)
ERROR: unable to parse command
Import the public key
To use your local repository you must import the public key with apt in the machine, where you want to use the repository. Donwload the key and import it with:
myuser@my-server-pc:~$ wget http://apt.example.com/key.pub
--2019-10-27 06:48:32-- http://apt.example.com/key.pub
Resolving apt.example.com (apt.example.com)... 127.0.0.1
Connecting to apt.example.com (apt.example.com)|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1688 (1.6K) [application/octet-stream]
Saving to: ‘key.pub’
key.pub 100%[========================================================================================>] 1.65K --.-KB/s in 0s
2019-10-27 06:48:32 (253 MB/s) - ‘key.pub’ saved [1688/1688]
myuser@my-server-pc:~$ sudo apt-key add key.pub
OK
myuser@my-server-pc:~$ apt-key list
/etc/apt/trusted.gpg
--------------------
pub rsa4096 2018-03-15 [SC] [expires: 2020-03-14]
26DA 9D86 3030 2E0B 86A7 A2CB ED75 B5A4 483D A07C
uid [ unknown] Andrey Smirnov <me@smira.ru>
sub rsa4096 2018-03-15 [E] [expires: 2020-03-14]
pub rsa4096 2019-10-27 [SCEA]
779C A018 6E67 47FC 7BBC E153 B0E1 0740 DD60 3265
uid [ unknown] MyCompanyName (aptly key no passphrase) <info@mycompanyname.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-archive.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
790B C727 7767 219C 42C8 6F93 3B4F E6AC C0B2 1F32
uid [ unknown] Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2012-cdimage.gpg
------------------------------------------------------
pub rsa4096 2012-05-11 [SC]
8439 38DF 228D 22F7 B374 2BC0 D94A A3F0 EFE2 1092
uid [ unknown] Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>
/etc/apt/trusted.gpg.d/ubuntu-keyring-2018-archive.gpg
------------------------------------------------------
pub rsa4096 2018-09-17 [SC]
F6EC B376 2474 EDA9 D21B 7022 8719 20D1 991B C93C
uid [ unknown] Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>
Troubleshooting
Getting this error when generating the key you missed to execute “screen” before the command.
aptly@srv:~$ gpg --batch --gen-key /srv/aptly/gpg2_generate_batch_file.txt gpg: directory '/srv/aptly/.gnupg' created gpg: keybox '/srv/aptly/.gnupg/pubring.kbx' created gpg: Generating a default key gpg: agent_genkey failed: Permission denied gpg: key generation failed: Permission denied gpg: done
This is probably because we switch the user with su and the tty is still to our old user:
aptly@srv:~# ls -al $(tty) crw--w---- 1 myuser tty 136, 0 Oct 27 07:02 /dev/pts/0
It’s “aptly” user, but the owner is the “myuser”.
Just execute screen to enter a new shell and generate the keys, then you can leave with exit.
We’ve included the screen package in the install line in the first step.
Hi,
I followed this tutorial for aptly can you make a tutorial on how to use s3 for storage.Next thing is how to upload files to aptly repo automatically form a pipeline. A script will do.