Tag: mirror

Copying partition table from one disk to another with older sfdisk under CentOS 7

Older version of sfdisk may still be used for msdos partition tables.
To copy the partition table from one disk to another using sfdisk a temporary file should be used to store the data for the partition table.

Here is an example of how to copy the msdos partition table from disk sda to disk sdb! Two simple commands

  1. Dump the source (sda) partition table to a temporary file.
  2. Redirect the standard input of the sfdisk utility with the above temporary file.

A copying partition table is really useful when recovering from a drive failure in a Linux software raid. Sometimes it is difficult or just easier to create the exact layout as the source mirror disk!

mdadm --add /dev/md1 /dev/sdb2
mdadm: /dev/sdb2 not large enough to join array

Errors such as the above are easily resolved with just two commands. The new versions of disk programs align the partitions, which may be a problem for a software RAID to join in a partition.

STEP 1) Dump the source partition table.

The source partition table is from /dev/sda:

[root@srv ~]# sfdisk -d /dev/sda > part_table_sda
sfdisk: Warning: extended partition does not start at a cylinder boundary.
DOS and Linux will interpret the contents differently.

There is a warning about not aligned partition, which may cause problems when creating from scratch, so copying the partition table is the best option in such cases.

Here is what the temporary file part_table_sda with the partition table information contains:

[root@srv ~]# cat part_table_sda 
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size= 67045376, Id=fd, bootable
/dev/sda2 : start= 67110912, size=  1048576, Id=fd
/dev/sda3 : start= 68159488, size= 62883840, Id=fd
/dev/sda4 : start=131043328, size=845729792, Id= f
/dev/sda5 : start=131076096, size=845434880, Id=fd

Keep on reading!

Mirror a PPA repositories using aptly – PHP (ppa:ondrej/php)

This is a simple example of how to mirror a PPA repository to a local server. The Ubuntu PPA to mirror is ppa:ondrej/php, which offers the user different PHP version generally not available in the Ubuntu installation. Of course, the user should be very careful about adding PPA repositories, because they are exactly what the abbreviation stands for Personal Package Archives.

If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps

What we are going to do – this is what you need to have a mirror of an external application repository:

  1. Install aptly in Ubuntu 18 LTS
  2. Create a mirror in aptly
  3. Create a snapshot of the mirror created before
  4. Publish the snapshot to be used in other servers.

and at the last step there is an example how to use the mirror in your local machines.

STEP 1) Install aptly in Ubuntu 18.04 LTS.

As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.
Change the user to aptly, because under this user the mirror process will happen.

root@srv ~ # su - aptly
aptly@srv:~$

STEP 2) Create a mirror in aptly.

Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):

aptly@srv:~$ gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver pool.sks-keyservers.net --recv-keys 4F4EA0AAE5267A6C
gpg: requesting key E5267A6C from hkp server pool.sks-keyservers.net
gpg: key E5267A6C: public key "Launchpad PPA for Ond\xc5\x99ej Sur�" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)

Here we’ve used the method to obtain the key from a GPG KEY server, but the key can be downloaded directrly from the original repository as suggested in the error message below.
If you are not sure where to download the key you could always just try to create the mirror ( in fact, this is in STEP 3) ) and get the error for missing key and how to obtain the key:
Keep on reading!

Mirror the official Ubuntu repositories using aptly

This article is to show mainly how to work with aptly by mirroring an official Ubuntu mirror. If you want to know how to install and a brief description of what is aptly you may want to read our previous article – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps

What we are going to do – this is what you need to have a mirror of an external application repository:

  1. Install aptly in Ubuntu 18 LTS
  2. Create a mirror in aptly
  3. Create a snapshot of the mirror created before
  4. Publish the snapshot to be used in other servers.

and at the last step there is an example how to use the mirror in your local machines.

STEP 1) Install aptly in Ubuntu 18.04 LTS.

As mentioned already you may follow our article on the subject – Install aptly under Ubuntu 18 LTS with Nginx serving the packages and the first steps. The following steps are based on this installation!
The aptly home directory is in “/srv/aptly”. We use the “aptly” user and change to it to manipulate the aptly installation.

STEP 2) Create a mirror in aptly.

Prepare the keys (aptly needs to have the Ubuntu keys in its trustedkeys keyring):

aptly@srv:~$ gpg --no-default-keyring --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg --export | gpg --no-default-keyring --keyring trustedkeys.gpg --import
gpg: key 3B4FE6ACC0B21F32: 3 signatures not checked due to missing keys
gpg: key 3B4FE6ACC0B21F32: public key "Ubuntu Archive Automatic Signing Key (2012) <ftpmaster@ubuntu.com>" imported
gpg: key D94AA3F0EFE21092: 3 signatures not checked due to missing keys
gpg: key D94AA3F0EFE21092: public key "Ubuntu CD Image Automatic Signing Key (2012) <cdimage@ubuntu.com>" imported
gpg: key 871920D1991BC93C: 1 signature not checked due to a missing key
gpg: key 871920D1991BC93C: public key "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>" imported
gpg: Total number processed: 3
gpg:               imported: 3
gpg: public key of ultimately trusted key 212A3D20E4D3351D not found
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u

Probably you would like to have “main” and “universe” for the three bionic, bionic updates and bionic security.
First, main and universe for bionic repository. main is ~16G and universe is ~136, these numbers will vary in future when more packages are added.
Two commands are need for the aptly mirror:

  1. create – create the mirror.
  2. update – download the repository contents locally.

Keep on reading!

Pages: Page 1, Page 2

SSD cache device to a software RAID5 using LVM2

Continuing our series LVM2 plus cache device:

  1. single hard disk with a SSD device SSD cache device to a hard disk drive using LVM, which uses SSD drive as a cache device to a single hard drive.
  2. Mirror LVM2 device with a SSD device for cache – SSD cache device to a software raid using LVM2 – software mirror across two devices with an additional SSD cache device over the mirror.

And now we show you how to do software RAID5 with SSD cache nvme using LVM2.

The goal:
Caching RAID5 consisting of three 8T hard drives with a single 1T NVME SSD drive. Caching reads, i.e. the write-through is enabled ().
Our setup:

  • 1 NVME SSD disk Samsung 1T. It will be used for writethrough cache device (you may use writeback, too, you do not care for the data if the cache device fails)!
  • 3 Hard disk drive 8T grouped in RAID5 for redundancy.

Keep on reading!

SSD cache device to a software raid using LVM2

Inspired by our article – SSD cache device to a hard disk drive using LVM, which uses SSD driver as a cache device to a single hard drive, we decided to make a new article, but this time using two hard drives in raid setup (in our case RAID1 for redundancy) and a single NVME SSD drive.
The goal:
Caching RAID1 consisting of two 8T hard drive with a single 1T NVME SSD drive. Caching reads and writes, i.e. the write-back is enabled.
Our setup:

  • 1 NVME SSD disk Samsung 1T. It will be used for writeback cache device (you may use writethrough, too, to maintain the redundancy of the whole storage)!
  • 2 Hard disk drive 8T grouped in RAID1 for redundancy.

STEP 1) Install lvm2 and enable the lvm2 service

Only this step is different on different Linux distributions. We included three of them:
Ubuntu 16+:

sudo apt update && apt upgrade -y
sudo apt install lvm2 -y
systemctl enable lvm2-lvmetad
systemctl start lvm2-lvmetad

CentOS 7:

yum update
yum install -y lvm2
systemctl enable lvm2-lvmetad
systemctl start lvm2-lvmetad

Gentoo:

emerge --sync
emerge -v sys-fs/lvm2
/etc/init.d/lvm start
rc-update add default lvm

Keep on reading!

aptly mirror: ERROR: unable to update: no candidates for debian-installer/binary-amd64/Packages found

Always check the source what supports when trying to mirror! We have lost some time before discovering that our source repository does not support udeb and source packages! If you create a mirror with “-with-sources=true -with-udebs=true” the update process will require files, which may not exists in the source repository if it does not offer udeb or source files and you’ll end up with broken mirror and error for missing file!

Downloading & parsing package files...
Downloading http://aptly.example.com/ubuntu/dists/xenial-myrepos/main/binary-amd64/Packages.bz2...
ERROR: unable to update: no candidates for http://aptly-master.example.com/ubuntu/dists/xenial-myrepo/main/debian-installer/binary-amd64/Packages found

If you get error for “debian-installer/binary-amd64/Packages” not found, check the source repository if it offers udeb and/or source packages – probably not, so drop your mirror and recreate it including one or the two options

-with-sources=false -with-udebs=false

Keep on reading!

aptly mirror – gpgv: Can’t check signature: public key not found

If you want to mirror repositories from your current aptly server to a new server you must import the GPG key from your old server because you are going to encounter the following error:

gpgv: Signature made Fri 22 Apr 2019 17:35:04 AM UTC using DSA key ID FDC7A25E
gpgv: Can't check signature: public key not found

Looks like some keys are missing in your trusted keyring, you may consider importing them from keyserver:

gpg --no-default-keyring --keyring trustedkeys.gpg --keyserver pool.sks-keyservers.net --recv-keys 181482CCFDC7A25E

Sometimes keys are stored in repository root in file named Release.key, to import such key:

wget -O - https://some.repo/repository/Release.key | gpg --no-default-keyring --keyring trustedkeys.gpg --import

ERROR: unable to fetch mirror: verification of detached signature failed: exit status 2

And the mirror command fails. The problem is

you must import the GPG key from your old server in trustedkeys.gpg (even if you have already imported it in the new server with apt-key!!!)

Here is how to list, export and import it (we are going to import it in default and trustedkeys.gpg, because it is more convenient, but it is not mandatory to be in the default).
Keep on reading!