Sign in Subscribe
Hetzner

Hetzner SX62 dedicated server with Software-RAID-5 and encrypted LVM on Debian 10

Martin Seener

• 6 min read
Hetzner SX62 dedicated server with Software-RAID-5 and encrypted LVM on Debian 10

A little over 2 years ago, we introduced a new Backup server for our PostgreSQL data. Check out my post on that here.

Now 2 years later, the disks got too small for our growing size of our PostgreSQL data which urged us to move on to a larger server.

We found one in Hetzners SX-line Server fleet - the SX62 with plenty of space. In fact with RAID-5 we will get 10 times the disk size for our backup storage.

In this guide we will set this guy up with a software-based RAID-5 using mdadm as well as [LVM] on an encrypted LUKS partition.

Basically the same setup we used before but this time with way larger disks (GPT vs. MBR) and a software-raid and not a hardware-raid plus a newer version of Debian.

This guide is written way shorter and mostly covers the steps without much information beside of that. Read my last post to get more insights.

Let's go!

Disclaimer

I don't take responsibilities for any damage that my tutorial may cause to your system. I've tested it twice on my system and it worked well all the time but the commands may be different on your system. So please keep that in mind while following my remarks but the commands should be somewhat identical on Debian derivates or older and newer versions of them.

Install the base system with initial RAID and LVM config

This setup here is a lot easier than for my old post, so boot up your new SX62 to Hetzners Rescue System.

Now let us directly invoke installimage and configure the SW RAID-5 and LVM and let it install the minimal Debian 10 installation.

installimage -a -n myserver.com \
-b grub \
-r yes \
-l 5 \
-d sda,sdb,sdc,sdd \
-i root/.oldroot/nfs/images/Debian-105-buster-64-minimal.tar.gz \
-p /boot:ext4:512M,lvm:vg0:all \
-v vg0:swap:swap:swap:8G,vg0:root:/:ext4:16G,vg0:var-log:/var/log:ext4:8G,vg0:tmp:/tmp:ext4:8G,vg0:var-lib-postgresql:/var/lib/postgresql:ext4:500G,vg0:var-lib-barman:/var/lib/barman:ext4:10T \
-f yes -s de -K /root/.ssh/robot_user_keys

What we defined here is:

  • -a makes it non-interactive
  • -n myserver.com defines the servers hostname
  • -b grub tells installimage to install grub as our preferred bootloader
  • -r yes tells installimage to setup a RAID
  • -l 5 defines the RAID's level. Here it's a RAID-5, so 30/40TB usable
  • -d sda.. defines the disks we want to use for the RAID
  • -i root/ defines the source image for the installation
  • -p defines all physical partitions we want to see on the SW-RAID
  • -v then defines all logical volumes we want to see on the LVM
  • -f tells installimage to format the disks
  • -s defines the preferred server language
  • -K tells installimage to install the following SSH robot keys for root

Encrypt everything and make it headless

Now that the system has been initially installed, reboot to your shiny new installation.

We will now install dropbear so we can headlessly decrypt the server over SSH at boot time.

apt update && apt-get -y upgrade
apt -y install busybox dropbear dropbear-initramfs cryptsetup-initramfs

You will see the following warning which is ok. We will fix it in a minute.

dropbear: WARNING: Invalid authorized_keys file, remote unlocking of cryptroot via SSH won't work!

Now reboot to the Rescue System again for the next steps.

At first we will make a full backup of our new installed system for a moment.

mkdir /rootbackup
mount /dev/vg0/root /mnt
mount /dev/vg0/var-log /mnt/var/log
mount /dev/vg0/tmp /mnt/tmp
mount /dev/vg0/var-lib-postgresql /mnt/var/lib/postgresql
mount /dev/vg0/var-lib/barman /mnt/var/lib/barman
rsync -av /mnt/ /rootbackup/

This makes a full backup now, because we need to remove the LVM again. Rsync shouldn't take too long but you can see it's output.

Now unmount everything and disable the LVM.

umount /mnt/var/lib/postgresql && umount /mnt/var/lib/barman && umount /mnt/tmp && umount /mnt/var/log  && umount /mnt
vgchange -a n vg0 # Otherwise recreation of LUKS on md1 will fail later

Since we now have two RAID-5's - md0 for boot and md1 for current LVM, we can just overwrite md1 with the LUKS device.

cryptsetup --cipher aes-xts-plain64 --key-size 512 --hash sha256 --iter-time 6000 luksFormat /dev/md1

Choose a long and safe password with pwgen 64 1 for the encryption.

When you're done, mount the new LUKS device and let us recreate the LVM and put the system back in place.

cryptsetup luksOpen /dev/md1 cryptroot # Open LUKS device
pvcreate /dev/mapper/cryptroot # Create physical LVM volume on LUKS
vgcreate vg0 /dev/mapper/cryptroot # Create our volume group again
# Create our logical volumes now
lvcreate -L 8G -n swap vg0
lvcreate -L 16G -n root vg0
lvcreate -L 8G -n var-log vg0
lvcreate -L 8G -n tmp vg0
lvcreate -L 500G -n var-lib-postgresql vg0
lvcreate -L 10T -n var-lib-barman vg0
# Format the volumes and create swap again
mkfs.ext4 /dev/vg0/root
mkfs.ext4 /dev/vg0/var-log
mkfs.ext4 /dev/vg0/tmp
mkfs.ext4 /dev/vg0/var-lib-postgresql
mkfs.ext4 /dev/vg0/var-lib-barman
mkswap /dev/vg0/swap

# Now lets mount everything and put back the system

mount /dev/vg0/root /mnt
mkdir -p /mnt/var/log
mount /dev/vg0/var-log /mnt/var/log
mkdir /mnt/tmp
mount /dev/vg0/tmp /mnt/tmp
mkdir -p /mnt/var/lib/postgresql
mount /dev/vg0/var-lib-postgresql /mnt/var/lib/postgresql
mkdir -p /mnt/var/lib/barman
mount /dev/vg0/var-lib-barman /mnt/var/lib/barman
rsync -av /rootbackup/ /mnt/

In the next step, let's add the boot partition to the party as well as the current devices, proc and sys to our chroot environment

mount /dev/md0 /mnt/boot
mount --bind /dev /mnt/dev
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
chroot /mnt

Configure LUKS for boot-time decryption

Now let's open /etc/crypttab and add the LUKS partition, so the system knows what to mount.

# <target name>	<source device>		<key file>	<options>
cryptroot /dev/md1 none luks,initramfs

Create /etc/rc.local and add some content. This will ensure proper network configuration after unlocking and booting the main system.

# Content of rc.local
/sbin/ifdown --force eth0
/sbin/ifup --force eth0

The last step is to add your ssh public keys to dropbear, so you can authenticate with it at boot time. For Debian 9 there was no ed25519 key support. For Debian 10 I did a short dig into the changelog but haven't found anything, so my best guess is, that there is still only RSA key support.

vim /etc/dropbear-initramfs/authorized_keys
# Add your SSH keys like you would in ~/.ssh/authorized_keys

Now let's update the initramfs with the dropbear config and keys and install grub again on all 4 disks, so we could boot from any single one.

For me I had to wait for the SW-RAID recovery to finish before I could install grub. So you may need to check for cat /proc/mdstat if it's finished already.

update-initramfs -u -k all
grub-install /dev/sda
grub-install /dev/sdb
grub-install /dev/sdc
grub-install /dev/sdd

Cleanup and reboot

We have done everything and we can cleanup our chroot environment and do the first boot into our new encrypted SX62.

exit # Exit Chroot environment
umount /mnt/var/lib/barman/
umount /mnt/var/lib/postgresql
umount /mnt/var/log
umount /mnt/tmp
umount /mnt/boot
umount /mnt/dev
umount /mnt/sys
umount /mnt/proc
umount /mnt
sync
shutdown -r now

After reboot, login as root and run crypttab-unlock to mount the LUKS device and continue to boot.

Conclusion

I hope this guide helps some people out there to further secure their servers. Normally a LUKS encrypted server that runs 24/7 doesn't really help against usual network attacks but if you ever need to have the data save when the server is down or you need to instruct Hetzner to replace a failing disk, you can be sure that your data on the drives are safe, even without a manual erase step. This is why I recommend everyone to go this extra mile.

Also check out my last post if you want to decrease attack surface and disable USB on this server.

Sign up for more like this.

Enter your email
Subscribe