Some context

As the title indicates, this post focuses on creating backups with btrbk, "a backup tool for btrfs subvolumes". btrfs is "a modern copy on write (CoW) filesystem". One of btrfs' main features is the ability to take snapshots which have some neat features:

Btrfs is a copy-on-write filing system designed to allow point-in-time snapshots of the filing system (or individual subvolumes, a term we used earlier in the guide). It’s genius in that you can instantly snapshot your data into a separate folder, and transfer that data to another btrfs filing system at a pure block level (IE: very efficiently). (Source)

I came across btrfs thanks to Gurucomputing's post. I followed the whole series on Docker at the beginning of my homelab journey and it was an amazing starting point. Many decisions I've made such as using Fedora and btrfs can be accredited to it. :)

It's worth pointing out that this post isn't a thorough guide on how to setup btrbk. Rather, I'll focus on how I set it up to send backups through SSH, the issues I came across because of my setup, and how I solved them. For more resources on btrbk, I recommend its documentation or Gurucomputing's guide.

So, what was the objective and what were the problems?

A quick rundown of my homelab: the most interesting parts lie on a HP USFF desktop that runs Proxmox as the hypervisor. It runs two main VMs: one for self-hosted applications and one for backups. Proxmox can create backups of the VMs and even snapshots while running, both of which can be easily restored. It's a great feature but considering the small SSD on which I installed it, I create the backup VM, attached an external drive, and moved the backups to it.

Why not simply add the drive to Proxmox though? Well, the self-hosting VM is on the main drive and I quickly started running out of space when I started copying my files to my Nextcloud instance. So I added an external drive to that VM and all was good. Except for the fact that this data wasn't being backed up by Proxmox. Maybe I missed some option to include this external drive, but since I had already read Gurucomputing's guide I knew that btrbk would work.

And so, I created a new VM, attached it another external drive, and set up both btrbk to backup the self-hosting VM and samba to backup the VM itself with Proxmox.

First btrbk setup

I only had one host to backup at the time but still decided to play with the fileserver-initiated backups, where the system storing the backups connects through SSH to the hosts you want to backup, creates the necessary snapshots, and transfers them to itself. You can also configure it the other way around and have the host to send its backups to the fileserver.

The only caveat is that btrfs commands usually require root, meaning either root access or passwordless sudo for unattended backups. PermitRootLogin yes in my sshd_config? Not a chance!¹

Fortunately the following section explains how to use a dedicated user. To set passwordless sudo for btrfs commands, one can add the necessary rules to the sudoers file or check this section for other options. For the sudoers file configuration, see for example #472.

I went the sudoers way: it worked without a hitch. So what exactly was the problem?

Moving to FreeIPA

At some point I decided test FreeIPA, interested by the idea of having Single Sign On to all my hosts. Since users are managed by FreeIPA, instead of using the local user I created to passwordless sudo I tested creating one on FreeIPA called bkuser. I added an HBAC rule to allow it to connect to both VMs through SSH, created an SSH key in the backup VM, and uploaded the public key to the IPA server. Public keys on FreeIPA are shared to hosts through SSSD which eliminates the need to manually copy them to the corresponding authorized_keys, meaning that bkuser could login to the self-hosting VM through the backup VM. Except it couldn't.

Sudo rules on FreeIPA

It wasn't a big deal: it was a typical case of not reading the documentation and finding out the hunch did not work out.

As mentioned, the recommended approach is to only give bkuser access to the btrfs subcommands it needs, like send and receive. The corresponding section on the sudoers file should look like:

Cmnd_Alias BTRFS_SEND = /usr/bin/btrfs send *
Cmnd_Alias BTRFS_RECEIVE = /usr/bin/btrfs receive *

bkuser ALL= NOPASSWD: BTRFS_SEND BTRFS_RECEIVE

The equivalent in FreeIPA are the sudo rules. While I thought I had to add the entire command just like in sudoers (e.g. /usr/sbin/btrfs send *), it turns out only the executable path is expected. This became clear when even the dry-runs were failing during the discovery stage with readlink. This means that specific subcommands cannot be specified, but adding the correct executable worked, at least to trigger the next problem.

SELinux silent denials

While bkuser could now usereadlink with passwordless sudo, I was getting a "permission denied" error. I logged in to the backup VM as bkuser and connected to the Docker host through SSH: sure enough I could not cd into the directory I wanted to backup. I checked that the permissions were set correctly and that my regular user was able to access that directory. I suspected an SELinux issue, but no denials were appearing on Cockpit's SELinux tab.

I looked up ways to troubleshoot SELinux and I found this documentation page, which explains that there are silent denials configured to stop common operations that "probe for more access than required to perform their tasks" from flooding the logs. These are called the dontaudit rules and be configured with:

# to disable
semodule -DB
# to enable
semodule -B

This either enables or disables the dontaudit rules and rebuilds the policy.

Running semodule -DB caused a bunch of AVC denials I didn't see on previous logs, and one stuck out:

SELinux is preventing bash from search access on the directory /srv/containers.

/srv/containers is where I store all my Docker-related files, from compose files to the bind mounts, which is exactly the directory bkuser had to access. But how to fix this issue? I checked the logs and found another clue:

SELinux is preventing btrfs from using the sys_admin capability.

Ah, so maybe my staff_u bkuser does not have enough permissions to use btrfs after all. SELinux has a variety of users, which is one of the parameters it uses to determine whether to allow any given action. You can check your user type on an SELinux-enabled system with id -Z. With a normal desktop system you will most likely see unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023. unconfined_u users are basically unrestricted, which is not great from a security perspective but does provide a better user experience as you're not triggering AVC denials left and right.

Looking to be consistent with my desire to restrict bkuser, I looked for other SELinux users and across a nice table on the Gentoo wiki. Considering that bkuser does need some administrative access, staff_u seemed like a good option.

The problem is that btrfs, and by extension btrbk, requires actual administrator access such as the one given to sysadm_u. This difference is clearer on the RHEL guide: the staff_r role is described in section 3.3: Confined non-administrator roles in SELinux while the sysadm_r role can be found in 3.4: Confined administrator roles in SELinux.

So I changed the SELinux user map to set bkuser as sysadm_u, probably had to clear the SSSD cache, and could now complete the backup!

To be honest, I'm not sure if this is the correct solution as giving more permissions is usually not the real answer. But as sudo operations are restricted by IPA rules it shouldn't be too much of an issue.²

Conclusion

All this troubleshooting forced me to better understand the features of FreeIPA, and how it interacts with both sudo and SELinux. Even so, I ran into a new problem when enrolling my laptop as a FreeIPA host. And you guessed it, it was SELinux again. But that will have to wait for a future post. Thanks for reading. :)