Fedora Linux 41 has been released!

In this guide, I will show you how to install Fedora 41 with full disk encryption, snapshot, and rollback Support.

So, if you mess up the system settings, install something that does not work, or run into problems after updating, you won't have to worry about breaking the system and spending hours trying to fix it. Simply restore the snapshot before the problematic change, and you are good to go.

Note: If you want to install Fedora 41 with snapshot and rollback support but do not want LUKS Full Disk Encryption, please see my other guide 'How to Install Fedora 41 with Snapshot and Rollback Support'.

Note: If you have previously installed Fedora 40 following my earlier tutorial and wish to upgrade to Fedora 41, please proceed directly to the upgrade section located at the end of this article to upgrade from Fedora 40 to Fedora 41. No additional steps are required.

In my last guide on installing Fedora 40 with full disk encryption, I chose to use the command line instead of the default Anaconda GUI installer. This was because the Anaconda installer doesn't allow for the encryption of the /boot directory, which is an important part of the installation process.

However, I have found a way to encrypt the /boot directory while still using the Anaconda installer.

So let's get started.

Table of Contents

1. Partitions and Subvolumes Layout

For this guide, I will use a hard disk with a capacity of 128 GiB.

I will create a 600 MiB EFI system partition and use the remaining disk space to create a partition for the Linux file system.

Fedora also creates a SwapOnZRAM during boot, so no separate swap partition is required.

This is how the disk partition looks:

NAME         SIZE  FSTYPE         PARTTYPENAME      MOUNTPOINT
/dev/vda 128G
/dev/vda1 600M vfat EFI System /boot/efi
/dev/vda2 127.4G btrfs (LUKS) Linux filesystem /

First, I will encrypt the Linux partition (/dev/vda2) with LUKS. Next, on the encrypted LUKS partition, I will create a btrfs file system. Then, on the btrfs file system, I'll create the following subvolumes to keep them out of root file system snapshots.

NAME             MOUNTPOINT                 TYPE
[main] / mainvolume
home /home subvolume
opt /opt subvolume
cache /var/cache subvolume
crash /var/crash subvolume
AccountsService /var/lib/AccountsService subvolume
gdm /var/lib/gdm subvolume
images /var/lib/libvirt/images subvolume
log /var/log subvolume
spool /var/spool subvolume
tmp /var/tmp subvolume
www /var/www subvolume
.mozilla /home/$USER/.mozilla subvolume
.snapshots /.snapshots subvolume
.snapshots /home/.snapshots subvolume

Info: Subvolumes, unlike standard partitions or LVM logical volumes, do not have a size; rather, they behave as directories with shared space. However, the resemblance between subvolumes and directories ends here. Each subvolume, much like a file system, has its own file tree, POSIX namespace, and inode pool. This means that hard links between subvolumes are not possible. From this perspective, a subvolume resembles a separate file system. Subvolumes are not block devices either.

The directories for which subvolumes are created and the reasons for doing so are listed below. For more information, visit this website.

/home

Contains user data. It will be created to keep user data separate while also protecting against data loss during system root rollbacks.

/opt

Contains applications installed by a third party. It will be created to keep these applications from being uninstalled during rollbacks.

/var/cache, /var/crash, /var/tmp

Directories containing temporary files and caches. They will be created so that the system root subvolume does not contain temporary files and caches. Fedora mounts /tmp with tmpfs on boot, so a separate subvolume is not required.

/var/lib/AccountsService, /var/lib/gdm

Contains login user and Gnome display information. These directories must be writeable at all times. When you try to boot a snapshot from the GRUB menu to rollback system root, you are booting a read-only snapshot, and because it cannot be written to, the system hangs just before the Gnome login screen appears.

If you are using a desktop environment other than Gnome, you will need to replace '/var/lib/gdm' with one specific to your desktop environment.

For KDE, it is '/var/lib/sddm'.

For XFCE, it is '/var/lib/lightdm' and '/var/lib/lightdm-data'.

To find out about other desktop environments, refer to this page.

/var/lib/libvirt/images

The default location for libvirt-managed virtual machine images. It will be created so that virtual machine images are not replaced with older versions during a rollback.

/var/log

Contains log files. It will be created to prevent the loss of log data during rollbacks.

/var/spool

Contains data that is awaiting some kind of later processing, such as mail, mail queues, printing, printer spool, and so on. It will be created to prevent the loss of mail, printing, and spool data following a rollback.

/var/www

Web server directory. It will be created to keep hosted web server data separate and prevent data loss during system root rollbacks.

/home/$USER/.mozilla

Fedora Workstation ships with the Firefox web browser by default. If you take a snapshot of your home directory and then undo the changes, you do not want to lose any new bookmarks, passwords, or other user data.

So I will create a subvolume for the '.mozilla' directory, which is where Firefox stores all of its data. This separates it from the home subvolume, avoiding data loss during undo operations on the home subvolume.

If you use other web browsers, you should consider creating separate subvolumes for them.

Chrome Browser: /home/$USER/.config/google-chrome
Brave Browser : /home/$USER/.config/BraveSoftware

Depending on your usage, you may also want to consider creating subvolumes in your home directory for the following directories:

GnuPG: /home/$USER/.gnupg
SSH: /home/$USER/.ssh
Thunderbird Mail: /home/$USER/.thunderbird
/.snapshots, /home/.snapshots

Contains snapshots of the root (/) and home (/home) directories.

2. Install Fedora Workstation 41

Boot your system using the Fedora 41 Workstation Live installer in UEFI mode.

In the 'Welcome to Fedora' window, select the 'Not Now' option to return to the Gnome desktop. Next, open the Terminal window to start preparing your system for Fedora 41 installation.

Switch to super user mode.

$ sudo -i

By default, the Anaconda installer does not let you encrypt the /boot directory. So, you must first enable this option before proceeding with the Anaconda installer.

# sed -i.bkp 's/encryption_support = False/encryption_support = True/' \
    /usr/lib64/python3.*/site-packages/pyanaconda/modules/storage/bootloader/base.py

# sed -i.bkp 's/self.raw_device.format._header_size/crypto.LUKS_METADATA_SIZE/g' \
    /usr/lib/python3.*/site-packages/blivet/devices/luks.py

Now that /boot encryption support is enabled, launch the Anaconda GUI installer. Keep the terminal window open for now. We will come back to it once the installation is complete.

On the welcome screen, select your Language, Keyboard, and configure your Time & Date. After that, select the 'Installation Destination' option.

You should now be on the INSTALLATION DESTINATION screen. To proceed, pick the Advanced Custom (Blivet-GUI) radio button and then hit the Done button.

On the BLIVET GUI PARTITIONING screen, create the partitions, file systems, and btrfs subvolumes necessary to install the Fedora Workstation 41.

First, you need to create and mount the EFI partition. Select the free space and click the + sign to create a partition.

Set the partition Size to 600 MiB, the Filesystem to EFI System Partition, and the Mountpoint to /boot/efi.

Then, you need to create a btrfs volume where you can create all the subvolumes needed to install Fedora Workstation 41.

Select the free space again and click on the + sign to create a Btrfs volume. Set the Filesystem to btrfs and the Mountpoint to /. I used the entire remaining space. However, you can specify the size you want for the btrfs volume.

Enable encryption by checking the Encrypt option. Select luks2 as the encryption type. GRUB currently supports devices encrypted with LUKS1 or LUKS2, but only with the PBKDF2 key derivation function. LUKS2 uses Argon2id by default. We will change it to PBKDF2 once the installation is complete.

Set the passphrase. Set a stronger passphrase with higher entropy (password entropy > 60 bits is suggested).

Next, you must create a home subvolume. (1) Select the Btrfs Volume from the left panel, and (2) click on the + sign on the right panel.

Create a home subvolume. Enter the Name as home and Mountpoint as /home. Click OK to finish.

I will create the remaining subvolumes when the installation is finished. This is because the Anaconda installer does not allow you to create subvolumes with slashes (/) in their names, such as var/log.

For now, click Done to finish creating subvolumes.

On the SUMMARY OF CHANGES screen, double-check that everything is properly defined. To finalize the changes, click the Accept Changes button.

You will be returned to the INSTALLATION SUMMARY screen. Press the Begin Installation button to start the installation process. The installation process will start. Just sit back and relax.

Once the installation is complete, close the Anaconda installer by clicking the Finish Installation button. Please do not reboot just yet. Now return to the terminal window.

As I previously stated, GRUB currently supports devices encrypted using the PBKDF2 key derivation function. As LUKS2 defaults to Argon2id, we must change it to PBKDF2. Your current installation is saved at /mnt/sysroot.

First, find your LUKS device and save it in a variable. For me, it is /dev/vda2. Yours will be different.

# LUKS_DEVICE="$(cryptsetup status \
    $(grub2-probe --target=device /mnt/sysroot) \
    | grep 'device:' | awk '{print $2}')" \
    ; echo $LUKS_DEVICE
/dev/vda2

Check the PBKDF.

# cryptsetup luksDump $LUKS_DEVICE | grep -A7 '0: luks2'
  0: luks2
	Key:        512 bits
	Priority:   normal
	Cipher:     aes-xts-plain64
	Cipher key: 512 bits
	PBKDF:      argon2id
	Time cost:  4
	Memory:     645584

As you can see, it is argon2id. We need to change it to pbkdf2.

To change it, run the following command. Also, for pbkdf2, the minimum iteration count is 1000 and the maximum is 4294967295. The more powerful your CPU, the higher the number of iterations. And, the higher the iteration count, the longer it takes to unlock the LUKS in the GRUB. So I have set the iteration count to 500000. You can set the iteration count to whatever you want. Lowering the iterations too much reduces security.

# cryptsetup luksChangeKey ${LUKS_DEVICE} \
    --pbkdf pbkdf2 \
    --pbkdf-force-iterations 500000

You will be asked the passphrase three times. The first one will authenticate, and the next two will set the new passphrase. Do not change the passphrase; keep it the same as before.

Check the PBKDF again.

# cryptsetup luksDump $LUKS_DEVICE | grep -A7 '0: luks2'
  0: luks2
	Key:        512 bits
	Priority:   normal
	Cipher:     aes-xts-plain64
	Cipher key: 512 bits
	PBKDF:      pbkdf2
	Hash:       sha256
	Iterations: 500000

As you can see, the PBKDF has been changed to pbkdf2, and the iteration count is set to 500000.

Finally, you must enable Crypto Disk and cryptomount. Execute the following commands sequentially.

# LUKS_UUID="$(cryptsetup luksUUID ${LUKS_DEVICE})" ; echo $LUKS_UUID

# sed -i.bkp1 "1i cryptomount -u ${LUKS_UUID//-/}" \
    /mnt/sysroot/boot/efi/EFI/fedora/grub.cfg

# echo 'GRUB_ENABLE_CRYPTODISK=y' >> /mnt/sysroot/etc/default/grub

The configuration is now complete. Reboot the system.

# reboot

You will be prompted for the passphrase twice. The first time is before loading GRUB, and the second time is before mounting the root file system. Later in this guide, I will enable the single passphrase prompt.

Depending on your computer's CPU, it may take up to 15 seconds for the GRUB to display the menu. So please be patient.

When you input your passphrase in the first LUKS prompt, no '*' cursor indication appears. So you should assume you have entered your passphrase correctly. Simply enter your LUKS passphrase blindly and hit the [Enter] key.

The last phase of the installation process will start. Click the Start Setup button to complete the remaining customization steps, such as setting a new login, password, etc.

You will then be logged into the Fedora Workstation 41 with the all-new Gnome 47 desktop interface.

3. Post-Installation Configurations

When booting your system, you may have noticed the following error message.

This is because you did not create a separate ext4 /boot partition and instead included it in the btrfs system root. GRUB preboot writes to /boot/grub2/grubenv if the boot was successful. This error occurs because of the GRUB btrfs.mod driver, unlike ext4, is read-only.

To resolve this, open the Gnome terminal and execute the following command.

$ sudo grub2-editenv - unset menu_auto_hide

Set the btrfs volume label. I will name the btrfs volume FEDORA, but you can name it whatever you want.

$ sudo btrfs filesystem label / FEDORA

$ sudo btrfs filesystem show /
Label: 'FEDORA'  uuid: d2808262-1fd0-436f-9676-5be297a8ef18
	Total devices 1 FS bytes used 7.15GiB
	devid    1 size 127.40GiB used 10.02GiB path /dev/mapper/luks-a9a402be-cc3d-4190-8487-50562dc43e22

Your setup should look something like this.

$ lsblk -p /dev/vda
NAME                                                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
/dev/vda                                                  252:0    0   128G  0 disk  
├─/dev/vda1                                               252:1    0   600M  0 part  /boot/efi
└─/dev/vda2                                               252:2    0 127.4G  0 part  
  └─/dev/mapper/luks-a9a402be-cc3d-4190-8487-50562dc43e22 253:0    0 127.4G  0 crypt /home
                                                                                     /

And the subvolumes will look like this.

$ sudo btrfs subvolume list /
ID 256 gen 49 top level 5 path home
ID 257 gen 37 top level 5 path var/lib/machines

In Fedora Workstation, systemd automatically creates the /var/lib/machines subvolume for systemd-nspawn containers.

Install the packages listed below.

$ sudo dnf install vim git inotify-tools make

Now, update your operating system.

$ sudo dnf update

Do not reboot your system just yet. There is an update that replaces the grub.cfg file in the ESP. First check if your grub.cfg in ESP has been replaced.

$ sudo cat /boot/efi/EFI/fedora/grub.cfg
cryptomount -u a9a402becc3d4190848750562dc43e22
search --no-floppy --fs-uuid --set=dev d2808262-1fd0-436f-9676-5be297a8ef18
set prefix=($dev)/boot/grub2

export $prefix
configfile $prefix/grub.cfg

You should see the cryptomount command highlighted in amber. If it is present, you can just reboot. You are good to go.

$ sudo reboot

If the cryptomount command is missing, it means your grub.cfg file has been replaced. Execute the commands in the order listed below. This is the only time you will need to do it.

$ LUKS_UUID="$(sudo grub2-probe --target=cryptodisk_uuid /)" \
    ; echo $LUKS_UUID
    
$ sudo sed -i.bkp2 "1i cryptomount -u ${LUKS_UUID//-/}" \
    /boot/efi/EFI/fedora/grub.cfg
    
$ sudo cat /boot/efi/EFI/fedora/grub.cfg

$ sudo reboot

4. Create the Additional Subvolumes

I will now proceed to create the remaining subvolumes on the system root.

These subvolumes are created on the system root to avoid being included in the rollback process. Some of these directories store temporary files and caches. Some contain data that you do not want to lose if you roll back the system root. And some should be in read-write mode when booting into a read-only snapshot.

Create the directory '/var/lib/libvirt'. Only applicable to other Fedora Spins; not required for Fedora Workstation.

$ sudo mkdir -vp /var/lib/libvirt

Get the UUID of your btrfs file system.

$ ROOT_UUID="$(sudo grub2-probe --target=fs_uuid /)" ; echo $ROOT_UUID
d2808262-1fd0-436f-9676-5be297a8ef18

Get the btrfs subvolume mount options from the /etc/fstab file.

$ OPTIONS="$(grep '/home' /etc/fstab \
    | awk '{print $4}' \
    | cut -d, -f2-)" \
    ; echo $OPTIONS
compress=zstd:1,x-systemd.device-timeout=0

Declare the remaining subvolumes you want to create in the 'SUBVOLUMES' bash array. Copy from 'SUBVOLUMES' to ')', paste into the terminal, and press [ENTER].

$ SUBVOLUMES=(
    "opt"
    "var/cache"
    "var/crash"
    "var/lib/AccountsService"
    "var/lib/gdm"
    "var/lib/libvirt/images"
    "var/log"
    "var/spool"
    "var/tmp"
    "var/www"
    "home/$USER/.mozilla"
)

If you are installing Fedora for KDE Desktop, replace "var/lib/gdm" with:

"var/lib/sddm"

If you are installing Fedora for XFCE Desktop, replace "var/lib/gdm" with:

"var/lib/lightdm"
"var/lib/lightdm-data"

If you intend to install Google Chrome web browser, you should also create a subvolume for it. Add the following to the SUBVOLMES array:

"home/$USER/.config/google-chrome"

If you intend to install Brave web browser, you should also create a subvolume for it. Add the following to the SUBVOLMES array:

"home/$USER/.config/BraveSoftware"

If you intend to install the Mozilla Thunderbird email client, add the following to the SUBVOLMES array:

"home/$USER/.thunderbird"

If you use the GnuPG encryption and signing tool, you must create a subvolume to avoid losing your keys.

"home/$USER/.gnupg"

Similarly, if you use the SSH remote login client, you must create a subvolume to prevent losing your keys.

"home/$USER/.ssh"

For this tutorial, however, I will only create the '.mozilla' subvolume in the user's home directory.

$ printf '%s\n' "${SUBVOLUMES[@]}"
opt
var/cache
var/crash
var/lib/AccountsService
var/lib/gdm
var/lib/libvirt/images
var/log
var/spool
var/tmp
var/www
home/madhu/.mozilla

Find the length of the longest element in the SUBVOLMES array.

$ MAX_LEN="$(printf '/%s\n' "${SUBVOLUMES[@]}" | wc -L)" ; echo $MAX_LEN
24

Now, create subvolumes and also add them to the /etc/fstab file. Copy from 'for' to 'done', paste it in the terminal, and hit [ENTER].

$ for dir in "${SUBVOLUMES[@]}" ; do
    if [[ -d "/${dir}" ]] ; then
        sudo mv -v "/${dir}" "/${dir}-old"
        sudo btrfs subvolume create "/${dir}"
        sudo cp -ar "/${dir}-old/." "/${dir}/"
    else
        sudo btrfs subvolume create "/${dir}"
    fi
    sudo restorecon -RF "/${dir}"
    printf "%-41s %-${MAX_LEN}s %-5s %-s %-s\n" \
        "UUID=${ROOT_UUID}" \
        "/${dir}" \
        "btrfs" \
        "subvol=${dir},${OPTIONS}" \
        "0 0" | \
        sudo tee -a /etc/fstab
done

Once you have finished creating subvolumes based on your requirements, change the ownership of the newly created directories in the user's home to your username.

$ sudo chown -cR $USER:$USER ~/$(ls -A)
$ sudo restorecon -vRF ~/$(ls -A)

If you have created subvolumes '.gnupg' and '.ssh', you must change their permissions.

$ sudo chmod -vR 0700 ~/{.gnupg,.ssh}

Examine your /etc/fstab file. This is what it should look like. The UUID and username will be unique to your system.

$ cat /etc/fstab
....
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /                        btrfs defaults,x-systemd.device-timeout=0 0 0
UUID=1D52-3A35                            /boot/efi                vfat  umask=0077,shortname=winnt 0 2
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /home                    btrfs subvol=home,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /opt                     btrfs subvol=opt,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/cache               btrfs subvol=var/cache,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/crash               btrfs subvol=var/crash,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/lib/AccountsService btrfs subvol=var/lib/AccountsService,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/lib/gdm             btrfs subvol=var/lib/gdm,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/lib/libvirt/images  btrfs subvol=var/lib/libvirt/images,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/log                 btrfs subvol=var/log,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/spool               btrfs subvol=var/spool,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/tmp                 btrfs subvol=var/tmp,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /var/www                 btrfs subvol=var/www,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /home/madhu/.mozilla     btrfs subvol=home/madhu/.mozilla,compress=zstd:1,x-systemd.device-timeout=0 0 0

Reload /etc/fstab to mount all the subvolumes.

$ sudo systemctl daemon-reload

$ sudo mount -va
/                        : ignored
/boot/efi                : already mounted
/home                    : already mounted
/opt                     : successfully mounted
/var/cache               : successfully mounted
/var/crash               : successfully mounted
/var/lib/AccountsService : successfully mounted
/var/lib/gdm             : successfully mounted
/var/lib/libvirt/images  : successfully mounted
/var/log                 : successfully mounted
/var/spool               : successfully mounted
/var/tmp                 : successfully mounted
/var/www                 : successfully mounted
/home/madhu/.mozilla     : successfully mounted

Check your subvolumes.

$ sudo btrfs subvolume list /
ID 256 gen 93 top level 5 path home
ID 257 gen 83 top level 5 path var/lib/machines
ID 258 gen 91 top level 5 path opt
ID 259 gen 92 top level 5 path var/cache
ID 260 gen 91 top level 5 path var/crash
ID 261 gen 91 top level 5 path var/lib/AccountsService
ID 262 gen 91 top level 5 path var/lib/gdm
ID 263 gen 92 top level 5 path var/lib/libvirt/images
ID 264 gen 92 top level 5 path var/log
ID 265 gen 92 top level 5 path var/spool
ID 266 gen 92 top level 5 path var/tmp
ID 267 gen 92 top level 5 path var/www
ID 268 gen 92 top level 256 path home/madhu/.mozilla

Your setup should now look somewhat like this.

$ lsblk -p /dev/vda
NAME                                                      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS
/dev/vda                                                  252:0    0   128G  0 disk  
├─/dev/vda1                                               252:1    0   600M  0 part  /boot/efi
└─/dev/vda2                                               252:2    0 127.4G  0 part  
  └─/dev/mapper/luks-a9a402be-cc3d-4190-8487-50562dc43e22 253:0    0 127.4G  0 crypt /home/madhu/.mozilla
                                                                                     /var/www
                                                                                     /var/tmp
                                                                                     /var/spool
                                                                                     /var/log
                                                                                     /var/lib/libvirt/images
                                                                                     /var/lib/gdm
                                                                                     /var/lib/AccountsService
                                                                                     /var/crash
                                                                                     /var/cache
                                                                                     /opt
                                                                                     /home
                                                                                     /

Now that everything appears to be in order, you can delete the old directories.

$ for dir in "${SUBVOLUMES[@]}" ; do
    if [[ -d "/${dir}-old" ]] ; then
        sudo rm -rvf "/${dir}-old"
    fi
done

5. Bypass the Additional Passphrase Prompt

While GRUB requests a passphrase in order to decrypt the encrypted /boot files, this information is not passed on to initramfs. As a result, the root (/) must be unlocked once more during the initramfs stage. This means that the user must enter his passphrase twice.

I will show you two methods for automatically decrypting your disk without requiring a passphrase: using a key file or TPM2. Choose your preferred method.

5.1 Using a Key File

If you want to decrypt your LUKS-encrypted volume using a key file, use this method. Secure Boot does not need to be enabled for this.

Change to super user.

$ sudo -i

Get the UUID of the LUKS device partition.

# LUKS_UUID="$(grub2-probe --target=cryptodisk_uuid /)" ; echo $LUKS_UUID
a9a402be-cc3d-4190-8487-50562dc43e22

Create the /etc/cryptsetup-keys.d directory if it does not already exist.

# mkdir -v /etc/cryptsetup-keys.d

Create a key file with 4KiB of random data. The key file has to be in volume.key format.

# dd if=/dev/random \
   of=/etc/cryptsetup-keys.d/luks-${LUKS_UUID}.key \
   bs=512 count=8

Ensure that only the root user has read access to the key file.

# chmod -c 0400 /etc/cryptsetup-keys.d/luks-${LUKS_UUID}.key

Attach the newly created key to the encrypted device’s available key slot. When prompted for a passphrase, enter the same LUKS passphrase.

# cryptsetup luksAddKey \
    --pbkdf pbkdf2 \
    --pbkdf-force-iterations 500000 \
    /dev/disk/by-uuid/${LUKS_UUID} \
    /etc/cryptsetup-keys.d/luks-${LUKS_UUID}.key

Add the key to /etc/dracut.conf.d/cryptodisk.conf file to include it in the initramfs image.

# echo \
  "install_items+=\" /etc/cryptsetup-keys.d/luks-${LUKS_UUID}.key \"" \
  > /etc/dracut.conf.d/cryptodisk.conf

# cat /etc/dracut.conf.d/cryptodisk.conf
install_items+=" /etc/cryptsetup-keys.d/luks-a9a402be-cc3d-4190-8487-50562dc43e22.key "

Rebuild the initramfs image.

# dracut -vf

Reboot the computer to ensure that everything is working properly.

# reboot

The LUKS passphrase should only be asked once before the GRUB menu appears.

5.2 Using TPM2

This method allows you to decrypt your LUKS-encrypted volumes using the TPM on your computer. You must have Secure Boot enabled for this method to work.

Change to super user.

$ sudo -i

Make sure Secure Boot is enabled.

# mokutil --sb-state
SecureBoot enabled

Make sure your computer supports TPM.

# systemd-cryptenroll --tpm2-device=list
PATH        DEVICE      DRIVER 
/dev/tpmrm0 MSFT0101:00 tpm_crb

Find the LUKS device UUID and save it in a variable.

# LUKS_UUID="$(grub2-probe --target=cryptodisk_uuid /)" \
    ; echo $LUKS_UUID
a9a402be-cc3d-4190-8487-50562dc43e22

Enroll the encrypted volume. When prompted for a passphrase, enter the same LUKS passphrase you set earlier.

# systemd-cryptenroll \
    --wipe-slot tpm2 \
    --tpm2-device /dev/tpmrm0 \
    /dev/disk/by-uuid/${LUKS_UUID}

According to the systemd-cryptenroll man page, it should be sufficient for most applications to bind against PCR 7, which is also the default. For more information, see man systemd-cryptenroll.

Verify that the TPM was added to the encrypted volume.

# systemd-cryptenroll /dev/disk/by-uuid/${LUKS_UUID}
SLOT TYPE    
   0 password
   1 tpm2

Add the TPM2 device node path to the kernel command line parameter 'rd.luks.options='.

# grubby --update-kernel=ALL \
    --args="rd.luks.options=tpm2-device=/dev/tpmrm0"

Verify that the TPM2 device node path has been added to the grub kernel parameter.

# grubby --info=DEFAULT
index=0
kernel="/boot/vmlinuz-6.11.5-300.fc41.x86_64"
args="ro rd.luks.uuid=luks-a9a402be-cc3d-4190-8487-50562dc43e22 rhgb quiet $tuned_params rd.luks.options=tpm2-device=/dev/tpmrm0"
root="UUID=d2808262-1fd0-436f-9676-5be297a8ef18"
initrd="/boot/initramfs-6.11.5-300.fc41.x86_64.img $tuned_initrd"
title="Fedora Linux (6.11.5-300.fc41.x86_64) 41 (Workstation Edition)"
id="170092c7bac7451489daeafac8d5dad8-6.11.5-300.fc41.x86_64"

Changes should also be reflected in the /etc/crypttab file.

# echo \
  "luks-${LUKS_UUID} UUID=${LUKS_UUID} none tpm2-device=/dev/tpmrm0" \
  > /etc/crypttab

# cat /etc/crypttab
luks-a9a402be-cc3d-4190-8487-50562dc43e22 UUID=a9a402be-cc3d-4190-8487-50562dc43e22 none tpm2-device=/dev/tpmrm0

Add the tpm2-tss module to /etc/dracut.conf.d/tpm2.conf file.

# echo 'add_dracutmodules+=" tpm2-tss "' > /etc/dracut.conf.d/tpm2.conf

# cat /etc/dracut.conf.d/tpm2.conf
add_dracutmodules+=" tpm2-tss "

Update the grub.cfg file.

# grub2-mkconfig -o /boot/grub2/grub.cfg

Run dracut to rebuild the initramfs.

# dracut -vf

Finally, reboot.

# reboot

The LUKS passphrase should only be asked once before the GRUB menu appears.

6. Install and Configure Snapper

Fedora 41 now uses dnf5 as its default package manager, replacing dnf4.

Unlike dnf4, the new dnf5 package is a fully featured package manager that does not require Python dependencies.

As a result, Fedora 41 no longer includes the snapper plugin python3-dnf-plugin-snapper. This plugin was in charge of creating pre and post snapshosts whenever packages were installed using the dnf4 package manager.

In dnf5, a new way to implement this function is introduced: the 'Actions plugin'. The Actions plugin is more powerful than the previous snapper plugin, as it can run external applications before or after transactions and interact with the dnf5 configuration.

So let us install the necessary packages.

$ sudo dnf install snapper libdnf5-plugin-actions

Create an Actions file called snapper.actions. Copy from 'sudo' to 'EOF', paste it into the terminal, and hit [Enter]. (Sources: 1 and 2)

$ sudo bash -c "cat > /etc/dnf/libdnf5-plugins/actions.d/snapper.actions" <<'EOF'
# Get snapshot description
pre_transaction::::/usr/bin/sh -c echo\ "tmp.cmd=$(ps\ -o\ command\ --no-headers\ -p\ '${pid}')"

# Creates pre snapshot before the transaction and stores the snapshot number in the "tmp.snapper_pre_number"  variable.
pre_transaction::::/usr/bin/sh -c echo\ "tmp.snapper_pre_number=$(snapper\ create\ -t\ pre\ -c\ number\ -p\ -d\ '${tmp.cmd}')"

# If the variable "tmp.snapper_pre_number" exists, it creates post snapshot after the transaction and removes the variable "tmp.snapper_pre_number".
post_transaction::::/usr/bin/sh -c [\ -n\ "${tmp.snapper_pre_number}"\ ]\ &&\ snapper\ create\ -t\ post\ --pre-number\ "${tmp.snapper_pre_number}"\ -c\ number\ -d\ "${tmp.cmd}"\ ;\ echo\ tmp.snapper_pre_number\ ;\ echo\ tmp.cmd
EOF

Create new snapper configurations named root and home for the btrfs volume at / and /home, respectively.

$ sudo snapper -c root create-config /
$ sudo snapper -c home create-config /home

The snapper configuration files will be saved in the /etc/snapper/configs/ directory.

Verify the snapper configuration files have been created.

$ sudo snapper list-configs
Config │ Subvolume
───────┼──────────
home   │ /home
root   │ /

Allow regular user to use snapper without requiring root privileges. Add your user name to the snapper's root and home configurations to set the ACL on the /.snapshots and /home/.snapshots directories.

$ sudo snapper -c root set-config ALLOW_USERS=$USER SYNC_ACL=yes
$ sudo snapper -c home set-config ALLOW_USERS=$USER SYNC_ACL=yes

Add the newly created snapshot subvolumes to the /etc/fstab file.

Execute the following commands sequentially. For the last command, copy from 'for' to 'done', paste it into your terminal, and press [ENTER].

$ ROOT_UUID="$(sudo grub2-probe --target=fs_uuid /)"

$ MAX_LEN="$(cat /etc/fstab | awk '{print $2}' | wc -L)"

$ OPTIONS="$(grep '/opt' /etc/fstab \
    | awk '{print $4}' \
    | cut -d, -f2-)"
    
$ for dir in '.snapshots' 'home/.snapshots' ; do
    printf "%-41s %-${MAX_LEN}s %-5s %-s %-s\n" \
        "UUID=${ROOT_UUID}" \
        "/${dir}" \
        "btrfs" \
        "subvol=${dir},${OPTIONS}" \
        "0 0" | \
        sudo tee -a /etc/fstab
done

Your /etc/fstab file should look like this. The UUIDs will be unique to your system.

$ cat /etc/fstab
....
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /.snapshots              btrfs subvol=.snapshots,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=d2808262-1fd0-436f-9676-5be297a8ef18 /home/.snapshots         btrfs subvol=home/.snapshots,compress=zstd:1,x-systemd.device-timeout=0 0 0

Reload the /etc/fstab file.

$ sudo systemctl daemon-reload
$ sudo mount -va

Your subvolumes will look like this:

$ sudo btrfs subvolume list /
....
ID 269 gen 126 top level 5 path .snapshots
ID 270 gen 126 top level 256 path home/.snapshots

Disable the indexing of the .snapshots directories by updatedb. It is enabled by default, which can cause significant slowdown and excessive memory usage if you have a large number of snapshots.

$ echo 'PRUNENAMES = ".snapshots"' | sudo tee -a /etc/updatedb.conf

Enable snapshot booting by appending the SUSE_BTRFS_SNAPSHOT_BOOTING="true" option to the /etc/default/grub file.

$ echo 'SUSE_BTRFS_SNAPSHOT_BOOTING="true"' | sudo tee -a /etc/default/grub

You must also make changes to the /boot/efi/EFI/fedora/grub.cfg file now that snapshot booting is enabled.

$ sudo sed -i.bkp1 '1i set btrfs_relative_path="yes"' /boot/efi/EFI/fedora/grub.cfg

Finally, update the grub.cfg file.

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Later, I'll enable snapper timeline snapshots. For now, the snapper configuration is complete.

List the snapshots for / volume. For the root, you may use snapper -c root ls or simply snapper ls. Both provide the same output.

$ snapper ls
# │ Type   │ Pre # │ Date │ User │ Cleanup │ Description │ Userdata
──┼────────┼───────┼──────┼──────┼─────────┼─────────────┼─────────
0 │ single │       │      │ root │         │ current     │

List the snapshots for /home subvolume.

$ snapper -c home ls
# │ Type   │ Pre # │ Date │ User │ Cleanup │ Description │ Userdata
──┼────────┼───────┼──────┼──────┼─────────┼─────────────┼─────────
0 │ single │       │      │ root │         │ current     │

At this stage, you do not have any snapshots, we will create some later.

7. Install and Configure Grub-Btrfs

The grub-btrfs package adds 'Fedora Linux snapshots' to the GRUB menu and allows you to test a snapshot in read-only mode before rolling back to it in read-write mode.

Clone the grub-btrfs repository.

$ git clone https://github.com/Antynea/grub-btrfs

The setup is pre-configured to work with Arch and Gentoo out of the box. You must make a few changes to the grub-btrfs config file to make it work with Fedora.

Change the directory to grub-btrfs.

$ cd grub-btrfs

Copy from 'sed' to 'config', paste it into the terminal, and press [ENTER].

$ sed -i.bkp \
-e '/#GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS/a \
GRUB_BTRFS_SNAPSHOT_KERNEL_PARAMETERS="systemd.volatile=state"' \
-e '/#GRUB_BTRFS_GRUB_DIRNAME/a \
GRUB_BTRFS_GRUB_DIRNAME="/boot/grub2"' \
-e '/#GRUB_BTRFS_MKCONFIG=/a \
GRUB_BTRFS_MKCONFIG=/usr/sbin/grub2-mkconfig' \
-e '/#GRUB_BTRFS_SCRIPT_CHECK=/a \
GRUB_BTRFS_SCRIPT_CHECK=grub2-script-check' \
config

Then install it.

$ sudo make install

Update grub.cfg and enable the grub-btrfsd.service.

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg
$ sudo systemctl enable --now grub-btrfsd.service

You will receive the 'No snapshots found' error since you have not yet created any snapshots, but don't worry, it will function properly after you do.

Your grub-btrfs installation is now complete. You may now delete the cloned grub-btrfs repository.

$ cd ..
$ rm -rvf grub-btrfs

8. Create a System Root Snapshot and Set It as the Default

Now that everything in Fedora has been installed and configured, it is time to take a subvolume snapshot of the main volume and set the subvolume as the default ‘/‘ file system. By manually creating a subvolume snapshot of the main volume and making it the default, you can easily rollback without having to specify the snapshot number and the 'ambit' during the rollback when using the snapper tool.

Create a directory named '1' in the /.snapshots directory.

$ sudo mkdir -v /.snapshots/1

Create an XML file called info.xml in the /.snapshots/1/ directory. Copy the following command from 'sudo' to 'EOF', paste it into your terminal, and hit the [ENTER] button.

$ sudo bash -c "cat > /.snapshots/1/info.xml" <<EOF
<?xml version="1.0"?>
<snapshot>
  <type>single</type>
  <num>1</num>
  <date>$(date -u +"%F %T")</date>
  <description>first root subvolume</description>
</snapshot>
EOF

Verify that the date is set properly and is in the Coordinated Universal Time (UTC) format.

$ cat /.snapshots/1/info.xml
<?xml version="1.0"?>
<snapshot>
  <type>single</type>
  <num>1</num>
  <date>2024-10-29 15:40:55</date>
  <description>first root subvolume</description>
</snapshot>

Now, create a read-write subvolume snapshot of the system root in the /.snapshots/1/ directory.

$ sudo btrfs subvolume snapshot / /.snapshots/1/snapshot

Get the subvolid of the /.snapshots/1/snapshot subvolume.

$ SNAP_1_ID="$(sudo btrfs inspect-internal rootid /.snapshots/1/snapshot)"

$ echo ${SNAP_1_ID}
271

Using this subvolume ID, set the /.snapshots/1/snapshot subvolume as the default subvolume for the root (/) filesystem.

$ sudo btrfs subvolume set-default ${SNAP_1_ID} /

Confirm that the /.snapshots/1/snapshot subvolume has been set as the default for the / filesystem.

$ sudo btrfs subvolume get-default /
ID 271 gen 148 top level 269 path .snapshots/1/snapshot

Finally, reboot your system.

$ sudo reboot

After rebooting, check the snapshots in snapper.

$ snapper ls
 # │ Type   │ Pre # │ Date                            │ User │ Cleanup │ Description          │ Userdata
───┼────────┼───────┼─────────────────────────────────┼──────┼─────────┼──────────────────────┼─────────
0  │ single │       │                                 │ root │         │ current              │
1* │ single │       │ Tue 29 Oct 2024 11:40:55 AM EDT │ root │         │ first root subvolume │

As you can see, the /.snapshots/1/snapshot subvolume is also visible as snapshot #1 in snapper. The asterisk (*) indicates that this snapshot is the default and is currently active.

The installation of Fedora 41 with snapshot and rollback support is now complete. You can now begin taking snapshots.

I ran some tests to determine the capabilities of the snapper tool as well as to demonstrate how to use it. If you're trying this out in a virtual machine, I highly recommend running these tests and any other scenarios you think of where you might want to explore different ways to destroy and recover.

The link is here: Snapper Tests. Although these tests are conducted on Fedora 40, they also apply to Fedora 41. There are no changes in Fedora 41.

9. Enable Automatic Timeline Snapshots

Now that you've finished everything, you can enable automatic timeline snapshots. When the timeline is enabled, a snapshot is created once every hour. Once per day, the timeline cleanup algorithm cleans up the snapshots.

Automatic Timeline Snapshots are enabled by default in both root and home configurations. Enable it only for the system root and disable it for the home. See the Arch Wiki page on 'Automatic timeline snapshots' for more information.

$ sudo snapper -c home set-config TIMELINE_CREATE=no
$ sudo systemctl enable --now snapper-timeline.timer
$ sudo systemctl enable --now snapper-cleanup.timer

That's all. Every hour from now on, a 'single' snapshot will be created and cleaned up every other day.

$ snapper ls
 # │ Type   │ Pre # │ Date                            │ User │ Cleanup  │ Description          │ Userdata
───┼────────┼───────┼─────────────────────────────────┼──────┼──────────┼──────────────────────┼─────────
0  │ single │       │                                 │ root │          │ current              │
1* │ single │       │ Tue 29 Oct 2024 11:40:55 AM EDT │ root │          │ first root subvolume │
2  │ single │       │ Tue 29 Oct 2024 12:00:02 PM EDT │ root │ timeline │ timeline             │
3  │ single │       │ Tue 29 Oct 2024 01:00:02 PM EDT │ root │ timeline │ timeline             │

If you want to stop timeline snapshots at any time, simply disable the snapper timers.

$ sudo systemctl disable --now snapper-timeline.timer
$ sudo systemctl disable --now snapper-cleanup.timer

The installation of Fedora 41 with snapshot and rollback support is now complete.

10. Issues and Possible Solutions

Issue: This usually occurs when a kernel update has just been released and all of the kernel's supporting files have not yet been propagated to a repository mirror near you.

This is caused by the kernel parameter '{extra_cmdline}'. The '{extra_cmdline}' parameter is added to the kernel when you add the 'SUSE_BTRFS_SNAPSHOT_BOOTING=true' key to the /etc/default/grub file.

Solution: From the GRUB menu, select the kernel version you want to boot, and then press 'e' to modify the command.

Go to the line that starts with the word 'linux', and move the parameter '{extra_cmdline}' to the end of the line. The parameter '{extra_cmdline}' should always be at the end of the line.

From this:

Install Fedora 41 with Full Disk Encryption Snapshot and Rollback Support - GRUB Error 1

To this:

Install Fedora 41 with Full Disk Encryption Snapshot and Rollback Support - GRUB Error 2

Then, on your keyboard, press Ctrl+x. You will be successfully booted into the operating system. Next, launch the terminal and execute the following command.

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

Reboot the computer to ensure the problem has been resolved.

Issue: The problem appears to be that GRUB does not support TPM 2.0 very well when enabled in UEFI System Settings.

Solution: I looked into it, and it appears that this solution works perfectly. All credit goes to Eric Renfro.

Create a script named 02_tpm in the /etc/grub.d/ directory with the following contents.

# vim /etc/grub.d/02_tpm
#!/usr/bin/sh -e
echo "rmmod tpm"

Set the execute permission on a file.

# chmod +x /etc/grub.d/02_tpm

Update the grub.cfg file.

# grub2-mkconfig -o /boot/grub2/grub.cfg

This issue should now be resolved.

11. Upgrade Fedora 40 to Fedora 41

If you have already installed Fedora 40 using an earlier version of this tutorial and want to upgrade to Fedora 41, this section is for you.

First, if you are using timeline snapshots, temporarily disable them. Once the upgrade is complete, you can re-enable them.

$ sudo systemctl disable --now snapper-timeline.timer
$ sudo systemctl disable --now snapper-cleanup.timer

Take a snapshot of the system root in its current state. So, if something goes wrong, you can always roll back and start over.

$ snapper create -d 'Fedora 40 restore point'

Update your Fedora 40 release. This is important. Do not skip this step.

$ sudo dnf upgrade --refresh -y

After the updates have been installed, reboot your computer.

$ sudo reboot

After the reboot, if you encounter the 'lexer.c:352:syntax' error, refer to Section 10 Problem 1 to boot normally.


After rebooting, open the terminal and run the following command to upgrade your Fedora 40 operating system to the most recent Fedora 41 version.

$ sudo dnf system-upgrade download --releasever=41 -y

Once the upgrades have been downloaded, trigger the upgrade process.

This will immediately reboot your computer, with no countdown or confirmation. So, before issuing the following command, close all other programs and save your work.

$ sudo dnf system-upgrade reboot

When the upgrade process is finished, your system will reboot and you will be directed to the grub prompt because your old grub.cfg file has been replaced with a new one. This new grub.cfg file does not include the cryptomount command, which is required to boot the encrypted luks volume.

Install Fedora 41 with Full Disk Encryption Snapshot and Rollback Support - GRUB Prompt

Locate your encrypted luks volume by using the command ls.

grub> ls
(memdisk) (proc) (hd0) (hd0,gpt2) (hd0,gpt1)

As I am using a single disk, the encrypted luks volume is on disk 0 and partition 2. For me, it will be (hd0,gpt2).

Use the cryptomount command to unlock your encrypted luks volume.

grub> cryptomount (hd0,gpt2)

You will be prompted for the LUKS passphrase. Enter your LUKS passphrase. Please be patient as the unlocking process takes some time.

Install Fedora 41 with Full Disk Encryption Snapshot and Rollback Support - LUKS PassPhrase

Then, use the normal command to boot normally.

grub> normal

You will now boot normally into Fedora 41.

Open the terminal and run the following command to confirm that your system has indeed been upgraded to Fedora 41 version.

$ cat /etc/fedora-release
Fedora release 41 (Forty One)

Check your grub.cfg file.

$ sudo cat /boot/efi/EFI/fedora/grub.cfg
search --no-floppy --root-dev-only --fs-uuid --set=dev 30cca32b-5982-4069-9ebe-c96d3579125f
set prefix=($dev)/.snapshots/1/snapshot/boot/grub2
export $prefix
configfile $prefix/grub.cfg

As you can see, the cryptomount command is missing. Also, notice that the snapshot's absolute path has been added to the prefix line, which is highlighted in amber.

In my case, it is snapshot number 1, from which I upgraded to Fedora 41. In your case, the snapshot number could be different.

The problem with this is that if you roll back, no matter which snapshot number you rollback from, you always end up rolling back to snapshot number 1 (in my case).

To resolve this, execute the commands listed below in order.

$ LUKS_DEVICE="$(sudo cryptsetup status \
    $(sudo grub2-probe --target=device /) \
    | grep 'device:' | awk '{print $2}')" \
    ; echo $LUKS_DEVICE
    
$ LUKS_UUID="$(sudo cryptsetup luksUUID ${LUKS_DEVICE})" \
    ; echo $LUKS_UUID
    
$ DEF_SNAP="$(sudo btrfs subvolume get-default / | awk '{print $NF}')/" \
    ; echo $DEF_SNAP
    
$ sudo sed -i.bkp3 \
    -e '1i set btrfs_relative_path="yes"' \
    -e "1i cryptomount -u ${LUKS_UUID//-/}" \
    -e "s|$DEF_SNAP||g" \
    /boot/efi/EFI/fedora/grub.cfg

Check your grub.cfg file again. It should now look like this.

$ sudo cat /boot/efi/EFI/fedora/grub.cfg
set btrfs_relative_path="yes"
cryptomount -u 20bb5f163489439d8cb396f7da6a6c63
search --no-floppy --root-dev-only --fs-uuid --set=dev 30cca32b-5982-4069-9ebe-c96d3579125f
set prefix=($dev)/boot/grub2
export $prefix
configfile $prefix/grub.cfg

Your grub.cfg file is now fixed.

Next, you must create an actions file because dnf5 no longer supports the snapper plugin python3-dnf-plugin-snapper.

Remove the package python3-dnf-plugin-snapper, as it is no longer supported in Fedora 41.

$ sudo dnf remove python3-dnf-plugin-snapper -y

And install the Libdnf5 plugin, which allows you to run actions.

$ sudo dnf install libdnf5-plugin-actions -y

Create an Actions file called snapper.actions. Copy from 'sudo' to 'EOF', paste it into the terminal, and hit [Enter].

$ sudo bash -c "cat > /etc/dnf/libdnf5-plugins/actions.d/snapper.actions" <<'EOF'
# Get snapshot description
pre_transaction::::/usr/bin/sh -c echo\ "tmp.cmd=$(ps\ -o\ command\ --no-headers\ -p\ '${pid}')"

# Creates pre snapshot before the transaction and stores the snapshot number in the "tmp.snapper_pre_number"  variable.
pre_transaction::::/usr/bin/sh -c echo\ "tmp.snapper_pre_number=$(snapper\ create\ -t\ pre\ -c\ number\ -p\ -d\ '${tmp.cmd}')"

# If the variable "tmp.snapper_pre_number" exists, it creates post snapshot after the transaction and removes the variable "tmp.snapper_pre_number".
post_transaction::::/usr/bin/sh -c [\ -n\ "${tmp.snapper_pre_number}"\ ]\ &&\ snapper\ create\ -t\ post\ --pre-number\ "${tmp.snapper_pre_number}"\ -c\ number\ -d\ "${tmp.cmd}"\ ;\ echo\ tmp.snapper_pre_number\ ;\ echo\ tmp.cmd
EOF

The snapper configuration is now complete.

Update the grub.cfg file.

$ sudo grub2-mkconfig -o /boot/grub2/grub.cfg

You can now re-enable timeline snapshots if you previously disabled them.

$ sudo systemctl enable --now snapper-timeline.timer
$ sudo systemctl enable --now snapper-cleanup.timer

That is it. Fedora has been successfully upgraded to version 41. Reboot your system and you are good to go.

12. Watch on YouTube


Subscribe
Notify of
guest
46 Comments
Newest
Oldest
Inline Feedbacks
View all comments