How to Build a Netboot Server, Part 2

The article How to Build a Netboot Server, Part 1 showed you how to create a netboot image with a “liveuser” account whose home directory lives in volatile memory. Most users probably want to preserve files and settings across reboots, though. So this second part of the netboot series shows how to reconfigure the netboot image from part one so that Active Directory user accounts can log in and their home directories can be automatically mounted from a NFS server.

Part 3 of this series will show how to make an interactive and centrally-configurable iPXE boot menu for the netboot clients.

Setup NFS4 Home Directories with KRB5 Authentication

Follow the directions from the previous post “Share NFS Home Directories Securely with Kerberos,” then return here.

Remove the Liveuser Account

Remove the “liveuser” account created in part one of this series:

$ sudo -i
# sed -i '/automaticlogin/Id' /fc28/etc/gdm/custom.conf
# rm -f /fc28/etc/sudoers.d/liveuser
# for i in passwd shadow group gshadow; do sed -i '/^liveuser:/d' /fc28/etc/$i; done

Configure NTP, KRB5 and SSSD

Next, we will need to duplicate the NTP, KRB5, and SSSD configuration that we set up on the server in the client image so that the same accounts will be available:

# MY_HOSTNAME=$(</etc/hostname)
# MY_DOMAIN=${MY_HOSTNAME#*.}
# dnf -y --installroot=/fc28 install ntp krb5-workstation sssd
# cp /etc/ntp.conf /fc28/etc
# chroot /fc28 systemctl enable ntpd.service
# cp /etc/krb5.conf.d/${MY_DOMAIN%%.*} /fc28/etc/krb5.conf.d
# cp /etc/sssd/sssd.conf /fc28/etc/sssd

Reconfigure sssd to provide authentication services, in addition to the identification service already configured:

# sed -i '/services =/s/$/, pam/' /fc28/etc/sssd/sssd.conf

Also, ensure none of the clients attempt to update the computer account password:

# sed -i '/id_provider/a \ \ ad_maximum_machine_account_password_age = 0' /fc28/etc/sssd/sssd.conf

Also, copy the nfsnobody definitions:

# for i in passwd shadow group gshadow; do grep "^nfsnobody:" /etc/$i >> /fc28/etc/$i; done

Join Active Directory

Next, you’ll perform a chroot to join the client image to Active Directory. Begin by deleting any pre-existing computer account with the same name your netboot image will use:

# MY_USERNAME=jsmith
# MY_CLIENT_HOSTNAME=$(</fc28/etc/hostname)
# adcli delete-computer "${MY_CLIENT_HOSTNAME%%.*}" -U "$MY_USERNAME"

Also delete the krb5.keytab file from the netboot image if it exists:

# rm -f /fc28/etc/krb5.keytab

Perform a chroot into the netboot image:

# for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc28/$i; done
# chroot /fc28 /usr/bin/bash --login

Perform the join:

# MY_USERNAME=jsmith
# MY_HOSTNAME=$(</etc/hostname)
# MY_DOMAIN=${MY_HOSTNAME#*.}
# MY_REALM=${MY_DOMAIN^^}
# MY_OU="cn=computers,dc=${MY_DOMAIN//./,dc=}"
# adcli join $MY_DOMAIN --login-user="$MY_USERNAME" --computer-name="${MY_HOSTNAME%%.*}" --host-fqdn="$MY_HOSTNAME" --user-principal="host/$MY_HOSTNAME@$MY_REALM" --domain-ou="$MY_OU"

Now log out of the chroot and clear the root user’s command history:

# logout
# for i in run sys proc dev/shm dev/pts dev; do umount /fc28/$i; done
# > /fc28/root/.bash_history

Install and Configure PAM Mount

We want our clients to automatically mount the user’s home directory when they log in. To accomplish this, we’ll use the “pam_mount” module. Install and configure pam_mount:

# dnf install -y --installroot=/fc28 pam_mount
# cat << END > /fc28/etc/security/pam_mount.conf.xml
<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE pam_mount SYSTEM "pam_mount.conf.xml.dtd">
<pam_mount>
<debug enable="0" />
<volume uid="1400000000-1499999999" fstype="nfs4" server="$MY_HOSTNAME" path="/home/%(USER)" mountpoint="/home/%(USER)" options="sec=krb5" />
<mkmountpoint enable="1" remove="0" />
<msg-authpw>Password:</msg-authpw>
</pam_mount>
END

Reconfigure PAM to use pam_mount:

# dnf install -y patch
# cp -r /fc28/usr/share/authselect/default/sssd /fc28/etc/authselect/custom
# echo 'initgroups: files' >> /fc28/etc/authselect/custom/sssd/nsswitch.conf
# patch /fc28/etc/authselect/custom/sssd/system-auth << END
@@ -12 +12,2 @@
-auth        sufficient                                   pam_sss.so forward_pass
+auth        requisite                                    pam_mount.so {include if "with-pammount"}
+auth        sufficient                                   pam_sss.so {if "with-pammount":use_first_pass|forward_pass}
@@ -35,2 +36,3 @@
 session     required                                     pam_unix.so
+session     optional                                     pam_mount.so {include if "with-pammount"}
 session     optional                                     pam_sss.so
END
# patch /fc28/etc/authselect/custom/sssd/password-auth << END
@@ -9 +9,2 @@
-auth        sufficient                                   pam_sss.so forward_pass
+auth        requisite                                    pam_mount.so {include if "with-pammount"}
+auth        sufficient                                   pam_sss.so {if "with-pammount":use_first_pass|forward_pass}
@@ -32,2 +33,3 @@
 session     required                                     pam_unix.so
+session     optional                                     pam_mount.so {include if "with-pammount"}
 session     optional                                     pam_sss.so
END
# chroot /fc28 authselect select custom/sssd with-pammount --force

Also ensure the NFS server’s hostname is always resolvable from the client:

# MY_IP=$(host -t A $MY_HOSTNAME | awk '{print $4}')
# echo "$MY_IP $MY_HOSTNAME ${MY_HOSTNAME%%.*}" >> /fc28/etc/hosts

Optionally, allow all users to run sudo:

# echo '%users ALL=(ALL) NOPASSWD: ALL' > /fc28/etc/sudoers.d/users

Convert the NFS Root to an iSCSI Backing-Store

Current versions of nfs-utils may have difficulty establishing a second connection from the client back to the NFS server for home directories when an nfsroot connection is already established. The client hangs when attempting to access the home directory. So, we will work around the problem by using a different protocol (iSCSI) for sharing our netboot image.

First chroot into the image to reconfigure its initramfs for booting from an iSCSI root:

# for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc28/$i; done
# chroot /fc28 /usr/bin/bash --login
# dnf install -y iscsi-initiator-utils
# sed -i 's/nfs/iscsi/' /etc/dracut.conf.d/netboot.conf
# sed -i '/^node.conn\[0\].timeo.noop_out_interval /s/=.*$/= 0/' /etc/iscsi/iscsid.conf
# sed -i '/^node.conn\[0\].timeo.noop_out_timeout /s/=.*$/= 0/' /etc/iscsi/iscsid.conf
# sed -i '/^node.session.timeo.replacement_timeout /s/=.*$/= 86400/' /etc/iscsi/iscsid.conf
# echo 'omit_drivers+=" qedi "' > /etc/dracut.conf.d/omit-qedi.conf
# echo 'blacklist qedi' > /etc/modprobe.d/blacklist-qedi.conf
# KERNEL=$(ls -c /lib/modules | head -n 1)
# INITRD=$(find /boot -name 'init*' | grep -m 1 $KERNEL)
# dracut -f $INITRD $KERNEL
# logout
# for i in run sys proc dev/shm dev/pts dev; do umount /fc28/$i; done
# > /fc28/root/.bash_history

The qedi driver broke iscsi during testing, so it’s been disabled here.

Next, create a fc28.img sparse file. This file serves as the iSCSI target’s backing store:

# FC28_SIZE=$(du -ms /fc28 | cut -f 1)
# dd if=/dev/zero of=/fc28.img bs=1MiB count=0 seek=$(($FC28_SIZE*2))

(If you have one available, a separate partition or disk drive can be used instead of creating a file.)

Next, format the image with a filesystem, mount it, and copy the netboot image into it:

# mkfs -t xfs -L NETROOT /fc28.img
# TEMP_MNT=$(mktemp -d)
# mount /fc28.img $TEMP_MNT
# cp -a /fc28/* $TEMP_MNT
# umount $TEMP_MNT

During testing using SquashFS, the client would occasionally stutter. It seems that SquashFS does not perform well when doing random I/O from a multiprocessor client. (See also The curious case of stalled squashfs reads.) If you want to improve throughput performance with filesystem compression, ZFS is probably a better option.

If you need extremely high throughput from the iSCSI server (say, for hundreds of clients), it might be possible to load balance a Ceph cluster. For more information, see Load Balancing Ceph Object Gateway Servers with HAProxy and Keepalived.

Install and Configure iSCSI

Install the scsi-target-utils package which will provide the iSCSI daemon for serving our image out to our clients:

# dnf install -y scsi-target-utils

Configure the iSCSI daemon to serve the fc28.img file:

# MY_REVERSE_HOSTNAME=$(echo $MY_HOSTNAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_HOSTNAME})
# cat << END > /etc/tgt/conf.d/fc28.conf
<target iqn.$MY_REVERSE_HOSTNAME:fc28>
  backing-store /fc28.img
  readonly 1
</target>
END

The leading iqn. is expected by /usr/lib/dracut/modules.d/40network/net-lib.sh.

Add an exception to the firewall and enable and start the service:

# firewall-cmd --add-service=iscsi-target
# firewall-cmd --runtime-to-permanent
# systemctl enable tgtd.service
# systemctl start tgtd.service

You should now be able to see the image being shared with the tgtadm command:

# tgtadm --mode target --op show

The above command should output something similar to the following:

Target 1: iqn.edu.example.server-01:fc28
    System information:
        Driver: iscsi
        State: ready
    I_T nexus information:
    LUN information:
        LUN: 0
            Type: controller
            SCSI ID: IET     00010000
            SCSI SN: beaf10
            Size: 0 MB, Block size: 1
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: No
            SWP: No
            Thin-provisioning: No
            Backing store type: null
            Backing store path: None
            Backing store flags: 
        LUN: 1
            Type: disk
            SCSI ID: IET     00010001
            SCSI SN: beaf11
            Size: 10488 MB, Block size: 512
            Online: Yes
            Removable media: No
            Prevent removal: No
            Readonly: Yes
            SWP: No 
            Thin-provisioning: No
            Backing store type: rdwr
            Backing store path: /fc28.img
            Backing store flags:
    Account information:
    ACL information:
        ALL

We can now remove the NFS share that we created in part one of this series:

# rm -f /etc/exports.d/fc28.exports
# exportfs -rv
# umount /export/fc28
# rmdir /export/fc28
# sed -i '/^\/fc28 /d' /etc/fstab

You can also delete the /fc28 filesystem, but you may want to keep it for performing future updates.

Update the ESP to use the iSCSI Kernel

Update the ESP to contain the iSCSI-enabled initramfs:

$ rm -vf $HOME/esp/linux/*.fc28.*
$ MY_KRNL=$(ls -c /fc28/lib/modules | head -n 1)
$ cp $(find /fc28/lib/modules -maxdepth 2 -name 'vmlinuz' | grep -m 1 $MY_KRNL) $HOME/esp/linux/vmlinuz-$MY_KRNL
$ cp $(find /fc28/boot -name 'init*' | grep -m 1 $MY_KRNL) $HOME/esp/linux/initramfs-$MY_KRNL.img

Update the boot.cfg file to pass the new root and netroot parameters:

$ MY_NAME=server-01.example.edu
$ MY_EMAN=$(echo $MY_NAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_NAME})
$ MY_ADDR=$(host -t A $MY_NAME | awk '{print $4}')
$ sed -i "s! root=[^ ]*! root=/dev/disk/by-path/ip-$MY_ADDR:3260-iscsi-iqn.$MY_EMAN:fc28-lun-1 netroot=iscsi:$MY_ADDR::::iqn.$MY_EMAN:fc28!" $HOME/esp/linux/boot.cfg

Now you just need to copy the updated files from your $HOME/esp/linux directory out to the ESPs of all your client systems. You should see results similar to what is shown in the below screenshot:

Upgrading the Image

First, make a copy of the current image:

# cp -a /fc28 /fc29

Chroot into the new copy of the image:

# for i in dev dev/pts dev/shm proc sys run; do mount -o bind /$i /fc29/$i; done
# chroot /fc29 /usr/bin/bash --login

Allow updating the kernel:

# sed -i 's/^exclude=kernel-\*$/#exclude=kernel-*/' /etc/dnf/dnf.conf

Perform the upgrade:

# dnf distro-sync -y --releasever=29

Prevent the kernel from being updated:

# sed -i 's/^#exclude=kernel-\*$/exclude=kernel-*/' /etc/dnf/dnf.conf

The above command is optional, but saves you from having to copy a new kernel out to the clients if you add or update a few packages in the image at some future time.

Clean up dnf’s package cache:

# dnf clean all

Exit the chroot and clear root’s command history:

# logout
# for i in run sys proc dev/shm dev/pts dev; do umount /fc29/$i; done
# > /fc29/root/.bash_history

Create the iSCSI image:

# FC29_SIZE=$(du -ms /fc29 | cut -f 1)
# dd if=/dev/zero of=/fc29.img bs=1MiB count=0 seek=$(($FC29_SIZE*2))
# mkfs -t xfs -L NETROOT /fc29.img
# TEMP_MNT=$(mktemp -d)
# mount /fc29.img $TEMP_MNT
# cp -a /fc29/* $TEMP_MNT
# umount $TEMP_MNT

Define a new iSCSI target that points to our new image and export it:

# MY_HOSTNAME=$(</etc/hostname)
# MY_REVERSE_HOSTNAME=$(echo $MY_HOSTNAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_HOSTNAME})
# cat << END > /etc/tgt/conf.d/fc29.conf
<target iqn.$MY_REVERSE_HOSTNAME:fc29>
  backing-store /fc29.img
  readonly 1
</target>
END
# tgt-admin --update ALL

Add the new kernel and initramfs to the ESP:

$ MY_KRNL=$(ls -c /fc29/lib/modules | head -n 1)
$ cp $(find /fc29/lib/modules -maxdepth 2 -name 'vmlinuz' | grep -m 1 $MY_KRNL) $HOME/esp/linux/vmlinuz-$MY_KRNL
$ cp $(find /fc29/boot -name 'init*' | grep -m 1 $MY_KRNL) $HOME/esp/linux/initramfs-$MY_KRNL.img

Update the boot.cfg in the ESP:

$ MY_DNS1=192.0.2.91
$ MY_DNS2=192.0.2.92
$ MY_NAME=server-01.example.edu
$ MY_EMAN=$(echo $MY_NAME | tr '.' "\n" | tac | tr "\n" '.' | cut -b -${#MY_NAME})
$ MY_ADDR=$(host -t A $MY_NAME | awk '{print $4}')
$ cat << END > $HOME/esp/linux/boot.cfg
#!ipxe

kernel --name kernel.efi \${prefix}/vmlinuz-$MY_KRNL initrd=initrd.img ro ip=dhcp rd.peerdns=0 nameserver=$MY_DNS1 nameserver=$MY_DNS2 root=/dev/disk/by-path/ip-$MY_ADDR:3260-iscsi-iqn.$MY_EMAN:fc29-lun-1 netroot=iscsi:$MY_ADDR::::iqn.$MY_EMAN:fc29 console=tty0 console=ttyS0,115200n8 audit=0 selinux=0 quiet
initrd --name initrd.img \${prefix}/initramfs-$MY_KRNL.img
boot || exit
END

Finally, copy the files from your $HOME/esp/linux directory out to the ESPs of all your client systems and enjoy!