Manual action needed to resolve boot failure for Fedora Atomic Desktops and Fedora IoT

Photo by Growtika on Unsplash (cropped)

Since the 39.20240617.0 and 40.20240617.0 updates for Atomic Desktops and the 40.20240617.0 update for IoT, systems with Secure Boot enabled may fail to boot if they have been installed before Fedora Linux 40. You might see the following error:

error: ../../grub-core/kern/efi/sb.c:182:bad shim signature.
error: ../../grub-core/loader/i386/efi/linux.c:258:you need to load the kernel first.

Press any key to continue...

Workaround

In order to resolve this issue, you must first boot into the previous version of your system. It should still be functional. In order to do this, reboot your system and select the previous boot entry in the selection menu displayed on boot. Its name should be something like:

Fedora Linux 39.20240610.0 (Silverblue)  (ostree:1)

Once you have logged in, search for the terminal application for your desktop and open a new terminal window. On Fedora IoT, log in via SSH or on the console. Make sure that you are not running in a toolbox for all the commands listed on this page.

If you are running a Fedora Atomic Desktop based on Fedora 39 and have not yet updated to Fedora 40, you first need to update to the latest working Fedora 39 version with those commands:

$ sudo rpm-ostree cleanup --pending
$ sudo rpm-ostree deploy 39.20240616.0

If you are running Fedora IoT, then first update to the latest working version with this command:

$ sudo rpm-ostree cleanup --pending
$ sudo rpm-ostree deploy 40.20240614.0

Then reboot your system.

Once you are logged in again on the latest working version, proceed with the following commands:

$ sudo -i
$ cp -rp /usr/lib/ostree-boot/efi/EFI /boot/efi
$ sync

Once completed, reboot your system. You should now be able to update again, as normal, using the graphical interface or the command line:

$ sudo rpm-ostree update

Why did this happen?

On Fedora Atomic Desktops and Fedora IoT systems, the components that are part of the boot chain (Shim, GRUB) are not (yet) automatically updated alongside the rest of the system. Thus, if you have installed a Fedora Atomic Desktop or a Fedora IoT system before Fedora 40, it uses an old versions of the Shim and bootloader binaries to boot your system.

When Secure Boot is enabled, the EFI firmware loads Shim first. Shim is signed by the Microsoft Third Party Certificate Authority so that it can be verified on most hardware out of the box. The Shim binary includes the Fedora certificates used to verify binaries signed by Fedora. Then Shim loads GRUB, which in turn loads the Linux kernel. Both are signed by Fedora.

Until recently, the kernel binaries where signed two times, with an older key and a newer one. With the 6.9 kernel update, the kernel is no longer signed with the old key. If GRUB or Shim is old enough and does not know about the new key, the signature verification fails.

See the initial report in the Fedora Silverblue issue tracker.

What are we doing to prevent it from happening again?

We have known for a while that not updating the bootloader was not a satisfying situation . We have been working on enabling bootupd for Fedora Atomic Desktops and Fedora IoT. bootupd is a small application that is responsible only for bootloader updates. While initially planned for Fedora Linux 38 (!), we had to delay enabling it due to various issues and missing functionality in bootupd itself and changes needed in Anaconda.

We are hoping to enable bootupd in Fedora Linux 41, hopefully by default, which should finally resolve this situation. See the Enable bootupd for Fedora Atomic Desktops and Fedora IoT Fedora Change page.

Note that the root issue also impacts Fedora CoreOS but steps have been put in place to force a bootloader update before the 6.9 kernel update. See the tracking issue for Fedora CoreOS.

For System Administrators

8 Comments

  1. br184

    Why does Fedora keep pushing breaking changes within a release cycle? Why can these changes not wait until Fedora 41? Would it have been so bad to sign all Fedora 40 kernels with both keys until it goes EOL and take the additional time to implement bootupd first?

    One does expect breakage when upgrading to a new release, but not within a fixed release. This is one of the main issues I have with Fedora. It is almost as unstable as rolling releases. It makes it really difficult to recommend Fedora to less enthusiastic Linux users that want a stable system, but still need/want relatively up-to-date software.

    With Fedora 40 I already had all Podman containers fail to start because of a breaking change with the upgrade to Podman 5.0. Since within the Fedora 39 cycle (kernel upgrade from 6.6 to 6.7) one of our older laptops no longer boots and instead only shows a black screen (bug is reported, had to downgrade the kernel to 6.6.14 for now, still unfixed). …

    • Matthew Phillips

      I’ll admit I’m biased toward Fedora but I didn’t find it nearly that big of a deal, just booted off a working image until the fix was out. Regarding Fedora pushing “breaking changes”, if you look around Fedora is usually one of the first willing to make a change for new technologies at all. IMO the risks Fedora is willing to take and an overwhelming track record for making those changes smooth is a reason to recommend Fedora to Linux users who want a stable system with the latest software.

      • br184

        Unless the primary purpose of your operating system is the system itself, it is a significant issue in my opinion.

        Most people need their OS to work predictably to do their activities. If your system suddenly stops booting for whatever reason, it can be very problematic, especially if you don’t have lots of experience.

        If someone does updates graphically on Fedora, how is one expected to find this article and successfully apply the fix in the terminal? I always thought Fedora cared about these users, as it is one of the few distributions that has graphical upgrades to the next release.

        It is okay that Fedora always tries to adopt new technologies first, but I don’t agree that this should be done mid-release.

        Isn’t the whole point of fixed release distributions to only introduce breaking changes with a new major version so that your system is predictable? Why even bother shipping a fixed release if it works the same as a rolling release?

        • Some find that they are exposed to fewer “breaking” changes by holding back one release version. The security updates will be back-ported to the N-1 release, but the feature updates typically are not. By holding back one release version, you can get the security updates as they come out, but the feature updates only when you upgrade to the next release. It is important not to linger on an N-2 release, however, because security updates will not be back-ported that far. (And because the upgrade from N-2 to N-1 won’t be “guaranteed”-to-work/tested for long after the N release comes out.) Ideally, you should upgrade from N-2 to N-1 within a month of N being released if you are choosing not to run the latest release.

        • Matthew Phillips

          Very respectfully the issue wasn’t caused by a change at all, unless you consider making an immutable operating system such a change. The whole article is a mea culpa explaining the facts, so again very respectfully if you are going to complain about it you should at least read it and speak on the facts.

    • VInícius

      as said the article, it’s an issue that fedora can’t update the bootloader, it’s a issue that is getting fixed, bootup is to launch in the next release, no more old bootloader issues after that

  2. Dominik

    I think the problem here is that IoT is supposed to work headless. So while using an usual Fedora breaking changes are expected, for devices that could (possibly) work in remote locations, this is something very serious.

Leave a Reply


The interval between posting a comment and its appearance will be irregular so please DO NOT resend the same post repeatedly. All comments are moderated but this site is not monitored continuously so comments will not appear as soon as posted.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions