Auto-Updating containers can be very useful in some cases. Podman provides mechanisms to take care of container updates automatically. This article demonstrates how to use Podman Auto-Updates for your setups.
Podman is a daemonless Docker replacement that can handle rootfull and rootless containers. It is fully aware of SELinux and Firewalld. Furthermore, it comes pre-installed with Fedora Linux so you can start using it right away.
If Podman is not installed on your machine, use one of the following commands to install it. Select the appropriate command for your environment.
# Fedora Workstation / Server / Spins $ sudo dnf install -y podman # Fedora Silverblue, IoT, CoreOS $ rpm-ostree install podman
Podman is also available for many other Linux distributions like CentOS, Debian or Ubuntu. Please have a look at the Podman Install Instructions.
Updating the Operating System on a regular basis is somewhat mandatory to get the newest features, bug fixes, and security updates. But what about containers? These are not part of the Operating System.
If you want to update your Operating System, it can be as easy as:
$ sudo dnf update
This will not take care of the deployed containers. But why should you take care of these? If you check the content of containers, you will find the application (for example MariaDB in the docker.io/library/mariadb container) and some dependencies, including basic utilities.
Running updates for containers can be tedious and time-consuming, since you have to:
- pull the new image
- stop and remove the running container
- start the container with the new image
This procedure must be done for every container. Updating 10 containers can easily end up taking 30-40 commands that must be run.
Automating these steps will save time and ensure, that everything is up-to-date.
Podman and systemd
Podman has built-in support for systemd. This means you can start/stop/restart containers via systemd without the need of a separate daemon. The Podman Auto-Update feature requires you to have containers running via systemd. This is the only way to automatically ensure that all desired containers are running properly. Some articles like these for Bitwarden and Matrix Server already had a look at this feature. For this article, I will use an even simpler Apache httpd container.
First, start the container with the desired settings.
# Run httpd container with some custom settings $ sudo podman container run -d -t -p 80:80 --name web -v web-volume:/usr/local/apache2/htdocs/:Z docker.io/library/httpd:2.4 # Just a quick check of the container $ sudo podman container ls CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 58e5b07febdf docker.io/library/httpd:2.4 httpd-foreground 4 seconds ago Up 5 seconds ago 0.0.0.0:80->80/tcp web # Also check the named volume $ sudo podman volume ls DRIVER VOLUME NAME local web-volume
Now, set up systemd to handle the deployment. Podman will generate the necessary file.
# Generate systemd service file $ sudo podman generate systemd --new --name --files web /home/USER/container-web.service
This will generate the file container-web.service in your current directory. Review and edit the file to your liking. Here is the file contents with added newlines and formatting to improve readability.
# container-web.service [Unit] Description=Podman container-web.service Documentation=man:podman-generate-systemd(1) Wants=network.target After=network-online.target RequiresMountsFor=%t/containers [Service] Environment=PODMAN_SYSTEMD_UNIT=%n Restart=on-failure TimeoutStopSec=70 ExecStartPre=/bin/rm -f %t/container-web.pid %t/container-web.ctr-id ExecStart=/usr/bin/podman container run \ --conmon-pidfile %t/container-web.pid \ --cidfile %t/container-web.ctr-id \ --cgroups=no-conmon \ --replace \ -d \ -t \ -p 80:80 \ --name web \ -v web-volume:/usr/local/apache2/htdocs/ \ docker.io/library/httpd:2.4 ExecStop=/usr/bin/podman container stop \ --ignore \ --cidfile %t/container-web.ctr-id \ -t 10 ExecStopPost=/usr/bin/podman container rm \ --ignore \ -f \ --cidfile %t/container-web.ctr-id PIDFile=%t/container-web.pid Type=forking [Install] WantedBy=multi-user.target default.target
Now, remove the current container, copy the file to the proper systemd directory, and start/enable the service.
# Remove the temporary container $ sudo podman container rm -f web # Copy the service file $ sudo cp container-web.service /etc/systemd/system/container-web.service # Reload systemd $ sudo systemctl daemon-reload # Enable and start the service $ sudo systemctl enable --now container-web # Another quick check $ sudo podman container ls $ sudo systemctl status container-web
Please be aware, that the container can now only be managed via systemd. Starting and stopping the container with the “podman” command may interfere with systemd.
Now that the general setup is out of the way, have a look at auto-updating this container.
The first thing to look at is manual auto-updates. Sounds weird? This feature allows you to avoid the 3 steps per container, but you will have full control over the update time and date. This is very useful if you only want to update containers in a maintenance window or on the weekend.
Edit the /etc/systemd/system/container-web.service file and add the label shown below to it.
The changed file will have a section appearing like this:
...snip... ExecStart=/usr/bin/podman container run \ --conmon-pidfile %t/container-web.pid \ --cidfile %t/container-web.ctr-id \ --cgroups=no-conmon \ --replace \ -d \ -t \ -p 80:80 \ --name web \ -v web-volume:/usr/local/apache2/htdocs/ \ --label "io.containers.autoupdate=registry" \ docker.io/library/httpd:2.4 ...snip...
Now reload systemd and restart the container service to apply the changes.
# Reload systemd $ sudo systemctl daemon-reload # Restart container-web service $ sudo systemctl restart container-web
After this setup you can run a simple command to update a running instance to the latest available image for the used tag. In this example case, if a new 2.4 image is available in the registry, Podman will download the image and restart the container automatically with a single command.
# Update containers $ sudo podman auto-update
Podman also provides a systemd timer unit that enables container updates on a schedule. This can be very useful if you don’t want to handle the updates on your own. If you are running a small home server, this might be the right thing for you, so you are getting the latest updates every week or so.
Enable the systemd timer for podman as follows:
# Enable podman auto update timer unit $ sudo systemctl enable --now podman-auto-update.timer Created symlink /etc/systemd/system/timers.target.wants/podman-auto-update.timer → /usr/lib/systemd/system/podman-auto-update.timer.
Optionally, you can edit the schedule of the timer. By default, the update will run every Monday morning, which is ok for me. Edit the timer module using this command:
$ sudo systemctl edit podman-auto-update.timer
This will bring up your default editor. Changing the schedule is beyond the scope of this article but the link to systemd.timer below will help. The Demo section of Systemd Timers for Scheduling Tasks contains details as well.
That’s it. Nothing more to do. Podman will now take care of image updates and also prune old images on a schedule.
Hints & Tips
Auto-Updating seems like the perfect solution for container updates, but you should consider some things, before doing so.
- avoid using the “latest” tag, since it can include major updates
- consider using tags like “2” or “2.4”, if the image provider has them
- test auto-updates beforehand (does the container support updates without additional steps?)
- consider having backups of your Podman volumes, in case something goes sideways
- auto-updates might not be very useful for highly productive setups, where you need full control over the image version in use
- updating a container also restarts the container and prunes the old image
- occasionally check if the updates are being applied
If you take care of the above hints, you should be good to go.
Docs & Links
If you want to learn more about this topic, please check out the links below. There is a lot of useful information in the official documentation and some blogs.
- Systemd Timers for Scheduling Tasks
As you can see, without the use of additional tools, you can easily run auto-updates on Podman containers manually or on a schedule. Scheduling allows unattended updates overnight, and you will get all the latest security updates, features, and bug fixes. Some setups I have tested successfully are: MariaDB, Ghost Blog, WordPress, Gitea, Redis, and PostgreSQL.
For the auto-update you don’t have to edit the systemd unit file. Instead you just pass the label as an argument when you create the container:
and that’s it. Generate your unit file and do the rest, but you don’t need to edit anything. This is how podman know what containers to update and which ones not to.
thanks for the addition. You are 100% correct. For the sake of this article, I opted to showcase the difference between the two different systemd files. 🙂
I am new to podman and containers, and I wonder, in the web-volume syntax, what “Z:” (-v web-volume:/usr/local/apache2/htdocs/:Z) stand for ?
I cannot see this parameter in the container-web.service file
I must miss something ..
Thanks for your response.
I must have missed it during the writing. Nevermind, the “:Z” is a volume option to satisfy some SELinux needs. It will generate the proper SELinux contexts, so the container volume can be used without conflicting with SELinux. for named volumes (as in my example) this is not needed. If you intend to use path volumes (/var/foo/bar:/var/www/html) this may/will be needed.
I hope this explains it a bit. You can read more about this here: https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options
How convenient, I did not know that podman was shipping with an auto-update systemd unit. I always created my own.
One tip: if using the btrfs filesystem, then one can easily create a snapshot before updating the containers.
Another thing to note: This auto update can also be used by users running podman containers, so it is not restricted to root containers.
Really useful to know, thanks. In the case where I have a custom image that I build using the podman build command, could you suggest a workflow that will achieve a similar result?
The run command is from the custom image, so unless I provide the run command with a server provided custom image that autoupdates using some sort CI system, I don’t think it is possible.
If you refer to “have a image only on the local machine” – Yes, the label supports another option “local” (–label “io.containers.autoupdate=local \”). This way, Podman will look for a local, matching image.
How do you find out the url for the ‘docker.io’ part ? As I undertand it, here you’re providing the url for Apache (docker.io/library/httpd:2.4) ?
What if I want to auto-update navidrome : https://hub.docker.com/r/deluan/navidrome ?
Thank you for youur help
If you search for the container image (ex.: podman search navidrome), you will get the url. In most cases, the format is “docker.io//”. For official images like httpd, nginx, mariadb, you can use “library” as the organization.
Very useful article. Thanks for sharing.
I noticed 2 typos:
At “Podman will generated the” I would remove the trailing d, so it reads “generate”.
At “the file container-web service” I would add a dot, so it reads “container-web.service”.
Thanks René. I’ve made the corrections.