This article explains how to transition a typical container setup from Raspberry Pi OS to Fedora Linux at the example of the Traefik reverse-proxy. We start with an already setup Fedora Linux to keep this one to the point, which is getting to know fundamental differences and options when first touching down on Fedora Linux and podman.
Introduction
Traefik is a “container-native” proxy and as such integrates neatly in container based systems. It can read almost everything required to do its job from the “docker”-socket. Some configuration loads automatically while you supply the rest with container labels on the containers you operate. Traefik then binds host ports and routes traffic (HTTP, TCP, UDP) to other containers.
Podman is the container orchestrator on Fedora Linux. It is a drop-in replacement for docker.
At its heart Fedora Linux provides some fundamental differences to Raspberry Pi OS:
- It’s close to upstream with the latest kernels, security updates and features.
- Fedora Server, Silverblue, Workstation, CoreOS and IoT Edition feel pretty similar. For example, they use the same package names and they build on the same system tools.
- Fedora Linux focuses on delivering good security and features.
One disadvantage is the lack of raspi-config which leaves you reading the (very good) documentation of Raspberry Pi and doing most of those configuration tasks manually.
Orchestration Options
All in all Fedora Linux gives you a range of options for operating containers:
- podman socket + docker-compose as a drop-in replacement for docker + docker-compose
- podman + systemd
- podman play with kubernetes-like pod configuration files
- rootless podman which works for all of the above
You’ll use podman socket + docker-compose as it involves the least amount of changes of concepts and technology. That’s a good starting point. Next take it a step further and migrate to systemd units. The podman play and rootless approaches will have to wait for another time. At the end of this article you will have a Fedora Linux system with a Traefik reverse-proxy running through docker-compose or systemd and the knowledge about specifics when it comes to operating containers with Fedora Linux.
Prerequisites
The article Use Docker Compose with Podman to Orchestrate Containers on Fedora Linux introduces you to the basics of running docker-compose on Fedora Linux. Make sure to check it out!
Let me give you the gist of that article. Install the packages podman, podman-docker and docker-compose. The podman package provides the commands to run the containers. The podman-docker package provides everything needed to disguise podman as docker for legacy tooling and applications. And docker-compose is the familiar player that won’t know it’s actually talking to podman.
On Fedora IoT and Fedora CoreOS run:
sudo rpm-ostree install podman podman-docker docker-compose sudo systemctl enable podman.socket sudo systemctl reboot
On Fedora Server (ARM) run:
sudo dnf install podman podman-docker docker-compose sudo systemctl enable podman.socket sudo systemctl reboot
Make sure that the service is active and the file docker.sock exists.
sudo systemctl status podman.socket ls -la /var/run/docker.socket
Migrate to podman + docker-compose
Move Traefik from your Raspberry Pi OS to Fedora Linux. That should include an acme folder, docker-compose.yml and any traefik configuration files. I’ll grab a cup of coffee in the meantime.
Place the files under, for example, /var/srv/traefik.
$ ls -la /var/srv/traefik total 8 drwxr-xr-x. 3 root root 63 Sep 3 09:04 . drwxr-xr-x. 15 root root 204 Sep 5 10:07 .. drwx--x--x. 2 root root 23 Aug 19 17:07 acme -rw-------. 1 root root 1813 Sep 3 09:04 docker-compose.yml -rw-------. 1 root root 2178 Aug 26 13:21 traefik.toml
Change the SELinux type label for all files you are going to mount in the container with the chcon command. On Fedora Linux, SELinux restricts access to files on the host system for container mounts by default. It’s good practice to also restrict the access mode to the certificate files and configuration (possibly containing secrets) as much as possible.
sudo chcon -Rt container_file_t /var/srv/proxy/acme /var/srv/proxy/traefik.toml sudo chmod 0700 /var/srv/proxy/acme sudo chmod 0600 /var/srv/proxy/docker-compose.yml /var/srv/proxy/traefik.toml
Create a podman network named proxy. This network serves traefik to route all requests to containers exposed on ports 80 and 443.
$ sudo podman network create proxy /etc/cni/net.d/proxy.conflist
Change the docker-compose file
The minimal target Traefik compose file will look like this. All it does is bind to port 80 on your host, implement a basic catchall router, configuring the docker provider and expose the dashboard, unsecured, on port 8080.
version: '2' services: proxy: image: docker.io/library/traefik:latest labels: - "traefik.enable=true" - "traefik.http.routers.http_catchall.rule=HostRegexp(`{any:.+}`)" - "traefik.http.routers.http_catchall.entrypoints=web" - "traefik.http.middlewares.compress.compress=true" security_opt: - label=disable ports: - "80:80" - "8080:8080" volumes: - "/var/srv/traefik/traefik.toml:/etc/traefik/traefik.toml:ro" - "/var/run/docker.sock:/var/run/docker.sock" command: - "--entryPoints.web.address=:80" - "--providers.docker=true" - "--providers.docker.exposedByDefault=false" - "--log.level=INFO" - "--api.insecure=true" networks: proxy: restart: always networks: proxy: external: true
There are subtle differences to a compose file on Raspberry Pi OS. Let’s go through them step by step.
In contrast to docker, the podman orchestrator considers multiple registries when pulling down an image from the internet. A short name like traefik:latest is ambiguous. I recommend you supply the full name to an image like docker.io/library/traefik:latest. Otherwise podman will try every registry and either use the first match or spawn an interactive input request. Since docker does not feature this behavior, docker-compose is utterly confused and just errors out.
You must supply the security_opt: label=disable flag. Applications like Traefik support reading the docker.sock to gain information about your orchestration. SELinux restricts this access. To grant full, insecure access one can disable SELinux labels inside the container.
Using the rootfull docker.sock inside a podman container opens up all kinds of security concerns just as it would with a docker container. A slightly more secure approach would be to use a trusted docker-socket-proxy container, which allows you to filter the allowed API methods on the socket. The most secure option that is currently available is to use a rootless user socket. But an explanation of how to do that will have to wait for its own article.
Next start traefik through docker-compose:
sudo docker-compose -f /var/srv/proxy/docker-compose.yml up -d
Navigate to your Raspberry Pi on Port 8080 to verify that Traefik is indeed up and running.
Migrate to systemd
An alternative approach to podman-docker and docker-compose is systemd units. It’s the recommended way of orchestrating containers on Fedora Linux; if there is one. This one might be difficult to wrap your head around though. Specifying containers and their dependencies in systemd is a bit more tedious than writing a single docker-compose file. But you gain a more native system on Fedora Linux when you use systemd.
There are several ways to approach a migration from docker-compose to systemd units. We’ll focus on using the built-in command podman generate systemd.
The handy command podman generate systemd takes a snapshot of the current orchestrated containers and persists their configuration in systemd units so you can start and stop them via systemctl. Every container started with podman containers run or podman pod create converts to valid systemd units following a recommended format.
After you have created one or more containers this way, you will discover that you can simply copy-paste the unit files and alter their contents to suit your needs. There is really nothing special about this process. I advise you to use podman generate systemd from time to time though. The command incorporates a lot of knowledge about how systemd works and it periodically receives updates. The updates might change the unit file formatting in small ways that improve stability or patch bugs.
podman generate systemd
To use the podman generate systemd command from scratch you have to start the containers once by manually composing a podman containers run statement. After a few times you may as well copy the unit files and just change small aspects to fit your applications.
Let’s assume you put the labels from the above compose file into a file /var/srv/traefik/labels and move the traefik CLI options to traefik.toml. A resulting podman run command from the above container spec looks like this:
sudo podman run \ --security-opt label=disable \ --label-file /var/srv/traefik/label \ --network proxy \ --network-alias=traefik \ --name traefik \ -v "/var/srv/traefik/traefik.toml:/etc/traefik/traefik.toml:ro" \ -v "/var/run/docker.sock:/var/run/docker.sock" \ -p 80:80 \ -p 8080:8080 \ -d \ docker.io/library/traefik:latest
This will give you a single running traefik container with podman. To confirm:
sudo podman ps -a sudo podman logs -f traefik curl -v http://localhost:8080/
Now generate a systemd unit from it and place it in the /etc/systemd/system directory. Enable and start it afterwards:
$ sudo podman generate systemd --new --name traefik | sudo tee /etc/systemd/system/container-traefik.service # container-traefik.service # autogenerated by Podman 3.3.1 # Wed Sep 15 08:58:59 CEST 2021 [Unit] Description=Podman container-traefik.service Documentation=man:podman-generate-systemd(1) Wants=network-online.target After=network-online.target RequiresMountsFor=%t/containers [Service] Environment=PODMAN_SYSTEMD_UNIT=%n Restart=on-failure TimeoutStopSec=70 ExecStartPre=/bin/rm -f %t/%n.ctr-id ExecStart=/usr/bin/podman run \ --cidfile=%t/%n.ctr-id \ --sdnotify=conmon \ --cgroups=no-conmon \ --rm \ --replace \ --security-opt label=disable \ --label-file /var/srv/traefik/label \ --network proxy \ --network-alias=traefik \ --name traefik \ -v /var/srv/traefik/traefik.toml:/etc/traefik/traefik.toml:ro \ -v /var/run/docker.sock:/var/run/docker.sock \ -p 80:80 \ -p 8080:8080 \ -d \ docker.io/library/traefik:latest ExecStop=/usr/bin/podman stop --ignore --cidfile=%t/%n.ctr-id ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=%t/%n.ctr-id Type=notify NotifyAccess=all [Install] WantedBy=multi-user.target default.target
Remove the initial container required to generate this unit file. Then enable and start the unit and verify the result:
sudo podman rm -f traefik sudo systemctl enable --now container-traefik.service sudo systemctl status container-traefik.service sudo podman ps -a sudo podman logs -f traefik
The traefik container is now operated by podman + systemd and will start automatically upon boot. Congrats! To control the startup order and dependencies between your containers you now have the systemd tools at your disposal.
Where are the pods?
You built a podman+systemd structure by using manually created networks and containers glued together by systemd dependencies. You might already wonder why it’s called podman although we don’t use anything related to pods. Especially for containers with more complex dependencies like applications with multiple databases, search engines and caches, podman knows the concept of a pod. A pod is essentially a space where all containers participate and view each other as if they were running on the same host. You can also turn that construct into a systemd managed one. For now I’ll leave that as an exercise for you.
It’s really up to you if you configure a pod-based setup and use podman generate systemd on that or if you compose your containers manually with networks and everything and turn that into systemd units.
Summary
Great you made it this far! By now, you have setup a docker-compose based, simple traefik installation using podman.sock. You learned about the typical changes required when transitioning from Raspberry Pi OS to Fedora Linux when it comes to plain container orchestration. After this first success, you took it a step further and migrated the docker-compose file into a systemd unit from scratch with podman generate systemd.
Systemd creates the traefik container through podman on boot and manages its dependencies. Traefik receives its configuration through the podman.socket as a drop-in replacement to docker and communicates with the other containers through the manually-created podman network proxy.
What’s next? You might want to take a look at how to operate the container stack with rootless podman. Maybe also the rootless podman.socket + docker-compose? It’s definitely possible.
Troubleshooting
When first migrating to Fedora (IoT) from a debian-based system like Raspberry Pi OS, there are several pitfalls that you might end up falling into. Let’s make sure you get out of there with dignity.
Permission denied on an application’s configuration file:
This is most probably an SELinux issue. Check that the label on the files in question is container_file_t via ls -laZ. If not, change that with chcon -Rt container_file_t.
Permission denied on /var/run/docker.sock:
Once more, this is probably SELinux related. Make sure to start the container with –security-opt label=disable or that the socket has the label container_file_t. Unfortunately, every time the socket is recreated, its label will be reset.
Docker-compose errors out:
There might be some quirks happening between podman and docker-compose. docker-compose does not know about some of podman’s functionality. This can lead to unexpected or misleading error messages. Run docker-compose in debug mode and check the logs with journalctl -xe.
Possible errors are:
- Inconsistent container state: try to use docker-compose down first
- Ambiguous image name: make sure that you use full image names including their registry
redbeard
Methinks you might want to correctly change the file labeling rather than just a one time
(e.g.
). Alternatively, just use one of the paths already in the context database:
/home/[^/]+/\.local/share/containers/storage/volumes/[^/]*/.* all files unconfined_u:object_r:container_file_t:s0
/srv/containers(/.*)? all files system_u:object_r:container_file_t:s0
/var/lib/containers/storage/volumes/[^/]*/.* all files system_u:object_r:container_file_t:s0
/var/lib/kubernetes/pods(/.*)? all files system_u:object_r:container_file_t:s0
/var/lib/origin(/.*)? all files system_u:object_r:container_file_t:s0
/var/lib/rkt/cas(/.*)? all files system_u:object_r:container_file_t:s0
/var/srv/containers(/.*)? all files system_u:object_r:container_file_t:s0
Alexander Wellbrock
Thank you very much for the hint! I especially like
as a already labeled destination for container data which makes things rather straight forward.
Matt
I’ve run Pi-hole on Fedora IOT for almost a year now and it’s been great!