systemd-resolved: introduction to split DNS

Photo by Ruvim Noga on Unsplash

Fedora 33 switches the default DNS resolver to systemd-resolved. In simple terms, this means that systemd-resolved will run as a daemon. All programs wanting to translate domain names to network addresses will talk to it. This replaces the current default lookup mechanism where each program individually talks to remote servers and there is no shared cache.

If necessary, systemd-resolved will contact remote DNS servers. systemd-resolved is a “stub resolver”—it doesn’t resolve all names itself (by starting at the root of the DNS hierarchy and going down label by label), but forwards the queries to a remote server.

A single daemon handling name lookups provides significant benefits. The daemon caches answers, which speeds answers for frequently used names. The daemon remembers which servers are non-responsive, while previously each program would have to figure this out on its own after a timeout. Individual programs only talk to the daemon over a local transport and are more isolated from the network. The daemon supports fancy rules which specify which name servers should be used for which domain names—in fact, the rest of this article is about those rules.

Split DNS

Consider the scenario of a machine that is connected to two semi-trusted networks (wifi and ethernet), and also has a VPN connection to your employer. Each of those three connections has its own network interface in the kernel. And there are multiple name servers: one from a DHCP lease from the wifi hotspot, two specified by the VPN and controlled by your employer, plus some additional manually-configured name servers. Routing is the process of deciding which servers to ask for a given domain name. Do not mistake this with the process of deciding where to send network packets, which is called routing too.

The network interface is king in systemd-resolved. systemd-resolved first picks one or more interfaces which are appropriate for a given name, and then queries one of the name servers attached to that interface. This is known as “split DNS”.

There are two flavors of domains attached to a network interface: routing domains and search domains. They both specify that the given domain and any subdomains are appropriate for that interface. Search domains have the additional function that single-label names are suffixed with that search domain before being resolved. For example, a lookup for “server” is treated as a lookup for “server.example.com” if the search domain is “example.com.” In systemd-resolved config files, routing domains are prefixed with the tilde (~) character.

Specific example

Now consider a specific example: your VPN interface tun0 has a search domain private.company.com and a routing domain ~company.com. If you ask for mail.private.company.com, it is matched by both domains, so this name would be routed to tun0.

A request for www.company.com is matched by the second domain and would also go to tun0. If you ask for www, (in other words, if you specify a single-label name without any dots), the difference between routing and search domains comes into play. systemd-resolved attempts to combine the single-label name with the search domain and tries to resolve www.private.company.com on tun0.

If you have multiple interfaces with search domains, single-label names are suffixed with all search domains and resolved in parallel. For multi-label names, no suffixing is done; search and routing domains are are used to route the name to the appropriate interface. The longest match wins. When there are multiple matches of the same length on different interfaces, they are resolved in parallel.

A special case is when an interface has a routing domain ~. (a tilde for a routing domain and a dot for the root DNS label). Such an interface always matches any names, but with the shortest possible length. Any interface with a matching search or routing domain has higher priority, but the interface with ~. is used for all other names. Finally, if no routing or search domains matched, the name is routed to all interfaces that have at least one name server attached.

Lookup routing in systemd-resolved

Domain routing

This seems fairly complex, partially because of the historic names which are confusing. In actual practice it’s not as complicated as it seems.

To introspect a running system, use the resolvectl domain command. For example:

$ resolvectl domain
Global:
Link 4 (wlp4s0): ~.
Link 18 (hub0):
Link 26 (tun0): redhat.com

You can see that www would resolve as www.redhat.com. over tun0. Anything ending with redhat.com resolves over tun0. Everything else would resolve over wlp4s0 (the wireless interface). In particular, a multi-label name like www.foobar would resolve over wlp4s0, and most likely fail because there is no foobar top-level domain (yet).

Server routing

Now that you know which interface or interfaces should be queried, the server or servers to query are easy to determine. Each interface has one or more name servers configured. systemd-resolved will send queries to the first of those. If the server is offline and the request times out or if the server sends a syntactically-invalid answer (which shouldn’t happen with “normal” queries, but often becomes an issue when DNSSEC is enabled), systemd-resolved switches to the next server on the list. It will use that second server as long as it keeps responding. All servers are used in a round-robin rotation.

To introspect a running system, use the resolvectl dns command:

$ resolvectl dns
Global:
Link 4 (wlp4s0): 192.168.1.1 8.8.4.4 8.8.8.8
Link 18 (hub0):
Link 26 (tun0): 10.45.248.15 10.38.5.26

When combined with the previous listing, you know that for www.redhat.com, systemd-resolved will query 10.45.248.15, and—if it doesn’t respond—10.38.5.26. For www.google.com, systemd-resolved will query 192.168.1.1 or the two Google servers 8.8.4.4 and 8.8.8.8.

Differences from nss-dns

Before going further detail, you may ask how this differs from the previous default implementation (nss-dns). With nss-dns there is just one global list of up to three name servers and a global list of search domains (specified as nameserver and search in /etc/resolv.conf).

Each name to query is sent to the first name server. If it doesn’t respond, the same query is sent to the second name server, and so on. systemd-resolved implements split-DNS and remembers which servers are currently considered active.

For single-label names, the query is performed with each of the the search domains suffixed. This is the same with systemd-resolved. For multi-label names, a query for the unsuffixed name is performed first, and if that fails, a query for the name suffixed by each of the search domains in turn is performed. systemd-resolved doesn’t do that last step; it only suffixes single-label names.

A second difference is that with nss-dns, this module is loaded into each process. The process itself communicates with remote servers and implements the full DNS stack internally. With systemd-resolved, the nss-resolve module is loaded into the process, but it only forwards the query to systemd-resolved over a local transport (D-Bus) and doesn’t do any work itself. The systemd-resolved process is heavily sandboxed using systemd service features.

The third difference is that with systemd-resolved all state is dynamic and can be queried and updated using D-Bus calls. This allows very strong integration with other daemons or graphical interfaces.

Configuring systemd-resolved

So far, this article talked about servers and the routing of domains without explaining how to configure them. systemd-resolved has a configuration file (/etc/systemd/resolv.conf) where you specify name servers with DNS= and routing or search domains with Domains= (routing domains with ~, search domains without). This corresponds to the Global: lists in the two listings above.

In this article’s examples, both lists are empty. Most of the time configuration is attached to specific interfaces, and “global” configuration is not very useful. Interfaces come and go and it isn’t terribly smart to contact servers on an interface which is down. As soon as you create a VPN connection, you want to use the servers configured for that connection to resolve names, and as soon as the connection goes down, you want to stop.

How does then systemd-resolved acquire the configuration for each interface? This happens dynamically, with the network management service pushing this configuration over D-Bus into systemd-resolved. The default in Fedora is NetworkManager and it has very good integration with systemd-resolved. Alternatives like systemd’s own systemd-networkd implement similar functionality. But the interface is open and other programs can do the appropriate D-Bus calls.

Alternatively, resolvectl can be used for this (it is just a wrapper around the D-Bus API). Finally, resolvconf provides similar functionality in a form compatible with a tool in Debian with the same name.

Scenario: Local connection more trusted than VPN

The important thing is that in the common scenario, systemd-resolved follows the configuration specified by other tools, in particular NetworkManager. So to understand how systemd-resolved names, you need to see what NetworkManager tells it to do. Normally NM will tell systemd-resolved to use the name servers and search domains received in a DHCP lease on some interface. For example, look at the source of configuration for the two listings shown above:

There are two connections: “Parkinson” wifi and “Brno (BRQ)” VPN. In the first panel DNS:Automatic is enabled, which means that the DNS server received as part of the DHCP lease (192.168.1.1) is passed to systemd-resolved. Additionally. 8.8.4.4 and 8.8.8.8 are listed as alternative name servers. This configuration is useful if you want to resolve the names of other machines in the local network, which 192.168.1.1 provides. Unfortunately the hotspot DNS server occasionally gets stuck, and the other two servers provide backup when that happens.

The second panel is similar, but doesn’t provide any special configuration. NetworkManager combines routing domains for a given connection from DHCP, SLAAC RDNSS, and VPN, and finally manual configuration and forward this to systemd-resolved. This is the source of the search domain redhat.com in the listing above.

There is an important difference between the two interfaces though: in the second panel, “Use this connection only for resources on its network” is checked. This tells NetworkManager to tell systemd-resolved to only use this interface for names under the search domain received as part of the lease (Link 26 (tun0): redhat.com in the first listing above). In the first panel, this checkbox is unchecked, and NetworkManager tells systemd-resolved to use this interface for all other names (Link 4 (wlp4s0): ~.). This effectively means that the wireless connection is more trusted.

Scenario: VPN more trusted than local network

In a different scenario, a VPN would be more trusted than the local network and the domain routing configuration reversed. If a VPN without “Use this connection only for resources on its network” is active, NetworkManager tells systemd-resolved to attach the default routing domain to this interface. After unchecking the checkbox and restarting the VPN connection:

$ resolvectl domain
Global:
Link 4 (wlp4s0):
Link 18 (hub0):
Link 28 (tun0): ~. redhat.com
$ resolvectl dns
Global:
Link 4 (wlp4s0):
Link 18 (hub0):
Link 28 (tun0): 10.45.248.15 10.38.5.26

Now all domain names are routed to the VPN. The network management daemon controls systemd-resolved and the user controls the network management daemon.

Additional systemd-resolved functionality

As mentioned before, systemd-resolved provides a common name lookup mechanism for all programs running on the machine. Right now the effect is limited: shared resolver and cache and split DNS (the lookup routing logic described above). systemd-resolved provides additional resolution mechanisms beyond the traditional unicast DNS. These are the local resolution protocols MulticastDNS and LLMNR, and an additional remote transport DNS-over-TLS.

Fedora 33 does not enable MulticastDNS and DNS-over-TLS in systemd-resolved. MulticastDNS is implemented by nss-mdns4_minimal and Avahi. Future Fedora releases may enable these as the upstream project improves support.

Implementing this all in a single daemon which has runtime state allows smart behaviour: DNS-over-TLS may be enabled in opportunistic mode, with automatic fallback to classic DNS if the remote server does not support it. Without the daemon which can contain complex logic and runtime state this would be much harder. When enabled, those additional features will apply to all programs on the system.

There is more to systemd-resolved: in particular LLMNR and DNSSEC, which only received brief mention here. A future article will explore those subjects.

For System Administrators Using Software

51 Comments

  1. Mark

    I haven’t had good experiences with systemd-resolved in the past and am not too thrilled Fedora is moving to it. Hopefully it has improved significantly since I last used it. I’ve had a lot of issues resolving internal hostnames over a VPN with split DNS (ie: git.mycompany.com resolves to a server that redirects to our website publicly, but resolves to our internal github server over the VPN). I’ve also had bad experiences with it leaking DNS queries all over the place. Also, RIP nslookup. systemd-resolved basically makes it useless. To troubleshoot DNS lookups now, you have to learn new systemd specific tools and use other systemd specific tools to view systemd-resolved logs. I can’t help but feel systemd-resolved is a solution in search of a problem.

    • issues resolving internal hostnames over a VPN with split DNS (ie: git.mycompany.com resolves to a server that redirects to our website publicly, but resolves to our internal github server over the VPN)

      If you tell resolved to resolve

      mycompany.com

      on the VPN link, this should “just work” when the VPN is enabled. Make sure that the VPN connection a) configures at least one dns server, b) configures a routing or search domain.

      I’ve also had bad experiences with it leaking DNS queries all over the place

      I’m sad to hear that. In general resolved is about not leaking queries randomly, but using a very predictable rules for where to look up what. Maybe if you provide details, we could figure out what went wrong.

      Also, RIP nslookup

      You can give an explicit server name to

      nslookup

      . With split dns you already had to this to some extent, since

      /etc/resolv.conf

      had no notion of split dns and query routing. Hopefully tools like

      nslookup

      will just learn to look at

      /run/systemd/system/resolv.conf

      when appropriate.

      To troubleshoot DNS lookups now, you have to learn new systemd specific tools and use other systemd specific tools to view

      systemd-resolved

      logs.

      Well, it’s 2020, you can’t really admin a linux machine without knowing

      journalctl

      . So the tool to look up logs is something that you probably already know quite well.

  2. Tomasz

    Thanks for superb explanation of systemd-resolved mistery! 😉

  3. Chris

    How does this work together with unbound

    • The functionality of both projects overlaps partially: both provide a caching stub resolver with DNSSEC validation. That is all systemd-resolved is trying to do, while unbound has additional functionality, in particular record filtering and a full recursive resolver. Systemd-resolved on the other hand provides a closer local implementation mechanism because it plugs directly into the nss stack, while unbound would be used with nss-dns like a normal dns server. So even though in theory both could be used in parallel, normally you would pick one or the other.

      • Petr

        Unbound is far superior to systemd-resolved. If you need good DNSSEC support, where you can verify Its working, use unbound. If you can use unbound-control, dynamic control Is easy.

        Unbound-host -D or delv don’t work with systemd-resolved.

        DNSSEC support Is Broken in resolved, because authors don’t like it. JUST note sentence: often becomes an issue when DNSSEC is enabled. No explanation what it does. Instead od proper negotiation, it blocks secure DNS by default.

  4. Venkata Subramani

    Yes, this will definitely improve DNS resolution job.

  5. David Both

    So this appears to be client side name resolution and does not replace BIND or other services of that nature for DNS servers. Is that correct?

    • ganguin

      I actually have an issue when using a VPN. Priority is given to the VPN, network is done by DHCP.

      When the VPN is disconnected everything works.

      When the VPN is connected resolved puts both the local DNS server and the VPN DNS server in resolve.conf. randomly choosing the order, therefore randomly failing, because when the VPN is up, the local network is not accessible anymore (with a policy routing rule).

      What am I doing wrong?

  6. xm4n

    hmmm, …. seems like systemd is feature-creeping on territories that it shouldn’t be. If it is true that systemd-resolve is the standard for Fedora 33, would there be a way to DISABLE it?
    I rather go with resolve.conf. I don’t need systemd to be touching other components of my system.

  7. Jake

    Using Fedora 33, the networkmanager-openconnect-gnome that used to work without changing any configuration, has stopped working.

    This article helps me make some sense of why, but I’m struggling with what to do to fix it.

    I want my be able to resolve https://something.something.corp.local

    Other distros that use resolved have given me similar issues with various workarounds depending on the distro to get .local addresses resolving. This is the most I’ve understood about what is happening, but as I said, I don’t know the “correct” way to resolve this.

    Any help would be much appreciated.

    • Alex D

      Just disable the darn thing , remove the /etc/resolv.conf file which is symlinked to the systemd stub resolver probably somewhere in /var/run/systemd , configure network-manager to update /etc/resolv.conf itself , check this link even if targeted for ubuntu , the steps should be the same https://gist.github.com/zoilomora/f7d264cefbb589f3f1b1fc2cea2c844c

    • .local is special because RFC6762 reserves this domain for exclusive MulticastDNS use. To make resolved resolve it using the classic DNS using some local server, it needs to be declared as a search or routing domain. See https://www.freedesktop.org/software/systemd/man/systemd-resolved.service.html#Protocols%20and%20Routing for a description of how this is supposed to work.

      • Jake

        #!/bin/bash

        /etc/NetworkManager/dispatcher.d/90-mycompany-vpn.sh

        IF=$1
        STATUS=$2
        if [ “$IF” == “vpn0” ]
        then
        case “$2” in
        up)
        resolvectl domain vpn0 mycompany.com corp.local test.local
        ;;
        *)
        ;;
        esac
        fi

        Using the above (https://www.techytalk.info/start-script-on-network-manager-successful-connection/) and learning a little more about NetworkManager and systemd-resolved along the way, I am using this script to add the .local stuff to my domain routes.

        • Pranith Kumar K

          Thanks a lot for your script! After looking at this I got the idea that we can add this information for the VPN using nm-connection-editor.

          Goto the VPN connection, in the IPv4/IPv6 settings, In ‘search domains’ field you can paste these things and it is working as expected whenever the interface is enabled.

          This is for people who don’t want to use a script.

  8. Anon Ymous

    Seriously, people. Instead of so many negative comments by people who think it is cool to complain about every article, how about less sarcastic complaining and more learning something. fedoramagazine . org is not supposed to be an outlet for complainers, it is supposed to be a place to learn. It is impossible to learn when doing childish sarcastic complaining. At the very least, shut up till you try Fedora 33 for a few months-, THEN your opinion might count.

    • Alexandru Duzsardi

      I agree, but systemd-resolved did break openvpn (vpn dns resolution in general) , we don’t just disable things for fun.
      Anyhow, reading some bug reports seems to be a network-manager systemd-resolved interoperability when using the network-manager openvpn plugin.

      • Michael Catanzaro

        In fact, when I proposed this systemd-resolved change, my primary goal was to fix OpenVPN. I would never have proposed this change had I not noticed that my VPNs were not working properly without it. See: https://fedoraproject.org/wiki/Changes/systemd-resolved#Benefit_to_Fedora. 70% of the Benefit to Fedora section discusses split DNS with VPNs. The shared DNS cache is useful even for people who don’t use VPNs, but otherwise all the benefit is for VPN users.

        I’m not aware of any complaints related to OpenVPN; the first I’ve heard about this problem is your comment here. We did have one bug report related to OpenConnect that took some time to solve, but it has been — excuse me — resolved 😉 and we only know of one total affected user, so presumably it was fine for most everyone else. So if you’re having trouble with OpenVPN, consider reporting a bug, because otherwise developers have no way to know there’s a problem.

        • Alex D

          I can’t really say it’s a Fedora issue, but this git repository (https://github.com/jonathanio/update-systemd-resolved) has the issues outlined

          We do have a mix of a distributions in our organization, Fedora, Ubuntu, Mint, OpenSuse,Elementary and maybe a couple of other. I do know this blog post is about Fedora 33 , but all of the ones mentioned above use systemd-resolved as the default dns stub resolver and we do have the issues described in the above mentioned github repository.

          Using the command line to connect to openvpn servers and using that script as the up and down directives in the vpn configuration file does work as expected on all linux variants.

          Do you know if there’s a specific version of systemd and NetworkManager that should have problems sorted out ?

        • I came here specifically to fix a VPN issue. We use a Palo Alto VPN and resolved seems to be treating the search domain provided by DHCP as a routing domain, but we have multiple domains so now when I try to access the secondary domain using a FQDN it has no idea how to route the DNS request so it just fails. I ran a command to add that domain and it is working now, but I see no way to persist the setting. I am hoping this is a documentation issue.

          resolvectl domain vpn0 ‘~domain.com’ ‘~otherdomain.com’

          This fixed it ^, but again, I don’t want to have to run that command whenever I connect.

          • Replying to my own comment, but I fixed the issue. It seems my issue is that the nm-connection-editor is no longer exposed via the gui. Via the default nm applet in Gnome you don’t have per-interface options for search domains. If you manually launch nm-connection-editor you can add them and it works as expected.

  9. A. Thanks for the great write up – I have turned off resolved on my system after fighting with it for a while, but that was two releases ago (Ubuntu). I’ll try to reenable it and see if my problems are now solved due to fixed and my better understanding of what is going on.
    B. Your said “For multi-label names, no suffixing is done” – that seems incorrect, and if true, is a serious problem: I’d expect a resolver to lookup all names, regardless of multi-label or not, and perform suffixing for names that get NXDOMAIN results. This process (that is what nss-dns did AFAIK) makes it possible to write short names in my local network DNS stub resolver (for example, my router is simply named “router” on the network) and also have multi-label searchable names – like in your example, the network admin might specify the search domain as “mycompany.com” and expect users to type “www.private” and “www.public”.

    • I have turned off resolved on my system after fighting with it for a while, but that was two releases ago (Ubuntu).

      Ubuntu sets up resolved in a different fashion: instead of using the nss-resolve plugin to let processes communicate directly with resolved over D-Bus, it configures resolved as a server in /etc/resolv.conf, so that nss-dns is still used. But because nss-dns and /etc/resolv.conf have no idea about split DNS, the effects are different. nss-dns just uses normal DNS protocol, while the D-Bus protocol allows additional metadata, so in some cases it can return smarter results. So please don’t assume that the results with the Ubuntu setup will be the same as the Fedora setup. If some queries are routed to the wrong place, feel free to open a bug.

  10. Muayyad Saleh Alsadi

    How to set caching features in systemd-resolved?

  11. Ticked Off

    Why does this not automatically update the DNS servers it has cached when you change them in Network Manager? I had to change Nics and this thing left me dead in the water with no way to even search for a solution becuase it nothing in this stupid thing updates automatically.

    Also congratulations for killing Myth TV with this nonsense. Looking forward to the systemd-mythTV fix.

    • Why does this not automatically update the DNS servers it has cached when you change them in Network Manager?

      Quoting the text above: “How does then systemd-resolved acquire the configuration for each interface? This happens dynamically, with the network management service pushing this configuration over D-Bus into systemd-resolved”. So the general answer is that this is up to NetworkManager. More specifically, NetworkManager will disable the old servers in systemd-resolved when it takes the interface down and add the new ones when bringing up the interface. This means that after name server configuration is changed in NetworkManager config, the interface needs to be reenabled for the config to take effect.

      MythTV

      I’m not at all familiar with MythTV, but I’m sure any problems can be fixed, once the maintainers have been notified about the bug.

  12. Piotr Dobrogost

    I’ve been going through similar configuration but using systemd-networkd instead of NetworkManager. I described what should be done step by step in such a scenario in the question (and answer) titled “How to configure systemd-resolved and systemd-networkd to use local DNS server for resolving local domains and remote DNS server for remote domains?” (https://unix.stackexchange.com/q/442598/5355).
    Also for people using OpenVPN but not using NetworkManager I highly recommend https://github.com/jonathanio/update-systemd-resolved as a way to update systemd-resolved on VPN link going up/down.

  13. no mDNS in Fedora33? so, no Avahi in fedora 33? Why?

    • As the article states, “MulticastDNS is implemented by nss-mdns4_minimal and Avahi.” These aren’t implemented in systemd-resolved (at least not yet), and in Fedora 33 they’ll continue to be handled as they are now. In the future, if systemd-resolved does implement mDNS, Fedora could consider changing the default handler. (I also use mDNS and it is working as usual on my local network from this Fedora 33 Beta box.)

      • Michael Catanzaro

        Well systemd-resolved actually does support mDNS, and we originally planned to use it for mDNS resolving while continuing to use avahi for mDNS responding. But mDNS resolving was not working as expected out-of-the-box. avahi is more mature, so it seemed better to stick with that for now.

  14. I find systemd-resolved’s split DNS option to be very useful. I’m glad that Fedora is switching to it by default. It is especially useful in combination with systemd-networkd and wireguard to route some non-public DNS queries over the encrypted wireguard VPN. Below are my config files as an example for anyone who may want to try to do something similar. Again, I think this is a wonderful/invaluable feature. Kudos to Fedora for providing it.

    [/etc/systemd]# cat resolved.conf.d/custom.conf
    [Resolve]
    DNS=8.8.8.8
    DNS=8.8.4.4
    FallbackDNS=
    Domains=~.
    LLMNR=no
    MulticastDNS=no
    DNSOverTLS=yes
    DNSSEC=allow-downgrade

    [/etc/systemd]# cat network/20-wg0.netdev
    [NetDev]
    Name=wg0
    Kind=wireguard

    [WireGuard]
    PrivateKey=XXX

    [WireGuardPeer]
    PublicKey=YYY
    Endpoint=146.163.0.1:9
    AllowedIPs=10.0.0.0/8,146.163.0.0/16,192.168.0.0/16

    [/etc/systemd]# cat network/25-wg0.network
    [Match]
    Name=wg0

    [Network]
    LinkLocalAddressing=0
    Domains=~siue.edu
    DNSOverTLS=no
    DNSSEC=no
    DNS=146.163.252.126
    DNS=146.163.252.127
    Address=172.16.0.1/16
    Address=172.17.0.1/16
    Address=172.18.0.1/16

    [Route]
    Destination=10.0.0.0/8
    Gateway=172.16.255.254

    [Route]
    Destination=146.163.0.0/16
    Gateway=172.17.255.254

    [Route]
    Destination=192.168.0.0/16
    Gateway=172.18.255.254

    [/etc/systemd]# cat network/80-enp2s0.network
    [Match]
    Name=enp2s0

    [Network]
    LinkLocalAddressing=0
    Address=172.31.0.1/24
    Gateway=172.31.0.254

    [Route]
    Destination=146.163.0.1
    Gateway=172.31.0.254

    P.S. I do find that there are a few things that google’s DNS servers don’t seem to resolve (e.g. amazon.com and duolingo.com addresses). For the few things that I’ve found such problems with, I’ve been able to work around the issue by using Firefox’s DoH to connect to NextDNS (I use Chrome normally and switch to Firefox for the things that don’t work with Chrome/GoogleDNS).

    • Alex D

      I’m sure it works well with systemd-networkd , unfortunately on the desktop we all use Network-Manager , and from what i can see … these two don’t have everything sorted out just yet.

      So before pushing systemd-resolved and systemd-networkd , maybe the guys at RedHat/Fedora and Debian and Canonical should have thought first resolving the issues with the interoperability, either fix Network-Manager so that bugs like DNS leaking doesn’t happen over VPN, domains that should be resolved over the VPN connection would work as expected and other similar issues, or make Network-Manager a compatible GUI for systemd-networkd , or just provide an alternative to Network-Manager as desktop application for systemd-networkd

      I think these are the issues , not that systemd{-networkd, -resolved} are bad but these were pushed without being tested enough, and they might even be working very well on a server where we do everything by editing configuration files, but not so much on the Desktop

      • Zbigniew Jędrzejewski-Szmek

        The change in F33 has nothing to do with systemd-networkd: it’s not being “pushed”, it is still disabled by default. There is no plan to make NetworkManager or something else a wrapper around systemd-networkd. They have very different philosophy, so that’s unlikely to be pursued.

        Also, the change in F33 is precisely about avoiding the situation where DNS leaking happens. This article didn’t go into the details of this, but in the previous setup DNS “leaks” were essentially unavoidable, because server priority depended on the order in which interfaces where brought up. The change page [https://fedoraproject.org/wiki/Changes/systemd-resolved#Split_DNS] describes this pretty well.

        The final rules of DNS resolution depend on the interaction between systemd-resolved, systemd-networkd (when used), NetworkManager, NetworkManager plugins, VPN connection configuration, nss module configuration, avahi, etc. So it’s likely that we will not get all the corner cases immediately correct. Enabling this in F33 has the goal of letting the developers find the cases there were not covered. So please report when something doesn’t work, but in a way that allows us to fix things. Asserting that “it’s broken” and “[nameless] guys at RedHat/Fedora and Debian and Canonical should have thought first” and so on is not helping.

    • Just a follow-up for anyone who may find my above comment in the future — I found out why some DNS names were not resolving in my setup. DNSSEC is completely broken. Even when in opportunistic mode (i.e. “DNSSEC=allow-downgrade”). Setting DNSSEC=no has resolved the problem that I mentioned at the end of my above comment.

      Fedora 33 System-Wide Change proposal

  15. John Cash

    Just for fun typed it in:

    [liveuser@localhost-live ~]$ resolvectl domain
    Failed to get global data: Could not activate remote peer.

    • Not sure what the issue is on the bootable Live system you’re testing on, but it works fine on my updated Fedora 33 (Beta+) system here:

      $ resolvectl domain
      Global:
      Link 2 (eno1): ~.
      Link 3 (crc):
      Link 4 (crc-nic):
      Link 5 (virbr0):
      Link 6 (virbr0-nic):
  16. Sergey

    Only 2 questions:
    1. As far as I remember dnsmasq could also cache dns records.
    2. How does the systemd-resolved link to DNS-over-HTTP ?

    • As far as I remember dnsmasq could also cache dns records.

      Yes. dnsmasq provides a lot of similar functionality. The way that integration with the rest of the system happens is different though: resolved allows everything to be configured over dbus, which is good for desktop integration. resolved also uses a custom nss module so processes which resolve names don’t do it using a network protocol, but instead communicate over dbus or varlink (a unix socket) locally. This allows us to sandbox things better.

      How does the systemd-resolved link to DNS-over-HTTP ?

      Right now not at all. It supports DNS-over-TLS though. DoH has been requested (https://github.com/systemd/systemd/issues/8639). I expect we’ll add support in the not-too-distant feature, but nobody seems to be working on that right now.

      • Petr Menšík

        dnsmasq can offer quite similar functionality. It is not fully configured over d-bus, but important bits are configured already from NetworkManager. It handles also split DNS, just set dns=dnsmasq in NetworkManager.conf. Try it yourself.

        It does not require any nss plugin, which introduces mostly only regressions. Instead, it leaves old files database to be used first, followed by DNS. Like it always did and how it always worked. Advantages of nss_resolve are not very clear to me. Especially because it changes search order for relative names including dot.

        Introducing new proprietary protocol were done ages before by lwres plugin. It never got used much and I think resolve plugin does not offer anything better. Communication via localhost is still provided, I don’t see any advantage in a new protocol.

  17. Antonio Dias

    I’m giving systemd-resolved a try (coming from an unbound installation, working flawless for my use case) and I’m trying to compensate on my station for a flawed VPN configuration on server side.

    I’m using a /etc/NetworkManager/dispatcher.d script to run “resolvectl dns” and “resolvectl domain” to add the DNS servers and domain that are behind the VPN connection. That is working but I’m wondering if there is a better systemd-resolved ish way to solve that kind of problem (the VPN server do no send the DNS servers addresses nor the domains they serve).

    Congratulates for the great article by the way.

    • NM allows dns servers to be set per connection (even graphically, as shown in the example in the article), and NM will forward those settings to resolved. There is also support for search domains, through commandline/configfile settings. So it should not be necessary to call ‘resolvectl dns’ or ‘resolvectl domain’. See https://wiki.gnome.org/Projects/NetworkManager/DNS for some more documentation.

  18. champtar

    Thanks a lot, finally split DNS working out of the box ! I’m constantly connected to 2/3 VPNs, so having working DNS is really nice (even if split DNS is evil and should have never been invented)

  19. Peter

    This change breaks IPSec VPN clients like Strongswan. The client or NetworkManger doesn’t create a VPN interface but uses filters, so resolved can’t send the dns query to the right interface.

  20. zeten30

    It works ‘ok’ in most cases, but brakes your reverse DNS resolution on VPN when VPN is used only for resources in it’s network. VPN -> mycompany.net resolves A/AAAA records but you’re not able to resolv PTR records as they go to your default DNS (not VPN). Workaround is to set resolvectl domain “~.” (instead of mycompany.net) . But all your DNS goes to VPN:( so no DNS split at the end. systemd-resolved should read routing table and resolve reverse zones properly VPN range -> VPN DNS, other ranges -. default connection DNS.

    • cretin

      Very clever remark. I would also be interested in them solving the reverse dns.
      Otherwise I must say that this article cleared for me a lot of things that I previously hadn’t understood, so I’m happy about that 🙂

    • Petr Menšík

      Reverse dns can be solved just like forward domains. Use ~10.in-addr.arpa. to direct all 10.0.0.0/8 hosts to given connection servers. arpaname from bind-utils can help you with conversion of ip address to reverse name. Then delete some parts on the start, depending on network mask.

      It should work if you specify then in nmcli c edit vpn-name, then set dns-search ~10.in-addr.arpa.

      It should work both with dnsmasq, unbound and systemd-resolved.

  21. cretin

    One question regarding Fedora 32, where I’m seeing that resolved-systemd is not being used.
    I gather that split-dns cannot be achieved without a third party tool like dnsmasq or powerdns. Is this correct?

    Thank you!

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions