Reconfiguring virtual machines with Cockpit

A virtual machine is created under certain assumptions about required memory, data storage, processing capacity, and so on. All too often, these assumptions need to be adjusted based on actual use practices, and additional configuration work by system administrator is required. Cockpit greatly simplifies and facilitates this work, especially for the Fedora Server editions, that themselves do not include a graphical interface.

Why Cockpit

Cockpit is not only a tool to create a virtual machine as  Karlis Kavacis and Paul W. Frields decribed some time ago. The authors showed to what extent Cockpit simplifies and facilitates this task. In the same way Cockpit is also extremely useful to manage a virtual machine after creation. Cockpit saves the memorization of the complex parameter zoo of e.g. virsh, the CLI management interface. It saves long typing on the keyboard including the correction of accidental typos. But even those who ‘speak virsh‘ fluently can benefit from Cockpit. It automatically provides a lot of status information about configuration and resource usage, which you would otherwise have to collect manually. This information is always present, so there is no need to keep this information in mind. Server administrators especially benefit from this tool, since Fedora servers do not include a graphical user interface out of the box.

The administrator only needs a web browser and thus does not need any special installation on the desktop. And it is available for any desktop OS. This is the significant distinction from virt-manager, which can be installed on Linux desktops and provides a graphical interface via ssh.

Manage virtual machines

In your local browser, open Cockpit on the host system using port 9090, e.g. https://example.com:9090. Log in either as root or with your administrative account. Select Virtual Machines in the left navigation column.

If there is no tab Virtual Machines, the corresponding Cockpit module, “cockpit-machines”, is not installed yet. In Cockpit, select Applications and install the component Machines.

The Virtual Machines overview page lists, in the central area, all virtual machines installed along with their current state. In the example above, there are 3 virtual machines, 2 of them running. The most frequently executed action, shutdown or launch, is directly accessible as a button. Additional commonly needed functions are offered in a drop-down menu (three vertical dots) to the right of it. It even offers a (simple) migration to another machine.

Libvirt structures the virtual machine files into storage pools. At the top left of the central content area is a box that shows the existence of 2 storage pools on the left and indicates 2 of them as active (up) and no pool inactive (down) on the right. By convention, the distribution files used to create virtual machines are stored in boot. The pool images, on the other hand, contains the disk images of the currently installed virtual machines. Later we will use this feature to create special virtual disks.

Manage a virtual machines basic host runtime environment

If you selecte a virtual machine’s name, a details page opens. In the top half it shows the basic runtime parameter along with a terminal window into the virtual machine.

Some of the parameters displayed are mostly informative, e.g. the hypervisor details, or you will hardly ever want to change them, e.g. the CPU type. These things can become useful when it comes to diagnosing issues.

More interesting is the option to quickly enable and disable auto-start on host reboot [1]. An example would be a (inactive) backup VM that can take over when problems with the main VM arise. A kind of ‘cold-standby’, maybe the only possibility in case of tight CPU and RAM resources on a machine. The action is taken immediately and is effective (on the next host reboot), no need for an explicit submit.

Very useful in everyday operations are the real-time information about the resource consumption for CPU and RAM [2] and the simultaneous ability to adjust these parameters [3]. Although, you will need to reboot the VM to make these changes or have them take effect.

Especially handy is the terminal window into the virtual machine. An uncomplicated way to quickly perform a CLI operation in the graphical interface – the best of both worlds.

Manage a virtual machines storage

A virtual machine usually uses a (virtual) disk provided as a disk image file in the Libvirt pool images (/var/lib/libvirt/images), either in qcow2 or raw format. Managing storage means manipulation of the respective file.

Increase a virtual disk

Cockpit is powerful but not (yet) perfect. The current version does not provide a graphical tool for adjusting a virtual disk. The system administrator still has to rely on CLI tools like qemu-img or the virt-* utilities of guestsfs-tools. Fortunately, the CLI tool is avaliable to increase the virtual disk image while the VM is stopped. Afterwards, we can start the VM again and use the VM’s Cockpit to reconfigure the storage.

As an example, using the vm2 VM Cockpit from above, display the current configuration of storage. Open http://vm2:9090 (again, adjust the address as appropriate) in the browser, log in, and select Storage from the left navigation column.

In the Filesystems section of the central content area [1] there are 2 file systems: root of 11 GiB and boot of 1 GiB. Additional information is in the right column. The Devices section indicates a Volume group of 11 GiB [2]. This implies that the root file system completely fills this Volume group. The Drives section lists a virtual disk of 12 GiB [3]. This implies that this disk is completely filled with the two file systems.

After shutting down vm2, the virtual disk is increased by 20 GiB to a total of 32 GiB in the host’s terminal.

[…]# qemu-img resize /var/lib/libvirt/images/vm2.qcow2 +20G
Image resized. 
[…]# qemu-img info /var/lib/libvirt/images/vm2.qcow2
file format: qcow2 
virtual size: 32 GiB (34359738368 bytes) 
disk size: 2.01 GiB 

As the info shows, the virtual disk size is now 32 GiB. The file still occupies only 2.01 GiB in the directory, due to the qcow2 dynamic file format.

Reconfigure an increased virtual disk

After starting vm2 again the Storage page now shows, unsurprisingly, a disk of 32 GiB, but nothing else changed. So the disk now has 20 GiB unallocated space.

Fedora Servers default storage setup is based on the concept of “segmenting storage” to facilitate system administration, prevent spillover of errors, and others. For details, see the Fedora Server Installation Guide. Therefore, if you need additional space for an application or service, e.g. web service, you would create a separate Logical volume in Fedora Server, following that rationale.

You have three options now:

1. Enlarge the existing partition and Volume group to add logical Volume as needed. That’s the way of Fedora Server’s default partitioning scheme. Unfortunately, you can’t enlarge partition and VG using Cockpit but have to use the terminal CLI in a one time action. Later, you can use to add or enlarge Logical volumes as needed using Cockpit.
2. Avoid using the terminal and create an additional partition and Volume Group using just Cockpit. As with 1. you can use Cockpit again to add or enlarge Logical volumes as needed.
3. Ignore the Fedora Server storage rationale and extend just everything to its maximal size.

Alternative 2 is used here. It is just as committed to the Fedora Server rationale as alternative 1, if not even stricter. It is, with Cockpit, the more comfortable variant and in the end implements the same actions, i.e. creating a Logical volume with file system and mounting it at a fitting position in the directory tree. Alternative 3 is conceptually the worst of all.

On the vm2 Cockkpit Storage page select the 32 Gib virtual disk and in the new page that opens click on the “Create partition” button right below the /dev/vda2 partition [1].

The only needed change is the Type to “No filesystem” [2] and select Create partition. It takes some time before a third partition is displayed in the list.

Go back to the Storage page. At the top of the right column, activate the menu icon next to Devices and select “Create volume group”.

Enter a Name for the volume group in the form that opens. The name “fedora_user” is used in the example as analogous to “fedora_fedora” of the system area. Select the newly created partition /dev/vda3, and select Create. The form closes and on the Storage page you find a new device, the “fedora_user” Volume group of 20.0 GiB.

Setup separated disk space for user data

As already mentioned, in compliance with the Fedora Server storage concept, a data area separate from system data should contain the service and user data. One File System Hierarchy-compatible way is to mount the /srv directory in a separate Logical Volume and store, therein, the respective data for all services. Another way is to mount a well-known dedicated application directory into its own Logical Volume. For example, in Fedora Server, PostgreSQL stores all its data into “/var/log/pgsql” or Apache httpd in “/var/www”. These directories are each mounted in their own Logical Volumes, preferably before the software gets installed.

As an example, httpd data is installed into a separate logical volume. Just in case the software is already installed, stop httpd and move the data temporarily out of the way. The directory “/var/www” should be empty.

The Storage page of the virtual machine (vm2) displays the Devices section in the top right column, and within the volume group “fedora_user” that was configured earlier. Selecting “fedora_user” opens the volume group configuration page. Below the box with details of the volume group is a list of the existing logical volumes of this volume group. It provides a link Create new logical volume to the right of the title, which opens a configuration form.

Replace the preset name by a descriptive term, here e.g. “webservice”, and choose a suitable size. Due to the dynamic memory allocation of the qcow2 format you may choose the size definitely a bit generous. However, a later extension with Cockpit is very easy to perform.

With the Purpose entry, the form offers the option to create a pool for “thinly provisioned volumes”, which in turn accommodates logical volumes allocating space dynamically. This does not make sense here, because qcow2 disk image files already provide for a kind of thin provisioning.

After creating the logical volume there will be an entry in the list “Logical volumes”. Expand the menu item using the “>” icon. You will find a Format Button. The associated form allows you to create and mount a file system in one pass.

You just have to provide a (descriptive) name and the mount point. After formatting, which takes some time you will find the volume group mounted and ready to use.

Cockpit handled all the various configuration steps involved, on its own. This saves the system administrator quite a bit of effort. It even includes creating the mount directory if it does not already exist. At the end of the day, Cockpit makes it very easy for the system administrator to implement the elaborate storage concept of Fedora Server.

Now is the time to install the appropriate programs. Software provided by Fedora distribution as rpm installs into the appropriate system directories via dnf tool. External software should install in either “/opt” or “/usr/local” to keep it separated from distribution files to prevent any potential mutual interference. In accordance with the storage rationale, you should also create a separate logical volume and mount it under “/opt”. An example of this would be the Wildfly application server, which stores program, configuration and possibly data in a common place.

Add a virtual disk

Expanding the virtual (system) disk is not the only way to provide additional storage, nor is it necessarily the best way. In the case of virtual disk images, creating a dedicated virtual disk instead of creating Logical volumes is a solid, often better alternative. It adds more flexibility. For example, in the case of the “cold standby” VM mentioned above, the system administrator can assign the (virtual) disk with the entire current dataset to the other VM in less than a minute.

Cockpit supports the creation of additional virtual disks (but unfortunately not to modify an existing one). On the host machine, go to the Virtual Machines details page and in the Disks section select the Add Disk button. It opens a form to fill in.

In the example shown above a 30gb disk for Web service is created. Selecting Add creates a disk image in /var/lib/libvirt/images and adds it to the Cockpit VM details page. You may check in the terminal window using lsblk. It shows a disk vdb without any partition, so far.

The host Cockpit session simply creates a new (virtual) hard disk and “plugs” it into the virtual machine. Configuration and setup of the hard disk occurs within the VM. On the VM, open a new tab in your browser, open Cockpit for this VM and select Storage  there.

On the right side you see the newly created disk listed as part of the (virtual hardware) equipment of the VM. You can work on this disk like on any other. A click opens a corresponding dialog form.

The system administrator can create a partition table and one or more partitions, can create a Volume group and Logical Volumes the same way as described in the previous section. It is also possible to set up and mount a file system directly without a partition table. Cockpit enables and supports all these options comfortably and efficiently.

Manage Libvirt disk pool

It is not only the disk images of virtual machines that need to be adjusted at times. The space on the host for these files can also run out and require expansion.

If the system administrator is committed to the Fedora Server concept, he has set up a dedicated separate file system for the image pool or overall for the libvirt directory. On the Storage page of the host’s Cockpit, identify the file system and select it. Enlarge the row using the “>” icon.

In the example above, the system administrator chose to provide a separate storage for the dedicated libvirt directory tree in “/dev/usrvg/libvirt”. Selecting Grow you get a form where you have to enter just the desired new size. Cockpit does everything else, including the adjustment of the file system, completely on its own.

It couldn’t be simpler or faster.

Manage network connectivity

Manage libvirt virtual network

The current version of Cockpit doesn’t offer many configuration options for a libvirt managed virtual network, so far. The virtual host Cockpit lists the networks for existing virtual machine under the “Network” link at the top, line by line. Mostly this is only the default network. By expanding the line, the editing options become visible.

For a production operation the most important thing might be to ensure or to be able to verify the start of the network on a reboot of the host. All other configuration options, in particular e.g. switching NAT on and off, require the use of ” virsh net-edit default” on the command line.

Quite interesting is the option to create a new virtual network. It opens a simple form collecting the most important options.

The system administrator can select a Forward mode, either NAT, Open, or none (isolated), which can be very convenient. The default Device is automatic, alternatively you can select an existing device. The automatic option creates a new device, virbr{x}. More advanced configuration such as domain name and name resolution would require working with CLI tools here, as well.

All in all, a system administrator will be rather reluctant to modify anything on the network infrastructure frequently, if at all. So nothing important for daily operation is missing here.

Manage a virtual machines network connectivity

The Host Cockpit offers broader support for the network configuration of virtual machines. You can add and delete a network interface and fully edit its properties, as well.

The network options are located at the bottom of the screen of the respective virtual machine, in most cases outside the viewable pane, and are easy to overlook.

Add an interface

You may have configured a virtual machine with just a public interface in order to provide some public service. According to the general recommendations it uses mac-vlan as an efficient connection type. In the host Cockpit GUI Virtual Machines Networks section, this shows up as “type: direct” and “Source: enp1s0”, the host’s physical interface [1].

After some time you may need to access a service provided by the host, e.g. a database service. Because the public interface uses mac-vlan, a direct communication between host and virtual machine is not possible (due to technical limitations of that connection type). Additionally, you may want to protect that communication and its data from the public. The solution is to add an interface to the virtual machine that connects to the libvirt internal virtual network, named default and using virbr0.

With Cockpit it is just a click on the “Add network interface” link right of the section title [2] and filling in a simple form.

In the case of an internal interface, leave everything as suggested, but activate “Always attach” to make it automatically available after reboot. Select the button “Add” to create the interface.

The interface shows up below the public interface. Inside the virtual machine an “ip a” shows an additional connection “enp7s0”. Cockpit sets everything up on its own, no further involvement of the system administrator required.

Modify an interface

You may have configured a virtual machine with just an internal interface, which works perfectly as (the only) interface enp1s0. Now you need public access instead. The easiest way is to edit the interface definition in Cockpit.

In the network section of the VM’s detail page there is just one interface with 3 buttons: Delete, Unplug, and Edit. The latter opens an interface properties form.

The form shows the definition of the current internal network. Just modify the Interface type from “Virtual network” to “Direct attachment” and Source from “default” to “enp1s0”, the host physical interface. Save your selection and reboot. Assuming DHCP works, the virtual machine has a perfectly working connection to the public network, including name resolution and default firewall configuration. Otherwise you have to define a static network configuration.

Even if you need a public interface in addition to the existing internal interface you may prefer to edit the existing interface and later add a new internal interface. That way you follow the common practice of always setting up the public interface as the “first” one, i.e. enp1s0 or eth0.

Delete an interface

As an example, the virtual machine is equipped with two interfaces, a public (enp1s0) and an internal (enp7s0).

Probably you don’t want a public interface but want the machine well protected from the public network. So you delete the public interface. A simple click on Delete will remove that interface from the virtual machine (after confirmation). If you check the network configuration inside the machine, an nmcli device show, lists only one device as attached to the still existing connection. And ip a lists just lo and enp7s0.

An nmcli conn show still lists the connection enp1s0, but without an attached device. The reason is that it was the first interface explicitly set up during installation. Therefore, it persists in “/etc/NetworkManager/system-connected” and the file is still there (although without any device configuration options). Cockpit does not delete a configuration file.

Simplified Access to VMs’ Cockpit

Typically, a (physical) server hosts not only one, but several or even a huge number of VMs. Each VM will have a running Cockpit Web Interface. To access it, you have to open a Browser window, connect and login to the respective VM you want to work with separately – even if you only want to do the most minimalist of tasks. It’s certainly not difficult, but it is cumbersome.

For convenience, Cockpit provides the option to access the Cockpit instance of the VMs from the web interface of one instance, conveniently the host. The communication between the involved Cockpit instances is not handled via HTTP protocol but via SSH. Therefore, authentication can also be done via a key file. This is more secure and also more convenient.

In the upper left corner you will see the name of the logged in user and the server, along with an expand icon. This opens a box where you can switch to another server or add a new one.

The Add new host link opens a simple form to fill in hostname and user. You can also assign a color. If you connect the VM using a direct public interface (mac-vlan) you can’t use that interface because a direct access from host to vm via mac-vlan is not supported for technical reasons. You have to use the internal interface (usually libvirt provided “virbr0”). As a user, you should select the unprivileged administrative account that can be created during the VM installation by Anaconda.

Cockpit will then connect and ask for the user’s password.

On this occasion, select “Automatic login” and check Authorize SSH key. Cockpit will create one for you if none exists or otherwise uses an existing one created previously for another Cockpit connection. Cockpit installs the public key in the VM, too. You don’t have to bother with all these details.

You can also log in as root. However, the default configuration blocks a password-supported root ssh login. You have to generate a keyfile pair manually in advance and also install the public part in the VM. You then select it for a root connection.

Security Considerations

A Fedora Installation makes several efforts to minimize public exposure. One of these is to prevent password-based root login via public interface. At the same time, Fedora Server installs Cockpit by default and opens the firewall accordingly. That way it introduces a password-based login again, including terminal access. That counteracts, in a way, the ssh measure.

It’s OK to make the post installation tasks as easy and painless as possible. But securing Cockpit access has to be one of the post installation tasks. All the more so as Cockpit is usually not used on a daily basis, but runs largely unused and unattended in the background.

There are several options.

1. The easiest way is to close the Cockpit firewall port.
   […]# firewall-cmd –permanent –remove-service=cockpit
   […]# firewall-cmd –reload
   To use Cockpit you have to ssh into the server and temporarily open the port.
   […]# firewall-cmd –add-service=cockpit
2. Since you log in via ssh anyway, you can also initialize an ssh tunnel in one go and use it for Cockpit if needed. As described above, you have the Cockpit service from the firewall removed.
   To use Cockpit, log into the server and temporarily open the port.
   […]# ssh $user@example.com -i $key -L 9090:example.com:9090
   You can access the Cockpit interface using https://localhost:9090 in your favorite browser.
3. You can install Cockpit on your local Fedora workstation or on a lab server shielded by a firewall. Configure this instance to access any of your Cockpit instances as described in the previous section. Use this shielded instance as a proxy to your public servers. As described, it uses a protected ssh connection in the public network. Of course, you have the Cockpit service removed from the firewall as well.

Concluding remarks

The examples in this post cover only the most common administrative tasks for virtual machine management. In particular, they show how the web interface can free tasks from tedious typing and speed them up. There is much more to discover. For example, Cockpit also supports the creation of snapshots, which a system administrator can use for interim backups during major reconfiguration work.

The software is regularly updated and extended so a task that is currently not supported might be included in the next release. It’s worth getting into the habit of briefly checking Cockpit’s support for each new task.