Remote dependency solving: Tapping into cloud computing

Many of us are familiar with the package managers in our desktops, such as

dnf

,

apt

,

yum

, and many others. An important part of these package managers is that they are able to find dependencies of software you are trying to install, and then install the dependencies for you. As a user, this makes less work for you, so you don’t have to look up the dependencies and install them manually. As a system administrator, this is a useful time saver.

However, sometimes the dependency resolution can take up valuable CPU cycles and memory on low-end and low-cost devices or machines. Trying to install new software or update can be a difficult task on these low-end machines. What is one to do?

Leverage the cloud: Remote dependency solving

Bringing in the power of the cloud, a group of developers from Red Hat Lab Q are excited to bring a new solution for this problem. Introducing server-side dependency solving, otherwise known as remote dependency solving! Remote dependency solving is an open-source project on GitHub that helps save memory and CPU on those devices with slower hardware.

This project was initiated by Jan Zeleny, and is maintained by Petr Hracek and students of FIT VUT Brno, including Josef Řídký, Michal Ruprich, and Šimon Matěj.

What’s the point? How does it work?

You have a device with low-cost hardware and you’ve recently installed Fedora on it. If you wanted to install a package or set of packages, but the device memory isn’t sufficient, memory could overlap. The device could also unexpectedly lose power or restart during the computing of dependencies. In case of failure or crash, dependencies are not determined properly. Sometimes, we might not have any dependencies at all!

Remote dependency solving solves this problem. Dependencies are computed remotely, as the name implies. This helps save those CPU cycles and memory on the machine by off-loading them to “the cloud”, where dependencies are determined and then sent back to the client.

First, the client sends information about the system and what packages to install. Next, the server computes dependencies and returns a set of packages with correct versions back to the client. Finally, the client installs like normal.

Caching

Another feature of this project is caching. This helps in a case where a second client wants to install the same set of packages and dependencies as another client that the server already computed. This saves computing time and results in a faster overall response to the client.

This is mainly useful for computing farms, where one computer solves dependencies and another one installs the set of packages with correct versions, as was computed before. Red Hat Satellite is an example of this type of usage.

A project like this requires infrastructure, but it is not currently within the scope of the development team. There are plans to work on this in the future.

Testing locally

The package is not yet part of the official Fedora repositories. There is work to get this packaged in Fedora, but the team is not there yet. Therefore, the remote dependency solving software needs to be installed from the GitHub repo.

Setting up

First, clone the project on a computer or embedded device with the following command.

$ git clone https://github.com/rh-lab-q/server-side-dependency-solving.git

This will clone the project on the device. Before project execution, you will need to install dependencies. Unfortunately, you can’t use remote dependency solving for this part! Run the following command to install all required packages.

$ sudo dnf install gcc cmake libsolv librepo-devel hawkey-devel glib2-devel json-glib json-glib-devel check-devel

Compiling and installing

Next, we need to switch to the build directory, or where we cloned the project. Then, we will need to run the

cmake

command, like the following example shows.

$ cd build
$ sudo cmake ../

The command verifies that we have all the required packages for the project. If there is no error, we can build the project using

make

.

$ sudo make

The command creates two binary files called

rds-client

and

rds-server

.

Running the remote dependency solving server and client

Once you have the two binary files created, you are ready to start. Open two terminals. One will be for

rds-server

and the other for

rds-client

. In order to install, erase, or update packages, the commands have to be executed by the root account.

In the first terminal, run the following command.

$ sudo ./ssds-server

In the second terminal, use this command.

$ sudo ./ssds-client

What’s going on in the background? On the terminal where the server is running, there will be communication with the

rds-client

terminal.

Install a package

After completing all previous steps, you are ready to install a package. For the example,

rds-client

will install

emacs

. Before installing

emacs

, make sure to check if it is installed or not. This can be done with the following command.

$ rpm -q emacs
package emacs is not installed

After confirming

emacs

is not already installed, run the following command to install the package. You should see a similar output like the one provided in this example.

$ sudo ./rds-client --install emacs
[2/2/16 14:30:27 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 1.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Sending initial message to server.
[MESSAGE]: Message sent.
[MESSAGE]: Installation of packages was selected.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
Number of packages to
install: 2
update: 0
erase: 0
maybe erase: 0
[MESSAGE]: Result from server:
Packages for install
emacs-common
emacs
[QUESTION]: Is it ok?
[y/n/d]: y
[MESSAGE]: Downloading preparation for package: emacs-common
[MESSAGE]: Downloading preparation for package: emacs
[MESSAGE]: Downloading packages.
emacs                        100% - Downloaded.
emacs-common 100% - Downloaded.
[MESSAGE]: All packages were downloaded successfully.
[MESSAGE]: Installing packages.
Preparing...                          ################################# [100%]
Updating / installing...
   1:emacs-common-1:24.5-6.fc23       ################################# [100%]
   2:emacs-1:24.5-6.fc23              ################################# [100%]

What does this look like on the server? If you go to check the server terminal window while running the command, you will see output similar to the below excerpt.

$ sudo ./rds-server
[2/2/16 14:26:06 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1
[MESSAGE]:

DEPENDENCY SOLVING.

pkg v query: emacs
[MESSAGE]: Dependencies for emacs are ok.
pred insert
{
  "code" : 11,
  "data" : {
    "install" : [
      {
        "pkg_name" : "emacs-common",
        "pkg_loc" : "Packages/e/emacs-common-24.5-6.fc23.x86_64.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=fedora-23&arch=x86_64"
      },
      {
        "pkg_name" : "emacs",
        "pkg_loc" : "Packages/e/emacs-24.5-6.fc23.x86_64.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=fedora-23&arch=x86_64"
      }
    ],
    "upgrade" : [
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}

After everything finishes, check to make sure

emacs

is installed.

$ rpm -q emacs
emacs-24.5-6.fc23.x86_64

Erase a package

Maybe you changed your mind and you want to remove a package. We will try erasing

emacs

from the system for this example. Run the following command to erase the package. You should have similar output in the terminal as in the example.

$ sudo ./rds-client --erase emacs
[3/2/16 21:11:55 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 1.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Erase of packages was selected.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
from server:
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
    ],
    "erase" : [
      {
        "pkg_name" : "emacs",
        "pkg_loc" : null,
        "base_url" : null,
        "metalink" : "@System"
      }
    ],
    "obsolete" : [
    ]
  }
}
Number of packages to
install: 0
update: 0
erase: 1
maybe erase: 0
[MESSAGE]: Result from server:
Packages for erase
emacs
[QUESTION]: Is it ok?
[y/n]: y
[MESSAGE]: Downloading packages.
[MESSAGE]: All packages were downloaded successfully.
[MESSAGE]: Erasing packages.
Preparing...                          ################################# [100%]
Cleaning up / removing...
   1:emacs-1:24.5-6.fc23              ################################# [100%]
[3/2/16 21:12:04 SSDS]: End of client.

Again, if you check your server terminal window, the output should look similar to the following block of text.

$ sudo ./rds-server
[2/2/16 14:26:06 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1

[MESSAGE]:

DEPENDENCY SOLVING.

pkg v query: emacs
[MESSAGE]: Dependencies for emacs are ok.
pred insert
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
    ],
    "erase" : [
      {
        "pkg_name" : "emacs",
        "pkg_loc" : null,
        "base_url" : null,
        "metalink" : "@System"
      }
    ],
    "obsolete" : [
    ]
  }
}

Finally, check if the package is uninstalled.

$ rpm -q emacs
package emacs is not installed

Updating the system

Updating the system can be done with one command. Run the following command in the client terminal window.

$ sudo ./rds-client --update
[9/2/16 09:44:36 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 0.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Update all packages was initiated.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
from server:
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
      {
        "pkg_name" : "tzdata-java",
        "pkg_loc" : "t/tzdata-java-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "tzdata",
        "pkg_loc" : "t/tzdata-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
... SNIP ...
      {
        "pkg_name" : "abi-dumper",
        "pkg_loc" : "a/abi-dumper-0.99.14-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      }
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}
Number of packages to
install: 0
update: 31
erase: 0
maybe erase: 0
[MESSAGE]: Result from server:
Packages for update
tzdata-java
tzdata
... SNIP ...
abi-dumper
[QUESTION]: Is it ok?
[y/n/d]: y
[MESSAGE]: Downloading preparation for package: tzdata-java
[MESSAGE]: Downloading preparation for package: tzdata
... SNIP ...
[MESSAGE]: Downloading preparation for package: abi-dumper
[MESSAGE]: Downloading packages.
[MESSAGE]: Packages are installed
[9/2/16 09:47:23 SSDS]: End of client.

Looking at the server terminal, you should see a similar output to this.

$ sudo ./rds-server
[9/2/16 09:44:21 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1
[MESSAGE]:

DEPENDENCY SOLVING.

Downloading repo: updates  - 100%
[MESSAGE]: Metadata for updates - download successfull (Destination dir: /tmp/ssds/updates).
Downloading repo: fedora  - 100%
[MESSAGE]: Metadata for fedora - download successfull (Destination dir: /tmp/ssds/fedora).
... SNIP ...
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
      {
        "pkg_name" : "tzdata-java",
        "pkg_loc" : "t/tzdata-java-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "tzdata",
        "pkg_loc" : "t/tzdata-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      ... SNIP ...
      {
        "pkg_name" : "abi-dumper",
        "pkg_loc" : "a/abi-dumper-0.99.14-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      }
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}

What’s next?

There is active development on this project and plenty of milestones to hit in the future. For those familiar with some of the more advanced features of DNF, there are plans to integrate those features into Remote Dependency Solving in the near future.

Some of the other items planned for future versions are…

  • Same commands as DNF
  • Secure communication between client and server
  • Package project into Fedora repositories
  • Cache request on the server-side so that we do not need to repeat dependency solving
  • Start
    rds-server

    as a daemon

  • Include project into Fedora Infrastructure

Learn more about remote dependency solving

You can read more about remote dependency solving by visiting the project repository on GitHub. Additionally, you can also watch the DevConf 2016 presentation first introducing the project.

https://www.youtube.com/watch?v=dOdbFQG9XlU

Using Software

5 Comments

  1. I don’t understand. How about the level of trustworthiness to your packages from your server ? Can be trust?

    • Petr Hracek

      Packages are downloaded from URL provided by the server.
      We work on security, though.

  2. Zbigniew Jędrzejewski-Szmek

    Why does the server require root privileges?

  3. Jim Becker

    This does not work on my machine. The ssds-server and ssds-client binaries are not present in the build directory after running cmake and make. However, the rds-server and rds-client binaries are present. Running rds-server works fine but running rds-client –install emacs crashes with: Error while opening file /var/cache/dnf/x86_64/23/@System.solv. There is no x86_64/23 directory present and therefore no such file @System.solv

Comments are Closed

The opinions expressed on this website are those of each author, not of the author's employer or of Red Hat. Fedora Magazine aspires to publish all content under a Creative Commons license but may not be able to do so in all cases. You are responsible for ensuring that you have the necessary permission to reuse any work on this site. The Fedora logo is a trademark of Red Hat, Inc. Terms and Conditions