Project: Remote Dependency Solving

February 18, 2016
Petr Hracek
Related topics:
Developer ToolsLinux
Related products:
Developer ToolsRed Hat Enterprise Linux

Abstract

This project (part of Red Hat Lab Q) was initiated by Jan Zeleny to accommodate low-end and low-cost devices, which have usually slower hardware, and has particular usefulness to Fedora. Three students (Josef Řídký, Michal Ruprich, Šimon Matěj) from Faculty of Information Technology (FIT VUT Brno, Czech Republic) began work on the project with me (Petr Hracek) as a leader of the team.

The aim

Let’s say we have a device with low-cost hardware and we have Fedora/EPEL Linux distribution installed on the device. If we wanted to install a package or a set of packages and device memory might not be sufficient, memory could be overlapped or the device could be restarted during computing of package dependencies. In case of failure or crash, dependencies are not determined properly or we don’t even have dependencies at all.

This project “Remote Dependency Solving” should solve it. Dependencies are computed remotely, the name of the project implies. First, the client sends information about the system and which packages it would like to install. Next, the server computes dependencies and returns to the client a set of packages with correct versions. Finally, the client can just install them without solving dependencies.

Another aim of the project is feature called “caching”. This is for the case where another client has already solved dependencies which are the same as ours. This is mainly useful for computer farms where one computer solves dependencies and another one will only install a set of packages with correct versions as was computed before. An example usage can be Red Hat Satellite.

Of course this project needs an infrastructure, but it is not in the scope right now. We are planning to do it later on.

How to test the project on your local machine?

Because the project is not part of Fedora/EPEL repository yet, we need to install them from GitHub repo. However, we are working hard on adding the project as a package to Fedora.

First of all, we need to clone the project on computer or embedded device with command

git clone https://github.com/rh-lab-q/server-side-dependency-solving

Now, we have cloned the project on our device.

Before project execution, we need to install dependencies. Unfortunately, we can not use Remote Dependency Solving project for it.

Run this command to install required packages:

sudo dnf install gcc cmake libsolv librepo-devel hawkey-devel glib2-devel json-glib json-glib-devel check-devel

Now, we are ready to compile and install project.

Project compilation:

First of all, we have to switch to build directory where we have cloned the project. The next step is to run cmake command, like this

$ cd build
$ sudo cmake ../

The command verifies that we have all required packages for the project. If there is no error we can build the project using make command:

$ sudo make

Command make creates two binary files with the names rds-client and rds-server.

How to use RDS to install/erase/update package(s)?

How to test the project on local computer over localhost?

First of all, we have to open two terminals. First one will be used for rds-server and second one for rds-client.
In order to install/erase/update package(s) we need to execute them under root account:

sudo ./ssds-server

and

sudo ./ssds-client

Now, let’s look how commands install, delete and update work and see what is going on on both terminals.
On the first terminal we run rds-server; we should see communication with rds-client.

Installation of emacs package

Before emacs installation, let’s check if emacs package is installed or not. This can be done by command rpm -q emacs:

$ rpm -q emacs
package emacs is not installed

Now, let’s install emacs package. We’ll run command rds-client --install emacs to install the package

Server communication log:

$ sudo ./rds-server
[2/2/16 14:26:06 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1
[MESSAGE]:

DEPENDENCY SOLVING.

pkg v query: emacs
[MESSAGE]: Dependencies for emacs are ok.
pred insert
{
  "code" : 11,
  "data" : {
    "install" : [
      {
        "pkg_name" : "emacs-common",
        "pkg_loc" : "Packages/e/emacs-common-24.5-6.fc23.x86_64.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=fedora-23&arch=x86_64"
      },
      {
        "pkg_name" : "emacs",
        "pkg_loc" : "Packages/e/emacs-24.5-6.fc23.x86_64.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=fedora-23&arch=x86_64"
      }
    ],
    "upgrade" : [
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}

Client communication log:

$ sudo ./rds-client --install emacs
[2/2/16 14:30:27 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 1.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Sending initial message to server.
[MESSAGE]: Message sent.
[MESSAGE]: Installation of packages was selected.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
Number of packages to
    install: 2
    update: 0
    erase: 0
    maybe erase: 0
[MESSAGE]: Result from server:
Packages for install
    emacs-common
    emacs
[QUESTION]: Is it ok?
[y/n/d]: y
[MESSAGE]: Downloading preparation for package: emacs-common
[MESSAGE]: Downloading preparation for package: emacs
[MESSAGE]: Downloading packages.
emacs                        100% - Downloaded.
emacs-common    100% - Downloaded.
[MESSAGE]: All packages were downloaded successfully.
[MESSAGE]: Installing packages.
Preparing...                          ################################# [100%]
Updating / installing...
   1:emacs-common-1:24.5-6.fc23       ################################# [ 50%]
   2:emacs-1:24.5-6.fc23              ################################# [100%]

Now, let’s check if emacs package is really installed.

$ rpm -q emacs
emacs-24.5-6.fc23.x86_64
$

Erasing of emacs package

Now, let’s erase emacs package. We’ll run command rds-client --erase emacs to erase the package

Server communication log:

$ sudo ./rds-server
[2/2/16 14:26:06 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1

[MESSAGE]:

DEPENDENCY SOLVING.

pkg v query: emacs
[MESSAGE]: Dependencies for emacs are ok.
pred insert
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
    ],
    "erase" : [
      {
        "pkg_name" : "emacs",
        "pkg_loc" : null,
        "base_url" : null,
        "metalink" : "@System"
      }
    ],
    "obsolete" : [
    ]
  }
}

Client communication log:

$ sudo ./rds-client --erase emacs
[3/2/16 21:11:55 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 1.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Erase of packages was selected.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
from server:
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
    ],
    "erase" : [
      {
        "pkg_name" : "emacs",
        "pkg_loc" : null,
        "base_url" : null,
        "metalink" : "@System"
      }
    ],
    "obsolete" : [
    ]
  }
}
Number of packages to
    install: 0
    update: 0
    erase: 1
    maybe erase: 0
[MESSAGE]: Result from server:
Packages for erase
    emacs
[QUESTION]: Is it ok?
[y/n]: y
[MESSAGE]: Downloading packages.
[MESSAGE]: All packages were downloaded successfully.
[MESSAGE]: Erasing packages.
Preparing...                          ################################# [100%]
Cleaning up / removing...
   1:emacs-1:24.5-6.fc23              ################################# [100%]
[3/2/16 21:12:04 SSDS]: End of client.

Now let’s check if package is really uninstalled.

$ rpm -q emacs
package emacs is not installed
$

How to update a system with RDS?

Update of the system will be done by command rds-client --update.

Server communication log:

  $ sudo ./rds-server
[9/2/16 09:44:21 SSDS]: Server started.
[MESSAGE]: Connection accepted from ip address 127.0.0.1
[MESSAGE]:

DEPENDENCY SOLVING.

Downloading repo: updates    - 100%
[MESSAGE]: Metadata for updates - download successfull (Destination dir: /tmp/ssds/updates).
Downloading repo: fedora     - 100%
[MESSAGE]: Metadata for fedora - download successfull (Destination dir: /tmp/ssds/fedora).
Downloading repo: adobe-linux-i386   - 100%
... SNIP ...
Downloading repo: mhlavink-developerdashboard    - 100%
[MESSAGE]: Metadata for mhlavink-developerdashboard - download successfull (Destination dir: /tmp/ssds/mhlavink-developerdashboard).
Downloading repo: rhpkg  - 100%
[MESSAGE]: Metadata for rhpkg - download successfull (Destination dir: /tmp/ssds/rhpkg).
Downloading repo: fedora-steam   - 100%
[MESSAGE]: Metadata for fedora-steam - download successfull (Destination dir: /tmp/ssds/fedora-steam).
Downloading repo: helber-atom    - 100%
[MESSAGE]: Metadata for helber-atom - download successfull (Destination dir: /tmp/ssds/helber-atom).
Downloading repo: google-chrome  - 100%
[MESSAGE]: Metadata for google-chrome - download successfull (Destination dir: /tmp/ssds/google-chrome).
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
      {
        "pkg_name" : "tzdata-java",
        "pkg_loc" : "t/tzdata-java-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "tzdata",
        "pkg_loc" : "t/tzdata-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "selinux-policy-targeted",
        "pkg_loc" : "s/selinux-policy-targeted-3.13.1-158.4.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "selinux-policy",
        "pkg_loc" : "s/selinux-policy-3.13.1-158.4.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      ... SNIP ...
      {
        "pkg_name" : "abi-dumper",
        "pkg_loc" : "a/abi-dumper-0.99.14-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      }
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}

Client communication log:

  $ sudo ./rds-client --update
[9/2/16 09:44:36 SSDS]: Client startup.
[MESSAGE]: Client startup. Required package count 0.
[MESSAGE]: Trying to connect to server...(1 of 3)
[MESSAGE]: Connection to server is established.
[MESSAGE]: Update all packages was initiated.
[MESSAGE]: Sending message with repo info to server.
[MESSAGE]: Waiting for answer from server.
from server:
{
  "code" : 11,
  "data" : {
    "install" : [
    ],
    "upgrade" : [
      {
        "pkg_name" : "tzdata-java",
        "pkg_loc" : "t/tzdata-java-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "tzdata",
        "pkg_loc" : "t/tzdata-2016a-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "selinux-policy-targeted",
        "pkg_loc" : "s/selinux-policy-targeted-3.13.1-158.4.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
      {
        "pkg_name" : "selinux-policy",
        "pkg_loc" : "s/selinux-policy-3.13.1-158.4.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      },
... SNIP ...
      {
        "pkg_name" : "abi-dumper",
        "pkg_loc" : "a/abi-dumper-0.99.14-1.fc23.noarch.rpm",
        "base_url" : null,
        "metalink" : "https://mirrors.fedoraproject.org/metalink?repo=updates-released-f23&arch=x86_64"
      }
    ],
    "erase" : [
    ],
    "obsolete" : [
    ]
  }
}
Number of packages to
    install: 0
    update: 31
    erase: 0
    maybe erase: 0
[MESSAGE]: Result from server:
Packages for update
    tzdata-java
    tzdata
    selinux-policy-targeted
    selinux-policy
... SNIP ...
    abi-dumper
[QUESTION]: Is it ok?
[y/n/d]: y
[MESSAGE]: Downloading preparation for package: tzdata-java
[MESSAGE]: Downloading preparation for package: tzdata
[MESSAGE]: Downloading preparation for package: selinux-policy-targeted
[MESSAGE]: Downloading preparation for package: selinux-policy
... SNIP ...
[MESSAGE]: Downloading preparation for package: abi-dumper
[MESSAGE]: Downloading packages.
[MESSAGE]: Packages are installed
[9/2/16 09:47:23 SSDS]: End of client.
$

The rest of DNF options will be implemented soon.

What next?

  • Use the same commands as DNF,
  • secure communication between client and server,
  • package the project into Fedora,
  • cache request on the server side so that we do not need to repeat dependency solving,
  • start rds-server as a daemon,
  • and include project into Fedora-Infrastructure.

References: