Fedora Modularity is a project within Fedora with the goal of Building a modular operating system with multiple versions of components on different lifecycles. Fedora 26 features the first look of the modularity vision: the Fedora 26 Boltron Server. However,  if you are jumping into the modularity world, creating and deploying your own modules and containers — your next question may be how to test these artifacts.  The Modularity Testing Framework (MTF) has been designed for testing artifacts such as modules, RPM base repos, containers, and other artifact types. It helps you to write tests easily, and the tests are also independent of the type of the module.

MTF is a minimalistic library built on the existing avocado and behave testing frameworks. enabling developers to enable test automation for various module aspects and requirements quickly. MTF adds basic support and abstraction for testing various module artifact types: RPM based, docker images, ISOs, and more. For detailed information about the framework, and how to use it check out the MTF Documentation.

Installing MTF

The Modularity Testing Framework is available in the official Fedora repositories. Install MTF using the command:

dnf install -y modularity-testing-framework

A COPR is available if you want to use the untested, unstable version. Install via COPR with the commands:

dnf copr enable phracek/Modularity-testing-framework
dnf install -y modularity-testing-framework

Writing a simple test

Creating a testing directory structure

First, create the tests/ directory in the root directory of the module. In the tests/ directory, create a Makefile file:

MODULE_LINT=/usr/share/moduleframework/tools/modulelint/*.py
TESTS=*.py (try not to use “*.py”, but use the test files with names such as sanity1.py sanity2.py... separated by spaces)

CMD=python -m avocado run $(MODULE_LINT) $(TESTS)

#
all:
    generator  # use it in case that tests are defined also in config.yaml file (described below)
    $(CMD)

In the root directory of the module, create a Makefile file containing a section test. For example:

.PHONY: build run default

IMAGE_NAME = memcached

MODULEMDURL=file://memcached.yaml

default: run

build:
    docker build --tag=$(IMAGE_NAME) .

run: build
    docker run -d $(IMAGE_NAME)

test: build
    # used for testing docker image available on Docker Hub. Dockerfile 
    cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker.io/modularitycontainers/memcached" make all
    # used for testing docker image available on locally.
    # Dockerfile and relavant files has to be stored in root directory of the module.
    cd tests; MODULE=docker MODULEMD=$(MODULEMDURL) URL="docker=$(IMAGE_NAME)" make all
    # This tests "modules" on local system.
    cd tests; MODULE=rpm MODULEMD=$(MODULEMDURL) URL="https://kojipkgs.fedoraproject.org/compose/latest-Fedora-Modular-26/compose/Server/x86_64/os/" make all

In the tests/ directory, place the config.yaml configuration file for module testing(Do adresare tests umisti configuracni soubor config.yaml pro testovani modulu) . See minimal-config.yaml. For example:

document: modularity-testing
version: 1
name: memcached
modulemd-url: http://raw.githubusercontent.com/container-images/memcached/master/memcached.yaml
service:
    port: 11211
packages:
    rpms:
        - memcached
        - perl-Carp
testdependecies:
    rpms:
        - nc
module:
    docker:
        start: "docker run -it -e CACHE_SIZE=128 -p 11211:11211"
        labels:
            description: "memcached is a high-performance, distributed memory"
            io.k8s.description: "memcached is a high-performance, distributed memory"
        source: https://github.com/container-images/memcached.git
        container: docker.io/modularitycontainers/memcached
    rpm:
        start: /usr/bin/memcached -p 11211 &
        repo:
           - https://kojipkgs.fedoraproject.org/compose/latest-Fedora-Modular-26/compose/Server/x86_64/os/

test:
    processrunning:
        - 'ls  /proc/*/exe -alh | grep memcached'
testhost:
    selfcheck:
        - 'echo errr | nc localhost 11211'
        - 'echo set AAA 0 4 2 | nc localhost 11211'
        - 'echo get AAA | nc localhost 11211'
    selcheckError:
        - 'echo errr | nc localhost 11211 |grep ERROR'

Add the simpleTest.py python file, which tests a service or an application, into the tests/ directory:

#!/usr/bin/python
# -*- coding: utf-8 -*-
#
# This Modularity Testing Framework helps you to write tests for modules
# Copyright (C) 2017 Red Hat, Inc.

import socket
from avocado import main
from avocado.core import exceptions
from moduleframework import module_framework


class SanityCheck1(module_framework.AvocadoTest):
    """
    :avocado: enable
    """

    def testSettingTestVariable(self):
        self.start()
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect(('localhost', self.getConfig()['service']['port']))
        s.sendall('set Test 0 100 4\r\n\n')
        #data = s.recv(1024)
        # print data

        s.sendall('get Test\r\n')
        #data = s.recv(1024)
        # print data
        s.close()

    def testBinExistsInRootDir(self):
        self.start()
        self.run("ls / | grep bin")

    def test3GccSkipped(self):
        module_framework.skipTestIf("gcc" not in self.getActualProfile())
        self.start()
        self.run("gcc -v")

if __name__ == '__main__':
    main()

Running tests

To execute tests from the root directory of the module, type

# run tests from a module root directory
$ sudo make test

The result looks like:

docker build --tag=memcached .
Sending build context to Docker daemon 268.3 kB
Step 1 : FROM baseruntime/baseruntime:latest
---> 0cbcd55844e4
Step 2 : ENV NAME memcached ARCH x86_64
---> Using cache
---> 16edc6a5f7b6
Step 3 : LABEL MAINTAINER "Petr Hracek" <phracek@redhat.com>
---> Using cache
---> 693d322beab2
Step 4 : LABEL summary "High Performance, Distributed Memory Object Cache" name "$FGC/$NAME" version "0" release "1.$DISTTAG" architecture "$ARCH" com.redhat.component $NAME usage "docker run -p 11211:11211 f26/memcached" help "Runs memcached, which listens on port 11211. No dependencies. See Help File below for more details." description "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load." io.k8s.description "memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load." io.k8s.diplay-name "Memcached 1.4 " io.openshift.expose-services "11211:memcached" io.openshift.tags "memcached"
---> Using cache
---> eea936c1ae23
Step 5 : COPY repos/* /etc/yum.repos.d/
---> Using cache
---> 920155da88d9
Step 6 : RUN microdnf --nodocs --enablerepo memcached install memcached &&     microdnf -y clean all
---> Using cache
---> c83e613f0806
Step 7 : ADD files /files
---> Using cache
---> 7ec5f42c0064
Step 8 : ADD help.md README.md /
---> Using cache
---> 34702988730f
Step 9 : EXPOSE 11211
---> Using cache
---> 577ef9f0d784
Step 10 : USER 1000
---> Using cache
---> 671ac91ec4e5
Step 11 : CMD /files/memcached.sh
---> Using cache
---> 9c933477acc1
Successfully built 9c933477acc1
cd tests; MODULE=docker MODULEMD=file://memcached.yaml URL="docker=memcached" make all
make[1]: Entering directory '/home/phracek/work/FedoraModules/memcached/tests'
Added test (runmethod: run): processrunning
Added test (runmethod: runHost): selfcheck
Added test (runmethod: runHost): selcheckError
python -m avocado run --filter-by-tags=-WIP /usr/share/moduleframework/tools/modulelint.py *.py
JOB ID     : 9ba3a3f9fd982ea087f4d4de6708b88cee15cbab
JOB LOG    : /root/avocado/job-results/job-2017-06-14T16.25-9ba3a3f/job.log
(01/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testDockerFromBaseruntime: PASS (1.52 s)
(02/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testDockerRunMicrodnf: PASS (1.53 s)
(03/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testArchitectureInEnvAndLabelExists: PASS (1.63 s)
(04/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testNameInEnvAndLabelExists: PASS (1.61 s)
(05/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testReleaseLabelExists: PASS (1.60 s)
(06/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testVersionLabelExists: PASS (1.45 s)
(07/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testComRedHatComponentLabelExists: PASS (1.64 s)
(08/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIok8sDescriptionExists: PASS (1.51 s)
(09/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIoOpenshiftExposeServicesExists: PASS (1.50 s)
(10/20) /usr/share/moduleframework/tools/modulelint.py:DockerfileLinter.testIoOpenShiftTagsExists: PASS (1.53 s)
(11/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testBasic: PASS (13.75 s)
(12/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testContainerIsRunning: PASS (14.19 s)
(13/20) /usr/share/moduleframework/tools/modulelint.py:DockerLint.testLabels: PASS (1.57 s)
(14/20) /usr/share/moduleframework/tools/modulelint.py:ModuleLintPackagesCheck.test: PASS (14.03 s)
(15/20) generated.py:GeneratedTestsConfig.test_processrunning: PASS (13.77 s)
(16/20) generated.py:GeneratedTestsConfig.test_selfcheck: PASS (13.85 s)
(17/20) generated.py:GeneratedTestsConfig.test_selcheckError: PASS (14.32 s)
(18/20) sanity1.py:SanityCheck1.testSettingTestVariable: PASS (13.86 s)
(19/20) sanity1.py:SanityCheck1.testBinExistsInRootDir: PASS (13.81 s)
(20/20) sanity1.py:SanityCheck1.test3GccSkipped: ERROR (13.84 s)
RESULTS    : PASS 19 | ERROR 1 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 144.85 s
JOB HTML   : /root/avocado/job-results/job-2017-06-14T16.25-9ba3a3f/html/results.html
Makefile:6: recipe for target 'all' failed
make[1]: *** [all] Error 1
Makefile:14: recipe for target 'test' failed
make: *** [test] Error 2
$

To execute tests from the tests/ directory, type:

# run Python tests from the tests/ directory
$ sudo MODULE=docker avocado run ./*.py

The result looks like:

$ sudo MODULE=docker avocado run ./*.py
[sudo] password for phracek:
JOB ID     : 2a171b762d8ab2c610a89862a88c015588823d29
JOB LOG    : /root/avocado/job-results/job-2017-06-14T16.43-2a171b7/job.log
(1/6) ./generated.py:GeneratedTestsConfig.test_processrunning: PASS (24.79 s)
(2/6) ./generated.py:GeneratedTestsConfig.test_selfcheck: PASS (18.18 s)
(3/6) ./generated.py:GeneratedTestsConfig.test_selcheckError: ERROR (24.16 s)
(4/6) ./sanity1.py:SanityCheck1.testSettingTestVariable: PASS (18.88 s)
(5/6) ./sanity1.py:SanityCheck1.testBinExistsInRootDir: PASS (17.87 s)
(6/6) ./sanity1.py:SanityCheck1.test3GccSkipped: ERROR (19.30 s)
RESULTS    : PASS 4 | ERROR 2 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0
JOB TIME   : 124.19 s
JOB HTML   : /root/avocado/job-results/job-2017-06-14T16.43-2a171b7/html/results.html