Tests

Tests may be written in different ways, but are exposed and invoked in a standard way as defined by the Standard Test Interface directly in the package git repository. It is also possible to enable pipeline for the tests namespace, see Testing Tests for details. To start working on tests you can clone a package repo directly:

You can also use the fedpkg to clone the repo. See the Package Maintenance Guide for more info about the tool:

Tests are enabled by including the tests.yml file under the tests directory:

Tests are wrapped or written as Ansible playbooks. Here is an example of a simple playbok which enables a single smoke test of the qrencode package:

Let’s now briefly look at the playbook to see which variables are defined in order to enable the smoke test:

It is possible to separate tests into multiple playbooks, each of them can represent a test or a part of a test. Testing system will run each playbook matching the glob tests/tests*.yml separately in a clean environment. Optionally you can have multiple playbooks without the tests prefix and link them from the tests.yml file. Let’s have a look at the gzip example:

Before running tests make sure you have the following dependencies installed on your system:

Although some playbooks may function without sudo, tests are always invoked as root. The test itself may set up users and/or drop permissions if a part of that test. But in general be sure to be root when invoking tests.

Running a test directly on the current system is easy:

To only run tests that are suited for classic systems installed by yum or dnf use the --tags argument:

See Standard Test Roles documentation for detailed instructions how to run tests for a specific Rpm Package, Docker Container or Atomic Host.

Test code itself can be stored directly in the dist-git (recommended as default) or fetched from another repository hosted in the Fedora infrastructure such as the Test Namespace. The simplest way to add a new test is by using one of the existing Standard Test Roles which take care of many implementatin details. If you want to create a custom test follow instructions below.

Once you’ve identified a dist-git repository you will be adding new tests to (above), you can start to write a new Ansible test. Create an Ansible playbook with a new name. Make sure the extension is .yml. Lets place the following example in test_pid_1.yml file.

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We ensure this directory exists in the test.

By use of tags we note what kind of systems this test is suitable to run on. When including additional tasks such as pre_tasks make sure you set appropriate tag as well. In addition to tags listed above it’s also possible to use always to denote the task should run for all environments. For example:

The block is the section that runs the actual test. In this example, we use a rather convoluted way of checking that PID 1 exists. However, by doing so, we place an extra test artifact in the artifacts directory.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on. Try running this example test against an Atomic Host or Docker Container. It should pass. Try changing the /proc/1 argument to another value, and the test should fail.

You can use most of the Ansible techniques in your playbooks. Take a look at the Standard Test Roles for Ansible roles to make writing your tests easier.

Marking the test to be run

Just having a .yml file in the right directory doesn’t yet mean it will be invoked. Make sure to reference or add it from a tests.yml playbook. This is the entry point that the testing or CI system will use to invoke all the tests for a given package.

If the tests.yml file doesn’t yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

You can now run this test with the standard commands above.

See the Quick Start Guide to get recommendations for contributing new tests.

Let’s say you have a script that runs a test. Its stdout and stderr is the test output, and an exit status of zero indicates success. Here’s how we would wrap that test to be invoked. Lets say we have a simple script like in a file called test-simple

We can write an Ansible wrapper for this script like this in test_simple.yml:

All tests have an artifacts directory where they place their output. The testing or CI system that invokes the test will fill in this variable with a directory that it will archive. We create ensure this directory exists in the test.

The block is the section that runs the actual test.

Lastly, we download the artifacts. Remember that the test is not always running on the same system that it was invoked on.

If the tests.yml file doesn’t yet exist, create it. Lets continue with our above example and create a tests.yml with the following content:

Try running this example test against an Atomic Host or Docker Container. It should pass.

See Standard Test Roles documentation for instructions how to wrap a BeakerLib and RHTS tests.

See the Quick Start Guide to get recommendations for contributing new tests.

If you need to do any adjustments to the system before testing, include an extra ansible task before the testing section. For example this will upgrade all packages on the system to the latest version: