Kushal Das4

FOSS and life. Kushal Das talks here.

State of tests for Fedora Cloud and Atomic in March 2016

Till Fedora 22 release we have tested our Cloud images only with manual help. The amazing Fedora QA team organized test days, and also published detailed documentation on the wiki about how to test the images. People tried to help as when possible, as not having access to a Cloud was a problem for many. The images are also big in size (than any random RPM), so that also meant only people with enough bandwidth can help.

During Fedora 23 release cycle, we worked on a change for Automated Two Week Atomic release. A part of it was about having automated testing, which we enabled using Autocloud project. This automatically tests every Cloud Base and Atomic qcow2 images, and libvirt, and Virtualbox based Vagrant images.

This post will explain the state of currently activated tests for the same. These tests are written as Python3 unittest cases.

At first we run 5 non-gating tests, failing any of these will not do a release blocking, but at the end of each run we get the summary for all of these non-gating tests. These include bzip2, cpio, diffutils, Audit.

Then we go ahead in testing the basics test cases, things like journald logging, package install (only on base image), SELinux should be enforcing, and no service should fail during startup of the image.

The next test is about checking that /tmp should be world writable. We then move into testing the mount status of the /tmp filesystem. If it is a link, then also it should behave properly. Our next test checks that for Vagrant images we are using predictable naming convention of network devices, basically checking the existing of eth0.

We then disable crond service, after a reboot we make sure that the service is still in disabled state, and do the rest of service manipulations. We also check that the user journald log file exists on that reboot, this comes from a regression test. Then we reboot the instance again.

After this second reboot, we test the status of the crond service once again, and then we move to our special tests cases related to Atomic host. Means these tests will run only on Atomic hosts (both cloud, and Vagrant ones).

In the first test we check if the package docker is at all installed or not. Yes, if you guessed that this is a regression test, then you are correct. We once had an image without the docker package inside :) Next, we test the docker storage setup, that should be up, and in running state. We then run the busybox image, and see that it should be able to run properly. The atomic command is used next to start the same container. Pulling in the latest Fedora image, and running it is the next test. As our next test, we try to mount / as /host in the container. The next test is about having /bin, /sbin, and /usr mounted as read only in the Atomic host. In our final test, we check that a privileged container should be able to talk to the host docker daemon.

We also have a github wiki page explaining how to write any new test case. Feel free to ping if you want to contribute, and make Fedora flying high in the Cloud.

Life of Tunir

This comics explains the birth of the project Tunir. When I started working in the Fedora Cloud SIG, I volunteered myself to help with the testing of the Cloud images. We have very clear guidelines about what to test, and how to test.

Basically, you have to boot up a cloud image (or the atomic image) in a Cloud or on your local computer, run a few commands in sequence, and check the output. For the first few times it was fun to do, but slowly I found it is difficult for me. Being a super lazy programmer, I thought why not use the computer to do this job, it is not music theory :) The birth of Tunir was the result of the conversation between me and even lazier me. I just had to convert the shell commands into some Python3 based unittest cases.

At beginning it could only handle the qcow2 images of cloud base and atomic image. But while working with two week atomic change, I found out that we need do the same for Vagrant based images. It is important to remember that Fedora project generates two different kind of Vagrant boxes, one for libvirt, another one for standard Virtualbox based .box file. So Tunir got the power to execute tests on any given Vagrant image. Using these features Tunir is being consumed by the Autocloud project, where we automatically test Fedora cloud and Atomic image builds. I should not forget to mention that Tunir can also connect to a remote system, and execute the tests there. Did I mention that Tunir is doing all of these with only one JSON file containing the job description, and one text file containing the commands used for the actual tests? Tunir started as a tool for a developer, who does not want to spend time in configuration, it remains in the same way.

Until a couple of releases back, Tunir actually had two types of tests in it, one type of commands which will return zero as a success value, we could also mark a set of commands as the ones which will return non-zero exit code. Right now Tunir can have a third set of commands, the non-gating tests, these commands may pass, or they may fail. But Tunir will continue executing the tests accordingly.

I wrote Tunir to test the actual OS images, but it is generic enough that it can be used to test any application. You can easily configure it to download all the dependencies of your project every time on a clean cloud/Vagrant image, and then it will build and test your application (or the full application stack along with configuration files).

Last week I have added a new weapon in Tunir’s arsenal. It can now test using a given AWS AMI ID. If you run your application on AWS cloud platform, you now can use Tunir to test it there. This code is in a separate branch, but will be merged in the master this week. Following the similar style Praveen Kumar submitted a patch using which Tunir can run tests on an Openstack Cloud. We will work on it a little bit more before merging it to master branch.

You can view the tests already written for Fedora in this repository. Thanks to Trishna Guha, and Farhaan Bukhsh we now have many more test cases. I have written a document explaining how to write more tests, I have another document which explains how to debug failed tests.

What is in the future? The best way to predict your future is to create it. We have the features which we, and our users use regularly. Tunir is still simple enough for anyone to understand in less than 10 minutes. If you just want to say hi, or you are looking forward for any new feature, come to #fedora-cloud on freenode.