Kushal Das

FOSS and life. Kushal Das talks here.


Quick way to get throw away VMs using Tunir

The latest Tunir package has a --debug option which can help us to get some quick vms up, where we can do some destructive work, and then just remove them. Below is an example to fire up two vms using Fedora Cloud base image using a quickjob.cfg file.

cpu = 1
ram = 1024

user = fedora
image = /home/Fedora-Cloud-Base-20141203-21.x86_64.qcow2

user = fedora
image = /home/Fedora-Cloud-Base-20141203-21.x86_64.qcow2

In the quickjob.txt file we just keep one command to check sanity :)

vm1 free -m

After we execute Tunir, we will something like below as output.

# tunir --multi quickjob
... lots of output ...

Non gating tests status:
DEBUG MODE ON. Destroy from /tmp/tmpiNumV2/destroy.sh

The above mention directory also has the temporary private key to login to the instances. The output also contains the IP addresses of the VM(s). We can login like

# ssh fedora@ -i /tmp/tmpiNumV2/private.pem -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no

The last two parts of the ssh command will make sure that we do not store the signature for the throwaway guests in the known_hosts file. To clean up afterwards we can do the following.

# sh /tmp/tmpiNumV2/destroy.sh

Tunir 0.13 is released and one year of development

Tunir 0.13 release is out. I already have a koji build in place for F23. There are two major feature updates in this release.

AWS support

Yes, we now support testing on AWS EC2. You will have to provide your access tokens in the job configuration along with the required AMI details. Tunir will boot up the instance, will run the tests, and then destroy the instance. There is documentation explaining the configuration details.

CentOS images

We now also support CentOS cloud images, and Vagrant images in Tunir. You can run the same tests on the CentOS images based on your need.

One year of development

I have started Tunir on Jan 12 2015, means it got more than one year of development history. At the beginning it was just a project to help me out with Fedora Cloud image testing. But it grew to a point where it is being used as the Autocloud backend to test Fedora Cloud, and Vagrant images. We will soon start testing the Fedora AMI(s) too using the same. Within this one year, there were total 7 contributors to the project. In total we are around 1k lines of Python code. I am personally using Tunir for various other projects too. One funny thing from the code commits timings, no commit on Sundays :)

Life of Tunir

This comics explains the birth of the project Tunir. When I started working in the Fedora Cloud SIG, I volunteered myself to help with the testing of the Cloud images. We have very clear guidelines about what to test, and how to test.

Basically, you have to boot up a cloud image (or the atomic image) in a Cloud or on your local computer, run a few commands in sequence, and check the output. For the first few times it was fun to do, but slowly I found it is difficult for me. Being a super lazy programmer, I thought why not use the computer to do this job, it is not music theory :) The birth of Tunir was the result of the conversation between me and even lazier me. I just had to convert the shell commands into some Python3 based unittest cases.

At beginning it could only handle the qcow2 images of cloud base and atomic image. But while working with two week atomic change, I found out that we need do the same for Vagrant based images. It is important to remember that Fedora project generates two different kind of Vagrant boxes, one for libvirt, another one for standard Virtualbox based .box file. So Tunir got the power to execute tests on any given Vagrant image. Using these features Tunir is being consumed by the Autocloud project, where we automatically test Fedora cloud and Atomic image builds. I should not forget to mention that Tunir can also connect to a remote system, and execute the tests there. Did I mention that Tunir is doing all of these with only one JSON file containing the job description, and one text file containing the commands used for the actual tests? Tunir started as a tool for a developer, who does not want to spend time in configuration, it remains in the same way.

Until a couple of releases back, Tunir actually had two types of tests in it, one type of commands which will return zero as a success value, we could also mark a set of commands as the ones which will return non-zero exit code. Right now Tunir can have a third set of commands, the non-gating tests, these commands may pass, or they may fail. But Tunir will continue executing the tests accordingly.

I wrote Tunir to test the actual OS images, but it is generic enough that it can be used to test any application. You can easily configure it to download all the dependencies of your project every time on a clean cloud/Vagrant image, and then it will build and test your application (or the full application stack along with configuration files).

Last week I have added a new weapon in Tunir's arsenal. It can now test using a given AWS AMI ID. If you run your application on AWS cloud platform, you now can use Tunir to test it there. This code is in a separate branch, but will be merged in the master this week. Following the similar style Praveen Kumar submitted a patch using which Tunir can run tests on an Openstack Cloud. We will work on it a little bit more before merging it to master branch.

You can view the tests already written for Fedora in this repository. Thanks to Trishna Guha, and Farhaan Bukhsh we now have many more test cases. I have written a document explaining how to write more tests, I have another document which explains how to debug failed tests.

What is in the future? The best way to predict your future is to create it. We have the features which we, and our users use regularly. Tunir is still simple enough for anyone to understand in less than 10 minutes. If you just want to say hi, or you are looking forward for any new feature, come to #fedora-cloud on freenode.

Introducing Autocloud

During Fedora 23 release cycle as part of Two week atomic image, we have developed, and deployed a new service in Fedora Infrastructure, called Autocloud. In simple words this services listens to fedmsg messages for successful koji builds of cloud base, and atomic images. When found, it downloads those images, and test them locally using Tunir. It tests the standard qcow2 images, and also the box files for vagrant. Yes, we test both libvirt, and Virtualbox based vagrant images (using tunir).

If you want to learn about how it is deployed, please visit this page. This link shows the output of currently running tests again a Fedora 23 atomic image. We do need help on writing new tests, and also doing the actual tests manually. My next post will have more details on that.

tunir 0.8 is out

Last week I have released tunir 0.8. In this release I have fixed few bugs related to better logging. But most of the work was wnet on vagrant support. We now support both Virtualbox, and libvirt based vagrant images. There is a new key in the job.json file called provider, if you give the value virtualbox, then tunir will use vagrant along with Virtualbox, otherwise it will try to use vagrant-libvirt provider (which is default for our usage). There is separate page in the docs which explains the configurations.

tunir 0.7 is out

Today I have released Tunir 0.7. Tunir is a simple CI which developers can even use in their laptops. There are few major changes in this release. The first one is about no database support. Tunir itself will not save any data anywhere, this also means --stateless command line argument is now unnecessary. Even if you do not pass that option, tunir will print out the output of the tests on STDOUT.

Second, and the biggest change is the ability to test on Vagrant boxes using vagrant-libvirt plugin. On a Fedora 22 system, you can install vagrant by the following command.

$ sudo dnf install vagrant-libvirt

The following is an example job configuration on Vagrant. This of course assumes that you have already downloaded that box file somewhere in the local filesystem. In case you have not noticed, we are now generating vagrant images for both Cloud base, and Atomic image in Fedora Project. You can download them from here.

  "name": "fedora",
  "type": "vagrant",
  "image": "/home/Fedora-Cloud-Atomic-Vagrant-22-20150521.x86_64.vagrant-libvirt.box",
  "ram": 2048,
  "user": "vagrant",
  "port": "22"

I have already built the rpms for Fedora, they are right now in the testing repo.

Tunir, a simple CI with less pain

One of my job requirement is to keep testing the latest Fedora cloud images. We have a list of tests from Fedora QA team. But the biggest problem is that I don't like doing these manually. I was looking for a way to run these automatically. We can do this by the normal CI systems, but there are two problems in that.

  • Most CI systems cannot handle cloud images, unless there is a real cloud running somewhere.
  • Maintaining the CI system & the cloud is a pain in my standard.

Tunir came out as a solution to these problems. It is a simple system, which can run predefined set of commands in a fresh cloud instance, or in a remote system. Btw, did I mention that you don't need a cloud to run these cloud instances in your local system? This is possible thanks to the code from Mike Ruckman.

Each job in Tunir requires two files, jobname.json and jobname.txt. The json file contains the details of the Cloud image (if any), or the remote system details, ram required for the vm etc. The .txt file contains the shell commands to run in the system. For now it has two unique commands for Tunir. You can write @@ in front of any command to mark that this command will return non zero exit code. We also have a SLEEP NUMBER_OF_SECONDS option, we use it when we reboot the system, and want Tunir to wait before executing the next command.

Tunir has a stateless mode, I use that all the time :) In stateless mode, it will not save the results in any database. It will directly print the result in the terminal.

$ tunir --job fedora --stateless

Tunir uses redis to store some configuration information, like available ports. Remember to execute createports.py to fill the configuration with available ports.

You can install Tunir using pip, a review request is also up for Fedora. If you are on Fedora 21, you can just test with my package.

I am currently using unittest for the Cloud testcases, they are available at my github. You can use fedora.json and fedora.txt from the same repo to execute the tests. Example of tests running inside Tunir is below (I am using this in the Fedora Cloud tests).

curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz
tar -xzvf tunirtests.tar.gz
python -m unittest tunirtests.cloudtests
sudo systemctl stop crond.service
@@ sudo systemctl disable crond.service
@@ sudo reboot
sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation
@@ sudo reboot
sudo python -m unittest tunirtests.cloudservice.TestServiceAfter

UPDATE: Adding the output from Tunir for test mentioned above.

sudo ./tunir --job fedora --stateless
[sudo] password for kdas: 
Got port: 2229
cleaning and creating dirs...
Creating meta-data...
downloading new image...
Local downloads will be stored in /tmp/tmpZrnJsA.
Downloading file:///home/Fedora-Cloud-Base-20141203-21.x86_64.qcow2 (158443520 bytes)
Succeeded at downloading Fedora-Cloud-Base-20141203-21.x86_64.qcow2
download: /boot/vmlinuz-3.17.4-301.fc21.x86_64 -> ./vmlinuz-3.17.4-301.fc21.x86_64
download: /boot/initramfs-3.17.4-301.fc21.x86_64.img -> ./initramfs-3.17.4-301.fc21.x86_64.img
/usr/bin/qemu-kvm -m 2048 -drive file=/tmp/tmpZrnJsA/Fedora-Cloud-Base-20141203-21.x86_64.qcow2,if=virtio -drive file=/tmp/tmpZrnJsA/seed.img,if=virtio -redir tcp:2229::22 -kernel /tmp/tmpZrnJsA/vmlinuz-3.17.4-301.fc21.x86_64 -initrd /tmp/tmpZrnJsA/initramfs-3.17.4-301.fc21.x86_64.img -append root=/dev/vda1 ro ds=nocloud-net -nographic
Successfully booted your local cloud image!
PID: 11880
Starting a stateless job.
Executing command: curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz
Executing command: tar -xzvf tunirtests.tar.gz
Executing command: python -m unittest tunirtests.cloudtests
Executing command: sudo systemctl stop crond.service
Executing command: @@ sudo systemctl disable crond.service
Executing command: @@ sudo reboot
Sleeping for 30.
Executing command: sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation
Executing command: @@ sudo reboot
Sleeping for 30.
Executing command: sudo python -m unittest tunirtests.cloudservice.TestServiceAfter

Job status: True

command: curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz
status: True

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  8019  100  8019    0     0   4222      0  0:00:01  0:00:01 --:--:--  4224

command: tar -xzvf tunirtests.tar.gz
status: True


command: python -m unittest tunirtests.cloudtests
status: True

Ran 4 tests in 0.036s

OK (skipped=1, unexpected successes=2)

command: sudo systemctl stop crond.service
status: True

command: @@ sudo systemctl disable crond.service
status: True

Removed symlink /etc/systemd/system/multi-user.target.wants/crond.service.

command: @@ sudo reboot
status: True

command: sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation
status: True

Ran 1 test in 0.282s


command: sudo python -m unittest tunirtests.cloudservice.TestServiceAfter
status: True

Ran 1 test in 0.070s