Kushal Das4

FOSS and life. Kushal Das talks here.

Running gotun inside Jenkins

By design gotun is a command line tool which can be called from other scripts, or any larger system. In the world of CI, Jenkins is the biggest name. So, one of the goals was also being able to execute within Jenkins for tests.

Setting up a Jenkins instance for test

vIf you don’t have a setup for Jenkins already, you can just create a new one for staging using the official container. For my example setup, I am using the same at http://status.kushaldas.in

Setting up the first job

My only concern was how to setup the secrets for authentication information on Jenkins (remember I am a newbie in Jenkins). This blog post helped me to get it done. In the first job, I am creating the configuration (if in future we add something dynamic like the image name there). The secrets are coming from the ENV variables as described in the gotun docs. In the job, I am running the Fedora Atomic tests on the image. Here is one example console output.

Running the upstream Atomic host tests in gotun inside Jenkins

My next task was to run the upstream Project Atomic host tests using the similar setup. All the configuration file for the tests are available on this git repo. As explained in a previous post, onevm.py creates the inventory file for Ansible, and then runsetup.sh executes the playbook. You can view the job output here.

For both the jobs, I am executing a Python script to create the job yaml files.

Keeping the tools simple

When I talk about programming or teach in a workshop, I keep repeating one motto: Try to keep things simple. Whenever I looked at the modern complex systems which are popular among the users, generally they are many simple tools working together solving a complex problem.

The Unix philosophy

Back in college days, I read about the Unix philosophy. It is a set of ideas and philosophical approaches to the software development. From the Wikipedia page, we can find four points.

  • Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new “features”.
  • Expect the output of every program to become the input to another, as yet unknown, program. Don’t clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don’t insist on interactive input.
  • Design and build software, even operating systems, to be tried early, ideally within weeks. Don’t hesitate to throw away the clumsy parts and rebuild them.
  • Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you’ve finished using them.

Do one thing

The first point is something many people decide to skip. The idea of a perfect system which can do everything leads to a complex over engineered software which takes many years to land in production. The first causality is the user of the system, then the team who has to maintain the system up & running. A simple system with proper documentation also attracts number of users. These first users also become your community. They try out the new features, provide valuable feedback. If you are working on an Open Source project, creating that community around your project is always important for the sustainability of the project.

Pipe and redirection between tools

Piping and redirection in Linux shells were another simple magic I learned during the early days in college. How a tool like grep can take the input stream and provide an output, which in turn can be used in the next tool, was one of the best thing I found in the terminal. As a developer, I spend a lot of time in the terminals, and we use piping and redirection in daily life many times.

Build and try and repeat

I think all of the modern agile followers know this point very well. Unless you are allowing your users to try out your tool and allow them to provide feedback, your tool will not be welcomed by the users. We, the programmers have this attitude that every problem can be solved by code, and our ideas are always the correct. No, that is not the case at all. Go out in the world, show your tool to as many people as possible, take feedback. Rewrite and rebuild your tool as required. If you wait for 3 years and hope that someone will force your tool to the users, that will not go well in long run.

Do One Thing and Do It Well

The whole idea of Do One Thing and Do It Well has been discussed many times. Search the term in your favorite search engine, and you will surely find many documents explaining the idea in details. Following this idea while designing tools or systems helped me till date. Tunir or gotun tried to follow the same ideas as much as possible. They are build to execute some command on a remote system and act accordingly to the exit codes. I think this is the one line description of both the tools. To verify if the tool is simple or not, I keep throwing the tool to the new users and go through the feedback.

Last night we received a mail from Dusty Mabe in the Fedora Cloud list, to test the updates-testing tree for Fedora Atomic. At the end of the email, he also gave the command to execute to rebase to the updates-testing tree.

# rpm-ostree rebase fedora-atomic/25/x86_64/testing/docker-host 

With that as input from upstream, it was just adding the command in one line on top of the current official Fedora Atomic tests, and followed by a reboot command and wait for the machine to come back online.

sudo rpm-ostree rebase fedora-atomic/25/x86_64/testing/docker-host
@@ sudo reboot
SLEEP 120
curl -O http://infrastructure.fedoraproject.org/infra/autocloud/tunirtests.tar.gz
tar -xzvf tunirtests.tar.gz
...

This helped me to find the regression in atomic command within the next few minutes while I was working on something else. As I reported the issue to the upstream, they are already working to find a solution (some discussion here). The simplicity of the tool helped me to get things done faster in this case.

Please let me know what do you think about this particular idea about designing software in the comments below.

Testing a redis container using gotun

Testing container is one of the major reason for the existence of the tool gotun. In this blog post, I am going to show you how easy it becomes to test a container. But, easy is still a relative term. If setting up your container requires very complex steps, then you will have to do those first. And then, finally, you can test the application. You can use Ansible to do the hard part of setting up the application.

Fedora Redis container

In our example, we will test the Fedora Redis container. Redis is an in-memory data structure store. We use it as key: value store, as a database. It can also hold some complex infrastructures.

Let me post the test commands first.

sudo docker run -d --name redis fedora/redis
sudo docker run  -d --link redis:rd --name client fedora/redis
COPY: ./redistests.py vm1:./
sudo python3 -m unittest redistests.RedisTest.test_set -v
sudo python3 -m unittest redistests.RedisTest.test_append -v
sudo docker rm -f client
sudo python3 -m unittest redistests.RedisTest.test_newclient -v

In the first line, I am starting a new redis container. In the second line, I am running a container named “client”. In the next two lines, I have two tests. I will be using this container in those two tests. Then I will delete the container, and as the part of the last test, we will create a new container, and connect to the redis server container to get the value for the key name.

In the third line, I am copying a Python3 unittest file over to the VM, and then executing the tests one by one. I wanted to check the output from the redis-cli command instead of the exit code, that is why I decided to use Python unittest module. It makes parsing the output text much easier, and we can do complex analysis of the output text. We follow the same style in the Fedora Atomic tests.

redistests.py

import unittest
import subprocess

def system(cmd):
    """
    Invoke a shell command.
    :returns: A tuple of output, err message, and return code
    """
    ret = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    out, err = ret.communicate()
    return out.decode("utf-8"), err.decode("utf-8"), ret.returncode


class RedisTest(unittest.TestCase):

    def test_set(self):
        out, err, eid = system("docker exec client redis-cli -h rd SET name kushal")
        out, err, eid = system("docker exec client redis-cli -h rd GET name")
        self.assertEqual(out, "kushal\n")

    def test_append(self):
        system("docker exec client redis-cli -h rd DEL clients")
        out, err, eid = system("docker exec client redis-cli -h rd APPEND clients ABC")
        self.assertEqual(out, "3\n")
        out, err, eid = system("docker exec client redis-cli -h rd APPEND clients XYZ")
        self.assertEqual(out, "6\n")
        out, ere, eid = system("docker exec client redis-cli -h rd GET clients")
        self.assertEqual(out, "ABCXYZ\n")

    def test_newclient(self):
        out, err, eid = system("docker run --rm --link redis:r2d2 --name newclient fedora/redis redis-cli -h r2d2 GET name")
        self.assertEqual(out, "kushal\n")

The output from the job.

$ ./gotun --job commands
Starts a new Tunir Job.

Server ID: cb0f491d-a921-46dd-b6e1-3c4353884d6c
Let us wait for the server to be in running state.
Time to assign a floating pointip.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Executing:  sudo docker run -d --name redis fedora/redis
Executing:  sudo docker run  -d --link redis:rd --name client fedora/redis
Executing COPY:  scp -o StrictHostKeyChecking=no -r -i /home/kdas/Downloads/kushal-testday.pem ./redistests.py fedora@209.132.184.210:./
Executing:  sudo python3 -m unittest redistests.RedisTest.test_set -v
Executing:  sudo python3 -m unittest redistests.RedisTest.test_append -v
Executing:  sudo docker rm -f client
Executing:  sudo python3 -m unittest redistests.RedisTest.test_newclient -v
---------------


Result file at: /tmp/tunirresult_210751937


Job status: true


command: sudo docker run -d --name redis fedora/redis
status:true

Unable to find image 'fedora/redis:latest' locally
Trying to pull repository docker.io/fedora/redis ... 
sha256:1bb7b6c03575dd43db6e8e3fc0b06c0b3df0178af9ea382b6f9df577d053bd22: Pulling from docker.io/fedora/redis
a3ed95caeb02: Pull complete 
cadab8e68e03: Pull complete 
9db1d534feda: Pull complete 
a4d5e4eefba6: Pull complete 
Digest: sha256:1bb7b6c03575dd43db6e8e3fc0b06c0b3df0178af9ea382b6f9df577d053bd22
Status: Downloaded newer image for docker.io/fedora/redis:latest
b83c3f13fa698627447575651e585f3308e795f579ea772c66d02ae0f1c72939


command: sudo docker run  -d --link redis:rd --name client fedora/redis
status:true

6096e414e2f7c314c9601ae6eaedbb3faf9fb265b70a4f77e5ba7b9a93b19209


command: sudo python3 -m unittest redistests.RedisTest.test_set -v
status:true

test_set (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK


command: sudo python3 -m unittest redistests.RedisTest.test_append -v
status:true

test_append (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.289s

OK


command: sudo docker rm -f client
status:true

client


command: sudo python3 -m unittest redistests.RedisTest.test_newclient -v
status:true

test_newclient (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 1.298s

OK


Total Number of Tests:6
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.

Testing Fedora Atomic Images using upstream Atomic Host tests

Project Atomic has a group of tests written in Ansible. In this blog post, I am going to show to use those along with gotun. I will be running the improved-sanity-test as suggested in the #atomic channel on Freenode.

createinventory.py

The following is the content of the createinventory.py file. We use this file to generate the inventory file for Ansible.

#!/usr/bin/env python

import json

data = None
with open("current_run_info.json") as fobj:
    data = json.loads(fobj.read())

user = data['user']
host1 = data['vm1']
key = data['keyfile']

result = """{0} ansible_ssh_host={1} ansible_ssh_user={2} ansible_ssh_private_key_file={3}
""".format(host1, host1,user,key)
with open("./inventory", "w") as fobj:
    fobj.write(result)

gotun job configuration and the actual job file

---
BACKEND: "openstack"
NUMBER: 1

OS_AUTH_URL: "https://fedorainfracloud.org:5000/v2.0"
USERNAME="username"
TENANT_ID="ID"
PASSWORD="PASSWORD"
OS_REGION_NAME: "RegionOne"
OS_IMAGE: "Fedora-Atomic-25-20170124.1.x86_64.qcow2"
OS_FLAVOR: "m1.medium"
OS_SECURITY_GROUPS:
    - "ssh-anywhere-cloudsig"
    - "default"
OS_NETWORK: "NETWORK ID"
OS_FLOATING_POOL: "external"
OS_KEYPAIR: "kushal-test"
key: "/home/kdas/kushal-test.pem"

Above fedora.yml configuration boots up the VM for us in the OpenStack environment. Then we have the actual test file called fedora.txt.

HOSTCOMMAND: ./createinventory.py
HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml

Now when we run the job (remember, it takes a lot of time to run).

$ gotun --job fedora
Starts a new Tunir Job.

Server ID: 3e8cd0c7-bc79-435e-9cf9-169c5bc66b3a
Let us wait for the server to be in running state.
Time to assign a floating point IP.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.


Executing:  HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml
---------------


Result file at: /tmp/tunirresult_326996419


Job status: true


command: ansible-playbook -i inventory tests/improved-sanity-test/main.yml
status:true


PLAY [Improved Sanity Test - Pre-Upgrade] **************************************

TASK [setup] *******************************************************************
ok: [209.132.184.162]

TASK [ansible_version_check : Fail if avc_major is not defined] ****************
skipping: [209.132.184.162]

TASK [ansible_version_check : Fail if avc_minor is not defined] ****************
skipping: [209.132.184.162]

TASK [ansible_version_check : Check if Ansible is the correct version (or newer)] ***
ok: [209.132.184.162] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [atomic_host_check : Determine if Atomic Host] ****************************
ok: [209.132.184.162]

<SKIPPING ALL THE OTHER OUTPUT>

TASK [var_files_present : Check for correct SELinux type] **********************
changed: [209.132.184.162]

PLAY RECAP *********************************************************************
209.132.184.162            : ok=260  changed=170  unreachable=0    failed=0   

[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm
 [WARNING]: Consider using get_url or uri module rather than running curl
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.


Total Number of Tests:1
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.

Previously, I blogged about how to use the upstream Kubernetes Ansible repo to test Kubernetes on Atomic images. Using this style, you can use any ansible playbook to do the real setup, and then use Ansible to test for you, or have your own set of test cases. Do let me know what do you think in the comments.

Testing Kubernetes on Fedora Atomic using gotun

Kubernetes is one of the major components of the modern container technologies. Previously, using Tunir I worked on ideas to test it automatically on Fedora Atomic images. The new gotun provides a better way to setup and test a complex system like Kubernetes.

In the following example I have used Fedora Infra OpenStack to setup 3 instances, and then used the upstream contrib/ansible repo to do the real setup of the Kubernetes. I have two scripts in my local directory where the ansible directory also exists. First, createinventory.py, which creates the ansible inventory file, and also a hosts file with the right IP addresses. We push this file to every VM and copy to /etc/ using sudo. You can easily do this over ansible, but I did not want to change or add anything to the git repo, that is why I am doing it like this.

#!/usr/bin/env python

import json

data = None
with open("current_run_info.json") as fobj:
    data = json.loads(fobj.read())

user = data['user']
host1 = data['vm1']
host2 = data['vm2']
host3 = data['vm3']
key = data['keyfile']

result = """kube-master.example.com ansible_ssh_host={0} ansible_ssh_user={1} ansible_ssh_private_key_file={2}
kube-node-01.example.com ansible_ssh_host={3} ansible_ssh_user={4} ansible_ssh_private_key_file={5}
kube-node-02.example.com ansible_ssh_host={6} ansible_ssh_user={7} ansible_ssh_private_key_file={8}

[masters]
kube-master.example.com
 
[etcd:children]
masters
 
[nodes]
kube-node-01.example.com
kube-node-02.example.com
""".format(host1,user,key,host2,user,key,host3,user,key)
with open("ansible/inventory/inventory", "w") as fobj:
    fobj.write(result)

hosts = """127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
{0} kube-master.example.com
{1} kube-node-01.example.com
{2} kube-node-02.example.com
""".format(host1,host2,host3)

with open("./hosts","w") as fobj:
    fobj.write(hosts)

Then, I also have a runsetup.sh script, which runs the actual script inside the ansible directory.

#!/usr/bin/sh
cd ./ansible/scripts/
export ANSIBLE_HOST_KEY_CHECKING=False
./deploy-cluster.sh

The following the job definition creates the 3VM(s) on the Cloud.

---
BACKEND: "openstack"
NUMBER: 3

OS_AUTH_URL: "https://fedorainfracloud.org:5000/v2.0"
TENANT_ID: "ID of the tenant"
USERNAME: "user"
PASSWORD: "password"
OS_REGION_NAME: "RegionOne"
OS_IMAGE: "Fedora-Atomic-25-20170124.1.x86_64.qcow2"
OS_FLAVOR: "m1.medium"
OS_SECURITY_GROUPS:
    - "ssh-anywhere-cloudsig"
    - "default"
    - "wide-open-cloudsig"
OS_NETWORK: "NETWORKID"
OS_FLOATING_POOL: "external"
OS_KEYPAIR: "kushal-testkey"
key: "/home/kdas/kushal-testkey.pem"

Then comes the final text file which contains all the actual test commands.

HOSTCOMMAND: ./createinventory.py
COPY: ./hosts vm1:./hosts
vm1 sudo mv ./hosts /etc/hosts
COPY: ./hosts vm2:./hosts
vm2 sudo mv ./hosts /etc/hosts
COPY: ./hosts vm3:./hosts
vm3 sudo mv ./hosts /etc/hosts
HOSTTEST: ./runsetup.sh
vm1 sudo kubectl get nodes
vm1 sudo atomic run projectatomic/guestbookgo-atomicapp
SLEEP 60
vm1 sudo kubectl get pods

Here I am using the old guestbook application, but you can choose to deploy any application for this fresh Kubernetes cluster, and then test if it is working fine or not. Please let me know in the comments what do you think about this idea. Btw, remember that the ansible playbook will take a long time to run.