Testing Fedora MariaDB layered image testing using gotun

Yesterday Adam Miller announced the availability of the latest Fedora Layered Image release. The following container images are available in the Fedora registry:

• registry.fedoraproject.org/f25/cockpit:130-1.3.f25docker
• registry.fedoraproject.org/f25/cockpit:130
• registry.fedoraproject.org/f25/cockpit
• registry.fedoraproject.org/f25/kubernetes-node:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-node:0.1
• registry.fedoraproject.org/f25/kubernetes-node
• registry.fedoraproject.org/f25/mariadb:10.1-2.f25docker
• registry.fedoraproject.org/f25/mariadb:10.1
• registry.fedoraproject.org/f25/mariadb
• registry.fedoraproject.org/f25/kubernetes-apiserver:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-apiserver:0.1
• registry.fedoraproject.org/f25/kubernetes-apiserver
• registry.fedoraproject.org/f25/kubernetes-master:0.1-5.f25docker
• registry.fedoraproject.org/f25/kubernetes-master:0.1
• registry.fedoraproject.org/f25/kubernetes-master
• registry.fedoraproject.org/f25/flannel:0.1-3.f25docker
• registry.fedoraproject.org/f25/flannel:0.1
• registry.fedoraproject.org/f25/flannel
• registry.fedoraproject.org/f25/kubernetes-proxy:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-proxy:0.1
• registry.fedoraproject.org/f25/kubernetes-proxy
• registry.fedoraproject.org/f25/etcd:0.1-5.f25docker
• registry.fedoraproject.org/f25/etcd:0.1
• registry.fedoraproject.org/f25/etcd
• registry.fedoraproject.org/f25/toolchain:1-2.f25docker
• registry.fedoraproject.org/f25/toolchain:1
• registry.fedoraproject.org/f25/toolchain

Now, I am going to show how we can add a set of tests for the MariaDB container under gotun on a Fedora Atomic host. I will be firing up the job in the Fedora Infra Cloud.

Because of the nature of the Atomic host, I decided to write a set of tests in golang and build a static binary which can be executed inside of the Atomic host. This way we do not have to worry about any dependency in the host.

$ ldd tunirtests.test
	not a dynamic executable

Source code of the test

The following is the source for mysql_test.go.

package main

import (
	"testing"
	"database/sql"
	 _ "github.com/go-sql-driver/mysql"
)

func TestMariadb(t *testing.T) {
	db, err := sql.Open("mysql", "user:password@/dbname")
	if err != nil {
		t.Fatal("Can not connect to mariadb", err.Error())
	}
	defer db.Close()
	stmt := "CREATE TABLE IF NOT EXISTS funny (id int(5) NOT NULL AUTO_INCREMENT, name varchar(250), PRIMARY KEY(id))"
	_, err = db.Exec(stmt)
	if err != nil {
		t.Fatal("Can not create table", err.Error())
	}

	_, err = db.Exec("INSERT INTO funny VALUES (?,?)",1,"kushal")
	if err != nil {
		t.Fatal("Can not insert data", err.Error())
	}
	_, err = db.Exec("INSERT INTO funny VALUES (?,?)",2,"Python")
	if err != nil {
		t.Fatal("Can not insert data", err.Error())
	}
	rows, err := db.Query("SELECT * from funny")
	for rows.Next() {
		var uid int
		var name string
		err := rows.Scan(&uid, &name)
		if err != nil {
			t.Fatal("Error in selecting data", err.Error())
		}
		if uid == 1 {
			if name != "kushal" {
				t.Fatal("Oops, data mismatch", name)
			}
		}
	}

}

As you can see above that I have hardcoded username, password and dub name in the source. This is because while starting the Mariadb container, I can pass those values to the container using environment variables.

I am creating a table, inserting 2 rows in it. Then selecting those rows back to the tool.

the gotun job test file

The following is the content of the mariadb.txt file.

curl -O https://kushal.fedorapeople.org/tunirtests.test
sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
SLEEP 30
chmod +x tunirtests.test
./tunirtests.test -test.run TestMariadb -test.v

At first, I am downloading the binary test file from my Fedora people space. We can also push it to the VM from the host using COPY directive. Next, I am running the docker command to fire up the container with all the required environment variables. After that, I am sleeping for 30 seconds as it takes time to get that container ready for usage. In the last line, I am executing the binary to test the container. Now, we can add more fancy command line arguments to the docker run command (like a data volume), and then test those too. But, I am going to keep those as an exercise for the reader :)

If you have any comment, feel free to drop me a mail, or tweet to @kushaldas.

The final output from the test is also given below.

./gotun --job mariadb
Starts a new Tunir Job.

Server ID: 9c4168ae-d8c4-4534-a53c-c2a291d4f6c5
Let us wait for the server to be in running state.
Time to assign a floating pointip.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Executing:  curl -O https://kushal.fedorapeople.org/tunirtests.test
Executing:  sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
Sleeping for  30
Executing:  chmod +x tunirtests.test
Executing:  ./tunirtests.test -test.run TestMariadb -test.v
---------------


Result file at: /tmp/tunirresult_586858839


Job status: true


command: curl -O https://kushal.fedorapeople.org/tunirtests.test
status:true

  %!T(MISSING)otal    %!R(MISSING)eceived %!X(MISSING)ferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4347k  100 4347k    0     0  3722k      0  0:00:01  0:00:01 --:--:-- 3725k


command: sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
status:true

Unable to find image 'registry.fedoraproject.org/f25/mariadb:latest' locally
Trying to pull repository registry.fedoraproject.org/f25/mariadb ... 
sha256:9a2c3bc162b6b1a1c286302c3e635d77c6e31cbca5d354ed0a839c659e1ecfdc: Pulling from registry.fedoraproject.org/f25/mariadb
be44cf43edd1: Pull complete 
a87762b3425a: Pull complete 
Digest: sha256:9a2c3bc162b6b1a1c286302c3e635d77c6e31cbca5d354ed0a839c659e1ecfdc
Status: Downloaded newer image for registry.fedoraproject.org/f25/mariadb:latest
e9c72f1f41b1ec763418f969359c3319bdc4c95d8123fc3b5e6617768b90d00f


command: chmod +x tunirtests.test
status:true



command: ./tunirtests.test -test.run TestMariadb -test.v
status:true

=== RUN   TestMariadbConnect
--- PASS: TestMariadbConnect (0.20s)
PASS


Total Number of Tests:4
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.

A few weeks ago I wrote about how can we run gotun inside Jenkins. Following the same path, today I am writing how can we run gotun inside Taskotron. Tim Flink & Mike Ruckman (roshi) helped me to do the initial setup.

Setting up the system

In my laptop I created a job.yml following an example Tim posted on IRC.

name: runs gotun inside taskotron

environment:
  rpm:
    - docker

actions:
  - name: run the job
    shell:
      - gotun --job commands

I also had the real definition of the gotun job in the commands.yml file, and the tests in commands.txt file. In this job, I am firing up an instance in the Fedora Infra Cloud. After this, I just ran the runtask command to start the job.

$ sudo runtask -i foo-1.2.3 -t koji_build job.yml

This way we can put gotun inside Taskotron, and then execute the same set of tests we run currently for testing Fedora Atomic images.

By design gotun is a command line tool which can be called from other scripts, or any larger system. In the world of CI, Jenkins is the biggest name. So, one of the goals was also being able to execute within Jenkins for tests.

Setting up a Jenkins instance for test

vIf you don’t have a setup for Jenkins already, you can just create a new one for staging using the official container. For my example setup, I am using the same at http://status.kushaldas.in

Setting up the first job

My only concern was how to setup the secrets for authentication information on Jenkins (remember I am a newbie in Jenkins). This blog post helped me to get it done. In the first job, I am creating the configuration (if in future we add something dynamic like the image name there). The secrets are coming from the ENV variables as described in the gotun docs. In the job, I am running the Fedora Atomic tests on the image. Here is one example console output.

Running the upstream Atomic host tests in gotun inside Jenkins

My next task was to run the upstream Project Atomic host tests using the similar setup. All the configuration file for the tests are available on this git repo. As explained in a previous post, onevm.py creates the inventory file for Ansible, and then runsetup.sh executes the playbook. You can view the job output here.

For both the jobs, I am executing a Python script to create the job yaml files.

Project Atomic has a group of tests written in Ansible. In this blog post, I am going to show to use those along with gotun. I will be running the improved-sanity-test as suggested in the #atomic channel on Freenode.

createinventory.py

The following is the content of the createinventory.py file. We use this file to generate the inventory file for Ansible.

#!/usr/bin/env python

import json

data = None
with open("current_run_info.json") as fobj:
    data = json.loads(fobj.read())

user = data['user']
host1 = data['vm1']
key = data['keyfile']

result = """{0} ansible_ssh_host={1} ansible_ssh_user={2} ansible_ssh_private_key_file={3}
""".format(host1, host1,user,key)
with open("./inventory", "w") as fobj:
    fobj.write(result)

gotun job configuration and the actual job file

---
BACKEND: "openstack"
NUMBER: 1

OS_AUTH_URL: "https://fedorainfracloud.org:5000/v2.0"
USERNAME="username"
TENANT_ID="ID"
PASSWORD="PASSWORD"
OS_REGION_NAME: "RegionOne"
OS_IMAGE: "Fedora-Atomic-25-20170124.1.x86_64.qcow2"
OS_FLAVOR: "m1.medium"
OS_SECURITY_GROUPS:
    - "ssh-anywhere-cloudsig"
    - "default"
OS_NETWORK: "NETWORK ID"
OS_FLOATING_POOL: "external"
OS_KEYPAIR: "kushal-test"
key: "/home/kdas/kushal-test.pem"

Above fedora.yml configuration boots up the VM for us in the OpenStack environment. Then we have the actual test file called fedora.txt.

HOSTCOMMAND: ./createinventory.py
HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml

Now when we run the job (remember, it takes a lot of time to run).

$ gotun --job fedora
Starts a new Tunir Job.

Server ID: 3e8cd0c7-bc79-435e-9cf9-169c5bc66b3a
Let us wait for the server to be in running state.
Time to assign a floating point IP.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.


Executing:  HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml
---------------


Result file at: /tmp/tunirresult_326996419


Job status: true


command: ansible-playbook -i inventory tests/improved-sanity-test/main.yml
status:true


PLAY [Improved Sanity Test - Pre-Upgrade] **************************************

TASK [setup] *******************************************************************
ok: [209.132.184.162]

TASK [ansible_version_check : Fail if avc_major is not defined] ****************
skipping: [209.132.184.162]

TASK [ansible_version_check : Fail if avc_minor is not defined] ****************
skipping: [209.132.184.162]

TASK [ansible_version_check : Check if Ansible is the correct version (or newer)] ***
ok: [209.132.184.162] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [atomic_host_check : Determine if Atomic Host] ****************************
ok: [209.132.184.162]

<SKIPPING ALL THE OTHER OUTPUT>

TASK [var_files_present : Check for correct SELinux type] **********************
changed: [209.132.184.162]

PLAY RECAP *********************************************************************
209.132.184.162            : ok=260  changed=170  unreachable=0    failed=0   

[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm
 [WARNING]: Consider using get_url or uri module rather than running curl
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.


Total Number of Tests:1
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.

Previously, I blogged about how to use the upstream Kubernetes Ansible repo to test Kubernetes on Atomic images. Using this style, you can use any ansible playbook to do the real setup, and then use Ansible to test for you, or have your own set of test cases. Do let me know what do you think in the comments.