During a session of the summer training this year, someone asked about the history of DGPLUG and how I started contributing to it. The story of the dpglug has an even longer back-story about my history with Linux. I'll start there, and then continue with the story of dgplug.

Seeing Linux for the first time

During my class 11-12, I used to spend a lot of time in the hostels of the Regional Engineering College, Durgapur, (or as we call it, REC, Durgapur.) This institute is now known as NIT, Duragpur. I got to lay my hands on and use a computer (mostly to play games) in my uncle’s hostel room. All the machines I saw there, were Windows only.

My Join Entrance Examination(JEE) center was the REC (year: 2001). That was a known & familiar environment for me. During breaks on day 1, I came back to hall 2 (my uncle’s room) for lunch. The room was unusually full; I think more than 7 people were looking at the computer with very anxious faces. One guy was doing something on the computer. My food was kept in the corner, and someone told me to eat quietly. I could not resist, and asked what was going on.

We’re installing Linux in the computer, and this is a very critical phase.
If the mouse works, then it will just work, or else we will not be able to use it in the system at all, someone replied

I had to ask again, What is this Linux? “Another Operating System” came the reply. I knew almost everyone in the room, I knew that they used computers daily. I also saw a few of them also writing programs to solve their chemical lab problems (my uncle was in Chemical Engineering department). There were people from the computer science department too. The thing stuck in my head, going back to the next examination, was that one guy, who knew something which others did not. It stuck deeper, than the actual exam in hand, and I kept thinking about it all day, that day. Later after the exam, I actually got some time to sit in front of the Linux computer, and then I tried to play with it, trying things and clicking around on the screen. Everything was so different than I used to see on my computer screen. That was it, my mind was set; I was going to learn Linux, since not many know about it. I will also be able to help others as required, just like my uncle’s friend.

Getting my own computer

After the results of the JEE were announced, I decided to join the Dr. B. C. Roy Engineering College, Durgapur. It was a private college, opened a year back. On 15th August, I also got my first computer, a Pentium III, with 128MB of RAM, and a 40GB hard drive. I also managed to convince my parents to get a mechanical keyboard somehow, (which was costly and unusual at the time) with this setup. I also got Linux (RHL 7.x IIRC) installed on that computer along with Windows 98. Once it got home, I kept clicking on everything in Linux, ran the box ragged and tried to find all that I could do with Linux.

After a few days, I had some trouble with Windows and had to reinstall it, and when I rebooted, I could not boot to Linux any more. The option had disappeared. I freaked out at first, but guessed that it had something to do with my Windows reinstall. As I had my Linux CDs with me, I went ahead and tried to install it again. Installing and reinstalling the operating systems over and over, gave me the idea that I will had to install Windows first, and only then, should I install Linux. Otherwise, we can not boot into Linux.

Introduction to the Command Line

I knew a few Windows commands by that time. Someone in REC pointed me to a book written by Sumitabha Das (I still have a copy at home, in my village). I started reading from there, and learning commands one by one.

Becoming the Linux Expert in college

This is around the same time when people started recognizing me as a Linux Expert; at least in my college.
Of course I knew how to install Linux, but the two major things that helped get that tag, were

• the mount command, I knew how to mount Windows partitions in Linux
• xmms-mp3 rpm package. I had a copy, and I could install it on anyone’s computer.

The same song, on the same hardware, but playing in XMMS always used to give much better audio quality than Windows ever did. Just knowing those two commands gave me a lot of advantage over my peers in that remote college (we never had Internet connection in the college, IIRC).

The Unix Lab & Introduction to computer class

We were introduced to computers in our first semester in some special class. Though many of my classmates saw a computer for the first time in their life, we were tasked to practice many (DOS) commands in the same day. I spent most of my time, helping others learn about the hardware and how to use it.

In our college hostel, we had a few really young professors who also stayed with us. Somehow I started talking a lot with them, and tried to learn as many things as I could. One of them mentioned something about a Unix lab in the College which we were supposed to use in the coming days. I went back to the college the very next day and managed to find the lab; the in-charge (same person who told me about it) allowed me to get in, and use the setup (there were 20 computers).

Our batch started using the lab, only for 2-3 days at the most. During the first day in the lab, I found a command to send out emails to the other users. I came back during some off hours, and wrote a long mail to one of my classmates (not going to talk about the details of the mail) and sent it out.

As we stopped using that lab, I was sure no one had read that mail. Except for one day, the lab in-charge asked me how my email writing was going on. I was stunned, how did he find out about the email? I was all splutteringly, tongue tied! Later at night, he explained to me, the idea of sysadmins, and all that a superuser can do in a Linux/Unix environment. I started thinking about privacy in the electronic world from that night itself :)

Learning from friends

The only other people who were excited about Linux, were two people from same batch in REC. Bipendra Shrestha, and Jitu. I used to spend a lot of time in their hostel and learned so many things from them.

Internet access and the start of dgplug

In 2004, I managed to get more regular access to the Internet (by saving up a bit more money to visit Internet cafes regularly). My weekly allowance was Rs.100, and regular one hour Internet access was around Rs.30-50.

While reading about Linux, I found the term LUG, or Linux User groups. As I was the only regular Linux user in college, I knew that I never had much chance to learn more on my own there, and that somehow I will have to find more people like me and learn together. Also around the same time, I started learning about words like upstream, contribution, Free Software, & FSF. I managed to contact Sankarshan Mukhopadhyay, who sent me a copy of Ankur Bangla, a Linux running in my mother tongue, Bengali. I also came to know about all the ilug chapters in India. That inspired me. Having our own LUG in Durgapur was my next goal. Soumya Kanti Chakrabarty was the first person, I convinced, to join with me to form this group.

The first website came up on Geocities (fun times), and we also had our yahoo group. Later in 2005, we managed to register our domain name; the money came in as a donation from my uncle (who by this time was doing his Ph.D. in IIT Kanpur).

I moved to Bangalore in July 2005, and Soumya was running the local meetings. After I started using IRC regularly, we managed to have our own IRC channel, and we slowly moved most of our discussion over to IRC only. I attended FOSS.IN in December 2005. I think I should write a complete post about that event, and how it changed my life altogether.

Physical meetings in 2006-2007

A day with Fedora on 4th April 2006 was the first big event for us. Sayamindu Dasgupta, Indranil Dasgupta, and Somyadip Modak came down for this event to Durgapur. This is the same time when we started the Bijra project, where we helped the school to have a Linux (LTSP) based setup, completely in Bengali. This was the first big project we took on as a group. This also gave us some media coverage back then. This led to the bigger meetup during 2007, when NRCFOSS members including lawgon, and Rahul Sundaram came down to Durgapur.

dgplug summer training 2008

In 2008, I pitched the idea of having a summer training over IRC, following the same rules of meetings as Fedora marketing on IRC. Shakthi Kannan also glommed onto the idea, and that started a new chapter in the history of dgplug.

Becoming the active contributor community

I knew many people who are better than me when it comes to brain power, but generally, there was no one to push the idea of always learning new things to them. I guess the motto of dgplug, “Learn and teach others” helped us go above this obstacle, and build a community of friends who are always willing to help.

শেখ এবং শেখাও (Learn & Teach others).

Our people are from different backgrounds, from various countries, but the idea of Freedom and Sharing binds us together in the group known as dgplug. Back in 2015 at PyCon India, we had a meeting of all the Python groups in India. After listening to the problems of all the other groups, I suddenly realized that we had none of those problems. We have no travel issues, no problems getting speakers, and no problem getting new people to join in. Just being on the Internet, helps a lot. Also, people in the group have strong opinions, this means healthy but long discussions sometimes :)

Now, you may have noticed that I did not call the group a GNU/Linux users group. Unfortunately, by the time I learned about the Free Software movement, and its history, it was too late to change the name. This year in the summer training, I will take a more in-depth session about the history of hacker ethics, and Free Software movement. and I know few other people will join in.

The future

I wish that DGPLUG continues to grow along with the members. The group does not limit itself only to be about software, or technology. Most of the regular members met each other in conferences, and we keep meeting every year in PyCon India, and PyCon Pune. We should be able to help other to learn and use the same freedom (be it in technology or in other walks of life) we have. The IRC channel should continue to be the happy place it always has been; where we all meet every day, and have fun together.

In Free and Open Source Software culture, conferences became an important part of the community. Most of the projects or communities do the work over this beautiful thing known as the Internet, people are taking part from the warmth of their home. Conferences are the only time when we all get a chance to meet, discuss new ideas, share the knowledge among ourselves. Conference speakers are generally the volunteers who agree to spend a lot of time to prepare and then give the talk, do the QA session. This also involves a lot of travel, for any mid-sized to a big conference, you will always find at least couple of speakers traveling half of the world to give those talks.

We, the organizers of many of these conferences can do a few things which helps the speakers to have a trouble free mind.

Inform the talk selection result as soon as possible(aka. visa takes time)

We should inform the speakers about the talk selection result as soon as possible. International traveling still requires visa for many countries, and generally, they are difficult to obtain in less time. For example, I am an Indian, and for getting a visa for most countries, I will have to submit my last three years income tax documents, last 6 months bank statements, office leave letters, and many other documents. Obtaining the documents take time, and we should make sure that the speakers have plenty of time to get this done.

Help with the travel timings

The organizers are local to the conference host city, it is better you provide some insight about travel timings to your conference speakers. They may not want to take a redeye flight, or maybe taking that early morning will provide a much better experience by skipping all the city traffic.

Be in touch during local travel

For all the conferences we organized in Pune (FUDCon 2011, 2015, PyCon Pune 2017), many of the international speakers landed in Mumbai, and then we organized cabs to pick them up, mostly all of these are between 12-4AM. They had to travel next 4 hours in the cab, we made sure to club at least 2 of the speakers in each cab, and also making sure that the drivers can speak English. Remember the language problem, as not every country speaks English fluently, and that goes same to all speakers too. If the speakers are coming on their own from the airport, make sure that they have all the details, and please try to have someone waiting for them at the airport. It is better that you introduce your volunteer to the speakers before the conference. That way when the speakers come out of the airport, they will see a familiar face waiting for them.

One of our tactics was to operate the whole speakers travel from the same hotel speakers were staying. I was awake for the 2 nights and started talking to the speakers as soon as they entered their cabs. We also talked to them during their trip to make sure that everything is okay, and they don’t uncomfortable. Only sad part was that due to lack of light many missed the beautiful view of Mumbai-Pune expressway.

During the conference days, before or after, we make sure that we have enough local volunteers to provide any help as required to the speakers. Many times it happens that the speakers prefer to visit some of the tourist locations before or after the conferences. As hosts, it goes also to our list of responsibilities, helping with the cab booking or providing suggestions for the sightseeing.

I remember the long list of speakers, and the checkbox(s) beside their names to mark that the speaker has boarded safely in the flight for their return journey. Our job is not done till the time they reach back to their home, and we make sure to keep an eye for any emergency. Many of the ideas also go to the speakers who are coming for the first time to the city from other parts of the country. In a country like India, where we have 24 official languages, it is difficult for most of Indians to understand or speak the local language in any state other than their own.

Previously, we had experiences where someone felt sick during the conference, and we have to take them to visit doctors, and making sure to check that they are okay. Telling that we have a large conference, and we can not take care of all of the people is easy, but remember the amount of effort these speakers are putting to make your conference as their own, and providing a great experience to the conference attendees.

FOSSASIA 2017 reminded me of foss.in. After a long time, finally, a conference which has the similar aspects. Similar kind of tight organizing team, the presence of upstream communities from different locations. The participation from the local Singapore tech community along with Hackerspace Singapore is a serious boost. This was my 4th FOSSASIA conference, and also 3rd time in Singapore. I should thank Mario, Hong, and rest of the organizers to make this event a very pleasant experience.

This time Sayan booked an Airbnb for Anwesha and me. Saptak + Medo + Siddhesh + Praveen Patil, and Pooja Yadav, & Pravin Kumar were also staying in the same Airbnb in the Chinatown. The conference venue was the Singapore Science Center just like last year. Having the conference in the same place helps as the MRT route is very easy to reach there on time.

The day before the conference we had a speakers meetup in the Singapore Microsoft office. We also received a tour of the office, the person in-charge also explained about how are managing an office without permanent seating positions.

Day one

The conference started at 9:24AM (as Hong asked us to remember the time). I attended the talks from Harish Pillay and Chan Cheow Hoe. The idea of having the CIO of the country coming to the conference and giving a talk on Open Source is a great feeling. In 2015 we had Minister for Foreign Affairs, Mr. Vivian Balakrishnan giving a keynote (and talking about the NodeJS code he wrote). The way govt. is taking part in the local community events is something other countries should try to learn. Of course, Singapore has the benefit of being small in size.

Though the day was full of talks related to AI and machine learning, there were two talks I was waiting to attend. After lunch, the first one was from Bunnie Huang, where he spoke about making technology more inclusive. He talked about Chibitronics. Before I traveled to Singapore, I actually asked him to get a copy of his new book, The Hardware Hacker. I got my copy signed by him after his talk :) (I already finished the book while coming back to India, more on that later in a separate blog post). I also met Xobs and found a Chibtronics Love-to-Code board in his pocket :)

Later in the day, Frank Karlitschek gave his keynote titled Protecting privacy with free software. He brought up the original idea of the Internet being decentralized. The last talk of the day was a panel discussion on Artificial Intelligence.

Day 2 & Day 3

I spent most of the time in the Python track, and in between jumping around different floors of the venue meeting people. I personally had a two-hour workshop on MicroPython and NodeMCU. Anwesha was busy in the PyLadies table along with Pooja. I forgot to show the poster of PyCon APAC in the Python track :( But you can still submit talks and attend the conference. Sadly this will clash with another conference for me.

Anwesha had her talk on day 3, and her laptop's display decided to crash just before the talk. But finally the slides came back :) I also attended the SELinux workshop from Jason Zaman. He and few BSD friends convinced me to try out ZFS, and then build a new home storage with FreeNAS.

Now I have to wait for the next edition of FOSSASIA. It is a great place where I can meet my friends from different parts of the world, and share ideas :)

Ubiquiti management interface does not allow to have emojis in the network SSID. After asking over twitter about it, Donald Stufft pointed me to a hack in their forum.

First, create a second wifi network, and then you will have to copy the /tmp/system.cfg file from the access point.

scp yourusername@192.168.1.IP:/tmp/system.cfg .

After this open up the system.cfg in your favorite editor, replace the newly created network name with the emoji text you want. Copy it back to the access point. After this ssh into the access point, it will give you a busybox prompt. Type save command in the prompt, and then reboot the access point from the web-console. This will give you the SSID you were looking for :)

Testing Fedora MariaDB layered image testing using gotun

Yesterday Adam Miller announced the availability of the latest Fedora Layered Image release. The following container images are available in the Fedora registry:

• registry.fedoraproject.org/f25/cockpit:130-1.3.f25docker
• registry.fedoraproject.org/f25/cockpit:130
• registry.fedoraproject.org/f25/cockpit
• registry.fedoraproject.org/f25/kubernetes-node:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-node:0.1
• registry.fedoraproject.org/f25/kubernetes-node
• registry.fedoraproject.org/f25/mariadb:10.1-2.f25docker
• registry.fedoraproject.org/f25/mariadb:10.1
• registry.fedoraproject.org/f25/mariadb
• registry.fedoraproject.org/f25/kubernetes-apiserver:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-apiserver:0.1
• registry.fedoraproject.org/f25/kubernetes-apiserver
• registry.fedoraproject.org/f25/kubernetes-master:0.1-5.f25docker
• registry.fedoraproject.org/f25/kubernetes-master:0.1
• registry.fedoraproject.org/f25/kubernetes-master
• registry.fedoraproject.org/f25/flannel:0.1-3.f25docker
• registry.fedoraproject.org/f25/flannel:0.1
• registry.fedoraproject.org/f25/flannel
• registry.fedoraproject.org/f25/kubernetes-proxy:0.1-3.f25docker
• registry.fedoraproject.org/f25/kubernetes-proxy:0.1
• registry.fedoraproject.org/f25/kubernetes-proxy
• registry.fedoraproject.org/f25/etcd:0.1-5.f25docker
• registry.fedoraproject.org/f25/etcd:0.1
• registry.fedoraproject.org/f25/etcd
• registry.fedoraproject.org/f25/toolchain:1-2.f25docker
• registry.fedoraproject.org/f25/toolchain:1
• registry.fedoraproject.org/f25/toolchain

Now, I am going to show how we can add a set of tests for the MariaDB container under gotun on a Fedora Atomic host. I will be firing up the job in the Fedora Infra Cloud.

Because of the nature of the Atomic host, I decided to write a set of tests in golang and build a static binary which can be executed inside of the Atomic host. This way we do not have to worry about any dependency in the host.

$ ldd tunirtests.test
	not a dynamic executable

Source code of the test

The following is the source for mysql_test.go.

package main

import (
	"testing"
	"database/sql"
	 _ "github.com/go-sql-driver/mysql"
)

func TestMariadb(t *testing.T) {
	db, err := sql.Open("mysql", "user:password@/dbname")
	if err != nil {
		t.Fatal("Can not connect to mariadb", err.Error())
	}
	defer db.Close()
	stmt := "CREATE TABLE IF NOT EXISTS funny (id int(5) NOT NULL AUTO_INCREMENT, name varchar(250), PRIMARY KEY(id))"
	_, err = db.Exec(stmt)
	if err != nil {
		t.Fatal("Can not create table", err.Error())
	}

	_, err = db.Exec("INSERT INTO funny VALUES (?,?)",1,"kushal")
	if err != nil {
		t.Fatal("Can not insert data", err.Error())
	}
	_, err = db.Exec("INSERT INTO funny VALUES (?,?)",2,"Python")
	if err != nil {
		t.Fatal("Can not insert data", err.Error())
	}
	rows, err := db.Query("SELECT * from funny")
	for rows.Next() {
		var uid int
		var name string
		err := rows.Scan(&uid, &name)
		if err != nil {
			t.Fatal("Error in selecting data", err.Error())
		}
		if uid == 1 {
			if name != "kushal" {
				t.Fatal("Oops, data mismatch", name)
			}
		}
	}

}

As you can see above that I have hardcoded username, password and dub name in the source. This is because while starting the Mariadb container, I can pass those values to the container using environment variables.

I am creating a table, inserting 2 rows in it. Then selecting those rows back to the tool.

the gotun job test file

The following is the content of the mariadb.txt file.

curl -O https://kushal.fedorapeople.org/tunirtests.test
sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
SLEEP 30
chmod +x tunirtests.test
./tunirtests.test -test.run TestMariadb -test.v

At first, I am downloading the binary test file from my Fedora people space. We can also push it to the VM from the host using COPY directive. Next, I am running the docker command to fire up the container with all the required environment variables. After that, I am sleeping for 30 seconds as it takes time to get that container ready for usage. In the last line, I am executing the binary to test the container. Now, we can add more fancy command line arguments to the docker run command (like a data volume), and then test those too. But, I am going to keep those as an exercise for the reader :)

If you have any comment, feel free to drop me a mail, or tweet to @kushaldas.

The final output from the test is also given below.

./gotun --job mariadb
Starts a new Tunir Job.

Server ID: 9c4168ae-d8c4-4534-a53c-c2a291d4f6c5
Let us wait for the server to be in running state.
Time to assign a floating pointip.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Executing:  curl -O https://kushal.fedorapeople.org/tunirtests.test
Executing:  sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
Sleeping for  30
Executing:  chmod +x tunirtests.test
Executing:  ./tunirtests.test -test.run TestMariadb -test.v
---------------


Result file at: /tmp/tunirresult_586858839


Job status: true


command: curl -O https://kushal.fedorapeople.org/tunirtests.test
status:true

  %!T(MISSING)otal    %!R(MISSING)eceived %!X(MISSING)ferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4347k  100 4347k    0     0  3722k      0  0:00:01  0:00:01 --:--:-- 3725k


command: sudo docker run -d --name mariadb_database -e MYSQL_USER=user -e MYSQL_PASSWORD=password -e MYSQL_DATABASE=dbname -p 3306:3306 registry.fedoraproject.org/f25/mariadb
status:true

Unable to find image 'registry.fedoraproject.org/f25/mariadb:latest' locally
Trying to pull repository registry.fedoraproject.org/f25/mariadb ... 
sha256:9a2c3bc162b6b1a1c286302c3e635d77c6e31cbca5d354ed0a839c659e1ecfdc: Pulling from registry.fedoraproject.org/f25/mariadb
be44cf43edd1: Pull complete 
a87762b3425a: Pull complete 
Digest: sha256:9a2c3bc162b6b1a1c286302c3e635d77c6e31cbca5d354ed0a839c659e1ecfdc
Status: Downloaded newer image for registry.fedoraproject.org/f25/mariadb:latest
e9c72f1f41b1ec763418f969359c3319bdc4c95d8123fc3b5e6617768b90d00f


command: chmod +x tunirtests.test
status:true



command: ./tunirtests.test -test.run TestMariadb -test.v
status:true

=== RUN   TestMariadbConnect
--- PASS: TestMariadbConnect (0.20s)
PASS


Total Number of Tests:4
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.

A few weeks ago I wrote about how can we run gotun inside Jenkins. Following the same path, today I am writing how can we run gotun inside Taskotron. Tim Flink & Mike Ruckman (roshi) helped me to do the initial setup.

Setting up the system

In my laptop I created a job.yml following an example Tim posted on IRC.

name: runs gotun inside taskotron

environment:
  rpm:
    - docker

actions:
  - name: run the job
    shell:
      - gotun --job commands

I also had the real definition of the gotun job in the commands.yml file, and the tests in commands.txt file. In this job, I am firing up an instance in the Fedora Infra Cloud. After this, I just ran the runtask command to start the job.

$ sudo runtask -i foo-1.2.3 -t koji_build job.yml

This way we can put gotun inside Taskotron, and then execute the same set of tests we run currently for testing Fedora Atomic images.

You may already hear about Kubernetes or you may be using it right now. OpenShift Origin is a distribution of Kubernetes, which is optimized for continuous development and multi-tenant deployment. It also powers the Red Hat OpenShift.

Minishift is the upcoming tool which will enable you to run OpenShift locally on your computer on a single node OpenShift cluster inside a VM. I am using it on a Fedora 25 laptop, with help of KVM. It can also be used on Windows or OSX. For KVM, I first had to install docker-machine-driver-kvm. Then downloaded the latest minishift from the releases page. Unzip, and put the binary in your path.

$ ./minishift start
Starting local OpenShift cluster using 'kvm' hypervisor...
E0209 20:42:29.927281    4638 start.go:135] Error starting the VM: Error creating the VM. Error creating machine: Error checking the host: Error checking and/or regenerating the certs: There was an error validating certificates for host "192.168.42.243:2376": tls: DialWithDialer timed out
You can attempt to regenerate them using 'docker-machine regenerate-certs [name]'.
Be advised that this will trigger a Docker daemon restart which might stop running containers.
. Retrying.
Provisioning OpenShift via '/home/kdas/.minishift/cache/oc/v1.4.1/oc [cluster up --use-existing-config --host-config-dir /var/lib/minishift/openshift.local.config --host-data-dir /var/lib/minishift/hostdata]'
-- Checking OpenShift client ... OK
-- Checking Docker client ... OK
-- Checking Docker version ... OK
-- Checking for existing OpenShift container ... OK
-- Checking for openshift/origin:v1.4.1 image ... 
   Pulling image openshift/origin:v1.4.1
   Pulled 0/3 layers, 3% complete
   Pulled 0/3 layers, 24% complete
   Pulled 0/3 layers, 45% complete
   Pulled 1/3 layers, 63% complete
   Pulled 2/3 layers, 81% complete
   Pulled 2/3 layers, 92% complete
   Pulled 3/3 layers, 100% complete
   Extracting
   Image pull complete
-- Checking Docker daemon configuration ... OK
-- Checking for available ports ... OK
-- Checking type of volume mount ... 
   Using Docker shared volumes for OpenShift volumes
-- Creating host directories ... OK
-- Finding server IP ... 
   Using 192.168.42.243 as the server IP
-- Starting OpenShift container ... 
   Creating initial OpenShift configuration
   Starting OpenShift using container 'origin'
   Waiting for API server to start listening
   OpenShift server started
-- Adding default OAuthClient redirect URIs ... OK
-- Installing registry ... OK
-- Installing router ... OK
-- Importing image streams ... OK
-- Importing templates ... OK
-- Login to server ... OK
-- Creating initial project "myproject" ... OK
-- Removing temporary directory ... OK
-- Server Information ... 
   OpenShift server started.
   The server is accessible via web console at:
       https://192.168.42.243:8443

   You are logged in as:
       User:     developer
       Password: developer

   To login as administrator:
       oc login -u system:admin

The oc binary is in ~/.minishift/cache/oc/v1.4.1/ directory, so you can add that in your PATH. If you open up the above-mentioned URL in your browser, you will find your OpenShift cluster is up and running well.

Now you can start reading the Using Minishift to start using your brand new OpenShift Cluster.

By design gotun is a command line tool which can be called from other scripts, or any larger system. In the world of CI, Jenkins is the biggest name. So, one of the goals was also being able to execute within Jenkins for tests.

Setting up a Jenkins instance for test

vIf you don’t have a setup for Jenkins already, you can just create a new one for staging using the official container. For my example setup, I am using the same at http://status.kushaldas.in

Setting up the first job

My only concern was how to setup the secrets for authentication information on Jenkins (remember I am a newbie in Jenkins). This blog post helped me to get it done. In the first job, I am creating the configuration (if in future we add something dynamic like the image name there). The secrets are coming from the ENV variables as described in the gotun docs. In the job, I am running the Fedora Atomic tests on the image. Here is one example console output.

Running the upstream Atomic host tests in gotun inside Jenkins

My next task was to run the upstream Project Atomic host tests using the similar setup. All the configuration file for the tests are available on this git repo. As explained in a previous post, onevm.py creates the inventory file for Ansible, and then runsetup.sh executes the playbook. You can view the job output here.

For both the jobs, I am executing a Python script to create the job yaml files.

When I talk about programming or teach in a workshop, I keep repeating one motto: Try to keep things simple. Whenever I looked at the modern complex systems which are popular among the users, generally they are many simple tools working together solving a complex problem.

The Unix philosophy

Back in college days, I read about the Unix philosophy. It is a set of ideas and philosophical approaches to the software development. From the Wikipedia page, we can find four points.

• Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features".
• Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input.
• Design and build software, even operating systems, to be tried early, ideally within weeks. Don't hesitate to throw away the clumsy parts and rebuild them.
• Use tools in preference to unskilled help to lighten a programming task, even if you have to detour to build the tools and expect to throw some of them out after you've finished using them.

Do one thing

The first point is something many people decide to skip. The idea of a perfect system which can do everything leads to a complex over engineered software which takes many years to land in production. The first causality is the user of the system, then the team who has to maintain the system up & running. A simple system with proper documentation also attracts number of users. These first users also become your community. They try out the new features, provide valuable feedback. If you are working on an Open Source project, creating that community around your project is always important for the sustainability of the project.

Pipe and redirection between tools

Piping and redirection in Linux shells were another simple magic I learned during the early days in college. How a tool like grep can take the input stream and provide an output, which in turn can be used in the next tool, was one of the best thing I found in the terminal. As a developer, I spend a lot of time in the terminals, and we use piping and redirection in daily life many times.

Build and try and repeat

I think all of the modern agile followers know this point very well. Unless you are allowing your users to try out your tool and allow them to provide feedback, your tool will not be welcomed by the users. We, the programmers have this attitude that every problem can be solved by code, and our ideas are always the correct. No, that is not the case at all. Go out in the world, show your tool to as many people as possible, take feedback. Rewrite and rebuild your tool as required. If you wait for 3 years and hope that someone will force your tool to the users, that will not go well in long run.

Do One Thing and Do It Well

The whole idea of Do One Thing and Do It Well has been discussed many times. Search the term in your favorite search engine, and you will surely find many documents explaining the idea in details. Following this idea while designing tools or systems helped me till date. Tunir or gotun tried to follow the same ideas as much as possible. They are build to execute some command on a remote system and act accordingly to the exit codes. I think this is the one line description of both the tools. To verify if the tool is simple or not, I keep throwing the tool to the new users and go through the feedback.

Last night we received a mail from Dusty Mabe in the Fedora Cloud list, to test the updates-testing tree for Fedora Atomic. At the end of the email, he also gave the command to execute to rebase to the updates-testing tree.

# rpm-ostree rebase fedora-atomic/25/x86_64/testing/docker-host 

With that as input from upstream, it was just adding the command in one line on top of the current official Fedora Atomic tests, and followed by a reboot command and wait for the machine to come back online.

sudo rpm-ostree rebase fedora-atomic/25/x86_64/testing/docker-host
@@ sudo reboot
SLEEP 120
curl -O http://infrastructure.fedoraproject.org/infra/autocloud/tunirtests.tar.gz
tar -xzvf tunirtests.tar.gz
...

This helped me to find the regression in atomic command within the next few minutes while I was working on something else. As I reported the issue to the upstream, they are already working to find a solution (some discussion here). The simplicity of the tool helped me to get things done faster in this case.

Please let me know what do you think about this particular idea about designing software in the comments below.

Testing container is one of the major reason for the existence of the tool gotun. In this blog post, I am going to show you how easy it becomes to test a container. But, easy is still a relative term. If setting up your container requires very complex steps, then you will have to do those first. And then, finally, you can test the application. You can use Ansible to do the hard part of setting up the application.

Fedora Redis container

In our example, we will test the Fedora Redis container. Redis is an in-memory data structure store. We use it as key: value store, as a database. It can also hold some complex infrastructures.

Let me post the test commands first.

sudo docker run -d --name redis fedora/redis
sudo docker run  -d --link redis:rd --name client fedora/redis
COPY: ./redistests.py vm1:./
sudo python3 -m unittest redistests.RedisTest.test_set -v
sudo python3 -m unittest redistests.RedisTest.test_append -v
sudo docker rm -f client
sudo python3 -m unittest redistests.RedisTest.test_newclient -v

In the first line, I am starting a new redis container. In the second line, I am running a container named “client”. In the next two lines, I have two tests. I will be using this container in those two tests. Then I will delete the container, and as the part of the last test, we will create a new container, and connect to the redis server container to get the value for the key name.

In the third line, I am copying a Python3 unittest file over to the VM, and then executing the tests one by one. I wanted to check the output from the redis-cli command instead of the exit code, that is why I decided to use Python unittest module. It makes parsing the output text much easier, and we can do complex analysis of the output text. We follow the same style in the Fedora Atomic tests.

redistests.py

import unittest
import subprocess

def system(cmd):
    """
    Invoke a shell command.
    :returns: A tuple of output, err message, and return code
    """
    ret = subprocess.Popen(cmd, shell=True, stdin=subprocess.PIPE,
                stdout=subprocess.PIPE, stderr=subprocess.PIPE, close_fds=True)
    out, err = ret.communicate()
    return out.decode("utf-8"), err.decode("utf-8"), ret.returncode


class RedisTest(unittest.TestCase):

    def test_set(self):
        out, err, eid = system("docker exec client redis-cli -h rd SET name kushal")
        out, err, eid = system("docker exec client redis-cli -h rd GET name")
        self.assertEqual(out, "kushal\n")

    def test_append(self):
        system("docker exec client redis-cli -h rd DEL clients")
        out, err, eid = system("docker exec client redis-cli -h rd APPEND clients ABC")
        self.assertEqual(out, "3\n")
        out, err, eid = system("docker exec client redis-cli -h rd APPEND clients XYZ")
        self.assertEqual(out, "6\n")
        out, ere, eid = system("docker exec client redis-cli -h rd GET clients")
        self.assertEqual(out, "ABCXYZ\n")

    def test_newclient(self):
        out, err, eid = system("docker run --rm --link redis:r2d2 --name newclient fedora/redis redis-cli -h r2d2 GET name")
        self.assertEqual(out, "kushal\n")

The output from the job.

$ ./gotun --job commands
Starts a new Tunir Job.

Server ID: cb0f491d-a921-46dd-b6e1-3c4353884d6c
Let us wait for the server to be in running state.
Time to assign a floating pointip.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Executing:  sudo docker run -d --name redis fedora/redis
Executing:  sudo docker run  -d --link redis:rd --name client fedora/redis
Executing COPY:  scp -o StrictHostKeyChecking=no -r -i /home/kdas/Downloads/kushal-testday.pem ./redistests.py fedora@209.132.184.210:./
Executing:  sudo python3 -m unittest redistests.RedisTest.test_set -v
Executing:  sudo python3 -m unittest redistests.RedisTest.test_append -v
Executing:  sudo docker rm -f client
Executing:  sudo python3 -m unittest redistests.RedisTest.test_newclient -v
---------------


Result file at: /tmp/tunirresult_210751937


Job status: true


command: sudo docker run -d --name redis fedora/redis
status:true

Unable to find image 'fedora/redis:latest' locally
Trying to pull repository docker.io/fedora/redis ... 
sha256:1bb7b6c03575dd43db6e8e3fc0b06c0b3df0178af9ea382b6f9df577d053bd22: Pulling from docker.io/fedora/redis
a3ed95caeb02: Pull complete 
cadab8e68e03: Pull complete 
9db1d534feda: Pull complete 
a4d5e4eefba6: Pull complete 
Digest: sha256:1bb7b6c03575dd43db6e8e3fc0b06c0b3df0178af9ea382b6f9df577d053bd22
Status: Downloaded newer image for docker.io/fedora/redis:latest
b83c3f13fa698627447575651e585f3308e795f579ea772c66d02ae0f1c72939


command: sudo docker run  -d --link redis:rd --name client fedora/redis
status:true

6096e414e2f7c314c9601ae6eaedbb3faf9fb265b70a4f77e5ba7b9a93b19209


command: sudo python3 -m unittest redistests.RedisTest.test_set -v
status:true

test_set (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.199s

OK


command: sudo python3 -m unittest redistests.RedisTest.test_append -v
status:true

test_append (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.289s

OK


command: sudo docker rm -f client
status:true

client


command: sudo python3 -m unittest redistests.RedisTest.test_newclient -v
status:true

test_newclient (redistests.RedisTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 1.298s

OK


Total Number of Tests:6
Total NonGating Tests:0
Total Failed Non Gating Tests:0

Success.