Kushal Das

FOSS and life. Kushal Das talks here.


Networking in podman 4.x

podman 4.0 has a new networking stack. It uses Netavark for network setup (this is a direct replacement for CNI), and also uses Aardvark DNS server. Both of these tools are written from scratch in Rust keeping the requirements of podman in mind.

podman logo

At the time of writing this blog post, we have podman-4.4.1 in Fedora 37, and podman-4.2.0 in Almalinux9.

Communication between two rootless containers

The default network for podman is called podman, this does not allow DNS based access between containers.

$ podman network ls
2f259bab93aa  podman      bridge

$ podman network inspect podman
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2023-02-20T07:36:58.054055322Z",
          "subnets": [
                    "subnet": "",
                    "gateway": ""
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"

This means if we start two containers, they will not be able to communicate with each other via their names.

The solution is to create a new network and use it.

$ podman network create project1

$ podman network inspect project1
          "name": "project1",
          "id": "1f0135a4fc3b1e58c1c8fcac762b15eb89a755959a4896fd321fa17f991de9fa",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2023-02-17T22:19:22.80494367Z",
          "subnets": [
                    "subnet": "",
                    "gateway": ""
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"

Noticed the dns_enabled is now true.

Let us test this out. We

$ podman run --rm -it --network project1 --name server42 fedora:37
[root@fc1869d78823 /]# cd /tmp/
[root@fc1869d78823 tmp]# mkdir hello
[root@fc1869d78823 tmp]# cd hello/
[root@fc1869d78823 hello]# echo "SELinux for win." > index.html
[root@fc1869d78823 hello]# python3 -m http.server
Serving HTTP on port 8000 ( ...

When we start this container, podman starts aardvark-dns automatically.

$ ps aux | grep aard
almalin+    1205  0.0  0.0 276428   212 ?        Ssl  Feb18   0:00 /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run

Now, we can start a second container on the same network and use the magical tool curl to fetch the data.

$ podman run --rm -it --network project1 fedora:37
[root@720fc9e63d72 /]# curl http://server42:8000/
SELinux for win.

As I heard, from the next release (4.5.0) of podman, we will be able to use DNS based communication even in the default network.

Reproducible wheel buidling failure on CircleCI container

At SecureDrop project we have Python wheels built for Python 3.7 on Buster in a reproducible way. We use the same wheels inside of the Debian packages. The whole process has checks to verify the sha256sums based on gpg signatures, and at the end we point pip to a local directory to find all the dependencies.

Now, I am working to update the scripts so that we can build wheels for Ubuntu Focal, and in future we can use those wheels in the SecureDrop server side packages. While working on this, in the CI I suddenly noticed that all wheels started failing on the reproducibility test on Buster. I did not make any change other than the final directory path, so was wondering what is going on. Diffoscope helped to find out the difference:

│ -6 files, 34110 bytes uncompressed, 9395 bytes compressed:  72.5%
│ +6 files, 34132 bytes uncompressed, 9404 bytes compressed:  72.4%
├── six-1.11.0.dist-info/METADATA
│ @@ -9,14 +9,15 @@
│  Platform: UNKNOWN
│  Classifier: Programming Language :: Python :: 2
│  Classifier: Programming Language :: Python :: 3
│  Classifier: Intended Audience :: Developers
│  Classifier: License :: OSI Approved :: MIT License
│  Classifier: Topic :: Software Development :: Libraries
│  Classifier: Topic :: Utilities
│ +License-File: LICENSE
│  .. image:: http://img.shields.io/pypi/v/six.svg
│     :target: https://pypi.python.org/pypi/six
│  .. image:: https://travis-ci.org/benjaminp/six.svg?branch=master
│      :target: https://travis-ci.org/benjaminp/six
├── six-1.11.0.dist-info/RECORD
│ @@ -1,6 +1,6 @@
│  six.py,sha256=A08MPb-Gi9FfInI3IW7HimXFmEH2T2IPzHgDvdhZPRA,30888
│  six-1.11.0.dist-info/LICENSE,sha256=Y0eGguhOjJj0xGMImV8fUhpohpduJUIYJ9KivgNYEyg,1066
│ -six-1.11.0.dist-info/METADATA,sha256=vfvF0GW2vCjz99oMyLbw15XSkmo1IxC-G_339_ED4h8,1607
│ +six-1.11.0.dist-info/METADATA,sha256=Beq9GTDD6nYVwLYrN3oOcts0HSPHotfRWQ_Zn8_9a7g,1629
│  six-1.11.0.dist-info/WHEEL,sha256=Z-nyYpwrcSqxfdux5Mbn_DQ525iP7J2DG3JgGvOYyTQ,110
│  six-1.11.0.dist-info/top_level.txt,sha256=_iVH_iYEtEXnD8nYGQYpYFUvkUW9sEO1GYbkeKSAais,4
│  six-1.11.0.dist-info/RECORD,,

It seems somehow the packages were picking up metadata about the LICENSE file. After looking into the environment, I found virtualenv is pulling latest setuptools into the virtualenv. Thus breaking the reproducibility. It was a quick fix. Yes, we do pin setuptools and pip in our wheels repository.

Next, the extension based wheels started failing, and I was going totally crazy to find out why there is some ABI change. I still don't know the correct reason, but noticed that the circleci/python:3.7-buster container image (and the upstream Python container) is using different flags to build the extensions than Debian Buster on vm/bare metal. For example, below on the top we have the command line from the container and then inside of the normal Debian Buster.

creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/home/circleci/project/.venv/include -I/usr/local/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
gcc -pthread -shared build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -L/usr/local/lib -lpython3.7m -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so

creating build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/processors.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/processors.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cprocessors.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cresultproxy' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/resultproxy.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/resultproxy.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cresultproxy.cpython-37m-x86_64-linux-gnu.so
building 'sqlalchemy.cutils' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/home/kdas/code/securedrop-debian-packaging/.venv/include -I/usr/include/python3.7m -c lib/sqlalchemy/cextension/utils.c -o build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.7/lib/sqlalchemy/cextension/utils.o -o build/lib.linux-x86_64-3.7/sqlalchemy/cutils.cpython-37m-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel

If you want to see the difference in the flags, you can try out the following command:

python3 -m sysconfig

Look for PY_CFLAGS and related flags in the command output. Reproducibility depends on the environment, and we should not expect the latest container image (with the latest Python 3.7.x release) creating the same thing like in Debian Buster, but the failure started recently, we did not notice this a few months. Also, this means in future we should enable reproducibility tests on nightlies in CI.

Using hexchat on Flatpak on Qubes OS AppVM

Flatpak is a system for building, distributing, and running sandboxed desktop applications on Linux. It uses BubbleWrap in the low level to do the actual sandboxing. In simple terms, you can think Flatpak as a as a very simple and easy way to use desktop applications in containers (sandboxing). Yes, containers, and, yes, it is for desktop applications in Linux. I was looking forward to use hexchat-otr in Fedora, but, it is not packaged in Fedora. That is what made me setup an AppVM for the same using flatpak.

I have installed the flatpak package in my Fedora 29 TemplateVM. I am going to use that to install Hexchat in an AppVM named irc.

Setting up the Flatpak and Hexchat

The first task is to add flathub as a remote for flatpak. This is a store where upstream developers package their application and publish.

flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo

And then, I installed the Hexchat from the store. I also installed the version of the OTR plugin required.

$ flatpak install flathub io.github.Hexchat
<output snipped>

$ flatpak install flathub io.github.Hexchat.Plugin.OTR//18.08
Installing in system:
io.github.Hexchat.Plugin.OTR/x86_64/18.08 flathub 6aa12f19cc05
Is this ok [y/n]: y
Installing: io.github.Hexchat.Plugin.OTR/x86_64/18.08 from flathub
[####################] 10 metadata, 7 content objects fetched; 268 KiB transferr
Now at 6aa12f19cc05.

Making sure that the data is retained after reboot

All of the related files are now available under /var/lib/flatpak. But, as this is an AppVM, this will get destroyed when I will reboot. So, I had to make sure that I can keep those between reboots. We can use the Qubes bind-dirs for this in the TemplateVMs, but, as this is particular for this VM, I just chose to use simple shell commands in the /rw/config/rc.local file (make sure that the file is executable).

But, first, I moved the flatpak directory under /home.

sudo mv /var/lib/flatpak /home/

Then, I added the following 3 lines in the /rw/config/rc.local file.

# For flatpak
rm -rf /var/lib/flatpak
ln -s /rw/home/flatpak /var/lib/flatpak

This will make sure that the flatpak command will find the right files even after reboot.

Running the application is now as easy as the following command.

flatpak run io.github.Hexchat

Feel free to try out other applications published in Flathub, for example, Slack or the Mark Text

Running Tor relay inside a docker container

The latest Tor project release is But, that is not available on all the different versions of different Linux distributions. For example, CentOS 7 has tor-, and only Fedora 28 has the latest Tor.

This is where a container can help. The official builds from Tor Project are for Debian, means we can build and use a Debian based container.

The Dockerfile

FROM debian:stretch
LABEL MAINTAINER Kushal Das <mail@kushaldas.in>

RUN apt-get update
RUN apt install vim gpg -y

RUN echo "deb http://deb.torproject.org/torproject.org stretch main\ndeb-src http://deb.torproject.org/torproject.org stretch main" > /etc/apt/sources.list.d/tor.list

# Let us get the key for Tor
RUN gpg --keyserver keys.gnupg.net --recv A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89
RUN gpg --export A3C4F0F979CAA22CDBA8F512EE8CBC9E886DDD89 > tor.gpg
RUN apt-key add tor.gpg

# Finally install Tor
RUN apt update
RUN apt install tor deb.torproject.org-keyring -y
ADD ./torrc /etc/tor/torrc

# Add the tor user
RUN groupadd -g 1000 tor && useradd -m -d /home/tor -g 1000 tor

# Now drop to the actual user
USER tor
RUN mkdir -p /home/tor/.tor/keys

VOLUME ["/home/tor/.tor"]

EXPOSE 9001 9051


I have a configuration file named torrc, you can copy the sample configuration and edit as required. I have the following entries there.


ORPort 9001



ExitRelay 0

Next, we will create a directory in the host system to keep the keys, and other files. We want to restart the container and still have the same details, mounting a directory from the host system into the container will help us in that.

mkdir /mnt/tor
chcon -R -t svirt_sandbox_file_t /mnt/tor

Please also make sure that you have correct ownership of that directory.

Running the container

docker run -d -v /etc/localtime:/etc/localtime -v /mnt/tor:/home/tor/.tor --restart always -p 9001:9001 -name relay kushaldas/tor:

After starting the container, you can check the logs for any error. If you can see the following message in the log, then it means that you configured the relay properly.

# docker logs -f relay

Self-testing indicates your ORPort is reachable from the outside.

The official relay guide

Tor project recently published an updated relay guide for anyone new to running a relay. Please go through that document first. If you need help, there is a mailing list of Tor relay operators, and #tor channel on OFTC IRC server is also very welcoming.

Testing Fedora Atomic Images using upstream Atomic Host tests

Project Atomic has a group of tests written in Ansible. In this blog post, I am going to show to use those along with gotun. I will be running the improved-sanity-test as suggested in the #atomic channel on Freenode.


The following is the content of the createinventory.py file. We use this file to generate the inventory file for Ansible.

#!/usr/bin/env python

import json

data = None
with open("current_run_info.json") as fobj:
    data = json.loads(fobj.read())

user = data['user']
host1 = data['vm1']
key = data['keyfile']

result = """{0} ansible_ssh_host={1} ansible_ssh_user={2} ansible_ssh_private_key_file={3}
""".format(host1, host1,user,key)
with open("./inventory", "w") as fobj:

gotun job configuration and the actual job file

BACKEND: "openstack"

OS_AUTH_URL: "https://fedorainfracloud.org:5000/v2.0"
OS_IMAGE: "Fedora-Atomic-25-20170124.1.x86_64.qcow2"
OS_FLAVOR: "m1.medium"
    - "ssh-anywhere-cloudsig"
    - "default"
OS_FLOATING_POOL: "external"
OS_KEYPAIR: "kushal-test"
key: "/home/kdas/kushal-test.pem"

Above fedora.yml configuration boots up the VM for us in the OpenStack environment. Then we have the actual test file called fedora.txt.

HOSTCOMMAND: ./createinventory.py
HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml

Now when we run the job (remember, it takes a lot of time to run).

$ gotun --job fedora
Starts a new Tunir Job.

Server ID: 3e8cd0c7-bc79-435e-9cf9-169c5bc66b3a
Let us wait for the server to be in running state.
Time to assign a floating point IP.
Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Polling for a successful ssh connection.

Executing:  HOSTTEST: ansible-playbook -i inventory tests/improved-sanity-test/main.yml

Result file at: /tmp/tunirresult_326996419

Job status: true

command: ansible-playbook -i inventory tests/improved-sanity-test/main.yml

PLAY [Improved Sanity Test - Pre-Upgrade] **************************************

TASK [setup] *******************************************************************
ok: []

TASK [ansible_version_check : Fail if avc_major is not defined] ****************
skipping: []

TASK [ansible_version_check : Fail if avc_minor is not defined] ****************
skipping: []

TASK [ansible_version_check : Check if Ansible is the correct version (or newer)] ***
ok: [] => {
    "changed": false, 
    "msg": "All assertions passed"

TASK [atomic_host_check : Determine if Atomic Host] ****************************
ok: []


TASK [var_files_present : Check for correct SELinux type] **********************
changed: []

PLAY RECAP *********************************************************************            : ok=260  changed=170  unreachable=0    failed=0   

[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
 [WARNING]: Consider using yum, dnf or zypper module rather than running rpm
 [WARNING]: Consider using get_url or uri module rather than running curl
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: always_run is deprecated. Use check_mode = no instead..

This feature will be removed in version 2.4. Deprecation warnings can be 
disabled by setting deprecation_warnings=False in ansible.cfg.

Total Number of Tests:1
Total NonGating Tests:0
Total Failed Non Gating Tests:0


Previously, I blogged about how to use the upstream Kubernetes Ansible repo to test Kubernetes on Atomic images. Using this style, you can use any ansible playbook to do the real setup, and then use Ansible to test for you, or have your own set of test cases. Do let me know what do you think in the comments.

rkt image build command reference

In my last post, I wrote about my usage of rkt. I have also posted the basic configuration to create your own container images. Today we will learn more about those various build commands of the .acb files. We use these commands with the acbuild tool.


begin starts a new build. The build information is stored inside the .acbuild directory in the current directory. By default, it starts with an empty rootfs. But we can pass some options to change that behavior. We can start with either a local filesystem, or a local aci image, or even from a remote aci image. To create the Fedora 25 aci image, I extracted the rootfs on a local directory and used that with begin command. Examples:

begin /mnt/fedora
begin ./fedora-25-linux-amd64.aci


dep command is used to add any separate aci as a dependency to the current aci. In the rootfs the current aci will be on top of any dependency image. The order of the dependencies is important, so keep an eye to that while working on a new aci image. For example to build any image on top of the Fedora aci image we use the following line

dep add kushal.fedorapeople.org/rkt/fedora:25


We can execute any command inside the container we are building using the run command. For example to install a package using dnf we will use the following line:

run -- dnf install htop -y

The actual command (which will run inside the container) is after --, anything before that is considered part of the dep command itself.


We can also add or remove any environment variable in the container image. We use environment command for the same.

environment add HOME /mnt
environment add DATAPATH /opt/data


copy command is used to copy a file or a directory from the local filesystem to the aci image. For example, here we are coping dnf.conf file to the /etc/dnf/ directory inside the container image.

copy ./dnf.conf /etc/dnf/dnf.conf


We use mount command to mark a location in the aci image which should be mounted while running the container. Remember one thing about mount points (this is true for ports too), they worked based on the name you give. Here, we are creating a mount point called apphome and then the next command we are actually specifying the host mount point for the same.

mount add apphome /opt/app/data
rkt run --volume apphome,kind=host,source=/home/kushal/znc,readOnly=false my-image.aci


Similar to the mount command, we can use the port command to mark any port of the container which can be mapped to the host system. We need to specify a name, the protocol (can be either udp or tcp) and finally the port number. We use the provided name to map it to a host port in the host.

port add http tcp 80
port add https tcp 443


set-user command specifies the user which will be used in the container environment.

set-user kushal

Remember to create the user before you try to use it.


Similar to the set-user command, it specifies the group which will be used to run the application inside the container.


set-working-directory is used to set the working directory for the application inside the container.

set-working-directory /opt/data


Using set-exec we specify a command to run as the application. In the below example we are running the znc command as the application in the container.

set-exec -- /usr/bin/znc --foreground


The final command for today is write. Using this command we create the final image from the current build environment. There is --overwrite flag, using which we can overwrite the image file we are creating.

write --overwrite znc-latest-linux-amd64.aci

I hope this post will help to understand the build commands, and you can use the same to build your own rkt images. In future, if I need to find the command reference, I can read this blog post itself.

Using rkt on my Fedora servers

Many of you already know that I moved all my web applications into containers on Fedora Atomic image based hosts. In the last few weeks, I moved a few of them from Docker to rkt on Fedora 25. I have previously written about trying out rkt in Fedora. Now I am going to talk about how can we build our own rkt based container images, and then use them in real life.

Installation of rkt

First I am going to install all the required dependencies, I added htop and tmux and vim on the list because I love to use them :)

$ sudo dnf install systemd-container firewalld vim htop tmux gpg wget
$ sudo systemctl enable firewalld
$ sudo systemctl start firewalld
$ sudo firewall-cmd --add-source= --zone=trusted
$ sudo setenforce Permissive

As you can see in the above-mentioned commands, rkt still does not work well with the SELinux on Fedora. We hope this problem will be solved soon.

Then install the rkt package as described in the upstream document.

$ sudo rkt run --interactive --dns= --insecure-options=image kushal.fedorapeople.org/rkt/fedora:25

The above-mentioned command downloads the Fedora 25 image I built and then executes the image. This is the base image for all of my other work images. You may not have to provide the DNS value, but I prefer to do so. The --interactive provides you an interactive prompt. If you forget to provide this flag on the command line, then your container will just exit. I was confused for some time and was trying to find out what was going on.

Building our znc container image

Now the next step is to build our own container images for particular applications. In this example first I am going to build one for znc. To build the images we will need acbuild tool. You can follow the instructions here to install it in the system.

I am assuming that you have your znc configuration handy. If you are installing for the first time, you can generate your configuration with the following command.

$ znc --makeconf

Now below is the znc.acb file for my znc container. We can use acbuild-script tool to create the container from this image.

#!/usr/bin/env acbuild-script

# Start the build with an empty ACI

# Name the ACI
set-name kushal.fedorapeople.org/rkt/znc
dep add kushal.fedorapeople.org/rkt/fedora:25

run -- dnf update -y
run -- dnf install htop vim znc -y
run -- dnf clean all

mount add znchome /home/fedora/.znc
port add znc tcp 6667

run --  groupadd -r fedora -g 1000 
run -- useradd -u 1000 -d /home/fedora -r -g fedora fedora

set-user fedora

set-working-directory /home/fedora/
set-exec -- /usr/bin/znc --foreground 

# Write the result
write --overwrite znc-latest-linux-amd64.aci

If you look closely to the both mount and port adding command, you will see that I have assigned some name to the mount point, and also to the port (along with the protocol). Remember that in the rkt world, all mount points or ports work based on these assigned names. So, for one image HTTP name can be assigned to the standard port 80, but in another image, the author can choose to use port 8080 with the same name. While running the image we choose to decide how to map the names to the host side or vice-versa. Execute the following command to build our first image.

$ sudo acbuild-script znc.acb

If everything goes well, you will find an image named znc-latest-linux-amd64.aci in the current directory.

Running the container

$ sudo rkt --insecure-options=image --debug run --dns=  --set-env=HOME=/home/fedora --volume znchome,kind=host,source=/home/kushal/znc,readOnly=false  --port znc:8010 znc-latest-linux-amd64.aci

Now let us dissect the above command. I am using --insecure-options=image option as I am not verifying the image, --debug flag helps to print some more output on the stdout. This helps to find any problem with a new image you are building. As I mentioned before I passed a DNS entry to the container using --dns= Next, I am overriding the $HOME environment value, I still have to dig more to find why it was pointing to /root/, but for now we will remember that --set-env can help us to set/override any environment inside the container.

Next, we mount /home/kushal/znc directory (which has all the znc configuration) in the mount name znchome and also specifying that it is not a readonly mount. In the same way we are doing a host port mapping of 8010 to the port named znc inside of the container. As the very last argument, I passed the image itself.

The following is the example where I am copying a binary (the ircbot application written in golang) into the image.

#!/usr/bin/env acbuild-script

# Start the build with an empty ACI

# Name the ACI
set-name kushal.fedorapeople.org/rkt/ircbot
dep add kushal.fedorapeople.org/rkt/fedora:25

copy ./ircbot /usr/bin/ircbot

mount add mnt /mnt

set-working-directory /mnt
set-exec -- /usr/bin/ircbot

# Write the result
write --overwrite ircbot-latest-linux-amd64.aci

In future posts, I will explain how can you run the containers as systemd services. For starting, you can use a tmux session to keep them running. If you have any doubt, remember to go through the rkt documents. I found them very informative. You can also try to ask your doubts in the #rkt channel on Freenode.net.

Now it is an exercise for the reader to find out the steps to create an SELinux module from the audit log, and then use the same on the system. The last step should be putting the SELinux back on Enforcing mode.

$ sudo setenforce Enforcing

One week with Fedora Atomic in production

I was using containers for over a year in my personal servers. I was running a few services in those. For the last one week, I moved all my personal servers into Fedora Atomic, and running more number of services in those.

Server hardware & OS

These are actually VM(s) with couple of GB(s) of RAM, and a few CPU(s). I installed using the Fedora Atomic ISO image (get it from here) over virt-manager.

The containers inside

You can find all the Dockerfiles etc in the repo. Note: I still have to clean up a few of those.

Issues faced

In day zero the first box I installed, stopped printing anything on STDOUT, after a reboot I upgraded with atomic host upgrade command. I never had any other problem still now. So, try to stay updated.

Building my own rpm-ostree repo

My next target was to compose my own rpm-ostree repo. I used Patrick's workstation repo files for the same. In my fork I added couple of files for my own tree, and the build script. The whole work is done on a Fedora 24 container. You can view the repo here. This whole thing is exposed via another apache container. I will explain more about the steps in a future blog post.

What is next?

First step is clean up my old Dockerfiles. I will add up any future service as containers in those boxes. Even though we are automatically testing our images using Autocloud, using this in my production environment will help me find bugs in more convenient manner.

Trying out rkt v1.14.0 on Fedora 24

Few days back we had rkt v1.14.0 release from CoreOS. You can read details about the release in their official blog post. I decided to give it a try on a Fedora 24 box, I followed the official documentation. The first step was to download rkt, and acbuild tools.

To download, and install acbuild tool, I did the following: (Btw, as it was a cloud instance, I just moved the binaries to my sbin path)

$ wget https://github.com/containers/build/releases/download/v0.4.0/acbuild-v0.4.0.tar.gz
$ tar xzvf acbuild-v0.4.0.tar.gz
$ sudo mv acbuild-v0.4.0/* /usr/sbin/

Now for rkt, do the following.

$ wget https://github.com/coreos/rkt/releases/download/v1.14.0/rkt-v1.14.0.tar.gz
$ tar xzvf rkt-v1.14.0.tar.gz
$ cd rkt-v1.14.0
$ ./rkt help
$ sudo cp -r init/systemd/* /usr/lib/systemd/

Now I had to modify a path inside ./scripts/setup-data-dir.sh file, at line 58 I wrote the following.

systemd-tmpfiles --create /usr/lib/systemd/tmpfiles.d/rkt.conf

Next step is to execute the script. This will create the required directories, and fix the permission issues. Before that I will also create a group, and add my current user to the group. Remember to logout, and login again for it.

$ sudo groupadd rkt
$ export WHOAMI=$(whoami); sudo gpasswd -a $WHOAMI rkt
$ sudo ./scripts/setup-data-dir.sh

rkt documentation suggests to disable SELinux for trying out, I tried to run it with SELinux in Enforcing mode, and then created local policy based on the errors. I have also opened a bug for the rawhide package.

# ausearch -c 'systemd' --raw | audit2allow -M localrktrawhide
# semodule -i localrktrawhide.pp

After all this we are finally in a state to start using rkt in the system.

The Try out document says to trust the signing key of etcd, I am going to do that, and then test by fetchin the image.

$ sudo ./rkt trust --prefix coreos.com/etcd
$ ./rkt fetch coreos.com/etcd:v2.3.7
image: searching for app image coreos.com/etcd
image: remote fetching from URL "https://github.com/coreos/etcd/releases/download/v2.3.7/etcd-v2.3.7-linux-amd64.aci"
image: keys already exist for prefix "coreos.com/etcd", not fetching again
image: downloading signature from https://github.com/coreos/etcd/releases/download/v2.3.7/etcd-v2.3.7-linux-amd64.aci.asc
Downloading signature: [=======================================] 490 B/490 B
Downloading ACI: [=============================================] 8.52 MB/8.52 MB
image: signature verified:
  CoreOS Application Signing Key <security@coreos.com>

You can view the images, with a image list subcommand.

$ ./rkt image list
ID                      NAME                                    SIZE    IMPORT TIME     LAST USED
sha512-5f362df82594     coreos.com/rkt/stage1-coreos:1.14.0     162MiB  1 day ago       1 day ago
sha512-86450bda7ae9     example.com/hello:0.0.1                 7.2MiB  15 hours ago    15 hours ago
sha512-7d28419b27d5     coreos.com/etcd:v2.3.7                  31MiB   48 seconds ago  48 seconds ago

From here, you can just follow the getting started guide. I used the debug flag to see what is going on.

$ sudo ./rkt --insecure-options=image --debug run ../hello/hello-0.0.1-linux-amd64.aci                                
image: using image from local store for image name coreos.com/rkt/stage1-coreos:1.14.0
image: using image from file ../hello/hello-0.0.1-linux-amd64.aci
stage0: Preparing stage1
stage0: Writing image manifest
stage0: Loading image sha512-86450bda7ae972c9507007bd7dc19a386011a8d865698547f31caba4898d1ebe
stage0: Writing image manifest
stage0: Writing pod manifest
stage0: Setting up stage1
stage0: Wrote filesystem to /var/lib/rkt/pods/run/d738b5e3-3fe9-4beb-ae5c-3e8f4153ee57
stage0: Pivoting to filesystem /var/lib/rkt/pods/run/d738b5e3-3fe9-4beb-ae5c-3e8f4153ee57
stage0: Execing /init
networking: loading networks from /etc/rkt/net.d
networking: loading network default with type ptp
Spawning container rkt-d738b5e3-3fe9-4beb-ae5c-3e8f4153ee57 on /var/lib/rkt/pods/run/d738b5e3-3fe9-4beb-ae5c-3e8f4153ee57/stage1/rootfs.
Press ^] three times within 1s to kill container.
Detected virtualization rkt.
Detected architecture x86-64.

Welcome to Linux!

Set hostname to <rkt-d738b5e3-3fe9-4beb-ae5c-3e8f4153ee57>.
[  OK  ] Listening on Journal Socket.
[  OK  ] Created slice system.slice.
         Starting Create /etc/passwd and /etc/group...
[  OK  ] Created slice system-prepare\x2dapp.slice.
[  OK  ] Started Pod shutdown.
[  OK  ] Started hello Reaper.
[  OK  ] Listening on Journal Socket (/dev/log).
         Starting Journal Service...
[  OK  ] Started Create /etc/passwd and /etc/group.
[  OK  ] Started Journal Service.
         Starting Prepare minimum environment for chrooted applications...
[  OK  ] Started Prepare minimum environment for chrooted applications.
[  OK  ] Started Application=hello Image=example.com/hello.
[  OK  ] Reached target rkt apps target.
[111534.724440] hello[5]: 2016/09/10 14:48:59 request from

While the above container was running, I tested it out from another terminal, and then stopped it.

$ ./rkt list
UUID            APP     IMAGE NAME              STATE   CREATED         STARTED         NETWORKS
865b862e        hello   example.com/hello:0.0.1 running 8 seconds ago   8 seconds ago   default:ip4=
$ curl
$ sudo ./rkt stop 865b862e

I hope this post will help you to try out rkt on a Fedora system. Feel free to comment if you have question, or ask over twitter.