Kushal Das

FOSS and life. Kushal Das talks here.


Networking in podman 4.x

podman 4.0 has a new networking stack. It uses Netavark for network setup (this is a direct replacement for CNI), and also uses Aardvark DNS server. Both of these tools are written from scratch in Rust keeping the requirements of podman in mind.

podman logo

At the time of writing this blog post, we have podman-4.4.1 in Fedora 37, and podman-4.2.0 in Almalinux9.

Communication between two rootless containers

The default network for podman is called podman, this does not allow DNS based access between containers.

$ podman network ls
2f259bab93aa  podman      bridge

$ podman network inspect podman
          "name": "podman",
          "id": "2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9",
          "driver": "bridge",
          "network_interface": "podman0",
          "created": "2023-02-20T07:36:58.054055322Z",
          "subnets": [
                    "subnet": "",
                    "gateway": ""
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": false,
          "ipam_options": {
               "driver": "host-local"

This means if we start two containers, they will not be able to communicate with each other via their names.

The solution is to create a new network and use it.

$ podman network create project1

$ podman network inspect project1
          "name": "project1",
          "id": "1f0135a4fc3b1e58c1c8fcac762b15eb89a755959a4896fd321fa17f991de9fa",
          "driver": "bridge",
          "network_interface": "podman1",
          "created": "2023-02-17T22:19:22.80494367Z",
          "subnets": [
                    "subnet": "",
                    "gateway": ""
          "ipv6_enabled": false,
          "internal": false,
          "dns_enabled": true,
          "ipam_options": {
               "driver": "host-local"

Noticed the dns_enabled is now true.

Let us test this out. We

$ podman run --rm -it --network project1 --name server42 fedora:37
[root@fc1869d78823 /]# cd /tmp/
[root@fc1869d78823 tmp]# mkdir hello
[root@fc1869d78823 tmp]# cd hello/
[root@fc1869d78823 hello]# echo "SELinux for win." > index.html
[root@fc1869d78823 hello]# python3 -m http.server
Serving HTTP on port 8000 ( ...

When we start this container, podman starts aardvark-dns automatically.

$ ps aux | grep aard
almalin+    1205  0.0  0.0 276428   212 ?        Ssl  Feb18   0:00 /usr/libexec/podman/aardvark-dns --config /run/user/1000/containers/networks/aardvark-dns -p 53 run

Now, we can start a second container on the same network and use the magical tool curl to fetch the data.

$ podman run --rm -it --network project1 fedora:37
[root@720fc9e63d72 /]# curl http://server42:8000/
SELinux for win.

As I heard from the next release (4.5.0) of podman, we will be able to use DNS based communication even in the default network.

Setting up local mTLS environment using mkcert

mTLS or mutual TLS is a way of doing mutual authentication. When we talk about TLS in general, we only about TLS for the servers/services. There the clients can verify that they are connected to the right server. But, the server does not know much about the clients themselves. This can be done via mTLS, say for services talking to each other. To know more please read the Cloudflare writeup on mTLS.

In this blog post we will see how we can use the mkcert from Filippo Valsorda to setup a local environment, so that you can play-around & learn.

Install nss-tools package for your system

For Fedora, I installed it via dnf.

$ sudo dnf install nss-tools -y

Getting mkcert

I grabbed the latest release from the gitub release page.

$ wget https://github.com/FiloSottile/mkcert/releases/download/v1.4.3/mkcert-v1.4.3-linux-amd64
$ mv mkcert-v1.4.3-linux-amd64 ~/bin/mkcert
$ chmod +x ~/bin/mkcert

Setting up the local CA

$ mkcert -install
Created a new local CA 💥
The local CA is now installed in the system trust store! ⚡️
The local CA is now installed in the Firefox trust store (requires browser restart)! 🦊

This will create two important files inside of your user home directory.

❯ ls -l .local/share/mkcert/
.r--------@ 2.5k kdas 20 Dec 12:14 rootCA-key.pem
.rw-r--r--@ 1.8k kdas 20 Dec 12:14 rootCA.pem

Note:: The rootCA-key.pem is an important file and it can allow people to decrypt traffic from your system. Do not share or randomly copy it around.

The rootCA.pem file contains the public key, we can use the openssl tool to inspect it.

❯ openssl x509 -text -noout -in ~/.local/share/mkcert/rootCA.pem
        Version: 3 (0x2)
        Serial Number:
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: O = mkcert development CA, OU = kdas@localhost.localdomain (Kushal Das), CN = mkcert kdas@localhost.localdomain (Kushal Das)
            Not Before: Dec 20 11:14:33 2021 GMT
            Not After : Dec 20 11:14:33 2031 GMT
        Subject: O = mkcert development CA, OU = kdas@localhost.localdomain (Kushal Das), CN = mkcert kdas@localhost.localdomain (Kushal Das)
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (3072 bit)
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Certificate Sign
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier: 
    Signature Algorithm: sha256WithRSAEncryption

If you look closely at the X509v3 extensions section of the output, you will notice two important things:

  • It is a CA certificate
  • pathlen 0 means it can not sign/create any new CA but only sign leaf certificates. Do man x509v3_config to learn more.

Setting up certificate for local development

❯ cd ~/code/mtls-example
❯ mkcert localhost ::1

Created a new certificate valid for the following names 📜
 - "localhost"
 - ""
 - "::1"

The certificate is at "./localhost+2.pem" and the key at "./localhost+2-key.pem" ✅

It will expire on 20 March 2024 🗓

Starting a nginx podman container with the certificate

Next we will start a podman nginx container to try to out the certificate. On my Fedora machine, I will also have take care of SELinux. First let us create a default.conf.

server {
  listen [::]:443 ssl http2 ipv6only=on;
  listen 443 ssl http2;
  server_name  localhost;
  ssl_protocols TLSv1.3;
  ssl_certificate /etc/nginx/conf.d/localhost+2.pem;
  ssl_certificate_key /etc/nginx/conf.d/localhost+2-key.pem;

  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
    root   /usr/share/nginx/html;

Then, I will copy the rootCA.pem file in the current directory and start the container.

❯ cp ~/.local/share/mkcert/rootCA.pem .
❯ chcon -Rt svirt_sandbox_file_t .
❯ podman run --rm -p 8080:443 -v $PWD:/etc/nginx/conf.d/ nginx

and from another terminal I can verify the setup using curl.

❯ curl --tlsv1.3 https://localhost:8080
<!DOCTYPE html>
<title>Welcome to nginx!</title>
html { color-scheme: light dark; }
body { width: 35em; margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif; }
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>

<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>

<p><em>Thank you for using nginx.</em></p>

Same via Python httpx module (includes commands to create and activate the Python virtualenv).

❯ python3 -m venv .venv
❯ source .venv/bin/activate
❯ python3 -m pip install httpx
>>> import httpx
>>> r = httpx.get("https://localhost:8080/", verify="./rootCA.pem")
>>> r
<Response [200 OK]>

Now let us enable client side certificate and verification in nginx

We will modify the default.conf to the following.

server {
  listen [::]:443 ssl http2 ipv6only=on;
  listen 443 ssl http2;
  server_name  localhost;
  ssl_protocols TLSv1.3;
  ssl_certificate /etc/nginx/conf.d/localhost+2.pem;
  ssl_certificate_key /etc/nginx/conf.d/localhost+2-key.pem;
  ssl_client_certificate /etc/nginx/conf.d/rootCA.pem;
  ssl_verify_client on;
  ssl_verify_depth  3;

  location / {
    root   /usr/share/nginx/html;
    index  index.html index.htm;
  error_page   500 502 503 504  /50x.html;
  location = /50x.html {
    root   /usr/share/nginx/html;

and restart the podman container.

Now, let us try the same curl command and Python code.

❯ curl --tlsv1.3 https://localhost:8080
<head><title>400 No required SSL certificate was sent</title></head>
<center><h1>400 Bad Request</h1></center>
<center>No required SSL certificate was sent</center>
>>> r = httpx.get("https://localhost:8080/", verify="./rootCA.pem")
>>> r
<Response [400 Bad Request]>

Creating a client side certificate and using the same

Here we are saying to use the name nehru in the client certificate. Note: I am running the commands in a different day, that is why the dates will not match with the CA certificate date :)

❯ mkcert -client nehru

Created a new certificate valid for the following names 📜
 - "nehru"

The certificate is at "./nehru-client.pem" and the key at "./nehru-client-key.pem" ✅

It will expire on 22 March 2024 🗓

If you use the openssl x509 -text -noout -in ./nehru-client.pem and check the details of the certificate, you will notice the following:

        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication

Next, we will use the same certificates in curl.

❯ curl --tlsv1.3 --key nehru-client-key.pem --cert nehru-client.pem https://localhost:8080

And then in Python.

>>> cert = ("./nehru-client.pem", "./nehru-client-key.pem")
>>> r = httpx.get("https://localhost:8080/", verify="./rootCA.pem", cert=cert)
>>> r
<Response [200 OK]>

I hope this will help you to start trying out mTLS on your local development environment. In future posts we will learn more in depth examples.

Running SecureDrop inside of podman containers on Fedora 33

Last week, while setting up a Fedora 33 system, I thought of running the SecureDrop development container there, but using podman instead of the Docker setup we have.

I tried to make minimal changes to our existing scripts. Added a ~/bin/docker file, with podman $@ inside (and the sha-bang line).

Next, I provided the proper label for SELinux:

sudo chcon -Rt container_file_t securedrop

The SecureDrop container runs as the normal user inside of the Docker container. I can not do the same here as the filesystem gets mounted as root, and I can not write in it. So, had to modify one line in the bash script, and also disabled another function call which deletes the /dev/random file inside of the container.

diff --git a/securedrop/bin/dev-shell b/securedrop/bin/dev-shell
index ef424bc01..37215b551 100755
--- a/securedrop/bin/dev-shell
+++ b/securedrop/bin/dev-shell
@@ -72,7 +72,7 @@ function docker_run() {
            -e LANG=C.UTF-8 \
            -e PAGE_LAYOUT_LOCALES \
            -e PATH \
-           --user "${USER:-root}" \
+           --user root \
            --volume "${TOPLEVEL}:${TOPLEVEL}" \
            --workdir "${TOPLEVEL}/securedrop" \
            --name "${SD_CONTAINER}" \
diff --git a/securedrop/bin/run b/securedrop/bin/run
index e82cc6320..0c11aa8db 100755
--- a/securedrop/bin/run
+++ b/securedrop/bin/run
@@ -9,7 +9,7 @@ cd "${REPOROOT}/securedrop"
 source "${BASH_SOURCE%/*}/dev-deps"
 run_redis &
 run_sass --watch &

This time I felt that build time for verifying each cached layer is much longer than what it used to be for podman. Maybe I am just mistaken. The SecureDrop web application is working very fine inside.

Package build containers

We also use containers to build Debian packages. And those molecule scenarios were failing as ansible.posix.synchronize module could not sync to a podman container. I asked if there is anyway to do that, and by the time I woke up, Adam Miller had a branch that fixed the issue. I directly used the same in my virtual environment. The package build was successful. Then, the testinfra tests failed due as it could not create the temporary directory inside of the container. I already opened an issue for the same.

Podman on Debian Buster

I use podman on all of my production servers, and also inside of the Qubes system in Fedora VMs. A few days ago I saw this post and thought of trying out the steps on my Debian Buster system.

But, it seems it requires one more installation step, so I am adding the updated installation steps for Debian Buster here.

Install all build dependencies

sudo apt -y install \
  gcc \
  make \
  cmake \
  git \
  btrfs-progs \
  golang-go \
  go-md2man \
  iptables \
  libassuan-dev \
  libc6-dev \
  libdevmapper-dev \
  libglib2.0-dev \
  libgpgme-dev \
  libgpg-error-dev \
  libostree-dev \
  libprotobuf-dev \
  libprotobuf-c-dev \
  libseccomp-dev \
  libselinux1-dev \
  libsystemd-dev \
  pkg-config \
  runc \
  uidmap \
  libapparmor-dev \
  libglib2.0-dev \
  libcap-dev \

Install latest Golang

Download and install latest golang and also make sure that you have a proper $GOPATH variable. You can read more here.

Install conmon

conmon is the OCI container runtime monitor. Install it via the following steps:

git clone https://github.com/containers/conmon
cd conmon
sudo make podman
sudo cp /usr/local/libexec/podman/conmon  /usr/local/bin/

Install CNI plugins

git clone https://github.com/containernetworking/plugins.git $GOPATH/src/github.com/containernetworking/plugins
cd $GOPATH/src/github.com/containernetworking/plugins
sudo mkdir -p /usr/libexec/cni
sudo cp bin/* /usr/libexec/cni

Setup the bridge

sudo mkdir -p /etc/cni/net.d
curl -qsSL https://raw.githubusercontent.com/containers/libpod/master/cni/87-podman-bridge.conflist | sudo tee /etc/cni/net.d/99-loopback.conf

Create the configuration files

Next, we need configuration files for the registries and also the policy file.

sudo mkdir -p /etc/containers
sudo curl https://raw.githubusercontent.com/projectatomic/registries/master/registries.fedora -o /etc/containers/registries.conf
sudo curl https://raw.githubusercontent.com/containers/skopeo/master/default-policy.json -o /etc/containers/policy.json

Installing slirp4netns

slirp4netns is used for user-mode networking for unprivileged network namespaces. At the time of the writing this blog post, the latest release is 0.4.2.

git clone https://github.com/rootless-containers/slirp4netns
cd slirp4netns
./configure --prefix=/usr
sudo make install

Installing podman

Finally we are going to install podman.

git clone https://github.com/containers/libpod/ $GOPATH/src/github.com/containers/libpod
cd $GOPATH/src/github.com/containers/libpod
sudo make install

Testing podman

Now you can test podman on your Debian system.

podman pull fedora:latest
podman run -it --rm /usr/bin/bash fedora:latest

Using podman for containers

Podman is one of the newer tool in the container world, it can help you to run OCI containers in pods. It uses Buildah to build containers, and runc or any other OCI compliant runtime. Podman is being actively developed.

I have moved the two major bots we use for dgplug summer training (named batul and tenida) under podman and they are running well for the last few days.


I am using a Fedora 28 system, installation of podman is as simple as any other standard Fedora package.

$ sudo dnf install podman

While I was trying out podman, I found it was working perfectly in my DigitalOcean instance, but, not so much on the production vm. I was not being able to attach to the stdout.

When I tried to get help in #podman IRC channel, many responded, but none of the suggestions helped. Later, I gave access to the box to Matthew Heon, one of the developer of the tool. He identified the Indian timezone (+5:30) was too large for the timestamp buffer and thus causing this trouble.

The fix was pushed fast, and a Fedora build was also pushed to the testing repo.


To learn about different available commands, visit this page.

First step was to build the container images, it was as simple as:

$ sudo podman build -t kdas/imagename .

I reused my old Dockerfiles for the same. After this, it was just simple run commands to start the containers.