Kushal Das

FOSS and life. Kushal Das talks here.


Congratulations Anwesha

The year 2022 gave me one of the happiest moments in my life, and I also felt proud as Anwesha joined the Ansible community team as a software engineer in Red Hat. Proud because she became the best example of someone to whom I taught things about computers (she has multiple mentors/friends who helped her during this journey). Though sometimes that created trouble at home, the output is super lovely. From a Masters in Law to a software engineer in Red Hat is a good story.

Also, Red Hat has a special place in our home. I left Red Hat more than 5 years ago, but still, you will notice how Red Hat (the friends there and the culture) changed my life and later the life of our family. Oh, the other special thing is that Anwesha will have her own Red Fedora now :)

Default values, documentation and Ansible

While testing my qubes_ansible project on the upcoming Qubes OS 4.1 project, I noticed something really strange. But, before getting into that, this Ansible module and the connection plugin are for Qubes OS only, and based on the excellent Python modules provided by the Qubes team.

The error goes like this during the fact gathering steps (reformatted for the blog):

fatal: [debian-10]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory.In some cases, you may have
    been able to authenticate and did not have permissions on the target
    directory. Consider changing the remote tmp path in ansible.cfg to a path
    rooted in \"/tmp\", for more error information use -vvv. Failed command
    was: ( umask 77 && mkdir -p \"` echo ~The *user* is the default user in
    Qubes./.ansible/tmp `\"&& mkdir \"` echo ~The *user* is the default user in
    Qubes./.ansible/tmp/ansible-tmp-1630548982.9355698-7707-90110802425258 `\"
    && echo ansible-tmp-1630548982.9355698-7707-90110802425258=\"` echo ~The
    *user* is the default user in
    Qubes./.ansible/tmp/ansible-tmp-1630548982.9355698-7707-90110802425258 `\"
    ), exited with result 1, stderr output: mkdir: cannot create directory
    ‘~The ~The *user* account as default in Qubes OS. ~The ~The *user* account
    as default in Qubes OS. account as default in Qubes OS. ~The ~The *user*
    account as default in Qubes OS. ~The ~The *user* account as default in
    Qubes OS. account as default in Qubes OS. account as default in Qubes OS.
    is the default user in Qubes.’: File name too long\n",
    "unreachable": true

Most important part is the default user's home directory part, echo ~The user is the default user in Qubes./.ansible/tmp. For a moment I totally freaked out, as this looks like documentation. After reading the code more, I can see it is coming from the DOCUMENTATION variable in my plugin. After playing around a bit more and trying out different values I can see that the default value mentioned in the documentation is becoming the default value in the Python code.

After searching more I can see that the Ansible developers want the documentation string to be the gold standard and the code is parsing it find the default values. In my mind this is more confusing. I would expect the default value to be declared inside of the code.

Parsing the DOCUMENTATION and then finding the default values there in a Python code still does not fit in my brain. Fixed the issue for now, let me see what other surprises are waiting in the future.

Exciting few weeks in the SecureDrop land

Eric Trump tweet

Last week there was an interesting tweet from Eric Trump, son of US President Donald Trump. Where he points out how Mr. David Fahrenthold, a journalist from Washington Post did some old school journalism and made sure that every Trump organization employee knows about how to securely leak information or talk to a journalist via SecureDrop.

I want to say thank you to him for this excellent advertisement for our work. There were many people over Twitter, cheering him for this tweet.

julian and matt's tweet Parker's tweet Harlo's tweet

If you don’t know what SecureDrop is, it is an open-source whistleblower submission system that media organizations and NGOs can install to securely accept documents from anonymous sources. It was originally created by the late Aaron Swartz and is now managed by Freedom of the Press Foundation. It is mostly written in Python and uses a lot of Ansible. Jennifer Helsby, the lead developer of SecureDrop and I took part in this week’s Python podcast along with our host Tobias. You can listen to learn about many upcoming features and plans.

If you are interested to contribute to the SecureDrop project, come over to our gitter channel and say hello.


Last month, during Defcon 27, there was a panel about DEF CON to help hackers anonymously submit bugs to the government, interestingly the major suggestion in that panel is to use SecureDrop (hosted by Defcon) so that the researchers can safely submit vulnerabilities to the US government. Watch the full panel discussion to learn more in details.

Using sops with Ansible for vars encryption

Sops is a secret management tool from Mozilla. According to the official Github page, it is defined as:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

In this blog post, I am going to show you how you can use it along with GPG to encrypt secrets in YAML files for your Ansible roles/playbooks.


Download the official binary from the release page, or you can build and install from the source code.

Basic usage (manual)

First let us create a .sops.yaml in the git repository root directory. In this example, I am saying that I can encrypt any .yml using the two given GPG fingerprints.

  # If vars file matchs "*.yml", then encrypt it more permissively.
    - path_regex: \.yml$
      pgp: "A85FF376759C994A8A1168D8D8219C8C43F6C5E1,2871635BE3B4E5C04F02B848C353BFE051D06C33"

Now to encrypt a file in place, I can use the following command:

sops -e -i host_vars/stg.dgplug.org.yml

If we open the file afterward, we will see something like below.

mysql_root_password: ENC[AES256_GCM,data:732TA7ps+qE=,iv:3azuZg4tqsLfe5IHDLJDKSUHmVk2c0g1Nl+oIcKOXRw=,tag:yD8iwmxmENww+waTs5Kzxw==,type:str]
mysql_user_password: ENC[AES256_GCM,data:YXBpO54=,iv:fQYATEWw4pv4lW5Ht8xiaBCliat8xdje5qdmb0Sff4Y=,tag:cncwg2Ops35l0lWegCSEJQ==,type:str]
mysql_user: ENC[AES256_GCM,data:cq/VgDlpRBxuHKM+cw==,iv:K+v6fkCIucMrMJ7pDRxFS/aHh0OCxqUcLJhZIgCsfA0=,tag:BD7l662OVOWRaHi2Rtw37g==,type:str]
mysql_db_name: ENC[AES256_GCM,data:hCgrKmU=,iv:/jnypeWdqUbIRy75q7OIODgZnaDpR3oTa0G/L8MRiZA=,tag:0k6cGNoDajUuKpKzrwQBaw==,type:str]
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-07-29T04:05:09Z'
    mac: ENC[AES256_GCM,data:qp9yV3qj0tYf/EaO0Q3JdlpPvm5WY4ev1zGCVNoo+Anm/esj0WHlR7M7SNg54xRTUwMhRRnirx7IsEC8EZW1lE+8DObnskemcXm93CJOBfVzQOX/RvCSR4rMp2FgBEPZADCDiba1X2K/9myU96lADME0nkdmX9YjhOLMFJ6ll4o=,iv:2WNKhl81FI/qw6mRKpK5wRYjqK16q1ASeCJYpEeuhj4=,tag:v4PlGT4ZmPUxj7aYIendVg==,type:str]
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----

            -----END PGP MESSAGE-----
        fp: A85FF376759C994A8A1168D8D8219C8C43F6C5E1
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----

            -----END PGP MESSAGE-----
        fp: 2871635BE3B4E5C04F02B848C353BFE051D06C33
    unencrypted_suffix: _unencrypted
    version: 3.1.1

If you look closely, we can see that sops only encrypts the values, not the keys in the YAML file.

Decrypting as required on runtime

We can use a small Python script to enable runtime decryption of the file as required in Ansible. Create a vars_plugin directory in the git repository root, and then put the following code in there as sops.py.

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type

    vars: sops
    version_added: "N/A"
    short_description: In charge of loading SOPS-encrypted vars
        - Loads SOPS-encrytped YAML vars into corresponding groups/hosts in group_vars/ and host_vars/ directories.
        - Only SOPS-encrypted vars files, with a top-level "sops" key, will be loaded.
        - Extends host/group vars logic from Ansible core.
        - SOPS binary must be on path (missing will raise exception).
        - Only supports YAML vars files (JSON files will raise exception).
        - Only host/group vars are supported, other files will not be parsed.
    options: []

import os
import subprocess
import yaml
from ansible.errors import AnsibleError, AnsibleParserError
from ansible.module_utils._text import to_bytes, to_native, to_text
from ansible.inventory.host import Host
from ansible.inventory.group import Group
from ansible.utils.vars import combine_vars

FOUND = {}

# Import host_group_vars logic for file-walking functions.
# We'll still need to copy/paste and modify the `get_vars` function
# and edit below to insert a call to sops cli.
from ansible.plugins.vars.host_group_vars import VarsModule as HostGroupVarsModule

# All SOPS-encrypted vars files will have a top-level key called "sops".
# In order to determine whether a file is SOPS-encrypted, let's inspect
# such a key if it is found, and expect the following subkeys.

class AnsibleSopsError(AnsibleError):

class VarsModule(HostGroupVarsModule):

    def get_vars(self, loader, path, entities, cache=True):
        Parses the inventory file and assembles host/group vars.

        Lifted verbatim from ansible.plugins.vars.host_group_vars, with a single
        in-line edit to support calling out to the SOPS CLI for decryption.
        Only SOPS-encrypted files will be handled.

        if not isinstance(entities, list):
            entities = [entities]

        super(VarsModule, self).get_vars(loader, path, entities)

        data = {}
        for entity in entities:
            if isinstance(entity, Host):
                subdir = 'host_vars'
            elif isinstance(entity, Group):
                subdir = 'group_vars'
                raise AnsibleParserError("Supplied entity must be Host or Group, got %s instead" % (type(entity)))

            # avoid 'chroot' type inventory hostnames /path/to/chroot
            if not entity.name.startswith(os.path.sep):
                    found_files = []
                    # load vars
                    b_opath = os.path.realpath(to_bytes(os.path.join(self._basedir, subdir)))
                    opath = to_text(b_opath)
                    key = '%s.%s' % (entity.name, opath)
                    if cache and key in FOUND:
                        found_files = FOUND[key]
                        # no need to do much if path does not exist for basedir
                        if os.path.exists(b_opath):
                            if os.path.isdir(b_opath):
                                self._display.debug("\tprocessing dir %s" % opath)
                                found_files = loader.find_vars_files(opath, entity.name)
                                FOUND[key] = found_files
                                self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))

                    for found in found_files:
                        # BEGIN SOPS-specific logic
                        if self._is_encrypted_sops_file(found):
                            new_data = self._decrypt_sops_file(found)

                            if new_data:  # ignore empty files
                                data = combine_vars(data, new_data)
                        # END SOPS-specific logic

                except Exception as e:
                    raise AnsibleParserError(to_native(e))
        return data

    def _is_encrypted_sops_file(self, path):
        Check whether given filename is likely a SOPS-encrypted vars file.
        Determined by presence of top-level 'sops' key in vars file.

        Assumes file is YAML. Does not support JSON files.
        is_sops_file_result = False
        with open(path, 'r') as f:
            y = yaml.safe_load(f)
            if type(y) == dict:
                # All SOPS-encrypted vars files will have top-level "sops" key.
                if 'sops' in y.keys() and type(y['sops'] == dict):
                    if all(k in y['sops'].keys() for k in SOPS_EXPECTED_SUBKEYS):
                        is_sops_file_result = True
            return is_sops_file_result

    def _decrypt_sops_file(self, path):
        Shells out to `sops` binary and reads decrypted vars from stdout.
        Passes back dict to vars loader.

        Assumes that a file is a valid SOPS-encrypted file. Use function
        `is_encrypted_sops_file` to check.

        Assumes file is YAML. Does not support JSON files.
        cmd = ["sops", "--input-type", "yaml", "--decrypt", path]
        real_yaml = None
            decrypted_yaml = subprocess.check_output(cmd)
        except OSError:
            msg = "Failed to call SOPS to decrypt file at {}".format(path)
            msg += ", ensure sops is installed in PATH."
            raise AnsibleSopsError(msg)
        except subprocess.CalledProcessError:
            msg = "Failed to decrypt SOPS file at {}".format(path)
            raise AnsibleSopsError(msg)
            real_yaml = yaml.safe_load(decrypted_yaml)
        except yaml.parser.ParserError:
            msg = "Failed to parse YAML from decrypted SOPS file at {},".format(path)
            msg += " confirm file is YAML format."
            raise AnsibleSopsError(msg)
        return real_yaml

From the next you will try to use any of the encrypted vars files in an Ansible run, it will ask for the GPG passphrase to decrypt the file as required.

Thank you Conor Schaefer for the original version of the Python script.

Fedora 29 on Qubes OS

I spent most of my life using Fedora as my primary operating system on my desktop/laptops. I use CentOS on my servers, sometimes even Fedora, and a few special cases, I use *BSD systems.

But, for the last one year I am running Qubes OS as my primary operating system on my laptop. That enables me to still keep using Fedora in the AppVMs as I want, and I can also have different work VMs in Debian/Ubuntu or even Windows as required. Moving to a newer version of Fedora is just about installing the new template and rebooting any AppVM with the newest template.

Fedora 29 will release on 30th October, and Qubes team already built a template for the same and pushed to the testing repository. You can install it by the following command.

$ sudo qubes-dom0-update qubes-template-fedora-29 --enablerepo=qubes-templates-itl-testing

After this, I just installed all the required packages and setup the template as I want using my Qubes Ansible project. It took only a few minutes to move all of my development related VMs into Fedora 29 and this still keeps the option open to go back to Fedora 28 the moment I want. This is one of the beauty of Qubes OS and of course there are the regular security aspects too.

If you are a software developer using Linux, and also care about security practices, give Qubes OS a try. It has also a very active and helpful user community. I am sure it will not disappoint you.

Using Ansible to maintain your Qubes system

From the time I have started using Qubes OS, How to create and setup new AppVMs in an efficient way? remained an open question for me. I was mostly using the command line tool to create any new AppVMs and then manually setting all the properties after creation. I also did the package installations and other setup inside of the VMs manually.

If you never heard of Qubes before, you should check it out. Qubes takes a different approach to security, security by compartmentalization, different applications are separated by Qubes (VMs) . The base is running Fedora and then all other VMs are on top of Xen. It also provides a very tight integration of the tools to give a pleasant experience.

When I asked about how people maintain different VMs or templateVMs (from which the normal VMs spawn off), the answer was mostly bash scripts. The tools provided by the Qubes team are friendly to scripting. Though the official way to managing VMs is done by Salt project.

As we (at Freedom of the Press Founation) are working towards a Qubes based desktop client for SecureDrop, we also started using Salt to maintain the states of the VMs. I personally found Salt to be very confusing and a bit difficult to learn.

From the mailing list I also found out about https://github.com/Rudd-O/ansible-qubes, but, as I started reading the README, I figured that Salt is being used here too in the background. That made me rethink about the Ansible as a choice to maintain my Qubes.

Last weekend I pinged Trishna for some pointers on writing new plugins for Ansible, and later at night I also talked with Toshio about the Ansible plugins + modules.

Introducing Qubes Ansible

The result of those chats is Qubes Ansible. It has a qubesos module and a qubes connection plugin for Ansible.

I already have a PR opened to add the connection plugin into Ansible.

The actual module will still require a lot of work to become feature complete with the existing command line tools and also with the Salt. This project is under active development.

Good thing is that I am getting feedback+patches from the #qubes IRC channel (on Freenode). From the Qubes development team, marmarek provided some real valuable input to make the plugin easier to use.

Example playbook

- hosts: localhost
  connection: local

    - name: Make sure the development VM is present
        guest: development2
        state: present
          memory: 1200
          maxmem: 1400
          netvm: 'sys-firewall'
          template: 'debian-9'
          label: "blue"

    - name: Run the VM
        guest: development2
        state: running

You can use the above playbook to create a development2 AppVM with the exact properties you want. The examples page has all the available options documented.

If you are using Qubes, please give it a try, and tell us how can we improve your experience of maintaining the system with Ansible. You can provide feedback in a Github issue or talk directly in the #qubes IRC channel.

Testing containers using Kubernetes on Tunir version 0.15

Today I have released Tunir 0.l5. This release got a major rewrite of the code, and has many new features. One of them is setting up multiple VM(s) from Tunir itself. We now also have the ability to use Ansible (using 2.x) from within Tunir. Using these we are going to deploy Kubernetes on Fedora 23 Atomic images, and then we will deploy an example atomicapp which follows Nulecule specification.

I am running this on Fedora 23 system, you can grab the latest Tunir from koji. You will also need the Ansible 2.x from the testing repository.

Getting Kubernetes contrib repo

First we will get the latest Kubernetes contrib repo.

$ git clone https://github.com/kubernetes/contrib.git

Inside we will make changes to a group_vars file at contrib/ansible/group_vars/all.yml

diff --git a/ansible/group_vars/all.yml b/ansible/group_vars/all.yml
index 276ded1..ead74fd 100644
--- a/ansible/group_vars/all.yml
+++ b/ansible/group_vars/all.yml
@@ -14,7 +14,7 @@ cluster_name: cluster.local
# Account name of remote user. Ansible will use this user account to ssh into
# the managed machines. The user must be able to use sudo without asking
# for password unless ansible_sudo_pass is set
-#ansible_ssh_user: root
+ansible_ssh_user: fedora

# password for the ansible_ssh_user. If this is unset you will need to set up
# ssh keys so a password is not needed.

Setting up the Tunir job configuration

The new multivm setup requires a jobname.cfg file as the configuration. In this case I have already downloaded a Fedora Atomic cloud .qcow2 file under /mnt. I am going to use that.

cpu = 1
ram = 1024
ansible_dir = /home/user/contrib/ansible

user = fedora
image = /mnt/Fedora-Cloud-Atomic-23-20160308.x86_64.qcow2
hostname = kube-master.example.com

user = fedora
image = /mnt/Fedora-Cloud-Atomic-23-20160308.x86_64.qcow2
hostname = kube-node-01.example.com

user = fedora
image = /mnt/Fedora-Cloud-Atomic-23-20160308.x86_64.qcow2
hostname = kube-node-02.example.com

The above configuration file is mostly self explanatory. All VM(s) will have 1 virtual CPU, and 1024 MB of RAM. I also put up the directory which contains the ansible source. Next we have 3 VM definitions. I also have hostnames setup for each, this are the same hostnames which are mentioned in the inventory file. The inventory file should exist on the same directory with name inventory. If you do not want to mention such big names, you can simply use vm1, vm2, vm3 in the inventory file.

Now if we remember, we need a jobname.txt file containing the actual commands for testing. The following is from our file.

PLAYBOOK cluster.yml
vm1 sudo atomic run projectatomic/guestbookgo-atomicapp

In the first line we are mentioning to run the cluster playbook. In the second line we are putting in the actual atomic command to deploy guestbook app on our newly setup Kubernetes. You can understand that we mention which VM to execute as the first part of the line. If no vm is marked, Tunir assumes that the command has to run on vm1.

Now if we just execute Tunir, you will be able to see Kubernetes being setup, and then the guestbook app being deployed. You can add few more commands in the above mentioned file to see how many pods running, or even about the details of the pods.

$ sudo tunir --multi jobname

Debug mode

For the multivm setup, Tunir now has a debug mode which can be turned on by passing --debug in the command line. This will not destroy the VM(s) at the end of the test. It will also create a destroy.sh script for you, which you can run to destroy the VM(s), and remove all temporary directories. The path of the file will be given at the end of the Tunir run.

DEBUG MODE ON. Destroy from /tmp/tmp8KtIPO/destroy.sh

Please try these new features, and comment for any improvements you want.