Kushal Das

FOSS and life. Kushal Das talks here.

kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion

git checkout to previous branch

We regularly move between git branches while working on projects. I always used to type in the full branch name, say to go back to develop branch and then come back to the feature branch. This generally takes a lot of typing (for the branch names etc.). I found out that we can use - like in the way we use cd - to go back to the previous directory we were in.

git checkout -

Here is a small video for demonstration.

I hope this will be useful for some people.

Adding directory to path in csh on FreeBSD

While I was trying to install rust on a FreeBSD box, I figured that I will have to update the path on the system with directory path of the ~/.cargo/bin. I added the following line in the ~/.cshrc file for the same.

set path = ( $path /home/kdas/.cargo/bin)

I am yet to learn much about csh, but, I can count this as a start.

Using sops with Ansible for vars encryption

Sops is a secret management tool from Mozilla. According to the official Github page, it is defined as:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

In this blog post, I am going to show you how you can use it along with GPG to encrypt secrets in YAML files for your Ansible roles/playbooks.

Installation

Download the official binary from the release page, or you can build and install from the source code.

Basic usage (manual)

First let us create a .sops.yaml in the git repository root directory. In this example, I am saying that I can encrypt any .yml using the two given GPG fingerprints.

creation_rules:
  # If vars file matchs "*.yml", then encrypt it more permissively.
    - path_regex: \.yml$
      pgp: "A85FF376759C994A8A1168D8D8219C8C43F6C5E1,2871635BE3B4E5C04F02B848C353BFE051D06C33"

Now to encrypt a file in place, I can use the following command:

sops -e -i host_vars/stg.dgplug.org.yml

If we open the file afterward, we will see something like below.

mysql_root_password: ENC[AES256_GCM,data:732TA7ps+qE=,iv:3azuZg4tqsLfe5IHDLJDKSUHmVk2c0g1Nl+oIcKOXRw=,tag:yD8iwmxmENww+waTs5Kzxw==,type:str]
mysql_user_password: ENC[AES256_GCM,data:YXBpO54=,iv:fQYATEWw4pv4lW5Ht8xiaBCliat8xdje5qdmb0Sff4Y=,tag:cncwg2Ops35l0lWegCSEJQ==,type:str]
mysql_user: ENC[AES256_GCM,data:cq/VgDlpRBxuHKM+cw==,iv:K+v6fkCIucMrMJ7pDRxFS/aHh0OCxqUcLJhZIgCsfA0=,tag:BD7l662OVOWRaHi2Rtw37g==,type:str]
mysql_db_name: ENC[AES256_GCM,data:hCgrKmU=,iv:/jnypeWdqUbIRy75q7OIODgZnaDpR3oTa0G/L8MRiZA=,tag:0k6cGNoDajUuKpKzrwQBaw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-07-29T04:05:09Z'
    mac: ENC[AES256_GCM,data:qp9yV3qj0tYf/EaO0Q3JdlpPvm5WY4ev1zGCVNoo+Anm/esj0WHlR7M7SNg54xRTUwMhRRnirx7IsEC8EZW1lE+8DObnskemcXm93CJOBfVzQOX/RvCSR4rMp2FgBEPZADCDiba1X2K/9myU96lADME0nkdmX9YjhOLMFJ6ll4o=,iv:2WNKhl81FI/qw6mRKpK5wRYjqK16q1ASeCJYpEeuhj4=,tag:v4PlGT4ZmPUxj7aYIendVg==,type:str]
    pgp:
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA/uCql0ybaddARAADkHqV/mEgVoTxVHkRKW3mjajReKaQ5Mz/VwMal3GsjKr
            8aGnghCjtgJE01wCBjrTfNKKmlf5JjMSFufy9pG0xE+OFOXt+pnJFDSro26QnPFG
            VlvkvnjTxw4gV/mIvxUXTT6rmijvQSHMXdyEGmyHu3kNprKuuN37xZ4SYSWg7pdO
            vee4DSOaw+XfdgYUF+YSEjKMZ+qevRtzJL4hg9SZvEsHMQObueMBywAc5pyR6LvS
            ZuAS5SS+cPXnyMJLemqfU7L+XMw2NMrmNYFXOWnMMny1Hez5fcmebJp+wjDqWJTX
            j9vJvXLuNAglFvL1yz2rNJYTb0mC9chLT7YxEa9Z9JHWezUB8ZYuOC6vRf18/Hz8
            e6Gd2sncl4rleCxvKZF9qECsmFzs4p7H5M+O31jnjWdnPBD8a84Du3wdeWiI5cRF
            d7/aUXEdGJQy7OVbzGE1alDOSIyDD2S73ou2du7s/79Wb11RwQV898OyGgmjWm0Y
            7hTsBiBXnayQdjtg6qKlvoWIn79PU0YmDYLujMiXDQPJLV3ta82dcK2B1yTCLuSO
            nGfFzNOSNemmniyEuMI16SrfYDsf9l/K3gec+TRNvEdc1GqO4gFblQWptPE7KIIC
            oBVMDLUJSpOf5yF7Izedy5wBb8ZmzJAvpchvTMUls2+En4ifYh90cauXxiP6edPS
            4AHkZMebg44aCefn4zMdKdUhbOFxbuA/4I3hQWLgmuJMFlcI4Orl5tsRjXwCfQ9S
            jOdGAUJ8usV9gT4IXj73WfN1JJHj7DTgoORXFtDMs2Yy/rPPR4H9msSL4reJGZLh
            dEAA
            =LoMY
            -----END PGP MESSAGE-----
        fp: A85FF376759C994A8A1168D8D8219C8C43F6C5E1
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA0HZfc7pf7hDARAAX6xF6qP9KQJ5wLn0kv5WVf8HgOkk+2ziIuBH411hVEir
            oN4bwwnL+DEYZm2aFvZl5zQElua7nGkQK041DecPbOCCBqThv3QKVMy2jScG5tTj
            jxGgh8W/KxdwIY7teRaCRNDkT18IhtBc4SO2smJEtPeGIptvDHLjETBFbDnZeo6/
            PG4QIA1Rfvm14n1NR56oibWduwvb1wrm4tGYPDx8mVgfugxeIhoqlZ87VrivrN+2
            40S/rwWQ/aEVM1A8q19DGkXYVBcxQA1dGBrKLDPtiCq/LCSDNY4iuzMjzQuHPjgM
            bS0lWv8fvSp6iIZlB3eSRPW9Ia8tRJpEfTLS8jiHlcCZ4Vy3fW6EijBf00iSy5hP
            Y54TCERscXt1/UOW2ACYTNhMfuiO+WuG6Vjdns1NsSUVazqxmf+kBMwl06/HyUwL
            KAYTOB2hipYUm6mlpSBfDgBKjQq8dgvxlWP0Ay0of0p4ZzFv7MepYkJA+gwb0hUt
            rui9GVE/uys8W8buYqfM2ABzIG4GrH3rELh8eW8oPwlu7rgS7YGhyog2xabJyQnj
            BZ65wwu5TzMq+n5v1+878teOzqqD/F+6X5dw0jF95bDHKdA64JR/Uxlj75sp59GH
            e/+3UDm0UwSILDrYJkcVaTnrt3wPjQAw4ynKZuN+k6KvmDCkGquHNaM+2hqvWq/S
            4AHkgpZHzmU14QmfJy7RN2HPduHhSeCh4LrhMzngJuJyH72G4IPlP4WwPJUhrKMt
            UI+Azl61rIZEm6n82oWbY1gIrrIygWLg1OSD/0Ly6KbO4/W1NipWU53w4nAo7abh
            3gkA
            =YXu1
            -----END PGP MESSAGE-----
        fp: 2871635BE3B4E5C04F02B848C353BFE051D06C33
    unencrypted_suffix: _unencrypted
    version: 3.1.1


If you look closely, we can see that sops only encrypts the values, not the keys in the YAML file.

Decrypting as required on runtime

We can use a small Python script to enable runtime decryption of the file as required in Ansible. Create a vars_plugin directory in the git repository root, and then put the following code in there as sops.py.

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type


DOCUMENTATION = '''
    vars: sops
    version_added: "N/A"
    short_description: In charge of loading SOPS-encrypted vars
    description:
        - Loads SOPS-encrytped YAML vars into corresponding groups/hosts in group_vars/ and host_vars/ directories.
        - Only SOPS-encrypted vars files, with a top-level "sops" key, will be loaded.
        - Extends host/group vars logic from Ansible core.
    notes:
        - SOPS binary must be on path (missing will raise exception).
        - Only supports YAML vars files (JSON files will raise exception).
        - Only host/group vars are supported, other files will not be parsed.
    options: []
'''


import os
import subprocess
import yaml
from ansible.errors import AnsibleError, AnsibleParserError
from ansible.module_utils._text import to_bytes, to_native, to_text
from ansible.inventory.host import Host
from ansible.inventory.group import Group
from ansible.utils.vars import combine_vars


FOUND = {}


# Import host_group_vars logic for file-walking functions.
# We'll still need to copy/paste and modify the `get_vars` function
# and edit below to insert a call to sops cli.
from ansible.plugins.vars.host_group_vars import VarsModule as HostGroupVarsModule


# All SOPS-encrypted vars files will have a top-level key called "sops".
# In order to determine whether a file is SOPS-encrypted, let's inspect
# such a key if it is found, and expect the following subkeys.
SOPS_EXPECTED_SUBKEYS = [
    "lastmodified",
    "mac",
    "version",
]



class AnsibleSopsError(AnsibleError):
    pass



class VarsModule(HostGroupVarsModule):


    def get_vars(self, loader, path, entities, cache=True):
        """
        Parses the inventory file and assembles host/group vars.


        Lifted verbatim from ansible.plugins.vars.host_group_vars, with a single
        in-line edit to support calling out to the SOPS CLI for decryption.
        Only SOPS-encrypted files will be handled.
        """


        if not isinstance(entities, list):
            entities = [entities]


        super(VarsModule, self).get_vars(loader, path, entities)


        data = {}
        for entity in entities:
            if isinstance(entity, Host):
                subdir = 'host_vars'
            elif isinstance(entity, Group):
                subdir = 'group_vars'
            else:
                raise AnsibleParserError("Supplied entity must be Host or Group, got %s instead" % (type(entity)))


            # avoid 'chroot' type inventory hostnames /path/to/chroot
            if not entity.name.startswith(os.path.sep):
                try:
                    found_files = []
                    # load vars
                    b_opath = os.path.realpath(to_bytes(os.path.join(self._basedir, subdir)))
                    opath = to_text(b_opath)
                    key = '%s.%s' % (entity.name, opath)
                    if cache and key in FOUND:
                        found_files = FOUND[key]
                    else:
                        # no need to do much if path does not exist for basedir
                        if os.path.exists(b_opath):
                            if os.path.isdir(b_opath):
                                self._display.debug("\tprocessing dir %s" % opath)
                                found_files = loader.find_vars_files(opath, entity.name)
                                FOUND[key] = found_files
                            else:
                                self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))


                    for found in found_files:
                        # BEGIN SOPS-specific logic
                        if self._is_encrypted_sops_file(found):
                            new_data = self._decrypt_sops_file(found)


                            if new_data:  # ignore empty files
                                data = combine_vars(data, new_data)
                        # END SOPS-specific logic


                except Exception as e:
                    raise AnsibleParserError(to_native(e))
        return data


    def _is_encrypted_sops_file(self, path):
        """
        Check whether given filename is likely a SOPS-encrypted vars file.
        Determined by presence of top-level 'sops' key in vars file.


        Assumes file is YAML. Does not support JSON files.
        """
        is_sops_file_result = False
        with open(path, 'r') as f:
            y = yaml.safe_load(f)
            if type(y) == dict:
                # All SOPS-encrypted vars files will have top-level "sops" key.
                if 'sops' in y.keys() and type(y['sops'] == dict):
                    if all(k in y['sops'].keys() for k in SOPS_EXPECTED_SUBKEYS):
                        is_sops_file_result = True
            return is_sops_file_result


    def _decrypt_sops_file(self, path):
        """
        Shells out to `sops` binary and reads decrypted vars from stdout.
        Passes back dict to vars loader.


        Assumes that a file is a valid SOPS-encrypted file. Use function
        `is_encrypted_sops_file` to check.


        Assumes file is YAML. Does not support JSON files.
        """
        cmd = ["sops", "--input-type", "yaml", "--decrypt", path]
        real_yaml = None
        try:
            decrypted_yaml = subprocess.check_output(cmd)
        except OSError:
            msg = "Failed to call SOPS to decrypt file at {}".format(path)
            msg += ", ensure sops is installed in PATH."
            raise AnsibleSopsError(msg)
        except subprocess.CalledProcessError:
            msg = "Failed to decrypt SOPS file at {}".format(path)
            raise AnsibleSopsError(msg)
        try:
            real_yaml = yaml.safe_load(decrypted_yaml)
        except yaml.parser.ParserError:
            msg = "Failed to parse YAML from decrypted SOPS file at {},".format(path)
            msg += " confirm file is YAML format."
            raise AnsibleSopsError(msg)
        return real_yaml


From the next you will try to use any of the encrypted vars files in an Ansible run, it will ask for the GPG passphrase to decrypt the file as required.

Thank you Conor Schaefer for the original version of the Python script.

Setting up authorized v3 Onion services

Just like v2 Onion services, we can also set up client authorization for Onion services v3. In simple terms, when you have a client authorization setup on an Onion service, only the Tor clients with the private token can access the service. Using this, you can run services (without opening up any port in your system) and only selected people can access that service, that is also being inside of totally encrypted Tor network. Last month, I did a workshop in Rootconf about the same topic, but, I demoed v2 Onion services. In this blog post, I am going to show you how you can do the same with the latest v3 services.

Setting up the Onion service

We assume that we are already running nginx or apache on port 80 of the server. Add the following two lines at the end of the /etc/tor/torrc file of your server.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 127.0.0.1:80

Then, restart the tor service.

systemctl restart tor

The above command will create the onion service at /var/lib/tor/hidden_service/ directory, and we can see the address from the hostname file.

cat /var/lib/tor/hidden_service/hostname 
cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion

It should also create a authorized_clients directory at the service directory.

Next, we will create keys of type x25519, and you can either use any of the following options to create the keys.

I used the Rust implementation, and I got the secret and the public key.

secret: "TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q"
public: "RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ"

Now, we will use the public key to create a clientname.auth file in /var/lib/tor/hidden_service/authorized_clients/ directory, I chose the name kushal.auth.

descriptor:x25519:RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ > /var/lib/tor/hidden_service/authorized_clients/kushal.auth

If you look closely, the file format is like below:

descriptor:x25519:public_key

Now, restart the tor service once again in the server.

systemctl restart tor

Setting up client authorization

The first step is to close down my Tor Browser as I will be manually editing the torrc file of the same. Then, I added the following line to the same file tor-browser_en-US/Browser/TorBrowser/Data/Tor/torrc.

ClientOnionAuthDir TorBrowser/Data/Tor/onion_auth

Next, we will create the directory.

mkdir tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth
chmod 0700 tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth

Then, add the following in kushal.auth_private file inside of the onion_auth directory.

cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid:descriptor:x25519:TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q

The format of the file:

onion_address_56_chars:descriptor:x25519:private_key

Now, start the Tor Browser, and you should be able to visit the authorized Onion service at cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion.

Use case for students

If you want to demo your web project to a selected group of people, but, don't want to spend money to get a web server or VPS, Onion services is a great way to showcase your work to the world. With the authenticated services, you can choose whom all can view the site/service you are running.

Using signify tool for sign and verification

We generally use GNUPG for sign and verify files on our systems. There are other tools available to do so; some tools are particularly written only for this purpose. signify is one such tool from the OpenBSD land.

How to install signify?

pkg install signify

I used the above command to install the tool on my FreeBSD system, and you can install it in your Debian system too, the tool is called signify-openbsd as Debian already has another tool with the same name. signify is yet to be packaged for Fedora, if you are Fedora packager, you may want to package this one for all of us.

Creating a public/private key pair

signify -G -s atest.sec -p atest.pub -c "Test key for blog post"

The command will also ask for a password for the secret key. -c allows us to add a comment in our key files. The following is the content of the public keyfile.

untrusted comment: Test key for blog post public key 
RWRjWJ28QRKKQCXxYPqwbnOqgsLYQSwvqfa2WDpp0dRDQX2Ht6Xl4Vz4

As it is very small in size, you can even create a QR code for the same.

Signing a file

In our demo directory, we have a hello.txt file, and we can use the newly generated key to create a signature.

signify -S -s atest.sec -m hello.txt

This will create a hello.txt.sig file as the signature.

Verifying the signature

$ signify -V -p atest.pub -m hello.txt
Signature Verified

This assumes the signature file in the same directory. You can find the OpenBSD signature files under /usr/local/etc/signify (or in /etc/signify/ if you are on Debian).

To know more about the tool, read this paper.