Kushal Das

FOSS and life. Kushal Das talks here.

kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion

Using sops with Ansible for vars encryption

Sops is a secret management tool from Mozilla. According to the official Github page, it is defined as:

sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.

In this blog post, I am going to show you how you can use it along with GPG to encrypt secrets in YAML files for your Ansible roles/playbooks.

Installation

Download the official binary from the release page, or you can build and install from the source code.

Basic usage (manual)

First let us create a .sops.yaml in the git repository root directory. In this example, I am saying that I can encrypt any .yml using the two given GPG fingerprints.

creation_rules:
  # If vars file matchs "*.yml", then encrypt it more permissively.
    - path_regex: \.yml$
      pgp: "A85FF376759C994A8A1168D8D8219C8C43F6C5E1,2871635BE3B4E5C04F02B848C353BFE051D06C33"

Now to encrypt a file in place, I can use the following command:

sops -e -i host_vars/stg.dgplug.org.yml

If we open the file afterward, we will see something like below.

mysql_root_password: ENC[AES256_GCM,data:732TA7ps+qE=,iv:3azuZg4tqsLfe5IHDLJDKSUHmVk2c0g1Nl+oIcKOXRw=,tag:yD8iwmxmENww+waTs5Kzxw==,type:str]
mysql_user_password: ENC[AES256_GCM,data:YXBpO54=,iv:fQYATEWw4pv4lW5Ht8xiaBCliat8xdje5qdmb0Sff4Y=,tag:cncwg2Ops35l0lWegCSEJQ==,type:str]
mysql_user: ENC[AES256_GCM,data:cq/VgDlpRBxuHKM+cw==,iv:K+v6fkCIucMrMJ7pDRxFS/aHh0OCxqUcLJhZIgCsfA0=,tag:BD7l662OVOWRaHi2Rtw37g==,type:str]
mysql_db_name: ENC[AES256_GCM,data:hCgrKmU=,iv:/jnypeWdqUbIRy75q7OIODgZnaDpR3oTa0G/L8MRiZA=,tag:0k6cGNoDajUuKpKzrwQBaw==,type:str]
sops:
    kms: []
    gcp_kms: []
    azure_kv: []
    lastmodified: '2019-07-29T04:05:09Z'
    mac: ENC[AES256_GCM,data:qp9yV3qj0tYf/EaO0Q3JdlpPvm5WY4ev1zGCVNoo+Anm/esj0WHlR7M7SNg54xRTUwMhRRnirx7IsEC8EZW1lE+8DObnskemcXm93CJOBfVzQOX/RvCSR4rMp2FgBEPZADCDiba1X2K/9myU96lADME0nkdmX9YjhOLMFJ6ll4o=,iv:2WNKhl81FI/qw6mRKpK5wRYjqK16q1ASeCJYpEeuhj4=,tag:v4PlGT4ZmPUxj7aYIendVg==,type:str]
    pgp:
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA/uCql0ybaddARAADkHqV/mEgVoTxVHkRKW3mjajReKaQ5Mz/VwMal3GsjKr
            8aGnghCjtgJE01wCBjrTfNKKmlf5JjMSFufy9pG0xE+OFOXt+pnJFDSro26QnPFG
            VlvkvnjTxw4gV/mIvxUXTT6rmijvQSHMXdyEGmyHu3kNprKuuN37xZ4SYSWg7pdO
            vee4DSOaw+XfdgYUF+YSEjKMZ+qevRtzJL4hg9SZvEsHMQObueMBywAc5pyR6LvS
            ZuAS5SS+cPXnyMJLemqfU7L+XMw2NMrmNYFXOWnMMny1Hez5fcmebJp+wjDqWJTX
            j9vJvXLuNAglFvL1yz2rNJYTb0mC9chLT7YxEa9Z9JHWezUB8ZYuOC6vRf18/Hz8
            e6Gd2sncl4rleCxvKZF9qECsmFzs4p7H5M+O31jnjWdnPBD8a84Du3wdeWiI5cRF
            d7/aUXEdGJQy7OVbzGE1alDOSIyDD2S73ou2du7s/79Wb11RwQV898OyGgmjWm0Y
            7hTsBiBXnayQdjtg6qKlvoWIn79PU0YmDYLujMiXDQPJLV3ta82dcK2B1yTCLuSO
            nGfFzNOSNemmniyEuMI16SrfYDsf9l/K3gec+TRNvEdc1GqO4gFblQWptPE7KIIC
            oBVMDLUJSpOf5yF7Izedy5wBb8ZmzJAvpchvTMUls2+En4ifYh90cauXxiP6edPS
            4AHkZMebg44aCefn4zMdKdUhbOFxbuA/4I3hQWLgmuJMFlcI4Orl5tsRjXwCfQ9S
            jOdGAUJ8usV9gT4IXj73WfN1JJHj7DTgoORXFtDMs2Yy/rPPR4H9msSL4reJGZLh
            dEAA
            =LoMY
            -----END PGP MESSAGE-----
        fp: A85FF376759C994A8A1168D8D8219C8C43F6C5E1
    -   created_at: '2019-07-29T04:05:06Z'
        enc: |-
            -----BEGIN PGP MESSAGE-----


            wcFMA0HZfc7pf7hDARAAX6xF6qP9KQJ5wLn0kv5WVf8HgOkk+2ziIuBH411hVEir
            oN4bwwnL+DEYZm2aFvZl5zQElua7nGkQK041DecPbOCCBqThv3QKVMy2jScG5tTj
            jxGgh8W/KxdwIY7teRaCRNDkT18IhtBc4SO2smJEtPeGIptvDHLjETBFbDnZeo6/
            PG4QIA1Rfvm14n1NR56oibWduwvb1wrm4tGYPDx8mVgfugxeIhoqlZ87VrivrN+2
            40S/rwWQ/aEVM1A8q19DGkXYVBcxQA1dGBrKLDPtiCq/LCSDNY4iuzMjzQuHPjgM
            bS0lWv8fvSp6iIZlB3eSRPW9Ia8tRJpEfTLS8jiHlcCZ4Vy3fW6EijBf00iSy5hP
            Y54TCERscXt1/UOW2ACYTNhMfuiO+WuG6Vjdns1NsSUVazqxmf+kBMwl06/HyUwL
            KAYTOB2hipYUm6mlpSBfDgBKjQq8dgvxlWP0Ay0of0p4ZzFv7MepYkJA+gwb0hUt
            rui9GVE/uys8W8buYqfM2ABzIG4GrH3rELh8eW8oPwlu7rgS7YGhyog2xabJyQnj
            BZ65wwu5TzMq+n5v1+878teOzqqD/F+6X5dw0jF95bDHKdA64JR/Uxlj75sp59GH
            e/+3UDm0UwSILDrYJkcVaTnrt3wPjQAw4ynKZuN+k6KvmDCkGquHNaM+2hqvWq/S
            4AHkgpZHzmU14QmfJy7RN2HPduHhSeCh4LrhMzngJuJyH72G4IPlP4WwPJUhrKMt
            UI+Azl61rIZEm6n82oWbY1gIrrIygWLg1OSD/0Ly6KbO4/W1NipWU53w4nAo7abh
            3gkA
            =YXu1
            -----END PGP MESSAGE-----
        fp: 2871635BE3B4E5C04F02B848C353BFE051D06C33
    unencrypted_suffix: _unencrypted
    version: 3.1.1


If you look closely, we can see that sops only encrypts the values, not the keys in the YAML file.

Decrypting as required on runtime

We can use a small Python script to enable runtime decryption of the file as required in Ansible. Create a vars_plugin directory in the git repository root, and then put the following code in there as sops.py.

from __future__ import (absolute_import, division, print_function)
__metaclass__ = type


DOCUMENTATION = '''
    vars: sops
    version_added: "N/A"
    short_description: In charge of loading SOPS-encrypted vars
    description:
        - Loads SOPS-encrytped YAML vars into corresponding groups/hosts in group_vars/ and host_vars/ directories.
        - Only SOPS-encrypted vars files, with a top-level "sops" key, will be loaded.
        - Extends host/group vars logic from Ansible core.
    notes:
        - SOPS binary must be on path (missing will raise exception).
        - Only supports YAML vars files (JSON files will raise exception).
        - Only host/group vars are supported, other files will not be parsed.
    options: []
'''


import os
import subprocess
import yaml
from ansible.errors import AnsibleError, AnsibleParserError
from ansible.module_utils._text import to_bytes, to_native, to_text
from ansible.inventory.host import Host
from ansible.inventory.group import Group
from ansible.utils.vars import combine_vars


FOUND = {}


# Import host_group_vars logic for file-walking functions.
# We'll still need to copy/paste and modify the `get_vars` function
# and edit below to insert a call to sops cli.
from ansible.plugins.vars.host_group_vars import VarsModule as HostGroupVarsModule


# All SOPS-encrypted vars files will have a top-level key called "sops".
# In order to determine whether a file is SOPS-encrypted, let's inspect
# such a key if it is found, and expect the following subkeys.
SOPS_EXPECTED_SUBKEYS = [
    "lastmodified",
    "mac",
    "version",
]



class AnsibleSopsError(AnsibleError):
    pass



class VarsModule(HostGroupVarsModule):


    def get_vars(self, loader, path, entities, cache=True):
        """
        Parses the inventory file and assembles host/group vars.


        Lifted verbatim from ansible.plugins.vars.host_group_vars, with a single
        in-line edit to support calling out to the SOPS CLI for decryption.
        Only SOPS-encrypted files will be handled.
        """


        if not isinstance(entities, list):
            entities = [entities]


        super(VarsModule, self).get_vars(loader, path, entities)


        data = {}
        for entity in entities:
            if isinstance(entity, Host):
                subdir = 'host_vars'
            elif isinstance(entity, Group):
                subdir = 'group_vars'
            else:
                raise AnsibleParserError("Supplied entity must be Host or Group, got %s instead" % (type(entity)))


            # avoid 'chroot' type inventory hostnames /path/to/chroot
            if not entity.name.startswith(os.path.sep):
                try:
                    found_files = []
                    # load vars
                    b_opath = os.path.realpath(to_bytes(os.path.join(self._basedir, subdir)))
                    opath = to_text(b_opath)
                    key = '%s.%s' % (entity.name, opath)
                    if cache and key in FOUND:
                        found_files = FOUND[key]
                    else:
                        # no need to do much if path does not exist for basedir
                        if os.path.exists(b_opath):
                            if os.path.isdir(b_opath):
                                self._display.debug("\tprocessing dir %s" % opath)
                                found_files = loader.find_vars_files(opath, entity.name)
                                FOUND[key] = found_files
                            else:
                                self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))


                    for found in found_files:
                        # BEGIN SOPS-specific logic
                        if self._is_encrypted_sops_file(found):
                            new_data = self._decrypt_sops_file(found)


                            if new_data:  # ignore empty files
                                data = combine_vars(data, new_data)
                        # END SOPS-specific logic


                except Exception as e:
                    raise AnsibleParserError(to_native(e))
        return data


    def _is_encrypted_sops_file(self, path):
        """
        Check whether given filename is likely a SOPS-encrypted vars file.
        Determined by presence of top-level 'sops' key in vars file.


        Assumes file is YAML. Does not support JSON files.
        """
        is_sops_file_result = False
        with open(path, 'r') as f:
            y = yaml.safe_load(f)
            if type(y) == dict:
                # All SOPS-encrypted vars files will have top-level "sops" key.
                if 'sops' in y.keys() and type(y['sops'] == dict):
                    if all(k in y['sops'].keys() for k in SOPS_EXPECTED_SUBKEYS):
                        is_sops_file_result = True
            return is_sops_file_result


    def _decrypt_sops_file(self, path):
        """
        Shells out to `sops` binary and reads decrypted vars from stdout.
        Passes back dict to vars loader.


        Assumes that a file is a valid SOPS-encrypted file. Use function
        `is_encrypted_sops_file` to check.


        Assumes file is YAML. Does not support JSON files.
        """
        cmd = ["sops", "--input-type", "yaml", "--decrypt", path]
        real_yaml = None
        try:
            decrypted_yaml = subprocess.check_output(cmd)
        except OSError:
            msg = "Failed to call SOPS to decrypt file at {}".format(path)
            msg += ", ensure sops is installed in PATH."
            raise AnsibleSopsError(msg)
        except subprocess.CalledProcessError:
            msg = "Failed to decrypt SOPS file at {}".format(path)
            raise AnsibleSopsError(msg)
        try:
            real_yaml = yaml.safe_load(decrypted_yaml)
        except yaml.parser.ParserError:
            msg = "Failed to parse YAML from decrypted SOPS file at {},".format(path)
            msg += " confirm file is YAML format."
            raise AnsibleSopsError(msg)
        return real_yaml


From the next you will try to use any of the encrypted vars files in an Ansible run, it will ask for the GPG passphrase to decrypt the file as required.

Thank you Conor Schaefer for the original version of the Python script.