Using sops with Ansible for vars encryption
Sops is a secret management tool from Mozilla. According to the official Github page, it is defined as:
sops is an editor of encrypted files that supports YAML, JSON, ENV, INI and BINARY formats and encrypts with AWS KMS, GCP KMS, Azure Key Vault and PGP.
In this blog post, I am going to show you how you can use it along with GPG to encrypt secrets in YAML files for your Ansible roles/playbooks.
Installation
Download the official binary from the release page, or you can build and install from the source code.
Basic usage (manual)
First let us create a .sops.yaml in the git repository root directory. In
this example, I am saying that I can encrypt any .yml using the two given
GPG fingerprints.
creation_rules:
# If vars file matchs "*.yml", then encrypt it more permissively.
- path_regex: \.yml$
pgp: "A85FF376759C994A8A1168D8D8219C8C43F6C5E1,2871635BE3B4E5C04F02B848C353BFE051D06C33"
Now to encrypt a file in place, I can use the following command:
sops -e -i host_vars/stg.dgplug.org.yml
If we open the file afterward, we will see something like below.
mysql_root_password: ENC[AES256_GCM,data:732TA7ps+qE=,iv:3azuZg4tqsLfe5IHDLJDKSUHmVk2c0g1Nl+oIcKOXRw=,tag:yD8iwmxmENww+waTs5Kzxw==,type:str]
mysql_user_password: ENC[AES256_GCM,data:YXBpO54=,iv:fQYATEWw4pv4lW5Ht8xiaBCliat8xdje5qdmb0Sff4Y=,tag:cncwg2Ops35l0lWegCSEJQ==,type:str]
mysql_user: ENC[AES256_GCM,data:cq/VgDlpRBxuHKM+cw==,iv:K+v6fkCIucMrMJ7pDRxFS/aHh0OCxqUcLJhZIgCsfA0=,tag:BD7l662OVOWRaHi2Rtw37g==,type:str]
mysql_db_name: ENC[AES256_GCM,data:hCgrKmU=,iv:/jnypeWdqUbIRy75q7OIODgZnaDpR3oTa0G/L8MRiZA=,tag:0k6cGNoDajUuKpKzrwQBaw==,type:str]
sops:
kms: []
gcp_kms: []
azure_kv: []
lastmodified: '2019-07-29T04:05:09Z'
mac: ENC[AES256_GCM,data:qp9yV3qj0tYf/EaO0Q3JdlpPvm5WY4ev1zGCVNoo+Anm/esj0WHlR7M7SNg54xRTUwMhRRnirx7IsEC8EZW1lE+8DObnskemcXm93CJOBfVzQOX/RvCSR4rMp2FgBEPZADCDiba1X2K/9myU96lADME0nkdmX9YjhOLMFJ6ll4o=,iv:2WNKhl81FI/qw6mRKpK5wRYjqK16q1ASeCJYpEeuhj4=,tag:v4PlGT4ZmPUxj7aYIendVg==,type:str]
pgp:
- created_at: '2019-07-29T04:05:06Z'
enc: |-
-----BEGIN PGP MESSAGE-----
wcFMA/uCql0ybaddARAADkHqV/mEgVoTxVHkRKW3mjajReKaQ5Mz/VwMal3GsjKr
8aGnghCjtgJE01wCBjrTfNKKmlf5JjMSFufy9pG0xE+OFOXt+pnJFDSro26QnPFG
VlvkvnjTxw4gV/mIvxUXTT6rmijvQSHMXdyEGmyHu3kNprKuuN37xZ4SYSWg7pdO
vee4DSOaw+XfdgYUF+YSEjKMZ+qevRtzJL4hg9SZvEsHMQObueMBywAc5pyR6LvS
ZuAS5SS+cPXnyMJLemqfU7L+XMw2NMrmNYFXOWnMMny1Hez5fcmebJp+wjDqWJTX
j9vJvXLuNAglFvL1yz2rNJYTb0mC9chLT7YxEa9Z9JHWezUB8ZYuOC6vRf18/Hz8
e6Gd2sncl4rleCxvKZF9qECsmFzs4p7H5M+O31jnjWdnPBD8a84Du3wdeWiI5cRF
d7/aUXEdGJQy7OVbzGE1alDOSIyDD2S73ou2du7s/79Wb11RwQV898OyGgmjWm0Y
7hTsBiBXnayQdjtg6qKlvoWIn79PU0YmDYLujMiXDQPJLV3ta82dcK2B1yTCLuSO
nGfFzNOSNemmniyEuMI16SrfYDsf9l/K3gec+TRNvEdc1GqO4gFblQWptPE7KIIC
oBVMDLUJSpOf5yF7Izedy5wBb8ZmzJAvpchvTMUls2+En4ifYh90cauXxiP6edPS
4AHkZMebg44aCefn4zMdKdUhbOFxbuA/4I3hQWLgmuJMFlcI4Orl5tsRjXwCfQ9S
jOdGAUJ8usV9gT4IXj73WfN1JJHj7DTgoORXFtDMs2Yy/rPPR4H9msSL4reJGZLh
dEAA
=LoMY
-----END PGP MESSAGE-----
fp: A85FF376759C994A8A1168D8D8219C8C43F6C5E1
- created_at: '2019-07-29T04:05:06Z'
enc: |-
-----BEGIN PGP MESSAGE-----
wcFMA0HZfc7pf7hDARAAX6xF6qP9KQJ5wLn0kv5WVf8HgOkk+2ziIuBH411hVEir
oN4bwwnL+DEYZm2aFvZl5zQElua7nGkQK041DecPbOCCBqThv3QKVMy2jScG5tTj
jxGgh8W/KxdwIY7teRaCRNDkT18IhtBc4SO2smJEtPeGIptvDHLjETBFbDnZeo6/
PG4QIA1Rfvm14n1NR56oibWduwvb1wrm4tGYPDx8mVgfugxeIhoqlZ87VrivrN+2
40S/rwWQ/aEVM1A8q19DGkXYVBcxQA1dGBrKLDPtiCq/LCSDNY4iuzMjzQuHPjgM
bS0lWv8fvSp6iIZlB3eSRPW9Ia8tRJpEfTLS8jiHlcCZ4Vy3fW6EijBf00iSy5hP
Y54TCERscXt1/UOW2ACYTNhMfuiO+WuG6Vjdns1NsSUVazqxmf+kBMwl06/HyUwL
KAYTOB2hipYUm6mlpSBfDgBKjQq8dgvxlWP0Ay0of0p4ZzFv7MepYkJA+gwb0hUt
rui9GVE/uys8W8buYqfM2ABzIG4GrH3rELh8eW8oPwlu7rgS7YGhyog2xabJyQnj
BZ65wwu5TzMq+n5v1+878teOzqqD/F+6X5dw0jF95bDHKdA64JR/Uxlj75sp59GH
e/+3UDm0UwSILDrYJkcVaTnrt3wPjQAw4ynKZuN+k6KvmDCkGquHNaM+2hqvWq/S
4AHkgpZHzmU14QmfJy7RN2HPduHhSeCh4LrhMzngJuJyH72G4IPlP4WwPJUhrKMt
UI+Azl61rIZEm6n82oWbY1gIrrIygWLg1OSD/0Ly6KbO4/W1NipWU53w4nAo7abh
3gkA
=YXu1
-----END PGP MESSAGE-----
fp: 2871635BE3B4E5C04F02B848C353BFE051D06C33
unencrypted_suffix: _unencrypted
version: 3.1.1
If you look closely, we can see that sops only encrypts the values, not the keys in the YAML file.
Decrypting as required on runtime
We can use a small Python script to enable runtime decryption of the file as required in Ansible. Create a vars_plugin directory in the git repository root, and then put the following code in there as sops.py.
from __future__ import (absolute_import, division, print_function)
__metaclass__ = type
DOCUMENTATION = '''
vars: sops
version_added: "N/A"
short_description: In charge of loading SOPS-encrypted vars
description:
- Loads SOPS-encrytped YAML vars into corresponding groups/hosts in group_vars/ and host_vars/ directories.
- Only SOPS-encrypted vars files, with a top-level "sops" key, will be loaded.
- Extends host/group vars logic from Ansible core.
notes:
- SOPS binary must be on path (missing will raise exception).
- Only supports YAML vars files (JSON files will raise exception).
- Only host/group vars are supported, other files will not be parsed.
options: []
'''
import os
import subprocess
import yaml
from ansible.errors import AnsibleError, AnsibleParserError
from ansible.module_utils._text import to_bytes, to_native, to_text
from ansible.inventory.host import Host
from ansible.inventory.group import Group
from ansible.utils.vars import combine_vars
FOUND = {}
# Import host_group_vars logic for file-walking functions.
# We'll still need to copy/paste and modify the `get_vars` function
# and edit below to insert a call to sops cli.
from ansible.plugins.vars.host_group_vars import VarsModule as HostGroupVarsModule
# All SOPS-encrypted vars files will have a top-level key called "sops".
# In order to determine whether a file is SOPS-encrypted, let's inspect
# such a key if it is found, and expect the following subkeys.
SOPS_EXPECTED_SUBKEYS = [
"lastmodified",
"mac",
"version",
]
class AnsibleSopsError(AnsibleError):
pass
class VarsModule(HostGroupVarsModule):
def get_vars(self, loader, path, entities, cache=True):
"""
Parses the inventory file and assembles host/group vars.
Lifted verbatim from ansible.plugins.vars.host_group_vars, with a single
in-line edit to support calling out to the SOPS CLI for decryption.
Only SOPS-encrypted files will be handled.
"""
if not isinstance(entities, list):
entities = [entities]
super(VarsModule, self).get_vars(loader, path, entities)
data = {}
for entity in entities:
if isinstance(entity, Host):
subdir = 'host_vars'
elif isinstance(entity, Group):
subdir = 'group_vars'
else:
raise AnsibleParserError("Supplied entity must be Host or Group, got %s instead" % (type(entity)))
# avoid 'chroot' type inventory hostnames /path/to/chroot
if not entity.name.startswith(os.path.sep):
try:
found_files = []
# load vars
b_opath = os.path.realpath(to_bytes(os.path.join(self._basedir, subdir)))
opath = to_text(b_opath)
key = '%s.%s' % (entity.name, opath)
if cache and key in FOUND:
found_files = FOUND[key]
else:
# no need to do much if path does not exist for basedir
if os.path.exists(b_opath):
if os.path.isdir(b_opath):
self._display.debug("\tprocessing dir %s" % opath)
found_files = loader.find_vars_files(opath, entity.name)
FOUND[key] = found_files
else:
self._display.warning("Found %s that is not a directory, skipping: %s" % (subdir, opath))
for found in found_files:
# BEGIN SOPS-specific logic
if self._is_encrypted_sops_file(found):
new_data = self._decrypt_sops_file(found)
if new_data: # ignore empty files
data = combine_vars(data, new_data)
# END SOPS-specific logic
except Exception as e:
raise AnsibleParserError(to_native(e))
return data
def _is_encrypted_sops_file(self, path):
"""
Check whether given filename is likely a SOPS-encrypted vars file.
Determined by presence of top-level 'sops' key in vars file.
Assumes file is YAML. Does not support JSON files.
"""
is_sops_file_result = False
with open(path, 'r') as f:
y = yaml.safe_load(f)
if type(y) == dict:
# All SOPS-encrypted vars files will have top-level "sops" key.
if 'sops' in y.keys() and type(y['sops'] == dict):
if all(k in y['sops'].keys() for k in SOPS_EXPECTED_SUBKEYS):
is_sops_file_result = True
return is_sops_file_result
def _decrypt_sops_file(self, path):
"""
Shells out to `sops` binary and reads decrypted vars from stdout.
Passes back dict to vars loader.
Assumes that a file is a valid SOPS-encrypted file. Use function
`is_encrypted_sops_file` to check.
Assumes file is YAML. Does not support JSON files.
"""
cmd = ["sops", "--input-type", "yaml", "--decrypt", path]
real_yaml = None
try:
decrypted_yaml = subprocess.check_output(cmd)
except OSError:
msg = "Failed to call SOPS to decrypt file at {}".format(path)
msg += ", ensure sops is installed in PATH."
raise AnsibleSopsError(msg)
except subprocess.CalledProcessError:
msg = "Failed to decrypt SOPS file at {}".format(path)
raise AnsibleSopsError(msg)
try:
real_yaml = yaml.safe_load(decrypted_yaml)
except yaml.parser.ParserError:
msg = "Failed to parse YAML from decrypted SOPS file at {},".format(path)
msg += " confirm file is YAML format."
raise AnsibleSopsError(msg)
return real_yaml
From the next you will try to use any of the encrypted vars files in an Ansible run, it will ask for the GPG passphrase to decrypt the file as required.
Thank you Conor Schaefer for the original version of the Python script.
