Yesterday Alex Birsan posted a blog explaining how he made a supply chain attack on various companies via dependencies. I was waiting for this blog from last August when we noticed the mentioned packages on PyPI (and removed). I reached out to Alex to figure out more about the packages, and he said he will write a blog post.

This is the same time when @rsobers also tweeted about how any similar attack works via DNS.

At SecureDrop project, we spend a lot of time figuring out a way to defend against similar attacks. My PyCon US 2019 talk explained the full process. In simple words, we are doing the following:

• Verify the source of every python dependency before adding/updating them. This is a manual step done by human eyes.
• Build wheels from the verified source and store them into a package repository along with OpenPGP signed sha256sums.
• Before building the Debian package, make sure to verify the OpenPGP signature, and also install only from the known package repository using verified (again via OpenPGP signature) wheel hashes (from the wheels we built ourselves).

Now, in the last few months, we modified these steps, and made it even easier. Now, all of our wheels are build from same known sources, and all of the wheels are then built as reproducible and stored in git (via LFS). The final Debian packages are again reproducible. Along with this and the above mentioned OpenPGP signature + package sha256sums verification via pip. We also pass --no-index the argument to pip now, to make sure that all the dependencies are getting picked up from the local directory.

Oh, and I also forgot to mention that all of the project source tarballs used in SecureDrop workstation package building are also reproducible. If you are interested in using the same in your Python project (or actually for any project's source tarball), feel free to have a look at this function.

There is also diff-review mailing list (very low volume), where we post a signed message of any source package update we review.

For more than a month, my primary task at SecureDrop land is to make the project ready for a distribution update. The current system runs on Ubuntu Xenial, and the goal is to upgrade to Ubuntu Focal. The deadline is around February 2021, and we will also disable Onion service v2 in the same release.

Tracking issue

There is a tracking issue related to all the changes for Focal. After the initial Python language-related updates, I had to fix the Debian packages for the same. Then comes the Ansible roles for the whole system, setting up the rebase + full integration tests for the system in CI. A lot of new issues were found when we managed to get Focal based staging instances in the CI.

OSSEC test failure

This particular test is failing on Focal staging. It checks for the port 1514 listening on UDP on the monitor server via OSSEC. The first thing we noticed that the output of the command is different in Xenial and on Focal. While looking at the details, we also noticed that the OSSEC service is failing on Focal. Now, this starts via a sysv script in the /etc/init.d/ directory. My initial try was to follow the solution mentioned here. But, the maild service was still failing most of the time. Later, I decided to move to a more straightforward forking based service file. Works well for the OSSEC monitoring service. So, I decided to have the same systemd service file for the agent in the app server.

But, the installation step via Ansible failed before any configuration. When we install the .deb packages, our configuration provider package tries to restart the service via the post-installation script. And that fails with a complaint that /var/ossec/etc/client.keys is not found. If I try to start the service manually, I get the same error with an exit code 1.

At this moment, I was trying to identify how was it working before, we are using the same codebase + the sysv on Xenial, but package installation works. The systemd generates the following service file:

# Automatically generated by systemd-sysv-generator

[Unit]
Documentation=man:systemd-sysv-generator(8)
SourcePath=/etc/init.d/ossec
Description=LSB: Start daemon at boot time
After=remote-fs.target

[Service]
Type=forking
Restart=no
TimeoutSec=5min
IgnoreSIGPIPE=no
KillMode=process
GuessMainPID=no
RemainAfterExit=yes
SuccessExitStatus=5 6
ExecStart=/etc/init.d/ossec start
ExecStop=/etc/init.d/ossec stop

After digging for hours, I noticed exit 0 at the end of the old sysv script. This file was last modified by late James Dolan around seven years ago, and we never had to touch the same.

Now I am planning to have 1 in the SuccessExitStaus= line of the service file for the agent. This will keep the behavior the same as the old sysv file.

[Unit]
Description=OSSEC service

[Service]
Type=forking
ExecStart=/var/ossec/bin/ossec-control start
ExecStop=/var/ossec/bin/ossec-control stop
RemainAfterExit=True
SuccessExitStatus=1

[Install]
WantedBy=multi-user.target

We use SQLite3 as the database in SecureDrop. We use SQLAlchemy to talk the database and Alembic for migrations. Some of those migrations are written by hand.

Most of my work time in the last month went to getting things ready for Ubuntu Focal 20.04. We currently use Ubuntu Xenial 16.04. During this, I noticed 17 test failures related to the Alembic on Focal but works fine on Xenial. After digging a bit more, these are due to the missing reference to temporary tables we used during migrations. With some more digging, I found this entry on the SQLite website:

Compatibility Note: The behavior of ALTER TABLE when renaming a table was enhanced in versions 3.25.0 (2018-09-15) and 3.26.0 (2018-12-01) in order to carry the rename operation forward into triggers and views that reference the renamed table. This is considered an improvement. Applications that depend on the older (and arguably buggy) behavior can use the PRAGMA legacy_alter_table=ON statement or the SQLITE_DBCONFIG_LEGACY_ALTER_TABLE configuration parameter on sqlite3_db_config() interface to make ALTER TABLE RENAME behave as it did prior to version 3.25.0.

This is what causing the test failures as SQLite upgraded to 3.31.1 on Focal from 3.11.0 on Xenial.

According to the docs, we can fix the error by adding the following in the env.py.

diff --git a/securedrop/alembic/env.py b/securedrop/alembic/env.py
index c16d34a5a..d6bce65b5 100644
--- a/securedrop/alembic/env.py
+++ b/securedrop/alembic/env.py
@@ -5,6 +5,8 @@ import sys
 
 from alembic import context
 from sqlalchemy import engine_from_config, pool
+from sqlalchemy.engine import Engine
+from sqlalchemy import event
 from logging.config import fileConfig
 from os import path
 
@@ -16,6 +18,12 @@ fileConfig(config.config_file_name)
 sys.path.insert(0, path.realpath(path.join(path.dirname(__file__), '..')))
 from db import db  # noqa
 
+@event.listens_for(Engine, "connect")
+def set_sqlite_pragma(dbapi_connection, connection_record):
+    cursor = dbapi_connection.cursor()
+    cursor.execute("PRAGMA legacy_alter_table=ON")
+    cursor.close()
+
 try:
     # These imports are only needed for offline generation of automigrations.
     # Importing them in a prod-like environment breaks things.

Later, John found an even simpler way to do the same for only the migrations impacted.

Last week, while setting up a Fedora 33 system, I thought of running the SecureDrop development container there, but using podman instead of the Docker setup we have.

I tried to make minimal changes to our existing scripts. Added a ~/bin/docker file, with podman $@ inside (and the sha-bang line).

Next, I provided the proper label for SELinux:

sudo chcon -Rt container_file_t securedrop

The SecureDrop container runs as the normal user inside of the Docker container. I can not do the same here as the filesystem gets mounted as root, and I can not write in it. So, had to modify one line in the bash script, and also disabled another function call which deletes the /dev/random file inside of the container.

diff --git a/securedrop/bin/dev-shell b/securedrop/bin/dev-shell
index ef424bc01..37215b551 100755
--- a/securedrop/bin/dev-shell
+++ b/securedrop/bin/dev-shell
@@ -72,7 +72,7 @@ function docker_run() {
            -e LANG=C.UTF-8 \
            -e PAGE_LAYOUT_LOCALES \
            -e PATH \
-           --user "${USER:-root}" \
+           --user root \
            --volume "${TOPLEVEL}:${TOPLEVEL}" \
            --workdir "${TOPLEVEL}/securedrop" \
            --name "${SD_CONTAINER}" \
diff --git a/securedrop/bin/run b/securedrop/bin/run
index e82cc6320..0c11aa8db 100755
--- a/securedrop/bin/run
+++ b/securedrop/bin/run
@@ -9,7 +9,7 @@ cd "${REPOROOT}/securedrop"
 source "${BASH_SOURCE%/*}/dev-deps"
 
 run_redis &
-urandom
+#urandom
 run_sass --watch &
 maybe_create_config_py
 reset_demo

This time I felt that build time for verifying each cached layer is much longer than what it used to be for podman. Maybe I am just mistaken. The SecureDrop web application is working very fine inside.

Package build containers

We also use containers to build Debian packages. And those molecule scenarios were failing as ansible.posix.synchronize module could not sync to a podman container. I asked if there is anyway to do that, and by the time I woke up, Adam Miller had a branch that fixed the issue. I directly used the same in my virtual environment. The package build was successful. Then, the testinfra tests failed due as it could not create the temporary directory inside of the container. I already opened an issue for the same.

SecureDrop right now runs on Ubuntu Xenial. We are working on moving to Ubuntu Focal. Here is the EPIC on the issue tracker.

While I was creating the Docker development environment on Focal, I noticed our tests were failing with the following message:

Traceback (most recent call last):                                                                                            
  File "/opt/venvs/securedrop-app-code/bin/pytest", line 5, in <module>              
    from pytest import console_main
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pytest/__init__.py", line 5, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/__init__.py", line 8, in <module>
    from _pytest.assertion import rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 31, in <module>
    from _pytest.assertion import util
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/util.py", line 14, in <module>
    import _pytest._code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/__init__.py", line 2, in <module>
    from .code import Code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/code.py", line 29, in <module>
    import pluggy
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 471, in <module>
    __version__ = version(__name__)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 438, in version
    return distribution(package).version
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 411, in distribution
    return Distribution.from_name(package)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 179, in from_name
    dists = resolver(name)
  File "<frozen importlib._bootstrap_external>", line 1382, in find_distributions
  File "/usr/lib/python3.8/importlib/metadata.py", line 466, in find_distributions
    found = cls._search_paths(context.name, context.path)
AttributeError: 'str' object has no attribute 'name'
make: *** [Makefile:238: test-focal] Error 1

Found out that pluggy dependency is too old. We update all application dependencies whenever there is a security update, but that is not the case with the development or testing requirements. These requirements only get installed on the developers' systems or in the CI. Then I figured that we are using a version of pytest 3 years old. That is why the code refuses to run on Python3.8 on Focal.

The update hell

Now, to update pluggy, I also had to update pytest and pytest-xdist, and that initial issue solved. But, this broke testinfra. Which we use in various molecule scenarios, say to test a staging or production server configurations or to test the Debian package builds. As I updated testinfra, molecule also required an update, which broke due to the old version of molecule in our pinned dependency. Now, to update I had to update molecule.yml and create.yml file for the different scenarios and get molecule-vagrant 0.3. Now, after I can run the molecule scenarios, I noticed that our old way of injecting variables to the pytest namespace via pytest_namespace function does not work. That function was dropped in between. So, had to fix that as the next step. This whole work is going on a draft PR, and meanwhile, some new changes merged with a new scenario. This means I will be spending more time to rebase properly without breaking these scenarios. The time takes to test each one of them, which frustrates me while fixing them one by one.

Lesson learned for me

We should look into all of our dependencies regularly and keep them updated. Otherwise, if we get into a similar situation again, someone else has to cry in a similar fashion :) Also, this is difficult to keep doing in a small team.

In July this year, I wrote a very initial Python module in Rust for OpenPGP, Johnnycanencrypt aka jce. It had very basic encryption, decryption, signing, verification, creation of new keys available. It uses https://sequoia-pgp.org library for the actual implementation.

I wanted to see if I can use such a Python module (which does not call out to the gpg2 executable) in the SecureDrop codebase.

First try (2 weeks ago)

Two weeks ago on the Friday, when I sat down to see if I can start using the module, within a few minutes, I understood it was not possible. The module was missing basic key management, more more refined control over creation, or expiration dates.

On that weekend, I wrote a KeyStore using file-based keys as backend and added most of the required functions to try again.

The last Friday

I sat down again; this time, I had a few friends (including Saptak, Nabarun) on video along with me, and together we tried to plug the jce inside SecureDrop container for Focal. After around 4 hours, we had around 5 failing tests (from 32) in the crypto-related tests. Most of the basic functionality was working, but we are stuck for the last few tests. As I was using the file system to store the keys (in simple .sec or .pub files), it was difficult to figure out the existing keys when multiple processes were creating/deleting keys in the same KeyStore.

Next try via a SQLite based KeyStore

Next, I replaced the KeyStore with an SQLite based backend. Now multiple processes can access the keys properly. With a few other updates, now I have only 1 failing test (where I have to modify the test properly) in that SecureDrop Focal patch.

While doing this experiment, I again found the benefits of writing the documentation of the library as I developed. Most of the time, I had to double-check against it to make sure that I am doing the right calls. I also added one example where one can verify the latest (10.0) Tor Browser download via Python.

In case you already use OpenPGP encryption in your tool/application, or you want to try it, please give jce a try. Works on Python3.7+. I tested on Linux and macOS, and it should work on Windows too. I have an issue open on that, and if you know how to do that, please feel free to submit a PR.

Right now, we are in the QA period for the SecureDrop 1.6.0 release. SecureDrop is an open-source whistleblower submission system that media organisations and NGOs can install to securely accept documents from anonymous sources. It was originally created by the late Aaron Swartz and is now managed by Freedom of the Press Foundation.

In this blog post I am going to explain how we do the QA for the release. I hope you can suggest some ways to improve the steps and make it better.

A few properties of SecureDrop to keep in mind during QA

• It is a complex web application that gets auto-updated via the Debian package on the servers running around the world.
• Security has the highest priority.
• We (the developers) do not have any access to those servers.
• Most of the servers are maintained by the system administrator from the news organization who very little time to manage and many times no Linux skill set.
• It is actually 2 servers per installation.
• Officially supported hardware list contains a few different generations of Intel NUCs, and Mac Minis.
• We also provide our own kernel package for these systems.

Test plan

The test plan for each release is tracked on the wiki and linked from the release ticket. Each time, we have the following categories of test cases:

• Application Acceptance Testing (related to general application usage)
• Basic Tails testing (for the tails updated GUI tool)
• Release specific changes for each release (detailed tests for new/updated features/fixes)
• Preflight check (for the release process itself)

We also have a private QA matrix, where we track things for each supported hardware (for update and new install) in a spreadsheet so that it becomes easier to understand if any basic test (say if the system is booting) is failing.

Please go through the test plan and the workflow. If you have any suggestions, you toot/tweet or email me. Your input is precious to make SecureDrop a better and safer tool for whistleblowers around the world.

SecureDrop workstation project's packages are reproducible. We use prebuilt wheels (by us) along with GPG signatures to verify and install them using pip during the Debian package building step. But, the way we built those wheels (standard pip command), they were not reproducible.

To fix this problem, Jennifer Helsby (aka redshiftzero) built a tool and the results are available at https://reproduciblewheels.com/. Every night her tool is building the top 100 + our dependency packages on Debian Buster and verifies the reproducibly of them. She has a detailed write up on the steps.

While this issue was fixed, a related issue was to have reproducible source tarballs. python3 setup.py sdist still does not give us a reproducible tarballs. Conor Schaefer, our CTO at the Freedom of the Press Foundation decided to tackle that issue using a few more lines of bash in our build scripts. Now we have reproducible wheels and source tarballs (based on specified timestamps) for our projects.

A few days ago, setuptools 50.0.0 release caused breakage to many projects. SecureDrop package builds was also broken. We use dh-virtualenv tool to build the packages. Initially, we tried to use the experimental build system from dh-virtualenv. We could specify the version of the setuptools to be installed in the virtualenv while creating it.

This approach worked for Xenial builds. As we are working to have proper builds on Focal (still work in progress), that was broken due to the above-mentioned change.

So, we again tried to use Python's venv module itself to create the virtual environment and use the wheels from the /usr/share/python-wheels directory to build the virtual environment. Which works very nicely on Xenial, but on Focal the default setuptools version is 44.0.0, which also failed to install the dependencies.

Now, we are actually getting the setuptools 46.0.0 wheel and replacing the build container's default setuptools wheel. The team spent a lot of time in debugging and finding a proper fix for the package builds. Hopefully, we will not get a similar breakage on the same kind of dependency error soon (the actual package dependencies are pinned via hashes).

A few weeks ago on August 12 Freedom of the Press had one event where we talked with Paul Lewis from The Gurdian about their use of SecureDrop project, and how it helps in doing the investigative journalism work. My dear friend Harlo was the host for the evening. You can watch the event on Youtube.

The second half of the event was a live demo of the new SecureDrop Workstation project.

SecureDrop is an open source whistleblower submission system that media organizations and NGOs can install to securely accept documents from anonymous sources. It was originally created by the late Aaron Swartz and is now managed by Freedom of the Press Foundation. SecureDrop is available in 20 languages.

The current SecureDrop is dependent heavily on air-gapped Tails systems. This means increased security but also means a lot of time in accessing the submissions by the journalists. SecureDrop Workstation is the next generation system coming up to help in reducing this and also provide much smoother user experience without giving up the security.

In simple words, it is a system built on top of the QubesOS, where journalists can access the submissions via a desktop application and can communicate with sources with much ease.

And the view for the journalists.

You can read in detail about the idea and reasoning behind this project in this whitepaper.

At the beginning of this year, we also published the news on the first (for alpha release) security audit. You can directly jump into the full audit report too.

If you follow DEF CON, you may remember the last panel in 2019 where the usage of SecureDrop and the security audit was discussed.

You can also watch the talk at USENIX Enigma 2020 by Jennifer Helsby.

Technologies used, and how can you help?

SecureDrop is a Free Software project built with similar technologies that you all see and use every day. The main server side is written in Python, with a lot of Ansible, and molecule to test. The web application is written in Flask and contains tests written in Pytest. The documentation is written in Sphinx and maintained via ReadTheDocs. Development setup can be created using Docker, and full-scale production vms (for testing) can be created using the Vagrant.

The translations are maintained in the Weblate by the amazing community at Localization Lab.

The SecureDrop Workstation client is written in PyQt. There are many related Python modules in our Github. The packages are reproducible builds, except the RPMS.

The sources and journalists access SecureDrop via Tor Project. In Qubes OS we use both Fedora and Debian VMs.

There are many issues opened in all of these project’s repositories, and by using/testing them, you most probably will be able to find more things to be fixed. There are problems which can be solved with people from different experiences.

We do daily standup at 16:00UTC in this meeting link. Please feel free to join in and say hi.