For the last few weeks, my blog feed was not showing up in the Fedora Planet. While trying to figure out what is wrong, Nirik pointed me to the 4 errors in the feed according to the W3C validator. If you don't know, I use a self developed Rust application called khata for my static blog. This means I had to fix these errors.

• Missing guid, just adding the guid to the feed items solved this.
• Relative URLs, this had to be fixed via the pulldown_cmark parser.
• Datetime error as error said "not RFC822" value. I am using chrono library, and was using to_rfc2822 call. Now, creating by hand with format RFC822 value.
• There is still one open issue dependent on the upstream fix.

The changes are in the git. I am using a build from there. I will make a release after the final remaining issue is fixed.

Oh, I also noticed how bad the code looks now as I can understand Rust better :)

Also, the other Planets, like Python and Tor, are still working for my feed.

SecureDrop right now runs on Ubuntu Xenial. We are working on moving to Ubuntu Focal. Here is the EPIC on the issue tracker.

While I was creating the Docker development environment on Focal, I noticed our tests were failing with the following message:

Traceback (most recent call last):                                                                                            
  File "/opt/venvs/securedrop-app-code/bin/pytest", line 5, in <module>              
    from pytest import console_main
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pytest/__init__.py", line 5, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/__init__.py", line 8, in <module>
    from _pytest.assertion import rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 31, in <module>
    from _pytest.assertion import util
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/util.py", line 14, in <module>
    import _pytest._code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/__init__.py", line 2, in <module>
    from .code import Code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/code.py", line 29, in <module>
    import pluggy
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 471, in <module>
    __version__ = version(__name__)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 438, in version
    return distribution(package).version
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 411, in distribution
    return Distribution.from_name(package)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 179, in from_name
    dists = resolver(name)
  File "<frozen importlib._bootstrap_external>", line 1382, in find_distributions
  File "/usr/lib/python3.8/importlib/metadata.py", line 466, in find_distributions
    found = cls._search_paths(context.name, context.path)
AttributeError: 'str' object has no attribute 'name'
make: *** [Makefile:238: test-focal] Error 1

Found out that pluggy dependency is too old. We update all application dependencies whenever there is a security update, but that is not the case with the development or testing requirements. These requirements only get installed on the developers' systems or in the CI. Then I figured that we are using a version of pytest 3 years old. That is why the code refuses to run on Python3.8 on Focal.

The update hell

Now, to update pluggy, I also had to update pytest and pytest-xdist, and that initial issue solved. But, this broke testinfra. Which we use in various molecule scenarios, say to test a staging or production server configurations or to test the Debian package builds. As I updated testinfra, molecule also required an update, which broke due to the old version of molecule in our pinned dependency. Now, to update I had to update molecule.yml and create.yml file for the different scenarios and get molecule-vagrant 0.3. Now, after I can run the molecule scenarios, I noticed that our old way of injecting variables to the pytest namespace via pytest_namespace function does not work. That function was dropped in between. So, had to fix that as the next step. This whole work is going on a draft PR, and meanwhile, some new changes merged with a new scenario. This means I will be spending more time to rebase properly without breaking these scenarios. The time takes to test each one of them, which frustrates me while fixing them one by one.

Lesson learned for me

We should look into all of our dependencies regularly and keep them updated. Otherwise, if we get into a similar situation again, someone else has to cry in a similar fashion :) Also, this is difficult to keep doing in a small team.

In July this year, I wrote a very initial Python module in Rust for OpenPGP, Johnnycanencrypt aka jce. It had very basic encryption, decryption, signing, verification, creation of new keys available. It uses https://sequoia-pgp.org library for the actual implementation.

I wanted to see if I can use such a Python module (which does not call out to the gpg2 executable) in the SecureDrop codebase.

First try (2 weeks ago)

Two weeks ago on the Friday, when I sat down to see if I can start using the module, within a few minutes, I understood it was not possible. The module was missing basic key management, more more refined control over creation, or expiration dates.

On that weekend, I wrote a KeyStore using file-based keys as backend and added most of the required functions to try again.

The last Friday

I sat down again; this time, I had a few friends (including Saptak, Nabarun) on video along with me, and together we tried to plug the jce inside SecureDrop container for Focal. After around 4 hours, we had around 5 failing tests (from 32) in the crypto-related tests. Most of the basic functionality was working, but we are stuck for the last few tests. As I was using the file system to store the keys (in simple .sec or .pub files), it was difficult to figure out the existing keys when multiple processes were creating/deleting keys in the same KeyStore.

Next try via a SQLite based KeyStore

Next, I replaced the KeyStore with an SQLite based backend. Now multiple processes can access the keys properly. With a few other updates, now I have only 1 failing test (where I have to modify the test properly) in that SecureDrop Focal patch.

While doing this experiment, I again found the benefits of writing the documentation of the library as I developed. Most of the time, I had to double-check against it to make sure that I am doing the right calls. I also added one example where one can verify the latest (10.0) Tor Browser download via Python.

In case you already use OpenPGP encryption in your tool/application, or you want to try it, please give jce a try. Works on Python3.7+. I tested on Linux and macOS, and it should work on Windows too. I have an issue open on that, and if you know how to do that, please feel free to submit a PR.

Right now, we are in the QA period for the SecureDrop 1.6.0 release. SecureDrop is an open-source whistleblower submission system that media organisations and NGOs can install to securely accept documents from anonymous sources. It was originally created by the late Aaron Swartz and is now managed by Freedom of the Press Foundation.

In this blog post I am going to explain how we do the QA for the release. I hope you can suggest some ways to improve the steps and make it better.

A few properties of SecureDrop to keep in mind during QA

• It is a complex web application that gets auto-updated via the Debian package on the servers running around the world.
• Security has the highest priority.
• We (the developers) do not have any access to those servers.
• Most of the servers are maintained by the system administrator from the news organization who very little time to manage and many times no Linux skill set.
• It is actually 2 servers per installation.
• Officially supported hardware list contains a few different generations of Intel NUCs, and Mac Minis.
• We also provide our own kernel package for these systems.

Test plan

The test plan for each release is tracked on the wiki and linked from the release ticket. Each time, we have the following categories of test cases:

• Application Acceptance Testing (related to general application usage)
• Basic Tails testing (for the tails updated GUI tool)
• Release specific changes for each release (detailed tests for new/updated features/fixes)
• Preflight check (for the release process itself)

We also have a private QA matrix, where we track things for each supported hardware (for update and new install) in a spreadsheet so that it becomes easier to understand if any basic test (say if the system is booting) is failing.

Please go through the test plan and the workflow. If you have any suggestions, you toot/tweet or email me. Your input is precious to make SecureDrop a better and safer tool for whistleblowers around the world.

SecureDrop workstation project's packages are reproducible. We use prebuilt wheels (by us) along with GPG signatures to verify and install them using pip during the Debian package building step. But, the way we built those wheels (standard pip command), they were not reproducible.

To fix this problem, Jennifer Helsby (aka redshiftzero) built a tool and the results are available at https://reproduciblewheels.com/. Every night her tool is building the top 100 + our dependency packages on Debian Buster and verifies the reproducibly of them. She has a detailed write up on the steps.

While this issue was fixed, a related issue was to have reproducible source tarballs. python3 setup.py sdist still does not give us a reproducible tarballs. Conor Schaefer, our CTO at the Freedom of the Press Foundation decided to tackle that issue using a few more lines of bash in our build scripts. Now we have reproducible wheels and source tarballs (based on specified timestamps) for our projects.

A few days ago, setuptools 50.0.0 release caused breakage to many projects. SecureDrop package builds was also broken. We use dh-virtualenv tool to build the packages. Initially, we tried to use the experimental build system from dh-virtualenv. We could specify the version of the setuptools to be installed in the virtualenv while creating it.

This approach worked for Xenial builds. As we are working to have proper builds on Focal (still work in progress), that was broken due to the above-mentioned change.

So, we again tried to use Python's venv module itself to create the virtual environment and use the wheels from the /usr/share/python-wheels directory to build the virtual environment. Which works very nicely on Xenial, but on Focal the default setuptools version is 44.0.0, which also failed to install the dependencies.

Now, we are actually getting the setuptools 46.0.0 wheel and replacing the build container's default setuptools wheel. The team spent a lot of time in debugging and finding a proper fix for the package builds. Hopefully, we will not get a similar breakage on the same kind of dependency error soon (the actual package dependencies are pinned via hashes).

I previously wrote about using the standard SOCKS proxy provided by the Tor Project on your system using Python, in particular using the requests module.

But, what about if you want to start a Tor SOCKS proxy only for your project (while the code is running), and use some existing code/module to do network calls (using sockets) over it?

Stem module to control the tor process

We can use Stem module to control the tor process in the system. In our example below, we will use PySocks module along with so that we can use urllib module to fetch some data. The code is based on one of the tutorial at the stem project. You will also notice that we are asking to use any exitnode from the Russia while creating the tor process.

Starting the tor process with a given SOCKS proxy port is super simple. And then we replace socket.socket with socks.socksocket (after the right configuration), and socket.getaddrinfo with our own implementation to make sure that we don't leak DNS information.

import io
import socket
import urllib.request

import socks
import stem.process
from stem.util import term

SOCKS_PORT = 7000


def query(url):
    """
    Uses urllib to fetch a site using the proxy on the SOCKS_PORT.
    """
    return urllib.request.urlopen(url).read()


def print_bootstrap_lines(line):
    if "Bootstrapped " in line:
        print(term.format(line, term.Color.BLUE))


def getaddrinfo(*args):
    "Let us do the actual DNS resolution in the SOCKS5 proxy"
    return [(socket.AF_INET, socket.SOCK_STREAM, 6, "", (args[0], args[1]))]


def main():
    # Start an instance of Tor configured to only exit through Russia. This prints
    # Tor's bootstrap information as it starts. Note that this likely will not
    # work if you have another Tor instance running.
    print(term.format("Starting Tor:\n", term.Attr.BOLD))

    tor_process = stem.process.launch_tor_with_config(
        config={
            "SocksPort": str(SOCKS_PORT),
            "ExitNodes": "{ru}",
        },
        init_msg_handler=print_bootstrap_lines,
    )

    print(term.format("\nChecking our endpoint:\n", term.Attr.BOLD))
    socks.set_default_proxy(
        socks.PROXY_TYPE_SOCKS5, "localhost", 7000, True, None, None
    )
    socket.socket = socks.socksocket
    socket.getaddrinfo = getaddrinfo
    try:
        print(term.format(query("https://icanhazip.com"), term.Color.BLUE))
    finally:
        tor_process.kill()  # stops tor


if __name__ == "__main__":
    main()

If you are surprised about the getaddrinfo function above, it is just returning the same domain name it received (instead of an IP address). The actual DNS resolution happens over Tor at the proxy level.

A few weeks ago on August 12 Freedom of the Press had one event where we talked with Paul Lewis from The Gurdian about their use of SecureDrop project, and how it helps in doing the investigative journalism work. My dear friend Harlo was the host for the evening. You can watch the event on Youtube.

The second half of the event was a live demo of the new SecureDrop Workstation project.

SecureDrop is an open source whistleblower submission system that media organizations and NGOs can install to securely accept documents from anonymous sources. It was originally created by the late Aaron Swartz and is now managed by Freedom of the Press Foundation. SecureDrop is available in 20 languages.

The current SecureDrop is dependent heavily on air-gapped Tails systems. This means increased security but also means a lot of time in accessing the submissions by the journalists. SecureDrop Workstation is the next generation system coming up to help in reducing this and also provide much smoother user experience without giving up the security.

In simple words, it is a system built on top of the QubesOS, where journalists can access the submissions via a desktop application and can communicate with sources with much ease.

And the view for the journalists.

You can read in detail about the idea and reasoning behind this project in this whitepaper.

At the beginning of this year, we also published the news on the first (for alpha release) security audit. You can directly jump into the full audit report too.

If you follow DEF CON, you may remember the last panel in 2019 where the usage of SecureDrop and the security audit was discussed.

You can also watch the talk at USENIX Enigma 2020 by Jennifer Helsby.

Technologies used, and how can you help?

SecureDrop is a Free Software project built with similar technologies that you all see and use every day. The main server side is written in Python, with a lot of Ansible, and molecule to test. The web application is written in Flask and contains tests written in Pytest. The documentation is written in Sphinx and maintained via ReadTheDocs. Development setup can be created using Docker, and full-scale production vms (for testing) can be created using the Vagrant.

The translations are maintained in the Weblate by the amazing community at Localization Lab.

The SecureDrop Workstation client is written in PyQt. There are many related Python modules in our Github. The packages are reproducible builds, except the RPMS.

The sources and journalists access SecureDrop via Tor Project. In Qubes OS we use both Fedora and Debian VMs.

There are many issues opened in all of these project’s repositories, and by using/testing them, you most probably will be able to find more things to be fixed. There are problems which can be solved with people from different experiences.

We do daily standup at 16:00UTC in this meeting link. Please feel free to join in and say hi.

Tomorrow at 17:00UTC, Tor Project is hosting the next session of PrivChat, titled "The Good, the Bad, and the Ugly of Censorship Circumvention". You can watch it live on Youtube.

This 2nd edition of PrivChat is about the Good, the Bad and the Ugly that is happening in the front lines of censorship circumvention. Cory Doctorow will be the host for the evening, and the following people will be participating:

• Felicia Anthonio (Access Now),
• Vrinda Bhandari (Internet Freedom Foundation ),
• Cecylia Bocovich (Tor Project ),
• Arturo Filastò (OONI )

Don't miss your chance to listen to them. You can ask questions via the Youtube chat.

I was using my dns-tor-proxy tool in the AppVMs in my QubesOS system. But, at the same time I was trying to figure out how to make it the default DNS system for the whole Qubes.

ahf provided me a shell script showing how he is forwarding the DNS requests to a VPN interface. I modified the same so that all of standard DNS queries become DoH queries over the Tor network.

Setting up sys-firewall

In the following example, I am setting up the sys-firewall service VM. All other AppVMs connected to this VM as netvm will be use dns-tor-proxy without any modification.

Make sure that the template for sys-firewall has the latest Tor installed. You can get it from the official Tor repository.

Download (or build) the latest dns-tor-proxy 0.3.0 release, and put the file (as executable) in /rw/config/ directory.

Next, modify the /rw/config/rc.local file & add the following lines.

systemctl start tor
sh /rw/config/dns.sh
/rw/config/dns-tor-proxy --doh &

As you can see, we are executing another script at /rw/config/dns.sh, which has the following content. Remember to modify the DNS value to the right IP for your sys-firewall vm.

#!/bin/sh

QUBES_DNS_SERVERS="10.139.1.1 10.139.1.2"
DNS=10.137.0.x

# accept DNS requests from the other vms

iptables -I INPUT -i vif+ -p udp --dport 53 -j ACCEPT
iptables -I INPUT -i vif+ -p tcp --dport 53 -j ACCEPT

# Clean up our NAT firewall rules.
iptables --flush PR-QBS --table nat

# We take incoming traffic on TCP and UDP port 53 and forward to
# our DNS server.
for QUBES_DNS_SERVER in ${QUBES_DNS_SERVERS} ; do
    iptables --append PR-QBS --table nat --in-interface vif+ --protocol tcp --destination "${QUBES_DNS_SERVER}" --dport 53 --jump DNAT --to-destination "${DNS}":53
    iptables --append PR-QBS --table nat --in-interface vif+ --protocol udp --destination "${QUBES_DNS_SERVER}" --dport 53 --jump DNAT --to-destination "${DNS}":53
done

# Log *other* DNS service connections. This part is optional, but ensures that
# you can monitor if one of your VM's is making any traffic on port 53 with
# either TCP or UDP. If you want to log *every* DNS "connection", including the
# ones to QUBES_DNS_SERVERS, you can either move these commands up before the
# for-loop in this file or change the --apend option to be an --insert instead.
iptables --append PR-QBS --table nat --in-interface vif+ --protocol tcp --dport 53 --jump LOG --log-level 1 --log-prefix 'DNS Query: '
iptables --append PR-QBS --table nat --in-interface vif+ --protocol udp --dport 53 --jump LOG --log-level 1 --log-prefix 'DNS Query: '

Now, restart your sys-firewall vm. And you are all set for your DNS queries.