Kushal Das

FOSS and life. Kushal Das talks here.


Update hell due to not updating for a long time

SecureDrop right now runs on Ubuntu Xenial. We are working on moving to Ubuntu Focal. Here is the EPIC on the issue tracker.

While I was creating the Docker development environment on Focal, I noticed our tests were failing with the following message:

Traceback (most recent call last):                                                                                            
  File "/opt/venvs/securedrop-app-code/bin/pytest", line 5, in <module>              
    from pytest import console_main
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pytest/__init__.py", line 5, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/__init__.py", line 8, in <module>
    from _pytest.assertion import rewrite
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/rewrite.py", line 31, in <module>
    from _pytest.assertion import util
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/assertion/util.py", line 14, in <module>
    import _pytest._code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/__init__.py", line 2, in <module>
    from .code import Code
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/_pytest/_code/code.py", line 29, in <module>
    import pluggy
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 471, in <module>
    __version__ = version(__name__)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 438, in version
    return distribution(package).version
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 411, in distribution
    return Distribution.from_name(package)
  File "/opt/venvs/securedrop-app-code/lib/python3.8/site-packages/importlib_metadata/__init__.py", line 179, in from_name
    dists = resolver(name)
  File "<frozen importlib._bootstrap_external>", line 1382, in find_distributions
  File "/usr/lib/python3.8/importlib/metadata.py", line 466, in find_distributions
    found = cls._search_paths(context.name, context.path)
AttributeError: 'str' object has no attribute 'name'
make: *** [Makefile:238: test-focal] Error 1

Found out that pluggy dependency is too old. We update all application dependencies whenever there is a security update, but that is not the case with the development or testing requirements. These requirements only get installed on the developers' systems or in the CI. Then I figured that we are using a version of pytest 3 years old. That is why the code refuses to run on Python3.8 on Focal.

The update hell

Now, to update pluggy, I also had to update pytest and pytest-xdist, and that initial issue solved. But, this broke testinfra. Which we use in various molecule scenarios, say to test a staging or production server configurations or to test the Debian package builds. As I updated testinfra, molecule also required an update, which broke due to the old version of molecule in our pinned dependency. Now, to update I had to update molecule.yml and create.yml file for the different scenarios and get molecule-vagrant 0.3. Now, after I can run the molecule scenarios, I noticed that our old way of injecting variables to the pytest namespace via pytest_namespace function does not work. That function was dropped in between. So, had to fix that as the next step. This whole work is going on a draft PR, and meanwhile, some new changes merged with a new scenario. This means I will be spending more time to rebase properly without breaking these scenarios. The time takes to test each one of them, which frustrates me while fixing them one by one.

Lesson learned for me

We should look into all of our dependencies regularly and keep them updated. Otherwise, if we get into a similar situation again, someone else has to cry in a similar fashion :) Also, this is difficult to keep doing in a small team.

Reproducible wheels at SecureDrop


SecureDrop workstation project's packages are reproducible. We use prebuilt wheels (by us) along with GPG signatures to verify and install them using pip during the Debian package building step. But, the way we built those wheels (standard pip command), they were not reproducible.

To fix this problem, Jennifer Helsby (aka redshiftzero) built a tool and the results are available at https://reproduciblewheels.com/. Every night her tool is building the top 100 + our dependency packages on Debian Buster and verifies the reproducibly of them. She has a detailed write up on the steps.

While this issue was fixed, a related issue was to have reproducible source tarballs. python3 setup.py sdist still does not give us a reproducible tarballs. Conor Schaefer, our CTO at the Freedom of the Press Foundation decided to tackle that issue using a few more lines of bash in our build scripts. Now we have reproducible wheels and source tarballs (based on specified timestamps) for our projects.

SecureDrop package build breakage due to setuptools

A few days ago, setuptools 50.0.0 release caused breakage to many projects. SecureDrop package builds was also broken. We use dh-virtualenv tool to build the packages. Initially, we tried to use the experimental build system from dh-virtualenv. We could specify the version of the setuptools to be installed in the virtualenv while creating it.

This approach worked for Xenial builds. As we are working to have proper builds on Focal (still work in progress), that was broken due to the above-mentioned change.

So, we again tried to use Python's venv module itself to create the virtual environment and use the wheels from the /usr/share/python-wheels directory to build the virtual environment. Which works very nicely on Xenial, but on Focal the default setuptools version is 44.0.0, which also failed to install the dependencies.

Now, we are actually getting the setuptools 46.0.0 wheel and replacing the build container's default setuptools wheel. The team spent a lot of time in debugging and finding a proper fix for the package builds. Hopefully, we will not get a similar breakage on the same kind of dependency error soon (the actual package dependencies are pinned via hashes).

Single file implementation of PEP582

During 2018 CPython core developer sprint, I worked on the PEP 582. The goal was to help all the newbie learners during their first day writing Python by skipping the whole complexity of virtual environments. The PEP contains a reference implementation. During the sprint itself, a few core developers did not like the idea of yet another feature focusing only on newbies. Instead, there was another discussion to create a single tool to solve all the problems in the packaging world.

Now, in 2020, we, the Python trainers, are still facing the same problem. How to explain the whole idea of virtual environments to the newbie? Should we teach the concept of the Operating system and shells and environments or teach Python?

A few nights ago, during a chat with Brett Cannon, he suggested having a single tool to do the same and see how people react.

Introducing project PEP582

PEP582 is a single file implementation of the above-mentioned idea. You can call it a stupid hack, but it works.

Installing the project and using it

First, get the latest copy of the source, and then you can install it (without any root/administrator access) using Python itself. If you are using Ubuntu or Debian system, it assumes that you already have python3-venv and python3-pip installed.

curl https://raw.githubusercontent.com/kushaldas/pep582/master/pep582.py -o pep582.py
python3 pep582.py --install
Successfully installed in /home/kdas/.local/lib/python3.7/site-packages/pep582.py

After this, in any directory, if you create a __pypackages__ directory, python executable will start using it. If you install any package via pip, it will also install in the __pypackages__ directory.

pep582 demo

It does not modify the PATH variable. If you want to install and use any executable. This is not a replacement for virtual environments. The tool is here to help the newbies to start programming fast. For more advanced work, they will have to learn about virtual environments.

Oh, this works on Windows too. I never tested on Mac yet.

pep582 demo

Please play around, and let me know any improvement you want to see. You are always welcome to open issues in the project repository.

A few things from last week

Last Monday I wrote some tips on security while working from home. in Economic Time. This includes some basic steps everyone of us can take. If you want to follow better practices, please read https://ssd.eff.org.

There was a fake news going around in WhatsApp about government surveillance, a story was published to bust this myth, and I answered a questions for that story. You can read it in either The Wire or Alt News site.

More power to you my friend

With Chelsea and Micah

Today marks 365 days of incarceration of Chelsea Manning, with around $256000 in fines. She has not been charged for any crime.

At home, Py (daughter) wanted to know why her Wonder Woman is still in the dungeon? We had to explain Grand jury subpoena to her in simple terms and explain that her Wonder Woman aunty is resisting it, even though that means she is in a dungeon (where most good people go when they fight for truth against the big powers of the world). Py now wants to go to USA and meet Chelsea. The other day, Py also declared that after growing up she will fight for others and go to dungeon just like Chelsea.

Chelsea spoke about her believes and principles again and again, and why did she feel that leaking the war crimes to the world was her duty. By the same principles, she is standing up to the secret hearings of grand juries now. Everyone knows that there is no good reason to put her back into the prison system, but the government still did that.

UN officials already accused the US government of using torture against Chelsea. The officials also mentioned to the US government:

believe that subjecting Chelsea to more punishment will change her mind, they are gravely mistaken.

After waking up today morning, I suddenly found she again tried to commit suicide and now recovering in the hospital. She had previously spent 7+years in prison including 11 months of solitary confinement, and these things in total causes a lot of mental health issues.

I hope for her speedy recovery and also hope someone in the US judicial system will see the injustice to her and release her soon. Meanwhile, we all can send her letters (on white paper, handwritten or drawn) to the following address:

Chelsea Elizabeth Manning
William G. Truesdale Adult Detention Center
2001 Mill Road
Alexandria, VA 22314

You can also read her statement released on Aaron Swartz Day 2019.

No summer training 2020

No summer training 2020 for me. Last year’s batch was beyond my capability to handle. Most of the participants did not follow anything we taught in the course, instead, they kept demanding more things.

I already started receiving mails from a few people who wants to join in the training in 2020. But, there is no positive answer from my side.

All the course materials are public, the logs are also available. We managed to continue this training for 12 years. This is way more than I could ever imagine.

As I was feeling a bit sad about this, the keynote at Railsconf 2019 from DHH actually helped me a lot to feel better.

5 months of Internet shutdown in Kashmir and more fascist attacks in India

From 5th August 2019, Kashmir is under a communication shutdown. SMS service for a particular connection provider is now available for postpaid users, but Internet is still down for all Indian citizens of Kashmir.

This is above 155 days of Internet shutdown. If you are reading this blog post, it means you have an active Internet connection, and you can connect to the different servers/services that are essential to modern life. Now, think about all of those citizens of India staying in Kashmir. Think about the problem when they have to access a website for job/medical/banking/travel or any other necessary work.

The current fascist regime of India kept shouting about “Digital India” for the last few years, and at the same time, making sure to use the Internet shutdown as a tool of oppression. By using a proper communication shutdown and blocking reporters, they made sure only the false stories from the state can be reached to the readers/viewers of news across the world. But, a few brave outside journalists and too many brave local journalists from Kashmir made sure that they kept pushing the real news from the ground. They tried their best to record atrocities.

This story in the New Yorker by Dexter Filkins should be the one for everyone to read. Take your time to read how brave Rana Ayyub and the author managed to sneak into Kashmir, and did the report.

Internet shutdowns across India

Now, if you think that the Indian government is doing this only in Kashmir, then you are totally wrong. In the last few years, India saw the highest number of Internet shutdowns across the country. Govt did not care about the reason. Given any chance, they shut down the Internet. During the current protests against the regime, they shut down the Internet in parts of Delhi, the capital of India. BBC did another story on why India gets the greatest number of Internet shutdowns.

To find all the instances of the shutdown, have a look at this site from SFLC India team.

Latest attack on students and professors of JNU

Jawaharlal Nehru University (JNU) is India’s topmost university, a place where leaders of many different fields got their education, including Nobel laureates. Yesterday evening a bunch of goons from the student wing (ABVP) of the party in power (BJP), went inside of the campus (with the full support of Delhi Police, who waited outside), and started attacking students and professors with rods and other weapons. They turned off all the street lights, but, as they forgot to shut down the Internet in the area, students managed to send across SOS messages. Search #SOSJNU on Twitter to see the amount of atrocity. Now, think for a second, what if they would have managed to shut down the Internet before the attack, just like they are doing now in Kashmir and many other parts of India. Economist and Nobel laureate Abhijit Banerjee commented how this “Echoes of Germany moving towards Nazi rule”.

Why should this matter to you, the technologist?

All of the technologies we are enjoying today, the modern world, the Internet is one of the major bounding material of the same. Think about the pain and oppression the people has to go through as this basic necessity is cut down from their lives.

Most people do not have a voice to raise for themselves. If we don’t know, then the whole country will be lost. And, we know from history what happens next.

People still count India as a democracy, actually the largest in the world. But, unless we raise up, the so-called democracy will be crushed the fascist regime in no-time.

Quick point about different mesh-network and other solutions available at Internet shutdown time

We need more documentation and examples (also translated in local languages) of the different tools available, which can help the citizens when the regime is trying their best to shut down the Internet. India is also known for random blocking of sites, and this is where free software like the Tor Project becomes so essential.