Kushal Das

FOSS and life. Kushal Das talks here.

kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion

vcrpy for web related tests

Couple of weeks ago, Jen pointed me to vcrpy. This is a Python implementation of Ruby’s library with same name.

What is vcrpy?

It is a Python module which helps to write faster and simple tests involving HTTP requests. It records all the HTTP interactions in plain text files (by default in a YAML file). This helps to write deterministic tests, and also to run them in offline.

It works well with the following Python modules.

  • requests
  • aiohttp
  • urllib3
  • tornado
  • urllib2
  • boto3

Usage example

Let us take a very simple test case.

import unittest
import requests

class TestExample(unittest.TestCase):

    def test_httpget(self):
        r = requests.get("https://httpbin.org/get?name=vcrpy&lang=Python")
        self.assertEqual(r.status_code, 200)
        data = r.json()
        self.assertEqual(data["args"]["name"], "vcrpy")
        self.assertEqual(data["args"]["lang"], "Python")


if __name__ == "__main__":
    unittest.main()

In the above code, we are making a HTTP GET request to the https://httpbin.org site and examining the returned JSON data. Running the test takes around 1.75 seconds in my computer.

$ python test_all.py
.
------------------------------------------------------------------
Ran 1 test in 1.752s

OK

Now, we can add vcrpy to this project.

import unittest
import vcr
import requests

class TestExample(unittest.TestCase):

    @vcr.use_cassette("test-httpget.yml")
    def test_httpget(self):
        r = requests.get("https://httpbin.org/get?name=vcrpy&lang=Python")
        self.assertEqual(r.status_code, 200)
        data = r.json()
        self.assertEqual(data["args"]["name"], "vcrpy")
        self.assertEqual(data["args"]["lang"], "Python")

if __name__ == "__main__":
    unittest.main()

We imported vcr module, and added a decorator vcr.use_cassette to our test function. Now, when we will execute the test again, vcrpy will record the HTTP call details in the mentioned YAML file, and use the same for the future test runs.

$ python test_all.py
.
------------------------------------------------------------------
Ran 1 test in 0.016s

OK

You all can also notice the time taken to run the test, around 0.2 second.

Read the project documentation for all the available options.

Job alert: Associate Site Reliability Engineer at FPF

We (at Freedom of the Press Foundation) are looking for an Associate Site Reliability Engineer.

This position is open to junior and entry-level applicants, and we recognize the need to provide on-the-job mentoring and support to help you familiarize yourself with the technology stack we use. In addition to the possibility of working in our New York or San Francisco offices, this position is open to remote work within American time zones.

Skills and Experience

  • Familiarity with remote systems administration of bare-metal or virtualized Linux servers.
  • Comfortable with shell and programming languages commonly used in an SRE context (e.g., Python, Go, Bash, Ruby).
  • Strong interest in honing skills required to empower a distributed software development and operations team through automation and systems maintenance.

For more details, please visit the job posting.

Are you thinking if you should apply or not?

YES, APPLY!. You are ready to apply for this position. You don’t have to ask anyone to confirm if you are ready or not. Unless you apply, you don’t have a chance to get the job.

So, the first step is to apply for the position, and then you can think about Impostor syndrome. We all have it. Some people will admit that in public, some people will not.

Using podman for containers

Podman is one of the newer tool in the container world, it can help you to run OCI containers in pods. It uses Buildah to build containers, and runc or any other OCI compliant runtime. Podman is being actively developed.

I have moved the two major bots we use for dgplug summer training (named batul and tenida) under podman and they are running well for the last few days.

Installation

I am using a Fedora 28 system, installation of podman is as simple as any other standard Fedora package.

$ sudo dnf install podman

While I was trying out podman, I found it was working perfectly in my DigitalOcean instance, but, not so much on the production vm. I was not being able to attach to the stdout.

When I tried to get help in #podman IRC channel, many responded, but none of the suggestions helped. Later, I gave access to the box to Matthew Heon, one of the developer of the tool. He identified the Indian timezone (+5:30) was too large for the timestamp buffer and thus causing this trouble.

The fix was pushed fast, and a Fedora build was also pushed to the testing repo.

Usage

To learn about different available commands, visit this page.

First step was to build the container images, it was as simple as:

$ sudo podman build -t kdas/imagename .

I reused my old Dockerfiles for the same. After this, it was just simple run commands to start the containers.

Tor Browser and Selenium

Many of us use Python Selenium to do functional testing of our websites or web applications. We generally test against Firefox and Google Chrome browser on the desktop. But, there is also a lot of people who uses Tor Browser (from Tor Project) to browse the internet and access the web applications.

In this post we will see how can we use the Tor Browser along with Selenium for our testing.

Setting up the environment

First step is to download and verify, and then extract the Tor Browser somewhere in your system. Next, download and extract geckodriver 0.17.0 somewhere in the path. For the current series of Tor Browsers, you will need this particular version of the geckodriver.

We will use pipenv to create the Python virtualenv and also to install the dependencies.

$ mkdir tortests
$ cd tortests
$ pipenv install selenium tbselenium
$ pipenv shell

The tor-browser-selenium is Python library required for Tor Browser Selenium tests.

Example code

import unittest
from time import sleep
from tbselenium.tbdriver import TorBrowserDriver


class TestSite(unittest.TestCase):
    def setUp(self):
        # Point the path to the tor-browser_en-US directory in your system
        tbpath = '/home/kdas/.local/tbb/tor-browser_en-US/'
        self.driver = TorBrowserDriver(tbpath, tbb_logfile_path='test.log')
        self.url = "https://check.torproject.org"

    def tearDown(self):
        # We want the browser to close at the end of each test.
        self.driver.close()

    def test_available(self):
        self.driver.load_url(self.url)
        # Find the element for success
        element = self.driver.find_element_by_class_name('on')
        self.assertEqual(str.strip(element.text),
                         "Congratulations. This browser is configured to use Tor.")
        sleep(2)  # So that we can see the page


if __name__ == '__main__':
    unittest.main()

In the above example, we are connecting to the https://check.torproject.org and making sure that it informs we are connected over Tor. The tbpath variable in the setUp method contains the path to the Tor Browser in my system.

You can find many other examples in the source repository.

Please make sure that you test web application against Tor Browser, having more applications which can run smoothly on top of the Tor Browser will be a great help for the community.

PyQt5 thread example

PyQt is the Python binding for Qt library. To write Qt5 code, we use PyQt5 module. Like many others, my first introduction to GUI application development was using PyQt. Back in foss.in 2005 a talk from Sirtaj introduced me to PyQt, and later fall in love with it.

I tried to help in a GUI application after 8 years (I think), a lot of things have changed in between. But, Qt/PyQt still seems to be super helpful when it comes to ease of development. Qt has one of the best documentation out there for any Open Source project.

Many students start developing GUI tools by replacing one of the command line tool they use. Generally the idea is very simple, take some input in the GUI, and then process it (using a subprocess call) on a button click, and then show the output. The subprocess call happens over a simple method, means the whole GUI gets stuck till the function call finishes. We can fix this issue by using a QThread. In the below example, we will just write a frontend for git clone command and then will do the same using QThread.

Setting up project directory

I have used qt creator to create a simple MainWindow form and saved it as mainwindow.ui in the project directory. Then, used pipenv to create a virtualenv and also installed the pyqt5 module. Next, used the pyuic5 command to create a Python file from UI file.

The code does not have error checks, the subprocess documentation should give you enough details about how to add them.

Doing git clone without any thread

The following code creates a temporary directory, and then git clones any given git repository into that.

#!/usr/bin/python3

import sys
import tempfile
import subprocess
from PyQt5 import QtWidgets

from mainwindow import Ui_MainWindow


class ExampleApp(QtWidgets.QMainWindow, Ui_MainWindow):

    def __init__(self, parent=None):
        super(ExampleApp, self).__init__(parent)
        self.setupUi(self)
        # Here we are telling to call git_clone method when
        # someone clicks on the pushButton.
        self.pushButton.clicked.connect(self.git_clone)

    # Here is the actual method which does git clone
    def git_clone(self):
        git_url = self.lineEdit.text()  # Get the git URL
        tmpdir = tempfile.mkdtemp()  # Creates a temporary directory
        cmd = "git clone {0} {1}".format(git_url, tmpdir)
        subprocess.check_output(cmd.split())  # Execute the command
        self.textEdit.setText(tmpdir)  # Show the output to the user


def main():
    app = QtWidgets.QApplication(sys.argv)
    form = ExampleApp()
    form.show()
    app.exec_()


if __name__ == '__main__':
    main()

Doing git clone with a thread

In the below example we added a new CloneThread class, it has a run method, which gets called when the thread starts. At the end of the run, we are emitting a signal to inform the main thread that the git clone operation has finished.

#!/usr/bin/python3

import sys
import tempfile
import subprocess
from PyQt5 import QtWidgets
from PyQt5.QtCore import QThread, pyqtSignal

from mainwindow import Ui_MainWindow


class CloneThread(QThread):
    signal = pyqtSignal('PyQt_PyObject')

    def __init__(self):
        QThread.__init__(self)
        self.git_url = ""

    # run method gets called when we start the thread
    def run(self):
        tmpdir = tempfile.mkdtemp()
        cmd = "git clone {0} {1}".format(self.git_url, tmpdir)
        subprocess.check_output(cmd.split())
        # git clone done, now inform the main thread with the output
        self.signal.emit(tmpdir)


class ExampleApp(QtWidgets.QMainWindow, Ui_MainWindow):

    def __init__(self, parent=None):
        super(ExampleApp, self).__init__(parent)
        self.setupUi(self)
        self.pushButton.setText("Git clone with Thread")
        # Here we are telling to call git_clone method when
        # someone clicks on the pushButton.
        self.pushButton.clicked.connect(self.git_clone)
        self.git_thread = CloneThread()  # This is the thread object
        # Connect the signal from the thread to the finished method
        self.git_thread.signal.connect(self.finished)

    def git_clone(self):
        self.git_thread.git_url = self.lineEdit.text()  # Get the git URL
        self.pushButton.setEnabled(False)  # Disables the pushButton
        self.textEdit.setText("Started git clone operation.")  # Updates the UI
        self.git_thread.start()  # Finally starts the thread

    def finished(self, result):
        self.textEdit.setText("Cloned at {0}".format(result))  # Show the output to the user
        self.pushButton.setEnabled(True)  # Enable the pushButton


def main():
    app = QtWidgets.QApplication(sys.argv)
    form = ExampleApp()
    form.show()
    app.exec_()


if __name__ == '__main__':
    main()

The example looks like the above GIF. You can find the source code here. You can find a bigger example in the journalist_gui of the SecureDrop project.

Fedora 28 template is available on QubesOS

In case you missed the news, Fedora 28 is now available as a template in Qubes OS 4.0. Fedora 26 will end of life on 2018-06-01, means this is a good time for everyone to upgrade. Use the following command in your dom0 to install the template. The template is more than 1GB in size, means it will take some time to download.

$ sudo qubes-dom0-update qubes-template-fedora-28

After installation, remember to start the template, and update, and also install all the required applications there. Next step would be to use this template everywhere.

Btw, we do have the latest Python 3.6.5 in Fedora 28 :)

$ python3
Python 3.6.5 (default, Mar 29 2018, 18:20:46)
[GCC 8.0.1 20180317 (Red Hat 8.0.1-0.19)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

SecureDrop development sprint in PyCon 2018

SecureDrop will take part in PyCon US development sprints (from 14th to 17th May). This will be first time for the SecureDrop project to present in the sprints.

If you never heard of the project before, SecureDrop is an open source whistleblower submission system that media organizations can install to securely accept documents from anonymous sources. Currently, dozens of news organizations including The Washington Post, The New York Times, The Associated Press, USA Today, and more, use SecureDrop to preserve the anonymous tipline in an era of mass surveillance. SecureDrop is installed on-premises in the news organizations, and journalists and source both use a web application to interact with the system. It was originally coded by the late Aaron Swartz and is now managed by Freedom of the Press Foundation.

How to prepare for the sprints

The source code of the project is hosted on Github.

The web applications, administration CLI tool, and a small Qt-based GUI are all written in Python. We use Ansible heavily for the orchestration. You can setup the development environment using Docker. This section of the documentation is a good place to start.

A good idea would be to create the initial Docker images for the development before the sprints. We have marked many issues for PyCon Sprints and also there are many documentation issues.

Another good place to look is the tests directorty. We use pytest for most of our test cases. We also have Selenium based functional tests.

Where to find the team?

Gitter is our primary communication platform. During the sprint days, we will in the same room of the CPython development (as I will be working on both).

So, if you are in PyCon sprints, please visit us to know more and maybe, start contributing to the project while in sprints.

Latest attempt to censor Internet and curb press freedom in India

A branch of the Indian government, the Ministry of Information and Broadcasting, is trying once again to censor Internet and Freedom of Speech. This time, it ordered to form a committee of 10 members who will frame regulations for online media/ news portals and online content.

This order includes these following Terms of Reference for the committee.

  • To delineate the sphere of online information dissemination which needs to be brought under regulation, on the lines applicable to print and electronic media.
  • To recommend appropriate policy formulation for online media / news portals and online content platforms including digital broadcasting which encompasses entertainment / infotainment and news/media aggregators keeping in mind the extant FDI norms, Programme & Advertising Code for TV Channels, norms circulated by PCI, code of ethics framed by NBA and norms prescribed by IBF; and
  • To analyze the international scenario on such existing regulatory mechanisms with a view to incorporate the best practices.

What are the immediate problems posed by this order?

If one reads carefully, one can see how vague are the terms, and specifically how they added the term online content into it.

online content means everything we can see/read/listen do over cyberspace. In the last few years, a number of new news organizations came up in India, whose fearless reporting have caused a lot of problems for the government and their friends. Even though they managed to censor publishing (sometimes self censored) news in the mainstream Indian media, but all of these new online media houses and individual bloggers and security researchers and activists kept informing the mass about the wrongdoings of the people in power.

With this latest attempt to restrict free speech over the internet, the government is trying to increase its reach even more. Broad terms like online content platforms or online media or news/media aggregators will include every person and websites under its watch. One of the impacts of mass indiscriminate surveillance like this is that people are shamed into reading and thinking only what is in line with the government, or popular thought .

How do you determine if some blog post or update in a social media platform is news or not? For me, most of things I read on the internet are news to me. I learn, I communicate my thoughts over these various platforms on cyberspace. To all those computer people reading this blog post, think about the moment when you will try to search about “how to do X in Y programming language?” on Internet, but, you can not see the result because that is blocked by this censorship.

India is also known for random blockades of different sites over the years. The Government also ordered to kill Internet for entire states for many days. For the majority of internet blockages, we, the citizens of India were neither informed the reasons nor given a chance to question the legality of those bans. India has been marked as acountry under surveillance by Reporters Without Borders back in 2012.

Also remember that this is the same Government, which was trying to fight at its best in the Supreme Court of India last year, to curb the privacy of every Indian citizen. They said that Indian citizens do not have any right to privacy. Thankfully the bench declared the following:

The right to privacy is protected as an intrinsic part of the right to life and personal liberty under Article 21 and as a part of the freedoms guaranteed by Part III of the Constitution.

Privacy is a fundamental right of every Indian citizen.

However, that fundamental right is still under attack in the name of another draconian law The Aadhaar act. A case is currently going on in the Supreme Court of India to determine the constitutional validity of Aadhaar. In the recent past, when journalists reported how the Aadhaar data can be breached, instead of fixing the problems, the government is criminally investigating the journalists.

A Declaration of the Independence of Cyberspace

Different governments across the world kept trying (and they will keep trying again and again) to curb free speech and press freedom. They are trying to draw borders and boundaries inside of cyberspace, and restrict the true nature of what is it referring to here?.

In 1996, late John Perry Barlow wrote A Declaration of the Independence of Cyberspace, and I think that fits in naturally in the current discussion.

Governments of the Industrial World, you weary giants of flesh and steel, I come from Cyberspace, the new home of Mind. On behalf of the future, I ask you of the past to leave us alone. You are not welcome among us. You have no sovereignty where we gather. -- John Perry Barlow

How can you help to fight back censorship?

Each and every one of us are affected by this, and we all can help to fight back and resist censorship. The simplest thing you can do is start talking about the problems. Discuss them with your neighbor, talk about it while commuting to the office. Explain the problem to your children or to your parents. Write about it, write blog posts, share across all the different social media platforms. Many of your friends (from other fields than computer technology) may be using Internet daily, but might not know about the destruction these laws can cause and the censorship imposed on the citizens of India.

Educate people, learn from others about the problems arising. If you are giving a talk about a FOSS technology, also talk about how a free and open Internet is helping all of us to stay connected. If that freedom goes away, we will lose everything. At any programming workshop you attend, share these knowledge with other participants.

In many cases, using tools to bypass censorship altogether is also very helpful (avoiding any direct confrontation). The Tor Project is a free software and open network which helps to keep freedom and privacy of the users. By circumventing surveillance and censorship, one can use it more for daily Internet browsing. The increase in Tor traffic will help all of the Tor network users together. This makes any attempt of tracking individuals even more expensive for any nation state actors. So, download the Tor Browser today and start using it for everything.

In this era of Public private partnership from hell, Cory Doctorow beautifully explained how internet is the nervous system of 21st century, and how we all can join together to save the freedom of internet. Listen to him, do your part.

Header image copyright: Peter Massas (CC-BY-SA)