Kushal Das

FOSS and life. Kushal Das talks here.


Introducing pyage-rust, a Python module for age encryption

age is a simple, modern and secure file encryption tool, it was designed by @Benjojo12 and @FiloSottile.

An alternative interoperable Rust implementation is available at github.com/str4d/rage

pyage-rust is a Python module for age, this is built on top of the Rust crate. I am not a cryptographer, and I prefer to keep this important thing to the specialists :)

pyage-rust demo


I have prebuilt wheels for both Linux and Mac, Python 3.7 and Python 3.8.

python3 -m pip install pyage-rust==0.1.0

Please have a look at the API documentation to learn how to use the API.

Building pyage-rust

You will need the nightly rust toolchain (via https://rustup.rs).

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install requirements-dev.txt
maturin develop && maturin build

Missing items in the current implementation

I am yet to add ways to use ssh keys or alias features, but those will come slowly in the future releases.

Another try at a new Python module for OpenPGP aka johnnycanencrypt

Using OpenPGP from Python is a pain. There are various documentation/notes on the Internet explaining why, including the famous one from isis agora lovecraft where they explained why they changed the module name to pretty_bad_protocol.

sequoia-pgp is a Rust project to do OpenPGP from scratch in Rust, and as library first approach. You can see the status page to see how much work is already done.

Using this and Pyo3 project I started writing an experimental Python module for OpenPGP called Johnny Can Encrypt.

I just did an release of 0.1.0. Here is some example code.

>>> import johnnycanencrypt as jce
>>> j = jce.Johnny("secret.asc")
>>> data = j.encrypt_bytes("kushal 🐍".encode("utf-8"))
>>> print(data)


>>> result = j.decrypt_bytes(data.encode("utf-8"), "mysecretpassword")
>>> print(result.decode("utf-8"))
kushal 🐍

The readme of the project page has build instruction, and more details about available API calls. We can create new keypair (RSA4096). We can encrypt/decrypt bytes and files. We can also sign/verify bytes/files. The code does not have much checks for error handling, this is super early stage.

You will need nettle (on Fedora) and libnettle on Debian (and related development packages) to build it successfully.

I published wheels for Debian Buster (Python3.7), and Fedora 32 (Python3.8).

Issues in providing better wheels for pip install

The wheels are linked against system provided nettle library. And every distribution has a different version. Means even if I am building a python3.7 wheel on Debian, it will not work on Fedora. I wish to find a better solution to this in the coming days.

As I said earlier in this post, this is just starting of this project. It will take time to mature for production use. And because of Sequoia, we will have better defaults of cipher/hash options.

Using Rust to access Internet over Tor via SOCKS proxy πŸ¦€

Tor provides a SOCKS proxy so that you can have any application using the same to connect the Onion network. The default port is 9050. The Tor Browser also provides the same service on port 9150. In this post, we will see how can we use the same SOCKS proxy to access the Internet using Rust.

You can read my previous post to do the same using Python.

Using reqwest and tokio-socks crates

I am using reqwest and tokio-socks crates in this example.

The Cargo.toml file.

name = "usetor"
version = "0.1.0"
authors = ["Kushal Das <mail@kushaldas.in>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

tokio = { version = "0.2", features = ["macros"] }
reqwest = { version = "0.10.4", features = ["socks", "json"] }
serde_json = "1.0.53"

The source code:

use reqwest;
use tokio;
use serde_json;

async fn main() -> Result<(), reqwest::Error> {
    let proxy = reqwest::Proxy::all("socks5://").unwrap();
    let client = reqwest::Client::builder()

    let res = client.get("https://httpbin.org/get").send().await?;
    println!("Status: {}", res.status());

    let text: serde_json::Value = res.json().await?;
    println!("{:#?}", text);


Here we are converting the response data into JSON using serde_json. The output looks like this.

✦ ❯ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/usetor`
Status: 200 OK
    "args": Object({}),
    "headers": Object({
        "Accept": String(
        "Host": String(
        "X-Amzn-Trace-Id": String(
    "origin": String(
    "url": String(

Instead of any normal domain, you can also connect to any .onion domain via the same proxy.

A few new generation command line tools

Many of us live on the terminal. We use tools which allows us to do things faster and let us stay productive. Most of these tools are old. Sometimes we do pick up a few new generation command line tools. Here is a small list of tools I am using daily. All of these are written in Rust .


ripgrep screenshot

ripgrep was the first Rust tool I started using daily as a replacement for grep. It is easy to use. The output looks nice, and also works with my vim.


exa is the replacement for ls. It includes many useful flags.

exa demo


bat is the one stop replacement for cat and less. It also provides syntax highlighting with nice colours. I do have an alias cat=/usr/bin/bat -p.

bat demo


zoxide allows to move around directories super fast.

zoxide demo


starship is the shell prompt you can see in all of the GIFs above. It allows a lot of customization.

All of these tools are packaged in Fedora 32 by the amazing fedora-rust SIG.

Writing Python module in Rust using PyO3

Back in 2017, I wrote about how to create Python modules using Rust. Things changed in between, especially PyO3 project now makes it super easy to create such extension modules.


I am using Python 3.7.3 and using the latest nightly build of Rust using rustup tool. Remember to use the nightly toolchain than stable.

rustup default nightly

The example source

use pyo3::prelude::*;
use pyo3::types::PyDict;
use pyo3::wrap_pyfunction;
use std::collections::HashMap;

/// Formats the sum of two numbers as string.
fn sum_as_string(a: usize, b: usize) -> PyResult<String> {
    Ok((a + b).to_string())

/// Formats the sum of two numbers as string.
fn get_result() -> PyResult<HashMap<String, String>> {
    let mut result = HashMap::new();
    result.insert("name".to_string(), "kushal".to_string());
    result.insert("age".to_string(), "36".to_string());

// Returns a Person class, takes a dict with {"name": "age", "age": 100} format.
fn give_me_a_person(data: &PyDict) -> PyResult<Person> {
    let name: String = data.get_item("name").unwrap().extract().unwrap();
    let age: i64 = data.get_item("age").unwrap().extract().unwrap();

    let p: Person = Person::new(name, age);

struct Person {
    #[pyo3(get, set)]
    name: String,
    #[pyo3(get, set)]
    age: i64,

impl Person {
    fn new(name: String, age: i64) -> Self {
        Person { name, age }

/// A Python module implemented in Rust.
fn myfriendrust(_py: Python, m: &PyModule) -> PyResult<()> {

In this example, we are creating one class Person, and 3 different function. You can checkout the whole source from this git repo.

get_result function returns a HashMap object from Rust. Where as give_me_person takes a Python dictionary as argument and then creates a Person class from it and then returns it to Python.

One thing to notice that we are using __new__ to create an instance of the Person class. You can read about the class creation in the documentation.

Building the module

$ cargo build --release
<omitting output>
Compiling myfriendrust v0.1.0 (/home/kdas/code/rust/myfriendrust)
Finished release [optimized] target(s) in 2.60s

$ ls target/release/
build deps examples incremental libmyfriendrust.d libmyfriendrust.so

This will create a libmodulename.so file, in this case the name of the file is libmyfriendrust.so. I have a helper shell script try.sh to rename the file to myfriendrust.so so that I can test it.

cd target/release
mv libmyfriendrust.so myfriendrust.so

Now, let us try it out.

To build a wheel for distribution

You can use maturin tool to create a wheel for distribution.

$ maturin build
πŸ”— Found pyo3 bindings
🐍 Found CPython 3.7m at python3.7
Compiling proc-macro2 v1.0.10
Compiling unicode-xid v0.2.0
Compiling syn v1.0.18
Compiling serde v1.0.106
Compiling ryu v1.0.4
Compiling proc-macro-hack v0.5.15
Compiling libc v0.2.69
Compiling regex-syntax v0.6.17
Compiling autocfg v1.0.0
Compiling itoa v0.4.5
Compiling scopeguard v1.1.0
Compiling smallvec v1.4.0
Compiling cfg-if v0.1.10
Compiling unindent v0.1.5
Compiling inventory v0.1.6
Compiling version_check v0.9.1
Compiling num-traits v0.2.11
Compiling lock_api v0.3.4
Compiling quote v1.0.3
Compiling regex v1.3.7
Compiling parking_lot_core v0.7.2
Compiling parking_lot v0.10.2
Compiling pyo3-derive-backend v0.9.2
Compiling serde_derive v1.0.106
Compiling ghost v0.1.1
Compiling ctor v0.1.14
Compiling paste-impl v0.1.11
Compiling inventory-impl v0.1.6
Compiling indoc-impl v0.3.5
Compiling paste v0.1.11
Compiling pyo3cls v0.9.2
Compiling indoc v0.3.5
Compiling serde_json v1.0.51
Compiling pyo3 v0.9.2
Compiling myfriendrust v0.1.0 (/home/kdas/code/rust/myfriendrust)
Finished dev [unoptimized + debuginfo] target(s) in 56.57s
πŸ“¦ Built wheel for CPython 3.7m to /home/kdas/code/rust/myfriendrust/target/wheels/myfriendrust-0.1.0-cp37-cp37m-manylinux1_x86_64.whl

This post is just an introduction. You can look into the documentation and start writing more complex code. If you say I write bad Rust code, then yes. I am still a very beginner in Rust.

mod_wsgi and a Python extention

I was working on a performance analysis of a web API. After I identified the possible issues, I tried to see if I can use a native extension for that part of code (it is a Flask application). In this case the extension was written in Rust. It worked very well. In both test environment and using mod_wsgi-express it was super fast. But it failed when I tried to use it in the production under nginx + mod_wsgi combination. The import modulename statement was just stuck. Causing a timeout in the application. There were no other error messages or log lines.

Found the cause only after having a chat with Graham. After listening to the problem, he told me the solution in seconds. To set the Application Group as %{GLOBAL}.

WSGIApplicationGroup %{GLOBAL}

This is to make sure that we are using the first (main) interpreter, instead of any of the sub-intepreters. The reason explained in the mod_wsgi documentation

The consequences of attempting to use a C extension module for Python which is implemented against the simplified API for GIL state management in any sub interpreter besides the first, is that the code is likely to deadlock or crash the process.

A new tool to render my blog


Back in 2013, I wrote Shonku in golang. It helped me in two important things:

  • Having a static blogging tool which works (worked for the last 5 years).
  • To help me to learn basics on golang.

Now, I think it worked for me. I could focus on writing the actual content of the posts than anything else. The tool has a few flaws, but, none of them had any issue with my blogging requirements. It just worked for me. I could have written it in Python (in much less time), but, learning a new language is always fun.

new new

As I am trying to write more and more Rust, I decided to write a new tool in Rust and use that for my blog https://kushaldas.in.

This is very initial code, and you can easily figure out that I still don’t know how to write more idiomatic Rust yet. However, this works. The last couple of the posts were made using this tool, and I also regenerated the whole site in between.

The cargo build --release command takes time, at the same time the release binary is insanely fast.


Setting up authorized v3 Onion services

Just like v2 Onion services, we can also set up client authorization for Onion services v3. In simple terms, when you have a client authorization setup on an Onion service, only the Tor clients with the private token can access the service. Using this, you can run services (without opening up any port in your system) and only selected people can access that service, that is also being inside of totally encrypted Tor network. Last month, I did a workshop in Rootconf about the same topic, but, I demoed v2 Onion services. In this blog post, I am going to show you how you can do the same with the latest v3 services.

Setting up the Onion service

We assume that we are already running nginx or apache on port 80 of the server. Add the following two lines at the end of the /etc/tor/torrc file of your server.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80

Then, restart the tor service.

systemctl restart tor

The above command will create the onion service at /var/lib/tor/hidden_service/ directory, and we can see the address from the hostname file.

cat /var/lib/tor/hidden_service/hostname 

It should also create a authorized_clients directory at the service directory.

Next, we will create keys of type x25519, and you can either use any of the following options to create the keys.

I used the Rust implementation, and I got the secret and the public key.


Now, we will use the public key to create a clientname.auth file in /var/lib/tor/hidden_service/authorized_clients/ directory, I chose the name kushal.auth.

descriptor:x25519:RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ > /var/lib/tor/hidden_service/authorized_clients/kushal.auth

If you look closely, the file format is like below:


Now, restart the tor service once again in the server.

systemctl restart tor

Setting up client authorization

The first step is to close down my Tor Browser as I will be manually editing the torrc file of the same. Then, I added the following line to the same file tor-browser_en-US/Browser/TorBrowser/Data/Tor/torrc.

ClientOnionAuthDir TorBrowser/Data/Tor/onion_auth

Next, we will create the directory.

mkdir tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth
chmod 0700 tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth

Then, add the following in kushal.auth_private file inside of the onion_auth directory.


The format of the file:


Now, start the Tor Browser, and you should be able to visit the authorized Onion service at cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion.

Use case for students

If you want to demo your web project to a selected group of people, but, don't want to spend money to get a web server or VPS, Onion services is a great way to showcase your work to the world. With the authenticated services, you can choose whom all can view the site/service you are running.

Writing Python Extensions in Rust

In December I spent few days with Rust. I wrote few lines of code and was trying to get in touch with the syntax and feeling of the language. One of the major things in my TODO list was figuring out how to write Python extensions in Rust. Armin Ronacher wrote this excellent post in the Sentry blog back in October, 2016. I decided to learn from the same code base. It is always much easier to make small changes and then see what actually change due the same. This is also my first usage of CFFI module. Before this, I always wrote Python C extensions from scratch. In this post I will assume that we already have a working Rust installation on your system, and then we will go ahead from that.

Creating the initial Rust project

I am already in my new project directory, which is empty.

$ cargo init
Created library project
$ ls
Cargo.toml src

Now, I am going to update the Cargo.toml file with the following content. Feel free to adjust based on your requirements.

name = "liblearn"
version = "0.1.0"
authors = ["Kushal Das <mail@kushaldas.in>"]

name = "liblearn"
crate-type = ["cdylib"]

Using the crate-type attribute we tell the Rust compiler what kind of artifact to generate. We will create a dynamic system library for our example. On my Linux computer it will create a *.so file. You can read more about the crate-types here.

Next we update our src/lib.rs file. Here we are telling that we also have a src/ksum.rs file.

mod tests {
    fn it_works() {

pub mod ksum;
use std::ffi::CStr;
use std::os::raw::{c_uint, c_char};

pub unsafe extern "C" fn sum(a: c_uint, b: c_uint) -> c_uint {
	println!("{}, {}", a, b);
	a + b

pub unsafe extern "C" fn onbytes(bytes: *const c_char) {
	let b = CStr::from_ptr(bytes);
	println!("{}", b.to_str().unwrap())

We have various types which can help us to handle the data coming from the C code. We also have two unsafe functions, the first is sum, where we are accepting two integers, and returning the addition of those values. We are also printing the integers just for our learning purpose.

We also have a onbytes function, in which we will take a Python bytes input, and just print it on the STDOUT. Remember this is just an example, so feel free to make changes and learn more :). The CStr::from_ptr function helps us with converting raw C string to a safe C string wrapper in Rust. Read the documentation for the same to know more.

All of the functions also have no_mangle attribute, so that Rust compiler does not mangle the names. This helps in using the functions in C code. Marking the functions extern will help in line of Rust FFI work. At this moment you should be able to build the Rust project with cargo build command.

Writing the Python code

Next we create a build.py file on the top directory, this will help us with CFFI. We will also need our C header file with proper definitions in it, include/liblearn.h


unsigned int sum(unsigned int a, unsigned int b);
void onbytes(const char *bytes);

The build.py

import sys
import subprocess
from cffi import FFI

def _to_source(x):
    if sys.version_info >= (3, 0) and isinstance(x, bytes):
        x = x.decode('utf-8')
    return x

ffi = FFI()
'cc', '-E', 'include/liblearn.h'],
ffi.set_source('liblearn._sumnative', None)

Feel free to consult the CFFI documentation to learn things in depth. If you want to convert Rust Strings to Python and return them, I would suggest you to have a look at the unpack function.

The actual Python module source

We have liblearn/init.py file, which holds the actual code for the Python extension module we are writing.

import os
from ._sumnative import ffi as _ffi

_lib = _ffi.dlopen(os.path.join(os.path.dirname(__file__), '_liblearn.so'))

def sum(a, b):
    return _lib.sum(a,b)

def onbytes(word):
    return _lib.onbytes(word)

setup.py file

I am copy pasting the whole setup.py below. Most of it is self explanatory. I also kept the original comments which explain various points.

import os
import sys
import shutil
import subprocess

    from wheel.bdist_wheel import bdist_wheel
except ImportError:
    bdist_wheel = None

from setuptools import setup, find_packages
from distutils.command.build_py import build_py
from distutils.command.build_ext import build_ext
from setuptools.dist import Distribution

# Build with clang if not otherwise specified.
if os.environ.get('LIBLEARN_MANYLINUX') == '1':
    os.environ.setdefault('CC', 'gcc')
    os.environ.setdefault('CXX', 'g++')
    os.environ.setdefault('CC', 'clang')
    os.environ.setdefault('CXX', 'clang++')

PACKAGE = 'liblearn'
EXT_EXT = sys.platform == 'darwin' and '.dylib' or '.so'

def build_liblearn(base_path):
    lib_path = os.path.join(base_path, '_liblearn.so')
    here = os.path.abspath(os.path.dirname(__file__))
    cmdline = ['cargo', 'build', '--release']
    if not sys.stdout.isatty():
    rv = subprocess.Popen(cmdline, cwd=here).wait()
    if rv != 0:
    src_path = os.path.join(here, 'target', 'release',
                            'libliblearn' + EXT_EXT)
    if os.path.isfile(src_path):
        shutil.copy2(src_path, lib_path)

class CustomBuildPy(build_py):
    def run(self):
        build_liblearn(os.path.join(self.build_lib, *PACKAGE.split('.')))

class CustomBuildExt(build_ext):
    def run(self):
        if self.inplace:
            build_py = self.get_finalized_command('build_py')

class BinaryDistribution(Distribution):
    """This is necessary because otherwise the wheel does not know that
    we have non pure information.
    def has_ext_modules(foo):
        return True

cmdclass = {
    'build_ext': CustomBuildExt,
    'build_py': CustomBuildPy,

# The wheel generated carries a python unicode ABI tag.  We want to remove
# this since our wheel is actually universal as far as this goes since we
# never actually link against libpython.  Since there does not appear to
# be an API to do that, we just patch the internal function that wheel uses.
if bdist_wheel is not None:
    class CustomBdistWheel(bdist_wheel):
        def get_tag(self):
            rv = bdist_wheel.get_tag(self)
            return ('py2.py3', 'none') + rv[2:]
    cmdclass['bdist_wheel'] = CustomBdistWheel

    description='Module to learn writing Python extensions in rust',
    author='Kushal Das',
        'Intended Audience :: Developers',
        'License :: OSI Approved :: BSD License',
        'Operating System :: OS Independent',
        'Programming Language :: Python',
        'Topic :: Software Development :: Libraries :: Python Modules'

Building the Python extension

$ python3 setup.py build
running build
running build_py
creating build/lib
creating build/lib/liblearn
copying liblearn/__init__.py -> build/lib/liblearn
Finished release [optimized] target(s) in 0.0 secs
generating cffi module 'build/lib/liblearn/_sumnative.py'
running build_ext

Now we have a build directory. We go inside of the build/lib directory, and try out the following.

$ python3
Python 3.5.2 (default, Sep 14 2016, 11:28:32)
[GCC 6.2.1 20160901 (Red Hat 6.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblearn
>>> liblearn.sum(12,30)
12, 30
>>> b = "Kushal in bengali কুূঢল".encode("utf-8")
>>> liblearn.onbytes(b)
Kushal in bengali কুূঢল

This post is only about how to start writing a new extension. My knowledge with Rust is very minimal. In future I will write more as I learn. You can find all the source files in github repo.

Thank you Siddhesh, and Armin for reviewing this post.

Learning Rust

Other than fighting with dust allergies, Rust took some of my time in the past few weeks. I kept it in the to learn list for a long time, and finally thought of looking at it due to a blog post by Armin. I do follow his personal blog, and my previous notions about the language came from his blog posts. But in this blog post on official Sentry site, Armin talked about how they fixed a Python performance issue with Rust (rather than writing in C).


I started with reading The book, which is also the official rust-lang documentation. One may also want to look at the screencasts at intro_rust(). The videos explained the idea behind ownership, and borrowing in a very simple manner. There is also a github project containing various resources related to Rust. I jumped into the Programming Rust book, which is in Early Release state. This book has great in-depth explanations of various features of the language. But always remember the #rust-beginners channel in IRC, there are people who can help you to understand things.

I try to learn any new thing by practicing. Learning a new programming language is not different. I prefer to write small, but usable code, which in turn can solve one of my issue. This is something I had trouble to keep doing in Rust, as most of the book is about explaining a lot of things in details, but not enough examples of code doing things. The nearest thing I found is Rust by Example. Maybe we can add more examples of code which does something useful for the user, or it is just in my mind.

First working code

One of the missing tool for me is an email address search tool for mutt. There are various scripts to search mail addresses from official LDAP server, or sometimes I used goobook for searching mail addresses from my gmail account. I wanted to have a command, which can find an address from any of my mail mail accounts. I have downloaded the google contacts in a CSV file. I also wrote another Python script using regex to find different mail address from my maildirs. But it was slow to print out from all the thousands of mails I have. So, I just tried to have the same in Rust, it was of course faster. Now I have all the unique mail addresses (a few with names too) in a plain text file.

My search tool just reads the whole file every time, and finds the matches. It prints one address in each line, that way I could just plug it in my mutt configuration.

set query_command="searchemailid %s"

Now if you look at the source code, you will find it is a mixture of copy-paste from various examples + a lot of lets change this, and see what happens next. I also thank Icefoz to help me out over IRC.

To end the post, you can install Rust in Fedora 25 using DNF.

# dnf install rust -y