Kushal Das

FOSS and life. Kushal Das talks here.

kushal76uaid62oup5774umh654scnu5dwzh4u2534qxhcbi4wbab3ad.onion

Using Rust to access Internet over Tor via SOCKS proxy 🦀

Tor provides a SOCKS proxy so that you can have any application using the same to connect the Onion network. The default port is 9050. The Tor Browser also provides the same service on port 9150. In this post, we will see how can we use the same SOCKS proxy to access the Internet using Rust.

You can read my previous post to do the same using Python.

Using reqwest and tokio-socks crates

I am using reqwest and tokio-socks crates in this example.

The Cargo.toml file.

[package]
name = "usetor"
version = "0.1.0"
authors = ["Kushal Das <mail@kushaldas.in>"]
edition = "2018"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]
tokio = { version = "0.2", features = ["macros"] }
reqwest = { version = "0.10.4", features = ["socks", "json"] }
serde_json = "1.0.53"

The source code:

use reqwest;
use tokio;
use serde_json;

#[tokio::main]
async fn main() -> Result<(), reqwest::Error> {
    let proxy = reqwest::Proxy::all("socks5://127.0.0.1:9050").unwrap();
    let client = reqwest::Client::builder()
        .proxy(proxy)
        .build().unwrap();

    let res = client.get("https://httpbin.org/get").send().await?;
    println!("Status: {}", res.status());

    let text: serde_json::Value = res.json().await?;
    println!("{:#?}", text);

    Ok(())
}

Here we are converting the response data into JSON using serde_json. The output looks like this.

✦ ❯ cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.06s
     Running `target/debug/usetor`
Status: 200 OK
Object({
    "args": Object({}),
    "headers": Object({
        "Accept": String(
            "*/*",
        ),
        "Host": String(
            "httpbin.org",
        ),
        "X-Amzn-Trace-Id": String(
            "Root=1-5ecc9e03-09dc572e6db5357f28eecf47",
        ),
    }),
    "origin": String(
        "51.255.45.144",
    ),
    "url": String(
        "https://httpbin.org/get",
    ),
})

Instead of any normal domain, you can also connect to any .onion domain via the same proxy.

Securing your Elastic services using authenticated onion services

Last year I set up an ElasticSearch box to monitor a few of my servers. The goal was to learn the basics of the elastic ecosystem. I know how powerful it is but never played enough with it before.

While doing the setup, I was wondering about how to secure communication between nodes. I can not send data over plain HTTP to the nodes, and also have to make sure to have some amount of authentication. I was a bit confused about the subscriptions options.

Authenticated onion services to rescue

I use authenticated onion services in many of my regular services. It provides an easy way to connect to services (over TCP) along with encryption and authentication.

Using the same in the logstash server is an even better option for me as I do not have to open up any port in the firewall. As the logstash was listening to 5044 on localhost, I added the following configuration to the /etc/tor/torrc in the logstash server. You should use v3 addresses, and this blog post will explain how to configure that.

HiddenServiceDir /var/lib/tor/logstash
HiddenServiceVersion 2
HiddenServicePort 5044 127.0.0.1:5044
HiddenServiceAuthorizeClient stealth logstash

In the client nodes, I first had to configure Tor to reach my Onion service (details is in the blog post above). Next, I added the server address and local proxy (from Tor) details to /etc/filebeat/filebeat.yml.

output.logstash:
  # The Logstash hosts
  hosts: ["youronionaddress.onion:5044"]
  proxy_url: socks5://localhost:9050
  proxy_use_local_resolver: false
  index: "filebeat-kushaldas"

And done :) Just start the logstash server, and also the filebeat service in every node. The data will start flowing in.

If you have query about the Tor Project, you can visit our new https://community.torproject.org/ site.

Tor rpm package repository for Fedora and CentOS/RHEL

Now we have official Tor RPM repositories for Fedora, CentOS/RHEL. The support documentation is already in place.

Using this repository, you can get the latest Tor build for your distribution from the upstream project itself. Tor already provides similar packages for Debian/Ubuntu systems.

How to enable the repository in your Fedora box?

Add the following to the /etc/yum.repos.d/tor.repo.

[tor]
name=Tor for Fedora $releasever - $basearch
baseurl=https://rpm.torproject.org/fedora/$releasever/$basearch
enabled=1
gpgcheck=1
gpgkey=https://rpm.torproject.org/fedora/public_gpg.key
cost=100

Then you can install the package via regular dnf command.

$ sudo dnf install tor

You will have to import the new keys used for signing these packages.

Importing GPG key 0x3621CD35:
Userid : "Kushal Das (RPM Signing key) <kushal@torproject.org>"
Fingerprint: 999E C8E3 14BC 8D46 022D 6C7D E217 C30C 3621 CD35
From : https://rpm.torproject.org/fedora/public_gpg.key
Is this ok [y/N]: y

If you run a Tor relay (which you all should, one of the easiest ways to contribute to the project and help people worldwide) on CentOS/RHEL, you can use similar repository configuration.

5 months of Internet shutdown in Kashmir and more fascist attacks in India

From 5th August 2019, Kashmir is under a communication shutdown. SMS service for a particular connection provider is now available for postpaid users, but Internet is still down for all Indian citizens of Kashmir.

This is above 155 days of Internet shutdown. If you are reading this blog post, it means you have an active Internet connection, and you can connect to the different servers/services that are essential to modern life. Now, think about all of those citizens of India staying in Kashmir. Think about the problem when they have to access a website for job/medical/banking/travel or any other necessary work.

The current fascist regime of India kept shouting about “Digital India” for the last few years, and at the same time, making sure to use the Internet shutdown as a tool of oppression. By using a proper communication shutdown and blocking reporters, they made sure only the false stories from the state can be reached to the readers/viewers of news across the world. But, a few brave outside journalists and too many brave local journalists from Kashmir made sure that they kept pushing the real news from the ground. They tried their best to record atrocities.

This story in the New Yorker by Dexter Filkins should be the one for everyone to read. Take your time to read how brave Rana Ayyub and the author managed to sneak into Kashmir, and did the report.

Internet shutdowns across India

Now, if you think that the Indian government is doing this only in Kashmir, then you are totally wrong. In the last few years, India saw the highest number of Internet shutdowns across the country. Govt did not care about the reason. Given any chance, they shut down the Internet. During the current protests against the regime, they shut down the Internet in parts of Delhi, the capital of India. BBC did another story on why India gets the greatest number of Internet shutdowns.

To find all the instances of the shutdown, have a look at this site from SFLC India team.

Latest attack on students and professors of JNU

Jawaharlal Nehru University (JNU) is India’s topmost university, a place where leaders of many different fields got their education, including Nobel laureates. Yesterday evening a bunch of goons from the student wing (ABVP) of the party in power (BJP), went inside of the campus (with the full support of Delhi Police, who waited outside), and started attacking students and professors with rods and other weapons. They turned off all the street lights, but, as they forgot to shut down the Internet in the area, students managed to send across SOS messages. Search #SOSJNU on Twitter to see the amount of atrocity. Now, think for a second, what if they would have managed to shut down the Internet before the attack, just like they are doing now in Kashmir and many other parts of India. Economist and Nobel laureate Abhijit Banerjee commented how this “Echoes of Germany moving towards Nazi rule”.

Why should this matter to you, the technologist?

All of the technologies we are enjoying today, the modern world, the Internet is one of the major bounding material of the same. Think about the pain and oppression the people has to go through as this basic necessity is cut down from their lives.

Most people do not have a voice to raise for themselves. If we don’t know, then the whole country will be lost. And, we know from history what happens next.

People still count India as a democracy, actually the largest in the world. But, unless we raise up, the so-called democracy will be crushed the fascist regime in no-time.

Quick point about different mesh-network and other solutions available at Internet shutdown time

We need more documentation and examples (also translated in local languages) of the different tools available, which can help the citizens when the regime is trying their best to shut down the Internet. India is also known for random blocking of sites, and this is where free software like the Tor Project becomes so essential.

Python function to generate Tor v3 onion service authentication keys

Here is a small Python function using the amazing Python Cryptography module to generate the Tor v3 Onion service authentication services.

from cryptography.hazmat.primitives import serialization
from cryptography.hazmat.primitives.asymmetric import x25519
import base64

def generate_tor_v3_keys():
    "Generates public, private keypair"
    private_key = x25519.X25519PrivateKey.generate()
    private_bytes = private_key.private_bytes(
        encoding=serialization.Encoding.Raw	,
        format=serialization.PrivateFormat.Raw,
        encryption_algorithm=serialization.NoEncryption())
    public_key = private_key.public_key()
    public_bytes = public_key.public_bytes(
        encoding=serialization.Encoding.Raw,
        format=serialization.PublicFormat.Raw)
    public = base64.b32encode(public_bytes).replace(b'=', b'') \
                       .decode("utf-8")
    private = base64.b32encode(private_bytes).replace(b'=', b'') \
                        .decode("utf-8")
    return public, private

You can follow my previous blog post to setup an authenticated Onion service.

Permanent Record: the life of Edward Snowden

book cover

The personal life and thinking of the ordinary person who did an extraordinary thing.

A fantastic personal narrative of his life and thinking process. The book does not get into technical details, but, it will make sure that people relate to the different events mentioned in the book. It tells the story of a person who is born into the system and grew up to become part of the system, and then learns to question the same system.

I bought the book at midnight on Kindle (I also ordered the physical copies), slept for 3 hours in between and finished it off in the morning. Anyone born in 80s will find so many similarities as an 80s kid. Let it be the Commodore 64 as the first computer we saw or basic as the first-ever programming language to try. The lucky ones also got Internet access and learned to roam around of their own and build their adventure along with the busy telephone lines (which many times made the family members unhappy).

If you are someone from the technology community, I don't think you will find Ed's life was not as much different than yours. It has a different scenario and different key players, but, you will be able to match the progress in life like many other tech workers like ourselves.

Maybe you are reading the book just to learn what happened, or maybe you want to know why. But, I hope this book will help to think about the decisions you make in your life and how that affects the rest of the world. Let it be a group picture posted on Facebook or writing the next new tool for the intelligence community.

Go ahead and read the book, and when you are finished, make sure you pass it across to your friend, or buy them new copies. If you have some free time, you may consider to run a Tor relay or a bridge, a simple step will help many around the world.

On a side note, the book mentions SecureDrop project at the very end, and today is also the release of SecureDrop 1.0.0 (the same day of the book release).

Announcing lymworkbook project

In 2017, I started working on a new book to teach Linux command line in our online summer training. The goal was to have the basics covered in the book, and the same time not to try to explain things which can be learned better via man pages (yes, we encourage people to read man pages).

Where to practice

This one question always came up, many times, the students managed to destroy their systems by doing random things. rm -rf is always one of the various commands in this regard.

Introducing lymworkbook

Now, the book has a new chapter, LYM Workbook, where the reader can set up VMs in the local machine via Vagrant, and go through a series of problems in those machines. One can then verify if the solution they worked on is correct or not. For example:

sudo lymsetup copypaste
sudo lymverify copypaste

We are starting with only a few problems, but I (and a group of volunteers) will slowly add many more problems. We will also increase the complexity by increasing the number of machines and having setup more difficult systems. This will include the basic system administration related tasks.

How can you help

Have a look at the issues, feel free to pick up any open issue or create issues with various problems which you think are good to learn. Things can be as easy as rsync a directory to another system, or setting up Tor Project and use it as a system proxy.

Just adding one problem as an issue is also a big help, so please spend 5 minutes of your free time, and add any problem you like.

Setting up authorized v3 Onion services

Just like v2 Onion services, we can also set up client authorization for Onion services v3. In simple terms, when you have a client authorization setup on an Onion service, only the Tor clients with the private token can access the service. Using this, you can run services (without opening up any port in your system) and only selected people can access that service, that is also being inside of totally encrypted Tor network. Last month, I did a workshop in Rootconf about the same topic, but, I demoed v2 Onion services. In this blog post, I am going to show you how you can do the same with the latest v3 services.

Setting up the Onion service

We assume that we are already running nginx or apache on port 80 of the server. Add the following two lines at the end of the /etc/tor/torrc file of your server.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 127.0.0.1:80

Then, restart the tor service.

systemctl restart tor

The above command will create the onion service at /var/lib/tor/hidden_service/ directory, and we can see the address from the hostname file.

cat /var/lib/tor/hidden_service/hostname 
cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion

It should also create a authorized_clients directory at the service directory.

Next, we will create keys of type x25519, and you can either use any of the following options to create the keys.

I used the Rust implementation, and I got the secret and the public key.

secret: "TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q"
public: "RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ"

Now, we will use the public key to create a clientname.auth file in /var/lib/tor/hidden_service/authorized_clients/ directory, I chose the name kushal.auth.

descriptor:x25519:RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ > /var/lib/tor/hidden_service/authorized_clients/kushal.auth

If you look closely, the file format is like below:

descriptor:x25519:public_key

Now, restart the tor service once again in the server.

systemctl restart tor

Setting up client authorization

The first step is to close down my Tor Browser as I will be manually editing the torrc file of the same. Then, I added the following line to the same file tor-browser_en-US/Browser/TorBrowser/Data/Tor/torrc.

ClientOnionAuthDir TorBrowser/Data/Tor/onion_auth

Next, we will create the directory.

mkdir tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth
chmod 0700 tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth

Then, add the following in kushal.auth_private file inside of the onion_auth directory.

cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid:descriptor:x25519:TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q

The format of the file:

onion_address_56_chars:descriptor:x25519:private_key

Now, start the Tor Browser, and you should be able to visit the authorized Onion service at cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion.

Use case for students

If you want to demo your web project to a selected group of people, but, don't want to spend money to get a web server or VPS, Onion services is a great way to showcase your work to the world. With the authenticated services, you can choose whom all can view the site/service you are running.

Indian news websites over Tor and HTTP vs HTTPS

Following the idea of Secure The News, I wanted to verify if the Indian news websites have proper certificates or can they be viewed over Tor network.

The major problem was to get the list of urls, and I managed to create that from the Wikipedia list of Indian news organizations. Next, I had to write a straightforward Python script to verify the sites over Tor.

I have 181 site URLs and out of those, 5 are down. Among the rest 176 sites, surprisingly all but 3 sites could not be open from Tor network. The following 3 sites are blocking the users from Tor, thus, compromising the privacy and security Tor Browser provides to their readers.

  • http://www.dnaindia.com/
  • http://www.financialexpress.com/
  • http://www.jagran.com/

On the other hand, when it comes to enabling HTTPS by default and redirecting people to that, most of these sites failed miserably.

117 sites do not provide their sites over HTTPS. If you do not know why it is important to provide the same, please read this. If you don’t know how to enable HTTPS on your website, you can read this guide.

This is the list of the URLs who are only on HTTP.

Tor Browser and Selenium

Many of us use Python Selenium to do functional testing of our websites or web applications. We generally test against Firefox and Google Chrome browser on the desktop. But, there is also a lot of people who uses Tor Browser (from Tor Project) to browse the internet and access the web applications.

In this post we will see how can we use the Tor Browser along with Selenium for our testing.

Setting up the environment

First step is to download and verify, and then extract the Tor Browser somewhere in your system. Next, download and extract geckodriver 0.17.0 somewhere in the path. For the current series of Tor Browsers, you will need this particular version of the geckodriver.

We will use pipenv to create the Python virtualenv and also to install the dependencies.

$ mkdir tortests
$ cd tortests
$ pipenv install selenium tbselenium
$ pipenv shell

The tor-browser-selenium is Python library required for Tor Browser Selenium tests.

Example code

import unittest
from time import sleep
from tbselenium.tbdriver import TorBrowserDriver


class TestSite(unittest.TestCase):
    def setUp(self):
        # Point the path to the tor-browser_en-US directory in your system
        tbpath = '/home/kdas/.local/tbb/tor-browser_en-US/'
        self.driver = TorBrowserDriver(tbpath, tbb_logfile_path='test.log')
        self.url = "https://check.torproject.org"

    def tearDown(self):
        # We want the browser to close at the end of each test.
        self.driver.close()

    def test_available(self):
        self.driver.load_url(self.url)
        # Find the element for success
        element = self.driver.find_element_by_class_name('on')
        self.assertEqual(str.strip(element.text),
                         "Congratulations. This browser is configured to use Tor.")
        sleep(2)  # So that we can see the page


if __name__ == '__main__':
    unittest.main()

In the above example, we are connecting to the https://check.torproject.org and making sure that it informs we are connected over Tor. The tbpath variable in the setUp method contains the path to the Tor Browser in my system.

You can find many other examples in the source repository.

Please make sure that you test web application against Tor Browser, having more applications which can run smoothly on top of the Tor Browser will be a great help for the community.