I did not upgrade our mastodon server to 4.2.0 from 4.1.9 for a long time. Finally while doing so in the morning, I got the following error with the bundle install command.

Your bundle is locked to json-canonicalization (0.3.2) from rubygems repository
https://rubygems.org/ or installed locally, but that version can no longer be
found in that source. That means the author of json-canonicalization (0.3.2) has
removed it. You'll need to update your bundle to a version other than
json-canonicalization (0.3.2) that hasn't been removed in order to install.

I have no clue about how Ruby works, but somehow only updating the lockfile via bundle lock --update json-canonicalization did not help. Finally updated the Gemfile.lock file to have json-canonicalization (0.3.3) manually. That solved the issue and I could continue with the update steps.

The year 2022 gave me one of the happiest moments in my life, and I also felt proud as Anwesha joined the Ansible community team as a software engineer in Red Hat. Proud because she became the best example of someone to whom I taught things about computers (she has multiple mentors/friends who helped her during this journey). Though sometimes that created trouble at home, the output is super lovely. From a Masters in Law to a software engineer in Red Hat is a good story.

Also, Red Hat has a special place in our home. I left Red Hat more than 5 years ago, but still, you will notice how Red Hat (the friends there and the culture) changed my life and later the life of our family. Oh, the other special thing is that Anwesha will have her own Red Fedora now :)

On 4th June, at 19:00 UTC, we are doing an online workshop to help out new relay operators. If you ever wanted to help the Tor Project, or just curious about what is required to become a relay/bridge operator, you should join into the workshop.

The workshop is specially geared towards folks who are new to the land of Internet facing services. You will get to chat with many other operators and people from the Tor Project, and ask any doubts you have.

Register for the event, and share the news at your local groups/lists. Ask your friends to join :)

Last year I set up an ElasticSearch box to monitor a few of my servers. The goal was to learn the basics of the elastic ecosystem. I know how powerful it is but never played enough with it before.

While doing the setup, I was wondering about how to secure communication between nodes. I can not send data over plain HTTP to the nodes, and also have to make sure to have some amount of authentication. I was a bit confused about the subscriptions options.

Authenticated onion services to rescue

I use authenticated onion services in many of my regular services. It provides an easy way to connect to services (over TCP) along with encryption and authentication.

Using the same in the logstash server is an even better option for me as I do not have to open up any port in the firewall. As the logstash was listening to 5044 on localhost, I added the following configuration to the /etc/tor/torrc in the logstash server. You should use v3 addresses, and this blog post will explain how to configure that.

HiddenServiceDir /var/lib/tor/logstash
HiddenServiceVersion 2
HiddenServicePort 5044 127.0.0.1:5044
HiddenServiceAuthorizeClient stealth logstash

In the client nodes, I first had to configure Tor to reach my Onion service (details is in the blog post above). Next, I added the server address and local proxy (from Tor) details to /etc/filebeat/filebeat.yml.

output.logstash:
  # The Logstash hosts
  hosts: ["youronionaddress.onion:5044"]
  proxy_url: socks5://localhost:9050
  proxy_use_local_resolver: false
  index: "filebeat-kushaldas"

And done :) Just start the logstash server, and also the filebeat service in every node. The data will start flowing in.

If you have query about the Tor Project, you can visit our new https://community.torproject.org/ site.

Just like v2 Onion services, we can also set up client authorization for Onion services v3. In simple terms, when you have a client authorization setup on an Onion service, only the Tor clients with the private token can access the service. Using this, you can run services (without opening up any port in your system) and only selected people can access that service, that is also being inside of totally encrypted Tor network. Last month, I did a workshop in Rootconf about the same topic, but, I demoed v2 Onion services. In this blog post, I am going to show you how you can do the same with the latest v3 services.

Setting up the Onion service

We assume that we are already running nginx or apache on port 80 of the server. Add the following two lines at the end of the /etc/tor/torrc file of your server.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 127.0.0.1:80

Then, restart the tor service.

systemctl restart tor

The above command will create the onion service at /var/lib/tor/hidden_service/ directory, and we can see the address from the hostname file.

cat /var/lib/tor/hidden_service/hostname 
cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion

It should also create a authorized_clients directory at the service directory.

Next, we will create keys of type x25519, and you can either use any of the following options to create the keys.

• Bash
• Rust
• Python

I used the Rust implementation, and I got the secret and the public key.

secret: "TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q"
public: "RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ"

Now, we will use the public key to create a clientname.auth file in /var/lib/tor/hidden_service/authorized_clients/ directory, I chose the name kushal.auth.

descriptor:x25519:RO7N45JLVI5UXOLALOK4V22JLMMF5ZDC2W6DXVKIAU3C7FNIVROQ > /var/lib/tor/hidden_service/authorized_clients/kushal.auth

If you look closely, the file format is like below:

descriptor:x25519:public_key

Now, restart the tor service once again in the server.

systemctl restart tor

Setting up client authorization

The first step is to close down my Tor Browser as I will be manually editing the torrc file of the same. Then, I added the following line to the same file tor-browser_en-US/Browser/TorBrowser/Data/Tor/torrc.

ClientOnionAuthDir TorBrowser/Data/Tor/onion_auth

Next, we will create the directory.

mkdir tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth
chmod 0700 tor-browser_en-US/Browser/TorBrowser/Data/Tor/onion_auth

Then, add the following in kushal.auth_private file inside of the onion_auth directory.

cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid:descriptor:x25519:TIICFSKY2PECECM2LOA7XLKQKJWHYTN4WLRSIIJKQFCCL3K2II2Q

The format of the file:

onion_address_56_chars:descriptor:x25519:private_key

Now, start the Tor Browser, and you should be able to visit the authorized Onion service at cz2eqjwrned6s7zy3nrmkk3fjoudzhvu53ynq6gdny5efdj26zxf4bid.onion.

Use case for students

If you want to demo your web project to a selected group of people, but, don't want to spend money to get a web server or VPS, Onion services is a great way to showcase your work to the world. With the authenticated services, you can choose whom all can view the site/service you are running.

Back in 2005 I joined my first job, in a software company in Bangalore. It was a backend of a big foreign bank. We trained heavily on different parts of software development during the first few months. At the same time, I had an altercation with the senior manager (about some Java code) who was in charge of the new joinees and their placement within the company. The result? Everyone else got a team but me, and I had to roam around within the office to find an empty seat and wait there till the actual seat owner came back. I managed to spend a lot of days in the cafeteria on the rooftop. But, then they made new rules that one can not sit there either, other than at lunch time.

So, I went asking around, talking to all the different people in the office (there were 500+ folks iirc) if they know any team who would take on a fresher. I tried to throw in words like Linux, open source to better my chances. And then one day, I heard that the research and development team was looking for someone with Linux and PHP skills. I went in to have a chat with the team, and they told me the problem (it was actually on DSpace, a Java based documentation/content repository system), and after looking at my resume decided to give me a desktop for couple of weeks. I managed to solve the problem in next few days, and after a week or so, I was told that I will join the team. There were couple of super senior managers and I was the only kid on that block. Being part of this team allowed me to explore different technologies and programming languages.

I will later write down my experiences in more detail, but for today, I want to focus on one particular incident. The kind of incident, which all system administrators experience at least once in their life (I guess). I got root access to the production server of the DSpace installation within a few weeks. I had a Windows desktop, and used putty to ssh in to the server. As this company was backend of the big bank, except for a few senior managers, no one else had access to Internet on their systems. There were 2 desktops in the kiosk in the ground floor, and one had to stand in a long queue to get a chance to access Internet.

One day I came back from the lunch (a good one), and was feeling a bit sleepy. I had taken down the tomcat server, pushed the changes to the application, and then wanted to start the server up again. Typed the whole path to startup.sh (I don’t remember the actual name, I’m just guessing it was startup.sh) and hit Enter. I was waiting for the long screens of messages this startup script spewed as it started up, but instead, I got back the prompt quickly. I was wondering what went wrong. Then, looking at the monitor very closely, I suddenly realised that I was planning to delete some other file and I had written rm at the beginning of the command prompt, forgotten it, and then typed the path of the startup.sh. Suddenly I felt the place get very hot and stuffy; I started sweating and all blood drained from my face in the next few moments. I was at panic level 9. I was wondering what to do. I thought about the next steps to follow. I still had a small window of time to fix the service. Suddenly I realized that I can get a copy of the script from the Internet (yay, Open Source!). So, I picked up a pad and a pen, ran down to the ground floor, and stood in the queue to get access to a computer with Internet. After getting the seat, I started writing down the whole startup.sh on the pad and double checked it. Ran right back up to my cubicle, feverishly typed in the script, (somehow miraculously without any typo in one go.) As I executed the script, I saw the familiar output, messages scrolling up, screen after joyful screen. And finally as it started up, I sighed a huge sigh of relief. And after the adrenalin levels came down, I wrote an incident report to my management, and later talked about it during a meeting.

From that day on, before doing any kind of destructive operation, I double check the command prompt for any typo. I make sure, that I don’t remove anything randomly and also make sure that I have my backups is place.

You can set up a Tor onion service in a VM on your home desktop, or on a Raspberry Pi attached to your home network. You can serve any website, or ssh service using the same. For example, in India most of the time if an engineering student has to demo a web application, she has to demo on her laptop or on a college lab machine. If you set up your web application project as an onion service, you can actually make it available to all of your friends. You don’t need an external IP or special kind of Internet connection or pay for a domain name. Of course, it may be slower than all the fancy website you have, but you don’t have to spend any extra money for this.

In this post, I am going to talk about how can you set up your own service using a Fedora 26 VM. The similar steps can be taken in Raspberry Pi or any other Linux distribution.

Install the required packages

I will be using Nginx as my web server. The first step is to get the required packages installed.

$ sudo dnf install nginx tor
Fedora 26 - x86_64 - Updates                     10 MB/s |  20 MB     00:01
google-chrome                                    17 kB/s | 3.7 kB     00:00
Qubes OS Repository for VM (updates)             98 kB/s |  48 kB     00:00
Last metadata expiration check: 0:00:00 ago on Wed Jan 17 08:30:23 2018.
Dependencies resolved.
================================================================================
 Package                Arch         Version                Repository     Size
================================================================================
Installing:
 nginx                  x86_64       1:1.12.1-1.fc26        updates       535 k
 tor                    x86_64       0.3.1.9-1.fc26         updates       2.6 M
Installing dependencies:
 gperftools-libs        x86_64       2.6.1-5.fc26           updates       281 k
 nginx-filesystem       noarch       1:1.12.1-1.fc26        updates        20 k
 nginx-mimetypes        noarch       2.1.48-1.fc26          fedora         26 k
 torsocks               x86_64       2.1.0-4.fc26           fedora         64 k

Transaction Summary
================================================================================
Install  6 Packages

Total download size: 3.6 M
Installed size: 15 M
Is this ok [y/N]:

Configuring Nginx

After installing the packages, the next step is to setup the web server. For a quick example, we will just show the default Nginx index page over this web service. Please read about Nginx to know more about how to configure Nginx with your web application.

Here we have the web server running on port 80 by default.

Configuring Tor

Next, we will set up the Tor onion service. The configuration file is located at /etc/tor/torrc. We will add the following two lines.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 80 127.0.0.1:80

We are redirecting port 80 in the onion service to the port 80 in the same system.

Starting the services

Remember to open up port 80 in the firewall before starting the services. I am going to keep it an exercise for the reader to find out how :)

We will start nginx and tor service as the next step, you can also watch the system logs to find out status of Tor.

$ sudo systemctl start nginx
$ sudo systemctl start tor
$ sudo journalctl -f -u tor
-- Logs begin at Thu 2017-12-07 07:13:58 IST. --
Jan 17 08:33:43 tortest Tor[2734]: Bootstrapped 0%: Starting
Jan 17 08:33:43 tortest Tor[2734]: Signaled readiness to systemd
Jan 17 08:33:43 tortest systemd[1]: Started Anonymizing overlay network for TCP.
Jan 17 08:33:43 tortest Tor[2734]: Starting with guard context "default"
Jan 17 08:33:43 tortest Tor[2734]: Opening Control listener on /run/tor/control
Jan 17 08:33:43 tortest Tor[2734]: Bootstrapped 5%: Connecting to directory server
Jan 17 08:33:44 tortest Tor[2734]: Bootstrapped 10%: Finishing handshake with directory server
Jan 17 08:33:44 tortest Tor[2734]: Bootstrapped 15%: Establishing an encrypted directory connection
Jan 17 08:33:45 tortest Tor[2734]: Bootstrapped 20%: Asking for networkstatus consensus
Jan 17 08:33:45 tortest Tor[2734]: Bootstrapped 25%: Loading networkstatus consensus
Jan 17 08:33:55 tortest Tor[2734]: I learned some more directory information, but not enough to build a circuit: We have no usable consensus.
Jan 17 08:33:55 tortest Tor[2734]: Bootstrapped 40%: Loading authority key certs
Jan 17 08:33:55 tortest Tor[2734]: Bootstrapped 45%: Asking for relay descriptors
Jan 17 08:33:55 tortest Tor[2734]: I learned some more directory information, but not enough to build a circuit: We need more microdescriptors: we have 0/6009, and can only build 0% of likely paths. (We have 0% of guards bw, 0% of midpoint bw, and 0% of exit bw = 0% of path bw.)
Jan 17 08:33:56 tortest Tor[2734]: Bootstrapped 50%: Loading relay descriptors
Jan 17 08:33:57 tortest Tor[2734]: Bootstrapped 56%: Loading relay descriptors
Jan 17 08:33:59 tortest Tor[2734]: Bootstrapped 65%: Loading relay descriptors
Jan 17 08:34:06 tortest Tor[2734]: Bootstrapped 72%: Loading relay descriptors
Jan 17 08:34:06 tortest Tor[2734]: Bootstrapped 80%: Connecting to the Tor network
Jan 17 08:34:07 tortest Tor[2734]: Bootstrapped 85%: Finishing handshake with first hop
Jan 17 08:34:07 tortest Tor[2734]: Bootstrapped 90%: Establishing a Tor circuit
Jan 17 08:34:08 tortest Tor[2734]: Tor has successfully opened a circuit. Looks like client functionality is working.
Jan 17 08:34:08 tortest Tor[2734]: Bootstrapped 100%: Done

There will be a private key and the hostname file for the onion service in the /var/lib/tor/hidden_service/ directory. Open up Tor browser, and visit the onion address. You should be able to see a page like below screenshot.

Remember to backup the private key file if you want to keep using the same onion address for a longer time.

What all things can we do with this onion service?

That actually depends on your imagination. Feel free to research about what all different services can be provided over Tor. You can start with writing a small Python Flask web application, and create an onion service for the same. Share the address with your friends.

Ask your friends to use Tor browser for daily web browsing. The more Tor traffic we can generate, the more difficult it will become for the nation-state actors to try to monitor traffics, that in turn will help the whole community.

WARNING on security and anonymous service

Remember that this tutorial is only for quick demo purpose. This will not make your web server details or IP or operating system details hidden. You will have to make sure of following proper operational security practices along with system administration skills. Riseup has a page describing best practices. But, please make sure that you do enough study and research before you start providing long-term services over the Tor.

Also please remember that Tor is developed and run by people all over the world and the project needs donation. Every little bit of help counts.

Choosing a new password is always an interesting topic. When I started using computers for the first time, my idea was to find some useful words which I can remember, maybe 2-3 of those words together. With time I found that the websites have different requirements when it comes to choosing a new password. But, in the last few years we also saw many examples where brute forcing a password is a rather simple thing. The modern powerful computers enable anyone to find a right combination of characters in a decent time frame.

What is a diceware password?

Diceware passwords are normal passwords (a few words together) generated from a list of words by either rolling a dice, or by computer. You can read more in the original Diceware website.

Using diceware project to generate your passphrases

If you notice, I have written passphrase instead of password. This is because passphrases are not only easier to remember than a complex password, but they also provide better security from bruteforce attacks. The following comics from XKCD explains it better than any words.

Installing diceware

diceware is a very simple command line tool written in Python. It can help you to choose a diceware passphrase easily. It was already packaged for Debian, last week I have packaged it for Fedora (thank you Parag for the review). Yesterday night it was pushed to stable. So, now you can install it using dnf.

$ sudo dnf install diceware

Using diceware

$ diceware 
MotorBolsterFountainThrowerPorridgeBattered

By default it is creating passphrases with 6 words in it, but you can increase by using -n command line argument. You should use at least 7 words in your passphrase. Read the story from Micah Lee to understand how this helps to increase the strength of your passphrases in many folds.

The man page of the diceware has more details about usage.

Start using a password manager

Now is a good time to start using a password manager. Save all the passwords/passphrases in one place, and secure it with a super long passphrase which you can remember. This article from Martin Shelton has many examples. The members of Fedora engineering team uses a command line tool called pass which uses gpg to encrypt the passwords.

Rootconf is the largest DevOps, and Cloud infrastructure related conference in India. This year's event happened on 14-15th April in the MLR convention center, Bangalore. I traveled on the day one of the event from Pune. Woke up by 3AM in the morning, and then took the first flight to Bangalore. Picked up Ramky on my way to the venue. Managed to skip most of the (in)famous Bangalore traffic thanks to a govt holiday.

I carried the Fedora standee, and the table cloth from Pune, we set up the Fedora table in the conference. Meanwhile the place was getting full with attendees. We found so many ex-foss.in friends. Premshree, Pankaj, Raj, Vasundhar and many others I met after years. There were 300+ attendees in the event.

I should also mention about the Cat5 cable lanyard.

The day started with Zainab welcoming everyone is super fast manner :) The first talk of the day was " Happiness through Crash-Only software" by Antoine Grondin. The whole idea that failure is part of life, and we will make our software even more complex by trying to avoid is something we all should keep in mind. He gave some ideas about how Digital Ocean works in the back. Instead of a normal start/end he explained how having recovering from failed state as start, and considering that the process will fail at the end can help. Seeing hand drawn slides was another exciting thing for me, I was doing that quite a lot in my previous talks.

The next talk was from Raj Shekhar, who gave an overview of Mesos. Even in the limited time, his talk was a very right fit while still describing the use case. This was the first time I listened to a talk about Mesos, and it was a very good one.

After this during tea break we moved out the booth/table. We had Ramky, Lalatendu, Aditya, /me, and rtnpro in the table talking to the attendees. The Fedora DVD(s), and badges went out like hot cakes :) Through out two days we had many questions related to Project Atomic, the great looking stickers actually helped to get more attention. In between, I went through the details of my talk to Raj, he provided some valuable input.

Food is always great in hasgeek events, but having so many food startups from Bangalore in one place seriously added new flavor to the conference. hasgeek is not only growing itself, but making sure that they have a great community all across the conference, from talks to food, everything.

Rest of the day 1 we spent talking to people in, and around booth area. hasgeek gets some great recording done of their talks, so we can view them in future :) In the evening we had the dinner at the venue, once again too tasty food :) I went to rtnpro's house by 10pm as I still wanted to work on my slides.

Day 2

(Photo by Lalatendu)

Woke up by 6:30AM, and went through my slides once again. The title of my talk was "Failure at Cloud and rescued by Python". The agenda of the talk was to encourage DevOps/Sys-admins to write Python scripts than shell scripts. It was the first talk of the day. I hope it went well, the slides are available here. As I mentioned Ansible in my talk, we had many Ansible questions asked in the Fedora booth.

Next was Premshree, who talked about "Continuous deployment at Scale" with the detailed example of how they handle things at Etsy. Glad to see that they use a IRC bot to do the deployment :)

After lunch I attended "Working in and with Open Source Communities" from Bernd Erk. This talk was full with tips about communities, how to treat everyone as member. One major point was about a community leader who should become the balance between the people who talk the most, and the silent ones of the community. He also emphasized about getting new members in the community as "Because you will die someday" :)

Rest of the day we again spent in and around of the Fedora booth. Met Tarun Dua after a long time. I came back in a late night flight to Pune. I will post the links to the talk videos when I get it.