Kushal Das

FOSS and life. Kushal Das talks here.


New importer design for darkserver

Previously darkserver project had a koji plugin, which used to collect the information from the koji-hubs itself. It used to process each rpm, find ELF files and extract GNU BUILD-ID from them.

But the major disadvantage on this method we found was package build time increase. For bigger packages like kernel it used to take more than one hour extra. Koji does not allow to do direct async-plugins, so I had to look for a complete separate import process.

The following image explains the new process

We have a darkproducer process running, which is checking koji for any new build (using API(s) provided by koji), if it finds a build is on, then it creates an entry in the Buildqueue, if the build is completed then it creates an entery in the Jobqueue.

darkbuildworker process is monitoring Buildqueue, and if it finds any completed job then it moves it to the Jobqueue or else puts it back in the Buildqueue.

Then finally we can have many darkjobworker processes, they can monitor the Jobqueue and imports any jobs landing in the queue. All of the processes work as daemons.

Broke the previous monolithic design into simple producer-consumer one. Using Redis to maintain the queues. I am also running a dev instance to to test the design

The codebase which contains queue creation and task management, is being moved into a separate python module to create/use redis based queues. I hope it will be useful for someone else trying to solve similar kind of problem.

FUDCon KL day 2 and day 3

Day 2 started with a talk from Joshua Wulf about the Fedora book written by the attendees. The idea seems to easy enough for the newbies to contribute but still they will require a bit knowledge of docbook I think.

Rest of the first half was spent discussing package Lisp modules in Fedora with Christoph Wickert. Now I am going to setup a repo with all the packages so that people can play around with them.

During lunch time I met another fellow Red Hatter, Luc de Louw. Lunch table discussions went on about various other FUDCons and funny incidents in them. The lighting talks session went really well. People submitted various ideas and started demoing various projects. While CWickert was showing LXDE, I decided to try it out and still now the performance looks amazing, though there are few small things needs to fixed.

The cloud talk by CWickert was hilarious. After the day finished we slowly moved to the hotel and FUDPub was supposed to start from 8pm in the same hotel. The food was nice and cake was delicious.

The number of attendees in the day 3 was smaller compared to the last two days. I had my talk on Darkserver in the morning. We had some good discussion about the future roadmap of the project during the talk. You can find the slides here. There was another good talk on various create applications available in FOSS world. In the second half of the day we decided to put up the talks and workshops for voting as number of people was less. I went ahead to do the python workshop as planned. Abu Mansur Manaf gave the ending keynote which should boost the newbies to become contributors.

Spent the evening in the hotel room discussing various functional programming languages and features of the languages and darkserver roadmap with hircus. Later in the night went out for dinner with the rest of the event crew members to a local food joint. Stayed up for the rest of the night as we had catch an early morning flight back to India.

Met many old friends and made many new ones, long discussions on project ideas, great food, sleep for only few hours everyday, this event was full on. The last FUDCon in India and this one, a large portion of the audience are total newbies and I hope they got good ideas on how to start contribute. My personal goals were to find new contributors for darkserver project and discuss the packaging ideas with CWickert and both went well.

Day 1 of FUDCON KL 2012

Getting up early in the morning, eating breakfast while sleeping is the most common thing of every event's day 1. It was exactly the same, but we managed to start from the hotel to the FUDCon venue on time. The registration desk was already opened. We slowly moved into Audi 3 for the keynote by Christoph Wickert

After the keynote people put up the talks they wanted to do in the barcamp and they started talking from 2:30pm (after lunch).

In between I met Joshua Wulf for the first time. 
While discussing about lekhonee-gnome with Ankur, we figured out that the latest build failed in F17 and I was trying to fix it. Suddenly someone from the left of my seat asked me to use pkgconfig in the spec file to find the gee-1.0 dependency and voila, it worked like magic. Before I could ask his name, he said very gently to me that he is the maintainer of vala and gee :)

i think this is the main reason people should come over to an event like FUDCon, meeting people in real and learn from them is always useful.

I did a small session on "Fedora for students" and tried to explain the advantages students can get while working on FOSS projects and Fedora in particular. I also attended Christoph's talk on "Improving Collaboration with other Open Source projects" and we discussed how things can go ahead for a project like darkserver

2 days of Fedora and python workshops in Durgapur

With help from local dgplug members we had 2 days of workshops on February 17th and 18th.

The idea was to introduce Fedora to the new students and various tools which they can use to develop their skills. This time we tried to get only interested students into the workshops.

Day1: We started around 11am in the seminar hall, had around 45 students attending the first talk on FOSS in general and then about Fedora. Existing contributors like Sayan, Biraj also talked about their experience, how they started their journey. After lunch break we started python workshop, we had to introduce the terminal and various commands to them before I started talking about python. Most of them were being able to catch up and solve the small problems I gave. We closed for the day around 5:30pm. I stayed back in the Boy's Hostel and had chats about various projects students are working on.

Day2: Though we started a bit early but we had almost every participant from previous day. First workshop of the day was on Vi. In the second half we again started with python but going into more details. Showing them how they can start working on a project, how to do their lab assignments using existing tools in Fedora. These students can see their seniors who already started contributing to various projects. I hope some of them will follow the path.


FUDCon Pune day 2

Group shot in FUDCon Pune 2011

Day 2 started in a slow motion, everyone was kind of sleepy in the hotel. Harish started the day with his talk on community. The students started coming in during his talk, I guess 9am was a bit too early for most of them.

Interested students moved in the talks they wanted to attend. Siddesh had all his talks almost filled up by the students, He gave 3 different talks one after another and people really loved how easily he went into details of gdb. Ankur gave 4 talks in the day :)

I attended Jared's talk on docbook and publican , he used my book as an example in the talk. in 2008 he only convinced me to use publican for the book and did lot of ground work and fixing for me.

The FUDPub in the evening was a crazy party , people kept dancing all over until the music was stopped. Photos and videos will be up later next week.

Right now Siddesh is working autotoolizing the glib based qpid library I am writing, /me and Jared are working on the pym.

Mukti 11.2 at NIT Durgapur

We reached Durgapur around 3:30AM on Friday, woke up by 6AM due to the really cold temperature. The inauguration event started few hours late and after that Mether started doing his RPM workshop.

My workshop started after 6:30PM , around 60+ students. Started with basics of python as a language. I did this workshop slowly so that everyone can pick the things we were discussing. 

Next day morning my second session started from 10AM and in the afternoon I had my PyGtk workshop in the lab. After finishing talking for more than 6 hours did the final talk for the night with the first year students. Total time I spoke was more than 7 hours , which is a record for me :)

3rd day again started at 10AM and I was surprised to see more than 30 students in the seminar hall where I was doing the python workshops. We discussed in details various modules in python and also discussed about projects for the students.

Generally I don't get many mails from the students after the workshops or even if they do mail , they don't continue. But here I found  3 students still continuing. I hope at least one of them will be able his/her journey in the path of freedom.  

Photos can be found here. Will update the post with flickr set when I upload there.

UPDATE: flickr set can be found here.

Report from PyCon India 2010

So I am back from PyCon India 2010. I missed last year, so was waiting for this year's event. Met many faces after very long time and to be in a place with so many other python lovers is always a nice experience. The total attendance was around 700 but the venue was too big for that number , so except the lunch time, corridors had lesser number of people discussing. The selection of the talks were also matching the environment as they came from different directions. We saw talks with hardware accessibility to web development to GUI application toolkits, network programming, scientific computing, terminal based works etc. 

Day 1 started a bit late, David Goodger gave the keynote, it was fun. Next talk I liked was from Dhananjay Nene on "Functional Programming with Python", he managed to show many things within the short time. 

I stood in the queue for lunch as soon as possible, it was late, too much crowd at the same time, but the taste of the food was delicious.  During and after the lunch I was discussing with Jace about Wing IDE, I had a thought of buying that previously also but the price is high for me. Wait, not anymore, they gave a 90% discount for the PyCon India :) Started using it right away with vi mode and it works like a charm.

After lunch, thought of going in Mahendra M's Twisted programming talk, but the room was full, no place for standing also. So hoped over to the auditorium for Vijay Kumar B.'s Device Interfacing with Python and ZIO , it was one of the best talk I ever attended.

Met DP and Jacob after long time :) 

This was first time my laptop's X crashed while trying to use the projector, so after few tries I started my talk with the black terminal on the big screen :) newt widgets looked great on that screen though :)

Day 2: I came a bit late as I had to meet few seniors from my college in the city (event was in a far corner of the city), I spent the day mostly in the corridors talking with people. I met so many Pune based python lovers there :) Strangely enough few speakers were absconding from the talks, so the organizers had to innovate to fill in.  

At the very end we had first AGM of Indian Python Software Society, too much fun in that :p. I applied for membership, I still need two votes. I also resumed my work on "Python for you and me" 2nd edition. You can also read ml-IN or zh-CN translation of the book.

Any python lover from the country should attend this event, next year it will be in Chennai. Photos will come later this week.

8051 development on Fedora

First install sdcc #yum install sdcc

Next is a small example to blink a LED connected to P1.7

#include <8051.h> void main(void) { int i=1; P1=0; /* P1.7 used as LED output */ while(1) { P1_7=!(P1_7); /* toggle P1.7*/

for(i=1;i<25000;i++); /* for delay */

} }

Compile it and get the hex file from intel hex $sdcc-sdcc blink.c $ sdcc-packihx blink.ihx > blink.hex

Now you can upload the hex to the programmer :D /me is also just starting up.

More on thumbnailing and optimization

Following my last post, spent most of the of the time on different imaging libraries to find a faster way of doing thumbnails and optimization.

I tested the following libraries to create image thumbnails, GdkPixbuf, imlib2, ImageMagic, epeg. Pixbuf gives somewhat nice timings , imlib2 is fast but was leaking too much memory. ImageMagic seems to the slowest among them. Last try was with epeg which can only handle jpegs and it came out as the fastest. So wrote a C function and using it from inside vala code using extern.

Next target was to find better way to get thumbnails from RAW images, tried libopenraw and LibRaw for that. But with help from yorba developers found the way to do it using gexiv2 only.

In between tried few tools for profiling the application, sayamindu told me about sysprof which seems to be the easiest for my purpose. Using it I found gexiv2_metadata_open_path is taking around 67% of time, inside it Exiv2::TiffImage:readMetaData is taking 51% of time.

Now coming to the point of speed , 1st run is on 1GB of RAW files

real	0m2.946s
user	0m2.542s
sys	0m0.116s

2nd run is on same 36GB of images , among them around half is RAW.

real	4m0.807s
user	0m54.283s
sys	1m24.789s

Now this is fast in my textbook :D I should not forget to tell about the great help I got from #vala and Adrien Bustany in the whole work.