Previously darkserver project had a koji plugin, which used to collect the information from the koji-hubs itself. It used to process each rpm, find ELF files and extract GNU BUILD-ID from them.

But the major disadvantage on this method we found was package build time increase. For bigger packages like kernel it used to take more than one hour extra. Koji does not allow to do direct async-plugins, so I had to look for a complete separate import process.

The following image explains the new process

We have a darkproducer process running, which is checking koji for any new build (using API(s) provided by koji), if it finds a build is on, then it creates an entry in the Buildqueue, if the build is completed then it creates an entery in the Jobqueue.

darkbuildworker process is monitoring Buildqueue, and if it finds any completed job then it moves it to the Jobqueue or else puts it back in the Buildqueue.

Then finally we can have many darkjobworker processes, they can monitor the Jobqueue and imports any jobs landing in the queue. All of the processes work as daemons.

Broke the previous monolithic design into simple producer-consumer one. Using Redis to maintain the queues. I am also running a dev instance to to test the design

The codebase which contains queue creation and task management, is being moved into a separate python module to create/use redis based queues. I hope it will be useful for someone else trying to solve similar kind of problem.

You can view the full set here.

Day 2 started with a talk from Joshua Wulf about the Fedora book written by the attendees. The idea seems to easy enough for the newbies to contribute but still they will require a bit knowledge of docbook I think.

Rest of the first half was spent discussing package Lisp modules in Fedora with Christoph Wickert. Now I am going to setup a repo with all the packages so that people can play around with them.

During lunch time I met another fellow Red Hatter, Luc de Louw. Lunch table discussions went on about various other FUDCons and funny incidents in them. The lighting talks session went really well. People submitted various ideas and started demoing various projects. While CWickert was showing LXDE, I decided to try it out and still now the performance looks amazing, though there are few small things needs to fixed.

The cloud talk by CWickert was hilarious. After the day finished we slowly moved to the hotel and FUDPub was supposed to start from 8pm in the same hotel. The food was nice and cake was delicious.

The number of attendees in the day 3 was smaller compared to the last two days. I had my talk on Darkserver in the morning. We had some good discussion about the future roadmap of the project during the talk. You can find the slides here. There was another good talk on various create applications available in FOSS world. In the second half of the day we decided to put up the talks and workshops for voting as number of people was less. I went ahead to do the python workshop as planned. Abu Mansur Manaf gave the ending keynote which should boost the newbies to become contributors.

Spent the evening in the hotel room discussing various functional programming languages and features of the languages and darkserver roadmap with hircus. Later in the night went out for dinner with the rest of the event crew members to a local food joint. Stayed up for the rest of the night as we had catch an early morning flight back to India.

Met many old friends and made many new ones, long discussions on project ideas, great food, sleep for only few hours everyday, this event was full on. The last FUDCon in India and this one, a large portion of the audience are total newbies and I hope they got good ideas on how to start contribute. My personal goals were to find new contributors for darkserver project and discuss the packaging ideas with CWickert and both went well.

Getting up early in the morning, eating breakfast while sleeping is the most common thing of every event's day 1. It was exactly the same, but we managed to start from the hotel to the FUDCon venue on time. The registration desk was already opened. We slowly moved into Audi 3 for the keynote by Christoph Wickert. 

With help from local dgplug members we had 2 days of workshops on February 17th and 18th.

The idea was to introduce Fedora to the new students and various tools which they can use to develop their skills. This time we tried to get only interested students into the workshops.

Day1: We started around 11am in the seminar hall, had around 45 students attending the first talk on FOSS in general and then about Fedora. Existing contributors like Sayan, Biraj also talked about their experience, how they started their journey. After lunch break we started python workshop, we had to introduce the terminal and various commands to them before I started talking about python. Most of them were being able to catch up and solve the small problems I gave. We closed for the day around 5:30pm. I stayed back in the Boy's Hostel and had chats about various projects students are working on.

Day2: Though we started a bit early but we had almost every participant from previous day. First workshop of the day was on Vi. In the second half we again started with python but going into more details. Showing them how they can start working on a project, how to do their lab assignments using existing tools in Fedora. These students can see their seniors who already started contributing to various projects. I hope some of them will follow the path.

Day 2 started in a slow motion, everyone was kind of sleepy in the hotel. Harish started the day with his talk on community. The students started coming in during his talk, I guess 9am was a bit too early for most of them.

Interested students moved in the talks they wanted to attend. Siddesh had all his talks almost filled up by the students, He gave 3 different talks one after another and people really loved how easily he went into details of gdb. Ankur gave 4 talks in the day :)

I attended Jared's talk on docbook and publican , he used my book as an example in the talk. in 2008 he only convinced me to use publican for the book and did lot of ground work and fixing for me.

The FUDPub in the evening was a crazy party , people kept dancing all over until the music was stopped. Photos and videos will be up later next week.

Right now Siddesh is working autotoolizing the glib based qpid library I am writing, /me and Jared are working on the pym.

We reached Durgapur around 3:30AM on Friday, woke up by 6AM due to the really cold temperature. The inauguration event started few hours late and after that Mether started doing his RPM workshop.

UPDATE: flickr set can be found here.

So I am back from PyCon India 2010. I missed last year, so was waiting for this year's event. Met many faces after very long time and to be in a place with so many other python lovers is always a nice experience. The total attendance was around 700 but the venue was too big for that number , so except the lunch time, corridors had lesser number of people discussing. The selection of the talks were also matching the environment as they came from different directions. We saw talks with hardware accessibility to web development to GUI application toolkits, network programming, scientific computing, terminal based works etc. 

First install sdcc #yum install sdcc

Next is a small example to blink a LED connected to P1.7

for(i=1;i<25000;i++); /* for delay */

} }

Compile it and get the hex file from intel hex $sdcc-sdcc blink.c $ sdcc-packihx blink.ihx > blink.hex

Now you can upload the hex to the programmer :D /me is also just starting up.

Following my last post, spent most of the of the time on different imaging libraries to find a faster way of doing thumbnails and optimization.

I tested the following libraries to create image thumbnails, GdkPixbuf, imlib2, ImageMagic, epeg. Pixbuf gives somewhat nice timings , imlib2 is fast but was leaking too much memory. ImageMagic seems to the slowest among them. Last try was with epeg which can only handle jpegs and it came out as the fastest. So wrote a C function and using it from inside vala code using extern.

Next target was to find better way to get thumbnails from RAW images, tried libopenraw and LibRaw for that. But with help from yorba developers found the way to do it using gexiv2 only.

In between tried few tools for profiling the application, sayamindu told me about sysprof which seems to be the easiest for my purpose. Using it I found gexiv2_metadata_open_path is taking around 67% of time, inside it Exiv2::TiffImage:readMetaData is taking 51% of time.

Now coming to the point of speed , 1st run is on 1GB of RAW files

real	0m2.946s
user	0m2.542s
sys	0m0.116s

2nd run is on same 36GB of images , among them around half is RAW.

real	4m0.807s
user	0m54.283s
sys	1m24.789s

Now this is fast in my textbook :D I should not forget to tell about the great help I got from #vala and Adrien Bustany in the whole work.