Speed , Vala, Sqlite3 and optimization

Wednesday, July 7th, 2010 | FOSS, vala

All started as just another stupid idea, writing an image indexer (reinventing the wheel ).  The code should do the following things:

  • Index images for any given folder with user provided tags
  • Extract and keep EXIF information from the files
  • Extract or generate image thumbnails for all files
  • Should be able to provide the thumbnail even if the original file is missing (may be in a usb hdd)
Wrote some test code to see how much time it takes to do the above for a folder with 36GB+ in size and 5044 images (both JPEG and NEF ) 

1st run 

real 30m39.178s
user 0m45.561s
sys 3m42.557s

2nd run with out thumbnail generation but with EXIF information

real 10m14.281s
user 0m58.208s
sys 0m45.969s

3rd run with out thumbnail generation and with out EXIF information

real 14m40.503s
user 0m1.585s
sys 0m7.535s

Here I am confused, managed to find out that transaction in sqlite can cause delay, so changed the code to do everything in single transaction
4th run with out thumbnail generation and with out EXIF information

real 0m1.032s
user 0m0.216s
sys 0m0.134s

5th run with out thumbnail generation but with EXIF information

real 3m1.191s
user 1m2.525s
sys 0m34.524s

6th run with everything 

real 16m47.241s
user 0m43.652s
sys 3m21.640s

So the major bottleneck is thumbnail generation, which I am currently doing Gdk.Pixbuf , for EXIF information I am using beautiful gexiv2 from awesome yorba guys.

Now to optimize I have to use some other library to generate thumbnails, which other libraries I can use ?

On the side note, I am not saving the thumbnails on disk but creating base64 encoded strings of them (I know I am bad, not following the thumbnail spec).

Tags: , ,

4 Comments to Speed , Vala, Sqlite3 and optimization

Fabian
July 7, 2010

What about all-famous ImageMagick or gegl?

kd
July 7, 2010

I totally forgot about imagemagic, thanks :D

Tweets that mention Speed , Vala, Sqlite3 and optimization | Kushal , kD & FOSS -- Topsy.com
July 8, 2010

[...] This post was mentioned on Twitter by DiPot, Planet Fedora. Planet Fedora said: Kushal Das: Speed , Vala, Sqlite3 and optimization: All started as just another stupid idea, writing an image inde… http://bit.ly/9GUbTe [...]

More on thumbnailing and optimization | Kushal , kD & FOSS
July 9, 2010

[...] my last post, spent most of the of the time on different imaging libraries to find a faster way of doing [...]

Leave a comment

Search

Archives

Categories