15
Jan

Scraping web pages - Part 3

As mentioned in earlier posts, we decided to use java-gnome library. This library provides Gtk bindings in java and works great. However it does not support webkit natively. We had to add support for Webkit and related libraries.

Unfortunately design and build mechanism of java-gnome library is not easily extensible. While default release is easy to build, adding new components is tedious. The library follows two class architecture for Gtk objects. You normally deal with the instance classes which proxy your function call to different class which in turn calls JNI methods to get work done.

The latter class is generated by a python script from a "defs" file. This file declares the functions which the script generator loads and generates java and c code.

Given large amount of work to support webkit, I had to devise some alternate mechanism. Enter "gir" files - or gnome introspection files. These files contain the methods exposed by Gtk objects along with parameter and type information. Using java code generation facilities, it was simple to generate first piece of puzzle. Latter piece was to generate defs files from same gir files.

Overall, it was successful attempt. After the code generation there were about 5 or so errors in the code which I fixed by hand.

Java-gnome is very good library, but difficult to extend. For a good reason, library tries to shield you from lot of underlying details. However shield in this is case is very strong and it becomes difficult to get through the shield to support the cases that java-gnome library does not support.

For example, if you have signals that return "non standard" values, java-gnome library does not support it. Its very difficult to get object for the signals that webkit supports. Most of the webkit signals pass other objects as parameters to handler. Java gnome mechanism returns you corresponding long pointer and has underlying mechanism to convert pointer to java object. However its not public. I am not sure why this was done, but fortunately methods are protected and I just had to add my plumbing class to override the default mechanism. Same case other way round, for the signals, library can not support, I had to write jni calls and do some pointer management. However it's difficult to get the actual pointer from java-gnome unless you are insider. Again solution was to extend same plumbing class so that I can handle underlying data.

All in all, library works great. View gets displayed properly and calls are executed correctly. Barring the difficulties to extend the library, very happy with library.