Random tidbits from Niranjan

As mentioned in earlier posts, we decided to use java-gnome library. This library provides Gtk bindings in java and works great. However it does not support webkit natively. We had to add support for Webkit and related libraries.

Unfortunately design and build mechanism of java-gnome library is not easily extensible. While default release is easy to build, adding new components is tedious. The library follows two class architecture for Gtk objects. You normally deal with the instance classes which proxy your function call to different class which in turn calls JNI methods to get work done.

The latter class is generated by a python script from a "defs" file. This file declares the functions which the script generator loads and generates java and c code.

Given large amount of work to support webkit, I had to devise some alternate mechanism. Enter "gir" files - or gnome introspection files. These files contain the methods exposed by Gtk objects along with parameter and type information. Using java code generation facilities, it was simple to generate first piece of puzzle. Latter piece was to generate defs files from same gir files.

Overall, it was successful attempt. After the code generation there were about 5 or so errors in the code which I fixed by hand.

Java-gnome is very good library, but difficult to extend. For a good reason, library tries to shield you from lot of underlying details. However shield in this is case is very strong and it becomes difficult to get through the shield to support the cases that java-gnome library does not support.

For example, if you have signals that return "non standard" values, java-gnome library does not support it. Its very difficult to get object for the signals that webkit supports. Most of the webkit signals pass other objects as parameters to handler. Java gnome mechanism returns you corresponding long pointer and has underlying mechanism to convert pointer to java object. However its not public. I am not sure why this was done, but fortunately methods are protected and I just had to add my plumbing class to override the default mechanism. Same case other way round, for the signals, library can not support, I had to write jni calls and do some pointer management. However it's difficult to get the actual pointer from java-gnome unless you are insider. Again solution was to extend same plumbing class so that I can handle underlying data.

All in all, library works great. View gets displayed properly and calls are executed correctly. Barring the difficulties to extend the library, very happy with library.

As described in earlier post scraping web pages is a very challenging task. A quick google search will bring you lot of search results of many libraries.  Here are some of the challenges that ultimately directed us to Webkit.

1. We need a good renderer. A renderer is something that will process all the Javascript, understand and work with CSS - in short what a browser does.This of course is apart from understanding markup and fetching child resources such as images, scripts and css files.
2. It should not be a browser, but should provide hooks for standard browser functionality. In other words, the renderer should allow us to override some of the functions that a browser normally performs - e.g. cookie and html5 data management. This is very important feature if you want to run jobs for multiple users at the same time from same machine.
3. A renderer should have events or notification mechanisms that will allow code to subscribe to when something interesting happens.
4. It should provide DOM access.
5. It should allow you to override/substitute a request and response. For example from scrapers perspective, live chat feature or facebook like buttons are not going to be important. Actually because many of these libraries makes lot of ajax calls, it ends up firing lot more events internally even though UI on the screen does not change.

Webkit fits the bill in every respect. It is perhaps; most widely used rendering engine in world if you consider all the mobile devices as Safari and Android default browsers are based on Webkit. Its actively developed and has a very good community support. We opted to go with Webkit GTK as it works on linux and has a very nice powerful API. Its also used by some browsers like Epiphany internally.

Our next challenge was finding Java bindings for Webkit. Eclipse SWT has a browser component which can use Webkit on linux, but did not suit our requirements. It does not provide DOM access and functionality to override cookies. Also are missing interesting events like page loaded or new window created. I have used this component in some of my applications and really like it, but it was not a good fit for this application.

Further search introduced as to Java-Gnome library. This was very promising. I had a GTK based hello world application running almost immediately. Only drawback, it does not have Webkit support, yet.

We will cover integration of Java-Gnome with Webkit in next post.