As described in earlier post scraping web pages is a very challenging task. A quick google search will bring you lot of search results of many libraries. Here are some of the challenges that ultimately directed us to Webkit.
- We need a good renderer. A renderer is something that will process all the Javascript, understand and work with CSS - in short what a browser does.This of course is apart from understanding markup and fetching child resources such as images, scripts and css files.
- It should not be a browser, but should provide hooks for standard browser functionality. In other words, the renderer should allow us to override some of the functions that a browser normally performs - e.g. cookie and html5 data management. This is very important feature if you want to run jobs for multiple users at the same time from same machine.
- A renderer should have events or notification mechanisms that will allow code to subscribe to when something interesting happens.
- It should provide DOM access.
- It should allow you to override/substitute a request and response. For example from scrapers perspective, live chat feature or facebook like buttons are not going to be important. Actually because many of these libraries makes lot of ajax calls, it ends up firing lot more events internally even though UI on the screen does not change.
Webkit fits the bill in every respect. It is perhaps; most widely used rendering engine in world if you consider all the mobile devices as Safari and Android default browsers are based on Webkit. Its actively developed and has a very good community support. We opted to go with Webkit GTK as it works on linux and has a very nice powerful API. Its also used by some browsers like Epiphany internally.
Our next challenge was finding Java bindings for Webkit. Eclipse SWT has a browser component which can use Webkit on linux, but did not suit our requirements. It does not provide DOM access and functionality to override cookies. Also are missing interesting events like page loaded or new window created. I have used this component in some of my applications and really like it, but it was not a good fit for this application.
Further search introduced as to Java-Gnome library. This was very promising. I had a GTK based hello world application running almost immediately. Only drawback, it does not have Webkit support, yet.
We will cover integration of Java-Gnome with Webkit in next post.
