Random tidbits from Niranjan

पूर्वी कीर्तनकार आफळे वगैरे असायचे, आता फॉरवर्डकार आफळे असतात

तंत्रज्ञानामुळे या सर्वांना आपले शिष्य तयार करून परंपरा सहज चालवता येते व असंख्य फॉरवर्डकार आफळे मिळतात

तेंव्हा रामयणातल्या छोट्या प्रसंगावर तासभर कीर्तन चालायचे

आता भक्त, छोट्या प्रसंगावर वा घटनांवर पाच पाच पानी मेसेज फॉरवर्ड करतात

तेंव्हा एक सोंगाड्या विनोदी पात्र म्हणून असायचा

आता ती जबाबदारी काही तरुण राजकीय मंडळी सांभाळतात

तेंव्हा आयुर्वेद व पथ्ये फक्त वैद्यानाच माहिती होती

आता जपानपासून ते अमेरिकेपर्यंत सगळे आयुर्वेदाची महती सांगतात

तेंव्हा गरम पाण्यापासून ते दही भातापर्यंतचे पदार्थ हे खाण्यापिण्या करता होते

आता तुमचे आरोग्य ठीक राहायला या गोष्टींची औषधे होत चालली आहेत

तेंव्हा मी मुकाट गप्प असायचो

आता, मी पण मनोगत लिहून लोकांना बोअर करतो

As mentioned in earlier posts, we decided to use java-gnome library. This library provides Gtk bindings in java and works great. However it does not support webkit natively. We had to add support for Webkit and related libraries.

Unfortunately design and build mechanism of java-gnome library is not easily extensible. While default release is easy to build, adding new components is tedious. The library follows two class architecture for Gtk objects. You normally deal with the instance classes which proxy your function call to different class which in turn calls JNI methods to get work done.

The latter class is generated by a python script from a "defs" file. This file declares the functions which the script generator loads and generates java and c code.

Given large amount of work to support webkit, I had to devise some alternate mechanism. Enter "gir" files - or gnome introspection files. These files contain the methods exposed by Gtk objects along with parameter and type information. Using java code generation facilities, it was simple to generate first piece of puzzle. Latter piece was to generate defs files from same gir files.

Overall, it was successful attempt. After the code generation there were about 5 or so errors in the code which I fixed by hand.

Java-gnome is very good library, but difficult to extend. For a good reason, library tries to shield you from lot of underlying details. However shield in this is case is very strong and it becomes difficult to get through the shield to support the cases that java-gnome library does not support.

For example, if you have signals that return "non standard" values, java-gnome library does not support it. Its very difficult to get object for the signals that webkit supports. Most of the webkit signals pass other objects as parameters to handler. Java gnome mechanism returns you corresponding long pointer and has underlying mechanism to convert pointer to java object. However its not public. I am not sure why this was done, but fortunately methods are protected and I just had to add my plumbing class to override the default mechanism. Same case other way round, for the signals, library can not support, I had to write jni calls and do some pointer management. However it's difficult to get the actual pointer from java-gnome unless you are insider. Again solution was to extend same plumbing class so that I can handle underlying data.

All in all, library works great. View gets displayed properly and calls are executed correctly. Barring the difficulties to extend the library, very happy with library.

I don't know the solution yet and I have posted a question on stackoverflow here. If anyone knows solution please either answer at stackoverflow or drop a comment here.

Basic story is there is C/C++ code which creates a JVM and loads and loads/executes some classes. I don't have any control over how the process is started. This process creates a JVM. It is this process to which I want to attach JDB or other java debuggers such as eclipse.

Long story. I am working on Java bindings for webkit. Specifically Webkit GTK V2. For version 2, webkit fires a web processes internally and communicates with it. To allow access to DOM, you can write extensions which are loaded by the web process dynamically.

I wrote the extension that creates instance of JVM and loads my classes. This works as expected. Now the problem is to debug the Java code I have to solely rely on println or log statements. If I run jps, web process is displayed as one the processes. So there is something in the world that knows its related to JVM even if everything is dynamic.

I would like to connect JDB to this process and debug my java code. If you know how to connect JDB to running process - which is not started with Java command, please let me know.

After some research,it turned out that solution was very simple. When you create VM, just pass the debugger option as follows. Connect debugger to port 9836 and you in the debugger.

JavaVMOption options[3];
options[0].optionString = ;
options[1].optionString = "-Xdebug";
options[2].optionString = "-agentlib:jdwp=transport=dt_socket,server=y,address=9836,

As described in earlier post scraping web pages is a very challenging task. A quick google search will bring you lot of search results of many libraries.  Here are some of the challenges that ultimately directed us to Webkit.

1. We need a good renderer. A renderer is something that will process all the Javascript, understand and work with CSS - in short what a browser does.This of course is apart from understanding markup and fetching child resources such as images, scripts and css files.
2. It should not be a browser, but should provide hooks for standard browser functionality. In other words, the renderer should allow us to override some of the functions that a browser normally performs - e.g. cookie and html5 data management. This is very important feature if you want to run jobs for multiple users at the same time from same machine.
3. A renderer should have events or notification mechanisms that will allow code to subscribe to when something interesting happens.
4. It should provide DOM access.
5. It should allow you to override/substitute a request and response. For example from scrapers perspective, live chat feature or facebook like buttons are not going to be important. Actually because many of these libraries makes lot of ajax calls, it ends up firing lot more events internally even though UI on the screen does not change.

Webkit fits the bill in every respect. It is perhaps; most widely used rendering engine in world if you consider all the mobile devices as Safari and Android default browsers are based on Webkit. Its actively developed and has a very good community support. We opted to go with Webkit GTK as it works on linux and has a very nice powerful API. Its also used by some browsers like Epiphany internally.

Our next challenge was finding Java bindings for Webkit. Eclipse SWT has a browser component which can use Webkit on linux, but did not suit our requirements. It does not provide DOM access and functionality to override cookies. Also are missing interesting events like page loaded or new window created. I have used this component in some of my applications and really like it, but it was not a good fit for this application.

Further search introduced as to Java-Gnome library. This was very promising. I had a GTK based hello world application running almost immediately. Only drawback, it does not have Webkit support, yet.

We will cover integration of Java-Gnome with Webkit in next post.

One of the project I work on involves connecting to various web sites on user's behalf and collecting or scraping data from these pages. Our users provide us credentials for the sites and we aggregate information on user's behalf.  Before you start, we are not doing anything illegal like stealing content and our users are actually providing us credentials so that that we can aggregate data.

It sounds very simple to start with. Almost every programming language out there,  allows you to make http calls.  Various method calls allow you to perform GET or POST operations that most of the sites needs. So obtaining basic contents of the pages is very easy. Heck, you can even use commands like wget to get the page and do your processing afterwards.

It really gets interesting and challenging if you consider dynamic nature of web sites these days. Web sites do all sort of things with the web content. Some of them use javascript to position content where as others bring in dynamic content by making ajax calls. This brings much better user experience but makes it very challenging for scrapers.

In general, if you are scraping any web site you will face following challenges

• Javascript - a users view can be completely different than what page says
• CSS - An element that appears in the markup may not be actually visible to user.
• Ajax calls - Finding our when you can analyse contents of the page can be very challenging. Content you are looking for can appear after few seconds.
• Customization of pages - Users can customize pages. Move around boxes

If you want to process data for multiple users at the same time, you will need to isolate data stores that browser manages for you. As all our efforts start after user logs in to site, its critical for us to isolate cookie store and html5 data store so that there a no mix and match between user sessions. You should also be able to reuse cookies for future processing so that sites think that your machine is "remembered" machine.

To make it more interesting, add complexities of page customization either by user or based on kind of account a user has. Web sites can throw different kind of information based on account type you have. Some account types may allow lot of customization and others may not allow much.

This is a series of small articles for the challenges we encountered and steps we took to address these challenges. One thing for sure, I can not declare this problem as solved as I am sure we have not encountered all the customized pages that sites can throw at you and sites do keep changing.