One of the project I work on involves connecting to various web sites on user's behalf and collecting or scraping data from these pages. Our users provide us credentials for the sites and we aggregate information on user's behalf. Before you start, we are not doing anything illegal like stealing content and our users are actually providing us credentials so that that we can aggregate data.
It sounds very simple to start with. Almost every programming language out there, allows you to make http calls. Various method calls allow you to perform GET or POST operations that most of the sites needs. So obtaining basic contents of the pages is very easy. Heck, you can even use commands like wget to get the page and do your processing afterwards.
It really gets interesting and challenging if you consider dynamic nature of web sites these days. Web sites do all sort of things with the web content. Some of them use javascript to position content where as others bring in dynamic content by making ajax calls. This brings much better user experience but makes it very challenging for scrapers.
In general, if you are scraping any web site you will face following challenges
- Javascript - a users view can be completely different than what page says
- CSS - An element that appears in the markup may not be actually visible to user.
- Ajax calls - Finding our when you can analyse contents of the page can be very challenging. Content you are looking for can appear after few seconds.
- Customization of pages - Users can customize pages. Move around boxes
If you want to process data for multiple users at the same time, you will need to isolate data stores that browser manages for you. As all our efforts start after user logs in to site, its critical for us to isolate cookie store and html5 data store so that there a no mix and match between user sessions. You should also be able to reuse cookies for future processing so that sites think that your machine is "remembered" machine.
To make it more interesting, add complexities of page customization either by user or based on kind of account a user has. Web sites can throw different kind of information based on account type you have. Some account types may allow lot of customization and others may not allow much.
This is a series of small articles for the challenges we encountered and steps we took to address these challenges. One thing for sure, I can not declare this problem as solved as I am sure we have not encountered all the customized pages that sites can throw at you and sites do keep changing.
