Poor Man's Webspider

March 1st, 2010

Webspiders are fun, but the learning curve is awfully steep. Websites don’t like crawlers stumbling about where they’re not wanted, and barriers as simple as a login screen can stymie a beginner. Add in checks on user agents strings and javaScipt-heavy links, and your weekend is over before you’ve gotten anything to work—side project over! This blog post will show you how to turn your browser and lamp server into a spider capable of taking you straight to the fun.

The Basic Idea

A greasemonkey script will pull data off the pages we’re interested in and send them to a php script. The php script will then tell the greasemonkey script what to do next: either open an alert box telling the user something went wrong, or move on to another url.