Warning: You are viewing old, legacy content. Kept for posterity. Information is out of date. Code samples probably don't work. My opinions have probably changed. Browse at your own risk.
Apr 11, 2008
cURL is amazing. If you're not familiar with this particular technology then let me introduce it now: cURL (and more specifically the PHP interface for cURL) allows the programmer to communicate with various servers using various different protocols. I've been using it recently to build webbots to automate some tedious tasks, and do things that no normal human user of the internet would ever want to do. Here's an example use of this technology, and how I went about building it...
I've been working on an accounting interface for a client. I've built a front end that allows my client's clients to login and view their recent sales activity. They can view summaries of best selling lines broken down by artist, and geographic region. They can also plot sales for any specified time period onto a Google map. It's pretty cool, and got some great feedback from users. Behind the scenes there's a few tables to store the sales information, and manage access control to the data. The tricky bit was getting the sales information into the database. Here's where the webbots come in useful!
The sales information is downloaded from third party sites where the actual sales take place. This can be done regularly, but requires a human to actually log into a secure site, select which reports to download, unpack them from their archive, and then convert and upload them to the sales database.
I programmed a webbot to automate this process, but it took a bit of trial and error, as the remote site extremely well secured! If you've tried to automate download from secure sites (over https with HTML form login) then this might help:
Now I've got a PHP program that uses cURL to login to the remote sites, and download any new sales reports. There's another program to unpack them and import the records to the database. I call both of these from the crontab and schedule downloads every night.
I'm now looking for more uses of webbots, as I'm sure they can automate lots more of the routine things that need doing.
If you're a PHP programmer start with the PHP documentation for cURL that I linked to above. Michael Schrenk's website has some useful wrapper functions that make cURL even easier to use.