Automating your scripts with WWW::Mechanize

CPAN provides a great module for Perl called WWW:Mechanize. To quote CPAN:

WWW::Mechanize, or Mech for short, helps you automate interaction with a website. It supports performing a sequence of page fetches including following links and submitting forms. Each fetched page is parsed and its links and forms are extracted. A link or a form can be selected, form fields can be filled and the next page can be fetched. Mech also stores a history of the URLs you’ve visited, which can be queried and revisited.

This post details how to install the module as well as a quick example of automation that logs in to a secure website (MyThree.com.au) and extracts information (Bill Info & Data Usage) without ever having to leave your console …

Installation
I like to use the ActiveState package of Perl, which comes pre-installed with LWP (if on Windows). If however you are installing on a non-windows platform, you may need to update your Perl installation with the LWP module. I think the easiest way to do this is to download the module direct from CPAN here. If you don’t do this, then your cpan console will most likely return the following type of error whenever you try to install a new module:
cpan> Could not fetch authors/ .gz
LWP not available

Once downloaded you can install libwww-perl using the normal Perl module distribution drill:
sudo perl Makefile.PL
sudo make
sudo make test
sudo make install

Now you can use the CPAN console to update modules in your perl installation easily.
If you are using a win32 platform, I wouldn’t bother with the cpan console, I would just use the Perl Package Manager and install the win32 bundles direct from the following repository:
ppm install http://theoryx5.uwinnipeg.ca/ppms/WWW-Mechanize.ppd
ppm install http://theoryx5.uwinnipeg.ca/ppms/Crypt-SSLeay.ppd

However if you’re on a *nix environment, then the cpan console is perhaps your best bet:
>sudo cpan
install Bundle::CPAN #this isn't mandatory, but it's good to update in general
install IO::Socket::SSL
install HTML::TokeParser
install WWW::Mechanize
install Crypt::SSLeay #if you're going to automate https transactions


Example Code
Now that you’ve installed the module, you can write some code that will automate actions performed on a typical web site that uses https communications. In the following example, I automate a user logging in to a secure website (www.my.three.com.au) who then obtains summary billing and usage information.

Make sure you include the use statements required to add this module’s functionality to your code:
use WWW::Mechanize;
use Crypt::SSLeay;

Create a new mech object:
$mech = WWW::Mechanize->new();


Open a URL and check the http return code:
$url = 'https://www.my.three.com.au/My3/jfn';
$result = $mech->get( $url );
die "GET failed\n" unless $result->is_success;

Submit a form on the web page after providing some html field details and check the http return code:
$result = $mech->submit_form(
form_name => 'login',
fields => {
login => $mobile,
password => $password
}
);
die "SUBMIT failed\n" unless $result->is_success;

Parse the string that is returned from a successful login for relevant information:
$content = $result->as_string();
if($content =~ /\\$ ([0-9\.]+)/)
{
$value = $1;
print "Total Balance Due \t\t\$$value \n";
}

Simulate clicking on an embedded link within a frame. To figure out the URL query parameters, and sub frame page information I like to use Firefox’s page info feature to determine the correct parameters and sequence. Another handy plugin is LiveHTTP headers, which lets you scrutinize transactions, capture them, replay them and so on.

$params="selectedLineNumber=$mobile&&oid=L7%3A3714843&ctx=L&mfunc=1055&jfnRC=7";
$url="https://www.my.three.com.au/My3/jfn?$params";
$result = $mech->get( $url );
die "GET failed\n" unless $result->is_success;

Parse for further content such as your data usage summary:
$content = $result->as_string();
if($content =~ /Mobile Broadband \(2048.*\n.*\n.*\n\s+(\d+\.\d+)/)
{
$value = $1;
print "Mobile Broadband Remaining \t$value GB \n";
}

Execution
Test and execute your code. The example code below prints billing and usage operation from the console. You can then automate this functionality into other aspects of your operating system (on Mac OSX I like to make widgets that achieve similar outcomes):
/Users/Koops/Documents/working/90KTS>./MyThree.pl
Opened login page ...
Passed 1st login ...
Passed 2nd login ...
Total Balance Due $49.00
Mobile Broadband Remaining 1108.083 MB

Complete Code Example
#!/usr/bin/perl
# $Id: MyThree.pl,v 1.0 2007/09/24 03:29:19 Koops Exp $
use WWW::Mechanize;
use Crypt::SSLeay;

$content = $result->as_string();
if($content =~ /Mobile Broadband \(2048.*\n.*\n.*\n\s+(\d+\.\d+)/)
{
$value = $1;
print "Mobile Broadband Remaining \t$value MB \n";
}