Perl is excellent glue-ware ... something I was reminded of towards the end of Friday's Perl Programming Course. A delegate asked me how easy it is to grab an XML resource from the web (such as an RSS feed) and extract data from it. Well - you could write the code yourself, or you could use standard Perl modules which - these days - are shipped with the distibution anyway.

I used the library for web processes to read in the XML feed data:

  use LWP::UserAgent;
  
  $agent = LWP::UserAgent->new; # Create me a browser
  $agent->agent("Well House Consultants"); # Set the browser name
  $req = HTTP::Request->new(GET => ($urlsource)); # Set up the request we'll run
  $res = $agent->request($req); # Run the request
  $page = $res -> content();

Which I then saved to a local file for caching purposes (I had decided not to check the feed more than once every quarter of an hour).

I then intepretter the data stream using the XML::Parser module:

  use XML::Parser;
  
  $parser = new XML::Parser(Handlers => {Start => \&entering, Char => \&handle_char});
  $parser -> parse($page);

And I provided subs called entering and handle_char to handle the tags of interest. Here's some sample output, showing the extracted data:

  text: Re: A good problem to have?
  url: http://www.firstgreatwestern.info/coffeeshop/index.php?topic=11326.msg118375#msg118375
  text: Re: Unique opportunity to travel the Portbury Branch Line - Saturday 29 Sept 2012
  url: http://www.firstgreatwestern.info/coffeeshop/index.php?topic=11167.msg118374#msg118374
  text: Re: A good problem to have?
  url: http://www.firstgreatwestern.info/coffeeshop/index.php?topic=11326.msg118373#msg118373
  text: Re: FGW (bad) experience 27/9/2012 - Maidenhead to Paddington

Full code is [here]. Other samples of LWP are [here] and [here]. Further XML::Parser examples are [here] (SAX) and [here] (DOM), and if you look at any of those examples you find links to further code and articles too.

We'll talk about XML and Web Processes briefly on any of our Per Courses is they're relevant to any of the delegates attending. If you need to get deeper into these modules, into Object Oriented Perl, and into handling large data flows, take a look at our Perl for Larger Projects course.
(written 2012-09-30, updated 2012-10-06)

 P668 - Handling XML in Perl
  [2378] Handling XML in Perl - introduction and early examples - (2009-08-27)
  [2555] Bookkeeping - (2009-12-29)

P402 - Perl - Writing Your Own Simple Client and Server
  [2047] Small Web Server in Perl - (2009-02-18)

Back to Henbury loop, Bristol - a freight railway line with passenger potential?

Previous and next or Horse's mouth home

Forward to Using CGI and Perl to put a simple application online. Sometimes still the best way.

From Structured to Object Oriented Programming.
Public Transport Services - from and to Melksham
October to December 2012 - Public Courses
Using CGI and Perl to put a simple application online. Sometimes still the best way.
Using Perl to read an RSS feed off a web site and extract data - via LWP and XML modules
Henbury loop, Bristol - a freight railway line with passenger potential?
Trains across Wiltshire - an update on the TransWilts
On getting noticed for the right reasons when you ask about job availability
Writing more maintainable Perl - naming fields from your data records
How have Melksham shops changed in 60 years?