Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Use LWP UserAgent to check and see if a remote site has changed
Standard Web Modules example from a Well House Consultants training course
More on Standard Web Modules [link]

This example is described in the following article(s):
   • Answering ALL the delegate's Perl questions - [link]
   • Automating access to a page obscured behind a holding page - [link]

Source code: cc2 Module: P408
use LWP::UserAgent;

# Get remote site name

$site = $ARGV[0] || "www.wellho.net";
$limit = 100;
@omit = ("cgi");

# Keep a local mirror for next time comparison

mkdir $site if (! -e $site) ;

# Grab robots.txt; alert user if it has changed and (manual) check is needed

$agent = LWP::UserAgent->new;
$agent->agent("Well House Consultants");
@alt = ("Not Available","Changed","Unchanged","Freshly Collected");
die "robots.txt was $alt[$altered]\n" if ($altered%2);

# Start at index.html ...

@pagequeue = ("http://$site/index.html");
while ($file = shift @pagequeue) {
        next if ($already{$file}); # Skip pages already checked
        last if (++$npages > $limit); # limit pages to handle
        sleep 5 unless ($npages%10); # Pause to prevent denial of service attack

        $already{$file} = $stat = getpage($file);
        if ($altered == 1 or $altered == 3) {
                print "$file: $stat, page was $alt[$altered]\n";
        # Convert links found so that they can be compared
        foreach (@links) {
                $fully = URI ->new_abs($_,$file) ->canonical;
                $fully =~ s/#.*//;
                $hopper = 1;
                foreach $leave (@omit) {
                        $hopper = 0 if ($fully =~ /$leave/)
                if ($fully =~ m!^http://$site/!i and $hopper) {
                        push @pagequeue,$fully;
$alert and print "alter count is $alert\n";


sub getpage {
$req = HTTP::Request->new(GET => ($pagewant=$_[0]));
$res = $agent->request($req);
$altered = 0; @links = ();
if (($status = $res->code()) == 200) {
        ($mirrorname = $pagewant) =~ tr!/:!%=!;
        $page = $res -> content();
        $altered = 3;
        if (-e "$site/$mirrorname") {
                open (FH,"$site/$mirrorname");
                read (FH, $oldpage, -s "$site/$mirrorname");
                $altered = ($page eq $oldpage) + 1;
        open (FH,">$site/$mirrorname");
        print FH $page;
        @links = ($page =~ m/href\s*=\s*"?([^">\s]+)/ig);
return ($status);
Learn about this subject
This module and example are covered on our public Using Perl on the Web course. If you have a group of three or more trainees who need to learn the subject, we can also arrange a private or on site course for you.

Books covering this topic
Yes. We have over 700 books in our library. Books covering Perl are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples
This example comes from our "Standard Web Modules" training module. You'll find a description of the topic and some other closely related examples on the "Standard Web Modules" module index page.

Full description of the source code
You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources
• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Web site author
This web site is written and maintained by Well House Consultants.

Purpose of this website
This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Conditions of use
Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/resources/ex.php • PAGE BUILT: Sun Oct 11 14:50:09 2020 • BUILD SYSTEM: JelliaJamb