Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
For 2023 - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Automating access to a page obscured behind a holding page

Question: "I have a web page that I visit which sends me an initial response to say that it's working on the results, and then the results appear a few seconds later. I want to use an automed process / program on my computer to visit this page and store the final results, rather than having to access it through the browser. How can I do this?"

What an excellent question - we've all visited pages that come back and say 'I'm working on those results' then 'here are you final results' a few seconds later, and wouldn't it be great to be able to call them up automatically to script the accesses to the page - that way you could make good use of some excellent resources.

The answer comes in two stages. Firstly - you need to understand which of several ways the server is sending the intermediate pages. It could be sending out a single multipart response, with a pause between each of the sections (that's server push). It could be sending out a series of pages, each one as a separate request from the client (that's client pull), or it could be sending out an initial page that includes Javascript which reads further data back from the server via an AJAX type connection. The particular example that I was asked to look at was client pull, and I wrote a short server side script to illustrate that. You can run it here and see the source code (it's in PHP) here. The source code is full commented!

Secondly (and really that is second - you must understand how your server is working before you try to script the front of it!), use a module such as Perl's LWP::UserAgent to automate the access to it. I've written a full example here which scripts the connection to the service that I've written as a demonstration in the paragraph above ... when I run it, the results I get are as follows:

-bash-3.2$ perl coojar
Intermediate: Running ... 0 seconds gone
Intermediate: Running ... 5 seconds gone
Intermediate: Running ... 10 seconds gone
Intermediate: Running ... 15 seconds gone
Intermediate: Completed ...
04:13:51 up 3 days, 12:18, 1 user, load average: 0.18, 0.13, 0.16
FINAL RESULT: Completed ...
04:13:51 up 3 days, 12:18, 1 user, load average: 0.18, 0.13, 0.16

Once again, the code is fully commented - so have a look at the source to see how it works. One thing of note is the concept of a cookie jar - this is the object in memory in which all the cookies are stored. We have chosen fot this particular application to have our own independent set of cookies rather than to share the ones that another browser has on the same client machine, but there are other Perl modules which let you connect to / share the cookie files (cookie jars) or other browsers. The setting of the agent name in the program is no more that a user politeness in most cases; please note that if you're runningthis against our server, you do need to specify an agent name, as we have additional security traps in place and defaulted LWP clients - intentionally - don't work with this script.

LWP::UserAgent is a very powerful automated browsing module and there are other examples in our trainee / source code directory ... including:

Simple LWP browser for testing
Grabbing current currency exchange rates
Validating all links from a page
Checking a static site for changes

Do remember whenever you use LWP::UserAgent that you are writing an automata, and you should check the robots.txt file at the server to ensure that your activities are acceptable to the server's operator. See here and here for more about robots.txt.
(written 2009-09-23)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
P408 - Perl - Standard Web Modules
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [2229] Do not re-invent the wheel - use a Perl module - (2009-06-11)
  [2402] Automated Browsing in Perl - (2009-09-11)
  [3485] Perl - retrieving and caching web resources - (2011-10-18)
  [4099] Perl Dancer - a Perl Framework - Installation and first test - (2013-05-23)
  [4100] Perl Dancer - from installation to your first real application - (2013-05-24)

H301 - PHP - Sticky fields and session
  [1739] Bath, Snake or Nag? - (2008-08-06)
  [1766] Diagrams to show you how - Tomcat, Java, PHP - (2008-08-22)
  [1911] Remember Me - PHP - (2008-11-28)
  [2738] What is all this SESSION stuff about? (PHP) - (2010-04-25)
  [3540] Easy session example in PHP - keeping each customers data apart - (2011-12-06)
  [3820] PHP sessions - a best practice teaching example - (2012-07-27)
  [3918] Multiple page web applications - maintaining state - PHP - (2012-11-10)
  [4070] Passing variable between PHP pages - hidden fields, cookies and sessions - (2013-04-26)

A805 - Web Application Deployment - Perl - modPerl, Mason, CGI.pm and LWP
  [45] CGI v mod_perl - (2004-09-11)
  [260] Embperl - (2005-03-28)

Back to
Variable names like i and j - why?
Previous and next
Horse's mouth home
Forward to
Java Programming Fundamentals
Some other Articles
Exceptions in Java - why and how
Where is my Java class?
Viv.java uses unchecked or unsafe operations - explanation and cure
Java Programming Fundamentals
Automating access to a page obscured behind a holding page
Variable names like i and j - why?
Hello World - a good traditional start to a Java course
Sanity checking the price, and selling up to increase income
Autumn Fruits and Bristol Old Station
Further North - long summer days and lovely countryside
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2023: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/2416_Aut ... -page.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb