Our web site is not best suited to off-line browsing these days - it may be flexible, but if you want to take a copy of it, but it onto a CD, then browse away from the Internet, please resist the temptation. Why is it NOT a good idea to 'blind mirror' us?

1. The Changing nature of our web site. Our pages are adaptive; if you browse from Aberdeen, you'll be offered pulldown menus asking if you're in Aberdeen, Inverness, Dundee, Perth, but if you're browsing from Bristol, you'll be offered Bristol, Bath, Newport, Taunton. If you're browsing with Internet Explorer, some adoption of the HTML will be made to accommodate non-standard features. Your previous visit history will be noted and you'll have different options highlighted as our page is presented in a way to help you navigate. None of these features can work from a mirror CD!

2. Our size. We've got around 15,000 different URLs on this web site ... pages ranging from pictures of Gosport Station to using Utility methods to construct objects of different type in Python, and it's unlikely that you'll want them all - so mirroring is a very slow and very blunt tool which hurts ...

3. Our bandwidth. It's a serious resource hog if you try to copy all of our pages. You're costing us a lot of bandwidth, you're slowing down others who are trying to use our site - basically, you're being antisocial (though probably not intentionally so!). And do you know the worst of it ...

4. Out of date. Your mirror copy will rapidly go out of date, as this is a dynamic site where new examples are added, links updated, and comments amended somewhere all the time. Having spent a lot of time creating a traffic jam, you'll find that the destination really wasn't worth going to.

5. Copyright issues. I am also concerned about our copyright issues; I appreciate that duplicating content is easy, but I would much rather provide a feed to people as they need pages than have - as I have found in the past - mirrored pages that have out-of-date or unaltered absolute links, and are said to be in our name - they get us a bad reputation when really they are an imitation, and ought to be the sincerest form of flattery.

If you're thinking of mirroring us ... please don't do it ... and if you have found this page unexpectedly ... our web site probably thinks that you are trying to mirror it, and is asking you not to do so!

How do we detect mirroring operations?

There are certain programs that do it, and we look for things like wget (link) and HTTrack (link) in our User Agent requests / logs. Such signals aren't going to find the people who try to hide what they're doing, but we have other flags that may find them. This is something we discuss on courses such as Linux Web Server which helps you with your httpd deployment.

How should you as a webmaster handle such bulk download requests?

First things first - work out what you want to do. Do you want to allow mirrors, allow part of the site to be mirrored, rudely lock and bolt the front door against mirroring, or hang up a polite sign that says 'please do not mirror'. And if you go for the latter, how do you get your mirrorers to actually read the sign?

If you've decided to restrict your users from mirroring, have a look at robots.txt, and have a look too at the environment variables that are set by the user agent and their use in conjunction with either deny directives or RewriteCond directives. And if you have common include files, you can put some database recording and monitoring in there to pick up unusual traffic flows that are the characteristic of mirroring attempts on larger sites ... all of which make very long subjects for a blog, but for excellent lunchtime discussions on a PHP techniques Workshop!
(written 2009-03-03 03:20:11)

 G911 - Well House Consultants - Search Engine Optimisation
  [2562] Tuning the web site for sailing on through this year - (2010-01-03)
  [2552] Web site traffic - real users, or just noise? - (2009-12-26)
  [2428] Diluting History - (2009-09-27)
  [2330] Update - Automatic feeds to Twitter - (2009-08-09)
  [2324] What search terms FAIL to bring visitors to our site, when they should? - (2009-08-05)
  [2137] Reaching the right people with your web site - (2009-04-23)
  [2107] How to tweet automatically from a blog - (2009-03-28)
  [2106] Learning to Twitter / what is Twitter? - (2009-03-28)
  [2045] Does robots.txt actually work? - (2009-02-16)
  [2019] Baby Caleb and Fortune City in your web logs? - (2009-01-31)
  [2000] 2000th article - Remember the background and basics - (2009-01-18)
  [1984] Site24x7 prowls uninvited - (2009-01-10)
  [1982] Cooking bodies and URLs - (2009-01-08)
  [1971] Telling Google which country your business trades in - (2009-01-02)
  [1969] Search Engines. Getting the right pages seen. - (2009-01-01)
  [1793] Which country does a search engine think you are located in? - (2008-09-11)
  [1344] Catching up on indexing our resources - (2007-09-10)
  [1029] Our search engine placement is dropping. - (2007-01-11)
  [1015] Search engine placement - long term strategy and success - (2006-12-30)
  [427] The Melksham train - a button is pushed - (2005-08-28)
  [165] Implementing an effective site search engine - (2005-01-01)

G902 - Well House Consultants - Web site techniques, utility and visibility
  [2668] Is it worth it? - (2010-03-09)
  [2569] How to run a successful online poll / petition / survey / consultation - (2010-01-10)
  [2532] Analysing Google arrivals by country of origin - (2009-12-10)
  [2519] Status Page / breaks of service in early December - (2009-11-30)
  [2410] Removal of technical resources from this site - (2009-09-19)
  [2389] Writing with our customers words - (2009-09-01)
  [2341] Koulutus, Open Source tietokone kielillä - (2009-08-09)
  [2340] ldning, Open Source dator språk - (2009-08-09)
  [2339] Opplæring, Open Source datamaskinen språk - (2009-08-09)
  [2338] Uddannelse, Open Source computer sprog - (2009-08-09)
  [2337] Opleiding, Open Source computertalen - (2009-08-09)
  [2336] Formação, Open Source computador línguas - (2009-08-09)
  [2335] Ausbildung, die Open-Source-Sprachen - (2009-08-09)
  [2334] Formazione, Open Source computer lingue - (2009-08-09)
  [2333] Formación, de los lenguajes de código abierto - (2009-08-09)
  [2332] Formation, des langages Open Source - (2009-08-09)
  [2225] How important is a front page ranking on a search engine? - (2009-06-09)
  [2056] Web Site Loading - experiences and some solutions shared - (2009-02-26)
  [1970] Plagarism - who is copying my pages? - (2009-01-02)
  [1961] Making our things easier to find - (2008-12-26)
  [1955] How to avoid duplicating web page maintainance - (2008-12-20)
  [1888] Find the link - (2008-11-16)
  [1856] A few of my favourite things - (2008-10-26)
  [1833] Web Bloopers - good form design - avoiding pitfalls - (2008-10-11)
  [1797] I have been working hard but I do not expect you noticed - (2008-09-14)
  [1756] Ever had One of THOSE mornings? - (2008-08-16)
  [1747] Who is watching you? - (2008-08-10)
  [1711] Rapid growth leads to server move - (2008-07-17)
  [1653] How do Google Ads work? - (2008-05-25)
  [1634] Kiss and Book - (2008-05-07)
  [1630] To provide external links, or not? - (2008-05-04)
  [1610] PHP course dot co, dot uk - (2008-04-13)
  [1554] Online hotel reservations - Melksham, Wiltshire (near Bath) - (2008-02-24)
  [1541] Colour, Composition or Content - (2008-02-16)
  [1534] Where in the world / country is my visitor from? - (2008-02-07)
  [1513] Perl, PHP or Python? No - Perl AND PHP AND Python! - (2008-01-20)
  [1506] Ongoing Image Copyright Issues, PHP and MySQL solutions - (2008-01-14)
  [1505] Script to present commonly used images - PHP - (2008-01-13)
  [1494] A time to update pictures - (2008-01-03)
  [1437] Above the fold with First Great Western - (2007-11-19)
  [1297] Stuffing content into a web page - easy maintainance - (2007-08-09)
  [1237] What proportion of our web traffic is robots? - (2007-06-19)
  [1212] What brought YOU to our web site? - (2007-06-01)
  [1207] Simple but effective use of mod_rewrite (Apache httpd) - (2007-05-27)
  [1198] From Web to Web 2 - (2007-05-21)
  [1186] Two new pages / sites - (2007-05-14)
  [1184] Finding resources - some pointers - (2007-05-13)
  [1177] Sorting out for a site map - (2007-05-05)
  [1104] Drawing dynamic graphs in PHP - (2007-03-09)
  [1055] Above the fold - (2007-01-28)
  [994] Training on Cascading Style Sheets - (2006-12-17)
  [976] Santa at the station - (2006-12-09)
  [916] Driving customers away - (2006-11-07)
  [893] Visibility - (2006-10-14)
  [800] Effective web campaign? - (2006-07-12)
  [767] Finding the language preference of a web site visitor - (2006-06-18)
  [757] Horse and Python training - (2006-06-12)
  [732] Where is a web site visitor browsing from - (2006-05-24)
  [718] Protecting images from theft - (2006-05-12)
  [681] Mirroring a dynamic site - (2006-04-12)
  [658] Keeping the visitors happy and browsing - (2006-03-26)
  [649] Denial of Service ''attack'' - (2006-03-17)
  [533] Bigger Box Campaign - (2005-12-18)
  [528] Getting favicon to work - avoiding common pitfalls - (2005-12-14)
  [510] Dynamic Web presence - next generation web site - (2005-11-29)
  [492] New Navigation Aid - Launch of My Wellho - (2005-11-11)
  [414] Form Madness - (2005-08-14)
  [376] What brings people to my web site? - (2005-07-13)
  [369] CMS - the minefield of Choices - (2005-07-05)
  [348] Graveyard pages - (2005-06-15)
  [347] Frightening and from-friend viruses and spams - (2005-06-14)
  [322] More maps - (2005-05-23)
  [320] Ordnance Survey - using a 'Get a map' - (2005-05-22)
  [314] What language is this written in? - (2005-05-17)
  [311] Growth pains - (2005-05-14)
  [288] Colour blindness for web developers - (2005-04-22)
  [284] The Iconish language - (2005-04-19)
  [278] Cover all the options - (2005-04-13)
  [276] An apology to Mr Boneparte - (2005-04-11)
  [274] Our most popular resources - (2005-04-10)
  [268] Information request forms, cleaning up spam - (2005-04-05)
  [261] Putting a form online - (2005-03-29)
  [259] Responding to spam - (2005-03-27)
  [222] Who are all these visitors? - (2005-02-20)
  [202] Searching for numbers - (2005-02-04)
  [197] Allow for peak traffic on your web site - (2005-02-01)
  [182] Your personal Google ranking - (2005-01-19)
  [179] The hunt for unique words - (2005-01-16)
  [173] Data Mining - (2005-01-09)
  [142] Colour for access - (2004-12-06)
  [117] A case of case - (2004-11-14)
  [109] URLs - a service and not a hurdle - (2004-11-04)
  [98] No more 'Error 404' pages. Something better. - (2004-10-24)
  [32] Web design platoon - (2004-08-29)
  [23] Skills and responsibilities - (2004-08-22)

Back to East of Melksham Countryside

Previous and next or Horse's mouth home

Forward to Melksham Industrial

Efficient calls to subs in Perl - avoid duplication, gain speed
Playing Catchup
Perl - lists do so much more than arrays
Melksham Industrial
Static mirroring through HTTrack, wget and others
East of Melksham Countryside
Internal Dummy Connections on Apache httpd
Virtual hosting and mod_proxy forwarding of different domains (httpd)
Tomcat 6 - Annotated Sample Configuration Files
Database connection Pooling, SSL, and command line deployment - httpd and Tomcat