Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
Python, Lua, Tcl, C and C++ training - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Java, Ruby, MySQL and Linux / Tomcat systems
 
Helping search engines with appropriate 400 error codes

Our web site's done its best over the years to help people who enter URLs that have slight errors in them - misspellings, wrong directory names, etc ... and we have quite a number of scrips that take parameters which can be modified to give different results - for example
  http://www.wellho.net/net/quote.html?where=L10+1RF
will get you a quotation for a course in Liverpool and
  http://www.wellho.net/net/quote.html?where=17013
will get you a quote for a course in Carlisle, Pennsylvania!

That's very flexible, but it means that there's a wide range of requests that can be made of our servers which in turn can result in the search engines and otherds finding a lot of pages that really aren't wanted / shouldn't be indexed. It just takes one person to put a link somewhere to set off a whole chain, or one malicious spider to start propogating valid but unwanted URLs.

I've spent some time this weekend cleaning up - making sure that we signal rather better to the search engines (and to other visitors too) which URLs are inappropriate. Firstly, some extras in our robots.txt file:
  Disallow: /net/recents.htm
  Disallow: /resources/recents.htm
And that asks any wellbehaved crawler (most are!) not to go into either or these URLs, or any URLs that start with that text. So as well as .html files (which were previously disallowed), we now request that .htm files are not indexed.

And here are some of the error codes that we now generate ...

400 - Your request is a bad one. Most probably you have tried to call up a script with an inappropriate parameter, such as giving us a URL when we expect a number of simple directory name. (Is this an injection attack going on?)

403 - Forbidden. You are not allowed access to this. We may have identified you as some sort of automatic process trying to mirror out site, and that's not appropriate as this is a dynamic site and the output changes.

404 - Not found. Most likely a typo in the URL or a missing page on our site. Possibly an automated probe to see if a page exists. If you get this from a link, please let us know!

410 - Gone. The resource you're looking for does not exist, and is never likely to exist. Please clear your cache of it and don't ask again. When an automated link has accidentally (or maliciously) been generated and points to a whole family of resources, we'll tell you to clear them out via a 410.

Some of these are generated via lines in the .htaccess files:

  # Any crap on the end of "recents" URLs - kill the page!
  RewriteRule ^recents\.html. /err/tiny410.html [R=410,L]
  RewriteRule ^recents\.htm/. /err/tiny410.html [R=410,L]

  # Get rid of query strings with long (injection?) paths
  RewriteCond %{QUERY_STRING} /.*/.*/
  RewriteRule ^smap /err/tiny400.html [R=400,L]
  RewriteRule ^smap.*%3 /err/tiny400.html [R=400,L]

  # If there's more than just a digit on the end, naughty
  RewriteCond %{QUERY_STRING} full=..
  RewriteRule ^landings\.htm /err/tiny403.html [R=403,L]

And sometimes one of our PHP scripts itself sends out a different header:

  header("HTTP/1.1 404 Not Found");

Covered on our PHP Techniques and Linux Web Server courses when appropriate to the delegate group. Also on private courses.
(written 2013-02-11)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A603 - Web Application Deployment - Further httpd Configuration
  [4307] Identifying and clearing denial of service attacks on your Apache server - (2014-09-27)
  [3955] Building up from a small PHP setup to an enterprise one - (2012-12-16)
  [3862] Forwarding a whole domain, except for a few directories - Apache http server - (2012-09-17)
  [3635] Parse error: parse error, unexpected T_STRING on brand new web site - why? - (2012-03-03)
  [3449] Apache Internal Dummy Connection - what is it and what should I do with it? - (2011-09-19)
  [3133] An image from a website that occasionally comes out as hyroglyphics - (2011-01-14)
  [2900] Redirecting a page - silent, temporary or permanent? - (2010-08-03)
  [2478] How did I do THAT? - (2009-10-26)
  [2272] Monitoring and loading tools for testing Apache Tomcat - (2009-07-07)
  [2060] Database connection Pooling, SSL, and command line deployment - httpd and Tomcat - (2009-03-01)
  [1974] Moving a directory on your web site - (2009-01-03)
  [1955] How to avoid duplicating web page maintainance - (2008-12-20)
  [1954] mod_rewrite for newcomers - (2008-12-20)
  [1939] mod_proxy_ajp and mod_proxy_balancer examples - (2008-12-13)
  [1778] Pointing all the web pages in a directory at a database - (2008-08-30)
  [1767] mod_proxy and mod_proxy_ajp - httpd - (2008-08-22)
  [1762] WEB-INF (Tomcat) and .htaccess (httpd) - (2008-08-20)
  [1707] Configuring Apache httpd - (2008-07-12)
  [1636] What to do if the Home Page is missing - (2008-05-08)
  [1619] User and Group settings for Apache httpd web server - (2008-04-22)
  [1566] Strange behaviour of web directory requests without a trailing slash - (2008-03-06)
  [1564] Default file (MiMe types) for Apache httpd and Apache Tomcat - (2008-03-04)
  [1554] Online hotel reservations - Melksham, Wiltshire (near Bath) - (2008-02-24)
  [1551] Which modules are loaded in my Apache httpd - (2008-02-23)
  [1381] Using a MySQL database to control mod_rewrite via PHP - (2007-10-06)
  [1377] Load Balancing with Apache mod_jk (httpd/Tomcat) - (2007-10-02)
  [1355] .php or .html extension? Morally Static Pages - (2007-09-17)
  [1351] Compressing web pages sent out from server. Is it worth it? - (2007-09-14)
  [1207] Simple but effective use of mod_rewrite (Apache httpd) - (2007-05-27)
  [1121] Sharing the load with Apache httpd and perhaps Tomcat - (2007-03-29)
  [1080] httpd.conf or .htaccess? - (2007-02-14)
  [1009] Passing GET parameters through Apache mod_rewrite - (2006-12-27)
  [934] Clustering, load balancing, mod_rewrite and mod_proxy - (2006-11-21)
  [853] To list a directory under httpd on a web server, or not? - (2006-09-02)
  [755] Using different URLs to navigate around a single script - (2006-06-11)
  [662] An unhelpful error message from Apache httpd - (2006-03-30)
  [649] Denial of Service ''attack'' - (2006-03-17)
  [631] Apache httpd to Tomcat - jk v proxy - (2006-03-03)
  [550] 2006 - Making business a pleasure - (2006-01-01)
  [526] Apache httpd - serving web documents from different directories - (2005-12-12)
  [466] Separating 'per instance' data from binaries and web sites - (2005-10-16)
  [345] Spotting a denial of service attack - (2005-06-12)

G902 - Well House Consultants - Web site techniques, utility and visibility
  [4492] Almost so wrong, but perhaps it's right for some? - (2015-05-11)
  [4474] Effect on external factors on traffic to our web sites - an update - (2015-04-26)
  [4401] Selecting RECENT and POPULAR news and trends for your web site users - (2015-01-19)
  [4376] Well House Consultants, Well House Manor, First Great Western Coffee shop, TransWilts / 2014 web site reports - (2015-01-01)
  [4239] Facebook marketing - early experiences - (2014-01-19)
  [4136] How do I post automatically from a PHP script to my Twitter account? - (2013-07-10)
  [4115] More or less back - what happened to our server the other day - (2013-06-14)
  [4076] Web site - fully back! - (2013-04-29)
  [3974] TV show appearance - how does it effect your web site? - (2013-01-13)
  [3896] An email marathon - (2012-10-15)
  [3776] Some traps it's so easy to fall into in designing your web site - (2012-06-23)
  [3745] Legal change - You need to obtain user consent if you use cookies on your website - (2012-06-01)
  [3744] Short Web Addresses for Melksham - (2012-05-30)
  [3734] QR codes with marketing logos embedded - (2012-05-16)
  [3623] Some TestWise examples - helping use Ruby code to check your web site operation - (2012-02-24)
  [3589] Promoting a single one of your domains on the search engines - (2012-01-22)
  [3563] How big is a web page these days? Does the size of your pages matter? - (2011-12-26)
  [3554] Learning more about our web site - and learning how to learn about yours - (2011-12-17)
  [3532] Sharing the user experience - designing a form with the customer in mind - (2011-11-29)
  [3491] Who is knocking at your web site door? Are you well set up to deal with allcomers? - (2011-10-21)
  [3426] Automed web site testing scripted in Ruby using watir-webdriver - (2011-09-09)
  [3367] Google +1 - what is it? - (2011-07-22)
  [3197] Finding and diverting image requests from rogue domains - (2011-03-08)
  [3149] Looking back at www.wellho.net - (2011-01-28)
  [3087] Making the most of critical emails - reading behind the scene - (2010-12-16)
  [3022] Retaining web site visitors - reducing the one page wonders - (2010-10-31)
  [2981] How to set up short and meaningfull alternative URLs - (2010-10-02)
  [2668] Is it worth it? - (2010-03-09)
  [2569] How to run a successful online poll / petition / survey / consultation - (2010-01-10)
  [2552] Web site traffic - real users, or just noise? - (2009-12-26)
  [2532] Analysing Google arrivals by country of origin - (2009-12-10)
  [2519] Status Page / breaks of service in early December - (2009-11-30)
  [2410] Removal of technical resources from this site - (2009-09-19)
  [2389] Writing with our customers words - (2009-09-01)
  [2341] Koulutus, Open Source tietokone kielillä - (2009-08-09)
  [2340] ldning, Open Source dator språk - (2009-08-09)
  [2339] Opplæring, Open Source datamaskinen språk - (2009-08-09)
  [2338] Uddannelse, Open Source computer sprog - (2009-08-09)
  [2337] Opleiding, Open Source computertalen - (2009-08-09)
  [2336] Formação, Open Source computador línguas - (2009-08-09)
  [2335] Ausbildung, die Open-Source-Sprachen - (2009-08-09)
  [2334] Formazione, Open Source computer lingue - (2009-08-09)
  [2333] Formación, de los lenguajes de código abierto - (2009-08-09)
  [2332] Formation, des langages Open Source - (2009-08-09)
  [2225] How important is a front page ranking on a search engine? - (2009-06-09)
  [2065] Static mirroring through HTTrack, wget and others - (2009-03-03)
  [2056] Web Site Loading - experiences and some solutions shared - (2009-02-26)
  [1982] Cooking bodies and URLs - (2009-01-08)
  [1970] Plagarism - who is copying my pages? - (2009-01-02)
  [1961] Making our things easier to find - (2008-12-26)
  [1888] Find the link - (2008-11-16)
  [1856] A few of my favourite things - (2008-10-26)
  [1833] Web Bloopers - good form design - avoiding pitfalls - (2008-10-11)
  [1797] I have been working hard but I do not expect you noticed - (2008-09-14)
  [1793] Which country does a search engine think you are located in? - (2008-09-11)
  [1756] Ever had One of THOSE mornings? - (2008-08-16)
  [1747] Who is watching you? - (2008-08-10)
  [1711] Rapid growth leads to server move - (2008-07-17)
  [1653] How do Google Ads work? - (2008-05-25)
  [1634] Kiss and Book - (2008-05-07)
  [1630] To provide external links, or not? - (2008-05-04)
  [1610] PHP course dot co, dot uk - (2008-04-13)
  [1541] Colour, Composition or Content - (2008-02-16)
  [1534] Where in the world / country is my visitor from? - (2008-02-07)
  [1513] Perl, PHP or Python? No - Perl AND PHP AND Python! - (2008-01-20)
  [1506] Ongoing Image Copyright Issues, PHP and MySQL solutions - (2008-01-14)
  [1505] Script to present commonly used images - PHP - (2008-01-13)
  [1494] A time to update pictures - (2008-01-03)
  [1437] Above the fold with First Great Western - (2007-11-19)
  [1297] Stuffing content into a web page - easy maintainance - (2007-08-09)
  [1237] What proportion of our web traffic is robots? - (2007-06-19)
  [1212] What brought YOU to our web site? - (2007-06-01)
  [1198] From Web to Web 2 - (2007-05-21)
  [1186] Two new pages / sites - (2007-05-14)
  [1184] Finding resources - some pointers - (2007-05-13)
  [1177] Sorting out for a site map - (2007-05-05)
  [1104] Drawing dynamic graphs in PHP - (2007-03-09)
  [1055] Above the fold - (2007-01-28)
  [1029] Our search engine placement is dropping. - (2007-01-11)
  [1015] Search engine placement - long term strategy and success - (2006-12-30)
  [994] Training on Cascading Style Sheets - (2006-12-17)
  [976] Santa at the station - (2006-12-09)
  [916] Driving customers away - (2006-11-07)
  [893] Visibility - (2006-10-14)
  [800] Effective web campaign? - (2006-07-12)
  [767] Finding the language preference of a web site visitor - (2006-06-18)
  [757] Horse and Python training - (2006-06-12)
  [732] Where is a web site visitor browsing from - (2006-05-24)
  [718] Protecting images from theft - (2006-05-12)
  [681] Mirroring a dynamic site - (2006-04-12)
  [658] Keeping the visitors happy and browsing - (2006-03-26)
  [533] Bigger Box Campaign - (2005-12-18)
  [528] Getting favicon to work - avoiding common pitfalls - (2005-12-14)
  [510] Dynamic Web presence - next generation web site - (2005-11-29)
  [492] New Navigation Aid - Launch of My Wellho - (2005-11-11)
  [414] Form Madness - (2005-08-14)
  [376] What brings people to my web site? - (2005-07-13)
  [369] CMS - the minefield of Choices - (2005-07-05)
  [348] Graveyard pages - (2005-06-15)
  [347] Frightening and from-friend viruses and spams - (2005-06-14)
  [322] More maps - (2005-05-23)
  [320] Ordnance Survey - using a 'Get a map' - (2005-05-22)
  [314] What language is this written in? - (2005-05-17)
  [311] Growth pains - (2005-05-14)
  [288] Colour blindness for web developers - (2005-04-22)
  [284] The Iconish language - (2005-04-19)
  [278] Cover all the options - (2005-04-13)
  [276] An apology to Mr Boneparte - (2005-04-11)
  [274] Our most popular resources - (2005-04-10)
  [268] Information request forms, cleaning up spam - (2005-04-05)
  [261] Putting a form online - (2005-03-29)
  [259] Responding to spam - (2005-03-27)
  [222] Who are all these visitors? - (2005-02-20)
  [202] Searching for numbers - (2005-02-04)
  [197] Allow for peak traffic on your web site - (2005-02-01)
  [182] Your personal Google ranking - (2005-01-19)
  [179] The hunt for unique words - (2005-01-16)
  [173] Data Mining - (2005-01-09)
  [165] Implementing an effective site search engine - (2005-01-01)
  [142] Colour for access - (2004-12-06)
  [117] A case of case - (2004-11-14)
  [109] URLs - a service and not a hurdle - (2004-11-04)
  [98] No more 'Error 404' pages. Something better. - (2004-10-24)
  [32] Web design platoon - (2004-08-29)
  [23] Skills and responsibilities - (2004-08-22)


Back to
9 years, and 4000 articles on
Previous and next
or
Horse's mouth home
Forward to
Images of Melksham you havent seen before
Some other Articles
New guest ... becoming returning guest
Annual Accounts - a big job for a small business
Web and console - same principle, same code - Ruby example
Images of Melksham you havent seen before
Helping search engines with appropriate 400 error codes
9 years, and 4000 articles on
Handling failures / absences of your backend server nicely
Travelogue - from the Netherlands to London via Hook and Harwich
Servlet v JSP (Java Server Page). What is the difference?
Tips on Tomcat - moving applications around
4754 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2017: 404 The Spa • Melksham, Wiltshire • United Kingdom • SN12 6QL
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4001_Hel ... codes.html • PAGE BUILT: Sat May 27 16:49:10 2017 • BUILD SYSTEM: WomanWithCat