Adding Expect on top of Tcl - what is it and where can I get a training course to learn about it?
Expect adds extra commands on top of Tcl - allowing the Tcl programmer to control additional processes as is they were really being run from the command line. But because they're being run from within a program, a whole lot of commands can be run with little effort, responses analysed easily, with the normal command output suppressed and replaced by a short report ...
We don't run a separate Expect course - but we do include Expect in our
Learning to program in Tcl course and our
Tcl programming course. The extra modules that cover the Expect extension account for sme 15% of the course - if Expect is not relevant to
any delegates, we just 'overview' it - otherwise it gets thorough coverage. And that can extend into an extra session on the penultimate evening of the course. For those for whom Expect is important, it is
very important!
Let's see an example - I want to check that various web servers that I look after are online from time to time. I could
ping each in turn, manually - for example:
munchkin:tcl12 grahamellis$ ping -c 3 www.wellho.net
PING www.wellho.net (83.170.95.163): 56 data bytes
64 bytes from 83.170.95.163: icmp_seq=0 ttl=53 time=36.290 ms
64 bytes from 83.170.95.163: icmp_seq=1 ttl=53 time=27.647 ms
64 bytes from 83.170.95.163: icmp_seq=2 ttl=53 time=142.831 ms
--- www.wellho.net ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 27.647/68.923/142.831/52.380 ms
munchkin:tcl12 grahamellis$
but if I have five or six (or many more computers) to check, that's too much typing, and prone to errors if I forget one of the hosts one time. And I also have to read back some of the data figures and analyse the results. Far better to write an Expect script to report as follows:
munchkin:tcl12 grahamellis$ expect qp
www.wellho.net came back with a trip time of 26.545
wellho.net came back with a trip time of 166.697
SERVER SLOW OR DISTANT
www.wellho.co.uk came back with a trip time of 33.698
www.firstgreatwestern.info came back with a trip time of 34.132
www.sheepbingo.co.uk came back with a trip time of 39.882
whm.wellho.net came back with a trip time of 18.160
munchkin:tcl12 grahamellis$
In that example, I have checked six hosts and had my Expect script analyse the results returned. Looking at one particular piece of data within the returned report, I've done some arithmetic so see if the server answer time is good, and if it isn't, I've flagged that there's a potential issue.
Taking this script a stage further, I could run it every hour under
cron so that it becomes a monitoring tool, and have it send an alert / text / email to the system admin of servers which fail to respond (we actually have a scheme like this in place - not only crosschecking servers, but also checking that our hotel and office networks are online even when we're not physically present).
The two main extra lines that I've added to what is otherwise just a Tcl script are:
spawn ping -c 3 $mysite
t instigate each ping and
expect "round-trip"
to wait for the returned information (up to the text "round-trip") which can then be analysed.
The
expect command returns a whole load of information in the
expect_out global array, and in particular my program looked at
$expect_out(buffer) member, which contains the text received prior to the words "round-trip".
A final extra note on this program - there's a line
log_user 0
in it. This sets a flag telling expect NOT to echo all spawned command output on the screen. When you're testing an Expect program, you need to see what's happening (in case the exchange hangs up because expect never receives what it's waiting for, for example) and the default setting which echos everything is a huge help. Then you'll add the
log_user 0 line when you're happy that the code works.
Complete source of the program described above -
[here]. Note that it relies on the data format returned by ping and if your operating system has a different format, my program may need amendment to suit. I am using OSX, by the way.
Expect programming sounds simple, and indeed it is if you're well trained / know what you're doing. However, there are a number of non-obvious mistakes that newcomers can make which can result in flaky performance - there's an example
[here] from our course to show such an issue. There are further issues other than the ones we can show in that sample demo - come on a course to learn ;-).
There is a further example of
expect_out use
[here].
You need also to consider
timeout and
eof conditions when you're writing an Expect script - these are beyond the simple examples already quoted.
If you are running a series of processes (such as ping) each of which takes a noticeable elapsed time to run, you
can set Expect to spawn multiple processes and then ask the
expect command to listen for a string on any of them - requires careful through in programming, but very effective. There's an example of this pulling in /robots.txt from multiple servers
[here]. And indeed there are a lot more resources on our site relating to expect processes
[here].
(written 2012-01-08, updated 2012-01-14)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
T211 - Tcl/Tk - What is Expect? Why use it? [286] Automating regular manual procedures - (2005-04-21)
[435] Expect for Windows - (2005-09-04)
[1173] Cheat Sheet / Check list for Expect maintainers - (2007-05-02)
[1174] Installing Tcl and Expect on Solaris 10 - a checklist - (2007-05-02)
[1409] What is Expect? - (2007-10-26)
[1411] Buffering of inputs to expect, and match order - (2007-10-27)
[1469] Curley brackets v double quotes - Tcl, Tk, Expect - (2007-12-12)
[1531] Expecting a item from a list of possibles - (2008-02-04)
[1602] Automating processes through Expect - (2008-04-05)
[2474] Using Tcl and Expect to automate repetitive jobs - (2009-10-24)
[2489] Parallel Pinging, using Python Threads or Expect spawn lists - (2009-11-02)
[3009] Expect in Perl - a short explanation and a practical example - (2010-10-22)
[3286] Should we cover expect and/or Tk on our public Tcl courses? - (2011-05-11)
[4405] Backup procedures - via backup server - (2015-01-24)
[4678] Expect with Ruby - a training example to get you started - (2016-05-18)
Some other Articles
Tcl - apparently odd behaviour of string trimleftMultiple buttons calling the same proc in wish (tcl/tk)Perl functions such as chop change their input parametersNew in Java 7 - and why we are not running public Java 7 coursesAdding Expect on top of Tcl - what is it and where can I get a training course to learn about it?Comparing loop commands in TclTrapping errors in Tcl - the safety net that catch providesImages of the new year in MelkshamTelling which ServerAlias your visitor used - useful during merging domainsFirst of the year