May 22, 2016

Saudi Arabia or Weymouth?

It's a quiet, cool Sunday morning ... great to be back in the UK having spent last week delivering a Ruby Course in Saudi Arabia. Walking out of my accommodation there just after 7 in the mrning, I was already ht by a wall of heat - in the high 30s celcius - for the 15 minute / 2 checkpoint walk to the training room, in the most magnificent air conditioned centre. The contrast today - I'm going to be on a a selection of rickety old trains as I go up to Swindon, then travel down via Chippenham and Melksham for a day at the seaside - the first "Weymouth" of the year. Advertising has been limited - a singlepost on the TransWilts facebook page; and I find my self excited tey worried as to whether we'll have an empty train , a full one, or somewhere between. So much depends on how the weather looks this morning. Report back to follow.

 

Posted by gje at 06:05 AM | Comments (0)
More about Graham Ellis of Well House Consultants
Related topics: via article database

May 20, 2016

One line scripts - Awk, Perl and Ruby

The art of the one line script, originating in awk and passing through Perl lives on in Ruby!

  WomanWithCat:4c grahamellis$ awk '/Upper/{c++}; END { print(c,"of",NR)}; BEGIN {c=0}' rstats2015.txt
  5 of 2537
  WomanWithCat:4c grahamellis$

  WomanWithCat:4c grahamellis$ perl -n -e '$c++ if /Upper/; END { print("$c of $.\n")}; BEGIN {$c=0}' rstats2015.txt
  5 of 2537
  WomanWithCat:4c grahamellis$

  WomanWithCat:4c grahamellis$ ruby -n -e 'c += 1 if /Upper/; END { puts "#{c} of #{$.}"}; BEGIN {c=0}' rstats2015.txt
  5 of 2537
  WomanWithCat:4c grahamellis$

I'm not a great one to encourage short scripts like this in general use ... but as a teaching exercise they're interesting in showing the flexibility of the language, and occasioanally used in a shell script, they can save the whole script having to be moved to another language just because of one analysis / calculation needed.

Posted by gje at 08:12 AM | Comments (0)
Useful links: Ruby training, Perl training
Related topics: via article database

May 19, 2016

Ruby testing with RSpec - a new example

If you're wanting to test your Ruby classes, I recommend writing a test suite using RSpec - fresh example from today's course [here].

* Write the tests first - that's "test driven development" and then write the code to ensure that all the tests pass.

* If you find fault in you code later, add another test to illustrate the fault and then correct it; running all the RSpec tests again wil ensure you haven't broken anything

* RSpec takes care of test set up, break down, checking and reporting - it's a complete test anagement system so that all you need to do is run the one enclsing program and if everything's good'll just be given a brief summary. Failures will be reported in an undestadable format.

* RSpec can be used with Selenium and / or watir-webdrivers to allow you to test a web site via Firefox or another browser - no need to sit there for hours to check that web site changes haven't damaged anything.

* Gherkin and Cucumber can also be used with RSpec to allow you to use those tools to define your system's behaviour prior for testing.

Posted by gje at 02:41 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

Processing data line by line - iterator in Ruby with yield

If you have a big file / flow of data in Ruby, you probably don't want to read it all in to an array before you do any processing. Not only would that mean you couldn't produce any output until you have read all the data, but it would also make for a big process - probably inefficient, possibly doing a lot of swapping, and perhaps even running out of memory.

Ruby's yield keyword allow you to suspend the operation of a method and pass control to a block defined in the calling code, which may in itself call another method. Using this technique, you can in effect write parallel code, where one method is the equivalent of a Python generator, and another method or methods perform as steps in the processing of each record of data released (one by one) by that generator.

Sample code [here].

Posted by gje at 02:27 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

May 18, 2016

Reading in XML in Ruby with xmlsimple

The xmlsimple Ruby gem allows you to load in a file (or IO handle object or string) of XML as a multidimensional collection - hashes of arrays of hashes. There's a sample program here which includes this code to load the data from after __END__:

  require 'xmlsimple'
  info = XmlSimple.xml_in(DATA)

If I now the structure of what I'm getting, I can then access elements directly, for example:

  p info["access"][0]["where"][0]

and if I don't know what the structure is, I can use Ruby to explore it. The source code at the link above contains a complete recursive piece of code for displaying a Hash that contains hashes, arrays and printable values to get you started!

There's an example using the alternative (standard) rexml library - [here].

Posted by gje at 08:56 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

Expect with Ruby - a training example to get you started

The Expect library, originally written for use wth Tcl, is also available in other languages and today I wrote an example to introduce how it's used in Ruby. Please note this is a spike solution - not going to fail (if it does) in a pretty way.

The principle of expect is that you spawn a process - passing back file handles - and you can then read and write exactly as if you were at the keyboard. Unlike open3 which works on a line by line basis and can give buffering issues, you can read a prompt up to a specific character or pattern in expect, giving you much more flexibility. Beyond this example, you have timeouts, EOF handling, the ability to wait for alternatives, and much more.

When using expect, I recommend you to run the job by hand so that you have an example of the terminal session you're emulating:

  # WomanWithCat:4c grahamellis$ ssh -l wellho www.wellho.net uptime
  # wellho@www.wellho.net's password:
  # 17:23:14 up 5 days, 2:08, 0 users, load average: 0.30, 0.49, 0.52
  # WomanWithCat:4c grahamellis$

You'll require to load the pty and expect modules:

  require 'pty'
  require 'expect'

And here's the automation

  uptimedata = []
  PTY.spawn("ssh -l wellho www.wellho.net uptime") do |ssh_read,ssh_write,pid|
    ssh_read.expect(/sword: /) { |msg| ssh_write.printf("#{ARGV[0]}\n") }
    loop do
      ssh_read.expect(/\n/) { |line| uptimedata.push line}
      break if uptimedata[-1].nil?
      p uptimedata[-1]
    end
  end

And the results:

  WomanWithCat:4c grahamellis$ ruby exprub password-here
  ["\r\n"]
  [" 19:22:11 up 5 days, 4:07, 0 users, load average: 0.15, 0.37, 0.45\r\n"]
  WomanWithCat:4c grahamellis$

Complete example (which does a ping as well) [here]. Although we may not cover this on our regular public ruby course, I would be very happy to cover it in an extra session on the penultimate evening for interested delegates, and also to cover it on private courses.

You'll find more about Expect and its principles under Tcl [here] and [here].

Posted by gje at 08:03 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

Swindon, Chippenham and Melksham - day trips by train to Weymouth start for 2016

Last Sunday, we didn't market a day trip to Weymouth on the TransWilts to celebrate the arrival of the new weekday train and arrival for the summer of an early Sunday train with a good connection to the seaside. With good cause because there were engineering works on the Heart of Wessex line, and with good luck because there were overruning engineering works that prevented the TransWilts train running.

For the coming 3 sundays, I've made a single Facebook post to promote the trip - from Swindon, Chippenham and Melksham to Weymouth. I've copied my post to our press and marketing contact at GWR so that they shouldn't be suprised if a few more people than normal turn up, but I admit (as always on these occasions) to being a little nervous as to how well we will have balanced passengers to capaity. I anticipate a class 153 train (a single carriage) so there isn't much scope for error.

Weymouth trips (with a change at Westbury) on 22nd and 29th May, on 5th June, and from July 3rd onwards. I'm going to see how this Sunday goes before doing any more promotion; I would rather have too much space than too little for people I'm tempting to use the train - perhaps for the first time in their lives.

Posted by gje at 09:59 AM | Comments (0)
Related topics: via article database

Running shell (operating system) commands from within Ruby

Ruby is an excellent systems admin / scripting tool, allowing other shells and processes to be run from within a Ruby program in various ways.

Using backtics or the %x notation, commeand output on STDOUT is routed back into a variable - for example:
  first = `grep -c option247 ac_20160516 ac_20160517`
or
  second = %x!grep -c option247 ac_20160516 ac_20160517!
and you can embed a whole shell script this way using a here document - see [here].

The system function allows you to run a command with STDOUT routes to Ruby's STDOUT, and have the job status returned
  third = system("grep -c option247 ac_20160516 ac_20160517")
and the exec syste function transfers control to the job you're running terminating your rubys process!
  exec "df -h"

To control STDIN and STDERR as well as STDOUT, you can use shell redirects, for example
  fourth = %x!grep -c option247 ac_20160516 ac_20160517 2>&1!
and to run a shell command returning data in parallel while that command is still running, you can use a pipe, for example
  handle = IO.popen("grep -c option247 ac_20160516 ac_20160517")

For ultimate flexibility, the open3 module provides yu with control over all three standard input/outputs:
  require 'open3'
  h_in,h_out,h_err = Open3.popen3("grep -c option247 ac_20160516 ac_20160517")

though you need to be very careful of pipe buffering if you're using this.

All the examples above in complete programs (and with sample output) [here] and [here]. Examples from our Ruby course material.

Update - further example [here] to show setting of environment variables to be passed into shell commands.

Posted by gje at 04:43 AM | Comments (0)
Useful link: Ruby training
Related topics: via article database

May 17, 2016

Exceptions in Ruby - throwing, catching and using

It's far better to use exceptions to trap run time irregularities than to try to forecast all possible errors, as exceptions form a sort of safety net. From today's Ruby Course, [here] is an example where the user is prompted to enter an integer, and I deliberately use the Integer function rather than to_i to extract that number so that I can catch incorrect entries through an exception (Integer throws exceptions, but to_i just returns a 0).

A further example - [here] - shows how exceptions can be caught in methods or thorn up to parents, and how methods can even raise their own exceptions. If you're writing methods, it's sensible to throw exceptions up as it forces the person using you code to take into account situations where no value is to be returned.

Posted by gje at 01:07 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

Alternating valuses / flip-flop / toggle - example in Ruby

If you want to alternate the value of a variable, here's two ways you can do it in Ruby - answers to a question during this week's Ruby Course

  # Too clever?
  
  t = 0
  (1..20).each do
    puts (((t=(t+1)%2)==0) ? "yes" : "no")
    end
  
  #¬†better?
  
  t = 0
  (1..20).each do
    t += 1
    t %= 2
    puts ((t==0) ? "YES" : "NO")
    end

Source code - [here].

Posted by gje at 05:34 AM | Comments (0)
Useful link: Ruby training
Related topics: via article database

May 16, 2016

Separating detailed data code from the main application - Ruby example

In almost every appication, you'll be writing a number of pieces of code that use the same data type. It's tempting initially to write all the detail of the individual fields and attributes of the data into a main program - especially if the application's not very long and you're in a hurry:

  fh = File.new "ac_20160516"
  
  counter = {}
  
  fh.each do |record|
    parts = record.split " "
    if parts[1] == "option247.uk"
      page = parts[6]
  
      if counter[page].nil?
        counter[page] = 1
      else
        counter[page] += 1
      end
      end
    end
  
  pages = counter.keys
  pages.sort!  {|this,that| this.length <=> that.length}
  
  pages.each do |page|
    puts "#{page} #{counter[page]}"
    end
  

Which works well enough, but ties the program down to a specific data format and implements the algorithms individually. If your data records are sometime available in a different way, you need to rewrite all the code, and if you want to use the counter technique elsewhere, you'll have to recode that too. Better to write the code a little longer and clearer and abstact the algorithms and data formats into separate classes.


  class Counter
    def initialize()
      @internal = {}
    end
  
    def count(item)
      @internal[item] = @internal[item].nil? ? 1 : @internal[item]+1
    end
  
    def getKeys()
      return @internal.keys.sort { |this,that| @internal[this] <=> @internal[that]}
    end
  
    def getKey(which)
      return @internal[which]
    end
  end
  class Hit
    def initialize(record)
      @fields = record.chomp.split(" ")
    end
    def issite?(which)
      return @fields[1] == which
    end
    def url
      return @fields[6]
    end
  end
  # ----------------------
  
  if __FILE__ == $0
    fh = File.new "ac_20160516"
    counter = Counter.new
  
    fh.each do |record|
      hit = Hit.new(record)
      parts = record.split " "
      if hit.issite?("option247.uk")
        counter.count(hit.url)
        end
      end
  
    pages = counter.getKeys
    pages.each do |page|
      puts "#{page} #{counter.getKey(page)}"
      end
  
    end

Complete source code of this example - the before [here] and the after [here]. From the Ruby Course I am presenting this week. The datafile is available [here].

Posted by gje at 11:08 PM | Comments (0)
Useful link: Ruby training
Related topics: via article database

May 15, 2016

Annual review of Melksham Matters for Chamber of Commerce AGM

So much has happened in the last 12 months in Melksham that I don't know where to start.

I'm missing the AGM on Tuesday 17th (18:00 at Well House Manor / open to all), so sending in my report by email - composed and checked from Saudi Arabia!

At the general election almost exactly a year ago, Michelle Donelan was elected as our new MP in place of Duncan Hames who had served the Chippenham Constituency so well for five years. Duncan was always going to be an exceptionally hard act to follow, but I'm delighted to report that Michelle has stepped into her role and taken up the mantle of listening to and supporting the community excellently, and hasn't been afraid to assist with causes and campaigns she believes in even before they become populist.

Moving from central to local government, Wiltshire Council has made major economies by cutting many optional budgets, reducing spend to a bare or statutory minimum, and passing responsibility for town and parish services to the town or parish councils where it can. Such cuts are partly due to policy decisions on council tax, partly due to cuts in central government grant funding, and partly due to the ever-rising cost of mandated social care which falls on the council. With staffing reductions too, the council has felt itself unable to apply for new central government funding such as the Total Transport Implementation Fund and the Local Sustainable Transport Interim fund which neighbouring authorities have successfully applied for and are using to oil the wheels for future improvements. It may be that Wiltshire is right to make cuts and keep council tax down at the expense of preparing for the future, or it may be that this is slimming one step to far - come May 2017, there are fresh electiond for Wiltshire Council and it will be interesting to see the popular view in the ballot box.

A merger was considered during this year between our Melksham Town and Melksham Without Parish councils, but was rejected after hearing evidence by Wiltshire Council's team than makes such decisions. I won't replay the arguments on both sides, which were passionately presented - but I will look forward to the two parishing working together for the common good of the town that serves the populations of both if them. Some new housing areas currently in Melksham Without will be transferring to the town in the near future - an agreed sensible move where the boundary went down the middle of a new road in what had become a built up area.

The two councils are now working together on the neighbourhood plan - a plan developed by the community to give some steer into the future development of the town, and with some potential to specify how and where the town should grow, rather than letting it grow almost randomly with little strategic thought.

Growth has continued this year with the opening of Herman Miller's new factory and there's other non-residential developement underway too - for example the coming of the Wiltshire Air Ambulance base on Semington Road. Other developments such as solar farms and a green energy system at Sandridge are here, in development or planned. And housing grows too with development underway at present on the old George Ward site, approved on other areas to the south and east of the town, and in planning on others. A decision on the planning application for the Melksham Link of the Wilts and Berks canal is imminent, and if approved that would be a major step in the development of Melksham for the future.

Wiltshire Council's development of new Rugby and Football facilities at Woolmore farm continues apace, with opening due in the next few months. This scheme has been funded (with commercial facilities for corporate events that will compete with other local businesses) by money from the pot originally earmarked for the Campus at Melksham House, which as been delayed and scaled back with the removal of indoor bowling facilities, and is to be renamed a "Health and Wellbeing Centre".

Melksham's town centre continues to thrive - there are few unoccupied premises available and those which are empty are more usually between lets than looking for tenants. We do notice a more towards cafes and restaurants - a note on the social change in the town perhaps - and we welcome the Water Meadow (a Marstons pub / restaurant) on the Eastern Development which has gained itself an excellent reputation within the first few weeks of opening.

Likewise, Melksham's 3 year trial train service improvement has thrived, already achieving twice the highest passenger forecasts by Wiltshire Council's consultants (Mouchel) prior to the funding application. TransWilts' expectations were higher than the councils, but even they have been hansomely exceeded with Melksham station being the 3rd fastest growing in the UK last year. Looking forward, the service will continue beyond the end of the trial in December - indeed there's an extra train started this week, and we can look forward to longer trains next year, and more exciting developements too.

The story on buses is not so good. Last year, we lost the Melksham Rail Link, where the town bus was making 2 extra runs to the station before its normal daytime service and bringing up to a dozen people to the train on their way to work. Not only were those passengers lost to the bus, but also to the train and in some cases they've had to find new jobs too. Again, the 234 bus service was lost last summer, and the last bus from Chippenham's now at 17:35 rather that 22:16 - too early for people who used to use the bus as the last leg of their commute home, and again people have had to give up jobs. The two early buses on route 272 from Bath were also lost - in this case against a spirited fight from a local councillor - but my personal view is that this was a reasonable cut as the buses were virtually empty.

Wiltshire Council is looking to cut about a half of its bus support budget next year, with potential removal of funding from route 271 and dramatic reduction on routes 14, 68 and 69. A consultation on that option, together with a complete removal of bus support, took part from January through March and brought 11,500 responses. A suggestion made by the option 24/7 group (of which I am a member) has suggested re-organising public transport at a network, allowing savings to be made and a system that encourages new users though better connections, integrated ticketing, information systems and efficiencies, and pulling in modest extra funding from rail designation sources. This approach is supported in pre-talk for the upcoming buses bill which will remove some legal hurdles, and also by Michelle Donealan. Wiltshire Council did not offer this option in their consultation, but has held several meetings with us and operator representatives to investigate further. Should the cuts consulted on proceed, Melksham is likely to see an end of evening and Sunday buses in summer 2017, with the town bus reduced to every 2 hours with 2 runs each day to Holt / Staverton / Trowbridge and towards Corsham.

Next month brings the "Brexit" vote, and it would be unwise at this stage to predict which way the vote will go - but we can be sure that we're in for an interesting aftermath. The following month, Melksham hosts the Party in the Park on 16th July, and the carnival parade is on 23rd - again with a gathering of the community in the park. It will be interest to see how the two events stack up against each other on different Saturdays rather than being combined on the same day.

Your chair and membership development committee member (Colin and Chris) have been doing a sterling job of approaching new businesses and arranging some interesting site visits for the Chamber over the year, and I'll leave it to them to fill you in on that side of things, and on the Wessex Association of Chambers, of which the Melksham Chamber of Commerce and Industry is a member. Many thank also to Phil McMullen for his secretarial and support work for the Chamber. Their work has allowed me to concentrate what little time I've been able to give on some of the matters listed above.

I've very much enjoyed my time as President of the Chamber, and would like to thank members for letting me take this role which I've thoroughly enjoyed. I'm sorry I can't be with you this evening, but I do look forward to seeing my friends at the Chamber again next month, and I thank you all for your support and helping to make my role an enjoyable one.

Posted by gje at 06:47 PM | Comments (0)
Related topics: via article database

Well House Consultants Ltd.
Copyright 2016