The other day, we suffered a gap in service from one of our dedicated hosts and after the event I wanted to go through the server logs to find how long the gap was, and whether there was any unusual activity prior to the outage. Problem was - I was looking in a 60 Mbyte log file with ovr 200,000 records covering some 60 hours of server use.
A quick Python script highlighted the length of the gap and the line numbers in the file, which I could then look at through an editor.
counter = 0
oldsid = 0
oldstamp = ""
for line in open("access_log"):
counter += 1
stamp = (line.split(" "))[4]
day,h,m,s = (stamp.split(":"))
sid = int(h) * 3600 + int(m) * 60 + int(s)
if oldsid + 300 < sid:
print oldstamp,stamp,counter
oldsid = sid
oldstamp = stamp
print counter
Gave me the results:
wizard:feb13 graham$ python gap
[24/Feb/2013:04:02:15 1
[25/Feb/2013:00:00:00 [24/Feb/2013:23:59:58 54148
[26/Feb/2013:19:31:58 [26/Feb/2013:20:09:34 199300
221290
wizard:feb13 graham$
Which told me the start time of the outage, the end time, and to look around line 199300 of the file. But the code is obfurscated, difficult to follow, and very much a hack solution that's not easily re-usable. Which is fine as a quick fix in a crisis. However, by switching to an object oriented approach, that same issue can be solved with neater and re-usable code - very much more followable.
Taking my original as a
"spike solution", I
"refactored" as follows:
counter = 0
oldsid = 0
oldlatest = None
for line in open("access_log"):
counter += 1
latest = access(line)
sid = latest.getseconds()
if oldsid + 300 < sid:
if oldlatest:
print oldlatest
print latest,counter
oldsid = sid
oldlatest = latest
print counter,"oo version 2"
Which ran as follows:
wizard:feb13 graham$ python oogap2
At [25/Feb/2013:00:00:00 to firstgreatwestern.info from 66.249.76.149
At [24/Feb/2013:23:59:58 to firstgreatwestern.info from 180.76.5.99 54148
At [26/Feb/2013:19:31:58 to firstgreatwestern.info from 65.55.213.220
At [26/Feb/2013:20:09:34 to firstgreatwestern.info from 132.185.160.100 199300
221290 oo version 2
wizard:feb13 graham$
Not only is that set of results more informative and cleaner, but also the code is easier to follow and to re-use. Alas, that code's no longer the full story as I've defined a class of access objects:
class access:
def __init__(this,line):
linebits = line.split(" ")
this.stamp = logtime(linebits[4])
this.sitevisited = linebits[0]
this.remoteip = linebits[1]
def getseconds(this):
return this.stamp.getseconds()
def __str__(this):
return "At " + str(this.stamp) + " to " + this.sitevisited + \
" from " + this.remoteip
Which in turn have made use of a class of logtime objects:
class logtime:
def __init__(this,timestring):
day,h,m,s = (timestring.split(":"))
this.seconds = int(h) * 3600 + int(m) * 60 + int(s)
this.printable = timestring
def getseconds(this):
return this.seconds
def __str__(this):
return this.printable
The end result is probably a more sophisticated approach than I would normally have taken (whether I needed to have
logtime as a separate object is a long discussion we could have) - but the whole example provided a neat, real-life xample for the
Python Programming Course which I have been running this week.
Original source code
[here]. First (intermediate) OO example using a single class
[here]. Final example as shown above -
[here].
(written 2013-02-28, updated 2013-03-01)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
Y106 - Object Oriented Python [477] Class, static and unbound variables - (2005-10-25)
[834] Python makes University Challenge - (2006-08-15)
[900] Python - function v method - (2006-10-20)
[1306] Python class rattling around - (2007-08-16)
[1348] Screw it or Glue it? Access to Object variables - a warning - (2007-09-12)
[1925] Introduction to Object Oriented Programming - (2008-12-06)
[2017] Python - a truly dynamic language - (2009-01-30)
[2169] When should I use OO techniques? - (2009-05-11)
[2604] Tips for writing a test program (Ruby / Python / Java) - (2010-01-29)
[3085] Object Oriented Programming for Structured Programmers - conversion training - (2010-12-14)
[3399] From fish, loaves and apples to money, plastic cards and BACS (Perl references explained) - (2011-08-20)
[3436] Moving from scripting to Object Orientation in Python - (2011-09-13)
[3673] Object oriented or structured - a comparison in Python. Also writing clean regular expressions - (2012-03-26)
[3878] From Structured to Object Oriented Programming. - (2012-10-02)
[3947] this or self - what are they, and what is the difference? (Python) - (2012-12-08)
[4028] Really Simple Class and Inheritance example in Python - (2013-03-04)
[4129] Simple OO demonstration in C++, comparison to Python - (2013-07-01)
[4448] What is the difference between a function and a method? - (2015-03-04)
[4591] From single block to structure and object oriented programming - (2015-12-02)
[4650] Why populate object with values as you construct them? - (2016-02-18)
[4721] When to check an object type - Python isinstance example - (2016-11-03)
Q908 - Object Orientation and General technical topics - Object Orientation: Design Patterns [485] North, Norther and Northest - PHP 5 Objects - (2005-11-04)
[1224] Object Relation Mapping (ORM) - (2007-06-09)
[2322] Looking for a practical standards course - (2009-08-05)
[2741] What is a factory? - (2010-04-26)
[2977] What is a factory method and why use one? - Example in Ruby - (2010-09-30)
[3608] Design Patterns - what are they? Why use them? - (2012-02-12)
[3716] Learning C++ - a design pattern for your first class - (2012-05-02)
[3810] Reading files, and using factories to create vectors of objects from the data in C++ - (2012-07-21)
[3843] Caching Design Patterns - (2012-08-20)
[4096] Perl design patterns example - (2013-05-20)
[4098] Using object orientation for non-physical objects - (2013-05-22)
[4325] Learning to program - what are algorithms and design patterns? - (2014-11-22)
[4330] Java - factory method, encapsulation, hashmap example - (2014-11-27)
[4356] Object factories in C++, Python, PHP and Perl - (2014-12-19)
[4359] How to avoid too many recalculations within an object - (2014-12-21)
[4377] Designing a base class and subclasses, and their extension, in C++ - (2015-01-01)
[4396] Java Utility class - flexible replacement for array. Also cacheing in objects and multiple catch clauses example. - (2015-01-16)
[4421] How healthy are the stars of stage and screen? - (2015-02-09)
[4581] Thin application, thick objects - keep you main code simple. Example in Ruby - (2015-11-21)
[4626] Singleton design pattern - examples and uses - (2016-01-20)
[4663] Easy data to object mapping (csv and Python) - (2016-03-24)
[4673] Separating detailed data code from the main application - Ruby example - (2016-05-16)
Q906 - Object Orientation and General technical topics - Object Orientation: Individual Objects [227] Bellringing and Programming and Objects and Perl - (2005-02-25)
[507] Introduction to Object Oriented Programming - (2005-11-27)
[1543] Learning Object Oriented Principles (and perhaps Java) - (2008-02-17)
[1864] Object Oriented Perl - First Steps - (2008-11-01)
[2171] Cleaning up redundant objects - (2009-05-11)
[2173] Basic OO principles - (2009-05-11)
[2393] A first demonstration of OO, including polymorphism - (2009-09-04)
[2651] Calculation within objects - early, last minute, or cached? - (2010-02-26)
[3721] Naming blocks of code, structures and Object Orientation - efficient coding in manageable chunks - (2012-05-06)
Some other Articles
Backups, Codebase, Strategy and more - dealing with forum incidentsSQL databases from Python - an SQLite exampleMelksham to Chippenham (Station) bus services - times and routesA request for all of Melksham transport groups to work integratedSpike solution, refactored and reusable, Python - ExampleMelksham Community - Annual Report / MCAPCoffee Shop in MelkshamAfternoon Teas and Breakfasts at Well House ManorAcceptable User Policy / vexatious interacterRed, yellow, green or pink dog? Yellow Dog Project