Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2021 - online Python 3 training - see ((here)).

Our plans were to retire in summer 2020 and see the world, but Coronavirus has lead us into a lot of lockdown programming in Python 3 and PHP 7.
We can now offer tailored online training - small groups, real tutors - works really well for groups of 4 to 14 delegates. Anywhere in the world; course language English.

Please ask about private 'maintenance' training for Python 2, Tcl, Perl, PHP, Lua, etc.
Long job, showing interactive monitoring
Optimising Python example from a Well House Consultants training course
More on Optimising Python [link]

This example is described in the following article(s):
   • Some tips and techniques for huge data handling in Python - [link]

Source code: longjob Module: Y305

import os
import time
import sys
import signal

# constants that we may want to change later!

analyse = "elogs"
tracey = sys.stderr

def tellme(sofar,stage):
        # Assumes that we have at least 1 200 and at least 1 404 each day!
        # \n in next line is to give tracey a bit of space!
        print "\nHere are the",stage,"results"
        for row in sofar:
                print row[0],row[1]["200"],row[1]["404"]
        # print is to leave a gap before tracey continues
        print

def sighandler(which, frame):

        # This could be run at ANY point ... don't do much in here!

        global interim
        interim = 1

        # If ^C is pressed twice within a second, really do kill it!

        now = time.time()
        sighandler.recent += 1
        if now - sighandler.recent < 1:
                sys.exit(0)
        sighandler.recent = now

# initialise an attribute of the function - midnight on 1st Jan 1970 ;-)

sighandler.recent = 0

# Pick up list of data files to be processed

sources = os.listdir(analyse)
sources.sort()
results = []

# Set up signal handler - if you get ^C, run sighandler in place of
# default handler which kills the program!

signal.signal(signal.SIGINT,sighandler)

# loop through the data, with progress bar
# to indicate proportion, we get a count of estimated work to do first

totalwork = len(sources)
counter = 0
interim = 0

for source in sources:

        # Fixed width format to avoid "droppings"
        # to stderr so that if you redirect, progress still goes to screen

        percent = 100.0 * counter / totalwork
        report = "Here we go ... {:8.2f}% of the way\r".format(percent)
        tracey.write(report)
        tracey.flush()

        # real work goes here
        # print "\nWe are on file",source,"...."
        counter += 1

        # work through the data

        statuscounters = {}

        for lyne in open(analyse + "/" + source):
                parts = lyne.split(" ")
                statuscode = parts[8]
                try:
                        statuscounters[statuscode] += 1
                except:
                        statuscounters[statuscode] = 1
        results.append([source,statuscounters])

        # next line is a dummy to delay us
        # time.sleep (1)

        if interim:
                tellme(results,"INTERIM")
                interim = 0

cleaner = " " * len(report) + "\r"
tracey.write(cleaner)

tellme(results,"FINAL")

""" Sample Output

munchkin:yi grahamellis$ python longjob
^Cre we go ... 20.00% of the way
Here are the INTERIM results
ac_20130401 39479 4475
ac_20130402 90888 13780
ac_20130403 84003 13105
ac_20130404 78890 12018
ac_20130405 83458 13296
ac_20130406 80828 11941
ac_20130407 71163 6850

Here are the FINAL results
ac_20130401 39479 4475
ac_20130402 90888 13780
ac_20130403 84003 13105
ac_20130404 78890 12018
ac_20130405 83458 13296
ac_20130406 80828 11941
ac_20130407 71163 6850
ac_20130408 76446 6394
ac_20130409 89485 10442
ac_20130410 92841 10281
ac_20130411 95830 11019
ac_20130412 89804 12077
ac_20130413 80315 9380
ac_20130414 79221 6031
ac_20130415 64148 2619
ac_20130416 75625 4403
ac_20130417 81016 3510
ac_20130418 92954 5391
ac_20130419 83012 4298
ac_20130420 101927 4034
ac_20130421 72699 2063
ac_20130422 74925 3029
ac_20130423 110545 3708
ac_20130424 106073 3494
ac_20130425 101609 2542
ac_20130426 100810 2835
ac_20130427 101800 2958
ac_20130428 85227 2437
ac_20130429 78672 3172
ac_20130430 95924 2971

munchkin:yi grahamellis$

"""



Learn about this subject
This module and example are covered on our public Intermediate Python course. If you have a group of three or more trainees who need to learn the subject, we can also arrange a private or on site course for you.

Books covering this topic
Yes. We have over 700 books in our library. Books covering Python are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples
This example comes from our "Optimising Python" training module. You'll find a description of the topic and some other closely related examples on the "Optimising Python" module index page.

Full description of the source code
You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources
• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Web site author
This web site is written and maintained by Well House Consultants.

Purpose of this website
This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Conditions of use
Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2022: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/resources/ex.php • PAGE BUILT: Sun Oct 11 14:50:09 2020 • BUILD SYSTEM: JelliaJamb