Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Some tips and techniques for huge data handling in Python

Python's an excellent tool for handling huge data sets and long-running programs, although some of the elements of the language that you'll use for such work aren't exactly things we teach on our Introduction to Python courses. Yesterday, however, I was teaching an Intermediate Python course, and had a chance to cover a number of these things.

Some elements of note:

a) Progress logging to stderr:

  tracey = sys.stderr


  percent = 100.0 * counter / totalwork
  report = "Here we go ... {:8.2f}% of the way\r".format(percent)

in my code. The output's to stderr rather than stdout so that it won't be redirected to file if there's any redirection done with >. It's output using \r rather than \n to ensure that reports overwrite one another, and I've added a flush so that the output doesn't hang around in buffers but is displayed straight away, even though there are no newlines (\n)s.

b) Reprogramming of ^C


which causes ^C to run a handler:

  def sighandler(which, frame):
    # This could be run at ANY point ... don't do much in here!
    global interim
    interim = 1
    # If ^C is pressed twice within a second, really do kill it!
    now = time.time()
    sighandler.recent += 1
    if now - sighandler.recent < 1:
    sighandler.recent = now

I've tried to do as little as possible in this handler, as the code could be called a just about any time. It tries to do little more than set a flag to indicate that an interim report is to be produced at an appropriate point. However, I have added extra code to pick up ^C twice in a second - if someone's hammering the keyboard then, sure, let the program exit.

Complete source code [here].

If you want ^C to generate an exception, see [here]. That's not a suitable trap where we want to resume execution straight away, as exceptions jump out from a piece of trapping code
(written 2013-05-15, updated 2013-05-18)

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
Y305 - Optimising Python
  [2277] Python classes / courses - what version do we train on? - (2009-07-10)
  [2369] Using a cache for efficiency. Python and PHP examples - (2009-08-21)
  [2462] Python - how it saves on compile time - (2009-10-20)
  [3766] Python timing - when to use a list, and when to use a generator - (2012-06-16)

Y201 - Python for DataMunging and System Admin
  [3479] Practical Extraction and Reporting - using Python and Extreme Programming - (2011-10-14)
  [4211] Handling JSON in Python (and a csv, marshall and pickle comparison) - (2013-11-16)
  [4438] Loving programming in Python - and ready to teach YOU how - (2015-02-22)

Back to
Python network programming - new FTP and socket level examples
Previous and next
Horse's mouth home
Forward to
Quick and easy - showing Python data hander output via a browser
Some other Articles
Identity in Python
A reminder of why we opened Well House Manor for our customers
Test Driven Development in Python - Customer Comes First
Quick and easy - showing Python data hander output via a browser
Some tips and techniques for huge data handling in Python
Python network programming - new FTP and socket level examples
Cacheing class for Python - using a local SQLite database as a key/value store
JSON from Python - first principles, easy example
New Pictures - Melksham Pack Horse Bridge
Spring at Well House Manor - Teas and Coffees, Museum, Garden, Rooms
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/4088_Som ... ython.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb