Python timing - when to use a list, and when to use a generator

If you're going to cook dinner for the family, it's much more efficient to cook everyone's meal together. If you're a family of four, you won't want to cook four separate pans of carrots, but rather you'll want to cook them all together. But now imagine you're holding a carrot street party, and providing carrots through the day for four hundred rather than four people. You could hire in (or purchase) special resources, but you're much more likely to adopt some sort of batch system. You won't cook all the carrots at once - you're pans aren't big enough, and even if they were, you've not got the capacity to store all of the cooked carrots until they're needed for serving.

The same thing applies when you're programming. In a program with a small amount of data, it's quicker and easier to handle it all via a list, but as the data size grows so would the amount of memory you need for a list, and it becomes quicker to handle the data as it becomes available - using a production line type approach known as an iterator or a generator.

Grand theory - how does that work in practise? In a language like Python, you can use a generator function. I've written about those before (examples HERE and HERE if you want to refer back). In Lua, you would use co-routines, and in any other language you could roll your own. But the question remains "Does it really make a difference?", and I thought I would try it out to see.

Let's look at the tools I'll use:

• Seven log files off our server, for periods from a minute (43 accesses) to a year (nearly 30 million accesses):

  munchkin:pyjun12 grahamellis$ wc -l ac[a-z]* | sort -bn

         43 acminute

       2671 achour

      79015 acday

     533818 acweek

     992112 acfortnight

    2481275 acmonth

   29775300 acyear

• Two programs dave - which stores a list and jenny - which processes each record as it reads it, using a generator. Both programs produce identical output files when run on the same input; I've checked this using the Unix diff command.

• Python's resource module, which is supplied with the standard Unix / Linux distributions, but needs to be imported into your program. See python documentation for full details.

The code I have added at the end of both dave and jenny, to report on resources, is as follows:

  import resource

  used = resource.getrusage(resource.RUSAGE_SELF)

  print "Maximum resident size:",used.ru_maxrss

  print "Run time:",used.ru_utime + used.ru_stime

  print "Page faults:",used.ru_minflt + used.ru_majflt

So - how does that run?

Maximum memory size while running:

Duration	list	generator
Minute	4116480	4112384
Hour	4837376	4108288
Day	30355456	4108288
Week	181055488	4108288
Fortnight	333402112	4112384
Month	837042176	4108288
Year	3351420928	4096000

Run time (sum of system and user time):

Duration	list	generator
Minute	0.055345	0.056894
Hour	0.064015	0.062101
Day	0.355556	0.296621
Week	2.088976	1.718086
Fortnight	3.833132	3.145695
Month	9.516904	7.700457
Year	353.116987	103.592527

Page faults (sum of minor and major):

Duration	list	generator
Minute	1479	1479
Hour	1653	1476
Day	8044	1476
Week	45638	1476
Fortnight	83796	1478
Month	209181	1476
Year	9143068	1491

With a generator, the amount of memory used remains around 4 Mbytes level no matter how much data is thrown at the program, but reading all the data into a list before processing any of it starts at around the same 4 Mbytes and rises to over 3 Gbytes - not really a surprise, since the data file's well over 8 Gbytes in size for the year. You'll notice too how the run time was roughly the same when the data set was small, but when using a list the application became significantly slower than the generator equivalent as the data set got large. In fact, best advice is always use a generator of the data set may get large - if the data set is small, a generator is almost as fast and the process is always quick. But if the data set is large, using a list will be much slower - and that's a slowing down of what's already a long process.

As a final thought ... look at the "page fault" table. Time shared operating systems such as linux run programs in "pages" and load and store pages from memory to allow for effective concurrency. The page fault figure that I've quotes tells me how many times my program has said "oops - I'll have go go off and find that page", and when I'm storing the whole list that's a very large number. With a generator, the figure's virtually unchanging - once the program's in memory it's ticking over sweetly and can carry on even if we throw a decade worth of data at it. I would expect severe problems if I through a decade of data at my list example ...

One of the very first Python Courses I delivered was to a company handling a huge data set, and by changing their lists to generators in existing code we made a huge difference for them. I had delegates from the same organisation on last week's Python course, and they still have very large data sets indeed. Thus this follow up blog, to help them put figures on - at least - a benchmark piece of code and data.

(written 2012-06-16) 25a8

Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles

Y305 - Optimising Python
  [4088] Some tips and techniques for huge data handling in Python - (2013-05-15)
  [2462] Python - how it saves on compile time - (2009-10-20)
  [2369] Using a cache for efficiency. Python and PHP examples - (2009-08-21)
  [2277] Python classes / courses - what version do we train on? - (2009-07-10)

Y105 - Python - Functions, Modules and Packages
  [4029] Exception, Lambda, Generator, Slice, Dict - examples in one Python program - (2013-03-04)
  [3945] vargs in Python - how to call a method with unknown number of parameters - (2012-12-06)
  [3931] Optional positional and named parameters in Python - (2012-11-23)
  [3885] Default local - a good choice by the author of Python - (2012-10-08)
  [3852] Static variables in Python? - (2012-08-29)
  [3695] Functions are first class variables in Lua and Python - (2012-04-13)
  [3662] Finding all the unique lines in a file, using Python or Perl - (2012-03-20)
  [3474] Python Packages - groupings of modules. An introduction - (2011-10-11)
  [3472] Static variables in functions - and better ways using objects - (2011-10-10)
  [3464] Passing optional and named parameters to python methods - (2011-10-04)
  [3459] Catching the fishes first? - (2011-09-27)
  [3280] Passing parameters to Python functions - the options you have - (2011-05-07)
  [3159] Returning multiple values from a function call in various languages - a comparison - (2011-02-06)
  [2998] Using an exception to initialise a static variable in a Python function / method - (2010-10-13)
  [2994] Python - some common questions answered in code examples - (2010-10-10)
  [2929] Passing a variable number of parameters in to a function / method - (2010-08-20)
  [2878] Program for reliability and efficiency - do not duplicate, but rather share and re-use - (2010-07-19)
  [2766] Optional and named parameters to Python functions/methods - (2010-05-15)
  [2718] Python - access to variables in the outer scope - (2010-04-12)
  [2520] Global and Enable - two misused words! - (2009-11-30)
  [2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
  [2481] Sample code with errors in it on our web site - (2009-10-29)
  [2440] Optional parameters to Python functions - (2009-10-07)
  [2439] Multiple returns from a function in Python - (2009-10-06)
  [2011] Conversion of OSI grid references to Eastings and Northings - (2009-01-28)
  [1879] Dynamic code - Python - (2008-11-11)
  [1871] Optional and named parameters in Python - (2008-11-05)
  [1870] What to do with a huge crop of apples - (2008-11-04)
  [1869] Anonymous functions (lambdas) and map in Python - (2008-11-04)
  [1790] Sharing variables with functions, but keeping them local too - Python - (2008-09-09)
  [1784] Global - Tcl, PHP, Python - (2008-09-03)
  [1464] Python Script - easy examples of lots of basics - (2007-12-08)
  [1202] Returning multiple values from a function (Perl, PHP, Python) - (2007-05-24)
  [1163] A better alternative to cutting and pasting code - (2007-04-26)
  [1134] Function / method parameters with * and ** in Python - (2007-04-04)
  [959] It's the 1st, not the 1nd 1rd or 1th. - (2006-12-01)
  [949] Sludge off the mountain, and Python and PHP - (2006-11-27)
  [913] Python - A list of methods - (2006-11-03)
  [912] Recursion in Python - (2006-11-02)
  [900] Python - function v method - (2006-10-20)
  [821] Dynamic functions and names - Python - (2006-08-03)
  [775] Do not duplicate your code - (2006-06-23)
  [749] Cottage industry or production line data handling methods - (2006-06-07)
  [745] Python modules. The distribution, The Cheese Shop and the Vaults of Parnassus. - (2006-06-05)
  [668] Python - block insets help with documentation - (2006-04-04)
  [561] Python's Generator functions - (2006-01-11)
  [418] Difference between import and from in Python - (2005-08-18)
  [386] What is a callback? - (2005-07-22)
  [340] Code and code maintainance efficiency - (2005-06-08)
  [308] Call by name v call by value - (2005-05-11)
  [303] Lambdas in Python - (2005-05-06)
  [294] Python generator functions, lambdas, and iterators - (2005-04-28)
  [105] Distance Learning - (2004-10-31)
  [96] Variable Scope - (2004-10-22)

580b

Back to
Christmas in June? Melksham hotel bookings and Santa train

Previous and next
or
Horse's mouth home

Forward to
How well do you know Melksham?

Some other Articles

Sample answers to training course exercises - available on our web site
Muttable v immutable and implications - Ruby
Melksham Chamber of Commerce - looking to our future shape. Pivotal meeting next Tuesday
How well do you know Melksham?
Python timing - when to use a list, and when to use a generator
Christmas in June? Melksham hotel bookings and Santa train
Shell, Awk, Perl of Python?
Spike solutions and refactoring - a Python example
Learning to program - the if statement. Python.
Melksham - placed 2254 out of 2255. What can be done about it?

4106 posts, page by page

Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83 at 50 posts per page

This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

Public Training Courses

Running regularly at our UK training Centre.
[Schedule] - [About] - [Book]

Like this? ??