If you're going to cook dinner for the family, it's much more efficient to cook everyone's meal together. If you're a family of four, you won't want to cook four separate pans of carrots, but rather you'll want to cook them all together. But now imagine you're holding a carrot street party, and providing carrots through the day for four hundred rather than four people. You could hire in (or purchase) special resources, but you're much more likely to adopt some sort of batch system. You won't cook all the carrots at once - you're pans aren't big enough, and even if they were, you've not got the capacity to store all of the cooked carrots until they're needed for serving.
The same thing applies when you're programming. In a program with a small amount of data, it's quicker and easier to handle it all via a list, but as the data size grows so would the amount of memory you need for a list, and it becomes quicker to handle the data as it becomes available - using a production line type approach known as an iterator or a generator.
Grand theory - how does that work in practise? In a language like Python, you can use a generator function. I've written about those before (examples HERE and HERE if you want to refer back). In Lua, you would use co-routines, and in any other language you could roll your own. But the question remains
"Does it really make a difference?", and I thought I would try it out to see.
Let's look at the tools I'll use:
• Seven log files off our server, for periods from a minute (43 accesses) to a year (nearly 30 million accesses):
munchkin:pyjun12 grahamellis$ wc -l ac[a-z]* | sort -bn
43 acminute
2671 achour
79015 acday
533818 acweek
992112 acfortnight
2481275 acmonth
29775300 acyear
• Two programs
dave - which stores a list and
jenny - which processes each record as it reads it, using a generator. Both programs produce identical output files when run on the same input; I've checked this using the Unix
diff command.
• Python's
resource module, which is supplied with the standard Unix / Linux distributions, but needs to be imported into your program. See
python documentation for full details.
The code I have added at the end of both dave and jenny, to report on resources, is as follows:
  import resource
  used = resource.getrusage(resource.RUSAGE_SELF)
  print "Maximum resident size:",used.ru_maxrss
  print "Run time:",used.ru_utime + used.ru_stime
  print "Page faults:",used.ru_minflt + used.ru_majflt
So - how does that run?
Maximum memory size while running:
Duration | list | generator |
Minute | 4116480 | 4112384 |
Hour | 4837376 | 4108288 |
Day | 30355456 | 4108288 |
Week | 181055488 | 4108288 |
Fortnight | 333402112 | 4112384 |
Month | 837042176 | 4108288 |
Year | 3351420928 | 4096000 |
Run time (sum of system and user time):
Duration | list | generator |
Minute | 0.055345 | 0.056894 |
Hour | 0.064015 | 0.062101 |
Day | 0.355556 | 0.296621 |
Week | 2.088976 | 1.718086 |
Fortnight | 3.833132 | 3.145695 |
Month | 9.516904 | 7.700457 |
Year | 353.116987 | 103.592527 |
Page faults (sum of minor and major):
Duration | list | generator |
Minute | 1479 | 1479 |
Hour | 1653 | 1476 |
Day | 8044 | 1476 |
Week | 45638 | 1476 |
Fortnight | 83796 | 1478 |
Month | 209181 | 1476 |
Year | 9143068 | 1491 |
With a generator, the amount of memory used remains around 4 Mbytes level no matter how much data is thrown at the program, but reading all the data into a list before processing any of it starts at around the same 4 Mbytes and rises to over 3 Gbytes - not really a surprise, since the data file's well over 8 Gbytes in size for the year. You'll notice too how the run time was roughly the same when the data set was small, but when using a list the application became significantly slower than the generator equivalent as the data set got large. In fact, best advice is
always use a generator of the data set may get large - if the data set is small, a generator is almost as fast and the process is always quick. But if the data set is large, using a list will be much slower - and that's a slowing down of what's already a long process.
As a final thought ... look at the "page fault" table. Time shared operating systems such as linux run programs in "pages" and load and store pages from memory to allow for effective concurrency. The page fault figure that I've quotes tells me how many times my program has said "oops - I'll have go go off and find that page", and when I'm storing the whole list that's a very large number. With a generator, the figure's virtually unchanging - once the program's in memory it's ticking over sweetly and can carry on even if we throw a decade worth of data at it. I would expect severe problems if I through a decade of data at my list example ...
One of the very first
Python Courses I delivered was to a company handling a huge data set, and by changing their lists to generators in existing code we made a huge difference for them. I had delegates from the same organisation on last week's Python course, and they still have very large data sets indeed. Thus this follow up blog, to help them put figures on - at least - a benchmark piece of code and data.
(written 2012-06-16)
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
Y305 - Optimising Python [2277] Python classes / courses - what version do we train on? - (2009-07-10)
[2369] Using a cache for efficiency. Python and PHP examples - (2009-08-21)
[2462] Python - how it saves on compile time - (2009-10-20)
[4088] Some tips and techniques for huge data handling in Python - (2013-05-15)
Y105 - Python - Functions, Modules and Packages [96] Variable Scope - (2004-10-22)
[105] Distance Learning - (2004-10-31)
[294] Python generator functions, lambdas, and iterators - (2005-04-28)
[303] Lambdas in Python - (2005-05-06)
[308] Call by name v call by value - (2005-05-11)
[340] Code and code maintainance efficiency - (2005-06-08)
[386] What is a callback? - (2005-07-22)
[418] Difference between import and from in Python - (2005-08-18)
[561] Python's Generator functions - (2006-01-11)
[668] Python - block insets help with documentation - (2006-04-04)
[745] Python modules. The distribution, The Cheese Shop and the Vaults of Parnassus. - (2006-06-05)
[749] Cottage industry or production line data handling methods - (2006-06-07)
[775] Do not duplicate your code - (2006-06-23)
[821] Dynamic functions and names - Python - (2006-08-03)
[900] Python - function v method - (2006-10-20)
[912] Recursion in Python - (2006-11-02)
[913] Python - A list of methods - (2006-11-03)
[949] Sludge off the mountain, and Python and PHP - (2006-11-27)
[959] It's the 1st, not the 1nd 1rd or 1th. - (2006-12-01)
[1134] Function / method parameters with * and ** in Python - (2007-04-04)
[1163] A better alternative to cutting and pasting code - (2007-04-26)
[1202] Returning multiple values from a function (Perl, PHP, Python) - (2007-05-24)
[1464] Python Script - easy examples of lots of basics - (2007-12-08)
[1784] Global - Tcl, PHP, Python - (2008-09-03)
[1790] Sharing variables with functions, but keeping them local too - Python - (2008-09-09)
[1869] Anonymous functions (lambdas) and map in Python - (2008-11-04)
[1870] What to do with a huge crop of apples - (2008-11-04)
[1871] Optional and named parameters in Python - (2008-11-05)
[1879] Dynamic code - Python - (2008-11-11)
[2011] Conversion of OSI grid references to Eastings and Northings - (2009-01-28)
[2439] Multiple returns from a function in Python - (2009-10-06)
[2440] Optional parameters to Python functions - (2009-10-07)
[2481] Sample code with errors in it on our web site - (2009-10-29)
[2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
[2520] Global and Enable - two misused words! - (2009-11-30)
[2718] Python - access to variables in the outer scope - (2010-04-12)
[2766] Optional and named parameters to Python functions/methods - (2010-05-15)
[2878] Program for reliability and efficiency - do not duplicate, but rather share and re-use - (2010-07-19)
[2929] Passing a variable number of parameters in to a function / method - (2010-08-20)
[2994] Python - some common questions answered in code examples - (2010-10-10)
[2998] Using an exception to initialise a static variable in a Python function / method - (2010-10-13)
[3159] Returning multiple values from a function call in various languages - a comparison - (2011-02-06)
[3280] Passing parameters to Python functions - the options you have - (2011-05-07)
[3459] Catching the fishes first? - (2011-09-27)
[3464] Passing optional and named parameters to python methods - (2011-10-04)
[3472] Static variables in functions - and better ways using objects - (2011-10-10)
[3474] Python Packages - groupings of modules. An introduction - (2011-10-11)
[3662] Finding all the unique lines in a file, using Python or Perl - (2012-03-20)
[3695] Functions are first class variables in Lua and Python - (2012-04-13)
[3852] Static variables in Python? - (2012-08-29)
[3885] Default local - a good choice by the author of Python - (2012-10-08)
[3931] Optional positional and named parameters in Python - (2012-11-23)
[3945] vargs in Python - how to call a method with unknown number of parameters - (2012-12-06)
[4029] Exception, Lambda, Generator, Slice, Dict - examples in one Python program - (2013-03-04)
[4161] Python varables - checking existance, and call by name or by value? - (2013-08-27)
[4212] Python functions - an introduction to how they work - (2013-11-16)
[4361] Multiple yields and no loops in a Python generator? - (2014-12-22)
[4407] Python - even named code blocks are objects - (2015-01-28)
[4410] A good example of recursion - a real use in Python - (2015-02-01)
[4441] Reading command line parameters in Python - (2015-02-23)
[4448] What is the difference between a function and a method? - (2015-03-04)
[4645] What are callbacks? Why use them? An example in Python - (2016-02-11)
[4662] Recursion in Python - the classic example - (2016-03-07)
[4719] Nesting decorators - (2016-11-02)
[4722] Embedding more complex code into a named block - (2016-11-04)
[4724] From and Import in Python - where is the module loaded from? - (2016-11-06)
Some other Articles
Sample answers to training course exercises - available on our web siteMuttable v immutable and implications - RubyMelksham Chamber of Commerce - looking to our future shape. Pivotal meeting next TuesdayHow well do you know Melksham?Python timing - when to use a list, and when to use a generatorChristmas in June? Melksham hotel bookings and Santa train Shell, Awk, Perl of Python?Spike solutions and refactoring - a Python exampleLearning to program - the if statement. Python.Melksham - placed 2254 out of 2255. What can be done about it?