Using a generator to process a big file

Functions, Modules and Packages example from a Well House Consultants training course
More on Functions, Modules and Packages [link]

This example is described in the following article(s):
• What to do with a huge crop of apples - [link]

Source code: dfg Module: Y105

"""

Last year, you had a good crop of apples on your tree .... what
did you do with them? Make apple pies for all the neighbours! How?

You collect all the apples and bring them into the kitchen.
You prepare the apples ready for the pies.
You make the pastry and apple pies.
You cook the pies
You distribute them.

This year, you had a huge crop of apples, and you wanted to make
apple pies ... not for all the neighbours, but for the whole
town. How did you do that? The same way that you did last
year? Alas, no. There were far too many appled to bring into
the kitchen, all your bowls would have been filled with apples
long before you had prepared them all, and your oven has only
got limited capacity. Far better to collect a few, prepare a
few, make a few pies, and distribute them ... and keep repeating
that until you have fed the whole town.

Which is what a generator function does in Python. When
you call a generator function (as the list provider in a for
statement) in Python, it will yield when it has the first result
available, which you can then process. Come back to the for
statement, and it will continue to run the generator code
('pick the next batch of apples') and return the next result,
time after time, until you're done.

Here's an example that's doing just this - opening a file in
a generator function, and returning at each call the next line
containing the word "Python". The loop in the main code is
then extracting the person's name from the line, printing it
out, and going back to the generator for the next "Python" line.

"""

def tap(file):
        fh = open (file,"r")
        # for line in fh.xreadlines():
        for line in fh.readlines(): # Use previous line Python 2
                if (line.find("Python") > 0):
                        yield line

print(tap("requests.xyz"))
for staff in tap("requests.xyz"):
        els = staff.split(" ")
        print(els[0])

""" Sample output:

<generator object tap at 0x100636630>
hazel
leane
olivia
adam
barry
harry
ken
nigel
rupert
WomanWithCat:y105 grahamellis$ python2 dfg
<generator object tap at 0x10b8d3d70>
hazel
leane
olivia
adam
barry
harry
ken
nigel
rupert
WomanWithCat:y105 grahamellis$

"""

Learn about this subject

This module and example are covered on the following public courses:
* Learning to program in Python
* Python Programming
Also available on on site courses for larger groups

Books covering this topic

Yes. We have over 700 books in our library. Books covering Python are listed here and when you've selected a relevant book we'll link you on to Amazon to order.

Other Examples

This example comes from our "Functions, Modules and Packages" training module. You'll find a description of the topic and some other closely related examples on the "Functions, Modules and Packages" module index page.

Full description of the source code

You can learn more about this example on the training courses listed on this page, on which you'll be given a full set of training notes.

Many other training modules are available for download (for limited use) from our download centre under an Open Training Notes License.

Other resources

• Our Solutions centre provides a number of longer technical articles.
• Our Opentalk forum archive provides a question and answer centre.
• The Horse's mouth provides a daily tip or thought.
• Further resources are available via the resources centre.
• All of these resources can be searched through through our search engine
• And there's a global index here.

Web site author

This web site is written and maintained by Well House Consultants.

Purpose of this website

This is a sample program, class demonstration or answer from a training course. It's main purpose is to provide an after-course service to customers who have attended our public private or on site courses, but the examples are made generally available under conditions described below.

Conditions of use

Past attendees on our training courses are welcome to use individual examples in the course of their programming, but must check the examples they use to ensure that they are suitable for their job. Remember that some of our examples show you how not to do things - check in your notes. Well House Consultants take no responsibility for the suitability of these example programs to customer's needs.

This program is copyright Well House Consultants Ltd. You are forbidden from using it for running your own training courses without our prior written permission. See our page on courseware provision for more details.

Any of our images within this code may NOT be reused on a public URL without our prior permission. For Bona Fide personal use, we will often grant you permission provided that you provide a link back. Commercial use on a website will incur a license fee for each image used - details on request.