If it sounds easy, it should be easy.
"Go through a file and echo the lines ... but if a line's duplicated, echo it only once". This was a sample application requested on today's Python course ... but one which has been troubling the folks for a while as they tried to filter the data in other languages.
The solution? Set up a dictionary, and as each line is read pretend that the whole line is a key ... and see if the key exists (i.e. if the line occurred previously). If it didn't, then output the line and add a member to the dictionary to indicate that you've done so - keyed on the line, and with a value of "1" which we're not really interested in - we're just looking at the presence or absence of a member. Source code [here]
and it's very short!
Such a use of dictionaries has provided a big leap forward for this application. What was previously a nighmare task is now just a few lines of code, and the hashing algorithm used by the dictionary means that it's going to be efficient even as the data set grows.
But there are lots of other ways that dictionaries can be used in this "unique line" application - just as there are lots of ways to get across the water once you find the narrowest and easiest crossing point ...
is the application expanded to give a count of the number of times each line occurs, sorted by the number of occurrences.
is an exended report, telling you all the line numbers on which a particular string occurs (makes up the line)
is the same report, but sorted by the number of occurrences of each line - so you have all the unique lines together, for example.
Oh - the sample data is [here]
Further pictures at St Budeaux / the Saltash ferry ...
Associated topics are indexed underY107 - Python - Dictionaries 
Exception, Lambda, Generator, Slice, Dict - examples in one Python program - (2013-03-04) 
Collections in Python - list tuple dict and string. - (2013-03-04) 
Multiple identical keys in a Python dict - yes, you can! - (2012-11-24) 
Finding all the unique lines in a file, using Python or Perl - (2012-03-20) 
Football league tables - under old and new point system. Python program. - (2011-12-18) 
Learning more about our web site - and learning how to learn about yours - (2011-12-17) 
Python sets and frozensets - what are they? - (2011-10-20) 
Passing optional and named parameters to python methods - (2011-10-04) 
Python - some common questions answered in code examples - (2010-10-10) 
Looking up a value by key - associative arrays / Hashes / Dictionaries - (2010-08-11) 
Python - fresh examples of all the fundamentals - (2009-08-20) 
Using a list of keys and a list of values to make a dictionary in Python - zip - (2007-04-13) 
Python dictionary for quick look ups - (2007-04-12) 
Python collections - mutable and imutable - (2006-11-29) 
Can't resist writing about Python - (2004-10-29)
Some other Articles
What are numpy and scipy?Oddballs in PlymouthNot mugged in London!Memorial to a day in 1999Python dictionaries - reaching to new usesRight place, right seasonCustomer Review - Hotel Room, MelkshamStrong arm tactics in the bus industry? Poor result for the customer!Travel, Transport and Tourism - A vision for Melksham, 2026How to set up short and meaningfull alternative URLs