Looks like large text files are "flavour of the week".

I was set the task of extracting all the records from a file containing some 13 million records that matched any one of some 200 different character strings in a certain field in the record.   It's not an uncommon sort of requirement, but it's not something that appears to be trivial when you first see it.  Do you sort the incoming file?  No way!  Do you traverse the incoming file 200 times? Preferably not!  Do you keep a list of all the strings you want to find, and loop through that list for each input line from the file?  No - that would be too slow too.

Solution - set up a hash which has keys for each of the strings you want to find - set the values to any true value that you like.   You can then parse your incoming data file and simply check whether a hash element exists to see if you want a record to be echoed on the output.

Here's a sample program that I wrote for filtering web site addresses and reporting on those hosted on certain shared servers:
Code:

# Set up a hash describing the records we want %wanted = (        "members.aol.com" => 1,        "www.geocities.com" => 1,        "groups.yahoo.com" => 1,        "home.earthlink.net" => 1        ); # Parse the incoming data looking for matches while ($line = <>) {        ($server) = ($line =~ m!//(.*?)/!);        if ($wanted{$server}) {                print $line;                }        }

To give you an idea, filtering a file with 3 million Urls in it (113 Mbytes), this program took 2 minutes on one of my training laptops to produce me an output file of 5 Mbytes containing around 113000 Urls.

---