"Data Monging" is a term that has come to be used for processing quantities of data - reformatting, extraction, etc. It's really what Perl's ALL about - the language has a number of features which make it especially good for the purpose. In this module, we highlight one or two of the more specialist of these features.


So you want to do the same thing to every element of an array?

In traditional languages which are not so full-featured as Perl, you'll use a loop, with a keyword such as "while" or "for", and a variable that steps up from 0 or 1 to the length of the array. You can do the same sort of thing in Perl as well if you wish:


Which gives:


Although you can use the traditional approach in Perl, there are other approaches too, which are often easier to code and more efficient at run time. You should remember that languages like C and Fortran had ARRAYS which were basic containers for a whole lot of variables all of the same type, whereas Perl uses LISTS which are much more flexible structures upon which operations can be performed in their own rights.

Have a look at these two, both of which perform exactly the same as the example above:



= for (or foreach) can be used as an iterator to pass through each element of a list, performing a statement or block on each.

= map iterates through each element of a list, performing a statement on each and returning the result of each statement into a new list

= grep iterates through each element of a list, performing a test on each and returns a list of all the elements for which a true value was returned by the tests.

Here's an example program that generates a list (of file names) and then user map, grep and for to modify, select and iterate through that list and its derivatives:


in operation:


To help you understand what's happening, we wrote that example to create a number of temporary lists, using a single map or grep in each Perl statement. The code could be shortened and will run more efficiently:



You may be used to using Perl's regular expressions to match patterns and extract matches from a line of text, perhaps stepping through all the lines of a file. Have you ever thought of using it on the whole contents of a file at one go? You can do so, provided you're sure that the file won't be so large that you'll fill your computer's memory / swap space.

Here's and example that reads a UK postcode file, and prints out in reverse order the names of all postal towns that come under the main Aberdeen office:




In all cases, the results look like this:



See also Perl for Larger Projects

Please note that articles in this section of our web site were current and correct to the best of our ability when published, but by the nature of our business may go out of date quite quickly. The quoting of a price, contract term or any other information in this area of our website is NOT an offer to supply now on those terms - please check back via our main web site

Perl - Lists
  [2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
  [2295] The dog is not in trouble - (2009-07-17)
  [2226] Revision / Summary of lists - Perl - (2009-06-10)
  [2067] Perl - lists do so much more than arrays - (2009-03-05)
  [1918] Perl Socket Programming Examples - (2008-12-02)
  [1917] Out of memory during array extend - Perl - (2008-12-02)
  [1828] Perl - map to process every member of a list (array) - (2008-10-09)
  [1703] Perl ... adding to a list - end, middle, start - (2008-07-09)
  [1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
  [1304] Last elements in a Perl or Python list - (2007-08-16)
  [968] Perl - a list or a hash? - (2006-12-06)
  [928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
  [773] Breaking bread - (2006-06-22)
  [762] Huge data files - what happened earlier? - (2006-06-15)
  [622] Queues and barrel rolls in Perl - (2006-02-24)
  [560] The fencepost problem - (2006-01-10)
  [463] Splitting the difference - (2005-10-13)
  [355] Context in Perl - (2005-06-22)
  [240] Conventional restraints removed - (2005-03-09)
  [230] Course sizes - beware of marketing statistics - (2005-02-27)
  [140] Comparison Chart for Perl programmers - list functions - (2004-12-04)
  [28] Perl for breakfast - (2004-08-25)

Perl - Handling Huge Data
  [2376] Long job - progress bar techniques (Perl) - (2009-08-26)
  [1924] Preventing ^C stopping / killing a program - Perl - (2008-12-05)
  [1920] Progress Bar Techniques - Perl - (2008-12-03)
  [1397] Perl - progress bar, supressing ^C and coping with huge data flows - (2007-10-20)
  [975] Answering ALL the delegate's Perl questions - (2006-12-09)
  [762] Huge data files - what happened earlier? - (2006-06-15)
  [639] Progress bars and other dynamic reports - (2006-03-09)

Perl - Data Munging
  [2129] Nothing beats Perl to solve a data manipulation requirement quickly - (2009-04-14)
  [1947] Perl substitute - the e modifier - (2008-12-16)
  [1509] Extracting information from a file of records - (2008-01-16)
  [1316] Filtering and altering Perl lists with grep and map - (2007-08-23)
  [597] Storing a regular expression in a perl variable - (2006-02-09)

resource index - Perl
Solutions centre home page

You'll find shorter technical items at The Horse's Mouth and delegate's questions answered at the Opentalk forum.

At Well House Consultants, we provide training courses on subjects such as Ruby, Lua, Perl, Python, Linux, C, C++, Tcl/Tk, Tomcat, PHP and MySQL. We're asked (and answer) many questions, and answers to those which are of general interest are published in this area of our site.