If you want a program to run quickly through a data set (that's the sort of thing you'll be doing in heavy scientific work), you'll want the data loaded into successive memory locations - but that means that you have to know how much space to allocate before you set the data up. Otherwise, you'll find that you're overriding data.

If you don't know how much data you'll have to work with in a program, so you cannot set up sequential memory locations, you'll have to use a scheme where each block of data includes a pointer to the next block so that you can keep adding blocks of data even if you've got the first part(s) loaded somewhere on the heap that can't be easily expanded.

Data stored in sequential / successive memory locations, and where each element of the data takes up the same amount of storage, is known to computer scientists as an ARRAY. Data stored with forward and backward pointers is known to them as a LIST. Unfortuatey, the terms have been muddied by the authors of programming languages, so the term "array" is or was used for a list in some languages ... and indeed Tcl's Arrays are a different type of collection all together!

Python uses lists. It calls them lists, and they ARE linked lists. A Tuple is a similar (but lighterweight) structure to a list, which is more efficient to acccess but lacks alteration facilities.

If you want to use the power of Python's scripting language to handle heavy scientific data, you can do so through the numpy module which you can download from the Scipy site - [here]; numpy supports true arrays, and also the basic data types of C, in a nice Python wrapper, giving you the best of both worlds.

But there's still the intrinsic problem that you have to know how much data you'll be saving into successive memory locations before you start to fill them ... and there are two ways of solving this:
1: by using an internal staging area to hold the data as you load it all in, then transferring it all to an array once you know who big an array to create
2: by looking ahead and working out how big the data set will be - typically by looking at the size of the incoming file.

There are examples that use these techniques on our web site - and I've added a couple that go along those lines to the numpy section of the site this morning, based on examples written for the course that I concluded on Friday. You'll also see from them how numpy may be used to read in binary data.

[source] - numpy loading an array
[source] - numpy loading a list of arrays / checking a file size


I have used my avatar (shown here) as the example file to load a 2 byte integers; if you look at the data shown in the source examples, you'll see that the 4th and 5th values are 72 and 73, showing that the image is 72 x 73 pixels. Of course, you do have to understand the format of an image file to make use of things in this way. Further details (in Perl) [here].
(written 2010-10-10)

 Y118 - Python - numpy, scipy and matplotlib
  [2990] What are numpy and scipy? - (2010-10-09)
  [2991] Loading and saving data - Python / numpy - (2010-10-09)
  [2992] Matplotlib - graphing in Python - teaching examples - (2010-10-10)
  [2997] 3D graphics - web site usage - simple matplotlib and python example - (2010-10-12)
  [3554] Learning more about our web site - and learning how to learn about yours - (2011-12-17)
  [4440] A first graph with Matplotlib in Python - (2015-02-22)
  [4445] Graphing presentations in Python - huge data, numpy and matplotlib - (2015-02-28)

Q110 - Object Orientation and General technical topics - Programming Algorithms
  [202] Searching for numbers - (2005-02-04)
  [227] Bellringing and Programming and Objects and Perl - (2005-02-25)
  [642] How similar are two words - (2006-03-11)
  [1157] Speed Networking - a great evening and how we arranged it - (2007-04-21)
  [1187] Updating a page strictly every minute (PHP, Perl) - (2007-05-14)
  [1391] Ordnance Survey Grid Reference to Latitude / Longitude - (2007-10-14)
  [1840] Validating Credit Card Numbers - (2008-10-14)
  [1949] Nuclear Physics comes to our web site - (2008-12-17)
  [2189] Matching disparate referencing systems (MediaWiki, PHP, also Tcl) - (2009-05-19)
  [2259] Grouping rows for a summary report - MySQL and PHP - (2009-06-27)
  [2509] A life lesson from the accuracy of numbers in Excel and Lua - (2009-11-21)
  [2586] And and Or illustrated by locks - (2010-01-17)
  [2617] Comparing floating point numbers - a word of caution and a solution - (2010-02-01)
  [2894] Sorting people by their names - (2010-07-29)
  [2951] Lots of way of converting 3 letter month abbreviations to numbers - (2010-09-10)
  [3042] Least Common Ancestor - what is it, and a Least Common Ancestor algorithm implemented in Perl - (2010-11-11)
  [3072] Finding elements common to many lists / arrays - (2010-11-26)
  [3093] How many toilet rolls - hotel inventory and useage - (2010-12-18)
  [3102] AND and OR operators - what is the difference between logical and bitwise varieties? - (2010-12-24)
  [3451] Why would you want to use a Perl hash? - (2011-09-20)
  [3620] Finding the total, average, minimum and maximum in a program - (2012-02-22)
  [3662] Finding all the unique lines in a file, using Python or Perl - (2012-03-20)
  [4325] Learning to program - what are algorithms and design patterns? - (2014-11-22)
  [4401] Selecting RECENT and POPULAR news and trends for your web site users - (2015-01-19)
  [4402] Finding sum, minimum, maximum and average in Python (and Ruby) - (2015-01-19)
  [4410] A good example of recursion - a real use in Python - (2015-02-01)
  [4652] Testing new algorithms in PHP - (2016-02-20)
  [4656] Identifying the first and last records in a sequence - (2016-02-26)
  [4707] Some gems from an introduction to Python - (2016-10-29)

P212 - Perl - More on Character Strings
  [453] Commenting Perl regular expressions - (2005-09-30)
  [583] Remember to process blank lines - (2006-01-31)
  [586] Perl Regular Expressions - finding the position and length of the match - (2006-02-02)
  [597] Storing a regular expression in a perl variable - (2006-02-09)
  [608] Don't expose your regular expressions - (2006-02-15)
  [737] Coloured text in a terminal from Perl - (2006-05-29)
  [928] C++ and Perl - why did they do it THAT way? - (2006-11-16)
  [943] Matching within multiline strings, and ignoring case in regular expressions - (2006-11-25)
  [1222] Perl, the substitute operator s - (2007-06-08)
  [1230] Commenting a Perl Regular Expression - (2007-06-12)
  [1251] Substitute operator / modifiers in Perl - (2007-06-28)
  [1305] Regular expressions made easy - building from components - (2007-08-16)
  [1336] Ignore case in Regular Expression - (2007-09-08)
  [1510] Handling Binary data (.gif file example) in Perl - (2008-01-17)
  [1727] Equality and looks like tests - Perl - (2008-07-29)
  [1735] Finding words and work boundaries (MySQL, Perl, PHP) - (2008-08-03)
  [1947] Perl substitute - the e modifier - (2008-12-16)
  [2230] Running a piece of code is like drinking a pint of beer - (2009-06-11)
  [2379] Making variables persistant, pretending a database is a variable and other Perl tricks - (2009-08-27)
  [2657] Want to do a big batch edit? Nothing beats Perl! - (2010-03-01)
  [2801] Binary data handling with unpack in Perl - (2010-06-10)
  [2834] Teaching examples in Perl - third and final part - (2010-06-27)
  [2874] Unpacking a Perl string into a list - (2010-07-16)
  [2877] Further more advanced Perl examples - (2010-07-19)
  [3059] Object Orientation in an hour and other Perl Lectures - (2010-11-18)
  [3100] Looking ahead and behind in Regular Expressions - double matching - (2010-12-23)
  [3322] How much has Perl (and other languages) changed? - (2011-06-10)
  [3332] DNA to Amino Acid - a sample Perl script - (2011-06-24)
  [3411] Single and double quotes strings in Perl - what is the difference? - (2011-08-30)
  [3546] The difference between dot (a.k.a. full stop, period) and comma in Perl - (2011-12-09)
  [3630] Serialsing and unserialising data for storage and transfer in Perl - (2012-02-28)
  [3650] Possessive Regular Expression Matching - Perl, Objective C and some other languages - (2012-03-12)
  [3707] Converting codons via Amino Acids to Proteins in Perl - (2012-04-25)
  [3927] First match or all matches? Perl Regular Expressions - (2012-11-19)
  [4452] Binary data handling - Python and Perl - (2015-03-09)

Back to Matplotlib - graphing in Python - teaching examples

Previous and next or Horse's mouth home

Forward to Python - some common questions answered in code examples

Copying - duplicating data, or just adding a name? Perl and Python compared
A river in Melksham is not just for boaters.
Python - some common questions answered in code examples
Arrays v Lists - what is the difference, why use one or the other
Oddballs in Plymouth
Not mugged in London!