Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
20.9.2014 - We have just updated our course layouts and descriptions and added our 2015 schedule.

Cottage industry or production line data handling methods

If you're running a cottage industry, for efficiency's sake you'll run the first process on each of your raw components first, and store the partially-completed elements in a basket as they're processed. When you've completed that first process, you'll then apply the second process to each element in turn, then the third process, and so on. For small scale production, that's a much MUCH better use of your time and resources that setting up all processes in what's probably a very small area and trying to take components through from end to end.

But if your level of throughput is to be several magnitudes greater, the cottage industry approach doesn't work and you'll want, in a bigger area, to set up a production line. You'll have more overheads, as each stage of the line will need someone to operate it, but you'll save on the need to store large numbers of intermediate components, and you'll save on the time spend putting down and picking up components at each stage too.

Data handling flows can resemble a cottage industry or a production line .... and it depends on how much data you have as to which approach will be the most effective.

In Shell Programming, using a series of operations each of which reads from a file, and saves out to a file with > style redirects, is your cottage industry approach. Using a pipe - | is more production line, with a buffer usually of 4k between the processes.

In Python, functions such as range and readlines return complete lists which you subsequently use - cottage industry. Alternatives such as xrange and xreadlines are generators which are running in parallel with their calling code and so are your production line. And you can write your own generator functions; you can spot them in existing code if you see the yield keyword.

As well as using the cottage industry / production line comparison, I also compare the one-at-a-time approach to being akin to filling up a reservoir from one process, then using the reservoir until it's empty from another. If you have a huge amount of data, then you're likely to overflow your reservoir and have your program fail. However, running the processes at the same time is rather like joining them with a pipe, with a tap that is turned on and off each time a new chunk of data is required. This is how I've successfully trained clients who have data files up to 10Gbytes in size to handle their data in easily written Python scripts.
(written 2006-06-07)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
A101 - Web Application Deployment - Linux -An Introduction For Users
  [3819] Packing a tar, jar or war file - best practise - (2012-07-26)
  [3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
  [3256] Displaying a directory or file system tree - Linux - (2011-04-22)
  [3179] Oops - I typed ci not vi, and have lost my file ... - (2011-02-21)
  [2831] Recording (a macro) in vi - (2010-06-27)
  [2636] Linux - useful tips including history and file name completion - (2010-02-15)
  [2494] Making Linux Politically correct - (2009-11-06)
  [2479] Accidentally typed ci rather than vi? - (2009-10-27)
  [2300] What does x on a linux directory mean? - (2009-07-21)
  [2299] How much space does my directory take - Linux - (2009-07-20)
  [2203] Always use su with minus. And where do programs come from? - (2009-05-27)
  [2201] Running straight from the jar, but not from a tar - (2009-05-26)
  [1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
  [1902] sstrwxrwxrwx - Unix and Linux file permissions - (2008-11-23)
  [1897] Keeping on an even keel - (2008-11-21)
  [1893] Some Linux and Unix tips - (2008-11-18)
  [1803] FTP passive mode - a sometimes cure for upload hangs - (2008-09-20)
  [1764] Yank and Push - copy and move in vi - (2008-08-21)
  [1651] ls command - favourite options - (2008-05-23)
  [1527] Selecting file names in a shell - one word or another - (2008-02-02)
  [1438] Copy and paste / cut and paste and other vi techniques - (2007-11-20)
  [1408] Wireless hotel tips - FTP and Skype connections failing - (2007-10-26)
  [1366] awk - a powerful data extraction and manipulation tool - (2007-09-25)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
  [1259] Where am I and how did I get here? - (2007-07-05)
  [1068] ls -l report, Linux / Unix - types and permssions - (2007-02-06)
  [1013] Copy multiple files - confusing error message from cp - (2006-12-30)
  [1012] Moving files between Windows / DOS and Linux / Unix - (2006-12-30)
  [984] Cardinal numbers and magic numbers - (2006-12-14)
  [711] THE home directory or MY home directory - (2006-05-06)
  [710] Linux training Glasgow, Python programming course Dundee - (2006-05-05)
  [703] Copying files and preserving ownership - (2006-04-28)
  [679] More or less on the edge of the page - (2006-04-11)
  [659] Web Application Components - (2006-03-28)
  [593] Finding where the disc space has gone - (2006-02-06)
  [431] File permissions of Linux and Unix systems - (2005-08-31)
  [430] Linux commands - some basics - (2005-08-31)
  [249] An easy way out - (2005-03-17)
  [152] Aladdin, or careful what you wish. - (2004-12-15)
  [74] pushd and popd - (2004-10-05)
  [73] vi - full circle - (2004-10-04)

A167 - Web Application Deployment - Shell Programming (bash)
  [1468] Lexical v Arithemetic testing, Bash and Perl - (2007-12-11)
  [1345] Perl and Shell coding standards / costs of an IT project - (2007-09-11)
  [827] No news is good news with Unix and Linux - (2006-08-10)
  [64] Shell Script for CGI on the web - (2004-09-26)
  [63] Almost like old times - (2004-09-26)

Y105 - Python - Functions, Modules and Packages
  [4212] Python functions - an introduction to how they work - (2013-11-16)
  [4161] Python varables - checking existance, and call by name or by value? - (2013-08-27)
  [4029] Exception, Lambda, Generator, Slice, Dict - examples in one Python program - (2013-03-04)
  [3945] vargs in Python - how to call a method with unknown number of parameters - (2012-12-06)
  [3931] Optional positional and named parameters in Python - (2012-11-23)
  [3885] Default local - a good choice by the author of Python - (2012-10-08)
  [3852] Static variables in Python? - (2012-08-29)
  [3766] Python timing - when to use a list, and when to use a generator - (2012-06-16)
  [3695] Functions are first class variables in Lua and Python - (2012-04-13)
  [3662] Finding all the unique lines in a file, using Python or Perl - (2012-03-20)
  [3474] Python Packages - groupings of modules. An introduction - (2011-10-11)
  [3472] Static variables in functions - and better ways using objects - (2011-10-10)
  [3464] Passing optional and named parameters to python methods - (2011-10-04)
  [3459] Catching the fishes first? - (2011-09-27)
  [3280] Passing parameters to Python functions - the options you have - (2011-05-07)
  [3159] Returning multiple values from a function call in various languages - a comparison - (2011-02-06)
  [2998] Using an exception to initialise a static variable in a Python function / method - (2010-10-13)
  [2994] Python - some common questions answered in code examples - (2010-10-10)
  [2929] Passing a variable number of parameters in to a function / method - (2010-08-20)
  [2878] Program for reliability and efficiency - do not duplicate, but rather share and re-use - (2010-07-19)
  [2766] Optional and named parameters to Python functions/methods - (2010-05-15)
  [2718] Python - access to variables in the outer scope - (2010-04-12)
  [2520] Global and Enable - two misused words! - (2009-11-30)
  [2506] Good example of recursion in Python - analyse an RSS feed - (2009-11-18)
  [2481] Sample code with errors in it on our web site - (2009-10-29)
  [2440] Optional parameters to Python functions - (2009-10-07)
  [2439] Multiple returns from a function in Python - (2009-10-06)
  [2011] Conversion of OSI grid references to Eastings and Northings - (2009-01-28)
  [1879] Dynamic code - Python - (2008-11-11)
  [1871] Optional and named parameters in Python - (2008-11-05)
  [1870] What to do with a huge crop of apples - (2008-11-04)
  [1869] Anonymous functions (lambdas) and map in Python - (2008-11-04)
  [1790] Sharing variables with functions, but keeping them local too - Python - (2008-09-09)
  [1784] Global - Tcl, PHP, Python - (2008-09-03)
  [1464] Python Script - easy examples of lots of basics - (2007-12-08)
  [1202] Returning multiple values from a function (Perl, PHP, Python) - (2007-05-24)
  [1163] A better alternative to cutting and pasting code - (2007-04-26)
  [1134] Function / method parameters with * and ** in Python - (2007-04-04)
  [959] It's the 1st, not the 1nd 1rd or 1th. - (2006-12-01)
  [949] Sludge off the mountain, and Python and PHP - (2006-11-27)
  [913] Python - A list of methods - (2006-11-03)
  [912] Recursion in Python - (2006-11-02)
  [900] Python - function v method - (2006-10-20)
  [821] Dynamic functions and names - Python - (2006-08-03)
  [775] Do not duplicate your code - (2006-06-23)
  [745] Python modules. The distribution, The Cheese Shop and the Vaults of Parnassus. - (2006-06-05)
  [668] Python - block insets help with documentation - (2006-04-04)
  [561] Python's Generator functions - (2006-01-11)
  [418] Difference between import and from in Python - (2005-08-18)
  [386] What is a callback? - (2005-07-22)
  [340] Code and code maintainance efficiency - (2005-06-08)
  [308] Call by name v call by value - (2005-05-11)
  [303] Lambdas in Python - (2005-05-06)
  [294] Python generator functions, lambdas, and iterators - (2005-04-28)
  [105] Distance Learning - (2004-10-31)
  [96] Variable Scope - (2004-10-22)


Back to
Getting rid of variables after you have finished with them
Previous and next
or
Horse's mouth home
Forward to
Almost everyone loses
Some other Articles
Python 3000 - the next generation
Over zealous police activity?
Want to be a technical trainer in the UK?
Almost everyone loses
Cottage industry or production line data handling methods
Getting rid of variables after you have finished with them
The Fag Packet Design Methodology
Domain Listing Center and Domain Registry of America
We can offer a room, but we can't operate on a dog
4292 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2014: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • FAX: 01144 1225 899360 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/749_Cott ... thods.html • PAGE BUILT: Thu Sep 18 15:30:25 2014 • BUILD SYSTEM: WomanWithCat