Home Accessibility Courses Diary The Mouth Forum Resources Site Map About Us Contact
Python and Tcl - public course schedule [here]
Private courses on your site - see [here]
Please ask about maintenance training for Perl, PHP, Lua, etc
awk - pattern scanning and formatting


awk (and its relatives gawk and nawk) is a pattern scanning and processing language that goes one step beyond grep; not only does it match, but it processes too. This means its a really powerful tool (and quick and easy too) for reformmating and reporting on data files.

Let's see an example. I would like to find all of my people who have Python skills in our sample data file:

[trainee@pasanda shell]$ awk /Python/ requests.txt
hazel PHP Python Perl Ruby ASP
leane PHP Python ASP Perl Java
olivia MySQL Python ASP PHP
adam Tcl/Tk Perl Python MySQL
barry Python XML Java Perl PHP
harry PHP Python Java
ken Tcl/Tk Python Java Perl
nigel PHP Python Java Perl
rupert Java Python MySQL
[trainee@pasanda shell]$

That's done just what a grep would do, but I can add a specification that I want to manipulate my output "just printing the names" I can do so:

[trainee@pasanda shell]$ awk '/Python/{print $1}' requests.txt
[trainee@pasanda shell]$

So that's "match a regular expression, and if it matches output as follows". Awk takes a series of pattern and operation statements and you'll usually wrap them in single quotes to protect them from the shell. Let's do something a bit more interesting:

[trainee@pasanda shell]$ awk 'BEGIN{print "Python Skills"; count=0};
 /Python/{count++; printf "%3d %-10s mainly %s\n",count,$1,$2;}'
Python Skills
  1 hazel mainly PHP
  2 leane mainly PHP
  3 olivia mainly MySQL
  4 adam mainly Tcl/Tk
  5 barry mainly Python
  6 harry mainly PHP
  7 ken mainly Tcl/Tk
  8 nigel mainly PHP
  9 rupert mainly Java
[trainee@pasanda shell]$

Here, the block marked BEGIN is run before any lines of input are read (a block marked END could also have been provided) and then each line of the incoming data is checked against the pattern and a block is performed if it matches. You can specify multiple pattern and match blocks if you wish; awk is a complete language. Rather than write long and complex instructions, you can save your awk program into a file:

[trainee@pasanda shell]$ awk -f demo.awk requests.txt
Python Skills
  1 hazel mainly PHP
  2 leane mainly PHP
  3 olivia mainly MySQL
  4 adam mainly Tcl/Tk
  5 barry mainly Python
  6 harry mainly PHP
  7 ken mainly Tcl/Tk
  8 nigel mainly PHP
  9 rupert mainly Java
Do not consider: antonia barbara cherry delia ethel florence
gloria iris jenny kerry margaret nina petra queenie rita
sally tina uva venus wendy xena yollanda zoe charles david
ed fred graham ivan john len morris orpheus peter quentin
steve tommy ulsyees victor william xavier yuri zachary
9 staff out of 52 have Python skills
[trainee@pasanda shell]$

Here's the complete awk script:

BEGIN{print "Python Skills"; yes=0; no=0; pass=""};
END{print "Do not consider:",pass;}
END{print yes,"staff out of",yes+no,"have Python skills"};
   if (/Python/) {
        printf "%3d %-10s mainly %s\n",yes,$1,$2;
   } else {
        pass = sprintf("%s %s",pass,$1);

See also Linux Basics Course

Please note that articles in this section of our web site were current and correct to the best of our ability when published, but by the nature of our business may go out of date quite quickly. The quoting of a price, contract term or any other information in this area of our website is NOT an offer to supply now on those terms - please check back via our main web site

Related Material

Web Application Deployment - Linux Utilities
  [4682] One line scripts - Awk, Perl and Ruby - (2016-05-20)
  [4586] Extending your bash shell with aliases, functions and extra commands - (2015-11-28)
  [3902] Shell - Grep - Sed - Awk - Perl - Python - which to use when? - (2012-10-22)
  [3764] Shell, Awk, Perl of Python? - (2012-06-14)
  [3446] Awk v Perl - (2011-09-18)
  [2638] Finding what has changed - Linux / Unix - (2010-02-17)
  [2484] Finding text and what surrounds it - contextual grep - (2009-10-30)
  [2320] Helping new arrivals find out about source code examples - (2009-08-03)
  [2145] Using the internet to remotely check for power failure at home (PHP) - (2009-04-29)
  [1690] Conversion of c/r line ends to l/f line ends - (2008-06-28)
  [1366] awk - a powerful data extraction and manipulation tool - (2007-09-25)
  [1361] Korn shell course - (2007-09-22)
  [71] Comparators in Linux and Unix - (2004-10-03)
  [63] Almost like old times - (2004-09-26)

Web Application Deployment - Linux -An Introduction For Users
  [3819] Packing a tar, jar or war file - best practise - (2012-07-26)
  [3791] The Kernel, Shells and Daemons. Greek Gods in computing - (2012-07-01)
  [3256] Displaying a directory or file system tree - Linux - (2011-04-22)
  [3179] Oops - I typed ci not vi, and have lost my file ... - (2011-02-21)
  [2831] Recording (a macro) in vi - (2010-06-27)
  [2636] Linux - useful tips including history and file name completion - (2010-02-15)
  [2494] Making Linux Politically correct - (2009-11-06)
  [2479] Accidentally typed ci rather than vi? - (2009-10-27)
  [2300] What does x on a linux directory mean? - (2009-07-21)
  [2299] How much space does my directory take - Linux - (2009-07-20)
  [2203] Always use su with minus. And where do programs come from? - (2009-05-27)
  [2201] Running straight from the jar, but not from a tar - (2009-05-26)
  [1904] Ruby, Perl, Linux, MySQL - some training notes - (2008-11-23)
  [1902] sstrwxrwxrwx - Unix and Linux file permissions - (2008-11-23)
  [1897] Keeping on an even keel - (2008-11-21)
  [1893] Some Linux and Unix tips - (2008-11-18)
  [1803] FTP passive mode - a sometimes cure for upload hangs - (2008-09-20)
  [1764] Yank and Push - copy and move in vi - (2008-08-21)
  [1651] ls command - favourite options - (2008-05-23)
  [1527] Selecting file names in a shell - one word or another - (2008-02-02)
  [1438] Copy and paste / cut and paste and other vi techniques - (2007-11-20)
  [1408] Wireless hotel tips - FTP and Skype connections failing - (2007-10-26)
  [1366] awk - a powerful data extraction and manipulation tool - (2007-09-25)
  [1288] Linux run states, shell special commands, and directory structures - (2007-08-03)
  [1287] Work and play at Well House Manor - Football and Shell Shortcuts - (2007-08-02)
  [1259] Where am I and how did I get here? - (2007-07-05)
  [1068] ls -l report, Linux / Unix - types and permssions - (2007-02-06)
  [1013] Copy multiple files - confusing error message from cp - (2006-12-30)
  [1012] Moving files between Windows / DOS and Linux / Unix - (2006-12-30)
  [984] Cardinal numbers and magic numbers - (2006-12-14)
  [749] Cottage industry or production line data handling methods - (2006-06-07)
  [711] THE home directory or MY home directory - (2006-05-06)
  [710] Linux training Glasgow, Python programming course Dundee - (2006-05-05)
  [703] Copying files and preserving ownership - (2006-04-28)
  [679] More or less on the edge of the page - (2006-04-11)
  [659] Web Application Components - (2006-03-28)
  [593] Finding where the disc space has gone - (2006-02-06)
  [431] File permissions of Linux and Unix systems - (2005-08-31)
  [430] Linux commands - some basics - (2005-08-31)
  [249] An easy way out - (2005-03-17)
  [152] Aladdin, or careful what you wish. - (2004-12-15)
  [74] pushd and popd - (2004-10-05)
  [73] vi - full circle - (2004-10-04)

resource index - Deployment
Solutions centre home page

You'll find shorter technical items at The Horse's Mouth and delegate's questions answered at the Opentalk forum.

At Well House Consultants, we provide training courses on subjects such as Ruby, Lua, Perl, Python, Linux, C, C++, Tcl/Tk, Tomcat, PHP and MySQL. We're asked (and answer) many questions, and answers to those which are of general interest are published in this area of our site.

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2019: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01225 708225 • FAX: 01225 793803 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/solutions/general- ... tting.html • PAGE BUILT: Wed Mar 28 07:47:11 2012 • BUILD SYSTEM: wizard