Home Accessibility Courses Twitter The Mouth Facebook Resources Site Map About Us Contact
 
For 2023 (and 2024 ...) - we are now fully retired from IT training.
We have made many, many friends over 25 years of teaching about Python, Tcl, Perl, PHP, Lua, Java, C and C++ - and MySQL, Linux and Solaris/SunOS too. Our training notes are now very much out of date, but due to upward compatability most of our examples remain operational and even relevant ad you are welcome to make us if them "as seen" and at your own risk.

Lisa and I (Graham) now live in what was our training centre in Melksham - happy to meet with former delegates here - but do check ahead before coming round. We are far from inactive - rather, enjoying the times that we are retired but still healthy enough in mind and body to be active!

I am also active in many other area and still look after a lot of web sites - you can find an index ((here))
Making a Lua program run more than 10 times faster

I made an extraordinary claim on Thursday. One of my delegates had written a program that analyzed a huge data file in a Lua script, and it was taking over half an hour to run. "Give me five minutes, and I could have that running in less that five minutes" I claimed.

It was all to do with pattern matching in Lua ... and I was pretty sure of myself, but my delegates were less sure of the claim. But it didn't really matter - it would have been a diversion from the main course if we had sidetracked to test my assertion.

But when I make a claim, and that claim is doubted, I worry. Was my claim reasonable, or wash it brash and overoptimistic? Well - how better to find out than to try it.

Try it, I did ... the original takes 3283 c.p.u. seconds to run - that's analyzing a 43 Mbyte log file looking for search engine arrivals, and extracting the search terms that were used.

In the original, the lookup was simple on each line:
  string.find(line,'(%S+).*"http.*%.google%..*[&?]q=(.-)[&"]')
in other words, grab the first value on the line, then look for the referer, check if it's google, and if it is grab the query string.

It works - well - but s-l-o-w-l-y.

Let's try to speed that up. Rather than analyse every line starting with wild cared groups of indefinite size (which are a VERY fuzzy, so slow) match, let's start off by looking specifically for the google reference:
  string.find(line,'%.google%.')
if that works, lets grab the first field off the line:
  string.find(line,'(%S+)')
and also the query string:
  string.find(line,'"http.-[&?]q=(.-)[&"]')

Now your natural reaction may be to say "that code is longer and more complicated" - and indeed it is - "so it will take longer", but it turns out THAT conclusion is utterly incorrect.

The original took 3283 seconds. The modified code took (bang the symbol, roll the drums) ... 3.25 seconds. That's right - my claim that I could reduce the time from 30 minutes to 5 minutes was dreadfully conservative - I actually got it down to less that 5 seconds, or put another way it now runs 1000 times faster.




Although this example is written with Lua's pattern matching as the examples, exactly the same principles apply to regular expressions in Perl, PHP, Python, Ruby, Tcl ... etc.

• Start matches with anchors and literals if possible
• Try to avoid too many long woolly matches such as .+
• Eliminate lines / cases quickly if you can
• Don't try to do it all in one long pattern - several shorter ones are often much faster.

Source code of slow version - [here]
Source code of fast version - [here]

(written 2010-04-16, updated 2010-05-14)

 
Associated topics are indexed as below, or enter http://melksh.am/nnnn for individual articles
U108 - Lua - Pattern matching
  [1744] Lua examples, Lua Courses - (2008-08-08)
  [1847] Lua - IAQ (Infrequently Answered Questions) - (2008-10-18)
  [2383] Lua Regular Expressions - (2009-08-28)
  [2702] First and last match with Regular Expressions - (2010-04-02)
  [3687] Binary / bitwise operations in Lua with the standard bit32 library - (2012-04-06)
  [4366] Changing what operators do on objects - a comparison across different programming languages - (2014-12-26)

Q806 - Regular Expression Cookbook
  [672] Keeping your regular expressions simple - (2006-04-05)
  [1230] Commenting a Perl Regular Expression - (2007-06-12)
  [1305] Regular expressions made easy - building from components - (2007-08-16)
  [1840] Validating Credit Card Numbers - (2008-10-14)
  [2165] Making Regular Expressions easy to read and maintain - (2009-05-10)
  [2563] Efficient debugging of regular expressions - (2010-01-04)
  [2608] Search and replace in Ruby - Ruby Regular Expressions - (2010-01-31)
  [2804] Regular Expression Myths - (2010-06-13)
  [3218] Matching a license plate or product code - Regular Expressions - (2011-03-28)
  [3788] Getting more than a yes / no answer from a regular expression pattern match - (2012-06-30)

Q804 - Object Orientation and General technical topics - Regular Expression Internals
  [1480] Next course - 7th January 2008, Regular Expressions - (2007-12-21)
  [2806] Macho matching - do not do it! - (2010-06-13)
  [3090] Matching to a string - what if it matches in many possible ways? - (2010-12-17)
  [3091] How do regular expressions work / Regular Expression diagrams - (2010-12-17)


Back to
Every cloud has a silver lining
Previous and next
or
Horse's mouth home
Forward to
Redirecting a home page using mod_rewrite
Some other Articles
Training Courses - time of change
Bird Watching on the Melksham Avon
Uploading a document or image to its own URL via a browser
Redirecting a home page using mod_rewrite
Making a Lua program run more than 10 times faster
Every cloud has a silver lining
How do our tactics help us meet the strategy, for the greater good?
Escapee Grandparents welcome - Henfield and Melksham
Returning multiple values from a function - Lua
Mixins example in Python
4759 posts, page by page
Link to page ... 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96 at 50 posts per page


This is a page archived from The Horse's Mouth at http://www.wellho.net/horse/ - the diary and writings of Graham Ellis. Every attempt was made to provide current information at the time the page was written, but things do move forward in our business - new software releases, price changes, new techniques. Please check back via our main site for current courses, prices, versions, etc - any mention of a price in "The Horse's Mouth" cannot be taken as an offer to supply at that price.

Link to Ezine home page (for reading).
Link to Blogging home page (to add comments).

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2024: 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho

PAGE: http://www.wellho.net/mouth/2727_Mak ... aster.html • PAGE BUILT: Sun Oct 11 16:07:41 2020 • BUILD SYSTEM: JelliaJamb