I need to be able  to take a character string for an algebraic expression,
e.g.  "a = height+2.5*sqr(b+2.3)"
and tokenise this into numeric constants, variables, math. operators,  functions etc.

I have seen this topic discussed by some of the very top Perl folks and the solution they came up with was to split the incoming string at the variable or constant tokens, for example:

Code:

#!/usr/bin/perl $string = "a = height+2.5*sqr(b+2.3)"; @parts = split(/([[:alpha:]]\w+|\d*\.?\d*)/,$string); foreach $el(@parts) {        print ("Element ",++$n,": $el\n");        }

which gives

Element 1: a
Element 2:
Element 3:
Element 4:
Element 5: =
Element 6:
Element 7:
Element 8: height
Element 9: +
Element 10: 2.5
Element 11: *
Element 12: sqr
Element 13: (
Element 14:
Element 15: b
Element 16:
Element 17: +
Element 18: 2.3
Element 19: )

or (even simpler - data changed to illustrate how numbers are handled):

Code:

#!/usr/bin/perl $string = "a = height+12.5*sqr(b+2.3)"; @parts = split(/(\w*)/,$string); foreach $el(@parts) {        print ("Element ",++$n,": $el\n");        }

Element 1:
Element 2: a
Element 3:
Element 4:
Element 5: =
Element 6:
Element 7:
Element 8: height
Element 9: +
Element 10: 12
Element 11: .
Element 12: 5
Element 13: *
Element 14: sqr
Element 15: (
Element 16: b
Element 17: +
Element 18: 2
Element 19: .
Element 20: 3
Element 21: )

---