22ea String Tokenizer - The Java Programming language
Training, Open Source computer languages
PerlPHPPythonMySQLApache / TomcatTclRubyJavaC and C++LinuxCSS 
Search for:
Home Accessibility Courses Diary The Mouth Forum Resources Site Map About Us Contact
String Tokenizer

Posted by ickle_girl_geek (ickle_girl_geek), 23 June 2003




I have read a file in, and stored it in an array, I now need to Tokenize on a String, that is I want to only split when ")," occur next to each other. Plus I seem to lose a token either side of the delimiter what I am doing wrong

The Input is as below

state(A,B,C,D) :-
  deriv(B,D), deriv(A,C), m_plus(A,C,[]), add(B,A,E,[]),
  m_plus(E,D,[]).
state(A,B,C,D) :-
  deriv(B,D), deriv(A,C), m_plus(A,C,[]), add(B,A,E,[]),
  m_minus(E,D,[]).
state(A,B,C,D) :-
  deriv(B,D), deriv(A,C), m_plus(A,C,[]), add(A,E,B,[]),
  m_plus(E,D,[]).

and I want it to be

state(A,B,C,D) :-
deriv(B,D),
deriv(A,C),
m_plus(A,C,[]),
add(B,A,E,[]),
m_plus(E,D,[]).

etc.

I think this may take more than one pass with a String Tokenizer but need help desperately (before I burst into tears and run away)

Please please please help me  

It would make me very happy  

Posted by admin (Graham Ellis), 23 June 2003
The String Tokenizer splits up strings at a single given character, and if you specify several characters it splits at any one of them - you want to split at a multicharacter string.  (i.e. the tokenizer uses an OR operation and you want to do an AND)

Possible solutions?

a) Use another string handling class - is this Java2 1.4?  If it is, I would be very tempted to use a regular expression

or

b) "Roll your own" ... here's some code I've cut and pasted from our training notes and modified a bit

Code:
public static void main (String [] args) {
     String you_said = "deriv(B,D), deriv(A,C), m_plus(A,C,[]), add(B,A,E,[]),";
     int old_posn=0,posn=0;
     while (posn>=0)
        {
        posn = you_said.indexOf("),",old_posn);
        String next_word = (posn>0) ?
              you_said.substring(old_posn,posn+2):
              you_said.substring(old_posn);
        System.out.println("Next word: "+next_word);
        old_posn = posn + 2;
        }
}
}


and here's what if output:

Code:
[localhost:~/jun03] graham% java Ic
Next word: deriv(B,D),
Next word:  deriv(A,C),
Next word:  m_plus(A,C,[]),
Next word:  add(B,A,E,[]),
Next word:
[localhost:~/jun03] graham%




This page is a thread posted to the opentalk forum at www.opentalk.org.uk and archived here for reference. To jump to the archive index please follow this link.
38be

You can Add a comment or ranking to this page

© WELL HOUSE CONSULTANTS LTD., 2013: Well House Manor • 48 Spa Road • Melksham, Wiltshire • United Kingdom • SN12 7NY
PH: 01144 1225 708225 • FAX: 01144 1225 899360 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho
0