Package com.imsl.io

Class Tokenizer

java.lang.Object

com.imsl.io.Tokenizer

public class Tokenizer
extends Object

Breaks a line into tokens.

The Tokenizer divides a line into tokens separated by deliminators. There can be any number of deliminators set. All of the deliminators are treated equally.

There can be at most one quote character set. If it is set then deliminators inside of a quoted string are treated as part of the string and not as deliminators. The quotes are not returned as part of the token. To escape a quote, repeat it.

See Also:

• Example: Space Separated Data

Constructor Summary

Constructors

Constructor	Description
Tokenizer(String deliminators, char quote, boolean mergeMultipleDeliminators)	Creates a Tokenizer.

Method Summary

Modifier and Type	Method	Description
int	countTokens()	Returns the number of times that the nextToken method can be called without generating an exception.
boolean	hasMoreTokens()	Returns true if a call to nextToken will not generate an exception.
String	nextToken()	Returns the next token.
void	parse(String line)	Sets the line to be tokenized.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

Tokenizer

public Tokenizer(String deliminators,
 char quote,
 boolean mergeMultipleDeliminators)

Creates a Tokenizer.

Parameters:

deliminators - is a String containing the deliminator characters.

quote - is a char containing the quote character. If 0 then quoting is disabled.

mergeMultipleDeliminators - is true if multiple consecutive deliminators are to be treated as a single deliminator.

Method Details

parse

public void parse(String line)

Sets the line to be tokenized. Any tokens left from the previous line are discarded.

Parameters:

line - is the line to be tokenized.

hasMoreTokens

public boolean hasMoreTokens()

Returns true if a call to nextToken will not generate an exception.

countTokens

public int countTokens()

Returns the number of times that the nextToken method can be called without generating an exception.

nextToken

public String nextToken()

Returns the next token.

Returns:

the next token.

Throws:

NoSuchElementException - if there are no more tokens to be returned.