edu.cmu.sphinx.alignment.tokenizer

Class CharTokenizer

java.lang.Object

edu.cmu.sphinx.alignment.tokenizer.CharTokenizer

All Implemented Interfaces:

java.util.Iterator<Token>

public class CharTokenizer
extends java.lang.Object
implements java.util.Iterator<Token>

Implements the tokenizer interface. Breaks an input sequence of characters into a set of tokens.

Field Summary

Fields

Modifier and Type	Field and Description
static java.lang.String	DEFAULT_POSTPUNCTUATION_SYMBOLS A string containing the default post-punctuation characters.
static java.lang.String	DEFAULT_PREPUNCTUATION_SYMBOLS A string containing the default pre-punctuation characters.
static java.lang.String	DEFAULT_SINGLE_CHAR_SYMBOLS A string containing the default single characters.
static java.lang.String	DEFAULT_WHITESPACE_SYMBOLS A string containing the default whitespace characters.
static int	EOF A constant indicating that the end of the stream has been read.

Constructor Summary

Constructors

Constructor and Description
CharTokenizer() Constructs a Tokenizer.
CharTokenizer(java.io.Reader file) Creates a tokenizer that will return tokens from the given file.
CharTokenizer(java.lang.String string) Creates a tokenizer that will return tokens from the given string.

Method Summary

Modifier and Type	Method and Description
java.lang.String	getErrorDescription() if hasErrors returns true, this will return a description of the error encountered, otherwise it will return null
boolean	hasErrors() Returns true if there were errors while reading tokens
boolean	hasNext() Returns true if there are more tokens, false otherwise.
boolean	isSentenceSeparator() Determines if the current token should start a new sentence.
Token	next() Returns the next token.
void	remove()
void	setInputReader(java.io.Reader reader) Sets the input reader
void	setInputText(java.lang.String inputString) Sets the text to tokenize.
void	setPostpunctuationSymbols(java.lang.String symbols) Sets the postpunctuation symbols of this Tokenizer to the given symbols.
void	setPrepunctuationSymbols(java.lang.String symbols) Sets the prepunctuation symbols of this Tokenizer to the given symbols.
void	setSingleCharSymbols(java.lang.String symbols) Sets the single character symbols of this Tokenizer to the given symbols.
void	setWhitespaceSymbols(java.lang.String symbols) Sets the whitespace symbols of this Tokenizer to the given symbols.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

EOF

public static final int EOF

A constant indicating that the end of the stream has been read.

See Also:

Constant Field Values

DEFAULT_WHITESPACE_SYMBOLS

public static final java.lang.String DEFAULT_WHITESPACE_SYMBOLS

A string containing the default whitespace characters.

See Also:

Constant Field Values

DEFAULT_SINGLE_CHAR_SYMBOLS

public static final java.lang.String DEFAULT_SINGLE_CHAR_SYMBOLS

A string containing the default single characters.

See Also:

Constant Field Values

DEFAULT_PREPUNCTUATION_SYMBOLS

public static final java.lang.String DEFAULT_PREPUNCTUATION_SYMBOLS

A string containing the default pre-punctuation characters.

See Also:

Constant Field Values

DEFAULT_POSTPUNCTUATION_SYMBOLS

public static final java.lang.String DEFAULT_POSTPUNCTUATION_SYMBOLS

A string containing the default post-punctuation characters.

See Also:

Constant Field Values

Constructor Detail

CharTokenizer

public CharTokenizer()

Constructs a Tokenizer.

CharTokenizer

public CharTokenizer(java.lang.String string)

Creates a tokenizer that will return tokens from the given string.

Parameters:

string - the string to tokenize

CharTokenizer

public CharTokenizer(java.io.Reader file)

Creates a tokenizer that will return tokens from the given file.

Parameters:

file - where to read the input from

Method Detail

setWhitespaceSymbols

public void setWhitespaceSymbols(java.lang.String symbols)

Sets the whitespace symbols of this Tokenizer to the given symbols.

Parameters:

symbols - the whitespace symbols

setSingleCharSymbols

public void setSingleCharSymbols(java.lang.String symbols)

Sets the single character symbols of this Tokenizer to the given symbols.

Parameters:

symbols - the single character symbols

setPrepunctuationSymbols

public void setPrepunctuationSymbols(java.lang.String symbols)

Sets the prepunctuation symbols of this Tokenizer to the given symbols.

Parameters:

symbols - the prepunctuation symbols

setPostpunctuationSymbols

public void setPostpunctuationSymbols(java.lang.String symbols)

Sets the postpunctuation symbols of this Tokenizer to the given symbols.

Parameters:

symbols - the postpunctuation symbols

setInputText

public void setInputText(java.lang.String inputString)

Sets the text to tokenize.

Parameters:

inputString - the string to tokenize

setInputReader

public void setInputReader(java.io.Reader reader)

Sets the input reader

Parameters:

reader - the input source

next

public Token next()

Returns the next token.

Specified by:

next in interface java.util.Iterator<Token>

Returns:

the next token if it exists, null if no more tokens

hasNext

public boolean hasNext()

Returns true if there are more tokens, false otherwise.

Specified by:

hasNext in interface java.util.Iterator<Token>

Returns:

true if there are more tokens false otherwise

remove

public void remove()

Specified by:

remove in interface java.util.Iterator<Token>

hasErrors

public boolean hasErrors()

Returns true if there were errors while reading tokens

Returns:

true if there were errors; false otherwise

getErrorDescription

public java.lang.String getErrorDescription()

if hasErrors returns true, this will return a description of the error encountered, otherwise it will return null

Returns:

a description of the last error that occurred.

isSentenceSeparator

public boolean isSentenceSeparator()

Determines if the current token should start a new sentence.

Returns:

true if a new sentence should be started