edu.stanford.nlp.process

Interface TokenizerFactory<T>

Type Parameters:

T - The type of the tokens returned by the Tokenizer

All Superinterfaces:

IteratorFromReaderFactory<T>, java.io.Serializable

All Known Implementing Classes:

ArabicTokenizer.ArabicTokenizerFactory, FrenchTokenizer.FrenchTokenizerFactory, PTBTokenizer.PTBTokenizerFactory, SpanishTokenizer.SpanishTokenizerFactory, TreeTokenizerFactory, WhitespaceTokenizer.WhitespaceTokenizerFactory

public interface TokenizerFactory<T>
extends IteratorFromReaderFactory<T>

A TokenizerFactory is a factory that can build a Tokenizer (an extension of Iterator) from a java.io.Reader. IMPORTANT NOTE: A TokenizerFactory should also provide two static methods: public static TokenizerFactory<? extends HasWord> newTokenizerFactory(); public static TokenizerFactory<Word> newWordTokenizerFactory(String options); These are expected by certain JavaNLP code (e.g., LexicalizedParser), which wants to produce a TokenizerFactory by reflection.

Author:

Christopher Manning

Method Summary

Modifier and Type	Method and Description
Tokenizer<T>	getTokenizer(java.io.Reader r) Get a tokenizer for this reader.
Tokenizer<T>	getTokenizer(java.io.Reader r, java.lang.String extraOptions) Get a tokenizer for this reader.
void	setOptions(java.lang.String options) Sets default options for how tokenizers built from this factory should behave.

Methods inherited from interface edu.stanford.nlp.objectbank.IteratorFromReaderFactory

getIterator

Method Detail

getTokenizer

Tokenizer<T> getTokenizer(java.io.Reader r)

Get a tokenizer for this reader.

Parameters:

r - A Reader (which is assumed to already by buffered, if appropriate)

Returns:

A Tokenizer

getTokenizer

Tokenizer<T> getTokenizer(java.io.Reader r,
                          java.lang.String extraOptions)

Get a tokenizer for this reader.

Parameters:

r - A Reader (which is assumed to already by buffered, if appropriate)

extraOptions - Options for how this tokenizer should behave

Returns:

A Tokenizer

setOptions

void setOptions(java.lang.String options)

Sets default options for how tokenizers built from this factory should behave.

Parameters:

options - Options for how this tokenizer should behave