edu.stanford.nlp.process

Class PTBTokenizer.PTBTokenizerFactory<T extends HasWord>

java.lang.Object

edu.stanford.nlp.process.PTBTokenizer.PTBTokenizerFactory<T>

Type Parameters:

T - The class of the returned tokens

All Implemented Interfaces:

IteratorFromReaderFactory<T>, TokenizerFactory<T>, java.io.Serializable

Enclosing class:

PTBTokenizer<T extends HasWord>

public static class PTBTokenizer.PTBTokenizerFactory<T extends HasWord>
extends java.lang.Object
implements TokenizerFactory<T>

This class provides a factory which will vend instances of PTBTokenizer which wrap a provided Reader. See the documentation for PTBTokenizer for details of the parameters and options.

See Also:

PTBTokenizer, Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected LexedTokenFactory<T>	factory
protected java.lang.String	options

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<T>	getIterator(java.io.Reader r) Returns a tokenizer wrapping the given Reader.
Tokenizer<T>	getTokenizer(java.io.Reader r) Returns a tokenizer wrapping the given Reader.
Tokenizer<T>	getTokenizer(java.io.Reader r, java.lang.String extraOptions) Get a tokenizer for this reader.
static PTBTokenizer.PTBTokenizerFactory<CoreLabel>	newCoreLabelTokenizerFactory(java.lang.String options) Constructs a new PTBTokenizer that returns CoreLabel objects and uses the options passed in.
static PTBTokenizer.PTBTokenizerFactory<CoreLabel>	newPTBTokenizerFactory(boolean tokenizeNLs, boolean invertible)
static <T extends HasWord> PTBTokenizer.PTBTokenizerFactory<T>	newPTBTokenizerFactory(LexedTokenFactory<T> tokenFactory, java.lang.String options) Constructs a new PTBTokenizer that uses the LexedTokenFactory and options passed in.
static TokenizerFactory<Word>	newTokenizerFactory() Constructs a new TokenizerFactory that returns Word objects and treats carriage returns as normal whitespace.
static PTBTokenizer.PTBTokenizerFactory<Word>	newWordTokenizerFactory(java.lang.String options) Constructs a new PTBTokenizer that returns Word objects and uses the options passed in.
void	setOptions(java.lang.String options) Sets default options for how tokenizers built from this factory should behave.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

factory

protected final LexedTokenFactory<T extends HasWord> factory

options

protected java.lang.String options

Method Detail

newTokenizerFactory

public static TokenizerFactory<Word> newTokenizerFactory()

Constructs a new TokenizerFactory that returns Word objects and treats carriage returns as normal whitespace. THIS METHOD IS INVOKED BY REFLECTION BY SOME OF THE JAVANLP CODE TO LOAD A TOKENIZER FACTORY. IT SHOULD BE PRESENT IN A TokenizerFactory.

Returns:

A TokenizerFactory that returns Word objects

newWordTokenizerFactory

public static PTBTokenizer.PTBTokenizerFactory<Word> newWordTokenizerFactory(java.lang.String options)

Constructs a new PTBTokenizer that returns Word objects and uses the options passed in. THIS METHOD IS INVOKED BY REFLECTION BY SOME OF THE JAVANLP CODE TO LOAD A TOKENIZER FACTORY. IT SHOULD BE PRESENT IN A TokenizerFactory.

Parameters:

options - A String of options

Returns:

A TokenizerFactory that returns Word objects

newCoreLabelTokenizerFactory

public static PTBTokenizer.PTBTokenizerFactory<CoreLabel> newCoreLabelTokenizerFactory(java.lang.String options)

Constructs a new PTBTokenizer that returns CoreLabel objects and uses the options passed in.

Parameters:

options - A String of options. For the default, recommended options for PTB-style tokenization compatibility, pass in an empty String.

Returns:

A TokenizerFactory that returns CoreLabel objects o

newPTBTokenizerFactory

public static <T extends HasWord> PTBTokenizer.PTBTokenizerFactory<T> newPTBTokenizerFactory(LexedTokenFactory<T> tokenFactory,
                                                                                             java.lang.String options)

Constructs a new PTBTokenizer that uses the LexedTokenFactory and options passed in.

Parameters:

tokenFactory - The LexedTokenFactory

options - A String of options

Returns:

A TokenizerFactory that returns objects of the type of the LexedTokenFactory

newPTBTokenizerFactory

public static PTBTokenizer.PTBTokenizerFactory<CoreLabel> newPTBTokenizerFactory(boolean tokenizeNLs,
                                                                                 boolean invertible)

getIterator

public java.util.Iterator<T> getIterator(java.io.Reader r)

Returns a tokenizer wrapping the given Reader.

Specified by:

getIterator in interface IteratorFromReaderFactory<T extends HasWord>

Parameters:

r - Where to read objects from

Returns:

An Iterator over the objects

getTokenizer

public Tokenizer<T> getTokenizer(java.io.Reader r)

Returns a tokenizer wrapping the given Reader.

Specified by:

getTokenizer in interface TokenizerFactory<T extends HasWord>

Parameters:

r - A Reader (which is assumed to already by buffered, if appropriate)

Returns:

A Tokenizer

getTokenizer

public Tokenizer<T> getTokenizer(java.io.Reader r,
                                 java.lang.String extraOptions)

Description copied from interface: TokenizerFactory

Get a tokenizer for this reader.

Specified by:

getTokenizer in interface TokenizerFactory<T extends HasWord>

Parameters:

r - A Reader (which is assumed to already by buffered, if appropriate)

extraOptions - Options for how this tokenizer should behave

Returns:

A Tokenizer

setOptions

public void setOptions(java.lang.String options)

Description copied from interface: TokenizerFactory

Sets default options for how tokenizers built from this factory should behave.

Specified by:

setOptions in interface TokenizerFactory<T extends HasWord>

Parameters:

options - Options for how this tokenizer should behave