TokenizerAdapter (Stanford JavaNLP API)

java.lang.Object
- edu.stanford.nlp.process.AbstractTokenizer<String>
- - edu.stanford.nlp.process.TokenizerAdapter

All Implemented Interfaces:

Tokenizer<String>, Iterator<String>

Direct Known Subclasses:

PennTreebankTokenizer
```
public class TokenizerAdapter
extends AbstractTokenizer<String>
```
This class adapts between a java.io.StreamTokenizer and a edu.stanford.nlp.process.Tokenizer.

Version:

2004/04/01

Author:

Christopher Manning

Field Summary

Fields
Modifier and Type Field Description

protected String eolString

protected StreamTokenizer st
- Fields inherited from class edu.stanford.nlp.process.AbstractTokenizer
  NEWLINE_TOKEN, nextToken

Constructor Summary

Constructors
Constructor Description

TokenizerAdapter(StreamTokenizer st)
Create a new TokenizerAdaptor.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method	Description
`String`	`getNext()`	Internally fetches the next token.
`boolean`	`isEol(String str)`	Say whether the `String` is the end-of-line token for this tokenizer.
`void`	`setEolString(String eolString)`	Set the `String` returned when the inner tokenizer returns an end-of-line token.

Methods inherited from class edu.stanford.nlp.process.AbstractTokenizer
hasNext, next, peek, remove, tokenize

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.util.Iterator
forEachRemaining

- Field Detail
  - st
```
protected final StreamTokenizer st
```
  - eolString
```
protected String eolString
```
- Constructor Detail
  - TokenizerAdapter
```
public TokenizerAdapter(StreamTokenizer st)
```
    Create a new TokenizerAdaptor. In general, it is recommended that the passed in StreamTokenizer should have had resetSyntax() done to it, so that numbers are returned as entered as tokens of type String, though this code will cope as best it can.
    
    Parameters:
    
    st - The internal java.io.StreamTokenizer
- Method Detail
  - getNext
```
public String getNext()
```
    Internally fetches the next token.
    
    Specified by:
    
    getNext in class AbstractTokenizer<String>
    
    Returns:
    
    the next token in the token stream, or null if none exists.
  - setEolString
```
public void setEolString(String eolString)
```
    Set the String returned when the inner tokenizer returns an end-of-line token. This will only happen if the inner tokenizer has been set to eolIsSignificant(true).
    
    Parameters:
    
    eolString - The String used to represent eol. It is not allowed to be null (which would confuse line ends and file end)
  - isEol
```
public boolean isEol(String str)
```
    Say whether the String is the end-of-line token for this tokenizer.
    
    Parameters:
    
    str - The String being tested
    
    Returns:
    
    Whether it is the end-of-line token