public class CHTBTokenizer extends AbstractTokenizer<java.lang.String>
NEWLINE_TOKEN, nextToken
Constructor and Description |
---|
CHTBTokenizer(java.io.Reader r)
Constructs a new tokenizer from a Reader.
|
Modifier and Type | Method and Description |
---|---|
protected java.lang.String |
getNext()
Internally fetches the next token.
|
static void |
main(java.lang.String[] args)
The main() method tokenizes a file in the specified Encoding
and prints it to standard output in the specified Encoding.
|
hasNext, next, peek, remove, tokenize
public CHTBTokenizer(java.io.Reader r)
ConvertEncodingThread
, or by specifying
the files encoding explicitly in the Reader with
java.io.InputStreamReader
.r
- Readerprotected java.lang.String getNext()
getNext
in class AbstractTokenizer<java.lang.String>
public static void main(java.lang.String[] args) throws java.io.IOException
java.io.IOException