public class ArabicTreebankTokenizer extends PennTreebankTokenizer
StreamTokenizer
.
This implementation is current as of the following LDC catalog numbers: LDC2008E61 (ATBp1v4), LDC2008E62 (ATBp2v3), and LDC2008E22 (ATBp3v3.1)
eolString, st
NEWLINE_TOKEN, nextToken
Constructor and Description |
---|
ArabicTreebankTokenizer(java.io.Reader r) |
Modifier and Type | Method and Description |
---|---|
java.lang.String |
getNext()
Internally fetches the next token.
|
static void |
main(java.lang.String[] args) |
isEol, setEolString
hasNext, next, peek, remove, tokenize
public java.lang.String getNext()
getNext
in class TokenizerAdapter
public static void main(java.lang.String[] args) throws java.io.IOException
java.io.IOException