edu.stanford.nlp.io

Interface Lexer

public interface Lexer

A Lexer interface to be used with LexerTokenizer. You can put a Reader inside a Lexer with the yyreset(java.io.Reader) method. An easy way to build classes implementing this interface is with JFlex (http://jflex.de). Just make sure to include the following in the JFlex source file In the Options and Macros section of the source file, include %class JFlexDummyLexer
%standalone
%unicode
%int

%implements edu.stanford.nlp.io.Lexer

%{
public void pushBack(int n) {
yypushback(n);
}

public int getYYEOF() {
return YYEOF;
}
%}
Alternatively, you can customize your own lexer and get lots of flexibility out.

Author:

Roger Levy

Field Summary

Fields

Modifier and Type	Field and Description
static int	ACCEPT
static int	IGNORE

Method Summary

Modifier and Type	Method and Description
int	getYYEOF() returns value for YYEOF
void	pushBack(int length) Pushes back length character positions in the lexer.
int	yylex() Gets the next token from input and returns an integer value signalling what to do with the token.
void	yyreset(java.io.Reader r) put a Reader inside the Lexer.
java.lang.String	yytext() returns the matched input text region

Field Detail

ACCEPT

static final int ACCEPT

See Also:

Constant Field Values

IGNORE

static final int IGNORE

See Also:

Constant Field Values

Method Detail

yylex

int yylex()
   throws java.io.IOException

Gets the next token from input and returns an integer value signalling what to do with the token.

Throws:

java.io.IOException

yytext

java.lang.String yytext()

returns the matched input text region

pushBack

void pushBack(int length)

Pushes back length character positions in the lexer. Conventionally used to push back exactly one token.

getYYEOF

int getYYEOF()

returns value for YYEOF

yyreset

void yyreset(java.io.Reader r)
      throws java.io.IOException

put a Reader inside the Lexer.

Throws:

java.io.IOException