Package edu.stanford.nlp.io

Interface Lexer

  • public interface Lexer
    A Lexer interface to be used with LexerTokenizer. You can put a Reader inside a Lexer with the yyreset(java.io.Reader) method. An easy way to build classes implementing this interface is with JFlex (http://jflex.de). Just make sure to include the following in the JFlex source file In the Options and Macros section of the source file, include %class JFlexDummyLexer
    %standalone
    %unicode
    %int

    %implements edu.stanford.nlp.io.Lexer

    %{
    public void pushBack(int n) {
    yypushback(n);
    }

    public int getYYEOF() {
    return YYEOF;
    }
    %}
    Alternatively, you can customize your own lexer and get lots of flexibility out.
    Author:
    Roger Levy

    • Field Summary

      Fields 
      Modifier and Type Field Description
      static int ACCEPT  
      static int IGNORE  

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      int getYYEOF()
      returns value for YYEOF
      void pushBack​(int length)
      Pushes back length character positions in the lexer.
      int yylex()
      Gets the next token from input and returns an integer value signalling what to do with the token.
      void yyreset​(Reader r)
      put a Reader inside the Lexer.
      String yytext()
      returns the matched input text region

    • Method Detail

      • yylex

        int yylex()
   throws IOException
        Gets the next token from input and returns an integer value signalling what to do with the token.
        Throws:
        IOException

      • yytext

        String yytext()
        returns the matched input text region

      • pushBack

        void pushBack​(int length)
        Pushes back length character positions in the lexer. Conventionally used to push back exactly one token.

      • getYYEOF

        int getYYEOF()
        returns value for YYEOF