Package edu.stanford.nlp.process

Interface LexedTokenFactory<T>

  • All Known Implementing Classes:
    CoreLabelTokenFactory, WordTokenFactory
    public interface LexedTokenFactory<T>
    Constructs a token (of arbitrary type) from a String and its position in the underlying text. This is used to create tokens in JFlex lexers such as PTBTokenizer.

    • Method Summary

      All Methods Instance Methods Abstract Methods 
      Modifier and Type Method Description
      T makeToken​(String str, int begin, int length)
      Constructs a token (of arbitrary type) from a String and its position in the underlying text.

    • Method Detail

      • makeToken

        T makeToken​(String str,
            int begin,
            int length)
        Constructs a token (of arbitrary type) from a String and its position in the underlying text. The int arguments are used just to record token character offsets in an underlying text. This method does not take a substring of str.
        Parameters:
        str - The String extracted by the lexer.
        begin - The offset in the document of the first character in this string.
        length - The number of characters the string takes up in the document.
        Returns:
        The token of type T.