edu.stanford.nlp.process

Class DocumentPreprocessor

java.lang.Object

edu.stanford.nlp.process.DocumentPreprocessor

All Implemented Interfaces:

java.lang.Iterable<java.util.List<HasWord>>

public class DocumentPreprocessor
extends java.lang.Object
implements java.lang.Iterable<java.util.List<HasWord>>

Produces a list of sentences from either a plain text or XML document. This class acts like a Reader: It allows you to make a single pass through a list of sentences in a document. If you need to pass through the document multiple times, then you need to create a second DocumentProcessor.

Tokenization: The default tokenizer is PTBTokenizer. If null is passed to setTokenizerFactory, then whitespace tokenization is assumed.

Adding a new document type requires two steps:

1. Add a new DocType.
2. Create an iterator for the new DocType and modify the iterator() function to return the new iterator.

NOTES: This document preprocessor is principally used in the Stanford Parser (and also a bit in the POS tagger). It is not used by CoreNLP. This implementation should not use external libraries since it is used in the parser.

Author:

Spence Green

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	DocumentPreprocessor.DocType

Constructor Summary

Constructors

Constructor and Description
DocumentPreprocessor(java.io.Reader input) Constructs a preprocessor from an existing input stream.
DocumentPreprocessor(java.io.Reader input, DocumentPreprocessor.DocType t)
DocumentPreprocessor(java.lang.String docPath)
DocumentPreprocessor(java.lang.String docPath, DocumentPreprocessor.DocType t)
DocumentPreprocessor(java.lang.String docPath, DocumentPreprocessor.DocType t, java.lang.String encoding) Constructs a preprocessor from a file at a path, which can be either a filesystem location, a classpath entry, or a URL.

Method Summary

Modifier and Type	Method and Description
java.util.Iterator<java.util.List<HasWord>>	iterator() Returns sentences until the document is exhausted.
static void	main(java.lang.String[] args) A simple, deterministic sentence-splitter.
void	setElementDelimiter(java.lang.String s) Only read text from inside these XML elements if in XML mode.
void	setEscaper(java.util.function.Function<java.util.List<HasWord>,java.util.List<HasWord>> e) Set an escaper.
void	setKeepEmptySentences(boolean keepEmptySentences) Set whether or not the tokenizer keeps empty sentences in whitespace mode.
void	setSentenceDelimiter(java.lang.String s) Make the processor assume that the document is already delimited by the supplied parameter.
void	setSentenceFinalPuncWords(java.lang.String[] sentenceFinalPuncWords) Sets the end-of-sentence delimiters.
void	setTagDelimiter(java.lang.String s) Split tags from tokens.
void	setTokenizerFactory(TokenizerFactory<? extends HasWord> newTokenizerFactory) Sets the factory from which to produce a Tokenizer.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Constructor Detail

DocumentPreprocessor

public DocumentPreprocessor(java.io.Reader input)

Constructs a preprocessor from an existing input stream.

Parameters:

input - An existing reader

DocumentPreprocessor

public DocumentPreprocessor(java.io.Reader input,
                            DocumentPreprocessor.DocType t)

DocumentPreprocessor

public DocumentPreprocessor(java.lang.String docPath)

DocumentPreprocessor

public DocumentPreprocessor(java.lang.String docPath,
                            DocumentPreprocessor.DocType t)

DocumentPreprocessor

public DocumentPreprocessor(java.lang.String docPath,
                            DocumentPreprocessor.DocType t,
                            java.lang.String encoding)

Constructs a preprocessor from a file at a path, which can be either a filesystem location, a classpath entry, or a URL.

Parameters:

docPath - The path

encoding - The character encoding used by Readers

Method Detail

setKeepEmptySentences

public void setKeepEmptySentences(boolean keepEmptySentences)

Set whether or not the tokenizer keeps empty sentences in whitespace mode. Useful for programs that want to echo blank lines. Not relevant for the non-whitespace model.

setSentenceFinalPuncWords

public void setSentenceFinalPuncWords(java.lang.String[] sentenceFinalPuncWords)

Sets the end-of-sentence delimiters.

For newline tokenization, use the argument {"\n"}.

Parameters:

sentenceFinalPuncWords - An array of words that count as sentence final punctuation.

setTokenizerFactory

public void setTokenizerFactory(TokenizerFactory<? extends HasWord> newTokenizerFactory)

Sets the factory from which to produce a Tokenizer. The default is PTBTokenizer.

NOTE: If a null argument is used, then the document is assumed to be tokenized and DocumentPreprocessor performs no tokenization.

setEscaper

public void setEscaper(java.util.function.Function<java.util.List<HasWord>,java.util.List<HasWord>> e)

Set an escaper.

Parameters:

e - The escaper

setSentenceDelimiter

public void setSentenceDelimiter(java.lang.String s)

Make the processor assume that the document is already delimited by the supplied parameter.

Parameters:

s - The sentence delimiter

setTagDelimiter

public void setTagDelimiter(java.lang.String s)

Split tags from tokens. The tag will be placed in the TagAnnotation of the returned label.

Note that for strings that contain two or more instances of the tag delimiter, the last instance is treated as the split point.

The tag delimiter should not contain any characters that must be escaped in a Java regex.

Parameters:

s - POS tag delimiter

setElementDelimiter

public void setElementDelimiter(java.lang.String s)

Only read text from inside these XML elements if in XML mode. Note: This class implements an approximation to XML via regex. Otherwise, text will read from all tokens.

iterator

public java.util.Iterator<java.util.List<HasWord>> iterator()

Returns sentences until the document is exhausted. Calls close() if the end of the document is reached. Otherwise, the user is required to close the stream.

Specified by:

iterator in interface java.lang.Iterable<java.util.List<HasWord>>

Returns:

An Iterator over sentences (each a List of word tokens). Although the type is given as List<HasWord>, in practice you get a List of CoreLabel, and you can cast down to that. (Someday we might manage to fix the generic typing....)

main

public static void main(java.lang.String[] args)
                 throws java.io.IOException

A simple, deterministic sentence-splitter. This method only supports the English tokenizer, so for other languages you should run the tokenizer first and then run this sentence splitter with the "-whitespaceTokenization" option.

Parameters:

args - Command-line arguments

Throws:

java.io.IOException