public class MaxentTagger extends Tagger implements ListProcessor<java.util.List<? extends HasWord>,java.util.List<TaggedWord>>, java.io.Serializable
edu/stanford/nlp/models/pos-tagger/english-bidirectional-distsim.tagger
.
Its accuracy was 97.32% on Penn Treebank WSJ secs. 22-24.edu/stanford/nlp/models/pos-tagger/english-left3words-distsim.tagger
This tagger runs a lot faster, and is recommended for general use.
Its accuracy was 96.92% on Penn Treebank WSJ secs. 22-24.MaxentTagger tagger = new MaxentTagger("models/left3words-wsj-0-18.tagger");
MaxentTagger tagger = new MaxentTagger(DEFAULT_NLP_GROUP_MODEL_PATH);
List<TaggedWord> taggedSentence = tagger.tagSentence(List<? extends HasWord> sentence)
List<TaggedWord> taggedSentence = tagger.apply(List<? extends HasWord> sentence)
List taggedList = tagger.process(List sentences)
String taggedString = tagger.tagString("Here's a tagged string.")
String taggedString = tagger.tagTokenizedString("Here 's a tagged string .")
tagString
method uses the default tokenizer (PTBTokenizer).
If you wish to control tokenization, you may wish to call
tokenizeText(Reader, TokenizerFactory)
and then to call
process()
on the result.
java edu.stanford.nlp.tagger.maxent.MaxentTagger -genpropsThis gets you a default properties file with descriptions of each parameter you can set in your trained model. You can modify the properties file, or use the default options. To train, run:
java -mx1g edu.stanford.nlp.tagger.maxent.MaxentTagger -props myPropertiesFile.propswith the appropriate properties file specified. Any argument you give in the properties file can also be specified on the command line. You must have specified a model using -model, either in the properties file or on the command line, as well as a file containing tagged words using -trainFile. Useful flags for controlling the amount of output are -verbose, which prints extra debugging information, and -verboseResults, which prints full information about intermediate results. -verbose defaults to false and -verboseResults defaults to true.
java edu.stanford.nlp.tagger.maxent.MaxentTagger -model modelFile -textFile textfileFor testing (evaluating against tagged text):
java edu.stanford.nlp.tagger.maxent.MaxentTagger -model modelFile -testFile testfileYou can use the same properties file as for training if you pass it in with the "-props" argument. The most important arguments for tagging (besides "model" and "file") are "tokenize" and "tokenizerFactory". See below for more details. Note that the tagger assumes input has not yet been tokenized and by default tokenizes it using a default English tokenizer. If your input has already been tokenized, use the flag "-tokenize false". Parameters can be defined using a Properties file (specified on the command-line with
-prop
propFile),
or directly on the command line (by preceding their name with a minus sign
("-") to turn them into a flag. The following properties are recognized:
Property Name | Type | Default Value | Relevant Phase(s) | Description |
model | String | N/A | All | Path and filename where you would like to save the model (training) or where the model should be loaded from (testing, tagging). |
trainFile | String | N/A | Train |
Path to the file holding the training data; specifying this option puts the tagger in training mode. Only one of 'trainFile','testFile','textFile', and 'dump' may be specified. There are three formats possible. The first is a text file of tagged data. Each line is considered a separate sentence. In each sentence, words are separated by whitespace. Each word must have a tag, which is separated from the token using the specified tagSeparator . This format, called TEXT, is the default format. The second format is a file of Penn Treebank formatted (i.e., s-expression) tree files. Trees are loaded one at a time and the tagged words in a tree are used as a training sentence. To specify this format, preface the filename with " format=TREES, ". The final possible format is TSV files (tab-separated columns). To specify a TSV file, set trainFile to "format=TSV,wordColumn=x,tagColumn=y,filename ".
Column numbers are indexed from 0, and sentences are separated with blank lines. The default wordColumn is 0 and default tagColumn is 1. If comments=true, then comment lines will be skipped (a common thing to appear in conllu files)
A file can be in a different character set encoding than the tagger's default encoding by prefacing the filename with "encoding=ENC," .
You can specify the tagSeparator character in a TEXT file by prefacing the filename with "tagSeparator=c,". Tree files can be fed through TreeTransformers and TreeNormalizers. To specify a transformer, preface the filename with "treeTransformer=CLASSNAME,". To specify a normalizer, preface the filename with "treeNormalizer=CLASSNAME,". You can also filter trees using a Filter<Tree> , which can be specified with "treeFilter=CLASSNAME,".
A specific range of trees to be used can be specified with treeRange=X-Y. Multiple parts of the range can be separated by : as opposed to the normal separator of ,.
For example, one could use the argument "-treeRange=25-50:75-100".
You can specify a TreeReaderFactory by prefacing the filename with "trf=CLASSNAME,". Note: If it includes a TreeNormalizer, you want to specify it as the treeNormalizer as well.Multiple files can be specified by making a semicolon separated list of files. Each file can have its own format specifiers as above. You will note that none of , ; or = can be in filenames. |
testFile | String | N/A | Test | Path to the file holding the test data; specifying this option puts the tagger in testing mode. Only one of 'trainFile','testFile','textFile', and 'dump' may be specified. The same format as trainFile applies, but only one file can be specified. |
textFile | String | N/A | Tag | Path to the file holding the text to tag; specifying this option puts the tagger in tagging mode. Only one of 'trainFile','testFile','textFile', and 'dump' may be specified. No file reading options may be specified for textFile |
dump | String | N/A | Dump | Path to the file holding the model to dump; specifying this option puts the tagger in dumping mode. Only one of 'trainFile','testFile','textFile', and 'dump' may be specified. |
genprops | boolean | N/A | N/A | Use this option to output a default properties file, containing information about each of the possible configuration options. |
tagSeparator | char | / | All | Separator character that separates word and part of speech tags, such as out/IN or out_IN. For training and testing, this is the separator used in the train/test files. For tagging, this is the character that will be inserted between words and tags in the output. |
encoding | String | UTF-8 | All | Encoding of the read files (training, testing) and the output text files. |
tokenize | boolean | true | Tag,Test | Whether or not the file needs to be tokenized. If this is false, the tagger assumes that white space separates words if and only if they should be tagged as separate tokens, and that the input is strictly one sentence per line. |
tokenizerFactory | String | edu.stanford.nlp. process.PTBTokenizer | Tag,Test | Fully qualified class name of the tokenizer to use. edu.stanford.nlp.process.PTBTokenizer does basic English tokenization. |
tokenizerOptions | String | Tag,Test | Known options for the particular tokenizer used. A comma-separated list. For PTBTokenizer, options of interest include americanize=false and quotes=ascii (for German). Note that any choice of tokenizer options that conflicts with the tokenization used in the tagger training data will likely degrade tagger performance. | |
sentenceDelimiter | String | null | Tag,Test | A marker used to separate a text into sentences. If not set (equal to null ), sentence breaking is done by content (looking for periods, etc.) Otherwise, it will break on this String, except that if the String is "newline", it breaks on the String "\\n". |
arch | String | generic | Train | Architecture of the model, as a comma-separated list of options, some with a parenthesized integer argument written k here: this determines what features are used to build your model. See ExtractorFrames and ExtractorFramesRare for more information. |
wordFunction | String | (none) | Train | A function to apply to the text before training or testing. Must inherit from java.util.function.Function<String, String> . Can be blank. |
lang | String | english | Train | Language from which the part of speech tags are drawn. This option determines which tags are considered closed-class (only fixed set of words can be tagged with a closed-class tag, such as prepositions). Defined languages are 'english' (Penn tag set), 'polish' (very rudimentary), 'french', 'chinese', 'arabic', 'german', and 'medline'. |
openClassTags | String | N/A | Train | Space separated list of tags that should be considered open-class. All tags encountered that are not in this list are considered closed-class. E.g. format: "NN VB" |
closedClassTags | String | N/A | Train | Space separated list of tags that should be considered closed-class. All tags encountered that are not in this list are considered open-class. |
learnClosedClassTags | boolean | false | Train | If true, induce which tags are closed-class by counting as closed-class tags all those tags which have fewer unique word tokens than closedClassTagThreshold. |
closedClassTagThreshold | int | int | Train | Number of unique word tokens that a tag may have and still be considered closed-class; relevant only if learnClosedClassTags is true. |
sgml | boolean | false | Tag, Test | Very basic tagging of the contents of all sgml fields; for more complex mark-up, consider using the xmlInput option. |
xmlInput | String | Tag, Test | Give a space separated list of tags in an XML file whose content you would like tagged. Any internal tags that appear in the content of fields you would like tagged will be discarded; the rest of the XML will be preserved and the original text of specified fields will be replaced with the tagged text. | |
outputFile | String | "" | Tag | Path to write output to. If blank, stdout is used. |
outputFormat | String | "" | Tag | Output format. One of: slashTags (default), xml (or inlineXML as a synonym), or tsv |
outputFormatOptions | String | "" | Tag | Output format options. Currently used: lemmatize, verbose, keepEmptySentences |
tagInside | String | "" | Tag | Tags inside elements that match the regular expression given in the String. |
search | String | cg | Train | Specify the search method to be used in the optimization method for training. Options are 'cg' (conjugate gradient), 'iis' (improved iterative scaling), or 'qn' (quasi-newton). |
sigmaSquared | double | 0.5 | Train | Sigma-squared smoothing/regularization parameter to be used for conjugate gradient search. Default usually works reasonably well. |
iterations | int | 100 | Train | Number of iterations to be used for improved iterative scaling. |
rareWordThresh | int | 5 | Train | Words that appear fewer than this number of times during training are considered rare words and use extra rare word features. |
minFeatureThreshold | int | 5 | Train | Features whose history appears fewer than this number of times are discarded. |
curWordMinFeatureThreshold | int | 2 | Train | Words that occur more than this number of times will generate features with all of the tags they've been seen with. |
rareWordMinFeatureThresh | int | 10 | Train | Features of rare words whose histories occur fewer than this number of times are discarded. |
veryCommonWordThresh | int | 250 | Train | Words that occur more than this number of times form an equivalence class by themselves. Ignored unless you are using ambiguity classes. |
debug | boolean | boolean | All | Whether to write debugging information (words, top words, unknown words, confusion matrix). Useful for error analysis. |
debugPrefix | String | N/A | All | File (path) prefix for where to write out the debugging information (relevant only if debug=true). |
nthreads | int | 1 | Test,Text | Number of threads to use when processing text. |
minWordsLockTags | int | 1 | All | If a word occurs at least this many times in the training data, the tagger will only propose known tags as candidates. |
Modifier and Type | Field and Description |
---|---|
static java.lang.String |
BASE_TAGGER_HOME
The directory from which to get taggers when using
DEFAULT_NLP_GROUP_MODEL_PATH.
|
static java.lang.String |
DEFAULT_DISTRIBUTION_PATH |
static java.lang.String |
DEFAULT_JAR_PATH |
static java.lang.String |
DEFAULT_NLP_GROUP_MODEL_PATH |
static java.lang.String |
TAGGER_HOME |
Constructor and Description |
---|
MaxentTagger() |
MaxentTagger(java.io.InputStream modelStream)
Constructor for a tagger, loading a model stored in a particular file,
classpath resource, or URL.
|
MaxentTagger(java.io.InputStream modelStream,
java.util.Properties config,
boolean printLoading)
Initializer that loads the tagger.
|
MaxentTagger(java.lang.String modelFile)
Constructor for a tagger, loading a model stored in a particular file,
classpath resource, or URL.
|
MaxentTagger(java.lang.String modelFile,
java.util.Properties config)
Constructor for a tagger using a model stored in a particular file,
with options taken from the supplied TaggerConfig.
|
MaxentTagger(java.lang.String modelFile,
java.util.Properties config,
boolean printLoading)
Initializer that loads the tagger.
|
MaxentTagger(TaggerConfig config) |
Modifier and Type | Method and Description |
---|---|
int |
addTag(java.lang.String tag)
Will return the index of a tag, adding it if it doesn't already exist
|
java.util.List<TaggedWord> |
apply(java.util.List<? extends HasWord> in)
Expects a sentence and returns a tagged sentence.
|
protected TokenizerFactory<? extends HasWord> |
chooseTokenizerFactory()
Figures out what tokenizer factory might be described by the
config.
|
protected static TokenizerFactory<? extends HasWord> |
chooseTokenizerFactory(boolean tokenize,
java.lang.String tokenizerFactory,
java.lang.String tokenizerOptions,
boolean invertible) |
java.lang.String |
getTag(int index) |
int |
getTagIndex(java.lang.String tag)
Will return the index of a tag if known, -1 if not already known
|
static void |
lemmatize(java.util.List<CoreLabel> sentence,
Morphology morpha)
Adds lemmas to the given list of CoreLabels, using the given
Morphology object.
|
static void |
main(java.lang.String[] args)
Command-line tagger interface.
|
int |
numTags() |
void |
outputTaggedSentence(java.util.List<? extends HasWord> sentence,
boolean outputLemmas,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle,
boolean outputVerbosity,
int numSentences,
java.lang.String separator,
java.io.Writer writer) |
java.util.List<java.util.List<TaggedWord>> |
process(java.util.List<? extends java.util.List<? extends HasWord>> sentences)
Tags the Words in each Sentence in the given List with their
grammatical part-of-speech.
|
protected void |
readModelAndInit(java.util.Properties config,
java.io.DataInputStream rf,
boolean printLoading)
This reads the complete tagger from a single model file, and inits
the tagger using a combination of the properties passed in and
parameters from the file.
|
protected void |
readModelAndInit(java.util.Properties config,
java.io.InputStream modelStream,
boolean printLoading)
This reads the complete tagger from a single model provided as an InputStream,
and initializes the tagger using a
combination of the properties passed in and parameters from the file.
|
protected void |
readModelAndInit(java.util.Properties config,
java.lang.String modelFileOrUrl,
boolean printLoading)
This reads the complete tagger from a single model stored in a file, at a URL,
or as a resource in a jar file, and initializes the tagger using a
combination of the properties passed in and parameters from the file.
|
void |
runTagger(java.io.BufferedReader reader,
java.io.BufferedWriter writer,
java.lang.String tagInside,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle)
This method runs the tagger on the provided reader and writer.
|
<X extends HasWord> |
runTagger(java.lang.Iterable<java.util.List<X>> document,
java.io.BufferedWriter writer,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) |
void |
runTaggerSGML(java.io.BufferedReader reader,
java.io.BufferedWriter writer,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) |
void |
runTaggerStdin(java.io.BufferedReader reader,
java.io.BufferedWriter writer,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) |
protected void |
saveModel(java.io.DataOutputStream file) |
protected void |
saveModel(java.lang.String filename) |
void |
tagAndOutputSentence(java.util.List<? extends HasWord> sentence,
boolean outputLemmas,
Morphology morpha,
PlainTextDocumentReaderAndWriter.OutputStyle outputStyle,
boolean outputVerbosity,
int numSentences,
java.lang.String separator,
java.io.Writer writer) |
void |
tagCoreLabels(java.util.List<CoreLabel> sentence)
Takes a sentence composed of CoreLabels and add the tags to the
CoreLabels, modifying the input sentence.
|
void |
tagCoreLabels(java.util.List<CoreLabel> sentence,
boolean reuseTags)
Takes a sentence composed of CoreLabels and add the tags to the
CoreLabels, modifying the input sentence.
|
java.util.List<? extends HasWord> |
tagCoreLabelsOrHasWords(java.util.List<? extends HasWord> sentence,
Morphology morpha,
boolean outputLemmas) |
void |
tagFromXML(java.io.InputStream input,
java.io.Writer writer,
java.lang.String... xmlTags)
Uses an XML transformer to turn an input stream into a bunch of
output.
|
void |
tagFromXML(java.io.Reader input,
java.io.Writer writer,
java.lang.String... xmlTags) |
java.util.List<TaggedWord> |
tagSentence(java.util.List<? extends HasWord> sentence)
Returns a new Sentence that is a copy of the given sentence with all the
words tagged with their part-of-speech.
|
java.util.List<TaggedWord> |
tagSentence(java.util.List<? extends HasWord> sentence,
boolean reuseTags)
Returns a new Sentence that is a copy of the given sentence with all the
words tagged with their part-of-speech.
|
java.util.Set<java.lang.String> |
tagSet() |
java.lang.String |
tagString(java.lang.String toTag)
Tags the input string and returns the tagged version.
|
java.lang.String |
tagTokenizedString(java.lang.String toTag)
Tags the tokenized input string and returns the tagged version.
|
static java.util.List<java.util.List<HasWord>> |
tokenizeText(java.io.Reader r)
Reads data from r, tokenizes it with the default (Penn Treebank)
tokenizer, and returns a List of Sentence objects, which can
then be fed into tagSentence.
|
static java.util.List<java.util.List<HasWord>> |
tokenizeText(java.io.Reader r,
TokenizerFactory<? extends HasWord> tokenizerFactory)
Reads data from r, tokenizes it with the given tokenizer, and
returns a List of Lists of (extends) HasWord objects, which can then be
fed into tagSentence.
|
public static final java.lang.String BASE_TAGGER_HOME
public static final java.lang.String TAGGER_HOME
public static final java.lang.String DEFAULT_NLP_GROUP_MODEL_PATH
public static final java.lang.String DEFAULT_JAR_PATH
public static final java.lang.String DEFAULT_DISTRIBUTION_PATH
public MaxentTagger()
public MaxentTagger(TaggerConfig config)
public MaxentTagger(java.lang.String modelFile)
modelFile
- Filename, classpath resource, or URL for the trained modelRuntimeIOException
- if I/O errors or serialization errorspublic MaxentTagger(java.io.InputStream modelStream)
modelStream
- The InputStream from which to read the modelRuntimeIOException
- if I/O errors or serialization errorspublic MaxentTagger(java.lang.String modelFile, java.util.Properties config)
modelFile
- Filename, classpath resource, or URL for the trained modelconfig
- The configuration for the taggerRuntimeIOException
- if I/O errors or serialization errorspublic MaxentTagger(java.lang.String modelFile, java.util.Properties config, boolean printLoading)
modelFile
- Where to initialize the tagger from.
Most commonly, this is the filename of the trained model, for example,
/u/nlp/data/pos-tagger/wsj3t0-18-left3words/left3words-wsj-0-18.tagger
.
However, if it starts with "https?://" it will be interpreted as a URL.
One can also load models directly from the classpath, as in loading from
edu/stanford/nlp/models/pos-tagger/bidirectional-distsim-wsj-0-18.tagger
.config
- TaggerConfig based on command-line argumentsprintLoading
- Whether to print a message saying what model file is being loaded and how long it took when finished.RuntimeIOException
- if I/O errors or serialization errorspublic MaxentTagger(java.io.InputStream modelStream, java.util.Properties config, boolean printLoading)
modelStream
- An InputStream for reading the model fileconfig
- TaggerConfig based on command-line argumentsprintLoading
- Whether to print a message saying what model file is being loaded and how long it took when finished.RuntimeIOException
- if I/O errors or serialization errorspublic int addTag(java.lang.String tag)
public int getTagIndex(java.lang.String tag)
public int numTags()
public java.lang.String getTag(int index)
public java.util.Set<java.lang.String> tagSet()
protected TokenizerFactory<? extends HasWord> chooseTokenizerFactory()
protected static TokenizerFactory<? extends HasWord> chooseTokenizerFactory(boolean tokenize, java.lang.String tokenizerFactory, java.lang.String tokenizerOptions, boolean invertible)
protected void saveModel(java.lang.String filename)
protected void saveModel(java.io.DataOutputStream file) throws java.io.IOException
java.io.IOException
protected void readModelAndInit(java.util.Properties config, java.lang.String modelFileOrUrl, boolean printLoading)
config
- The tagger configmodelFileOrUrl
- The name of the model file. This routine opens and closes it.printLoading
- Whether to print a message saying what model file is being loaded and how long it took when finished.RuntimeIOException
- if I/O errors or serialization errorsprotected void readModelAndInit(java.util.Properties config, java.io.InputStream modelStream, boolean printLoading)
config
- The tagger configmodelStream
- The model provided as an InputStreamprintLoading
- Whether to print a message saying what model file is being loaded and how long it took when finished.RuntimeIOException
- if I/O errors or serialization errorsprotected void readModelAndInit(java.util.Properties config, java.io.DataInputStream rf, boolean printLoading)
config
- The tagger configrf
- DataInputStream to read from. It's the caller's job to open and close this stream.printLoading
- Whether to print a message saying what model file is being loaded and how long it took when finished.RuntimeIOException
- if I/O errors or serialization errorspublic java.lang.String tagTokenizedString(java.lang.String toTag)
toTag
- The untagged input Stringpublic java.lang.String tagString(java.lang.String toTag)
toTag
- The untagged input Stringpublic java.util.List<TaggedWord> apply(java.util.List<? extends HasWord> in)
apply
in interface java.util.function.Function<java.util.List<? extends HasWord>,java.util.List<TaggedWord>>
apply
in class Tagger
in
- This needs to be a sentence (List of words)public java.util.List<java.util.List<TaggedWord>> process(java.util.List<? extends java.util.List<? extends HasWord>> sentences)
WordToSentenceProcessor
.process
in interface ListProcessor<java.util.List<? extends HasWord>,java.util.List<TaggedWord>>
sentences
- A List of Sentencepublic java.util.List<TaggedWord> tagSentence(java.util.List<? extends HasWord> sentence)
sentence
- sentence to tagpublic java.util.List<TaggedWord> tagSentence(java.util.List<? extends HasWord> sentence, boolean reuseTags)
sentence
- sentence to tagreuseTags
- whether or not to reuse the given tagpublic void tagCoreLabels(java.util.List<CoreLabel> sentence)
public void tagCoreLabels(java.util.List<CoreLabel> sentence, boolean reuseTags)
public static void lemmatize(java.util.List<CoreLabel> sentence, Morphology morpha)
public static java.util.List<java.util.List<HasWord>> tokenizeText(java.io.Reader r)
r
- Reader where untokenized text is readpublic static java.util.List<java.util.List<HasWord>> tokenizeText(java.io.Reader r, TokenizerFactory<? extends HasWord> tokenizerFactory)
r
- Reader where untokenized text is readtokenizerFactory
- Tokenizer. This can be null
in which case
the default English tokenizer (PTBTokenizerFactory) is used.public void tagFromXML(java.io.InputStream input, java.io.Writer writer, java.lang.String... xmlTags)
public void tagFromXML(java.io.Reader input, java.io.Writer writer, java.lang.String... xmlTags)
public void runTaggerStdin(java.io.BufferedReader reader, java.io.BufferedWriter writer, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) throws java.io.IOException
java.io.IOException
public void runTaggerSGML(java.io.BufferedReader reader, java.io.BufferedWriter writer, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) throws java.io.IOException
java.io.IOException
public <X extends HasWord> void runTagger(java.lang.Iterable<java.util.List<X>> document, java.io.BufferedWriter writer, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) throws java.io.IOException
java.io.IOException
public void runTagger(java.io.BufferedReader reader, java.io.BufferedWriter writer, java.lang.String tagInside, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle) throws java.io.IOException
reader
, applies the
tagger to it one sentence at a time (determined using
documentPreprocessor), and writes the output to the given
writer
.
The document is broken into sentences using the sentence
processor determined in the tagger's TaggerConfig.
tagInside
makes the tagger run in XML mode.... If set
to non-empty, instead of processing the document as one large
text blob, it considers each region in between the given tag to
be a separate text blob.java.io.IOException
public java.util.List<? extends HasWord> tagCoreLabelsOrHasWords(java.util.List<? extends HasWord> sentence, Morphology morpha, boolean outputLemmas)
public void tagAndOutputSentence(java.util.List<? extends HasWord> sentence, boolean outputLemmas, Morphology morpha, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle, boolean outputVerbosity, int numSentences, java.lang.String separator, java.io.Writer writer)
public void outputTaggedSentence(java.util.List<? extends HasWord> sentence, boolean outputLemmas, PlainTextDocumentReaderAndWriter.OutputStyle outputStyle, boolean outputVerbosity, int numSentences, java.lang.String separator, java.io.Writer writer)
public static void main(java.lang.String[] args) throws java.lang.Exception
args
- Command-line argumentsjava.io.IOException
- If any file problemsjava.lang.Exception