Stanford Log-linear Part-Of-Speech Tagger

About | Questions | Mailing lists | Download | Extensions | Release history | FAQ

About

A Part-Of-Speech Tagger (POS Tagger) is a piece of software that reads text in some language and assigns parts of speech to each word (and other token), such as noun, verb, adjective, etc., although generally computational applications use more fine-grained POS tags like 'noun-plural'. This software is a Java implementation of the log-linear part-of-speech taggers described in these papers (if citing just one paper, cite the 2003 one):

Kristina Toutanova and Christopher D. Manning. 2000. Enriching the Knowledge Sources Used in a Maximum Entropy Part-of-Speech Tagger. In Proceedings of the Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-2000), pp. 63-70.
Kristina Toutanova, Dan Klein, Christopher Manning, and Yoram Singer. 2003. Feature-Rich Part-of-Speech Tagging with a Cyclic Dependency Network. In Proceedings of HLT-NAACL 2003, pp. 252-259.

The tagger was originally written by Kristina Toutanova. Since that time, Dan Klein, Christopher Manning, William Morgan, Anna Rafferty, Michel Galley, and John Bauer have improved its speed, performance, usability, and support for other languages.

The system requires Java 1.8+ to be installed. Depending on whether you're running 32 or 64 bit Java and the complexity of the tagger model, you'll need somewhere between 60 and 200 MB of memory to run a trained tagger (i.e., you may need to give java an option like java -mx200m). Plenty of memory is needed to train a tagger. It again depends on the complexity of the model but at least 1GB is usually needed, often more.

Several downloads are available. The basic download contains two trained tagger models for English. The full download contains three trained English tagger models, an Arabic tagger model, a Chinese tagger model, a French tagger model, and a German tagger model. Both versions include the same source and other required files. The tagger can be retrained on any language, given POS-annotated training text for the language.

Part-of-speech name abbreviations: The English taggers use the Penn Treebank tag set. Here are some links to documentation of the Penn Treebank English POS tag set: 1993 Computational Linguistics article in PDF, AMALGAM page, Aoife Cahill's list. See the included README-Models.txt in the models directory for more information about the tagsets for the other languages.

The tagger is licensed under the GNU General Public License (v2 or later). Source is included. Source is included. The package includes components for command-line invocation, running as a server, and a Java API. The tagger code is dual licensed (in a similar manner to MySQL, etc.). Open source licensing is under the full GPL, which allows many free uses. For distributors of proprietary software, commercial licensing is available. If you don't need a commercial license, but would like to support maintenance of these tools, we welcome gift funding.


Questions

For documentation, first take a look at the included README.txt.

Matthew Jockers has kindly produced an example and tutorial for running the tagger. This particularly concentrates on command-line usage with XML and (Mac OS X) xGrid.

Galal Aly wrote a tagging tutorial focused on usage in Java with Eclipse.

For more details, look at our included javadocs, particularly the javadoc for MaxentTagger.

There is a brief FAQ. Additional questions, feedback, and bugs/bug-fixes) can be sent to our mailing lists.


Mailing Lists

We have 3 mailing lists for the Stanford POS Tagger, all of which are shared with other JavaNLP tools (with the exclusion of the parser). Each address is at @lists.stanford.edu:

  1. java-nlp-user This is the best list to post to in order to ask questions, make announcements, or for discussion among JavaNLP users. You have to subscribe to be able to use it. Join the list via this webpage or by emailing java-nlp-user-join@lists.stanford.edu. (Leave the subject and message body empty.) You can also look at the list archives.
  2. java-nlp-announce This list will be used only to announce new versions of Stanford JavaNLP tools. So it will be very low volume (expect 1-3 message a year). Join the list via via this webpage or by emailing java-nlp-announce-join@lists.stanford.edu. (Leave the subject and message body empty.)
  3. java-nlp-support This list goes only to the software maintainers. It's a good address for licensing questions, etc. For general use and support questions, please join and use java-nlp-user. You cannot join java-nlp-support, but you can mail questions to java-nlp-support@lists.stanford.edu.

Download

Download basic English Stanford Tagger version 3.5.0 [22 MB]

Download full Stanford Tagger version 3.5.0 [123 MB]


The basic download is a 22 MB zipped file with support for tagging English. The full download is a 123 MB zipped file, which includes additional English models and trained models for Arabic, Chinese, French, Spanish, and German. In both cases most of the file size is due to the trained model files. The only difference between the two downloads is the number of trained models included. If you unpack the tar file, you should have everything needed. This software provides a GUI demo, a command-line interface, and an API. Simple scripts are included to invoke the tagger. For more information on use, see the included README.txt.


Extensions: Packages by others using Stanford Tagger


Release History


VersionDateDescription
3.5.0 2014-10-26 Upgrade to Java 8
English / Full
3.4.1 2014-08-27 Add Spanish model
English / Full
3.4 2014-06-16 French model uses CC tagset
English / Full
3.3.1 2014-01-04 Bugfix release
English / Full
3.3.0 2013-11-12 imperatives included in English model
English / Full
3.2.0 2013-06-20 improved speed & size of all models
English / Full
3.1.5 2013-04-04 ctb7 model, -nthreads option, improved speed
English / Full
3.1.4 2012-11-11 Improved Chinese model
English / Full
3.1.3 2012-07-09 Minor bug fixes
English / Full
3.1.2 2012-05-22 Included some "tech" words in the latest model
English / Full
3.1.1 2012-03-09 Caseless models added for English
English / Full
3.1.0 2012-01-06 French tagger added, tagging speed improved
English / Full
3.0.4 2011-09-14 Compatible with other recent Stanford releases.
English / Full
3.0.32011-06-19 Compatible with other recent Stanford releases.
English / Full
3.0.22011-05-15 Addition of TSV input format.
English / Full
3.0.12011-04-20 Faster Arabic and German models. Compatible with other recent Stanford releases.
English / Full
3.02010-05-21 Tagger is now re-entrant. New tagger objects are loaded with tagger = new MaxentTagger(path) and then used with tagger.tagMethod...
English / Full
2.02009-12-24 An order of magnitude faster, slightly more accurate best model, more options for training and deployment.
English / Full
1.62008-09-28 A fraction better, a fraction faster, more flexible model specification, and quite a few less bugs.
English / Full
1.5.12008-06-06 Tagger properties are now saved with the tagger, making taggers more portable; tagger can be trained off of treebank data or tagged text; fixes classpath bugs in 2 June 2008 patch; new foreign language taggers released on 7 July 2008 and packaged with 1.5.1.
English / Full / Updated models
1.52008-05-21 Added taggers for several languages, support for reading from and writing to XML, better support for changing the encoding, distributional similarity options, and many more small changes; patched on 2 June 2008 to fix a bug with tagging pre-tokenized text.
English / Full
1.02006-01-10 First cleaned-up release after Kristina graduated.
Old School
0.12004-08-16 First release.