Tokenizer for Icelandic text (2.0.3)

  Authors

Þorsteinsson, Vilhjálmur

  Item identifier

http://hdl.handle.net/20.500.12537/11

 Project URL

https://github.com/mideind/Tokenizer/releases/tag/2.0.3

 Date issued

2020-01-14

 Type

toolService

 Language(s)

Icelandic

 Description

Tokenizer is a compact pure-Python (2 and 3) executable program and module for tokenizing Icelandic text. It converts input text to streams of tokens, where each token is a separate word, punctuation sign, number/amount, date, e-mail, URL/URI, etc. It also segments the token stream into sentences, considering corner cases such as abbreviations and dates in the middle of sentences.

 Publisher

Miðeind ehf.

 Subject(s)

tokenization token detection sentence detection

 Collection(s)

Clarin IS

 

This item is replaced by a newer submission:

http://hdl.handle.net/20.500.12537/65

List all versions  

Show full item record