Tokenizer for Icelandic text (3.6.4) (2026-09-30)

 
Clarin IS Repository
  Authors
Þorsteinsson, Vilhjálmur ; et al.show everyone Þorsteinsson, Vilhjálmur ; Óladóttir, Hulda ; Þórðarson, Sveinbjörn ; Ragnarsson, Pétur Orri ; Jónsson, Haukur Páll ; Eyjólfsson, Logi
  Item identifier
http://hdl.handle.net/20.500.12537/404
 Project URL
https://github.com/mideind/Tokenizer/releases/tag/v3.6.4
 Date issued
2024-07-10
 Type
toolService
 Language(s)
Icelandic
 Description
Tokenizer is a compact pure-Python 3 executable program and module for tokenizing Icelandic text. It converts input text to streams of tokens, where each token is a separate word, punctuation sign, number/amount, date, e-mail, URL/URI, etc. It also segments the token stream into sentences, considering corner cases such as abbreviations and dates in the middle of sentences. More information at: https://github.com/mideind/Tokenizer Tokenizer er pakki fyrir Python 3, ásamt skipanalínutóli, sem sér um tilreiðslu íslensks texta. Pakkinn umbreytir inntakstexta í tókastraum. Hver tóki er stakt orð, greinarmerki, tala/upphæð, dags-/tímasetning, netfang, vefslóð o.s.frv. Tólið skiptir tókastraumnum einnig í setningar og tekur tillit til jaðartilvika eins og skammstafana og dagsetninga í miðjum setningum. Frekari upplýsingar á: https://github.com/mideind/Tokenizer
 Publisher
Miðeind ehf.
 Acknowledgement

Ministry of Education, Science and Culture

Project code: Tokenizer (I3)

Project name: Language Technology for Icelandic 2019-2023

 Subject(s)
tokenization sentence detection token detection
 Collection(s)
Clarin IS
 Other versions
Show full item record
 

 Files in this item

 Download all files in item (561.19 KB)
This item is
Publicly Available
and licensed under:
The MIT License (MIT)
Icon
Name
Tokenizer-3.6.4.zip
Size
290.36 KB
Format
application/zip
Description
ZIP version
MD5
4609fad993833b611853080c9fd19605
 Download file  Preview
 File Preview  
  • Tokenizer-3.6.4
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py27 kB
        • tokenizer.py138 kB
        • __init__.py2 kB
        • py.typed0 B
        • Abbrev.conf46 kB
    • README.md37 kB
    • .gitignore1 kB
    • pyproject.toml3 kB
    • CLAUDE.md3 kB
    • test
      • toktest_edgecases.txt6 kB
      • toktest_large_gold_perfect.txt576 kB
      • test_composite_glyphs.py8 kB
      • test_helper_functions.py2 kB
      • test_dashes.py14 kB
      • Overview.txt34 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_index_calculation.py21 kB
      • toktest_edgecases_diff.txt751 B
      • test_cli.py7 kB
      • toktest_sentences.txt21 kB
      • test_abbrev.py1 kB
      • toktest_large.txt559 kB
      • test_tokenizer.py101 kB
      • toktest_normal.txt12 kB
      • toktest_normal_gold_expected.txt13 kB
      • example.txt3 kB
      • test_detokenize.py2 kB
      • test_tokenizer_tok.py18 kB
    • .github
      • workflows
        • release.yml1 kB
        • python-package.yml907 B
    • LICENSE.txt1 kB
    • perf.py994 B
    • MANIFEST.in104 B
Icon
Name
Tokenizer-3.6.4.tar.gz
Size
270.82 KB
Format
application/gzip
Description
GZipped TAR file
MD5
84678619534afdd73440b78a6da7dc9b
 Download file  Preview
 File Preview  
  • Tokenizer-3.6.4
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py27 kB
        • tokenizer.py138 kB
        • __init__.py2 kB
        • py.typed0 B
        • Abbrev.conf46 kB
    • README.md37 kB
    • .gitignore1 kB
    • pyproject.toml3 kB
    • CLAUDE.md3 kB
    • test
      • toktest_edgecases.txt6 kB
      • toktest_large_gold_perfect.txt576 kB
      • test_composite_glyphs.py8 kB
      • test_helper_functions.py2 kB
      • test_dashes.py14 kB
      • Overview.txt34 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_index_calculation.py21 kB
      • toktest_edgecases_diff.txt751 B
      • test_cli.py7 kB
      • toktest_sentences.txt21 kB
      • test_abbrev.py1 kB
      • toktest_large.txt559 kB
      • test_tokenizer.py101 kB
      • toktest_normal.txt12 kB
      • toktest_normal_gold_expected.txt13 kB
      • example.txt3 kB
      • test_detokenize.py2 kB
      • test_tokenizer_tok.py18 kB
    • .github
      • workflows
        • release.yml1 kB
        • python-package.yml907 B
    • LICENSE.txt1 kB
    • perf.py994 B
    • MANIFEST.in104 B
    • pax_global_header52 B

Partners, Coordination, Funding

Repository

More

CLARIN-IS is fully supported by the Ministry of Culture and Business Affairs

Copyright (c) 2023. Arni Magnusson Institute for Icelandic Studies. All rights reserved.