Tokenizer for Icelandic text (3.5.5) (2025-12-09)

 
Clarin IS Repository
  Authors
Þorsteinsson, Vilhjálmur ; et al.show everyone Þorsteinsson, Vilhjálmur ; Óladóttir, Hulda ; Þórðarson, Sveinbjörn ; Ragnarsson, Pétur Orri ; Jónsson, Haukur Páll ; Eyjólfsson, Logi
  Item identifier
http://hdl.handle.net/20.500.12537/372
 Project URL
https://github.com/mideind/Tokenizer/releases/tag/3.5.5
 Date issued
2024-08-15
 Type
toolService
 Language(s)
Icelandic
 Description
Tokenizer is a compact pure-Python 3 executable program and module for tokenizing Icelandic text. It converts input text to streams of tokens, where each token is a separate word, punctuation sign, number/amount, date, e-mail, URL/URI, etc. It also segments the token stream into sentences, considering corner cases such as abbreviations and dates in the middle of sentences. More information at: https://github.com/mideind/Tokenizer Tokenizer er pakki fyrir Python 3, ásamt skipanalínutóli, sem sér um tilreiðslu íslensks texta. Pakkinn umbreytir inntakstexta í tókastraum. Hver tóki er stakt orð, greinarmerki, tala/upphæð, dags-/tímasetning, netfang, vefslóð o.s.frv. Tólið skiptir tókastraumnum einnig í setningar og tekur tillit til jaðartilvika eins og skammstafana og dagsetninga í miðjum setningum. Frekari upplýsingar á: https://github.com/mideind/Tokenizer
 Publisher
Miðeind ehf.
 Acknowledgement

Ministry of Education, Science and Culture

Project code: Tokenizer (I3)

Project name: Language Technology for Icelandic 2019-2023

 Subject(s)
tokenization sentence detection token detection
 Collection(s)
Clarin IS
 Other versions
Show full item record
 

 Files in this item

 Download all files in item (560.28 KB)
This item is
Publicly Available
and licensed under:
The MIT License (MIT)
Icon
Name
Tokenizer-3.5.5.zip
Size
289.69 KB
Format
application/zip
Description
Unknown
MD5
0bd11287ef446264fe392eda81ee755e
 Download file  Preview
 File Preview  
  • Tokenizer-3.5.5
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py28 kB
        • tokenizer.py135 kB
        • __init__.py2 kB
        • Abbrev.conf46 kB
    • README.md37 kB
    • .gitignore1 kB
    • pyproject.toml3 kB
    • CLAUDE.md3 kB
    • test
      • toktest_edgecases.txt6 kB
      • toktest_large_gold_perfect.txt576 kB
      • test_composite_glyphs.py8 kB
      • test_helper_functions.py2 kB
      • test_dashes.py14 kB
      • Overview.txt34 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_index_calculation.py22 kB
      • toktest_edgecases_diff.txt751 B
      • test_cli.py7 kB
      • toktest_sentences.txt21 kB
      • test_abbrev.py1 kB
      • toktest_large.txt559 kB
      • test_tokenizer.py103 kB
      • toktest_normal.txt12 kB
      • toktest_normal_gold_expected.txt13 kB
      • example.txt3 kB
      • test_detokenize.py2 kB
      • test_tokenizer_tok.py18 kB
    • .github
    • LICENSE.txt1 kB
    • perf.py977 B
    • MANIFEST.in104 B
Icon
Name
Tokenizer-3.5.5.tar.gz
Size
270.59 KB
Format
application/gzip
Description
Unknown
MD5
3cba60656f8c51fa219b051930fd8e33
 Download file  Preview
 File Preview  
  • Tokenizer-3.5.5
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py28 kB
        • tokenizer.py135 kB
        • __init__.py2 kB
        • Abbrev.conf46 kB
    • README.md37 kB
    • .gitignore1 kB
    • pyproject.toml3 kB
    • CLAUDE.md3 kB
    • test
      • toktest_edgecases.txt6 kB
      • toktest_large_gold_perfect.txt576 kB
      • test_composite_glyphs.py8 kB
      • test_helper_functions.py2 kB
      • test_dashes.py14 kB
      • Overview.txt34 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_index_calculation.py22 kB
      • toktest_edgecases_diff.txt751 B
      • test_cli.py7 kB
      • toktest_sentences.txt21 kB
      • test_abbrev.py1 kB
      • toktest_large.txt559 kB
      • test_tokenizer.py103 kB
      • toktest_normal.txt12 kB
      • toktest_normal_gold_expected.txt13 kB
      • example.txt3 kB
      • test_detokenize.py2 kB
      • test_tokenizer_tok.py18 kB
    • .github
    • LICENSE.txt1 kB
    • perf.py977 B
    • MANIFEST.in104 B
    • pax_global_header52 B

Partners, Coordination, Funding

Repository

More

CLARIN-IS is fully supported by the Ministry of Culture and Business Affairs

Copyright (c) 2023. Arni Magnusson Institute for Icelandic Studies. All rights reserved.