Tokenizer for Icelandic text (3.4.1) (2022-05-31)

Clarin IS Repository

Authors: Þorsteinsson, Vilhjálmur ; et al.show everyone
Þorsteinsson, Vilhjálmur ; Óladóttir, Hulda ; Þórðarson, Sveinbjörn ; Ragnarsson, Pétur Orri ; Jónsson, Haukur Páll ; Eyjólfsson, Logi

Item identifier: http://hdl.handle.net/20.500.12537/219

Project URL: https://github.com/mideind/Tokenizer/releases/tag/3.4.1

Date issued: 2022-05-31

Type: toolService

Language(s): Icelandic

Description: Tokenizer is a compact pure-Python (2.7 and 3) executable program and module for tokenizing Icelandic text. It converts input text to streams of tokens, where each token is a separate word, punctuation sign, number/amount, date, e-mail, URL/URI, etc. It also segments the token stream into sentences, considering corner cases such as abbreviations and dates in the middle of sentences. More information at: https://github.com/mideind/Tokenizer

Tokenizer er pakki fyrir Python 2.7 og 3, ásamt skipanalínutóli, sem sér um tilreiðslu íslensks texta. Pakkinn umbreytir inntakstexta í tókastraum. Hver tóki er stakt orð, greinarmerki, tala/upphæð, dags-/tímasetning, netfang, vefslóð o.s.frv. Tólið skiptir tókastraumnum einnig í setningar og tekur tillit til jaðartilvika eins og skammstafana og dagsetninga í miðjum setningum. Frekari upplýsingar á: https://github.com/mideind/Tokenizer

Publisher: Miðeind ehf.

Subject(s): tokenization sentence detection token detection

Collection(s): Clarin IS

This item is replaced by a newer submission:

http://hdl.handle.net/20.500.12537/262

Show full item record

Files in this item

Download all files in item (532.29 KB)

This item is

Publicly Available

and licensed under:
The MIT License (MIT)

Name: Tokenizer-3.4.1.tar.gz
Size: 258.02 KB
Format: application/gzip
Description: Unknown
MD5: 91bb9fa5417e9aca9041953a9b3c486a

Download file Preview

File Preview

Tokenizer-3.4.1
- src
  - tokenizer
    - abbrev.py13 kB
    - main.py9 kB
    - definitions.py27 kB
    - tokenizer.py125 kB
    - __init__.py2 kB
    - version.py22 B
    - py.typed0 B
    - Abbrev.conf46 kB
- setup.py3 kB
- .gitignore1 kB
- README.rst39 kB
- setup.cfg64 B
- test
  - toktest_large.txt559 kB
  - test_tokenizer_tok.py18 kB
  - toktest_normal.txt12 kB
  - test_index_calculation.py20 kB
  - toktest_normal_gold_expected.txt13 kB
  - toktest_edgecases.txt6 kB
  - toktest_edgecases_gold_expected.txt6 kB
  - test_detokenize.py2 kB
  - Overview.txt34 kB
  - toktest_large_gold_perfect.txt576 kB
  - toktest_large_gold_acceptable.txt583 kB
  - toktest_sentences.txt21 kB
  - example.txt3 kB
  - test_tokenizer.py102 kB
  - toktest_edgecases_diff.txt751 B
- .github
  - workflows
    - python-package.yml1 kB
- release.sh251 B
- LICENSE1 kB
- MANIFEST.in74 B
- pax_global_header52 B

Name: Tokenizer-3.4.1.zip
Size: 274.26 KB
Format: application/zip
Description: Unknown
MD5: 1578d146018d976c5eb4be0e346784f5

Download file Preview

File Preview

Tokenizer-3.4.1
- src
  - tokenizer
    - abbrev.py13 kB
    - main.py9 kB
    - definitions.py27 kB
    - tokenizer.py125 kB
    - __init__.py2 kB
    - version.py22 B
    - py.typed0 B
    - Abbrev.conf46 kB
- setup.py3 kB
- .gitignore1 kB
- README.rst39 kB
- setup.cfg64 B
- test
  - toktest_large.txt559 kB
  - test_tokenizer_tok.py18 kB
  - toktest_normal.txt12 kB
  - test_index_calculation.py20 kB
  - toktest_normal_gold_expected.txt13 kB
  - toktest_edgecases.txt6 kB
  - toktest_edgecases_gold_expected.txt6 kB
  - test_detokenize.py2 kB
  - Overview.txt34 kB
  - toktest_large_gold_perfect.txt576 kB
  - toktest_large_gold_acceptable.txt583 kB
  - toktest_sentences.txt21 kB
  - example.txt3 kB
  - test_tokenizer.py102 kB
  - toktest_edgecases_diff.txt751 B
- .github
  - workflows
    - python-package.yml1 kB
- release.sh251 B
- LICENSE1 kB
- MANIFEST.in74 B

Partners, Coordination, Funding

Repository

More

CLARIN-IS is fully supported by the Ministry of Culture and Business Affairs

Copyright (c) 2023. Arni Magnusson Institute for Icelandic Studies. All rights reserved.