Files in this item

 Download all files in item (526.56 KB)
This item is
Publicly Available
and licensed under:
The MIT License (MIT)
Icon
Name
Tokenizer-3.3.2.tar.gz
Size
254.99 KB
Format
application/gzip
Description
Unknown
MD5
ceca503cfa798eb43225163fc56aad76
 Download file  Preview
 File Preview  
  • Tokenizer-3.3.2
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py26 kB
        • tokenizer.py120 kB
        • __init__.py2 kB
        • version.py22 B
        • py.typed0 B
        • Abbrev.conf46 kB
    • setup.py3 kB
    • .gitignore1 kB
    • README.rst39 kB
    • setup.cfg64 B
    • test
      • toktest_large.txt559 kB
      • test_tokenizer_tok.py12 kB
      • toktest_normal.txt12 kB
      • test_index_calculation.py20 kB
      • toktest_normal_gold_expected.txt13 kB
      • toktest_edgecases.txt6 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_detokenize.py2 kB
      • Overview.txt34 kB
      • toktest_large_gold_perfect.txt576 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_sentences.txt21 kB
      • example.txt3 kB
      • test_tokenizer.py97 kB
      • toktest_edgecases_diff.txt751 B
    • .github
    • release.sh251 B
    • LICENSE1 kB
    • MANIFEST.in74 B
    • pax_global_header52 B
Icon
Name
Tokenizer-3.3.2.zip
Size
271.57 KB
Format
application/zip
Description
Unknown
MD5
b3a2d787e407dd8e4cf3fb58dc0746fc
 Download file  Preview
 File Preview  
  • Tokenizer-3.3.2
    • src
      • tokenizer
        • abbrev.py13 kB
        • main.py9 kB
        • definitions.py26 kB
        • tokenizer.py120 kB
        • __init__.py2 kB
        • version.py22 B
        • py.typed0 B
        • Abbrev.conf46 kB
    • setup.py3 kB
    • .gitignore1 kB
    • README.rst39 kB
    • setup.cfg64 B
    • test
      • toktest_large.txt559 kB
      • test_tokenizer_tok.py12 kB
      • toktest_normal.txt12 kB
      • test_index_calculation.py20 kB
      • toktest_normal_gold_expected.txt13 kB
      • toktest_edgecases.txt6 kB
      • toktest_edgecases_gold_expected.txt6 kB
      • test_detokenize.py2 kB
      • Overview.txt34 kB
      • toktest_large_gold_perfect.txt576 kB
      • toktest_large_gold_acceptable.txt583 kB
      • toktest_sentences.txt21 kB
      • example.txt3 kB
      • test_tokenizer.py97 kB
      • toktest_edgecases_diff.txt751 B
    • .github
    • release.sh251 B
    • LICENSE1 kB
    • MANIFEST.in74 B