IGC-2026-MTeng-1 - CC BY 4.0 subset
====================================================

Compiled by: Steinþór Steingrímsson and Hinrik Hafsteinsson
Árni Magnússon Institute for Icelandic Studies

Contents
--------
IGC2026-MTeng-1.jsonl contains 67,447,502 items. Each item is
one Icelandic sentence (or, in a minority of cases, a run of 2-3
consecutive sentences) from the Icelandic Gigaword Corpus 2026 (IGC 2026),
together with its automatic English translation and a pointer back to
exactly where in the corpus it came from.

This file covers the seven IGC 2026 subcorpora released under a
Creative Commons Attribution 4.0 International licence (CC BY 4.0,
https://creativecommons.org/licenses/by/4.0/):

  IGC-Adjud     - judgments of the Icelandic District Courts, the Court
                  of Appeal and the Supreme Court
  IGC-Journals  - scientific and scholarly journals
  IGC-Law       - law, bills, proposals and resolutions submitted to
                  Althingi (the Icelandic parliament)
  IGC-News1     - online news media (subset 1)
  IGC-Parla     - transcripts of speeches from Althingi
  IGC-Social    - social media: blogs, forum posts and tweets
  IGC-Wiki      - Icelandic Wikipedia

(The other two IGC 2026 subcorpora, IGC-Books and IGC-News2, are
released under a separate, restricted IGC licence - see
IGC2026-MTeng-2.jsonl and its accompanying README.)

The Icelandic Gigaword Corpus itself is a product of the Árni Magnússon
Institute for Icelandic Studies: https://igc.arnastofnun.is

Format
------
One JSON object per line (UTF-8 JSON Lines). Fields:

  source            the Icelandic sentence (or short run of sentences)
  target            its automatic English translation
  perplexity        perplexity of `source` under the KenLM model used for
                     filtering (see below)
  filename          the IGC 2026 TEI source-XML file this text came from
  segment_id        the xml:id of the paragraph/segment (<p> for most
                     subcorpora, <seg> for IGC-Parla) that `source` was
                     taken from
  sentence_position where within that paragraph/segment this text sits,
                     as "i/N" (the i-th of N sentences in that
                     paragraph/segment) or "i-j/N" when `source` is a run
                     of consecutive sentences i through j. Comparing the
                     position numbers actually present for a given
                     filename+segment_id across this dataset shows which
                     sentences of that paragraph were NOT included here
                     (e.g. a gap between "1/5" and "4/5" means sentences
                     2 and 3 were filtered out or otherwise did not
                     survive the pipeline below).

If the exact same source sentence was found in more than one place in
the corpus, the item has no single filename/segment_id/sentence_position;
instead it carries a "matches" array, e.g.:

  {"source": "...", "target": "...", "perplexity": ...,
   "matches": [{"filename": "...", "segment_id": "...", "sentence_position": "..."},
               {"filename": "...", "segment_id": "...", "sentence_position": "..."}]}

How this dataset was compiled
------------------------------
1. Icelandic sentences were extracted from the IGC 2026 unannotated
   subcorpora (paragraphs/segments split into sentences).
2. Each sentence was translated from Icelandic into English using
   TranslateGemma-27b-it, 6-bit quantized (i1-GGUF, Q6_K):
   https://huggingface.co/mradermacher/translategemma-27b-it-i1-GGUF
3. The resulting pairs were filtered:
   - by perplexity of the Icelandic source sentence under a KenLM
     language model:
     Steinþór Steingrímsson. 2026. IGC-2024 5-gram: Icelandic KenLM
     language model trained on the Icelandic Gigaword Corpus (2024).
     CLARIN-IS.
   - by language identification, using langdetect with a custom
     Icelandic profile: https://github.com/steinst/langdetect_profiles
     (to remove sentences not actually in Icelandic)


If you use this dataset. Please cite the dataset itself, as well as the Icelandic Gigaword Corpus paper:

Steinþór Steingrímsson and Hinrik Hafsteinsson. 2026. IGC2026-MTeng 1. CLARIN-IS. http://hdl.handle.net/20.500.12537/401.

Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson and Jón Guðnason. 2018. Risamálheild: A Very Large Icelandic Text Corpus. Proceedings of LREC 2018. Myazaki, Japan. p. 4361-4366.


BibTeX:

@misc{20.500.12537/401,
   title = {{IGC2026-MTeng 1}},
   author = {Steingr{\'{\i}}msson, Stein{\th}{\'o}r and and Hafsteinsson, Hinrik},
   url = {http://hdl.handle.net/20.500.12537/401},
   note = {{CLARIN}-{IS}},
   year = {2026} 
}

@inproceedings{steingrimsson-etal-2018-risamalheild,
    title = {{Risam{\'a}lheild: A Very Large {I}celandic Text Corpus}},
    author = {Steingr{\'\i}msson, Stein{\th}{\'o}r  and
      Helgad{\'o}ttir, Sigr{\'u}n  and
      R{\"o}gnvaldsson, Eir{\'\i}kur  and
      Barkarson, Starka{\dh}ur  and
      Gu{\dh}nason, J{\'o}n},
    booktitle = "Proceedings of the Eleventh International Conference on Language Resources and Evaluation ({LREC} 2018)",
    month = may,
    year = "2018",
    address = "Miyazaki, Japan",
    publisher = "European Language Resources Association (ELRA)",
    url = "https://aclanthology.org/L18-1690",
}