IGC-2026-MTeng-2 - IGC licence subset
=======================================================

Compiled by: Steinþór Steingrímsson and Hinrik Hafsteinsson
Árni Magnússon Institute for Icelandic Studies

Contents
--------
IGC2026-MTeng-2.jsonl contains 40,472,616 items. Each item is
one Icelandic sentence (or, in a minority of cases, a run of 2-3
consecutive sentences) from the Icelandic Gigaword Corpus 2026 (IGC 2026),
together with its automatic English translation and a pointer back to
exactly where in the corpus it came from.

This file covers the two IGC 2026 subcorpora released under the
restricted IGC licence
(https://repository.clarin.is/licenses/userlicense_igc_restricted_download_en.pdf):

  IGC-Books  - published books
  IGC-News2  - online news media (subset 2)

Redistribution of this file is subject to that IGC licence, not CC BY
4.0. (The other seven IGC 2026 subcorpora are released under CC BY 4.0 -
see IGC2026-MTeng-1.jsonl and its accompanying README.)

The Icelandic Gigaword Corpus itself is a product of the Árni Magnússon
Institute for Icelandic Studies: https://igc.arnastofnun.is

Format
------
One JSON object per line (UTF-8 JSON Lines). Fields:

  source            the Icelandic sentence (or short run of sentences)
  target            its automatic English translation
  perplexity        perplexity of `source` under the KenLM model used for
                     filtering (see below)
  filename          the IGC 2026 TEI source-XML file this text came from
  segment_id        the xml:id of the paragraph (<p>) that `source` was
                     taken from
  sentence_position where within that paragraph this text sits, as "i/N"
                     (the i-th of N sentences in that paragraph) or
                     "i-j/N" when `source` is a run of consecutive
                     sentences i through j. Comparing the position
                     numbers actually present for a given
                     filename+segment_id across this dataset shows which
                     sentences of that paragraph were NOT included here
                     (e.g. a gap between "1/5" and "4/5" means sentences
                     2 and 3 were filtered out or otherwise did not
                     survive the pipeline below).

If the exact same source sentence was found in more than one place in
the corpus, the item has no single filename/segment_id/sentence_position;
instead it carries a "matches" array, e.g.:

  {"source": "...", "target": "...", "perplexity": ...,
   "matches": [{"filename": "...", "segment_id": "...", "sentence_position": "..."},
               {"filename": "...", "segment_id": "...", "sentence_position": "..."}]}

How this dataset was compiled
------------------------------
1. Icelandic sentences were extracted from the IGC 2026 unannotated
   subcorpora (paragraphs split into sentences).
2. Each sentence was translated from Icelandic into English using
   TranslateGemma-27b-it, 6-bit quantized (i1-GGUF, Q6_K):
   https://huggingface.co/mradermacher/translategemma-27b-it-i1-GGUF
3. The resulting pairs were filtered:
   - by perplexity of the Icelandic source sentence under a KenLM
     language model:
     Steinþór Steingrímsson. 2026. IGC-2024 5-gram: Icelandic KenLM
     language model trained on the Icelandic Gigaword Corpus (2024).
     CLARIN-IS.
   - by language identification, using langdetect with a custom
     Icelandic profile: https://github.com/steinst/langdetect_profiles
     (to remove sentences not actually in Icelandic)


If you use this dataset. Please cite the dataset itself, as well as the Icelandic Gigaword Corpus paper:

Steinþór Steingrímsson and Hinrik Hafsteinsson. 2026. IGC2026-MTeng 2. CLARIN-IS. http://hdl.handle.net/20.500.12537/402.

Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson and Jón Guðnason. 2018. Risamálheild: A Very Large Icelandic Text Corpus. Proceedings of LREC 2018. Myazaki, Japan. p. 4361-4366.


BibTeX:

@misc{20.500.12537/402,
   title = {{IGC2026-MTeng 2}},
   author = {Steingr{\'{\i}}msson, Stein{\th}{\'o}r and and Hafsteinsson, Hinrik},
   url = {http://hdl.handle.net/20.500.12537/402},
   note = {{CLARIN}-{IS}},
   year = {2026} 
}

@inproceedings{steingrimsson-etal-2018-risamalheild,
    title = {{Risam{\'a}lheild: A Very Large {I}celandic Text Corpus}},
    author = {Steingr{\'\i}msson, Stein{\th}{\'o}r  and
      Helgad{\'o}ttir, Sigr{\'u}n  and
      R{\"o}gnvaldsson, Eir{\'\i}kur  and
      Barkarson, Starka{\dh}ur  and
      Gu{\dh}nason, J{\'o}n},
    booktitle = "Proceedings of the Eleventh International Conference on Language Resources and Evaluation ({LREC} 2018)",
    month = may,
    year = "2018",
    address = "Miyazaki, Japan",
    publisher = "European Language Resources Association (ELRA)",
    url = "https://aclanthology.org/L18-1690",
}