IGC2024-Filtered-1

Authors: Jón Friðrik Daðason, Steinþór Steingrímsson and Hinrik Hafsteinsson
Item identifier: http://hdl.handle.net/20.500.12537/381
Published by: The Árni Magnússon Institute for Icelandic Studies, March 31, 2026

Description in English
---
This is a JSONL version of the 2024 release of the Icelandic Gigaword Corpus (IGC), prepared for language model training. The archive contains training and validation sets of unannotated, CC-BY-licensed documents from the IGC.
The corpus has been filtered, deduplicated, and normalized to remove content unsuitable for training. Documents were excluded if they contained unintended code (e.g., HTML, CSS, or JavaScript), optical character recognition errors, character encoding issues, highly repetitive n-gram sequences, or a very low word count, or if they were duplicates or near-duplicates of other documents in the IGC. In addition, recurring boilerplate text, such as lists of related articles and social media sharing links, has been removed where possible, along with author bylines and image captions. The remaining text has been normalized for whitespace, non-printable and control characters, and other similar issues.
 

Lýsing á íslensku
---
Þetta er útgáfa af Íslensku risamálheildinni (RMH) frá 2024 á JSONL sniði, ætluð til þjálfunar á mállíkönum. Hún samanstendur af ómörkuðum skjölum úr RMH með CC-BY leyfi sem hefur verið skipt í þjálfunar- og þróunargögn.
Málheildin hefur verið síuð og normalíseruð til að fjarlægja efni sem hentar illa til þjálfunar. Skjölum var sleppt ef þau innihéldu forritunarkóða (t.d. HTML, CSS eða JavaScript), ljóslestrarvillur, stafasettsvandamál, hátt hlutfall af endurteknum n-stæðum, eða ef þau voru mjög stutt. Endurteknar útgáfur af sama skjali voru einnig fjarlægðar. Þar að auki hefur fastatexti (e. boilerplate text), eins og listar yfir tengdar greinar og hlekkir til að deila efni á samfélagsmiðlum, auk höfundalína og myndatexta, verið fjarlægður þar sem kostur var á. Textinn var að lokum normalíseraður með tilliti til bilstafa, ósýnilegra stafa, stýristafa og annarra svipaðra atriða.


If you use this version of the corpus, please cite the following papers:

1) Steinþór Steingrímsson, Sigrún Helgadóttir, Eiríkur Rögnvaldsson, Starkaður Barkarson and Jón Guðnason. 2018. Risamálheild: A Very Large Icelandic Text Corpus. In Proceedings of LREC 2018. Myazaki, Japan. p. 4361-4366.
2) Jón Friðrik Daðason and Steinþór Steingrímsson. 2026. Optimized for AI: Curating the Icelandic Gigaword Corpus for Stable LLM Training. In Proceedings of In Proceedings of the Workshop on Challenges in the Management of Large Corpora (CMLC-12), Palma de Mallorca. European Language Resource Association.