# IGC 5-gram — Icelandic KenLM Language Model

A 5-gram back-off language model for Icelandic, trained with
[KenLM](https://github.com/kpu/kenlm) on the **Icelandic Gigaword Corpus (IGC),
2024 version** (*Risamálheild*).

- **Author:** Steinþór Steingrímsson
- **Training data:** Icelandic Gigaword Corpus (IGC), 2024 version
- **Model type:** 5-gram, interpolated modified Kneser–Ney smoothing
- **Format:** KenLM binary — trie with quantization and array-compressed pointers
- **Toolkit:** KenLM (`lmplz` / `build_binary`)
- **Built:** April 2026
- **License:** [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)
- **Distributed by:** CLARIN-IS
- **Handle:** [http://hdl.handle.net/20.500.12537/399](http://hdl.handle.net/20.500.12537/399)

---

## 1. What is in this release

| File | Size | Description |
|---|---|---|
| `igc_5gram_pruned_quantized_trie.bin` | 3.9 GiB | The model. KenLM binary, quantized array trie. |
| `prepare_text.py` | — | Tokenizes and lowercases raw Icelandic text so it matches the training data (§5). |
| `score_text.py` | — | Computes perplexity of prepared text under the model (§6). |
| `requirements.txt` | — | Python dependencies for the two scripts. |
| `README.md` | — | This file. |

The model is memory-mapped at query time and needs roughly 4 GiB, so it runs
comfortably on an ordinary workstation. It is pruned (singleton 3-, 4- and
5-grams removed) and quantized to 8 bits; see §3 for exactly what that means.

### N-gram counts

| Order | N-grams |
|---:|---:|
| 1 | 8,746,874 |
| 2 | 122,876,081 |
| 3 | 151,245,526 |
| 4 | 219,313,864 |
| 5 | 228,161,039 |
| **Total** | **730,343,384** |

The vocabulary is **8,746,874 types**, including `<unk>`, `<s>` and `</s>`.
Unigrams and bigrams are not pruned, so every token seen in the training corpus
is in the vocabulary — there is no frequency cutoff on the vocabulary itself.

---

## 2. Training data and preprocessing

The model is trained on the **Icelandic Gigaword Corpus (IGC), 2024 version**, a
large corpus of Icelandic text compiled at the Árni Magnússon Institute for
Icelandic Studies.

The corpus was extracted to a single plain-text file (UTF-8) with the following
properties:

- **One segment per line.** KenLM wraps each line in `<s> … </s>`, so each line
  is treated as one sentence-like unit.
- **Tokenized**, with punctuation split off as separate tokens
  (`mál þetta , sem dómtekið var 13. febrúar 2015 , er höfðað …`).
- **Lowercased.** The model is case-insensitive; there are no uppercase forms
  in the vocabulary.

Size of the training text: **119,737,776 lines / 2,275,021,331 whitespace-separated
tokens** (13.2 GiB).

**This preprocessing matters.** Any text scored with this model must be
tokenized and lowercased the same way, or the OOV rate and perplexity will be
misleadingly high. `prepare_text.py` (§5) does this for you.

---

## 3. How the model was built

KenLM was used at commit
[`4cb443e`](https://github.com/kpu/kenlm/commit/4cb443e60b7bf2c0ddf3c745378f76cb59e254e5)
(`windows-1418-g4cb443e`, 2025-03-30), built with a maximum supported order of 6.
Estimation and binarization were run on 2026-04-29/30 on a 20-core machine with
128 GiB of RAM.

The build is two steps: estimate an ARPA model from the corpus, then convert it
to the quantized trie. The intermediate ARPA file is not part of this release.

```bash
# 1. Estimate the pruned 5-gram model
bin/lmplz -o 5 -S 60% -T /tmp --prune 0 0 1 1 \
  < igc_corpus.txt \
  > igc_5gram_pruned.arpa

# 2. Convert to a quantized, array-compressed trie
bin/build_binary trie -q 8 -b 8 -a 64 \
  igc_5gram_pruned.arpa \
  igc_5gram_pruned_quantized_trie.bin
```

### 3.1 Estimation parameters (`lmplz`)

| Flag | Value | Meaning |
|---|---|---|
| `-o` | `5` | Model order — up to 5-grams. |
| `-S` | `60%` | Memory budget for the on-disk sort: 60 % of physical RAM. |
| `-T` | `/tmp` | Directory for temporary sort files. |
| `--prune` | `0 0 1 1` | Pruning thresholds per order. |

Smoothing is KenLM's default: **interpolated modified Kneser–Ney**. No
`--discount_fallback` was used, and no vocabulary limit (`--limit_vocab_file`,
`--vocab_estimate`) was applied.

On `--prune 0 0 1 1`: KenLM takes one threshold per order and **applies the last
value to all remaining orders**, so with `-o 5` this is equivalent to
`--prune 0 0 1 1 1`. In effect: keep all unigrams and bigrams, and discard 3-,
4- and 5-grams that occur only once (count ≤ 1).

### 3.2 Binarization parameters (`build_binary`)

| Flag | Value | Meaning |
|---|---|---|
| *(type)* | `trie` | Trie data structure, rather than the default probing hash table. |
| `-q` | `8` | Quantize n-gram probabilities to 8 bits. |
| `-b` | `8` | Quantize back-off weights to 8 bits. |
| `-a` | `64` | Compress trie pointers with an array of offsets, at most 64 bits encoded by the array. |

Quantization and pointer compression are what bring the model down to 3.9 GiB.
They are lossy: probabilities and back-off weights are rounded to one of 256
bins per order, so scores differ slightly from the unquantized ARPA model.

---

## 4. Installation

The model is read by the standard KenLM tools and bindings. The two helper
scripts in this release need Python 3.8 or newer.

```bash
pip install -r requirements.txt
```

That installs:

- **[tokenizer](https://github.com/mideind/Tokenizer)** — the Icelandic sentence
  splitter and tokenizer from Miðeind, used by `prepare_text.py`. The training
  data was tokenized with version 3.6.2.
- **[kenlm](https://github.com/kpu/kenlm)** — the Python bindings, used by
  `score_text.py`. There is no official KenLM release on PyPI, so this installs
  from the upstream repository and compiles from source; a C++ compiler and
  CMake are required.

If you would rather use the KenLM command-line tools (`query`) instead of the
Python bindings, build them from source:

```bash
git clone https://github.com/kpu/kenlm.git
cd kenlm
cmake -B build && cmake --build build -j$(nproc)
```

---

## 5. Preparing text: `prepare_text.py`

The model was trained on text that was **sentence-split and tokenized with the
[Icelandic tokenizer from Miðeind](https://github.com/mideind/Tokenizer), then
lowercased.** Any text you score has to be processed the same way. If you score
raw text, most tokens will not match the vocabulary — punctuation will be glued
to words and capitalized words will be unknown — and the perplexity will be far
too high to mean anything.

`prepare_text.py` does exactly that preprocessing, so you do not have to:

```bash
python prepare_text.py input.txt -o prepared.txt
```

It also reads stdin and writes stdout, so it composes:

```bash
cat input.txt | python prepare_text.py > prepared.txt
```

| Option | Meaning |
|---|---|
| `input` | Input file, or `-` for stdin (default: stdin). |
| `-o`, `--output` | Output file, or `-` for stdout (default: stdout). |
| `-q`, `--quiet` | Suppress the summary line printed to stderr. |

What it does, in order:

1. Splits the input into sentences and tokenizes it, with `normalize=True` so
   that straight quotes become the Icelandic `„` and `“` used in the training
   data.
2. Lowercases the result.
3. Writes one sentence per line, tokens separated by single spaces.

Example:

```
$ cat raw.txt
Forsetinn flutti ávarp á Alþingi í gær. Hann sagði "þetta er mikilvægt mál" og
vísaði til XXII. kafla laganna nr. 19/1940.

$ python prepare_text.py raw.txt
forsetinn flutti ávarp á alþingi í gær .
hann sagði „ þetta er mikilvægt mál “ og vísaði til xxii. kafla laganna nr. 19 / 1940 .
```

Note that abbreviations and ordinals keep their periods (`nr.`, `kr.`,
`xxii.`, `13.`), numbers keep their separators (`250.000`, `3,5%`), and `/` is
spaced out. That is the tokenizer's behaviour and it is what the model expects.

> **Order matters: tokenize first, lowercase second.** `prepare_text.py` feeds
> the tokenizer the text with its original casing and lowercases afterwards.
> Lowercasing first changes how the sentence splitter treats roman numerals and
> initials — `XXII. kafla` stays one sentence, but `xxii. kafla` is split in two
> — and would not match the training data.

---

## 6. Calculating perplexity: `score_text.py`

`score_text.py` scores prepared text and reports perplexity, token count and
out-of-vocabulary count:

```bash
python score_text.py igc_5gram_pruned_quantized_trie.bin prepared.txt
```

```
Sentences:                      4
Tokens:                         51
OOVs:                           1       (1.96%)
Total log10 prob:               -103.9851
Perplexity including OOVs:      109.3762
Perplexity excluding OOVs:      73.8667
```

Or as a single pipeline from raw text:

```bash
python prepare_text.py raw.txt -q | python score_text.py model.bin -
```

| Option | Meaning |
|---|---|
| `model` | Path to the `.bin` model file. |
| `input` | Prepared text file, or `-` for stdin (default: stdin). |
| `--per-line` | Also print log10 probability, token count and OOVs for each line. |
| `--lazy` | Memory-map the model lazily instead of loading it up front. |

The script warns on stderr if the input still contains uppercase characters,
which usually means `prepare_text.py` was not run.

### 6.1 Reading the numbers

- **Perplexity including OOVs** — out-of-vocabulary tokens are scored with the
  `<unk>` probability and counted. This is the number to report for
  open-vocabulary evaluation, and the one to use when comparing against other
  models over the same test set.
- **Perplexity excluding OOVs** — OOV tokens are skipped entirely. Only
  meaningful if you are deliberately evaluating closed-vocabulary behaviour; it
  is not comparable across models with different vocabularies.
- **OOV rate** — on ordinary Icelandic text this should be a few percent at
  most. A much higher rate almost always means the text was not prepared.

Each line is treated as one sentence and wrapped in `<s> … </s>`. The closing
`</s>` is a predicted token and is counted; the opening `<s>` is context only
and is not. This is why a 47-token file reports 51 tokens above — four
sentences, four `</s>`. Miscounting this is the usual reason a hand-rolled
perplexity disagrees with KenLM.

`score_text.py` uses the same arithmetic as KenLM's own `query` tool and
produces identical numbers:

```
perplexity including OOVs = 10 ** (-total_logprob / tokens)
perplexity excluding OOVs = 10 ** (-(total_logprob - oov_logprob) / (tokens - oovs))
```

### 6.2 Using KenLM's `query` tool directly

If you have the KenLM command-line tools built, `query` is a faster alternative
for large files and gives the same results:

```bash
kenlm/build/bin/query -v summary igc_5gram_pruned_quantized_trie.bin < prepared.txt
```

Useful flags:

```bash
# per-sentence scores as well as the summary
bin/query -v sentence -v summary igc_5gram_pruned_quantized_trie.bin < prepared.txt

# per-word detail: word=vocab_id ngram_length log10(p)
bin/query -v word -v summary igc_5gram_pruned_quantized_trie.bin < prepared.txt

# do NOT wrap lines in <s> ... </s> (e.g. for continuous text)
bin/query -n -v summary igc_5gram_pruned_quantized_trie.bin < prepared.txt
```

### 6.3 Memory

The model is memory-mapped and needs about 4 GiB resident. The default loading
method is fine. On a memory-constrained machine you can map it lazily and let
the page cache do the work — `--lazy` for `score_text.py`, `-l lazy` for
`query`:

```bash
python score_text.py --lazy igc_5gram_pruned_quantized_trie.bin prepared.txt
bin/query -l lazy -v summary igc_5gram_pruned_quantized_trie.bin < prepared.txt
```

Lazy loading trades query speed for memory, so keep the model on an SSD if you
use it.

---

## 7. Intended use and limitations

The model is suitable for:

- scoring and re-ranking Icelandic text (ASR, MT, spelling and grammar tools),
- corpus filtering and quality estimation of synthetic or scraped text,
- perplexity-based domain and fluency measurement.

Limitations to be aware of:

- **Lowercased and tokenized only.** The model carries no information about
  capitalization, and it expects the same tokenization used in training.
- **Pruned.** Singleton 3-, 4- and 5-grams are absent; rare-but-valid
  constructions are scored through lower-order back-off.
- **Quantized.** Probabilities and back-off weights are rounded to 8 bits, so
  scores differ slightly from an unquantized model.
- **No content filtering.** The IGC includes news, legal texts, parliamentary
  proceedings, social media and web text. The model reflects the distribution of
  that material, including its biases, named entities and any errors in it.
- **Order 5.** This is an n-gram model and carries no long-range context.

---

## 8. License

The model is released under the
**[Creative Commons Attribution 4.0 International (CC BY 4.0)](https://creativecommons.org/licenses/by/4.0/)**
license.

You are free to share and adapt it, including commercially, provided you give
appropriate credit to the author and to the Icelandic Gigaword Corpus.

### Why CC BY 4.0

The Icelandic Gigaword Corpus is not released under a single license. Its
subcorpora carry one of two:

| License | Subcorpora |
|---|---|
| **CC BY 4.0** | IGC-Adjud, IGC-Laws, IGC-Journals, IGC-News1, IGC-Parla, IGC-Social, IGC-Wiki |
| **Custom MIM license** | IGC-Books, IGC-News2 |

The difference that matters here is redistribution of *text*: material under the
custom license may not be republished. Crucially, **both licenses explicitly
permit building language models from the corpus and publishing those models**,
as well as other language technology and linguistic research use.

That makes CC BY 4.0 the right choice for this release:

- It is permitted. Publishing a model trained on the corpus is allowed under
  both IGC licenses, so no more restrictive license is required.
- It matches the majority of the source material and preserves the attribution
  obligation the CC BY subcorpora carry, which the model inherits as a
  derivative work.
- It adds nothing the source does not require. CC BY-SA would impose a copyleft
  obligation the IGC does not ask for; CC0 or MIT/Apache would drop the
  attribution requirement that CC BY material does ask for.

The training corpus itself is not part of this release and is not covered by
this license. Obtain it from
[CLARIN-IS](https://repository.clarin.is/repository/xmlui/handle/20.500.12537/359)
under its own terms.

---

## 9. Citation

If you use this model, please cite the model, the underlying corpus and KenLM.

### The model

```bibtex
@misc{steingrimsson_igc5gram_2026,
  author    = {Steingrímsson, Steinþór},
  title     = {{IGC-2024} 5-gram: Icelandic {KenLM} Language Model trained on the
               Icelandic Gigaword Corpus (2024)},
  year      = {2026},
  note      = {CLARIN-IS},
  url       = {http://hdl.handle.net/20.500.12537/399}
}
```

### The Icelandic Gigaword Corpus

The datasets the model was trained on:

```bibtex
@misc{barkarson_igc2024ext,
  author    = {Barkarson, Starkaður and Steingrímsson, Steinþór},
  title     = {Icelandic Gigaword Corpus ({IGC}-2024ext) --- unannotated version},
  year      = {2024},
  publisher = {The Árni Magnússon Institute for Icelandic Studies},
  note      = {CLARIN-IS},
  url       = {http://hdl.handle.net/20.500.12537/359}
}
```

```bibtex
@misc{barkarson_igc2022,
  author    = {Barkarson, Starka{\dh}ur and Steingr{\'{\i}}msson, Stein{\th}{\'o}r and Andr{\'e}sd{\'o}ttir, {\TH}{\'o}rd{\'{\i}}s Dr{\"o}fn and Hafsteinsd{\'o}ttir, Hildur and Ingimundarson, Finnur {\'A}g{\'u}st and Magn{\'u}sson, {\'A}rni Dav{\'{\i}}{\dh}},
  title     = {Icelandic Gigaword Corpus ({IGC}-2022) - unannotated version},
  year      = {2022},
  publisher = {The Árni Magnússon Institute for Icelandic Studies},
  note      = {CLARIN-IS},
  url       = {http://hdl.handle.net/20.500.12537/253}
}
```


And the paper describing the corpus and its versions:

```bibtex
@inproceedings{barkarson-etal-2022-evolving,
  author    = {Barkarson, Starkaður and Steingrímsson, Steinþór and
               Hafsteinsdóttir, Hildur},
  title     = {Evolving Large Text Corpora: Four Versions of the {I}celandic
               {G}igaword Corpus},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation
               Conference ({LREC} 2022)},
  address   = {Marseille, France},
  pages     = {2371--2381},
  year      = {2022},
  url       = {https://aclanthology.org/2022.lrec-1.254/}
}
```

### KenLM

```bibtex
@inproceedings{heafield2011kenlm,
  author    = {Heafield, Kenneth},
  title     = {{KenLM}: Faster and Smaller Language Model Queries},
  booktitle = {Proceedings of the Sixth Workshop on Statistical Machine
               Translation},
  pages     = {187--197},
  address   = {Edinburgh, Scotland},
  year      = {2011},
  url       = {https://aclanthology.org/W11-2123/}
}
```

---

## 10. Acknowledgements

Built with [KenLM](https://github.com/kpu/kenlm) by Kenneth Heafield. Trained on
the Icelandic Gigaword Corpus, compiled at the Árni Magnússon Institute for
Icelandic Studies.