The Icelandic WinoGrande dataset v. 1.0

The WinoGrande dataset (Sakaguchi et al., 2020), used for evaluating common sense capabilities of neural language models, is inspired by the original WinoGrad
dataset (Levesque et al., 2012), but its problems are designed to minimize biases which the models may rely on when solving them. We systematically go through
the WinoGrande test set (1767 examples) and translate and adapt sentences to work in Icelandic. While the English WinoGrande problems are not always constructed as pairs, in our adaptation, we create sentence pairs where it is feasible. We also found some of the examples to be specific to culture, subjective, or otherwise inapplicable for translation. Those examples were either adjusted or skipped. The result is a dataset of 1095 examples. The size of the Icelandic dataset is closes in size to the small variant of the English dataset (640 examples). Included in the dataset is a five-fold split and a python script that should be used to generate train and development sets.

--

The dataset is described in the following paper which has been submitted for review.

@misc{snæbjarnarson2022warm,
      title={A Warm Start and a Clean Crawled Corpus -- A Recipe for Good Language Models}, 
      author={Vésteinn Snæbjarnarson and Haukur Barri Símonarson and Pétur Orri Ragnarsson and Svanhvít Ingólfsdóttir and Haukur Páll Jónsson and Vilhjálmur Þorsteinsson and Hafsteinn Einarsson},
      year={2022},
      eprint={2201.05601},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}