A question answering dataset intended to measure a large language model's knowledge of Icelandic facts on culture and history and its ability to answer questions correctly. 

The dataset is split into two parts, a gold corpus and a silver corpus. The gold corpus consists of 2,000 pairs of manually reviewed questions and answers while the silver corpus consists of 10,644 pairs of questions and answers which have not been manually reviewed All pairs were originally automatically created by GPT-4-turbo based on Icelandic Wikipedia articles and online news from RÚV, which are included in the Icelandic Gigaword Corpus (http://hdl.handle.net/20.500.12537/236). 

In the gold corpus, 1,900 pairs are from Wikipedia articles while 100 pairs are from news texts. In the silver corpus, 9,610 pairs are from Wikipedia articles and 1,034 pairs are from news texts.

The gold and silver corpora are published as JSONL files, where each question and answer pair is a JSON object with the following elements:
- "query"
- "answer"
- "question_id": A uuid for the question and answer pair
- "question_score": This is not included in the gold subset, but is otherwise a score given by GPT-4-turbo
- "document_score": A score given by GPT-4-turbo
- "url": The original URL of a Wikipedia or news article
- "xml_id": This is only included in the news data, and references the XML ID of an original Icelandic Gigaword Corpus XML file containing the article
- "title": The title of the original article
- "context": The original article

The gold corpus is additionally published in formats compatible with BIG-bench and OpenAI-evals. Both formats consist of JSON objects.

The BIG-bench format consists of the following elements:
- "input": the question
- "target": the answer

The OpenAI-evals format consists of the following elements:
- "input": a list consisting of two dicts, one for the system prompt and one for the question. The system prompt is in all cases "Þú ert vandvirk aðstoðarmanneskja. Svaraðu eftirfarandi spurningu með hnitmiðuðu svari.", which translates to "You are a helpful assistant. Answer the following question with a concise answer.".
- "ideal": the answer