Icelandic Culture and History QA Dataset 24.10

  Authors

Arnardóttir, Þórunn and Einarsson, Elías Bjartur

  Item identifier

http://hdl.handle.net/20.500.12537/347

 Date issued

2024-10-01

 Type

corpus, text

 Size

12644 entries

 Language(s)

Icelandic

 Description

A question answering dataset intended to measure a large language model's knowledge of Icelandic culture and history and its ability to answer questions correctly. The dataset is split into two parts, a gold corpus and a silver corpus. The gold corpus consists of 2,000 pairs of manually reviewed questions and answers while the silver corpus consists of 10,644 pairs of questions and answers which have not been manually reviewed. All pairs were originally automatically created by GPT-4-turbo based on Icelandic Wikipedia articles and online news from RÚV, which are included in the Icelandic Gigaword Corpus (http://hdl.handle.net/20.500.12537/236). In the gold corpus, 1,900 pairs are from Wikipedia articles while 100 pairs are from news texts. In the silver corpus, 9,610 pairs are from Wikipedia articles and 1,034 pairs are from news texts. The dataset is published as JSONL files in a format compatible with many question answering datasets. Additionally, the gold corpus is published in formats compatible with BIG-bench and OpenAI-evals. For more information on the formats, see the README. Spurningarsvörunarmálheild sem er ætluð til að mæla þekkingu risamállíkans á íslenskri menningu og sögu og getu þess til þess að svara spurningum rétt. Málheildinni er skipt í tvennt, í gullgögn og silfurgögn. Gullgögnin innihalda 2.000 pör af handyfirförnum spurningum og svörum en silfurgögnin innihalda 10.644 pör af óyfirförnum spurningum og svörum. Öll pörin voru upphaflega búin til á sjálfvirkan hátt með GPT-4-turbo og eru búin til upp úr íslenskum Wikipedia-greinum og fréttum af RÚV, sem eru hluti af Risamálheildinni (http://hdl.handle.net/20.500.12537/236). Í gullgögnunum eru 1.900 pör úr Wikipedia-greinum og 100 pör úr fréttum. Í silfurgögnunum eru 9.610 pör úr Wikipedia-greinum og 1.034 pör úr fréttum. Málheildin er gefin út í JSONL-skjölum á sniði sem samrýmist mörgum spurningarsvörunarmálheildum. Auk þess eru gullgögnin gefin út á sniðum sem samrýmast BIG-bench og OpenAI-evals. Sjá README-skjal fyrir frekari upplýsingar um sniðin.

 Publisher

Miðeind ehf.

 Acknowledgement

Ministry of Education, Science and Culture

Project code: Benchmarking datasets for LLMs (G12)

Project name: Language Technology for Icelandic 2019-2023

 Subject(s)

question answering qa large language models llm culture history

 Collection(s)

Clarin IS

Show full item record