Long Context Synthetic Translation Pairs for English and Icelandic (22.09)

  Authors

Jónsson, Haukur Páll ; et al.show everyone

Jónsson, Haukur Páll ; Símonarson, Haukur Barri ; Ragnarsson, Pétur Orri ; Ingólfsdóttir, Svanhvít Lilja ; Þorsteinsson, Vilhjálmur ; Snæbjarnarson, Vésteinn

  Item identifier

http://hdl.handle.net/20.500.12537/260

 Project URL

https://velthyding.is

 Demo URL

https://velthyding.is

 Date issued

2022-09-23

 Type

corpus, text

 Size

17197440 sentences

 Language(s)

English , Icelandic

 Description

This backtranslation corpus was created by extracting publicly available and open datasets which contain texts with context. For English we sampled whole speeches from Europarl V8, whole news articles from NewsCrawl and whole articles from Wikipedia. For Icelandic we sampled whole stories from the Icelandic sagas and Icelandic e-books, we sampled whole documents from the Icelandic Gigaword Corpus (without sports) and whole articles from Wikipedia. The texts were then translated using a translation system which was trained to translate with context. The system translated 3 sentences at once to produce the backtranslations. The corpus consists of multiple *.tsv files in which the first column contains text in the source language and the second column is the corresponding translation. A newline is between each segment (usually a single sentence). An additional newline is placed after each paragraph. A third newline is placed between documents (article, story, etc.). Note that not all the corpora contain paragraph information, and the whole document is considered a single paragraph. # The corpora | Monolingual dataset | Source language | Tokens (millions) | | ----- | ---------------- | ----------------- | | The Icelandic Gigaword Corpus (Without sport) (IGC) | Icelandic | 118.4 | | Wikipedia | Icelandic | 8.9 | | Icelandic sagas | Icelandic | 1.4 | | Icelandic e-books | Icelandic | 1.6 | | NewsCrawl | English | 44.5 | | Wikipedia | English | 53.3 | | EuroPARL | English | 58.4 | --- Þessi málheild með bakþýðingum var mynduð með því að bakþýða opnar málheildir í samhengi. Ensku gögnin voru fengin með því að nýta ræður úr Europarl V8 málheildinni, fréttir úr NewsCrawl og heilar greinar úr Wikipedia. Íslensku frumgögnin eru fengin úr íslendingasögunum, opnum rafbókum, Risamálheildinni og af Wikipedia. Textarnir eru þýddir með þýðingarlíkani sem hefur verið þjálfað til að þýða lengra samhengi, þ.e. meira en eina málsgrein í einu. Bakþýðingarnar voru myndaðar með því að þýða þrjár aðliggjandi málsgreinar í einu. Gögnin eru á *.tsv sniði með frummálið í fyrsta dálki og myndaða textann í seinni dálkinum. Tóm lína er á milli efnisgreina og tvær tómar línur á milli skjala (grein, frétt, saga osfrv.) Ath. að ekki eru öll skjöl með efnisgreinaskiptingu.

 Publisher

Miðeind ehf

 Acknowledgement

Ministry of Education, Science and Culture

Project code: Back-translation data selection and filtering (V2b)

Project name: Language Technology for Icelandic 2019-2023

 Subject(s)

machine translation nmt backtranslations synthetic data

 Collection(s)

Clarin IS

Sýna fulla færslu atriðis