EXAMS

EXAMS is a new benchmark dataset for cross-lingual and multilingual question answering for high school examinations. It contains more than 24,000 high-quality high school exam questions in 26 languages, covering 8 language families and 24 school subjects from Natural Sciences and Social Sciences, among others. EXAMS offers a fine-grained evaluation framework across multiple languages and subjects, which allows precise analysis and comparison of various models.

Identifier Task Type Metric License Website Code Download
EXAMS Multiple-choice QA Accuracy CC-BY-SA 4.0

Data Source

Matriculation exams and online tests. The task is combination between [1] (EXAMS) and [2] (Additional questions from online quizzes).

Data Description

# Train Dev Test
Examples 1,512 365 1,472
  #Choices per split
Train 3.88
Validation 4.00
Test 4.00

Vocabulary Overlap

Number of common words in the row and column divided by the total number of unique words in the row.

   train validation test
train 1.000 0.539 0.432
dev 0.222 1.000 0.226
test 0.426 0.544 1.000

Subject Distribution

  train validation test
History 0.400 0.197 0.189
Philosophy 0.161 0.181 0.218
Geography 0.153 0.241 0.198
Biology 0.109 0.132 0.151
Physics 0.100 0.140 0.130
Chemistry 0.078 0.110 0.115

Example

{
   "id":"31f92c5d-650d-4031-8987-f709e928f5ea",
   "question":{
      "stem":"Кои пълководци оглавяват наказателния поход на Симеон срещу възникналата сръбска опасност през 917 г.?",
      "choices":[
         {
            "text":"Теодор Сигрица и Мармаис",
            "label":"A",
            "para":"..."
         },
         {
            "text":"Кракра и Алусиан",
            "label":"B",
            "para":"..."
         },
         {
            "text":"Ивац и Никулица",
            "label":"C",
            "para":"..."
         },
         {
            "text":"Книн, Имник и Ицвоклий",
            "label":"D",
            "para":"..."
         }
      ]
   },
   "answerKey":"A",
   "info":{
      "subject":"History",
      "language":"Bulgarian"
   }
}

Citation

[1] Momchil Hardalov, Todor Mihaylov, Dimitrina Zlatkova, Yoan Dinkov, Ivan Koychev, and Preslav Nakov. 2020. "EXAMS: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering". In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5427–5444, Online. Association for Computational Linguistics.

@inproceedings{hardalov-etal-2020-exams,
    title = "{EXAMS}: A Multi-subject High School Examinations Dataset for Cross-lingual and Multilingual Question Answering",
    author = "Hardalov, Momchil  and
      Mihaylov, Todor  and
      Zlatkova, Dimitrina  and
      Dinkov, Yoan  and
      Koychev, Ivan  and
      Nakov, Preslav",
    booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://www.aclweb.org/anthology/2020.emnlp-main.438",
    pages = "5427--5444",
    series = "EMNLP~'20"
}

[2] Momchil Hardalov, Ivan Koychev, and Preslav Nakov. 2019. "Beyond English-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for Bulgarian. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 447–459, Varna, Bulgaria.

@inproceedings{hardalov-etal-2019-beyond,
    title = "Beyond {E}nglish-Only Reading Comprehension: Experiments in Zero-shot Multilingual Transfer for {B}ulgarian",
    author = "Hardalov, Momchil  and
      Koychev, Ivan  and
      Nakov, Preslav",
    booktitle = "Proceedings of the International Conference on Recent Advances in Natural Language Processing",
    month = sep,
    year = "2019",
    address = "Varna, Bulgaria",
    publisher = "INCOMA Ltd.",
    url = "https://aclanthology.org/R19-1053",
    doi = "10.26615/978-954-452-056-4_053",
    pages = "447--459",
    series = "RANLP~'19"
}

License

The dataset is licensed under Attribution-ShareAlike 4.0 International (CC-BY-SA 4.0). The code in this repository is licenced under the Apache 2.0 License.