Pith. sign in

REVIEW 3 major objections 5 minor 35 references

A Collection of Question Answering Datasets for Norwegian

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new suite of four Norwegian QA datasets, built by native speakers, covers both Bokmål and Nynorsk and reveals that language models perform worse in Nynorsk, struggle with commonsense reasoning, and often repeat falsehoods.

desk verdict A solid, genuinely useful Norwegian QA resource that fills a real gap, but the empirical model-ranking claims should be read with the paper's own caveats about localization and annotation quality in mind. read the letter →

arxiv 2501.11128 v1 pith:3JPTLZ2N submitted 2025-01-19 cs.CL cs.AI

Vladislav Mikhailov , Petter M{ae}hlum , Victoria Ovedie Chruickshank Lang{o} , Erik Velldal , Lilja {O}vrelid This is my paper · ORCID
classification cs.CLcs.AI
keywords NorwegianquestionansweringBokmålNynorskcommonsensereasoningtruthfulnesslanguagemodelevaluationdatasetcreationmultiple-choiceQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces four new Norwegian question-answering datasets — NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA — that together cover world knowledge, commonsense reasoning, truthfulness, and Norway-specific knowledge in both written standards of Norwegian, Bokmål and Nynorsk. The 10.5k question-answer pairs were created by a team of native speakers through manual translation, localization, and creative adaptation of English benchmarks, plus curation of NRK quiz data. Evaluating 11 language models, the authors find that most perform better in Bokmål than Nynorsk, struggle most on commonsense reasoning, and often generate or select false answers. If the datasets measure what they claim, Norwegian language modeling now has a benchmark that goes beyond reading comprehension and can track progress in reasoning and truthfulness for both written standards.

What carries the argument

The central object is the collection itself, built by a two-stage annotation process: first, native-speaker annotators manually translated, localized, and creatively rewrote items from OpenBookQA, CommonSenseQA, and TruthfulQA; second, a curation stage filtered low-quality examples and checked spelling and grammar. NRK-Quiz-QA was assembled from more than 500 quizzes from the Norwegian public broadcaster, with temporal references adjusted and image-dependent items removed. The evaluation uses multiple-choice scoring by token probability for three datasets and free-form generation scored by ROUGE-L for the generation subset, across a pool of prompts per language standard.

What would settle it

Have a group of bilingual Norwegian–English speakers answer a random sample of the original English OpenBookQA and CommonSenseQA items and the corresponding Norwegian versions, and compare human accuracy and item difficulty; if accuracy or difficulty diverge sharply, the Norwegian datasets are not measuring the same constructs. Also, double-annotate a sample of the curated items to measure inter-annotator agreement on quality; low agreement would weaken the curation claim.

Watch

Extended reading notes

Core claim

On the paper's own terms: Norwegian QA resources have been limited to extractive or translated reading-comprehension datasets in Bokmål; this collection is the first to cover both official written standards and to target commonsense reasoning, truthfulness, and Norwegian-specific knowledge, with items written by native speakers rather than machine-translated. The reported evaluations show consistent performance gaps that suggest these abilities are not yet well supported in Norwegian language models.

Load-bearing premise

The pipeline that translates, localizes, and creatively rewrites English items into Norwegian preserves the abilities the original datasets measure, and the single-annotator curation of 80% of examples is sufficient to ensure item quality.

Editorial extensions

If this is right

  • Norwegian language models can now be benchmarked on commonsense and truthfulness separately from reading comprehension, in both Bokmål and Nynorsk.
  • The consistent Bokmål-over-Nynorsk gap identifies minority-language coverage as a specific weakness to target in pretraining and evaluation.
  • The low truthfulness scores indicate that Norwegian models reproduce human misconceptions, so mitigation efforts need Norwegian-specific data.
  • The few-shot results on NorOpenBookQA suggest that adding demonstrations is not monotonic; 4-shot often beats 16-shot, informing practical evaluation protocols.
  • The public release under a permissive license lets other researchers reuse the datasets for training and cross-lingual comparison.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because localization can change question difficulty, the claim that models struggle with commonsense reasoning could be an artifact of changed item difficulty; a bilingual human-answer study comparing item-level difficulty against the English originals would test this directly.
  • The Nynorsk subsets are small (e.g., 95 examples for NorCommonSenseQA and 57 for NorTruthfulQA multiple-choice), so NN score differences should be read with wide error bars; expanding NN coverage is a natural next step.
  • The NRK quiz items are zeitgeist-dependent and may leak into pretraining corpora; a contamination check on those items would harden the benchmark.
  • The NorOpenBookQA training split enables future instruction tuning or continued pretraining on Norwegian world-knowledge data, which the authors do not explore.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents four new Norwegian QA datasets: NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA (multiple-choice and generation), and NRK-Quiz-QA, covering both Bokmål and Nynorsk. The datasets are created by native-speaker annotators through manual translation and localization of OpenBookQA, CommonSenseQA, and TruthfulQA, together with newly written examples and adapted material from NRK quizzes. The authors describe the annotation pipeline, report dataset statistics, and evaluate 11 language models in zero- and few-shot regimes. The main empirical findings are that most models score higher in Bokmål than Nynorsk, that commonsense reasoning is the weakest skill, and that models often reproduce human falsehoods. All datasets and annotation materials are publicly released.

Significance. If the resource is valid, it makes a substantial contribution: it is the first Norwegian QA collection covering both written standards, with multiple task formats and Norwegian-specific content, and it provides a reusable evaluation harness. The paper is commendably transparent about its limitations, including the 80% curation rate, lack of inter-annotator agreement, and absence of human baselines. The public release under a permissive license and the integration with NorEval are concrete strengths that will benefit the community. However, the strength of the empirical conclusions is currently limited by validation gaps; the resource itself is valuable, but the evidence that the adapted datasets measure the intended constructs is incomplete.

major comments (3)
  1. [§3.1.1, §3.6, §7] The paper's central empirical claims—that LMs struggle most with commonsense reasoning and are often untruthful in Norwegian—presuppose that NorCommonSenseQA and NorTruthfulQA measure the same constructs as their English source datasets. Section 3.1.1 explicitly permits annotators to localize and creatively adapt items, and Section 3.6 reports that the Norwegian questions are 'somewhat shorter' and 'less complex' than the English originals (e.g., 'Hvor kommer kumelk fra?'). This is internal evidence that adaptation can lower the cognitive demand of individual items. Because Section 7 states that only 80% of examples were curated, each by a single annotator, with no inter-annotator agreement, and no human baseline was collected, there is currently no quantitative check on construct validity. I ask the authors to add (i) a human baseline on a random sample, (ii) an item-level difficulty comparison with the English originals (e.g., correlation of per-item model accuracy), and (iii) a report of how many items were translated, localized, or newly created, with an analysis of whether localization changes answer distributions.
  2. [Table 2, §5] The Nynorsk subsets are very small: NorOpenBookQA has 253 NN examples, NorCommonSenseQA has 95, and NorTruthfulQA Multiple-choice has 57. Claims such as 'most LMs perform better in NB than NN' and the NN-specific observations in Section 5 (e.g., only three models surpass 40% on NorCommonSenseQA NN) rest on samples where a difference of 5–8 percentage points is within binomial noise. The authors should report confidence intervals or significance tests for the NB–NN comparisons, and should soften or qualify the corresponding conclusions, particularly for datasets with fewer than roughly 100 NN examples.
  3. [§4 Result Aggregation] The evaluation protocol reports the maximum accuracy and ROUGE-L across the pool of 50 prompts for each model. Maximum-of-prompts is an optimistic summary and can make model rankings sensitive to a single prompt; it is also not accompanied by any variance estimate. Since the paper's empirical conclusions are based on these numbers, I request that the authors also report mean and standard deviation (or per-prompt distributions), and that they fix and document the random seed used for k-shot demonstration sampling.
minor comments (5)
  1. [Throughout] The dataset name is spelled inconsistently as 'NorCommonsenseQA' and 'NorCommonSenseQA'; please unify to a single spelling.
  2. [§3.1] The name 'NortruthfulQA' appears with a lowercase 't' at the start of Section 3.1; use 'NorTruthfulQA' consistently throughout.
  3. [§3.6] The manual comparison of 100 examples should state which datasets and annotators were involved and how the sample was drawn, so readers can interpret the claim about question length and complexity.
  4. [§4] The k-shot demonstration examples are sampled randomly without a stated seed; this makes the few-shot results non-reproducible as reported.
  5. [Table 4] The column header for NorOpenBookQA appears to duplicate 'NB'; check that the two k-shot blocks are labeled unambiguously.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the dataset creation and model evaluation are independent of any fitted target or self-referential evidence.

full rationale

This is a resource paper, not a derivation. The claimed contributions are four new Norwegian QA datasets created by native-speaker translation, localization, and original writing, followed by an empirical evaluation of publicly available language models. There is no mathematical derivation or fitted parameter that is later presented as a prediction. The benchmark accuracy and ROUGE-L scores in Table 4 are computed on held-out test sets with no tuning to any desired outcome, so no result reduces by construction to its inputs. The only self-references are citations to the authors' own prior models (e.g., Samuel et al., 2025, for NORA.LLM) and to their NorEval evaluation toolkit; these are used as evaluation subjects and tooling, not as evidence that the datasets measure the intended abilities. The paper explicitly discloses the absence of human baselines and inter-annotator agreement, and these are acknowledged validity limitations rather than circular reasoning. No quoted step in the paper exhibits the formal pattern of an output being equivalent to its input, whether through definition, fitting, or a self-citation chain. The central claim that the datasets cover world knowledge, commonsense, truthfulness, and Norwegian-specific knowledge rests on the annotation design and is independently assessable against the released data and external benchmarks.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

No fitted parameters; the central claims rest on annotation-quality and evaluation-metric assumptions rather than numerical fitting. The key axioms are listed.

assumptions (5)
  • domain assumption Manual translation and localization preserves the validity and difficulty of the original English QA tasks.
    Invoked in Section 3.1.1; if localization changes difficulty, the benchmarks do not measure the same skills as their English sources.
  • domain assumption Single-annotator validation on 80% of examples is sufficient to guarantee dataset quality.
    Acknowledged in Section 7 as a limitation; the paper does not compute inter-annotator agreement.
  • domain assumption NRK quiz data from 2017 to 2024 is high quality and suitable for benchmarking after temporal adjustment.
    Section 3.1.2 states the data is high quality; leakage is acknowledged in Section 7.
  • domain assumption Maximum accuracy across 50 prompts is a valid performance measure.
    Section 4: the paper reports the maximum score across prompts, which can overstate performance and hides variance.
  • domain assumption ROUGE-L against reference answers is a valid measure of truthfulness.
    Section 4; ROUGE-L measures lexical overlap, not truthfulness directly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Collection of Question Answering Datasets for Norwegian." pith.science (2026). https://pith.science/paper/3JPTLZ2N

@misc{pith2026250111128,
  author       = {Pith},
  title        = {Pith review of: A Collection of Question Answering Datasets for Norwegian},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3JPTLZ2N}},
  note         = {Machine review of arXiv:2501.11128}
}
read the original abstract

This paper introduces a new suite of question answering datasets for Norwegian; NorOpenBookQA, NorCommonSenseQA, NorTruthfulQA, and NRK-Quiz-QA. The data covers a wide range of skills and knowledge domains, including world knowledge, commonsense reasoning, truthfulness, and knowledge about Norway. Covering both of the written standards of Norwegian - Bokm{\aa}l and Nynorsk - our datasets comprise over 10k question-answer pairs, created by native speakers. We detail our dataset creation approach and present the results of evaluating 11 language models (LMs) in zero- and few-shot regimes. Most LMs perform better in Bokm{\aa}l than Nynorsk, struggle most with commonsense reasoning, and are often untruthful in generating answers to questions. All our datasets and annotation materials are publicly available.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 11 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. https://aclanthology.org/2024.acl-long.44 The belebele benchmark: a parallel reading comprehension dataset in 122 language variants . In Proceedings of the 62nd Annual Meeting of the ...

    work page 2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  5. [5]

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1300 B ool Q : Exploring the surprising difficulty of natural yes/no questions . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language...

    doi:10.18653/v1/n19-1300 2019

  6. [6]

    Marta R Costa-juss \`a , James Cross, Onur C elebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No Language Left Behind: Scaling Human-centered Machine Translation . arXiv preprint arXiv:2207.04672

    arXiv 2022

  7. [7]

    Martin d ' Hoffschmidt, Wacim Belblidia, Quentin Heinrich, Tom Brendl \'e , and Maxime Vidal. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.107 FQ u AD : F rench question answering dataset . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1193--1208, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.findings-emnlp.107 2020

  8. [8]

    Pavel Efimov, Andrey Chertok, Leonid Boytsov, and Pavel Braslavski. 2020. SberQuAD – Russian Reading Comprehension Dataset: Description and Analysis . In International Conference of the Cross-Language Evaluation Forum for European Languages, pages 3--15. Springer

    work page 2020

Show all 35 references
  1. [9]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  2. [10]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://openreview.net/forum?id=d7KBjmI3GmQ Measuring Massive Multitask Language Understanding . In International Conference on Learning Representations

    2021

  3. [11]

    Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja vrelid. 2023. https://aclanthology.org/2023.nodalida-1.17 N or Q u AD : N orwegian question answering dataset . In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pag...

    2023

  4. [12]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  5. [13]

    Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. https://doi.org/10.18653/v1/P17-1147 T rivia QA : A large scale distantly supervised challenge dataset for reading comprehension . In Proceedings of the 55th Annual Meeting of the Association for Computational...

    2017 doi

  6. [14]

    Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. https://doi.org/10.18653/v1/2020.acl-main.560 The state and fate of linguistic diversity and inclusion in the NLP world . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  7. [15]

    Marzena Karpinska, Nader Akoury, and Mohit Iyyer. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.97 The perils of using M echanical T urk to evaluate open-ended text generation . In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, page...

    2021 doi

  8. [16]

    Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav...

    2019 doi

  9. [17]

    Seungyoung Lim, Myungji Kim, and Jooyoul Lee. 2019. KorQuAD1.0: Korean QA Dataset for Machine Reading Comprehension . arXiv preprint arXiv:1909.07005

    2019 arXiv

  10. [18]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  11. [19]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://doi.org/10.18653/v1/2022.acl-long.229 T ruthful QA : Measuring how models mimic human falsehoods . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

    2022 doi

  12. [20]

    Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang

    Peng Liu, Lemei Zhang, Terje Farup, Even W. Lauvrak, Jon Espen Ingvaldsen, Simen Eide, Jon Atle Gulla, and Zhirong Yang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.317 NLEB ench+ N or GLM : A comprehensive empirical analysis and benchmark dataset for generative language...

    2024 doi

  13. [21]

    Shayne Longpre, Yi Lu, and Joachim Daiber. 2021. https://doi.org/10.1162/tacl_a_00433 MKQA : A linguistically diverse benchmark for multilingual open domain question answering . Transactions of the Association for Computational Linguistics, 9:1389--1406

    2021 doi

  14. [22]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. https://doi.org/10.18653/v1/D18-1260 Can a suit of armor conduct electricity? a new dataset for open book question answering . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language P...

    2018 doi

  15. [23]

    Timo M \"o ller, Julian Risch, and Malte Pietsch. 2021. https://doi.org/10.18653/v1/2021.mrqa-1.4 G erman Q u AD and G erman DPR : Improving non- E nglish question answering and passage retrieval . In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, p...

    2021 doi

  16. [24]

    Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. https://doi.org/10.18653/v1/P18-2124 Know what you don ' t know: Unanswerable questions for SQ u AD . In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages ...

    2018 doi

  17. [25]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-...

    2016 doi

  18. [26]

    Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension . ACM Computing Surveys, 55(10):1--45

    2023

  19. [27]

    Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter M hlum, Magnus Breder Birkenes, Andrey Kutuzov, et al. 2025. The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective . In Proceed...

    2025

  20. [28]

    Sebastian Ruder. 2021. Challenges and Opportunities in NLP Benchmarking

    2021

  21. [29]

    David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja vrelid, Lucas Georges Gabriel Charpentier, and Andrey Kutuzov. 2025. Small Languages, Big Models: A Study of Continual Training on Languages of Norway . In Proceedings of the Joint 25th Nordic Conference on Computational L...

    2025

  22. [30]

    Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. https://openreview.net/forum?id=zWqr3MQuNs Detecting Pretraining Data from Large Language Models . In The Twelfth International Conference on Learning Re...

    2024

  23. [31]

    ByungHoon So, Kyuhong Byun, Kyungwon Kang, and Seongjin Cho. 2022. JaQuAD: Japanese Question Answering Dataset for Machine Reading Comprehension . arXiv preprint arXiv:2202.01764

    2022 arXiv

  24. [32]

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. https://doi.org/10.18653/v1/N19-1421 C ommonsense QA : A question answering challenge targeting commonsense knowledge . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associa...

    2019 doi

  25. [33]

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. https://doi.org/10.1162/tacl_a_00021 Constructing datasets for multi-hop reading comprehension across documents . Transactions of the Association for Computational Linguistics, 6:287--302

    2018 doi

  26. [34]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 doi

  27. [35]

    Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.149 AGIE val: A human-centric benchmark for evaluating foundation models . In Findings of the Association for...

    2024 doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.