Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Retrieving a larger model's reasoning traces as context lets several small models beat a GPT-4 baseline on an expert radiation and cancer biology exam.

desk verdict Useful pipeline and new benchmark, but the headline claim is unsupported and the retrieval setup leaks the answer—send to review as a major-revision candidate. read the letter →

arxiv 2509.10744 v1 pith:OAO7BMNN submitted 2025-09-12 cs.CL cs.AI

Ozan Gokdemir , Neil Getty , Robert Underwood , Sandeep Madireddy , Franck Cappello , Arvind Ramanathan , Ian T. Foster , Rick L. Stevens This is my paper · ORCID
classification cs.CLcs.AI
keywords multiple-choicequestionansweringretrieval-augmentedgenerationreasoningdistillationtracessmalllanguagemodelsscientificbenchmarkdomainadaptationradiationandcancerbiology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a fully automated pipeline that turns a large corpus of scientific papers into thousands of multiple-choice questions, then tests whether small language models (1.1B–14B parameters) do better when retrieval pulls in source chunks or when it pulls in reasoning traces written by a much larger model with the final answer removed. On 16,680 synthetic questions from radiation and cancer biology, and on the 2023 ASTRO expert exam, reasoning-trace retrieval consistently beats both no-retrieval and source-chunk retrieval across most models, with the largest gains for the smallest models. Several small models reach or exceed a GPT-4 reference accuracy on the expert exam when given reasoning traces. The paper frames this as a scalable way to keep benchmarks current and to distill domain knowledge into cheaper, more efficient models.

What carries the argument

The reasoning trace is the paper's central object: a GPT-4.1-generated, per-question chain of thought in one of three modes (detailed, focused, efficient), stored in a separate vector store and retrieved by the small model as context, with the final answer explicitly excluded. This sits inside an automated benchmark pipeline that parses PDFs, semantically chunks them with a biomedical encoder, generates questions with distractors, filters by quality scores, and keeps provenance links to source chunks. The mechanism being tested is whether question-aligned reasoning text transfers more domain knowledge to a small model at inference time than the source chunks themselves do.

What would settle it

Take a fresh expert-written MCQA set, generate retrieval traces from the source corpus alone (never from the test questions), and compare small-model accuracy with and without those traces; if the boost over chunk retrieval disappears, the reported gains require query-specific traces and do not transfer to unseen questions.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central finding is that the choice of retrieval source can matter more than model scale: replacing paper-derived semantic chunks with reasoning traces written by GPT-4.1 (with the final answer token removed) raises multiple-choice accuracy across a 1.1B–14B model suite on both a synthetic benchmark and an expert-written exam. The effect is strongest for the smallest models—TinyLlama-1.1B nearly quadruples its baseline accuracy with trace retrieval—and persists on the 2023 ASTRO Radiation and Cancer Biology exam, particularly on the 189-question non-mathematical subset, where every model improves over both baseline and chunk retrieval. The authors interpret this

Load-bearing premise

The gains depend on treating a large model's reasoning about the exact question being asked as a fair retrieval source; because the trace is generated from that question and only the final answer token is deleted, the approach may be measuring open-book access to answer-relevant reasoning rather than true domain adaptation.

Editorial extensions

If this is right

  • Small models, including a 1.1B model, can reach competitive MCQA accuracy on a specialized scientific domain when their retrieval context is a distilled reasoning trace rather than raw source text.
  • The pipeline can generate thousands of provenance-linked MCQs from a corpus automatically, so benchmarks can be updated as new papers appear without expensive manual curation.
  • The pattern holds on an expert-written exam, especially on non-mathematical questions, suggesting the effect is not merely an artifact of the synthetic benchmark.
  • Retrieval-source choice matters as much as model size: chunk retrieval gives smaller gains and is sometimes worse than baseline on the expert exam, while trace retrieval is the more consistently beneficial source.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The method's practical value hinges on whether traces generated for one set of questions help on questions the teacher model never saw; the paper does not test this, so a natural next experiment is to generate traces only from the source corpus and evaluate on a held-out expert exam.
  • The synthetic benchmark is open-book by construction—questions are generated from the same chunks that are later retrieved—so the large gains there may overstate real-world retrieval; the expert-exam results are the more conservative evidence.
  • If trace quality rather than trace existence drives the effect, distilling traces from cheaper or open teacher models could make the pipeline self-sustaining and avoid dependence on a proprietary model.
  • A natural extension, which the paper's own future-work section gestures toward, is to use the traces not only as retrieval context but also as training data for supervised fine-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a modular pipeline for automatically generating multiple-choice question-answering (MCQA) benchmarks from scientific corpora, with provenance tracking and quality filtering, and applies it to radiation and cancer biology, producing 16,680 questions from 22,000 papers/abstracts. It evaluates eight small and mid-size open-weight models (1.1B–14B parameters) under three conditions: no retrieval, RAG from paper-derived semantic chunks, and RAG from GPT-4.1 reasoning traces with final answers excluded. Accuracy is reported on the synthetic benchmark and on the 2023 ASTRO Radiation and Cancer Biology exam. The central claim is that reasoning-trace retrieval consistently improves accuracy and that several small models surpass GPT-4 on the ASTRO exam.

Significance. If the evaluation were clean, the benchmark-generation pipeline would be a useful contribution: it is scalable, provenance-linked, and the artifacts are released. The finding that compact reasoning traces can serve as effective retrieval context would also be of interest. However, the central results are compromised by an oracle-like retrieval design, and the headline comparison to GPT-4 is not supported by the reported data. The paper therefore does not currently establish either the domain-adaptation claim or the 'surpass GPT-4' claim. The framework itself may still be valuable, but the empirical evaluation as presented is not.

major comments (4)
  1. [§2, Figure 1, §2.2] Oracle leakage in RAG-Traces: The traces are generated by prompting GPT-4.1 on each of the 16,680 synthetic and 335 ASTRO questions. At evaluation time the question text is used as the FAISS query, so the top retrieved item is the trace generated for that exact question. Removing the final answer token does not remove the answer from the chain of thought. The observed RAG-Traces gains are therefore compatible with the model extracting the correct answer from the retrieved context; they do not measure domain adaptation. A protocol is needed in which retrieved traces are not generated on the test questions, or the claims must be reframed accordingly.
  2. [§2, §3.1, Table 2] The synthetic benchmark is a lookup by construction: questions are generated from the same chunks that are later indexed for RAG-Chunks. Retrieval in the RAG-Chunks condition is therefore a search for the exact source chunk, not an open-book test over a corpus. This inflates the observed RAG-Chunks gains (e.g., TinyLlama 17.6%→43.4% in Table 2) and overstates the difficulty of the baseline condition. An external held-out split or a corpus that is disjoint from the question-generation source is required to support the claimed improvements.
  3. [Abstract, §3.2, Tables 3–4, Ref. [5]] The claim that several small models 'surpass GPT-4' on the ASTRO exam is unsupported. Tables 3 and 4 contain no GPT-4 accuracy column or baseline. The cited 'GPT-4 baseline [5]' is reference [5], Beattie et al. 2024, 'Using Large Language Models to Create Patient Centered Consent Forms', which is not an ASTRO exam benchmark. The paper must report GPT-4 (or GPT-4.1) accuracy under the identical evaluation protocol and provide a correct citation, or the headline claim must be removed.
  4. [§2, Figure 2] The quality-control loop is self-referential: GPT-4.1 generates the questions and GPT-4.1 also scores their clarity, accuracy, and educational value. No human validation or comparison to existing expert benchmarks is reported, so the quality of the new benchmark is not independently established. Reporting a human-validated sample or comparing a subset against an existing expert benchmark would be necessary to support the claim that this is a high-quality benchmark.
minor comments (5)
  1. [§3.1.2] Text says TinyLlama reaches '71%–71.5%' while Table 2 lists RAG-RT-Detail as 0.710, RAG-RT-Focused 0.699, and RAG-RT-Efficient 0.581. The numbers should be reconciled.
  2. [§2.1] Typo: 'Qwen-1.5-14B-Chat]' has a stray closing bracket.
  3. [§1] In the contributions list, 'improves small models towards domain ,' is incomplete; the intended phrase appears to be cut off.
  4. [§2, Figure 1 caption] The caption says 'An arbitrary LLM judge performs the grading.' The judge model should be identified, and it should be stated whether it is the same GPT-4.1 model that generates the questions.
  5. [§2.2] The no-math subset is defined as identified automatically by GPT-5, but the prompt, threshold, and validation of this classification are not described. Please provide details for reproducibility.

Circularity Check

2 steps flagged · score 7.0 of 10

RAG-Traces is an oracle lookup: GPT-4.1 traces are generated on the exact test questions, so the reported gains measure answer-following rather than domain adaptation; the 'surpass GPT-4' claim is also unsupported by the tables.

  1. fitted input called prediction [Section 2 (Figure 3 reasoning-trace schema; 'We extract reasoning traces...') and Section 2.2 Evaluation Protocol ('RAG-Traces')]
    "We extract reasoning traces by prompting GPT-4.1 to answer all 16,680 MCQs, explicitly excluding the final answer to prevent leakage. Three reasoning modes are generated simultaneously (detailed, focused, efficient) and stored in separate FAISS databases for evaluation."

    The reasoning trace for each benchmark question is generated by GPT-4.1 on that exact question. At evaluation time the question text is the retrieval query, so the top retrieved item is the very trace produced for that question. Although the final answer letter is removed, the chain-of-thought text contains the reasoning that leads to the answer. RAG-Traces accuracy therefore reduces by construction to how well the small model can extract an oracle-provided answer from its context, not to any learned or retrieved domain adaptation. The paper's central result—that reasoning-trace retrieval 'consistently improves performance'—is thus forced by the experimental setup rather than demonstrated as a transferable finding.

  2. other [Section 2 (workflow: 'Chunks are fed to GPT-4.1... generates an MCQ'; 'RAG from paper-derived chunks')]
    "Chunks are fed to GPT-4.1 in batches through the Argo-Proxy API [8]. The structured prompt first summarizes and expands the chunk, then generates an MCQ with one correct answer and distractors..."

    On the synthetic benchmark, each MCQ is generated from a specific semantic chunk, and the RAG-Chunks condition retrieves from exactly those same paper-derived chunks. Retrieving the generating chunk is a lookup into the source of the question rather than retrieval of independent evidence, so the synthetic benchmark cannot validate domain adaptation. The same issue compounds for RAG-Traces, since the traces are generated from the same questions. The external Astro exam avoids the chunk-generation loop but remains subject to the oracle-trace problem described above.

full rationale

The paper's headline empirical claim—that reasoning-trace retrieval consistently improves small-model accuracy and enables small models to surpass GPT-4—rests on a retrieval source that is constructed from the exact test questions. GPT-4.1 is prompted on each benchmark question to produce a reasoning trace; the final answer token is stripped, but the chain of thought still contains the answer. Because evaluation queries are the same questions, the FAISS retrieval returns an oracle hint, making the RAG-Traces gain a by-construction artifact rather than evidence of domain adaptation. The synthetic-benchmark evaluation is additionally circular in the chunk-retrieval condition because the MCQs are generated from the same chunks later retrieved. The 'surpass GPT-4' statement is also unsupported: Tables 3 and 4 contain no GPT-4 accuracy column, and reference [5] is a consent-form paper rather than an ASTRO exam baseline; this is a missing-support/correctness issue rather than a circularity per se. No load-bearing self-citation or uniqueness-theorem circularity was found. The framework's chunk-RAG results and benchmark-generation infrastructure do have independent content, but the central advertised evaluation is substantially circular, yielding a score of 7.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new physical or conceptual entities are postulated. The synthetic benchmark and framework are artifacts, not entities requiring independent falsifiable evidence. The central empirical results rest on the listed assumptions about corpus representativeness, self-validated question quality, retrieval correctness, and leakage-free traces, plus two design thresholds that shape the reported numbers.

free parameters (2)
  • question quality threshold = 7/10
    The pipeline keeps only GPT-4.1 self-scored questions at or above 7/10; this cut is chosen without sensitivity analysis and directly determines the 16,680-item benchmark.
  • no-math subset definition = GPT-5 classification (not quantified)
    The ASTRO exam is split into math and no-math subsets using GPT-5, and the no-math subset shows larger gains; the classifier's accuracy and prompting are not specified.
assumptions (5)
  • domain assumption The Semantic Scholar corpus (14,115 full papers plus 8,433 abstracts) is a representative source of current radiation and cancer biology knowledge.
    The benchmark's domain validity assumes the corpus covers the relevant knowledge; no coverage analysis is given (Section 2).
  • domain assumption GPT-4.1-generated questions are acceptable without human validation because GPT-4.1's own quality score is reliable.
    Quality control is performed by the same model that generates the questions; this creates a self-validating loop (Section 2, Figure 2).
  • domain assumption The retrieved chunk or trace is the correct, answer-bearing document for each question.
    For the synthetic benchmark, the question was generated from the chunk, making correct retrieval near certain; for the ASTRO exam, retrieval quality is not reported, yet the results assume the trace is useful (Section 2.2).
  • domain assumption Excluding the final answer token from GPT-4.1 reasoning traces prevents answer leakage.
    The paper states final answers are excluded to prevent leakage, but the reasoning trace itself derives the answer, so the assumption is questionable (Section 2).
  • domain assumption AdaParse accurately parses all 14k PDFs without semantic loss.
    The pipeline has no reported parser error rate for this corpus (Section 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models." pith.science (2026). https://pith.science/paper/OAO7BMNN

@misc{pith2026250910744,
  author       = {Pith},
  title        = {Pith review of: Automated MCQA Benchmarking at Scale: Evaluating Reasoning Traces as Retrieval Sources for Domain Adaptation of Small Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OAO7BMNN}},
  note         = {Machine review of arXiv:2509.10744}
}
read the original abstract

As scientific knowledge grows at an unprecedented pace, evaluation benchmarks must evolve to reflect new discoveries and ensure language models are tested on current, diverse literature. We propose a scalable, modular framework for generating multiple-choice question-answering (MCQA) benchmarks directly from large corpora of scientific papers. Our pipeline automates every stage of MCQA creation, including PDF parsing, semantic chunking, question generation, and model evaluation. As a case study, we generate more than 16,000 MCQs from 22,000 open-access articles in radiation and cancer biology. We then evaluate a suite of small language models (1.1B-14B parameters) on these questions, comparing baseline accuracy with retrieval-augmented generation (RAG) from paper-derived semantic chunks and from reasoning traces distilled from GPT-4.1. We find that reasoning-trace retrieval consistently improves performance on both synthetic and expert-annotated benchmarks, enabling several small models to surpass GPT-4 on the 2023 Astro Radiation and Cancer Biology exam.

Figures

Figures reproduced from arXiv: 2509.10744 by the authors.

Figure 1
Figure 1. Workflow overview. PDFs are parsed into text and [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Reasoning-trace JSON schema. Supports three rea [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Percent accuracy improvement on the synthetic [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Percent accuracy improvement on all questions of the 2023 ASTRO Radiation and Cancer Biology Study Guide MCQA benchmark, comparing reasoning-trace retrieval to both baseline performance and retrieval from source docu￾ments for each evaluated model. 3.2.1 All Astro Ques…
Figure 6
Figure 6. Figure 6: Percent accuracy improvement on non￾mathematical questions of the 2023 ASTRO Radiation and Cancer Biology Study Guide MCQA benchmark, compar￾ing reasoning-trace retrieval to both baseline performance and retrieval from source documents for each evaluated model. 3.2.2 A…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 2 canonical work pages

  1. [5]

    Beattie, S

    J. Beattie, S. Neufeld, D.X. Yang, C. Chukwuma, N.B. Desai, M. Dohopolski, and S.B. Jiang. 2024. Using Large Language Models to Create Patient Centered Consent Forms.International Journal of Radiation Oncology*Biology*Physics120, 2 (Oct. 2024), e612. https://doi.org/10.1016/j.ijrobp.2024.07.1346

    work page doi:10.1016/j.ijrobp.2024.07.1346 2024

  2. [1]

    Allen Institute for AI. 2015. Semantic Scholar. https://www.semanticscholar.org/. Accessed: 2025-08-22

    2015

  3. [2]

    2023.RADIATION AND CAN- CER BIOLOGY STUDY GUIDE

    American Society for Radiation Oncology. 2023.RADIATION AND CAN- CER BIOLOGY STUDY GUIDE. https://www.astro.org/ASTRO/media/ASTRO/ AffiliatePages/arro/PDFs/RadBio_StudyGuide_23.pdf

    2023

  4. [3]

    Yadu Babuji, Anna Woodard, Zhuozhao Li, Ben Clifford, Rohan Kumar, Lukasz Lacinski, Ryan Chard, Justin Wozniak, Ian Foster, Michael Wilde, Daniel Katz, and Kyle Chard. 2019. Parsl: Pervasive Parallel Programming in Python. InACM International Symposium on High-Performance Parallel and Distributed Comput- ing

    2019

  5. [4]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng X...

    arXiv 2023

  6. [6]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Øyvind Tafjord. 2018. Think you have Solved Question Answer- ing? Try ARC, the AI2 Reasoning Challenge. InNorth American Chapter of the Association for Computational Linguistics (NAACL), Workshop. Dataset: ARC (AI2 Reasoning Challenge)

    2018

  7. [7]

    2024.Science and Engineering Indicators 2024: The State of U.S

    Steven Deitz and Christina Freyman. 2024.Science and Engineering Indicators 2024: The State of U.S. Science and Engineering. Technical Report NSB-2024-3. National Science Foundation, Alexandria, VA. https://ncses.nsf.gov/pubs/nsb20243

    2024

  8. [8]

    Peng Ding. 2024. argo-proxy. https://github.com/Oaklight/argo-proxy. Accessed: 2025-08-23

    2024

Show all 28 references
  1. [9]

    Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. arXiv:2401.08281 [cs.LG]

    2024 arXiv

  2. [10]

    Ozan Gokdemir, Carlo Siebenschuh, Alexander Brace, Azton Wells, Brian Hsu, Kyle Hippe, Priyanka Setty, Aswathy Ajith, J. Gregory Pauloski, Varuni Sastry, Sam Foreman, Huihuo Zheng, Heng Ma, Bharat Kale, Nicholas Chia, Thomas Gibbs, Michael Papka, Thomas Brettin, Francis Alexan...

    2025

  3. [11]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, A...

    2025 arXiv

  4. [12]

    Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021. Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing.ACM Transactions on Computing for Healthcare3, 1 (Oct. 2021),...

    2021 doi

  5. [13]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Un- derstanding.arXiv preprint arXiv:2009.03300(2021). Dataset: MMLU

    2021 arXiv

  6. [14]

    HuggingFaceTB. 2025. SmolLM3-3B. https://huggingface.co/HuggingFaceTB/ SmolLM3-3B. Accessed: 2025-08-23

    2025

  7. [15]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    2023 arXiv

  8. [16]

    Qiao Jin, Bhuwan Dhingra, William W Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics, 2567–25...

    2019

  9. [17]

    Jakub Lála, Odhran O’Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G Ro- driques, and Andrew D White. 2023. PaperQA: Retrieval-Augmented Generative Agent for Scientific Research. (2023)

    2023

  10. [18]

    Esther Landhuis. 2016. Scientific literature: Information overload.Nature535 (07 2016), 457–458. https://doi.org/10.1038/nj7612-457a

    2016 doi

  11. [19]

    OpenAI. 2025. Introducing GPT-5. https://openai.com/index/introducing-gpt-5/

    2025

  12. [20]

    Aurora Pi-Groeneveld, Luca De Martini, Jesse Dodge, Oyvind Tafjord, Ben Hutchinson, Matt Gardner, Noah Smith, Luke Zettlemoyer, and Nathan Schnei- der. 2024. OLMo: Accelerating the Science of Language Models. (2024). arXiv:2402.00838 [cs.CL]

    2024 arXiv

  13. [21]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A Graduate-Level Google-Proof Q&A Benchmark. arXiv:2311.12022 [cs.AI] https: //arxiv.org/abs/2311.12022

    2023 arXiv

  14. [22]

    Carlo Siebenschuh, Kyle Hippe, Ozan Gokdemir, Alexander Brace, Arham Khan, Khalid Hossain, Yadu Babuji, Nicholas Chia, Venkatram Vishwanath, Rick Stevens, Arvind Ramanathan, Ian Foster, and Robert Underwood. 2025. AdaParse: An Adaptive Parallel PDF Parsing and Resource Scaling...

    2025 arXiv

  15. [23]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieil- lard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon...

    2025 arXiv

  16. [24]

    Yuan-Sen Ting, Tuan Dung Nguyen, Tirthankar Ghosal, Rui Pan, Hardik Arora, Zechang Sun, Tijmen de Haan, Nesar Ramachandra, Azton Wells, Sandeep Madireddy, and Alberto Accomazzi. 2024. AstroMLab 1: Who Wins Astron- omy Jeopardy!? https://doi.org/10.48550/ARXIV.2407.11194

    2024 doi arXiv

  17. [25]

    Shansong Wang, Mingzhe Hu, Qiang Li, Mojtaba Safari, and Xiaofeng Yang

  18. [26]

    Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2017. Crowdsourcing Multiple Choice Science Questions.Transactions of the Association for Computa- tional Linguistics5 (2017), 301–312. Dataset: SciQ

    2017

  19. [27]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. TinyLlama: An Open-Source Small Language Model. arXiv:2401.02385 [cs.CL] https://arxiv. org/abs/2401.02385

    2024 arXiv

  20. [2024]

    InExtended Semantic Web Conference

    Large Language Models for Scientific Question Answering: An Extensive Analysis of the SciQA Benchmark. InExtended Semantic Web Conference

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.