Pith. sign in

REVIEW 3 major objections 5 minor 20 references

SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper constructs SciCUEval, an 11,343-question benchmark that tests language models on four scientific context-understanding skills, and reports that the reasoning model DeepSeek-R1 leads while nearly all models fail at detecting…

desk verdict Useful benchmark idea with a plausible headline result, but the fine-grained tables are internally inconsistent and the dataset isn't released; it needs revision before the fine-grained claims can be trusted. read the letter →

arxiv 2505.15094 v1 pith:BPXQQZHP submitted 2025-05-21 cs.CL

Jing Yu , Yuqi Tang , Kehua Feng , Mingyang Rao , Lei Liang , Zhiqiang Zhang , Mengshu Sun , Wen Zhang
show 3 more authors
Qiang Zhang Keyan Ding Huajun Chen
This is my paper
classification cs.CL
keywords scientificcontextunderstandingLLMevaluationbenchmarknoiseinjectionknowledgegraphsstructuredtablesinformation-absencedetectionmulti-sourceintegrationDeepSeek-R1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that existing AI benchmarks do not measure how language models actually use scientific context, because they mostly ask direct questions over clean text. To fix that, it constructs SciCUEval, a benchmark of 11,343 instances spanning biology, chemistry, physics, biomedicine, and materials science, and combining unstructured papers, structured tables, and knowledge graphs. Each instance is tied to one of four competencies: finding relevant information, detecting that information is absent, integrating multiple sources, and inferring conclusions from context. The paper evaluates 18 models and reports that DeepSeek-R1 is the strongest, that almost all models are strikingly bad at refusing to answer when context is insufficient, and that domain-specialized scientific models lag general-purpose ones.

What carries the argument

The carrying mechanism is the dataset construction pipeline, organized around a two-dimensional grid: four evaluation competencies (relevant information identification, information-absence detection, multi-source information integration, and context-aware inference) crossed with three data modalities (unstructured text, structured tables, and knowledge graphs). For each sampled entry, an LLM generates a question and answer from a per-competency prompt; a Sentence-BERT embedding model retrieves the Top-k most similar but irrelevant entries, which are injected into the context as distractors (k=200–300 for tables and knowledge graphs, k=5 for text); and a two-stage quality filter keeps only instances where a judge LLM confirms the answer is context-supported and five PhD reviewers confirm the question tests the intended competency with the answer fully supported. The noise injection is what makes the benchmark about context understanding rather than factual recall: the correct answer is present in a sea of plausible near-neighbours, so a model must filter, abstain, integrate, or infer.

What would settle it

Re-annotate a random sample of, say, 300 retained instances with independent domain experts who do not see the original labels; if a non-negligible fraction (over about 5%) fail on 'answer fully supported by context' or 'question tests the intended competency', the accuracy differences between models become suspect. A second test: replace the embedding-similarity distractors in a subset with randomly chosen entries; if model accuracy does not change, the benchmark is not actually testing noise-robust identification.

Watch

Extended reading notes

Core claim

The paper's central claim is that scientific context understanding is a measurable skill distinct from general question answering, and that SciCUEval measures it. Evaluated on its own terms, the benchmark shows 18 LLMs with a consistent profile: models are good at pulling a relevant fact out of a noisy context, bad at saying 'I cannot answer' when the context lacks the fact (most score below 20% on correct refusals), stronger on prose than on tables or knowledge graphs, and led overall by DeepSeek-R1, which reaches 69.72% accuracy and beats GPT-4o, Claude-3.5-Sonnet, and domain-specialized scientific models alike. The paper reads the first two patterns as evidence that current models are overconfident and hallucinate rather than abstain, and that reasoning-augmented training is currently the most effective route to scientific context understanding.

Load-bearing premise

The benchmark's validity rests on the quality-control step: an AI judge and a one-week human review must catch every question whose answer is not actually supported by its context, because if hidden errors remain in the 11,343 questions, the scores do not measure what the paper says they measure.

Editorial extensions

If this is right

  • If the benchmark is accepted, reasoning-augmented training (for example DeepSeek-R1 or Qwen3 with explicit thinking) becomes a viable substitute for expensive scientific-domain pretraining, since both outperformed proprietary and domain-specific models.
  • Information-absence detection is a widespread weakness: most models score below 20% on correct refusals, meaning current LLMs are prone to fabricate scientific answers when evidence is missing.
  • Structured data is the bottleneck: models perform best on unstructured text and markedly worse on tables and knowledge graphs, so progress in scientific AI depends on improving structured-data reasoning.
  • Domain-specialized scientific models (ChemLLM, SciGLM, LlaSMol, ChemDFM) do not currently deliver their promised advantage on scientific context understanding, which the paper attributes to shallow reasoning and narrow training scope.
  • The direct-context comparison in the paper shows a large gap between answering without context and answering with context (for DeepSeek-R1, from roughly 6–17% direct to 58–75% with context), so the benchmark is measuring context use rather than memorized answers, at least in part.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: because the distractors are retrieved by embedding similarity, SciCUEval can double as an adversarial benchmark for retrieval-augmented systems; a model's score on information identification measures how well it resists near-miss retrievals, which is exactly the failure mode of dense retrieval in production.
  • Editorial extension: the quality-control pass itself depends on the kind of model the benchmark evaluates, with GPT-4o serving as judge; an independent re-annotation by fresh human experts on a random sample would test whether the reported 90.83% acceptance rate holds and whether hidden errors are concentrated in any one competency.
  • Editorial extension: the poor absence-detection scores suggest a cheap intervention that could be tested directly on the dataset: adding an explicit 'answer only if the context supports it' instruction or a confidence threshold should be evaluated on the information-absence instances before any retraining is attempted.
  • Editorial extension: because the text sub-datasets draw from arXiv and bioRxiv, some questions may be answerable from pretraining memory even with the context removed; a contamination audit against the exact source papers would tighten the benchmark's claim that it measures context use rather than memorization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces SciCUEval, a benchmark dataset of 11,343 questions across ten scientific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, with three data modalities (text, tables, knowledge graphs) and four target competencies (relevant information identification, information-absence detection, multi-source information integration, context-aware inference). The construction pipeline samples scientific data from public sources, uses LLMs to generate questions and answers, injects distractor entries through embedding-based similarity search, and applies a two-stage quality-control process (LLM-as-judge plus manual review by five PhD researchers). The authors evaluate 18 proprietary, general-purpose, and scientific-domain LLMs, reporting that DeepSeek-R1 achieves the highest overall accuracy and that reasoning-augmented models outperform both proprietary and scientific-domain models. The paper also provides fine-grained analyses by competency and by modality.

Significance. If the dataset is released and the reported numbers are reconciled, SciCUEval would fill a genuine gap: existing scientific benchmarks mostly test direct question answering, whereas a benchmark that systematically probes context understanding, absence detection, and multi-source integration across heterogeneous scientific data would be useful to the community. The paper has several commendable elements, including detailed generation prompts in Appendix B, a clearly described noise-injection procedure, transparent documentation of the human-review protocol and compensation in Appendix C, and a fairly broad model evaluation. However, the central contribution is the benchmark itself, and the manuscript currently does not release it and contains internal inconsistencies in the fine-grained evaluation tables. Until those issues are fixed, the empirical and diagnostic claims are not verifiable, so the paper's current significance is conditional on the revision.

major comments (3)
  1. [Section 4.3 and Appendix A, Table 6 vs. Table 3] The fine-grained evaluation tables are internally inconsistent, and this undermines the competency and modality analysis that the paper advertises as a key contribution. In Table 6, SciGLM-6B is assigned MatText All=22.67 and BioText All=39.94, whereas Table 3 gives SciGLM-6B MatText=31.35 and BioText=44.17; the 22.67/39.94 pair is exactly ChemLLM-7B-Chat's pair in Table 3. Conversely, Table 6 gives ChemLLM-7B-Chat MatText All=31.35 and BioText All=44.17, which are SciGLM-6B's Table 3 values. In addition, Table 6 reports Qwen2.5-7B-it BioText All=66.18 while Tables 3 and 7 report 68.18, and Qwen2.5-7B-it's Table 6 MatText/BioText competency columns duplicate its PriKG/HipKG rows. There are further mismatches: Llama3.1-70B-it MatText is 51.59 in Table 6 but 51.91 in Table 3, Gemma2-9B-it overall is 42.20 in Table 4 but 42.21 in Table 3, and Qwen2.5-7B-it overall is 46.24 in Table 5 but 46.62 in Tables 3 and 4. Because the dataset is not released, the reader cannot tell which table reflects the actual runs; these discrepancies must be resolved before the fine-grained conclusions in Sections 4.3 and 4.4 are supported.
  2. [Section 3.4 and Appendix E] The central artifact, the SciCUEval dataset itself, is not released or linked anywhere in the manuscript. Section 3.4 gives aggregate statistics and Appendix E lists source databases, but there is no repository, data URL, or download instruction. This blocks independent inspection of the constructed instances, the noise-injection parameters, and the quality-control results (including the claimed 90.83% acceptance rate). Since the paper's contribution is a benchmark dataset, releasing a stable version with a persistent identifier is necessary for the evaluation results to be reproducible and for the benchmark to be usable by the community.
  3. [Section 3.3 and Appendix C] The quality-control pipeline has a potential self-referential bias that is load-bearing for the benchmark's validity. Questions and answers are generated by LLMs (Appendix B), and the automated judge in Appendix C is GPT-4o; if the same model family generates and validates the data, systematic errors in that family's answers may pass both stages. The human review by five PhD researchers is a substantial mitigating step, but the manuscript reports only a 90.83% overall acceptance rate and does not provide per-criterion pass rates, inter-annotator agreement, or examples of rejected instances. I ask the authors to add a quantitative validation audit, for example a second human re-annotation of a random sample with agreement metrics and a breakdown of the 9.17% rejected cases, so that the reader can assess whether the retained instances truly measure the intended competencies with fully context-supported answers.
minor comments (5)
  1. [Table 2 vs. Table 8] Table 2 lists the BioText source as 'Biorxiv', while Table 8 lists 'bio-protocol.org'; these should be reconciled.
  2. [Appendix B, reasoning prompt] The context-aware inference prompt instructs that the answer 'should not be directly deducible' and 'should require complex reasoning', while the quality-control judge requires answers to be 'explicitly extractable or logically deducible'; the authors should clarify how these two standards are reconciled for the inference competency.
  3. [Table 7] The column header 'Direct Context' is ambiguous and should read 'Direct / Context', and one model name is misspelled as 'Deepsee-V3'.
  4. [Section 4.5, Modality Discrepancy] The sentence 'models struggle to extract, synthesize, and infer information effectively from unstructured data' appears to refer to structured tables and knowledge graphs, not unstructured data; please correct the wording.
  5. [Section 4.5 and Table 7] The text says that 'the integration of context consistently enhances performance', but Table 7 contains counterexamples, such as ChemDFM-v1.5-8B on PhaKG (direct 44.53 vs. context 40.43); the claim should be softened or quantified.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark instances and reported model scores are not derived from each other; minor GPT-4o-as-judge overlap is a validity risk, not a circular reduction.

full rationale

SciCUEval is a benchmark-construction and evaluation paper rather than a derivation from first principles, and I could not exhibit any step where a claimed result reduces to its own input by construction. The four competencies are introduced as measurement categories (Sec. 3.1), and the dataset instances are produced by prompting LLMs (Eq. 2), augmented with embedding-similar distractors (Eqs. 3-4), then filtered by GPT-4o as judge and by five PhD reviewers (Sec. 3.3, App. C). None of these steps fits a parameter to the model scores reported later; the accuracies in Table 3 are direct measured outputs on the constructed instances, not predictions of the generator or the judge. The one mildly self-referential element, that GPT-4o is used in the LLM-as-judge filter and is also one of the evaluated models, is a contamination or benchmark-validity risk rather than circularity, because the retained instances passed an independent human-review gate and the scores are not mathematically forced by the filter. The self-citations in the related-work list (e.g., Feng et al. 2024) are descriptive and not load-bearing. The reviewer-flagged numerical inconsistencies between Tables 3 and 6 are a reproducibility and correctness concern, not a circularity, since they do not make any result equivalent to its inputs. Under the strict requirement to quote a specific reduction, no circular step can be identified.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the quality of LLM-generated instances, the effectiveness of embedding-based distractors, and the choice of metrics. No new scientific entities are proposed. The only explicit numeric design parameter is the distractor count k.

free parameters (1)
  • distractor count k = 5 for text; 200-300 for tables and knowledge graphs
    Chosen by hand to control context length and noise difficulty. It is a design choice, not fitted to data, but it directly affects benchmark difficulty.
assumptions (4)
  • domain assumption LLM-generated question-answer pairs, after LLM-as-judge and human review, are valid and unbiased test instances.
    The benchmark's validity rests entirely on this. GPT-4o generated the questions and answers and also served as a judge; the human review was fast (five reviewers, one week, for 11k questions).
  • domain assumption Sentence-BERT embeddings retrieve distractors that are semantically similar but not answer-relevant.
    Noise injection assumes that similar embeddings identify good misleading entries. If distractors accidentally contain the answer or are too easy, the difficulty is miscalibrated.
  • domain assumption Accuracy and rejection rate are appropriate metrics for all question formats.
    For multi-answer list questions, exact-match accuracy is used, but the paper does not specify how partial credit or ordering is handled.
  • domain assumption The four competencies adopted from RGB (Chen et al., 2024) are sufficient to characterize scientific context understanding.
    The paper adopts this framework without an independent argument that these four skills cover the full construct of scientific context understanding.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models." pith.science (2026). https://pith.science/paper/BPXQQZHP

@misc{pith2026250515094,
  author       = {Pith},
  title        = {Pith review of: SciCUEval: A Comprehensive Dataset for Evaluating Scientific Context Understanding in Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BPXQQZHP}},
  note         = {Machine review of arXiv:2505.15094}
}
read the original abstract

Large Language Models (LLMs) have shown impressive capabilities in contextual understanding and reasoning. However, evaluating their performance across diverse scientific domains remains underexplored, as existing benchmarks primarily focus on general domains and fail to capture the intricate complexity of scientific data. To bridge this gap, we construct SciCUEval, a comprehensive benchmark dataset tailored to assess the scientific context understanding capability of LLMs. It comprises ten domain-specific sub-datasets spanning biology, chemistry, physics, biomedicine, and materials science, integrating diverse data modalities including structured tables, knowledge graphs, and unstructured texts. SciCUEval systematically evaluates four core competencies: Relevant information identification, Information-absence detection, Multi-source information integration, and Context-aware inference, through a variety of question formats. We conduct extensive evaluations of state-of-the-art LLMs on SciCUEval, providing a fine-grained analysis of their strengths and limitations in scientific context understanding, and offering valuable insights for the future development of scientific-domain LLMs.

Figures

Figures reproduced from arXiv: 2505.15094 by the authors.

Figure 1
Figure 1. Overview of the SciCUEval dataset. It spans five scientific domains, supports three data modalities [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration of data generation pipeline in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance of LLMs across four competencies on SciCUEval. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance of LLMs across three modalities on SciCUEval. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 16 canonical work pages

  1. [1]

    Given the follow- ing four materials: mp-xxxxx, mp-xxxxx, mp-xxxxx, mp-xxxxx

    The question should be in Q&A format, starting with sentence like "Given the follow- ing four materials: mp-xxxxx, mp-xxxxx, mp-xxxxx, mp-xxxxx" or "Which of the fol- lowing materials, mp-xxxxx, mp-xxxxx, mp- xxxxx, mp-xxxxx"

    work page

  2. [2]

    The question should focus on a single numeric property of the materials that is rep- resentative of the material and comparable

    work page

  3. [3]

    The question should involve comparing the values of this property and identifying the result

    work page

  4. [4]

    arXiv preprint arXiv:2402.17944

    Large language models (llms) on tabular data: Prediction, generation, and understanding–a survey. arXiv preprint arXiv:2402.17944. Kehua Feng, Keyan Ding, Weijie Wang, Xiang Zhuang, Zeyuan Wang, Ming Qin, Yu Zhao, Jianhua Yao, Qiang Zhang, and Huajun Chen. 2024. Sciknoweval: Evaluating multi-level scientific knowledge of large language models.arXiv prepri...

    arXiv 2024

  5. [5]

    [Unable to generate question]

    If you feel you cannot generate a question or are uncertain about the correctness of the question, please output “[Unable to generate question]”

    work page

  6. [6]

    [Unable to generate question]

    The question should be very difficult. If you feel you cannot provide a high-difficulty question, please output “[Unable to generate question]”. Please output the question in JSON format only. Do not output anything other than the JSON format. The JSON format should look like this: { "question": "[Question or rejection]", "options": { "A": "[Option A]", "...

    work page

  7. [9]

    question

    The answer should be the material ID of the material with the correct value, and the answer must be one of the materials listed in the question. Please output the question in JSON format only. Do not output anything other than the JSON format. The JSON format should look like this: { "question": "[Question or rejection]", "answer": "[Answer]" } Next is th...

    work page

  8. [10]

    The question should require at least a university-level edu- 16 cation to answer

    The difficulty level of the question is high and should involve summarizing, generaliz- ing, and reasoning, rather than simple infor- mation retrieval or verification. The question should require at least a university-level edu- 16 cation to answer

    work page

Show all 20 references
  1. [11]

    It should not be directly deducible but should require complex reasoning to ar- rive at the correct answer

    The answer to the question should not be directly available from the first part para- graphs. It should not be directly deducible but should require complex reasoning to ar- rive at the correct answer

  2. [12]

    The incorrect options should sound reasonable, but the content must be wrong

    Incorrect options should contain errors or deviations from the original content. The incorrect options should sound reasonable, but the content must be wrong

  3. [15]

    The answer must be explicitly extractable or logically deducible from the provided context

  4. [16]

    The answer must adhere strictly to the relevant information in the context and be factually correct

  5. [17]

    Yes". If it does not meet the standard, output

    If the answer meets the standard, output "Yes". If it does not meet the standard, output "No". [Relevant Context start] {Context} [Relevant Context end] [Question start] {Question} [Question end] [Answer start] {Answer} [Answer end] Please evaluate and output either "Yes" or "...

  6. [18]

    Whether the question effectively tests the in- tended competency, ensuring that it is aligned with the targeted skill or knowledge domain and accurately reflects the underlying con- struct it aims to assess

  7. [19]

    Whether the question is expressed clearly and logically, such that its wording is unam- biguous, coherent, and easily understood by both human evaluators and automated sys- tems, thereby minimizing potential misinter- pretations

  8. [20]

    nbc 1" and

    Whether the given contexts fully support the given answer and is factually correct, which requires that the answer not only directly de- rives from or can be logically inferred based on the supporting materials, but also adheres to facts and scientific evidence. Together, thes...

  9. [2021]

    Finqa: A dataset of numerical reasoning over financial data.arXiv preprint arXiv:2109.00122. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen,...

    2024 arXiv

  10. [2023]

    Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun

    Building a knowledge graph to enable pre- cision medicine.Scientific Data, 10(1):67. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun

  11. [2024]

    InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17754–17762

    Benchmarking large language models in retrieval-augmented generation. InProceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pages 17754–17762. Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting...

    2025 arXiv

  12. [2025]

    question_type

    Benchmarking retrieval-augmented genera- tion for chemistry.arXiv preprint arXiv:2505.07671. 11 Appendix A More Results on SciCUEval Table 4 and 5 present the quantitative evaluation re- sults of LLMs across four competencies and three modalities on SciCUEval, respectively. Ta...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.