Pith. sign in

REVIEW 5 major objections 6 minor 15 references

MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 1,757-question graduate materials benchmark exposes a large LLM capability gap.

desk verdict Useful new benchmark with a real audit problem: the black-box scores are computed on a 1,471-question subset while the open-source numbers use the full 1,757, so the headline capability gap is not apples-to-apples. read the letter →

arxiv 2505.23982 v1 pith:433GHL2W submitted 2025-05-29 cs.AI

Jerry Junyang Cheung , Shiyao Shen , Yuchen Zhuang , Yinghao Li , Rampi Ramprasad , Chao Zhang This is my paper
classification cs.AI
keywords MSQAmaterialssciencelargelanguagemodelsquestionansweringgraduate-levelreasoningretrieval-augmentedgenerationdomain-specificfine-tuningTrue/Falseevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MSQA is a benchmark of 1,757 graduate-level materials science questions, each offered in a long-answer form that demands a multi-step explanation and a binary True/False form that demands a precise judgment. The paper's central claim is that this is the first benchmark to test both factual knowledge and reasoning together for advanced materials science, and that current language models leave a large gap on it. In the authors' experiments, commercial API-based models reach 84.5% accuracy on long answers, open-source models peak near 60.5%, and domain-specialized fine-tuned models often fall below 30%, which the authors trace to overfitting and distribution shift. A sympathetic reader would care because the benchmark supplies a reusable yardstick for whether LLMs can actually be trusted for synthesis planning, property analysis, and materials design.

What carries the argument

The load-bearing object is the MSQA dataset and its generation pipeline. The authors sample 3,000 papers from a large materials-science corpus, summarize their abstracts with gpt-4o, generate candidate answers with gpt-4o, gemini-2.0-pro, and deepseek-v3, and merge those answers with gpt-4o in a multi-model self-consistency step. A three-stage quality pass (regular-expression filtering, keyword filtering, then human review of 50 sampled pairs) is what converts these machine-drafted answers into the benchmark's gold standard. The dual format is the second piece of machinery: the same underlying claim appears as a detailed explanatory question and as a balanced True/False question, so a model cannot pass by pattern-matching a single response style.

What would settle it

Have two independent materials-science researchers verify every gold long answer against the source paper it came from; if the verified-correct rate is substantially below the reported 92.86%, or errors cluster in one subfield, the benchmark's headline accuracy numbers can no longer be read as measuring graduate-level reasoning.

Watch

Extended reading notes

Core claim

The paper's central claim is that MSQA is the first benchmark to jointly test factual knowledge and multi-step reasoning at graduate level in materials science, and that current LLMs fall well short of it. In the reported experiments, the best API-based model reaches 84.5% accuracy on long-answer questions, the best open-source model reaches 60.5%, and the three domain-specialized models trail far behind, with Honeybee at 19.5% on long answers. Retrieval-augmented generation sharply improves two open-source models, while chain-of-thought prompting lowers accuracy for all open-source and domain-specific models on binary questions and yields only a small gain for black-box models. The paper also reports that gpt-4o as judge agrees with human experts in about 77.4% of long-answer evaluations.

Load-bearing premise

The 1,757 gold answers are factually correct and graduate-level, yet they were drafted and merged by LLMs and only 50 randomly selected pairs were checked by human experts, with 92.86% judged satisfactory.

Editorial extensions

If this is right

  • Retrieval-augmented generation is currently the clearest lever for small open models: with BM25 context, Llama-3-8B jumps from 39.4% to 85.2% on long answers and Qwen-2.5-7B from 51.3% to 87.5%.
  • Chain-of-thought prompting is not a reliable fix for materials-science binary questions; it lowers accuracy for all open-source and domain-specific models tested, so reasoning gains depend on underlying knowledge.
  • Domain-specific fine-tuning, in its current form, can actively hurt a model's ability to answer graduate-level materials questions, since all three domain models scored far below their general-purpose counterparts.
  • The binary True/False format is harder than it looks: several general models score near or below 70%, with DeepSeek-R1-distilled-Llama3 at 52.7%, close to chance.
  • Because gpt-4o and gpt-4o-mini disagree substantially as judges, with the mini model showing verbosity bias and tolerance of vague answers, the choice of evaluator changes reported long-answer accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the reported proprietary-versus-open gap is not a same-protocol comparison: black-box models were only tested without retrieval, so a retrieval-augmented open-source model might narrow or even close the gap.
  • The 50-pair human validation implies the true quality of the other 1,707 gold answers is unmeasured; a full audit could raise or lower every headline accuracy by a meaningful margin.
  • The paired long/binary formats make a consistency score easy to compute: how often a model that answers the True/False item correctly also produces a factually complete explanation. That measure of shallow versus deep correctness is not reported in the paper but is directly enabled by the dataset.
  • A testable extension: re-ask the same questions with deliberately wrong retrieved context. Models that cannot resist the wrong context have memorized associations, while models that override it show genuine reasoning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces MSQA, a benchmark of 1,757 graduate-level materials science question-answer pairs in long-answer and binary True/False formats. The questions and gold answers are generated from a curated corpus of 3,000 materials science papers using gpt-4o, gemini-2.0-pro, and deepseek-v3, with answers merged by gpt-4o and filtered through regex, LLM refinement, and a small human annotation step. The authors evaluate ten LLMs (open-source, domain-specific, and black-box) under direct prompting and retrieval-augmented generation, using gpt-4o as an automated judge for long-answer responses. They report that proprietary models reach up to 84.5% accuracy, open-source models peak at about 60.5%, and domain-specific models underperform, and they provide error analyses.

Significance. If the reported results are valid, MSQA would be a useful addition to the small set of graduate-level scientific reasoning benchmarks, and the finding that domain-specific fine-tuned models underperform general-purpose models is practically important for the LLM-for-materials-science community. The paper's strengths include a clearly documented dataset-generation pipeline, the open release of data and code, and a concrete error analysis with worked examples. However, the headline capability gap rests on a table whose numbers are internally inconsistent, on a comparison that mixes direct and retrieval-augmented settings, and on a human validation of only 2.8% of the data; these issues must be resolved before the benchmark's empirical claims can be trusted.

major comments (5)
  1. [§4.2, Table 4] The printed counts in Table 4 do not support the reported accuracies. Claude and Gemini tuples sum to 1,471 rather than the dataset size of 1,757, Grok's tuple sums to 1,743, and the Grok value (363+885)/1,743 = 71.6% is far from the reported 84.84%. The text also reports 84.46% for Grok, matching neither value. Because this table is the basis for the abstract's headline that proprietary models reach 84.5% while open-source models peak at 60.5%, the authors must report the number of valid responses per model, recompute all accuracies on the same denominator, and clarify how missing or unparseable outputs were treated (excluded vs. counted as incorrect).
  2. [Abstract and §4.2] The headline comparison is not apples-to-apples with respect to retrieval augmentation. Open-source models under direct prompting reach 60.50% for Deepseek-R1-distilled-Llama3 on long-answer questions, but under RAG they reach 85.71% (Table 3), which is comparable to the reported black-box performance. Black-box models were evaluated only under direct prompting, so the abstract's claim that 'open-source LLMs peak around 60.5%' is misleading. The authors should either report RAG results for black-box models or explicitly restrict the claim to the direct-prompting setting.
  3. [§3.4] The human validation is too thin and internally inconsistent. Section 3.4 states that 50 question-answer pairs were randomly selected and evaluated, but the reported 92.86% satisfaction rate cannot be an integer count of 50 pairs or of 100 independent judgments (it would be 46.43 or 92.86, respectively). With only 2.8% of the dataset human-validated, the gold-answer correctness is not established at the level required for a benchmark. The authors should report the exact number of judgments, provide an inter-annotator agreement metric (e.g., Cohen's kappa), and analyze the cases that failed the quality criteria.
  4. [§4.3] The LLM-as-judge agreement of 77.38% with human judgments is a serious threat to the accuracy numbers, since all long-answer accuracies are produced by GPT-4o. The paper should present the confusion matrix numerically (per-class agreement and misclassification patterns) and provide a sensitivity analysis: for example, recompute model accuracies when 'mostly correct' is treated conservatively, or when only the human-validated subset is used.
  5. [Table 3, §4.1] The extremely low domain-specific accuracies (e.g., Mol-Instructions-Molecule at 0.23% and 6.66% on long-answer tasks) may be partly an artifact of output-format failures rather than scientific reasoning failure. The paper gives one qualitative example for Llasmol but does not quantify how many responses were unparseable, off-topic, or in the wrong format. The authors should provide a format-level breakdown and show that the underperformance persists after excluding format failures before attributing it to overfitting and distribution shift.
minor comments (6)
  1. [Abstract] The final sentence contains a wording error: 'crucial for LLMs in advanced materials science' repeats 'LLMs' and obscures the intended meaning; it should probably read 'crucial for advancing materials science.'
  2. [§4.2] The text reports Grok's accuracy as 84.46%, while Table 4 reports 84.84%; these values should be reconciled.
  3. [Table 3] Honeybee's long-answer RAG accuracy (2.73%) is much lower than its direct-prompting accuracy (19.53%), whereas RAG typically improves or maintains performance for other models; the paper does not explain this anomaly.
  4. [§4.1] The binary-answer evaluation relies on exact keyword matching for 'YES' or 'NO'; the paper should state how responses like 'True'/'False' or 'Yes, because...' were handled and report the proportion of unparseable responses.
  5. [Introduction] The claim that MSQA is 'the first benchmark to jointly evaluate the factual and reasoning capabilities' should be softened or situated relative to existing benchmarks such as MaScQA (Zaki et al., 2023) and SciBench, as discussed in related work, to avoid overclaiming.
  6. [References] Several references are incompletely formatted, with 'and 1 others' in the author lists (e.g., Abdin et al., Grattafiori et al., Gemini Team); these should be corrected to full author lists or standard 'et al.' formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity found; the LLM-as-judge loop is a validity risk rather than a definitional reduction.

full rationale

MSQA is a benchmark-construction and evaluation paper, not a derivation of predictions from fitted inputs. The closest self-referential element is the evaluation loop: gold long answers are generated by gpt-4o, gemini-2.0-pro, and deepseek-v3 and merged by gpt-4o (Sections 3.3-3.4), and long-answer scores are produced by gpt-4o acting as an LLM judge (Section 4.1). This design can bias rankings, but it is not circular by construction: accuracy is an empirical agreement rate between each tested model's output and a fixed gold set, the gold set is anchored to 3,000 source articles, 50 pairs received independent human review (92.86% satisfactory, Section 3.4), and the judge was separately checked against human judgments (77.38% agreement, Section 4.3). No fitted parameter is later relabeled as a prediction, and no uniqueness claim or central premise rests on a self-citation. The Table 4 inconsistency (printed tuples sum to 1,471 responses rather than 1,757, and Grok's tuple yields 71.6% rather than 84.84%) is a reproducibility and auditing problem, not a circularity problem. Accordingly, no enumerated circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on assumptions about the correctness and difficulty of LLM-generated gold answers, the reliability of an LLM judge, and the representativeness of the sampled literature. These are domain assumptions rather than fitted mathematical parameters, but they are load-bearing: if the gold answers are not reliably graduate-level and correct, the reported accuracy numbers and model ordering do not measure what the paper claims.

free parameters (3)
  • Human validation sample size = 50 QA pairs
    Only 50 of 1,757 pairs were expert-checked, yet the paper uses this sample to claim gold-answer quality; the central claim depends on this small hand-selected sample.
  • RAG context count = top-5 BM25 paragraphs
    Retrieval-augmented results in Table 3 depend on this hand-set number of retrieved paragraphs.
  • Article cluster count = 10 clusters
    The 3,000 source articles are sampled from 10 K-means clusters chosen by silhouette score, which shapes the topical diversity of the benchmark.
assumptions (4)
  • domain assumption LLM-generated gold answers are factually correct and graduate-level
    Only 50 of 1,757 pairs were human-validated; the remaining answers are accepted without expert verification because they were generated and merged by LLMs.
  • domain assumption GPT-4o as judge reliably scores long-answer responses
    The paper reports 77.38% agreement with human judgments on a subset, but no inter-annotator agreement or error analysis for the judge is provided.
  • domain assumption Abstracts and selected sections of 3,000 papers constitute graduate-level materials science knowledge
    No human expert verified that the sampled literature covers graduate-level curriculum or that the questions derived from abstracts have appropriate difficulty.
  • domain assumption Binary True/False labels generated by GPT-4o are valid
    Balanced counts (878 True, 879 False) are reported, but the correctness of the False statements and the absence of answer-revealing clues are not human-validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge." pith.science (2026). https://pith.science/paper/433GHL2W

@misc{pith2026250523982,
  author       = {Pith},
  title        = {Pith review of: MSQA: Benchmarking LLMs on Graduate-Level Materials Science Reasoning and Knowledge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/433GHL2W}},
  note         = {Machine review of arXiv:2505.23982}
}
read the original abstract

Despite recent advances in large language models (LLMs) for materials science, there is a lack of benchmarks for evaluating their domain-specific knowledge and complex reasoning abilities. To bridge this gap, we introduce MSQA, a comprehensive evaluation benchmark of 1,757 graduate-level materials science questions in two formats: detailed explanatory responses and binary True/False assessments. MSQA distinctively challenges LLMs by requiring both precise factual knowledge and multi-step reasoning across seven materials science sub-fields, such as structure-property relationships, synthesis processes, and computational modeling. Through experiments with 10 state-of-the-art LLMs, we identify significant gaps in current LLM performance. While API-based proprietary LLMs achieve up to 84.5% accuracy, open-source (OSS) LLMs peak around 60.5%, and domain-specific LLMs often underperform significantly due to overfitting and distributional shifts. MSQA represents the first benchmark to jointly evaluate the factual and reasoning capabilities of LLMs crucial for LLMs in advanced materials science.

Figures

Figures reproduced from arXiv: 2505.23982 by the authors.

Figure 1
Figure 1. Example question-answer pairs from MSQA. The long-answer response (top) demonstrates multi-step reasoning with explanations, while the binary-answer response (bottom) shows a concise yes/no determination on a related concept. (NER) (Weston et al., 2019; Shetty et al., 2023), classification (Venugopal et al., 2021; Gupta et al., 2022b), and relation extraction (RE) (Cheung et al., 2023; Song et al., 2023a). Although … view at source ↗
Figure 2
Figure 2. Overview of data generation and quality evaluation in MSQA. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The top 20 most common root verbs (inner [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: T-SNE visualization of material and chemical mentions from long-form answers, embedded using [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Confusion matrix between human and GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Confusion matrix between two material sci [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: Annotation Survey Guideline 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Annotation Survey Format 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

  1. [1]

    Summarize the purpose of the paper in clear and concise terms

    work page

  2. [2]

    <method>

    Classify the purpose as emphasizing "<method>" or "<result>"

    work page

  3. [3]

    abstract

    A general-purpose material property data ex- traction pipeline from large polymer corpora using natural language processing. npj Computational Ma- terials, 9(1):52. Yu Song, Santiago Miret, and Bang Liu. 2023a. Matsci- nlp: Evaluating scientific language models on ma- terials science language tasks using text-to-schema modeling. arXiv preprint arXiv:2305....

    arXiv

  4. [4]

    question

    Ensure your answer is clear, concise, and structured logically. ### Here is the problem: "question": {question}, ### Reference Solutions: Solution 1: {answer1} Solution 2: {answer2} Solution 3: {answer3} Prompt for LLM-as-Judge: Your task is to evaluate the accuracy of LLM-generated answers to materials science questions by comparing them to expert-valida...

    work page

  5. [6]

    Abstract

    Identify research questions relevant to the abstract's themes and materials science interests. "Abstract": {paper[key_abstract]} Prompt for generating candidate answers: I will provide the purpose of a materials science paper, related research questions, and a detailed section of the paper. Your tasks:

    work page

  6. [7]

    Select the Most Relevant Question: Choose the research question that is most specific, clearly phrased, and directly related to the provided section

    work page

  7. [8]

    - Standalone and unambiguous

    Refine the Question: Modify the selected question to ensure it is: - Grounded on information from the provided section, but answerable even without using the provided section. - Standalone and unambiguous. Do not use definite articles when referring to compounds. - Clearly phrased for precision

    work page

  8. [9]

    this study

    Generate a Direct Answer: Provide a concise and well-structured response that: - Directly answers the question. - Is based on the provided section but remains meaningful out of context. - Avoids vague references such as "this study" or "this paragraph." - Clearly conveys the information without requiring the reader to see the original section. Present the...

    work page

Show all 15 references
  1. [10]

    Review the above solutions. 11

  2. [11]

    Enclose the solution within <SOLUTION> and </SOLUTION> tag

    Generate an improved and refined solution by aggregating the strengths from the provided solutions. Enclose the solution within <SOLUTION> and </SOLUTION> tag

  3. [12]

    Provide a brief explanation of your reasoning

  4. [14]

    This is because the hy- drophobic core can interact more strongly with hydrophobic parts of proteins, leading to enhanced protein adsorption

    **Protein Adsorption:** * Shorter PSA blocks: When the PSA block is relatively short (e.g., 4-10 monomers), the block copolymer tends to adsorb more proteins from the surrounding environment due to the rel- atively hydrophobic nature of the core block (of- ten a hydrophobic po...

  5. [15]

    Good Question

    **Surface Passivation:** * Shorter PSA blocks: Short PSA blocks can lead to poor surface passivation due to the under- lying hydrophobic core being more accessible to the environment, resulting in a higher likelihood of protein adsorption and biofouling. * Longer PSA blocks: L...

  6. [2020]

    In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 747– 748

    Cluster quality analysis using silhouette score. In 2020 IEEE 7th International Conference on Data Science and Advanced Analytics (DSAA), pages 747– 748. Pranav Shetty, Arunkumar Chitteth Rajan, Chris Kuen- neth, Sonakshi Gupta, Lakshmi Prerana Panchumarti, Lauren Holm, Chao Z...

    2020

  7. [2023]

    arXiv preprint arXiv:2307.07889

    Llm comparative assessment: Zero-shot nlg evaluation through pairwise comparisons using large language models. arXiv preprint arXiv:2307.07889. 9 Santiago Miret and Nandan M Krishnan. 2024. Are llms ready for real-world materials discovery? arXiv preprint arXiv:2402.05200. Gih...

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.