{"id":"87450baa-82c9-494c-bf4a-38ca01ebcdad","arxiv_id":"2504.16414","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 971-question chemistry benchmark shows that even the best large language models, given full context, still fail on many multi-step reasoning questions.","lead":"This paper introduces a chemistry question-answering benchmark built from research papers and tests 13 AI models on multi-step reasoning tasks. Even when given all the needed text, the best model answered only 84 percent correctly, showing that combining multiple chemical facts remains a weak point for current AI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Residual error rates may reflect invalid or ambiguous benchmark items rather than compositional reasoning failures; benchmark validity is checked on only 40 expert-scored questions.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing point is not exactly the reader's weakest assumption. The reader worries that removing all-models-fail questions biases difficulty; that filter actually makes the qualitative 'models struggle' claim conservative. The sharper threat to the central claim is that the dataset may contain invalid or ambiguous items, which would make observed residual errors look like reasoning failures. The paper's own numbers support this concern: 12.5% of the expert-reviewed set rated Poor, and 23% of the pilot items were set aside for low confidence. With only 40/971 expert-validated questions, per-item validity is not established. The model-family circularity (OpenAI models generate, verify, and grade) reinforces the need for independent validation. I would not reject the paper: the qualitative finding that context helps and that no model is perfect is likely robust, and the central claim is directionally supported even if the precise residual error rate is uncertain. However, the quantitative residual-error claim should be conditional on an independent validity audit. Hence verdict CONDITIONAL, agreeing with the reader's overall assessment but with a different emphasis on why the benchmark validity is the critical uncertainty.","tokens_in":17952,"tokens_out":7857,"duration_ms":87934,"concrete_test":"Draw a uniform random sample of 150-200 questions from the released ChemKGMultiHopQA dataset. Have two independent chemistry PhDs, not involved in the paper, rate each question as (a) factually correct answer given the provided context, (b) unambiguous single-answer, and (c) genuinely requiring multiple hops. Recompute the best context/no-context model correctness rates restricted to questions both experts pass. If the best context error rate drops substantially (e.g., below 5-10%) on the valid subset, the 'reasoning errors remain' claim loses quantitative support. As a secondary check, retrieve the S7.3 rejected-question pool and have experts label a random sample for validity; if most are valid hard questions, the outcome-dependent filter is confirmed as a difficulty bias.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that 'even perfect retrieval accuracy with full context does not eliminate reasoning errors' rests on residual errors in Table 3 (best context score 84.35%, so 15.65% errors). This inference requires that incorrect answers are genuine compositional reasoning failures, not artifacts of invalid, ambiguous, or incorrectly grounded questions. The paper's quality evidence is thin: an automated LLM verifier from the same model family used for generation, an outcome-dependent filter that removes questions all evaluated models answer incorrectly (S7.3), and expert review of only 40 high-confidence questions from 971 (Section 5.1, S7.4). In the expert pilot, 12/52 (23%) were dropped for low evaluator confidence and 5/40 (12.5%) were rated Poor; if even a modest fraction of the final 971 items are invalid, a large share of the observed 15.65% residual error could be dataset defect rather than reasoning failure. The S7.3 filter also admits that excluded questions were only partly manually assessed ('most categorized as having multiple valid answers'), so some valid hard questions were removed, biasing difficulty estimates and model comparisons. The benchmark therefore does not yet establish that the residual errors are attributable to multi-hop compositional reasoning.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ChemKGMultiHopQA, a chemistry-domain multi-hop question-answering benchmark of 971 questions generated from ChemRxiv articles through named entity recognition, knowledge graph construction, and OpenAI o3-mini/gpt-4o-based question generation and verification. The authors evaluate 13 large language models in two settings: with full supporting context and without context. They report that providing context nearly doubles correctness rates, that the best model with context (Claude Sonnet 3.7 with extended thinking) reaches 84.35%, and that the best no-context model (o3-mini) reaches 47.58%. The central claim is that even state-of-the-art models struggle with multi-hop compositional reasoning in chemistry, and that even perfect retrieval with full context does not eliminate reasoning errors.","tokens_in":18149,"tokens_out":5179,"duration_ms":52575,"significance":"If the benchmark is valid, it fills a real gap in domain-specific multi-hop reasoning evaluation: existing resources such as ChemLitQA-multi are smaller, mostly single-entity chains, or not chemistry-specific. The automated pipeline is a plausible contribution, and the design choices—short answers, distinct source documents per hop, and context/no-context comparison—are sensible. The evaluation of 13 models and the public release of the dataset and code are strengths. However, the validity evidence is currently too thin to support the strongest claim that residual errors are attributable to compositional reasoning failures rather than to ambiguous, invalid, or incorrectly grounded benchmark items. The paper is honest in disclosing the outcome-dependent filter (S7.3) and the generation-model overlap (S7.5), but those disclosures directly weaken the headline inference.","major_comments":[{"comment":"The filtering rule described in S7.3—'To minimize ambiguity, we excluded questions that were answered incorrectly by all evaluated models from the benchmark'—is outcome-dependent and affects the central difficulty and validity claims. The paper acknowledges that only a subset of the excluded questions was manually assessed and that 'most' were categorized as having multiple valid answers, leaving open the possibility that valid hard questions were removed. Because the reported correctness rates are computed on the post-filter set, they overstate model capability relative to the original generated pool, and the observation that models make residual errors on the remaining questions cannot be interpreted as evidence about compositional reasoning unless the authors show that the filter does not preferentially remove valid difficult items. Please report statistics on the full generated pool, or re-run the evaluation on a random sample not subjected to this filter, and quantify how the headline numbers change.","section":"S7.3"},{"comment":"The expert validation is too small to support the inference that the residual errors are reasoning failures. Out of 52 pilot questions, 12 (23%) were dropped due to low evaluator confidence, and of the remaining 40, 5 (12.5%) were rated Poor. If a similar rate holds in the full benchmark, roughly 120 of the 971 items could be defective, a number comparable to or larger than the best model's 15.65% error rate with context. Please report the residual error rates restricted to the expert-validated Good and Ok items, release the per-question expert annotations, and conduct a substantially larger validation sample before attributing residual errors to compositional reasoning.","section":"5.1 and S7.4"},{"comment":"There is a circularity concern in the evaluation design: the questions are generated with o3-mini and verified with gpt-4o, and the same models are then evaluated on those questions. Section S7.5 acknowledges that 'o3-mini was primarily used to generate the questions, which may have introduced a slight bias, resulting in its minor improvement in correctness.' This undermines the no-context model comparison, particularly the claim that OpenAI models perform better because of richer pretraining data. Please quantify the effect by evaluating on a held-out set generated with different models, or by reporting performance separately on items that the evaluated OpenAI models did and did not generate, and disclose the generation overlap per question.","section":"3.2 and S7.5"},{"comment":"The correctness metric uses exact match followed by a GPT-4o binary assessment for non-matching answers. Since gpt-4o is also one of the evaluated models and is part of the generation and verification pipeline, the judge may have systematic preferences for outputs from the same model family, potentially inflating the correctness rates of OpenAI models. Please provide a sample of judge decisions with human agreement rates, and clarify how near-miss or partially correct answers are treated.","section":"4, evaluation metric"}],"minor_comments":[{"comment":"Mistral Large's performance on the HotpotQA chemistry subset (4.59% without context and 0.92% with context) is dramatically lower than all other models and is inconsistent with its ChemKGMultiHopQA results; this anomaly is never discussed and could distort the dataset comparison in Figure 4.","section":"4.2 and Table S5"},{"comment":"The text contains an unresolved 'Figure ??' placeholder in the sentence referencing Figure S13; this should be fixed.","section":"S7.9"},{"comment":"The HotpotQA-Chemistry row lists 'no chemical entities' under entity types, which is confusing because the subset is defined by chemistry categories; please clarify whether the intended meaning is that the questions are not grounded in chemistry-specific knowledge bases.","section":"Table 2"},{"comment":"Model names are used inconsistently, e.g., 'Qwen QWEN 32B' and 'Deepseek R1 Distil QWEN 32B'; please standardize the model names and match them exactly to the provider names in Table 3.","section":"4.1"},{"comment":"The 'Avg. Corr.' columns in Table S4 are not directly interpretable because the denominator (number of models considered) is not stated; please report the number of models and standard deviations or confidence intervals.","section":"S7.4, Table S4"},{"comment":"The statement 'The full Q&A dataset, along with the evaluation code, is accessible here' does not include a URL or repository identifier in the manuscript text; please provide a working link.","section":"4"}],"recommendation":"major_revision","confidential_remarks":"The paper is transparent about its limitations, and that transparency is commendable, but the disclosed limitations (outcome-dependent filtering in S7.3 and generation-model overlap in S7.5) directly undermine the headline claim that residual errors are evidence of compositional reasoning failure. This is fixable with additional validation and sensitivity analysis, so I see major revision rather than rejection as appropriate. The benchmark could be a useful contribution if the authors can demonstrate that the reported error rates survive on a rigorously validated subset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful new chemistry multi-hop QA benchmark with a mostly automated pipeline, and the headline result—context helps a lot, but even oracle context leaves 15%+ errors—is plausible. The benchmark's difficulty numbers are softened by an outcome-dependent filter, though, so the residual errors should not yet be read as pure reasoning failure.\n\nWhat's new and good: ChemKGMultiHopQA, 971 questions spanning 1–4 hops, built from ChemRxiv literature via NER, gpt-4o relation extraction, and PubChem/Wikipedia enrichment, with each hop drawn from a distinct source text. That last property is genuinely valuable. The comparison against a chemistry-filtered HotpotQA subset and ChemLitQA-multi gives useful context. The evaluation of 13 models with and without provided context is broad, cleanly reported, and the finding that context roughly doubles accuracy is solid. The authors also acknowledge in the appendix that o3-mini's generation role may bias its no-context score, which is more honest than most.\n\nSoft spots: the S7.3 filter removes every question all evaluated models answered incorrectly. The authors say most such questions were ambiguous, but that filter is applied before scoring, so the reported correctness rates are upper bounds on a curated subset. This undercuts the claim that perfect retrieval does not eliminate reasoning errors, because some residual errors may be bad questions rather than bad reasoning. Expert validation is thin: 40 high-confidence questions, 5 rated Poor, and no sensitivity analysis for the final pool. The generator-verifier-evaluator overlap with OpenAI models is a real confound for the no-context ranking, though less so for the context-provided comparison.\n\nThe math and tables are consistent; the graph statistics are carefully reported. The citations are appropriate. The appendix is forthcoming about limitations.\n\nBottom line: this deserves a serious referee. The benchmark and pipeline are reusable, and the qualitative conclusion is probably right. I would want revisions: report accuracy on the full generated pool including excluded questions, expand expert validation to a random sample rather than high-confidence-only, and add a control for generator-evaluator overlap. If those are addressed, this becomes a solid benchmark paper. I would not cite it in my own work, but I would send it to review.","headline":"A useful new chemistry multi-hop QA benchmark whose headline difficulty numbers are softened by an outcome-dependent exclusion filter; worth refereeing with revisions.","tokens_in":18715,"tokens_out":1917,"would_cite":false,"duration_ms":20569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A chemistry benchmark shows that even with the right context, LLMs struggle to chain several facts.","keywords":["multi-hop reasoning","chemistry question answering","knowledge graph construction","named entity recognition","retrieval-augmented generation","large language model evaluation","compositional reasoning","benchmark generation"],"falsifier":"Ask expert chemists to judge a random sample of the questions that the paper excluded because all evaluated models answered them incorrectly. If a substantial share of those excluded questions are judged unambiguous, factual, and answerable from the provided context, then the exclusion rule biases the benchmark toward easier questions and the reported error rates need to be recalculated.","tokens_in":17742,"feed_emoji":"🧪","tokens_out":4610,"duration_ms":46767,"temperature":0.7,"pith_summary":"This paper introduces a new chemistry dataset of 971 multi-hop questions built from recent chemistry preprints and a knowledge graph, then evaluates 13 large language models in two settings: with the full source text supplied and with no context at all. The authors report that even the best model answers only about 84% correctly when given perfect context, and the best no-context model falls below 48%. Their central claim is that compositional reasoning—joining facts from different sources into one answer—remains a genuine bottleneck for current models, and that retrieval context roughly doubles accuracy but does not eliminate errors. The paper also presents the data-generation pipeline as a reusable recipe for building hard multi-hop reasoning benchmarks in other scientific domains.","feed_headline":"Perfect context doesn't fix LLM multi-hop chemistry reasoning","feed_subtitle":"Best model scores 84% with full text, 48% from memory alone.","key_machinery":"The central mechanism is a knowledge graph built from the introductions of chemistry preprints. Chemical entities are extracted and verified through a named entity recognition model refined by a language model, relations between co-occurring entities are extracted as triplets, and nodes are enriched with external database information. A randomized breadth-first search samples paths whose edges come from different source documents, guaranteeing that no single passage can answer the question. One-hop questions are then generated per triplet and combined into a single multi-hop question by reverse chaining, so the final answer is the first entity in the path and the question cannot be solved without composing evidence across documents.","core_discovery":"The paper's claim, stated on its own terms, is that even state-of-the-art large language models, including reasoning-specialised models that scale test-time computation, face significant challenges in multi-hop compositional reasoning within chemistry. The strongest evidence is that with full relevant context supplied, the top correctness rate is 84.35%, while without context the top rate is 47.58%; context nearly doubles average performance across models. Because even perfect retrieval still leaves a substantial error rate, the paper concludes that the limiting factor is compositional reasoning itself rather than access to the necessary facts. The authors further claim that their automated pipeline—named entity recognition, relation extraction, knowledge graph construction, and reverse-chained question aggregation—produces questions requiring one to four hops from distinct source documents, and that an expert-filtered subset supports the benchmark's validity.","pith_inferences":["The exclusion of questions that every evaluated model answered incorrectly could inflate measured capability if some of those questions were valid but simply hard; re-testing the excluded pool with stronger future models, or with expert chemists, would calibrate the benchmark's difficulty.","The reverse-chaining construction always produces a single entity as the final answer, so an extension to relational, numerical, or procedural answers would test a different and potentially harder form of composition.","A natural next experiment is iterative retrieval, where the model receives one source at a time and decides what to fetch next, matching the paper's stated limitation of using a single full-context block.","If the error floor persists when context is guaranteed correct, a useful diagnostic is to probe each hop independently before asking the model to combine the answers, thereby separating failure to retrieve a fact from failure to compose facts."],"forward_implications":["Providing full relevant context roughly doubles correctness for both reasoning and non-reasoning models, so retrieval augmentation remains a high-leverage intervention for multi-hop scientific question answering.","Because even perfect context leaves a measurable error floor, multi-hop QA systems should not assume that retrieving the right documents is sufficient for a correct answer.","Reasoning-specialised models use fewer thinking tokens when context is supplied, suggesting that external evidence reduces the burden of internal knowledge search during reasoning.","Accuracy is higher for single-hop than for multi-hop questions in the context-provided setting, but remains fairly flat across two to four hops, indicating that hop count alone does not fully capture difficulty.","The pipeline can generate large domain-specific reasoning datasets with only a sample of expert verification, lowering the cost of building new benchmarks in specialised fields."],"supporting_citations":[{"why":"Provides the general multi-hop QA benchmark from which a chemistry subset is drawn for comparison.","marker":"[19]"},{"why":"The prior chemistry QA dataset used as the main comparison baseline for domain-specific multi-hop questions.","marker":"[21]"},{"why":"Supplies the method of composing single-hop questions into multi-hop questions that the pipeline adapts.","marker":"[27]"},{"why":"The named entity recognition model used to identify candidate chemical entities from the text.","marker":"[35]"},{"why":"The biomedical pretrained architecture underlying the named entity recognition model.","marker":"[36]"},{"why":"The external chemistry database used to enrich graph nodes with names, identifiers, and property descriptions.","marker":"[37]"}],"fun_headline_variants":["Full context still trips up LLM multi-hop chemistry reasoning","Perfect retrieval can't fix LLM chemistry reasoning gaps","LLMs fail chemistry multi-hop even with all facts supplied","Context boosts LLM chemistry reasoning but errors persist","Multi-hop chemistry reasoning: LLMs need more than context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's scores are only meaningful if questions that every tested model got wrong were genuinely ambiguous or flawed; if some were merely hard, the reported correctness rates are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Full context still trips up LLM multi-hop chemistry reasoning","Perfect retrieval can't fix LLM chemistry reasoning gaps","LLMs fail chemistry multi-hop even with all facts supplied","Context boosts LLM chemistry reasoning but errors persist","Multi-hop chemistry reasoning: LLMs need more than context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1212,"prompt_tokens":907,"completion_tokens":305,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":227}},"tokens_in":523,"tokens_out":305,"duration_ms":3150,"temperature":1.0,"reasoning_tokens":227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:03:56.968916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask expert chemists to judge a random sample of the questions that the paper excluded because all evaluated models answered them incorrectly. If a substantial share of those excluded questions are judged unambiguous, factual, and answerable from the provided context, then the exclusion rule biases the benchmark toward easier questions and the reported error rates need to be recalculated.","supporting_citations":[{"cited_title":"Chemlit-qa: A human evaluated dataset for chemistry rag tasks","cited_arxiv_id":null,"evidence_quote":"The prior chemistry QA dataset used as the main comparison baseline for domain-specific multi-hop questions."},{"cited_title":"Nilinker: attention-based approach to nil entity linking","cited_arxiv_id":null,"evidence_quote":"The named entity recognition model used to identify candidate chemical entities from the text."},{"cited_title":"Pubchem in 2021: new data content and improved web interfaces","cited_arxiv_id":null,"evidence_quote":"The external chemistry database used to enrich graph nodes with names, identifiers, and property descriptions."}],"review_version":1}