{"id":"81f03b69-0847-4ba1-84c6-d04cdd14f5c5","arxiv_id":"2608.07370","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LitTraceQA is a scientific QA benchmark that requires systems to retrieve relevant papers, ground answers in typed evidence, and produce verifiable answers, with each stage scored separately.","lead":"This paper introduces LitTraceQA, a benchmark that evaluates AI question answering over scientific papers in three connected stages: finding the relevant papers, locating specific evidence such as tables or equations, and producing checkable answers. It matters because answer-only evaluations can hide whether AI systems are genuinely grounding their claims in cited research.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold annotations rest on a single LLM with no human review; automated checks verify evidence strings, not that answers are entailed by evidence, so any ground-truth errors corrupt all three evaluation stages.","rationale":"The reader's weakest assumption—gold annotations are correct despite single-LLM generation and no human review—identifies the same load-bearing concern that I find. The paper's central value depends on the integrity of P*, E*, and a* as a connected trace, and the current construction validates only that evidence text is recoverable from the declared paper, not that the answer is entailed by the evidence or that the locator is accurate. This is especially fragile for non-textual evidence (figures, tables, equations), where substring checks over captions or surrounding text cannot detect a wrong numeric reading or an incorrect formal interpretation. The paper itself flags this in §8, so the conditional verdict is appropriate: the benchmark is promising, but its ground-truth validity is unverified. My concrete test—an expert audit of a sample—would directly settle whether this concern lands. I do not see an internal inconsistency in the task definition, and I do not believe the concern requires a harsher verdict than CONDITIONAL; hence UNCHANGED relative to the reader's assessment.","tokens_in":15989,"tokens_out":4558,"duration_ms":43217,"concrete_test":"Take a stratified random sample of 100 records from the 4,978-record local collection (or all 55 public-split examples) and have two domain-expert annotators independently verify each record against the source PDF: (1) the evidence locator points to the claimed table, figure, text span, equation, or citation context; (2) the gold answer is correct and follows from that evidence; and (3) the question is unambiguous. Compute the gold-error rate and Cohen's kappa. If the gold-error rate exceeds 5% or kappa is below 0.8, the benchmark's ground truth cannot support the claimed strict trace evaluation; if the error rate is near zero, the acknowledged limitation is real but non-fatal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"LitTraceQA's central claim—that it provides a testbed for verifiable, traceable scientific QA—requires the gold traces (P*, E*, a*) to be trustworthy. The construction in §4 relies on an open-book generator (Claude opus 4) producing questions, evidence, and answers; automated grounding checks only confirm that evidence strings are recoverable from the declared paper. They do not verify that the evidence locator actually contains the claimed content (especially for figures and tables, where needle matching over captions cannot detect misread values), nor that the gold answer is entailed by the cited evidence, nor that the question is unambiguous. Appendix B shows the verification metadata is 'generate loop' with no human or expert step, and §8 explicitly lists human review and inter-annotator agreement as an open requirement. If a non-negligible fraction of gold answers are wrong or disconnected from their evidence, then every stage metric—retrieval F1, grounding F1, answer accuracy, and joint success—is unreliable: a perfect system would score low, and error analysis would misattribute failures to retrieval, grounding, or answer stages. This is the load-bearing assumption because no evaluation protocol can compensate for incorrect ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LitTraceQA, a benchmark task for multi-stage scientific question answering in which a system must output canonical paper identifiers, typed evidence locations, and an answer in a requested format (free-form, multiple choice, or structured table). The public development split contains 55 examples, and the paper also analyzes a larger local annotation collection of 4,978 unique questions over 4,859 gold papers, with five evidence types (table, figure, text span, equation/algorithm, citation context) and closed-book hardness metadata produced by three challenger models. The evaluation protocol separates paper retrieval, evidence grounding, and answer correctness, and defines a strict joint-success criterion plus oracle diagnostic settings. The paper reports corpus statistics but no retrieval-augmented baseline results, and it explicitly lists human quality control, normalized locators, official splits, licensing, and baseline evaluations as open requirements for a final release.","tokens_in":16184,"tokens_out":3871,"duration_ms":37508,"significance":"If the gold traces are trustworthy, LitTraceQA addresses a real gap: existing scientific QA benchmarks typically evaluate either retrieval or answer accuracy, while LitTraceQA explicitly separates paper retrieval, typed evidence grounding, and answer correctness, and requires a checkable trace. The paper's strengths include a clearly specified evaluation contract with separated metrics and an oracle diagnostic ladder, internally consistent corpus statistics, and an unusually candid limitations section that identifies the release requirements. The task design around typed evidence artifacts (tables, figures, equations, citation contexts) is a useful contribution to the community. However, the central claim that LitTraceQA is a testbed for verifiable scientific QA is not yet empirically demonstrated: no end-to-end or retrieval-augmented baseline is run, and the gold annotations rest on a single LLM generator with automated string-matching checks rather than human verification or answer-entailment validation. These issues are load-bearing because every evaluation metric depends on the correctness of the gold trace.","major_comments":[{"comment":"The gold answers are produced by a single open-book generator (Claude opus 4) and validated only by automated grounding checks that confirm evidence strings are recoverable from the declared paper text; the verification metadata in every appendix example is 'generate loop' with no human review. Because the evaluation protocol in §6 scores predicted answers against these gold answers, any incorrect or non-entailed gold answer corrupts retrieval F1, grounding F1, answer accuracy, and joint success simultaneously. Section 8 acknowledges human quality control as an open requirement, but this is not optional for the central claim that LitTraceQA is a benchmark; the paper should either add an expert spot-check or entailment-validation stage or explicitly restrict its claim to a development set.","section":"§4 and Appendix B"},{"comment":"No retrieval-augmented baseline is evaluated anywhere in the paper; the closed-book challenger results are construction-time hardness metadata and are explicitly not leaderboard results. Without at least one end-to-end retrieval-augmented system evaluated under the §6 metrics (paper retrieval, evidence grounding, answer correctness, joint success), the central claim that LitTraceQA provides a testbed for verifiable scientific QA is not empirically demonstrated. The authors should add baseline results on the public development split, for example using a PaperQA-style or OpenScholar-style pipeline, or should soften the contribution claim to one of task definition and corpus construction.","section":"§8 and §7"},{"comment":"The automated grounding checks verify that distinctive strings are recoverable from local paper text, but they do not verify that a table or figure locator contains the claimed value, nor that the gold answer is entailed by the cited evidence. For figure and table evidence, needle matching over captions or abstract sentences cannot detect misread values or incorrect row/column associations, as the appendix examples themselves show. Since LitTraceQA's value depends on the correctness of the trace, the paper should add value-level verification for non-textual evidence or report a manual audit on a representative sample.","section":"§4, ev_001–ev_004 and Appendix B"}],"minor_comments":[{"comment":"The sentence 'This supports the intended evaluation target' overstates what the closed-book labels show; the labels only demonstrate that three specific models cannot answer from memory without paper context, not that retrieval-grounded systems will be rewarded. Consider rephrasing to say the labels are consistent with the goal but require retrieval-augmented evaluation to confirm.","section":"§7, Finding 3"},{"comment":"The oracle-paper and oracle-evidence diagnostic settings are well defined, but the paper reports no results under any of these settings; including illustrative diagnostic numbers on the public development split in a future revision would help validate the protocol.","section":"§6"},{"comment":"The manuscript does not provide a data URL, code URL, or hosted version of the public development split; for reproducibility, the final version should include release links and a detailed schema description file.","section":"General"},{"comment":"The extracted title and some section headers contain spacing artifacts such as 'FORMULTI-STAGE'; an editorial pass should ensure the camera-ready rendering is clean.","section":"Title and headers"}],"recommendation":"major_revision","confidential_remarks":"The paper is closer to a benchmark-development report than a completed benchmark paper. The two cited related benchmarks VISTAQA and Wang et al. (2026) share authors with this paper; that is not improper, but the novelty comparison in Table 1 relies in part on self-cited prior work and would be stronger with independent external validation. The main barrier to acceptance is empirical: without baseline retrieval-augmented results and human verification of gold traces, the central 'testbed' claim is unsupported. I believe these issues are addressable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper defines a genuinely useful evaluation contract and is honest about its own incompleteness, but it is not yet a working benchmark. The genuinely new piece is the strict three-stage trace: paper IDs, typed evidence locators, and answer components all have to be jointly correct, with separate scores so failures can be localized. Table 1 makes the gap clear, and the oracle diagnostic ladder (end-to-end vs. oracle paper vs. oracle evidence) is a well-designed evaluation framework that should become standard for this kind of resource.\n\nThe task schema covers five evidence types and three answer formats, and the corpus statistics are internally consistent. Section 8 is unusually candid about what remains for a real release, and the 4,978-record collection is well characterized. The citation pattern is fine; the two self-cited related works are directly relevant and the rest of the references are broad enough.\n\nThe soft spots are real and load-bearing. Gold answers and evidence come from a single LLM (Claude Opus 4) with no human review or inter-annotator agreement. Automated grounding checks verify that evidence strings appear in the declared paper, but they do not verify that the locator actually contains the claimed content or that the gold answer follows from the evidence. If a non-trivial fraction of gold traces are wrong, every stage metric is corrupted. No retrieval-augmented baselines are reported, so the claim that LitTraceQA is a testbed is not empirically demonstrated. The public split of 55 examples is very small, and there is no download link. The closed-book hardness filtering is partly self-referential because the same model families are used as challengers, but this is a minor issue and the paper does not oversell it.\n\nThis deserves serious peer review as a benchmark design contribution, but it needs major revisions and a completed release. The evaluation protocol and task definition are likely to influence future scientific-QA work even if the current dataset does not ship in this form.","headline":"A well-specified evaluation contract for traceable scientific QA, but the dataset is not yet a working benchmark: no baselines, no human-validated gold, and a tiny public split.","tokens_in":16747,"tokens_out":2262,"would_cite":false,"duration_ms":20575,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LitTraceQA reframes scientific question answering as a three-stage trace: retrieve the papers, ground the answer in typed evidence, and only then generate.","keywords":["literature-grounded QA","evidence grounding","paper retrieval","multi-paper QA","typed evidence","closed-book hardness filtering","benchmark","trace evaluation"],"falsifier":"Take a random sample of, say, 100 LitTraceQA gold records, have two independent expert annotators re-locate the evidence and re-derive the gold answer from the declared paper, and measure agreement; if a nontrivial fraction of gold answers cannot be recovered from the cited evidence, the ground-truth assumption fails.","tokens_in":15775,"feed_emoji":"📚","tokens_out":5107,"duration_ms":41841,"temperature":0.7,"pith_summary":"LitTraceQA reframes scientific question answering as a traceable three-stage task: a system must retrieve the right papers from a metadata pool, locate typed evidence inside those papers (tables, figures, text spans, equations or algorithms, citation contexts), and return an answer in the requested format. The paper claims this is the only benchmark in its comparison that combines paper retrieval, multi-paper QA, typed artifact evidence, checkable answers, strict trace evaluation, and closed-book hardness filtering. The public development split has 55 examples; a larger local collection of 4,978 unique-question records over 4,859 gold papers is analyzed for scale and composition. Because each stage is scored separately and joint success requires all three, a fluent answer without the correct paper and evidence is counted as a failure. The paper is explicit that this is a benchmark-development draft, with locator normalization, official splits, human quality control, and retrieval-augmented baselines still to come before a full release.","feed_headline":"QA benchmark scores retrieval, evidence, and answer separately","feed_subtitle":"LitTraceQA: fluent answers without the right paper and typed evidence count as failures.","key_machinery":"The trace contract is the load-bearing object: each gold record is a triple of gold papers, typed evidence items, and a gold answer, and the system output is scored component-wise against that triple. The five typed evidence categories, table, figure, text span, equation or algorithm, and citation context, define what counts as a grounding location, with coarse locators (page plus object ID where supported) used for scoring. The second load-bearing mechanism is the closed-book hardness filter: candidate questions that challenger models can answer from memory are removed, so the retained collection is aimed at questions that require actual retrieval and grounding.","core_discovery":"The central claim is that scientific-paper QA should be evaluated as a retrieval-grounding problem, not an answer-generation problem. LitTraceQA requires three connected outputs: canonical paper identifiers, typed evidence locations, and an answer in a requested format (free-form, multiple-choice, or structured table), and it scores each stage separately while also requiring a strict conjunction of all three for joint success. The evaluation includes an oracle diagnostic ladder (end-to-end, oracle paper, oracle evidence) to isolate where a system fails. The paper further claims that closed-book hardness filtering, which retains only questions that challenger models could not answer reliably without paper access, changes the benchmark target so that retrieval-grounded answering is rewarded more than answering from model memory.","pith_inferences":["If the trace contract works, the same schema could be extended to other evidence types, such as code execution outputs, dataset cards, or supplementary materials, without changing the scoring logic.","The oracle diagnostic ladder suggests a development protocol: optimize the full pipeline, then use oracle-paper and oracle-evidence scores to decide whether to invest in retriever or reader improvements.","A direct validity test would be an expert human agreement study checking whether the gold evidence locators uniquely determine the gold answers; the paper does not yet provide such a study.","The closed-book hardness filter has a shelf life: as models improve, questions once hard without paper access may become answerable from memory, so the hardness labels would need periodic re-challenging."],"forward_implications":["Answer-only accuracy becomes insufficient: a system that retrieves the wrong paper or grounds the answer in the wrong evidence is marked failed even if the answer string is correct.","Failure diagnosis is localized: the oracle-paper and oracle-evidence settings separate retrieval errors from grounding errors from answer-generation errors, telling developers which stage to fix.","Multi-paper questions dominate the local collection at 64.85%, so the benchmark mostly tests cross-paper retrieval and alignment across papers.","The closed-book hardness labels, with all challenger models wrong on 61.9% of records, indicate that the collection targets questions that cannot be answered from parametric memory alone.","The paper's own release checklist identifies what remains before a complete benchmark release: normalized locators, official splits, deliberate handling of the single-paper subset, documented licensing, human quality control, and retrieval-augmented baseline evaluations."],"supporting_citations":[{"why":"QASPER supplies the prior setting of information-seeking QA over research papers with supporting evidence, which LitTraceQA extends by adding retrieval.","marker":"Dasigi et al. (2021)"},{"why":"QASA provides full-paper reasoning without paper-pool retrieval, framing the retrieval gap LitTraceQA targets.","marker":"Lee et al. (2023)"},{"why":"M3SciQA is the closest multi-paper, multimodal scientific QA comparator that LitTraceQA positions against.","marker":"Li et al. (2024)"},{"why":"LitSearch supplies the scientific literature retrieval benchmark that LitTraceQA contrasts with its evidence-and-answer contract.","marker":"Ajith et al. (2024)"},{"why":"AirQA is the broad academic-paper QA comparator with multiple settings and instance-level evaluation.","marker":"Huang et al. (2025)"},{"why":"SPIQA grounds the figure-and-table QA dimension that LitTraceQA includes as typed evidence.","marker":"Pramanick et al. (2024)"},{"why":"VISTAQA motivates the need to measure evidence identification explicitly rather than inferring it from answer correctness.","marker":"Nasr Azadani et al. (2026)"},{"why":"ALCE supplies the answer-attribution evaluation dimension for citation-supported generation that LitTraceQA adapts to typed paper artifacts.","marker":"Gao et al. (2023)"}],"fun_headline_variants":["QA benchmark splits success into retrieval, evidence, and answer","Ground answers in papers: retrieval, evidence, and response scored","Scientific QA needs paper IDs, evidence spots, and faithful answers","Benchmark: find the paper, locate the proof, then answer","Three-part test: paper, evidence, and answer must all link"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is only as good as the automatically generated gold answers and evidence, which come from a single language-model generator with no human review, so if that generator produced confident but wrong labels the entire benchmark would be unreliable despite its careful evaluation protocol.","fun_headline_variants_meta":{"raw":{"variants":["QA benchmark splits success into retrieval, evidence, and answer","Ground answers in papers: retrieval, evidence, and response scored","Scientific QA needs paper IDs, evidence spots, and faithful answers","Benchmark: find the paper, locate the proof, then answer","Three-part test: paper, evidence, and answer must all link"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000312,"raw_usage":{"total_tokens":1760,"prompt_tokens":917,"completion_tokens":843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":756}},"tokens_in":533,"tokens_out":843,"duration_ms":6913,"temperature":1.0,"reasoning_tokens":756,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:27:19.040087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 100 LitTraceQA gold records, have two independent expert annotators re-locate the evidence and re-derive the gold answer from the declared paper, and measure agreement; if a nontrivial fraction of gold answers cannot be recovered from the cited evidence, the ground-truth assumption fails.","supporting_citations":[],"review_version":2}