{"id":"3873d220-1bfb-4483-a8c9-d7a198566718","arxiv_id":"2507.07155","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 105-question cosmology benchmark ranks nine retrieval-augmented generation systems, placing an OpenAI configuration first at 91.4 percent accuracy, with a calibrated AI judge that tracks human evaluation.","lead":"This paper tests nine AI question-answering systems on 105 cosmology questions and finds the OpenAI-based system is the most accurate, scoring 91.4 percent. It also builds an AI judge that mimics human expert grading, and releases the questions, grades, and code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'best configuration' claim is not statistically supported: top scores differ within binomial noise, and the LLM judge is calibrated and evaluated on the same 945 responses; no significance tests or held-out validation are reported.","rationale":"The reader's weakest_assumption focuses on the single co-author human evaluation as a potential source of bias, while my concern is that the headline ranking is statistically non-significant even under ideal labels. The reader's rationale does mention 'differences among the top four systems are within binomial noise', so there is partial overlap, but I identify statistical significance as the more load-bearing issue because it affects the primary 'best configuration' claim regardless of human-label validity. The LLM-as-a-Judge overfitting (same data for prompt selection and concordance reporting) further compounds the problem. The benchmark, pipelines, and human evaluation results are fully released, which is strong practical support and means the paper's contribution is salvageable through re-analysis rather than invalid. Therefore, the existing CONDITIONAL verdict remains appropriate: accept with required significance testing, independent or blinded human evaluation, and held-out judge validation.","tokens_in":11879,"tokens_out":3596,"duration_ms":34138,"concrete_test":"Use the released human scores from the CosmoPaperQA repository to compute paired differences between OpenAI and each of OpenAIPDF, VertexAI, and HybridOAIGem, using McNemar's test on the 2x2 contingency tables. Report exact two-sided p-values and bootstrap 95% CIs for accuracy differences. If any pairwise p > 0.05 or CI includes 0, the statement 'currently the best RAG agent configuration' must be softened to 'numerically highest, not statistically distinguishable'. Additionally, re-run the LLM judge evaluation with a leave-one-system-out split: tune the CoT prompt variant on 8 systems and measure agreement on the held-out 9th; if Cohen's kappa drops substantially across folds, the 'robust proxy' claim is not yet established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the OpenAI embedding+generation configuration is 'currently the best' (91.4% accuracy) rests on a single binary human evaluation of 105 questions per system. With n=105 and p≈0.9, the Wilson 95% CI spans roughly ±5.5 percentage points, so the observed differences between OpenAI (91.4%), OpenAIPDF (90.5%), VertexAI (86.7%), and HybridOAIGem (85.7%) are all within sampling noise. No confidence intervals, McNemar tests, or bootstrap comparisons are provided anywhere in the paper or appendices. Therefore, even granting the human labels are perfectly unbiased, the data cannot distinguish the top systems. The secondary claim that the calibrated LLM-as-a-Judge is a 'robust proxy for human evaluation' is also under-supported: Appendix D describes prompt selection ('CoT typically enhances evaluation accuracy') and the reported concordance is computed on the same 945 responses used to select the protocol, so the agreement is likely overfit. This concern is distinct from the benchmark contribution itself, which is released and reproducible; the issue is the strength of the comparative and proxy claims.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces CosmoPaperQA, a benchmark of 105 expert-curated cosmology question-answer pairs built from five influential papers, and evaluates nine RAG agent configurations on it. Each configuration generates answers for all questions, yielding 945 responses that are scored by a single human expert (co-author BB). The authors report that the OpenAI embedding/generation configuration achieves the highest accuracy (91.4%), followed closely by OpenAIPDF (90.5%), VertexAI (86.7%), and hybrid systems (84.8–85.7%). They further calibrate an LLM-as-a-Judge (LLMaaJ) system using the human labels and claim that it can serve as a 'robust proxy for human evaluation' because its rankings correlate nearly perfectly (Pearson r > 0.99) with the human ranking. The paper also releases the dataset, pipelines, and judge prompts.","tokens_in":12104,"tokens_out":1869,"duration_ms":18904,"significance":"If the central claims held, the paper would provide a valuable benchmark and a scalable automated evaluation method for RAG systems in astrophysics. The benchmark itself, CosmoPaperQA, is a genuine contribution: it is built from real research papers, covers diverse cosmology topics, and is released publicly. The SciRag framework and the systematic comparison of nine configurations, including cost analysis, are also useful practical resources for the community. The paper's primary limitations are statistical and methodological: the headline 'best configuration' claim and the 'robust proxy' claim are not supported by the evidence presented. The benchmark and the software release remain useful even if those two claims are weakened to 'suggestive' or 'preliminary.'","major_comments":[{"comment":"The claim that the OpenAI configuration is 'currently the best' with 91.4% accuracy is not statistically supported. Each system is evaluated on the same 105 questions, and the accuracy differences among the top systems (91.4%, 90.5%, 86.7%, 85.7%) are within binomial sampling noise; for n=105 and p≈0.9, the 95% Wilson interval spans roughly ±5 percentage points. No confidence intervals, McNemar tests, bootstrap comparisons, or any other significance testing is reported anywhere in the paper or appendices. Even under the assumption that the human labels are perfectly unbiased, the data cannot distinguish the top systems. The authors should either add appropriate uncertainty quantification and significance tests, or temper the 'best' claim to a ranking that is presented as suggestive rather than definitive.","section":"Abstract and §4.1"},{"comment":"The claim that the calibrated LLM-as-a-Judge is a 'robust proxy for human evaluation' is undermined by a circularity: the judge prompt (including the choice to use chain-of-thought) was selected based on concordance with the same 945 human labels that are then used to report the Pearson r > 0.99 concordance. This is an in-sample evaluation with no held-out validation. To support the proxy claim, the authors should split the 945 responses into a calibration set and a test set (e.g., by system or by question), select the judge configuration on the calibration set, and report concordance on the held-out set. As written, the reported near-perfect correlation is likely inflated by prompt selection.","section":"Appendix D and §4.2"},{"comment":"The entire evaluation rests on a single human evaluator, who is also a co-author and a builder of the benchmark. The paper reports no inter-rater reliability check, no second evaluator, and no discussion of how the evaluator's familiarity with the benchmark or the system configurations might bias the labels. Since all downstream claims—the ranking, the LLMaaJ calibration, and the cost-performance trade-offs—depend on these labels, the lack of any reliability assessment is a load-bearing gap. The authors should either provide additional evidence of label reliability (e.g., a second evaluator on a subset, or a consistency check such as re-evaluating a sample after a time interval) or explicitly discuss this as a major limitation that prevents strong conclusions about system ranking.","section":"Appendix B and Author Contributions"},{"comment":"Even if the ranking is preserved across the three evaluation methods, the LLM judges show large systematic biases (e.g., Gemini judge overrates baseline Gemini by 11.4 percentage points). The paper claims a 'robust proxy,' but a proxy that systematically overrates or underrates absolute accuracy by 5–15 points cannot be used to compare systems across different configurations or to set thresholds for correctness. The 'robust proxy' claim should be limited to ranking preservation, and the authors should report the limits of agreement (e.g., Bland-Altman style) rather than only correlation coefficients.","section":"§4.2 and Appendix E"}],"minor_comments":[{"comment":"The abstract states 'yielding 91.4% accuracy' without any uncertainty. Consider adding 'point estimate' or a confidence interval, as the accuracy is based on 105 binary judgments.","section":"Abstract"},{"comment":"Typo: 'benchamrks' should be 'benchmarks.'","section":"§2.1"},{"comment":"Typo: 'uch as PaperQA2' should be 'such as PaperQA2.'","section":"§6"},{"comment":"The caption is dense and the vertical dashed lines grouping systems into categories are hard to parse if the figure is printed in grayscale; consider adding category labels directly on the horizontal axis.","section":"Figure 1 caption"},{"comment":"The text refers to 'Table E' but the table is labeled 'Table 1'; please fix the cross-reference.","section":"Appendix E"},{"comment":"The description of the HybridGemGem Assistant states that Gemini's text-embedding-001 is 'leading embedding model on MTEB' but the citation is to MTEB itself; please clarify that this is the model's ranking as of a specific date, as retrieval dates are given elsewhere.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the benchmark is the contribution, not the rankings. CosmoPaperQA — 105 expert-curated QA pairs from five real cosmology papers, with questions spanning factual retrieval, synthetic reasoning, and analytical interpretation — fills a real gap, and the released pipelines, dataset, and cost curves are genuinely useful for anyone building RAG systems for astronomy. The 9-way comparison with a single human evaluating all 945 responses is also a first for this domain.\n\nWhat the paper does well: it is transparent about its own limitations, admitting that many questions explicitly name their source papers (giving retrieval cues) and that the five-paper corpus is smaller than real research contexts. The cost analysis is practical, and the modular SciRag framework should make follow-up work easy.\n\nWhere it gets soft: the claim that OpenAI is 'currently the best' is a point estimate on 105 questions per system. The Wilson interval is roughly ±5 percentage points, so 91.4%, 90.5%, 86.7%, and 85.7% are all within noise; no confidence intervals or significance tests are reported. The single human evaluator is also co-author BB, who helped design the benchmark and evaluation criteria, and there is no inter-rater reliability check. That matters for the ground-truth labels, though it does not sink the benchmark itself.\n\nThe LLM-as-a-judge 'robust proxy' claim has a more specific problem: Appendix D describes tuning the prompt (adding chain-of-thought) based on concordance with the 945 labels, then reports concordance on those same labels (Pearson r > 0.99). That is calibration-set agreement, not validation. A held-out set or cross-validation is needed before calling the judge robust.\n\nWho this is for: researchers building RAG pipelines for astrophysics, who need a starting point for evaluation and a sense of cost-accuracy trade-offs. They will get real value from the assets and the honest discussion.\n\nRecommendation: worth a serious referee — the released assets justify it — but the authors should add uncertainty quantification, an independent or blinded evaluator, and held-out judge validation before the ranking and proxy claims are cited as evidence. Send it to review with expectation of revision.","headline":"A useful, reproducible RAG-evaluation benchmark with released assets, but the headline ranking and 'robust proxy' judge claims outrun the statistics.","tokens_in":12684,"tokens_out":1623,"would_cite":true,"duration_ms":19465,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper evaluates nine retrieval-augmented generation agents on a new 105-question cosmology benchmark and finds the OpenAI-based configuration best, at 91.4% accuracy, with a calibrated LLM judge that reproduces the human ranking.","keywords":["retrieval-augmented generation","LLM-as-a-Judge","CosmoPaperQA","cosmology question answering","autonomous scientific discovery","RAG evaluation","astrophysics benchmark"],"falsifier":"Have two independent cosmology experts, blinded to system identity, re-score a random 100 of the 945 responses; if their agreement with the original human labels is near chance (e.g., Cohen's kappa below 0.6), the accuracy ranking and judge calibration built on those labels are not supported.","tokens_in":11686,"feed_emoji":"🔭","tokens_out":5829,"duration_ms":56584,"temperature":0.7,"pith_summary":"The paper tries to establish which retrieval-augmented generation (RAG) configuration best answers expert-level cosmology questions, and whether an LLM-based judge can replace a human expert in grading such answers. On a new benchmark of 105 question-answer pairs built from five landmark cosmology papers, the OpenAI embedding and generative model configuration scores 91.4% accuracy across all 945 generated answers. The paper also claims that a calibrated LLM-as-a-Judge system, trained on the human evaluation results, preserves the human ranking of systems and can be scaled to thousands of questions. If correct, this gives astrophysics a reproducible way to select RAG agents for autonomous discovery pipelines and a reusable AI evaluator for future benchmarks.","feed_headline":"OpenAI RAG agent tops cosmology QA at 91.4 percent","feed_subtitle":"Nine configurations, 945 expert-scored answers, and an AI judge that matches the human ranking.","key_machinery":"The load-bearing object is CosmoPaperQA, a benchmark of 105 expert-curated question-answer pairs drawn from five highly cited cosmology papers, combined with SciRag, a modular pipeline that runs nine retrieval-augmented configurations over the same five-paper corpus. Evaluation uses a binary 100/0 correctness protocol applied by a human cosmologist, and the same protocol is given to two LLM judges with chain-of-thought prompting to produce the calibrated LLM-as-a-Judge system.","core_discovery":"The paper's central claim is that a retrieval-augmented generation agent configured with OpenAI embeddings and an OpenAI generative model is currently the best of nine tested configurations for answering expert-level cosmology questions, scoring 91.4% on a new benchmark of 105 question-answer pairs built from five landmark cosmology papers. A human cosmologist scored all 945 generated answers (105 questions across 9 systems) as correct or incorrect, and those scores place commercial solutions (OpenAI 89.5–91.4%, VertexAI 86.7%) and hybrid architectures (84.8–85.7%) above academic tools PaperQA2 (81.9%) and far above non-RAG baselines (16.2–17.1%). Using the 945 human labels, the paper calibrates two LLM judges (OpenAI o3 mini and Gemini) with a binary chain-of-thought protocol; both judges preserve the human ranking with Pearson r > 0.99, with the OpenAI judge scoring conservatively and the Gemini judge overrating. The authors conclude that the calibrated AI judge can serve as a proxy for human evaluation in this domain, enabling scale-up to thousands of cosmology QA pairs and systematic configuration selection for autonomous discovery agents.","pith_inferences":["Editorial extension: a multi-evaluator replication would be the natural next test; a single evaluator's labels may carry systematic bias that calibrating an AI judge would then inherit.","Editorial extension: because many benchmark questions name their source papers, real-world retrieval on a large corpus would likely score lower; the 91.4% should be read as an upper bound for unconstrained research queries.","Editorial extension: the published 945 scored responses are reusable as training labels for future automatic evaluators, so the benchmark's value may outlive the specific ranking."],"forward_implications":["If the central claim is correct, the OpenAI embedding-plus-generation configuration is the default choice for cosmology RAG among the nine tested, with 91.4% accuracy.","VertexAI's 86.7% accuracy at about 1/137 the per-query cost makes it the strongest cost-performance option for large-scale deployments.","Non-RAG baselines (16.2% and 17.1% accuracy) are inadequate for expert-level cosmology questions, so retrieval augmentation is essential for this task.","The calibrated LLM-as-a-Judge preserves the human ranking (Pearson r > 0.99) and can replace human evaluation for scaling to thousands of question-answer pairs.","The released CosmoPaperQA dataset, human scores, pipelines, and judge system provide a reproducible public benchmark for future astronomy RAG evaluations."],"supporting_citations":[{"why":"One of the five source papers whose content generates CosmoPaperQA questions.","marker":"(Aghanim et al., 2020)"},{"why":"Source papers for CAMELS simulation questions in the benchmark.","marker":"(Villaescusa-Navarro et al., 2021; 2022)"},{"why":"Source paper for local Hubble constant measurement questions.","marker":"(Riess et al., 2016)"},{"why":"Source paper for Atacama Cosmology Telescope DR6 questions.","marker":"(Calabrese et al., 2025)"},{"why":"Basis of the PaperQA2 academic RAG baseline compared in the evaluation.","marker":"(L´ala et al., 2023)"},{"why":"Extends PaperQA2 to language agents, the baseline the paper claims to outperform.","marker":"(Skarlinski et al., 2024)"},{"why":"Provides the LLM-as-a-Judge methodology the paper calibrates for astronomy.","marker":"(Zheng et al., 2023)"},{"why":"Survey of LLM-as-a-judge methods informing the judge design.","marker":"(Gu et al., 2025)"},{"why":"OCR tool used in document preprocessing for all RAG pipelines.","marker":"(Mistral AI, 2025)"}],"fun_headline_variants":["OpenAI RAG agent leads cosmology QA with 91.4% accuracy","Calibrated AI judge matches human evaluation for cosmology RAG","New cosmology QA benchmark ranks RAG agents, OpenAI on top","RAG agent with OpenAI beats academic tools on cosmology questions","945 expert-scored answers enable accurate AI judge for cosmology RAG"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the single cosmologist who scored all 945 responses provides an unbiased, consistent ground truth; if that expert's judgments are not reproducible, every accuracy ranking and judge calibration in the paper is unsupported.","fun_headline_variants_meta":{"raw":{"variants":["OpenAI RAG agent leads cosmology QA with 91.4% accuracy","Calibrated AI judge matches human evaluation for cosmology RAG","New cosmology QA benchmark ranks RAG agents, OpenAI on top","RAG agent with OpenAI beats academic tools on cosmology questions","945 expert-scored answers enable accurate AI judge for cosmology RAG"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00067,"raw_usage":{"total_tokens":3065,"prompt_tokens":969,"completion_tokens":2096,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2007}},"tokens_in":585,"tokens_out":2096,"duration_ms":15063,"temperature":1.0,"reasoning_tokens":2007,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:48:54.805131+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two independent cosmology experts, blinded to system identity, re-score a random 100 of the 945 responses; if their agreement with the original human labels is near chance (e.g., Cohen's kappa below 0.6), the accuracy ranking and judge calibration built on those labels are not supported.","supporting_citations":[],"review_version":1}