{"id":"9e836fad-77d9-4e9d-aafb-1bf7aa7f34b9","arxiv_id":"2505.17139","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EarthSE provides a two-level QA benchmark and an open-ended dialogue benchmark for Earth science and shows current LLMs perform poorly on both.","lead":"The paper introduces EarthSE, a three-tier benchmark for testing how well large language models handle Earth science, from basic multiple-choice questions to open-ended research dialogues. It evaluates 11 leading models and finds they struggle on difficult Earth science reasoning and on proposing research improvements.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Earth-Gold's SES rests on a single GPT-4-generated reference answer and a GPT-4 judge, so 'open-ended exploration' may reduce to matching the source paper; without independent human validation the exploration-deficiency conclusion is unsecured.","rationale":"The reader's weakest assumption identifies the same circularity: Earth-Gold's reference is the paper's own GPT-4-structured method and limitations, and ranking above it is taken as evidence of better exploration. I agree this is the most load-bearing issue because it directly supports the paper's flagship novelty and the headline claim about open-ended exploration deficiencies. The QA datasets (Earth-Iron/Earth-Silver) still provide reasonable evidence that complex Earth science reasoning is hard, but the Earth-Gold/SES contribution is not independently validated. The concern is addressable with human expert evaluation and reference-perturbation checks, so the appropriate verdict remains CONDITIONAL rather than REJECT; since the reader already reached CONDITIONAL, no adjustment is needed.","tokens_in":16766,"tokens_out":6246,"duration_ms":52841,"concrete_test":"Run a human-validation study on 100 sampled Earth-Gold questions: three Earth-science domain experts, blind to the paper reference and model identity, score each model response on a pre-registered exploration rubric (specificity, novelty, methodological soundness, depth of limitation analysis). Compute Kendall's tau between expert mean scores and GPT-4-derived retention ranks, and Spearman correlation between expert scores and SES. If tau < 0.3 or expert scores do not favor the models GPT-4 ranks above the reference, SES is not measuring exploration as claimed; if agreement is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the claim that Earth-Gold measures “open-ended scientific exploration” (§3.4). For each dialogue, the reference answer is built from the GPT-4-extracted M0/L0/M1/L1 components of the source paper, i.e., one specific paper's method and limitation analysis. Retention rate r is computed by having GPT-4 rank model outputs against that single reference and reading off where the reference falls (r=(i−1)/M). This conflates “explores well” with “produces text GPT-4 prefers over the paper's own summary.” Alternative but valid proposals for the same research direction are penalized unless they beat the paper's method in GPT-4's judgment, and the judge is the same model family that generated the reference. The diversity term d is computed on the model's own responses and is independent of exploration quality; SES=r×d is an unvalidated multiplicative normalization with fragile (s̄−0.9) scaling and no error bars. Construction-time human validation only filters dialogue quality; it does not validate the evaluation metric, so the central finding of “significant deficiencies in open-ended scientific exploration” is carried entirely by a self-referential LLM ranking.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents EarthSE, a three-tier benchmark for evaluating LLMs' Earth science knowledge and open-ended scientific exploration. From a corpus of roughly 103,000 Earth science papers, the authors construct two QA datasets — Earth-Iron (4,133 questions; five spheres, 114 sub-disciplines, 11 task categories) for broad foundational assessment and Earth-Silver (built from papers in high-impact journals) for difficult, professional-level assessment — and Earth-Gold, a set of open-ended two-turn dialogues derived from 1,000 highly cited papers. For Earth-Gold the paper introduces the Scientific Exploration Score (SES), defined as the product of a retention rate, obtained by having GPT-4 rank model responses against a GPT-4-generated reference answer, and a diversity term based on average embedding cosine similarity across M sampled responses. Eleven proprietary and open-weight LLMs are evaluated; the main empirical findings are strong multiple-choice accuracy on Earth-Iron (roughly 90-94%), sharply lower accuracy on Earth-Silver and on fill-in-the-blank items, and low retention and diversity on Earth-Gold, which the paper interprets as significant deficiencies in complex Earth science reasoning and open-ended scientific exploration.","tokens_in":16985,"tokens_out":17787,"duration_ms":125760,"significance":"The resource itself is substantial and, if the measurement concerns below are addressed, would be a useful addition to the field: it is publicly released on HuggingFace, built at a scale and disciplinary breadth (103K papers, 114 sub-disciplines, five spheres, 11 task categories) that exceeds existing Earth-science-specific benchmarks such as ClimaQA and OceanBench, and it is the first to attempt a dedicated assessment of open-ended exploration in this domain. The paper is transparent about its construction, shipping the full QA and dialogue prompts in the appendices together with worked examples, and the three-tier Iron/Silver/Gold design is a sensible operationalization of 'foundational to advanced' capability. The CoT-step-hint experiment (Table 6) is a clean, falsifiable result. The weaknesses are concentrated in the unvalidated measurement chain behind SES and, to a lesser degree, the QA evaluation; they are fixable with additional human-validation studies and a corrected, stabilized metric.","major_comments":[{"comment":"The paper's headline conclusion — that current LLMs show 'significant deficiencies in ... open-ended scientific exploration' (Abstract; §5) — rests entirely on Earth-Gold's SES, whose measurement chain is unvalidated. In §3.4 the reference answer for each dialogue is GPT-4's own extraction of the source paper's M0/L0/M1/L1 components, polished into a two-turn dialogue (Appendix C, Figures 11-12), and retention rate r is computed by having GPT-4 rank M model-generated responses against that single reference. 'Retention' therefore conflates exploration quality with proximity to one GPT-4 paraphrase of one specific paper's method and limitation analysis; a model that proposes a different but scientifically valid direction is penalized unless GPT-4 prefers it over the paper's own approach, and the judge belongs to the same model family that generated the reference. The construction-time expert validation in §3.4 filters dialogue quality but does not validate the ranking or the metric, and the diversity term d is computed only on the model's own responses, so it is orthogonal to exploration quality. The natural fix, which I would regard as necessary for the claim, is (i) a stratified human-expert study measuring agreement with GPT-4's rankings on Earth-Gold and (ii) either multiple references per dialogue (expert-written or from several strong models) or a robustness analysis of SES to the choice of reference.","section":"§3.4 (Earth-Gold, SES); §4.3; Conclusion"},{"comment":"The SES definition in §3.4 is internally inconsistent and numerically fragile. The text defines d = 1/s̄ via Eq. (1), then says that 'we normalize s̄ to [0.9,1]' and gives SES = r/(10(s̄−0.9)); these are different functions — over s̄ in [0.9,1], 1/s̄ ranges only over [1,1.11], while 1/(10s̄−9) ranges from infinity down to 1 and diverges at s̄ = 0.9. The diversity values in Table 5 (up to 3.98) are impossible under the former but consistent with the divergent form, so the formula actually used differs from the one printed. In addition, the tabulated SES values do not equal the product of the tabulated retention and diversity (e.g., Llama-3.1-8B: 0.08 × 3.98 = 0.318 versus the reported 0.3301), leaving it unclear whether SES is averaged per dialogue or computed from aggregate means. With M = 3 sampled responses per question, the diversity estimate has very high variance, and Table 5 reports no error bars or sensitivity analysis. The authors should state the exact formula used, justify or remove the [0.9,1] clipping, and report per-dialogue variability (e.g., bootstrap intervals).","section":"§3.4, Eq. (1) and SES normalization; Table 5"},{"comment":"The QA evaluation chain that supports the 'complex Earth science reasoning' part of the headline claim is also model-centric, and the difficulty-screening procedure is under-specified. §3.3 reports removing questions whose LLM accuracy exceeds 80% and requiring expert review for the 60-80% band, yet Earth-Iron multiple-choice accuracy for the evaluated models reaches 93-94% (Table 4); this apparent tension needs to be resolved by stating which models and which accuracy definition were used for screening. More substantively, GPT-4 generates the reference answers and CoTs, GPT-4 is the judge for the free-response Win Rate (Appendix D), and no human verification of answer correctness on a sample of Earth-Iron/Earth-Silver is reported, nor are inter-annotator statistics for the expert retention decisions. A small-scale human audit of stratified samples of answers, and of judge decisions, would substantiate that the reported deficiencies reflect genuine knowledge gaps rather than artifacts of the generation-evaluation loop.","section":"§3.3 (difficulty screening); §4.2; Table 4; Appendix D"}],"minor_comments":[{"comment":"In 'Pbase, Phj, Phj are used to construct Earth-Iron, Earth-Silver, Earth-Gold respectively', the third collection should presumably be Phc; the same typo should be checked throughout the paper.","section":"§3.2"},{"comment":"The journal list in Table 7 includes CELL, NATURE MEDICINE, NATURE CELL BIOLOGY, and NATURE BIOTECHNOLOGY, which are not Earth science venues and sit oddly with the claim that Phj comprises papers from 'high-impact Earth science journals'; please clarify the selection procedure or correct the table.","section":"Table 7 (Appendix A)"},{"comment":"The size of Earth-Silver and the number of dialogues in Earth-Gold are never reported; Table 1's '4133' question count appears to cover only Earth-Iron.","section":"§3.3 / Table 1"},{"comment":"Please specify whether SES is computed per user turn and then averaged over the two dialogue turns, and report the decoding settings used for the GPT-4 ranking calls (e.g., temperature, number of calls, aggregation).","section":"§3.4 / §4.3"},{"comment":"The human-expert steps (question retention, dialogue validation) are described without quantitative detail; the number of experts, items reviewed, and agreement statistics should be added.","section":"§3.3 and §3.4"},{"comment":"The sentence 'Earth-Gold reveals below low retention and diversity in open-ended dialogues' contains a typo ('below low').","section":"§5"}],"recommendation":"major_revision","confidential_remarks":"The resource is real and the paper is likely salvageable, but the journal should not take the current headline finding at face value: the Earth-Gold results are produced by an unvalidated, self-referential pipeline (GPT-4 writes the reference, GPT-4 judges, and the diversity term is independent of quality), and the SES formula is printed in two inconsistent forms. I would require the human-judge agreement study and the formula correction before accepting. The Table 7 journal-list oddity is a minor flag, but I would also ask the authors to audit the corpus selection to confirm that Earth-Silver and Earth-Gold are drawn from genuinely Earth-science venues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read EarthSE. The useful thing here is the scale and the structure: 100k papers distilled into two QA tiers and a dialogue set, covering five spheres and 114 disciplines. That breadth is new. The QA construction is fairly standard GPT-4-from-abstracts, but the difficulty filtering and human checks give it some value as a stress test. The results tables give a plausible picture: models do okay on MC/TF, poorly on FIB and calculation, and the harder Silver set separates them.\n\nThe soft spot is Earth-Gold. The stress-test note is right: the reference answer is built from the same paper's method and limitation text, and a single GPT-4 judge decides whether a model's response beats it. So SES measures something like 'produces text GPT-4 prefers over the paper's own summary,' not open-ended exploration in any validated sense. The diversity term is a weak multiplicative add-on with arbitrary normalization. Without human judgments on a sample of model responses, the claim that 'LLMs are deficient at scientific exploration' is not secured. Also missing: code and error bars on the rankings. Those are fixable.\n\nI'd push back on one part of the reader's pessimism: the QA tiers are not ruined by the circularity. The reference answers are derived from papers, but for fact-like questions that's fine; the issue is mainly the free-response win rate and the Gold dialogues. Also, difficulty filtering is standard practice, though it does mean the benchmark is calibrated to be hard, not representative.\n\nBottom line: this is a useful resource for anyone benchmarking Earth science LLMs, and the dialogue track is a reasonable idea that needs revalidation. Send it to review, but with a strong request for human evaluation of Gold responses, an independent judge (or a different model family as judge), and code. If those are added, it becomes a solid benchmark paper.","headline":"A big, useful Earth science QA benchmark dragged down by an unvalidated 'exploration' metric built on GPT-4 judging GPT-4.","tokens_in":17534,"tokens_out":1575,"would_cite":true,"duration_ms":12295,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EarthSE claims LLMs are strong at Earth-science facts but weak at open-ended scientific exploration.","keywords":["Earth science benchmark","large language models","scientific exploration","open-ended dialogue","question answering","scientific exploration score","Earth-Gold","chain-of-thought"],"falsifier":"Compare SES rankings on Earth-Gold against a version where each reference answer is replaced by a purely extractive summary of the same paper with all 'novel method' and 'limitation' sentences removed; if model rankings and scores barely change, the metric is measuring summary fidelity, not exploration.","tokens_in":16563,"feed_emoji":"🌍","tokens_out":8503,"duration_ms":69172,"temperature":0.7,"pith_summary":"EarthSE is a three-tier benchmark that tries to measure scientific exploration in LLMs, not just factual recall. It is built from a corpus of roughly 100,000 Earth science papers, organized into five spheres and 114 disciplines; from those papers the authors construct two question-answer datasets (Earth-Iron for broad coverage, Earth-Silver for professional depth) and one open-ended dialogue dataset (Earth-Gold). The central claim is that current leading LLMs handle Earth science facts reasonably well, especially in multiple-choice format, but degrade sharply on exact-term generation, multi-step calculation, and open-ended exploration dialogues, where most model responses are vaguer and more repetitive than the paper-derived reference answers. A sympathetic reader should take away that the bottleneck in AI for Earth science is not stored knowledge but the ability to criticise a field's methods and propose new ones.","feed_headline":"LLMs know Earth facts but fail at open-ended exploration","feed_subtitle":"Three-tier EarthSE benchmark from 100,000 papers finds weak reasoning and generic answers beyond basic recall.","key_machinery":"The carrying object is the tiered paper collection and the formal exploration cycle. Starting from a base collection $P_{\\mathrm{base}}$ of 100,000 papers, the authors select $P_{\\mathrm{hj}}$ (10,000 papers in high-impact journals) for Earth-Silver and $P_{\\mathrm{hc}}$ (1,000 most-cited papers) for Earth-Gold. Scientific exploration is formalized as $(M_{i+1}, L_{i+1}) = \\mathrm{LLM}(M_i, L_i)$: a model takes a methodology $M$ and its limitation analysis $L$ and produces the next methodology with its own limitations. Each Earth-Gold dialogue is constructed from the paper's $M_0, L_0, M_1, L_1$ components, and scored by the Scientific Exploration Score $\\mathrm{SES} = r \\times d$, where retention $r$ is the rank of the reference answer among sampled model responses and diversity $d=1/\\bar{s}$ is the inverse of the mean cosine similarity of the responses to their own centroid; in practice the paper rescales $\\bar{s}$ to $[0.9,1]$, writing $\\mathrm{SES}=r/(10(\\bar{s}-0.9))$.","core_discovery":"The paper's central discovery is that scientific exploration can be decomposed and measured as an iterated self-negation process, and that current LLMs are weak at it. EarthSE operationalizes exploration through Earth-Gold, where each highly cited paper is reduced to four components — existing method summary $M_0$, limitation analysis $L_0$, new method proposal $M_1$, and new-method limitations $L_1$ — and then cast as a two-turn dialogue. Across 11 leading models, the evaluation finds strong multiple-choice performance but consistently low accuracy on fill-in-the-blank and calculation, and retention rates under 50% on Earth-Gold, meaning the majority of model-generated exploration responses fail to beat the paper-derived reference. The authors interpret this as evidence that LLMs have not yet acquired robust scientific exploration ability, and they show that giving a few chain-of-thought steps at inference time substantially raises performance on the hardest closed-form questions.","pith_inferences":["A testable extension the authors do not run: replace the Earth-Gold reference answers with abstractive summaries of the same papers (with the novel-method sentences removed) and recompute SES; if rankings barely shift, the metric is tracking summarization rather than exploration.","The same $M_0/L_0/M_1/L_1$ decomposition could be inverted to make training data: Earth-Gold dialogues could be used as supervised fine-tuning examples to teach self-critique, rather than only as an evaluation set.","Because the pipeline only depends on papers having related-work, method, and discussion sections, EarthSE's construction can be transferred to other empirical sciences; the exploration cycle is not Earth-specific."],"forward_implications":["Earth-Gold's low retention scores imply that current LLMs, when asked to reflect on a field's methods and propose improvements, usually produce answers that are more generic than the paper-derived reference; open-ended scientific exploration is not yet a usable LLM capability.","The Earth-Silver results imply that professional-depth Earth science is far from solved: even the strongest tested models sit near 54% accuracy on multiple-choice questions and near 11% on fill-in-the-blank.","The CoT-hint experiments imply that a substantial share of the apparent difficulty on fill-in-the-blank questions is inference-time reasoning, not missing knowledge, since a few prompted reasoning steps roughly double or triple accuracy.","The SES metric implies that exploration quality and conceptual diversity can be traded off explicitly: models that repeat themselves are penalized even when individual answers are strong."],"supporting_citations":[{"why":"Supplies the 100,000-paper Earth science corpus from which all three EarthSE datasets are built.","marker":"[14]"},{"why":"The LLM used to generate QA pairs, structure papers into M0/L0/M1/L1, build dialogues, and rank free responses.","marker":"[1]"},{"why":"The document extraction tool that converts the 100,000 PDFs into structured JSON for corpus construction.","marker":"[40]"},{"why":"Source of the recurrent research pattern 'analyze limitations of existing work, then propose novel methods' that defines scientific exploration in Earth-Gold.","marker":"[49]"},{"why":"Supports the claim that genuine scientific discovery requires iterative self-critique and self-improvement.","marker":"[21]"},{"why":"Chain-of-thought reasoning is the format used in QA construction and in the CoT-hint experiments that improve fill-in-the-blank accuracy.","marker":"[44]"},{"why":"Motivates sampling M diverse responses at temperature=0.6 for the retention and diversity computation in SES.","marker":"[31]"},{"why":"Used in the paper as the reference for the win-rate metric on free-response questions.","marker":"[2]"}],"fun_headline_variants":["LLMs ace Earth trivia but fail at open-ended exploration","New benchmark proves LLMs struggle with scientific exploration","EarthSE: Where LLMs stumble on Earth science exploration","Open-ended Earth science tasks stump today's LLMs","EarthSE reveals LLMs can't explore Earth science beyond recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the paper-derived reference answers for Earth-Gold — an LLM's summary of a paper's method and limitations — are the right gold standard for open-ended scientific exploration, so that ranking a model's response above them measures exploration rather than paraphrasing.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace Earth trivia but fail at open-ended exploration","New benchmark proves LLMs struggle with scientific exploration","EarthSE: Where LLMs stumble on Earth science exploration","Open-ended Earth science tasks stump today's LLMs","EarthSE reveals LLMs can't explore Earth science beyond recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000572,"raw_usage":{"total_tokens":2721,"prompt_tokens":979,"completion_tokens":1742,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1663}},"tokens_in":595,"tokens_out":1742,"duration_ms":10532,"temperature":1.0,"reasoning_tokens":1663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:03:37.124615+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare SES rankings on Earth-Gold against a version where each reference answer is replaced by a purely extractive summary of the same paper with all 'novel method' and 'limitation' sentences removed; if model rankings and scores barely change, the metric is measuring summary fidelity, not exploration.","supporting_citations":[{"cited_title":"Iterative large language models evolution through self-critique","cited_arxiv_id":null,"evidence_quote":"Supports the claim that genuine scientific discovery requires iterative self-critique and self-improvement."}],"review_version":1}