Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read EarthSE claims LLMs are strong at Earth-science facts but weak at open-ended scientific exploration.

desk verdict A big, useful Earth science QA benchmark dragged down by an unvalidated 'exploration' metric built on GPT-4 judging GPT-4. read the letter →

arxiv 2505.17139 v3 pith:G6PKJUTN submitted 2025-05-22 cs.CL cs.AI

Wanghan Xu , Xiangyu Zhao , Yuhao Zhou , Xiaoyu Yue , Ben Fei , Fenghua Ling , Wenlong Zhang , Lei Bai This is my paper
classification cs.CLcs.AI
keywords Earthsciencebenchmarklargelanguagemodelsscientificexplorationopen-endeddialoguequestionansweringscoreEarth-Goldchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EarthSE is a three-tier benchmark that tries to measure scientific exploration in LLMs, not just factual recall. It is built from a corpus of roughly 100,000 Earth science papers, organized into five spheres and 114 disciplines; from those papers the authors construct two question-answer datasets (Earth-Iron for broad coverage, Earth-Silver for professional depth) and one open-ended dialogue dataset (Earth-Gold). The central claim is that current leading LLMs handle Earth science facts reasonably well, especially in multiple-choice format, but degrade sharply on exact-term generation, multi-step calculation, and open-ended exploration dialogues, where most model responses are vaguer and more repetitive than the paper-derived reference answers. A sympathetic reader should take away that the bottleneck in AI for Earth science is not stored knowledge but the ability to criticise a field's methods and propose new ones.

What carries the argument

The carrying object is the tiered paper collection and the formal exploration cycle. Starting from a base collection $P_{\mathrm{base}}$ of 100,000 papers, the authors select $P_{\mathrm{hj}}$ (10,000 papers in high-impact journals) for Earth-Silver and $P_{\mathrm{hc}}$ (1,000 most-cited papers) for Earth-Gold. Scientific exploration is formalized as $(M_{i+1}, L_{i+1}) = \mathrm{LLM}(M_i, L_i)$: a model takes a methodology $M$ and its limitation analysis $L$ and produces the next methodology with its own limitations. Each Earth-Gold dialogue is constructed from the paper's $M_0, L_0, M_1, L_1$ components, and scored by the Scientific Exploration Score $\mathrm{SES} = r \times d$, where retention $r$ is the rank of the reference answer among sampled model responses and diversity $d=1/\bar{s}$ is the inverse of the mean cosine similarity of the responses to their own centroid; in practice the paper rescales $\bar{s}$ to $[0.9,1]$, writing $\mathrm{SES}=r/(10(\bar{s}-0.9))$.

What would settle it

Compare SES rankings on Earth-Gold against a version where each reference answer is replaced by a purely extractive summary of the same paper with all 'novel method' and 'limitation' sentences removed; if model rankings and scores barely change, the metric is measuring summary fidelity, not exploration.

Watch

Extended reading notes

Core claim

The paper's central discovery is that scientific exploration can be decomposed and measured as an iterated self-negation process, and that current LLMs are weak at it. EarthSE operationalizes exploration through Earth-Gold, where each highly cited paper is reduced to four components — existing method summary $M_0$, limitation analysis $L_0$, new method proposal $M_1$, and new-method limitations $L_1$ — and then cast as a two-turn dialogue. Across 11 leading models, the evaluation finds strong multiple-choice performance but consistently low accuracy on fill-in-the-blank and calculation, and retention rates under 50% on Earth-Gold, meaning the majority of model-generated exploration responses fail to beat the paper-derived reference. The authors interpret this as evidence that LLMs have not yet acquired robust scientific exploration ability, and they show that giving a few chain-of-thought steps at inference time substantially raises performance on the hardest closed-form questions.

Load-bearing premise

The load-bearing premise is that the paper-derived reference answers for Earth-Gold — an LLM's summary of a paper's method and limitations — are the right gold standard for open-ended scientific exploration, so that ranking a model's response above them measures exploration rather than paraphrasing.

Editorial extensions

If this is right

  • Earth-Gold's low retention scores imply that current LLMs, when asked to reflect on a field's methods and propose improvements, usually produce answers that are more generic than the paper-derived reference; open-ended scientific exploration is not yet a usable LLM capability.
  • The Earth-Silver results imply that professional-depth Earth science is far from solved: even the strongest tested models sit near 54% accuracy on multiple-choice questions and near 11% on fill-in-the-blank.
  • The CoT-hint experiments imply that a substantial share of the apparent difficulty on fill-in-the-blank questions is inference-time reasoning, not missing knowledge, since a few prompted reasoning steps roughly double or triple accuracy.
  • The SES metric implies that exploration quality and conceptual diversity can be traded off explicitly: models that repeat themselves are penalized even when individual answers are strong.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the authors do not run: replace the Earth-Gold reference answers with abstractive summaries of the same papers (with the novel-method sentences removed) and recompute SES; if rankings barely shift, the metric is tracking summarization rather than exploration.
  • The same $M_0/L_0/M_1/L_1$ decomposition could be inverted to make training data: Earth-Gold dialogues could be used as supervised fine-tuning examples to teach self-critique, rather than only as an evaluation set.
  • Because the pipeline only depends on papers having related-work, method, and discussion sections, EarthSE's construction can be transferred to other empirical sciences; the exploration cycle is not Earth-specific.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents EarthSE, a three-tier benchmark for evaluating LLMs' Earth science knowledge and open-ended scientific exploration. From a corpus of roughly 103,000 Earth science papers, the authors construct two QA datasets — Earth-Iron (4,133 questions; five spheres, 114 sub-disciplines, 11 task categories) for broad foundational assessment and Earth-Silver (built from papers in high-impact journals) for difficult, professional-level assessment — and Earth-Gold, a set of open-ended two-turn dialogues derived from 1,000 highly cited papers. For Earth-Gold the paper introduces the Scientific Exploration Score (SES), defined as the product of a retention rate, obtained by having GPT-4 rank model responses against a GPT-4-generated reference answer, and a diversity term based on average embedding cosine similarity across M sampled responses. Eleven proprietary and open-weight LLMs are evaluated; the main empirical findings are strong multiple-choice accuracy on Earth-Iron (roughly 90-94%), sharply lower accuracy on Earth-Silver and on fill-in-the-blank items, and low retention and diversity on Earth-Gold, which the paper interprets as significant deficiencies in complex Earth science reasoning and open-ended scientific exploration.

Significance. The resource itself is substantial and, if the measurement concerns below are addressed, would be a useful addition to the field: it is publicly released on HuggingFace, built at a scale and disciplinary breadth (103K papers, 114 sub-disciplines, five spheres, 11 task categories) that exceeds existing Earth-science-specific benchmarks such as ClimaQA and OceanBench, and it is the first to attempt a dedicated assessment of open-ended exploration in this domain. The paper is transparent about its construction, shipping the full QA and dialogue prompts in the appendices together with worked examples, and the three-tier Iron/Silver/Gold design is a sensible operationalization of 'foundational to advanced' capability. The CoT-step-hint experiment (Table 6) is a clean, falsifiable result. The weaknesses are concentrated in the unvalidated measurement chain behind SES and, to a lesser degree, the QA evaluation; they are fixable with additional human-validation studies and a corrected, stabilized metric.

major comments (3)
  1. [§3.4 (Earth-Gold, SES); §4.3; Conclusion] The paper's headline conclusion — that current LLMs show 'significant deficiencies in ... open-ended scientific exploration' (Abstract; §5) — rests entirely on Earth-Gold's SES, whose measurement chain is unvalidated. In §3.4 the reference answer for each dialogue is GPT-4's own extraction of the source paper's M0/L0/M1/L1 components, polished into a two-turn dialogue (Appendix C, Figures 11-12), and retention rate r is computed by having GPT-4 rank M model-generated responses against that single reference. 'Retention' therefore conflates exploration quality with proximity to one GPT-4 paraphrase of one specific paper's method and limitation analysis; a model that proposes a different but scientifically valid direction is penalized unless GPT-4 prefers it over the paper's own approach, and the judge belongs to the same model family that generated the reference. The construction-time expert validation in §3.4 filters dialogue quality but does not validate the ranking or the metric, and the diversity term d is computed only on the model's own responses, so it is orthogonal to exploration quality. The natural fix, which I would regard as necessary for the claim, is (i) a stratified human-expert study measuring agreement with GPT-4's rankings on Earth-Gold and (ii) either multiple references per dialogue (expert-written or from several strong models) or a robustness analysis of SES to the choice of reference.
  2. [§3.4, Eq. (1) and SES normalization; Table 5] The SES definition in §3.4 is internally inconsistent and numerically fragile. The text defines d = 1/s̄ via Eq. (1), then says that 'we normalize s̄ to [0.9,1]' and gives SES = r/(10(s̄−0.9)); these are different functions — over s̄ in [0.9,1], 1/s̄ ranges only over [1,1.11], while 1/(10s̄−9) ranges from infinity down to 1 and diverges at s̄ = 0.9. The diversity values in Table 5 (up to 3.98) are impossible under the former but consistent with the divergent form, so the formula actually used differs from the one printed. In addition, the tabulated SES values do not equal the product of the tabulated retention and diversity (e.g., Llama-3.1-8B: 0.08 × 3.98 = 0.318 versus the reported 0.3301), leaving it unclear whether SES is averaged per dialogue or computed from aggregate means. With M = 3 sampled responses per question, the diversity estimate has very high variance, and Table 5 reports no error bars or sensitivity analysis. The authors should state the exact formula used, justify or remove the [0.9,1] clipping, and report per-dialogue variability (e.g., bootstrap intervals).
  3. [§3.3 (difficulty screening); §4.2; Table 4; Appendix D] The QA evaluation chain that supports the 'complex Earth science reasoning' part of the headline claim is also model-centric, and the difficulty-screening procedure is under-specified. §3.3 reports removing questions whose LLM accuracy exceeds 80% and requiring expert review for the 60-80% band, yet Earth-Iron multiple-choice accuracy for the evaluated models reaches 93-94% (Table 4); this apparent tension needs to be resolved by stating which models and which accuracy definition were used for screening. More substantively, GPT-4 generates the reference answers and CoTs, GPT-4 is the judge for the free-response Win Rate (Appendix D), and no human verification of answer correctness on a sample of Earth-Iron/Earth-Silver is reported, nor are inter-annotator statistics for the expert retention decisions. A small-scale human audit of stratified samples of answers, and of judge decisions, would substantiate that the reported deficiencies reflect genuine knowledge gaps rather than artifacts of the generation-evaluation loop.
minor comments (6)
  1. [§3.2] In 'Pbase, Phj, Phj are used to construct Earth-Iron, Earth-Silver, Earth-Gold respectively', the third collection should presumably be Phc; the same typo should be checked throughout the paper.
  2. [Table 7 (Appendix A)] The journal list in Table 7 includes CELL, NATURE MEDICINE, NATURE CELL BIOLOGY, and NATURE BIOTECHNOLOGY, which are not Earth science venues and sit oddly with the claim that Phj comprises papers from 'high-impact Earth science journals'; please clarify the selection procedure or correct the table.
  3. [§3.3 / Table 1] The size of Earth-Silver and the number of dialogues in Earth-Gold are never reported; Table 1's '4133' question count appears to cover only Earth-Iron.
  4. [§3.4 / §4.3] Please specify whether SES is computed per user turn and then averaged over the two dialogue turns, and report the decoding settings used for the GPT-4 ranking calls (e.g., temperature, number of calls, aggregation).
  5. [§3.3 and §3.4] The human-expert steps (question retention, dialogue validation) are described without quantitative detail; the number of experts, items reviewed, and agreement statistics should be added.
  6. [§5] The sentence 'Earth-Gold reveals below low retention and diversity in open-ended dialogues' contains a typo ('below low').

Circularity Check

1 steps flagged · score 5.0 of 10

Earth-Gold's SES is anchored to a paper-derived reference and a GPT-4 judge, so the 'open-ended exploration deficiency' finding is partly self-definitional; the Earth-Iron/Silver QA results are not circular.

  1. self definitional [Section 3.1 and Section 3.4, 'A New Metric: Scientific Exploration Score (SES)']
    "Following the scientific exploration task definition in Section 3.1, we decompose each paper in P hc into four structured components: Existing Method Summary (M 0), Limitation Analysis (L0), Novel Method Proposal (M 1), New Method Limitations (L1). ... With M 0, L0, M 1, and L1 extracted, we employ GPT-4 to generate two-turn dialogues simulating human-AI collaborative scientific exploration. ... Retention Rate ( r): GPT-4 ranks M generated answers and the reference answer by reflective depth and innovation."

    Section 3.1 formalizes scientific exploration as the iterative self-negation process (M^{i+1}, L^{i+1}) = LLM(M^i, L^i). Earth-Gold's reference answers are built by extracting M0/L0/M1/L1 from one paper and having GPT-4 generate the dialogue. The retention component r of SES is the rank of that same paper-derived reference among the evaluated model's responses, assigned by GPT-4. A high SES thus means, by construction, that GPT-4 ranks the model's text at or above one specific paper's method/limitation summary; it does not independently establish open-ended exploration ability. A model proposing a different but valid next step is scored only through GPT-4's preference against that single reference, and the diversity term d (Eq.

full rationale

EarthSE is primarily a benchmark-construction paper rather than a formal derivation, so most of its pipeline is non-circular: Earth-Iron/Earth-Silver use standard accuracy, win-rate, and semantic-similarity evaluations against GPT-4-generated QA pairs, and no load-bearing self-citation or imported uniqueness theorem was found. The circularity concern is confined to Earth-Gold's SES, the paper's stated core innovation: the reference answer is generated from the same M0/L0/M1/L1 components that define the exploration task, and GPT-4 both creates the reference and ranks model outputs against it. This makes the conclusion that LLMs lack open-ended scientific exploration partly self-definitional. The 80%-accuracy difficulty filter is a selection-bias issue and not a fitted-parameter prediction, and the diversity term is only a self-similarity statistic, so the score is 5 rather than 6-8. Independent human validation of the SES rankings or an external gold standard would remove the remaining circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central benchmark claims rest on unvalidated modeling choices: GPT-4-generated ground truth, journal and citation proxies, GPT-4 as judge, keyword sphere classification, and embedding-based diversity. None of these are externally benchmarked in the paper.

free parameters (3)
  • Difficulty screening threshold = 80%
    Questions with LLM accuracy above 80% were removed; this threshold shapes benchmark difficulty and is chosen by the authors, not derived.
  • SES normalization lower bound = 0.9
    Diversity is normalized to the [0.9, 1] cosine-similarity interval; the 0.9 constant directly affects every SES value and is arbitrary.
  • Number of responses M = 3
    M equals 3 for retention and diversity calculations in Table 5; this choice controls metric granularity.
assumptions (5)
  • domain assumption GPT-4-generated QA pairs from paper content are factually accurate enough to serve as benchmark ground truth.
    Section 3.3 uses GPT-4 to generate questions and answers and relies on LLM-based cleaning, with no reported human agreement.
  • domain assumption Journal impact factor and citation counts are valid proxies for paper quality and scientific exploration value.
    Section 3.2 selects Phj by high-impact journals and Phc by top-cited papers.
  • domain assumption GPT-4 can reliably rank open-ended responses by reflective depth and innovation.
    Section 3.4 defines retention using GPT-4 rankings, with no human correlation study.
  • domain assumption Semantic keyword classification into five spheres is accurate.
    Section 3.2 and Figure 7 use keyword thresholds to classify papers, with no validation accuracy reported.
  • domain assumption Sentence embedding cosine similarity captures conceptual diversity of scientific responses.
    Section 3.4 uses sentence-transformers embeddings to define diversity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs." pith.science (2026). https://pith.science/paper/G6PKJUTN

@misc{pith2026250517139,
  author       = {Pith},
  title        = {Pith review of: EarthSE: A Benchmark for Evaluating Earth Scientific Exploration Capability of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G6PKJUTN}},
  note         = {Machine review of arXiv:2505.17139}
}
read the original abstract

Advancements in Large Language Models (LLMs) drive interest in scientific applications, necessitating specialized benchmarks such as Earth science. Existing benchmarks either present a general science focus devoid of Earth science specificity or cover isolated subdomains, lacking holistic evaluation. Furthermore, current benchmarks typically neglect the assessment of LLMs' capabilities in open-ended scientific exploration. In this paper, we present a comprehensive and professional benchmark for the Earth sciences, designed to evaluate the capabilities of LLMs in scientific exploration within this domain, spanning from fundamental to advanced levels. Leveraging a corpus of 100,000 research papers, we first construct two Question Answering (QA) datasets: Earth-Iron, which offers extensive question coverage for broad assessment, and Earth-Silver, which features a higher level of difficulty to evaluate professional depth. These datasets encompass five Earth spheres, 114 disciplines, and 11 task categories, assessing foundational knowledge crucial for scientific exploration. Most notably, we introduce Earth-Gold with new metrics, a dataset comprising open-ended multi-turn dialogues specifically designed to evaluate the advanced capabilities of LLMs in scientific exploration, including methodology induction, limitation analysis, and concept proposal. Extensive experiments reveal limitations in 11 leading LLMs across different domains and tasks, highlighting considerable room for improvement in their scientific exploration capabilities. The benchmark is available on https://huggingface.co/ai-earth .

Figures

Figures reproduced from arXiv: 2505.17139 by the authors.

Figure 1
Figure 1. Cross-domain Evaluation of Mainstream LLMs on EarthSE. (a) EarthSE evaluates the capabilities of different LLMs in Earth’s five spheres. (b) The multi-task evaluation demonstrates pronounced capability limitations in tasks such as calculation and term explanation. Abstract Advancements in Large Language Models (LLMs) drive interest in scientific ap￾plications, necessitating specialized benchmarks such as Earth scien… view at source ↗
Figure 2
Figure 2. EarthSE Covers 114 Disciplines in Earth Sciences. ScienceQA [27], SciBench [41] and MMLU-Pro [42] cover disciplines from elementary to graduate levels. For instance, ScienceQA includes over 21,000 multi-modal questions spanning natural, social, and language sciences. MMLU-Pro extends MMLU [16] with broader college-level subjects to evaluate deeper reasoning. SciBench emphasize college-level problem solving, drawing … view at source ↗
Figure 3
Figure 3. Construction Process of EarthSE. Automated construction of three-tiered benchmarks (Earth-Iron, Earth-Silver, Earth-Gold) from 100K papers, enabling comprehensive evaluation of knowledge coverage, professional proficiency, and scientific exploration capabilities. focusing on a comprehensive evaluation of fundamental scientific exploration abilities. Earth-Silver (QA) features more difficult and challenging questions… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Citations of Papers in Phc. 3.3 Earth-Iron/Silver: QA Benchmark for Foundational Tasks Earth-Iron and Earth-Silver are constructed from Pbase and Phj respectively, using the same pipeline as shown in the first row of [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Case Study from Earth-Silver. Some models make mistakes when applying formulas in multiple steps, resulting in low accuracy in calculation questions. Compared to Earth-Iron, Earth-Silver is constructed from a higher-quality corpus of original papers, resulting in a sig…
Figure 6
Figure 6. Figure 6: Case Study from Earth-Gold. Earth-Gold is used to evaluate the performance of LLMs in open scientific exploration. For the same user questions, the reference answers contain more details, such as the yellow text, while the bad answers are very general. 4.4 CoT Guidance…
Figure 7
Figure 7. Figure 7: Positive and Negative Keywords. From the initial paper collection Pbase, we select a subset of papers Phj published in high-quality journals based on multiple criteria including impact factor (IF) [11], disciplinary ranking, and expert assessment. Representative journa…
Figure 8
Figure 8. Figure 8: Prompt for Different Question Formats. The prompts for different tasks are shown in [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 10
Figure 10. Figure 10: QA Refine Prompt. C Scientific Exploration Dialogue Construction The Earth-Gold dataset construction from the high-citation paper collection Phc involves a two-phase processing pipeline: 1. Paper Structuring Phase: Decompose each paper into four core components: • Sum…
Figure 11
Figure 11. Figure 11: Paper Structuring Prompt. The structured paper content enables the generation of two progressive dialogue rounds: • Round 1: The LLM summarizes existing methods (M0) and analyzes their limitations (L0). This round evaluates the model’s understanding of domain-specific…
Figure 12
Figure 12. Figure 12: Dialogue Construction Prompt. The <INPUT1> to <INPUT5> in the prompt are replaced by the values in the dictionary obtained in paper structuring. D General Metrics Accuracy (Acc.) For closed-form questions (multiple-choice, fill-in-the-blank, and true/false) with deter…
Figure 13
Figure 13. Figure 13: QA Example 1. QA Example 2 (True or False) Question Atmospheric humidity in Nanjing’s urban canopy layer demonstrates significant nighttime differences among local climate zones, particularly due to moisture deficits during colder months, and shows clear seasonal patt…
Figure 15
Figure 15. Figure 15: QA Example 3. QA Example 4 (Free QA) Question How does the redistribution of aerosol particles captured through nucleation and impaction scavenging into various drop size categories affect the dynamics and ultimate removal process of aerosol particles in the cloud env…
Figure 17
Figure 17. Figure 17: Scientific Exploration Dialogue Example 1. Earth-Gold: Scientific Exploration Dialogue Example 2 User Please examine the interplay and responsiveness of the convective boundary layer (CBL) under diverse atmospheric and oceanic conditions, including sea surface tempera…
Figure 18
Figure 18. Figure 18: Scientific Exploration Dialogue Example 2. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_18.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A research agenda calling for geo-temporal reasoning in deep research systems, with no experiments or system implementation.

Reference graph

Works this paper leans on

53 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Oceangpt: A large language model for ocean science tasks.arXiv preprint arXiv:2310.02031, 2023

    Zhen Bi, Ningyu Zhang, Yida Xue, Yixin Ou, Daxiong Ji, Guozhou Zheng, and Huajun Chen. Oceangpt: A large language model for ocean science tasks.arXiv preprint arXiv:2310.02031, 2023

    arXiv 2023

  3. [3]

    Matplotlib and seaborn

    Ekaba Bisong. Matplotlib and seaborn. InBuilding machine learning and deep learning models on google cloud platform: A comprehensive guide for beginners, pages 151–165. Springer, 2019

    work page 2019

  4. [4]

    This reference does not exist: an exploration of llm citation accuracy and relevance

    Courtni Byun, Piper Vasicek, and Kevin Seppi. This reference does not exist: an exploration of llm citation accuracy and relevance. InProceedings of the Third Workshop on Bridging Human–Computer Interaction and Natural Language Processing, pages 28–39, 2024

    work page 2024

  5. [5]

    Sciassess: Benchmarking llm proficiency in scientific literature analysis.arXiv preprint arXiv:2403.01976, 2024

    Hengxing Cai, Xiaochen Cai, Junhan Chang, Sihang Li, Lin Yao, Changxin Wang, Zhifeng Gao, Hongshuai Wang, Yongge Li, Mujie Lin, et al. Sciassess: Benchmarking llm proficiency in scientific literature analysis.arXiv preprint arXiv:2403.01976, 2024

    arXiv 2024

  6. [6]

    On the design and analysis of llm-based algorithms.arXiv preprint arXiv:2407.14788, 2024

    Yanxi Chen, Yaliang Li, Bolin Ding, and Jingren Zhou. On the design and analysis of llm-based algorithms.arXiv preprint arXiv:2407.14788, 2024

    work page arXiv 2024

  7. [7]

    Grok, gemini, chatgpt and deepseek: Com- parison and applications in conversational artificial intelligence.INTELIGENCIA ARTIFICIAL, 2(1), 2025

    Murillo Edson de Carvalho Souza and Li Weigang. Grok, gemini, chatgpt and deepseek: Com- parison and applications in conversational artificial intelligence.INTELIGENCIA ARTIFICIAL, 2(1), 2025

    work page 2025

  8. [8]

    K2: A foundation language model for geoscience knowledge understanding and utilization

    Cheng Deng, Tianhang Zhang, Zhongmou He, Qiyuan Chen, Yuanyuan Shi, Yi Xu, Luoyi Fu, Weinan Zhang, Xinbing Wang, Chenghu Zhou, et al. K2: A foundation language model for geoscience knowledge understanding and utilization. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 161–170, 2024

    work page 2024

Show all 53 references
  1. [9]

    A deep learning model based on bert and sentence transformer for semantic keyphrase extraction on big social data.IEEE Access, 9:165252–165261, 2021

    R Devika, Subramaniyaswamy Vairavasundaram, C Sakthi Jay Mahenthar, Vijayakumar Varadarajan, and Ketan Kotecha. A deep learning model based on bert and sentence transformer for semantic keyphrase extraction on big social data.IEEE Access, 9:165252–165261, 2021

    2021

  2. [10]

    Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    Xinrun Du, Yifan Yao, Kaijing Ma, Bingli Wang, Tianyu Zheng, King Zhu, Minghao Liu, Yiming Liang, Xiaolong Jin, Zhenlin Wei, et al. Supergpqa: Scaling llm evaluation across 285 graduate disciplines.arXiv preprint arXiv:2502.14739, 2025

    2025 arXiv

  3. [11]

    The impact factor.Current contents, 25(20):3–7, 1994

    Eugene Garfield et al. The impact factor.Current contents, 25(20):3–7, 1994

    1994

  4. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Llm-based code generation method for golang compiler testing

    Qiuhan Gu. Llm-based code generation method for golang compiler testing. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, pages 2201–2203, 2023

    2023

  6. [14]

    Opendatalab: Empowering general artificial intelligence with open datasets.arXiv preprint arXiv:2407.13773, 2024

    Conghui He, Wei Li, Zhenjiang Jin, Chao Xu, Bin Wang, and Dahua Lin. Opendatalab: Empowering general artificial intelligence with open datasets.arXiv preprint arXiv:2407.13773, 2024

    2024 arXiv

  7. [15]

    The accuracy, robustness, and readability of llm-generated sustainability-related word definitions.arXiv preprint arXiv:2502.00916, 2025

    Alice Heiman. The accuracy, robustness, and readability of llm-generated sustainability-related word definitions.arXiv preprint arXiv:2502.00916, 2025

    2025 arXiv

  8. [16]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020. 10

    2009 arXiv

  9. [17]

    The era5 global reanalysis.Quarterly journal of the royal meteorological society, 146(730):1999–2049, 2020

    Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz- Sabater, Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, et al. The era5 global reanalysis.Quarterly journal of the royal meteorological society, 146(730):1999–2049, 2020

    1999

  10. [18]

    Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024

    Raisa Islam and Owana Marzia Moushi. Gpt-4o: The cutting-edge advancement in multimodal llm.Authorea Preprints, 2024

    2024

  11. [19]

    Enhancing large language models with climate resources.arXiv preprint arXiv:2304.00116, 2023

    Mathias Kraus, Julia Anna Bingler, Markus Leippold, Tobias Schimanski, Chiara Colesanti Senni, Dominik Stammbach, Saeid Ashraf Vaghefi, and Nicolas Webersinke. Enhancing large language models with climate resources.arXiv preprint arXiv:2304.00116, 2023

    2023 arXiv

  12. [20]

    Ryo Kurokawa, Yuji Ohizumi, Jun Kanzawa, Mariko Kurokawa, Yuki Sonoda, Yuta Nakamura, Takao Kiguchi, Wataru Gonoi, and Osamu Abe. Diagnostic performances of claude 3 opus and claude 3.5 sonnet from patient history and key images in radiologyâ ˘A ´Zs â ˘AIJdiagnosis pleaseâ ˘A˙...

    2024

  13. [21]

    Iterative large language models evolution through self-critique

    Qianxi Li. Iterative large language models evolution through self-critique. 2024

    2024

  14. [22]

    Llm with relation classifier for document-level relation extraction.arXiv preprint arXiv:2408.13889, 2024

    Xingzuo Li, Kehai Chen, Yunfei Long, and Min Zhang. Llm with relation classifier for document-level relation extraction.arXiv preprint arXiv:2408.13889, 2024

    2024 arXiv

  15. [23]

    Perception, reason, think, and plan: A survey on large multimodal reasoning models.arXiv preprint arXiv:2505.04921, 2025

    Yunxin Li, Zhenyu Liu, Zitao Li, Xuanyu Zhang, Zhenran Xu, Xinyu Chen, Haoyuan Shi, Shenyuan Jiang, Xintong Wang, Jifang Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models.arXiv preprint arXiv:2505.04921, 2025

    2025 arXiv

  16. [24]

    Bryan Lim, Ishith Seth, Molly Maxwell, Roberto Cuomo, Richard J Ross, and Warren M Rozen. Evaluating the efficacy of large language models in generating medical documentation: A comparative study of chatgpt-4, chatgpt-4o, and claude.Aesthetic Plastic Surgery, pages 1–12, 2025

    2025

  17. [25]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    2024 arXiv

  18. [26]

    The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scien- tist: Towards fully automated open-ended scientific discovery.arXiv preprint arXiv:2408.06292, 2024

    2024 arXiv

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  20. [28]

    The five environmental spheres

    Stanley E Manahan. The five environmental spheres. InEnvironmental Science and Technology, pages 59–84. CRC Press, 2006

    2006

  21. [29]

    Climaqa: An auto- mated evaluation framework for climate foundation models.arXiv preprint arXiv:2410.16701, 2024

    Veeramakali Vignesh Manivannan, Yasaman Jafari, Srikar Eranky, Spencer Ho, Rose Yu, Duncan Watson-Parris, Yian Ma, Leon Bergen, and Taylor Berg-Kirkpatrick. Climaqa: An auto- mated evaluation framework for climate foundation models.arXiv preprint arXiv:2410.16701, 2024

    arXiv

  22. [30]

    Seafloorai: A large-scale vision- language dataset for seafloor geological survey.Advances in Neural Information Processing Systems, 37:22107–22123, 2024

    Kien Nguyen, Fengchun Qiao, Arthur Trembanis, and Xi Peng. Seafloorai: A large-scale vision- language dataset for seafloor geological survey.Advances in Neural Information Processing Systems, 37:22107–22123, 2024

    2024

  23. [31]

    Is temperature the creativity parameter of large language models?arXiv preprint arXiv:2405.00492, 2024

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. Is temperature the creativity parameter of large language models?arXiv preprint arXiv:2405.00492, 2024

    2024 arXiv

  24. [32]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025. 11

    2025 arXiv

  25. [33]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  26. [34]

    From calculation to adjudication: Examining llm judges on mathematical reasoning tasks.arXiv preprint arXiv:2409.04168, 2024

    Andreas Stephan, Dawei Zhu, Matthias Aßenmacher, Xiaoyu Shen, and Benjamin Roth. From calculation to adjudication: Examining llm judges on mathematical reasoning tasks.arXiv preprint arXiv:2409.04168, 2024

    2024 arXiv

  27. [35]

    Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    2022 arXiv

  28. [36]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  29. [37]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  30. [38]

    Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting. Large language models in medicine.Nature medicine, 29(8):1930–1940, 2023

    1930

  31. [39]

    Climatext: A dataset for climate change topic detection.arXiv preprint arXiv:2012.00483, 2020

    Francesco S Varini, Jordan Boyd-Graber, Massimiliano Ciaramita, and Markus Leippold. Climatext: A dataset for climate change topic detection.arXiv preprint arXiv:2012.00483, 2020

    2012 arXiv

  32. [40]

    Mineru: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction.arXiv preprint arXiv:2409.18839, 2024

    2024 arXiv

  33. [41]

    Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023

    Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models.arXiv preprint arXiv:2307.10635, 2023

    2023 arXiv

  34. [42]

    Mmlu-pro: A more robust and challenging multi-task language understanding benchmark

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, et al. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. InThe Thirty-eight Conference on Neural Information Pro...

    2024

  35. [43]

    Climatebert: A pretrained language model for climate-related text.arXiv preprint arXiv:2110.12010, 2021

    Nicolas Webersinke, Mathias Kraus, Julia Anna Bingler, and Markus Leippold. Climatebert: A pretrained language model for climate-related text.arXiv preprint arXiv:2110.12010, 2021

    2021 arXiv

  36. [44]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  37. [45]

    Measuring and reducing llm hallucination without gold-standard answers.arXiv preprint arXiv:2402.10412, 2024

    Jiaheng Wei, Yuanshun Yao, Jean-Francois Ton, Hongyi Guo, Andrew Estornell, and Yang Liu. Measuring and reducing llm hallucination without gold-standard answers.arXiv preprint arXiv:2402.10412, 2024

    2024 arXiv

  38. [46]

    Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering.arXiv preprint arXiv:2404.14741, 2024

    Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering.arXiv preprint arXiv:2404.14741, 2024

    2024 arXiv

  39. [47]

    The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    Yutaro Yamada, Robert Tjarko Lange, Cong Lu, Shengran Hu, Chris Lu, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist-v2: Workshop-level automated scientific discovery via agentic tree search.arXiv preprint arXiv:2504.08066, 2025

    2025 arXiv

  40. [48]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 12

    2024 arXiv

  41. [49]

    Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses.arXiv preprint arXiv:2410.07076, 2024

    Zonglin Yang, Wanhao Liu, Ben Gao, Tong Xie, Yuqiang Li, Wanli Ouyang, Soujanya Poria, Erik Cambria, and Dongzhan Zhou. Moose-chem: Large language models for rediscovering unseen chemistry scientific hypotheses.arXiv preprint arXiv:2410.07076, 2024

  42. [50]

    Easytool: Enhancing llm-based agents with concise tool instruction.arXiv preprint arXiv:2401.06201, 2024

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dongsheng Li, and Deqing Yang. Easytool: Enhancing llm-based agents with concise tool instruction.arXiv preprint arXiv:2401.06201, 2024

    2024 arXiv

  43. [51]

    Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852, 2024

    Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, et al. Chemllm: A chemical large language model.arXiv preprint arXiv:2402.06852, 2024

    2024 arXiv

  44. [52]

    Towards llm-based fact verification on news claims with a hierarchi- cal step-by-step prompting method.arXiv preprint arXiv:2310.00305, 2023

    Xuan Zhang and Wei Gao. Towards llm-based fact verification on news claims with a hierarchi- cal step-by-step prompting method.arXiv preprint arXiv:2310.00305, 2023

    2023 arXiv

  45. [53]

    earth" as a positive keyword, astronomy-related papers may incorrectly appear relevant. To address this, we introduce

    Yifan Zhang, Cheng Wei, Shangyou Wu, Zhengting He, and Wenhao Yu. Geogpt: Understanding and processing geospatial tasks through an autonomous gpt.arXiv preprint arXiv:2307.07930, 2023. 13 A Paper Selection and Tiered Classification The proposed benchmark derives from 100,000 p...

    1973 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.