Pith. sign in

REVIEW 2 major objections 6 minor 30 references

General code-search benchmarks miss scientific needs; new multi-domain datasets show retrieval quality swings sharply with field-specific documentation culture.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 01:26 UTC pith:MXS4ET3A

load-bearing objection Solid, usable IR infrastructure for scientific code discovery: real SME queries, public multi-domain corpus, and baselines that cleanly show domain and documentation-culture gaps. the 2 major comments →

arxiv 2607.05443 v1 pith:MXS4ET3A submitted 2026-07-03 cs.IR cs.AIcs.SE

Scientific Code Search at Scale: A Multi-Domain Dataset and Benchmark

Nishan Pantha , Pranath Reddy Kumbam , Sajil Awale , Pushwitha Krishnappa , Muthukumaran Ramasubramanian , Nidhi Jha , Emily Foshee , Ankur Kumar
show 3 more authors
Rachel Slank Ashkbiz Danehkar Rahul Ramachandran
This is my paper
classification cs.IR cs.AIcs.SE
keywords code searchinformation retrievalscientific softwarebenchmark datasetscode retrievalrepository searchdomain-specific vocabularydocumentation enrichment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Scientists need to find open research software among hundreds of millions of public repositories, but everyday search tools and existing code-search benchmarks are built for general software engineering, not for mission names, instrument acronyms, and specialized data formats. This paper builds a curated set of thousands of domain-classified scientific repositories and two evaluation tasks grounded in real scientific information needs: one for finding whole tools from expert queries, and one for retrieving functions and classes from natural-language descriptions and identifiers across seven languages. Baselines show large, systematic gaps—repository search is much stronger in fields with standardized documentation and weaker where READMEs assume insider knowledge—while code-level search succeeds on docstrings but collapses on terse scientific naming conventions. The released corpus and benchmarks give the community a way to measure and improve scientific tool discovery rather than treating all code search as the same problem.

Core claim

The paper establishes that scientific code discovery is a distinct information-retrieval problem whose difficulty is driven by domain vocabulary and documentation culture, not only by model choice. Using a curated multi-domain scientific repository corpus, expert-curated repository queries, and a large code-snippet retrieval set, baseline lexical and dense methods exhibit large performance variation across scientific fields and between docstring versus identifier queries, with context enrichment helping repository search and general language-model embeddings leading on code snippets.

What carries the argument

A multi-source curated scientific repository corpus plus two IR benchmarks built from it: expert repository-search queries with multi-relevance ground truth, and a large snippet corpus of function/class units paired with docstring and identifier queries, evaluated with standard ranking metrics under lexical, dense, and hybrid retrieval.

Load-bearing premise

The repository-search labels assume that experts starting from tools they already know will mark enough of the truly relevant repositories for each query.

What would settle it

Have independent domain experts, given each query and full corpus access without prior tool lists, re-label relevant repositories; if many additional true matches appear—especially in the harder domains—or if domain ranking order changes, the reported gaps and scores would not hold.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Scientific tool discovery systems need evaluation on expert scientific queries and multi-relevance labels, not only general software-engineering benchmarks.
  • Enriching sparse repository documentation with cleaned READMEs, topics, and linked external context is a practical lever that improves retrieval across methods.
  • Code retrieval for science must handle abbreviated domain naming; docstring-style queries work far better than identifier queries under current single-pass methods.
  • Domain-specific scientific-text embeddings help repository-level search more than code-level search, so model choice should match the granularity of the task.
  • Public multi-domain benchmarks enable measuring progress on agentic multi-step retrieval and retrieval-augmented scientific workflows.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Fields with sparse project-level docs but strict API/documentation standards may invert the usual ranking: hard for repository discovery, easier for function-level search.
  • If identifier-query failure is mostly naming convention, hybrid pipelines that expand abbreviations against domain glossaries could close much of the docstring–identifier gap without new pretraining.
  • Under-represented scientific communities will keep looking artificially hard until collection sources match the registries used for better-covered fields.
  • The same documentation-culture signal that drives these gaps likely affects reproducibility and reuse outside search: tools that are hard to retrieve are also hard to credit and re-adopt.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper argues that general-purpose code-search benchmarks miss scientific vocabulary and needs, and addresses this by releasing a curated corpus of 5,264 domain-classified GitHub repositories across five NASA SMD divisions, plus two IR benchmarks: a repository-search set of 219 SME-curated queries with multi-relevance labels, and a code-snippet set of 117,950 snippets / 119,720 queries across seven languages. Multi-source collection, LLM domain classification (validated F1 0.85/0.93 on two 100-sample sets), README cleaning, and external-context enrichment are described; baselines (BM25, MiniLM, INDUS, hybrids, Qwen3, SFR-Code) show large domain and query-type gaps (e.g., repository MRR@10 ~.87 Astrophysics vs ~.22 Planetary; docstring vs identifier ~.76 vs ~.25).

Significance. If the released resources hold up under community use, this is a clear contribution to scientific IR and open science: the first multi-domain NASA-grounded code-search benchmarks with expert queries, public HuggingFace datasets, evaluation scripts, and multi-method baselines. Strengths include multi-source collection with explicit quality filters, validated LLM classification with reasoning fields, multi-relevance SME labels, BEIR/MTEB-compatible formats, hybrid baselines, and transparent limitations (domain imbalance, incomplete multi-relevance, English-only). The empirical patterns (documentation-culture effects, context-enrichment gains, docstring/identifier gap) are actionable for retrieval and documentation practice.

major comments (2)
  1. Section 4.1: repository-to-query SME annotation based on prior familiarity is disclosed as favoring precision over exhaustive recall, so relevant repositories outside annotators' knowledge may be missing. Absolute Recall/NDCG are therefore soft, and some domain gaps could partly be annotation artifacts. The directional patterns (Astro ≫ Planetary; docstring ≫ identifier; context-enrichment gains) remain load-bearing, but the paper should quantify incompleteness where possible (e.g., inter-annotator expansion, pool-based re-judgment on top-k from strong systems, or leave-one-annotator-out) or more explicitly bound claims to precision-oriented evaluation.
  2. Tables 2 and 4 vs Sections 6.2–6.3: repository-search queries cover only three of five SMD divisions (~140 Earth / ~50 Astro / ~30 Planetary), while Heliophysics and Bio/Physical are present in the corpus but not in the SME query set. Cross-domain claims and 'holistic' aggregates are therefore driven by the three annotated domains; the paper should either add SME queries for the missing divisions or clearly scope headline domain-variation claims to the three annotated fields.
minor comments (6)
  1. Table 1 totals 5,242 (99.58%) while the abstract and Table 2 report 5,264; reconcile the residual ~22 repositories and the percentage wording.
  2. Table 9: the 'readme' and 'readme + topics' blocks appear identical for all methods/domains; if topics add no signal, state that explicitly rather than duplicating rows.
  3. Section 6.2 vs Table 10: hybrid results use indus-sde-st-v0.2 while the main dense comparison emphasizes nasa-smd-ibm-st-v2; clarify model choice and keep naming consistent.
  4. Appendix F reports large intermediate extraction counts (e.g., 761k function-identifier samples) while the main benchmark is 117,950 snippets / 119,720 queries; a short filtering/sampling paragraph would help readers map appendix tables to the released splits.
  5. Ethical Considerations correctly note redaction of original README content under CC-BY-4.0 with regeneration scripts; ensure the HuggingFace cards and README state this prominently so users do not expect full text in the public dump.
  6. Minor polish: 'Curatedcorpus' / 'novelinformation' spacing in the abstract; standardize 'all-MiniLM-L6-v2' hyphenation; fix 'A vailable' in footnote 5.

Circularity Check

0 steps flagged

No circularity: dataset/benchmark paper with independent expert labels and extracted pairs; baselines do not reduce to inputs by construction.

full rationale

This is a corpus-and-benchmark paper, not a first-principles derivation. The central claims are (i) construction of a multi-source scientific repository corpus and two IR benchmarks, and (ii) empirical baseline variation across domains, query types, and languages. Ground truth for repository search comes from SME repository-to-query annotation (Section 4.1); code pairs come from tree-sitter extraction of docstrings/identifiers (Section 5). Metrics (MRR, Recall, NDCG) are computed against those held-out labels and do not equal any fitted training objective by construction. INDUS embeddings appear only as evaluated baselines among BM25, MiniLM, Qwen3, and SFR-Code; they are not used to justify the benchmark’s validity or uniqueness. Self-citations to prior INDUS work and collection pipelines are ordinary tooling references, not load-bearing uniqueness theorems or ansatz imports. No fitted parameter is renamed a prediction; no definition equates a claimed result to its input. Score 0 is the correct outcome.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

As an empirical IR/dataset paper, load-bearing choices are curation thresholds, taxonomy, annotation protocol, and LLM tooling rather than fitted physical constants. The central comparative claims rest on accepting NASA SMD divisions as the domain axis, expert familiarity-based multi-labels as relevance truth, and fixed preprocessing/LLM decisions as defining the released artifacts. No new physical entities are postulated.

free parameters (4)
  • README minimum length filter = 50 words
    Repositories with fewer than 50 words in README are removed; this hand-chosen cutoff defines the final 5,264-repo corpus and can exclude short but valid scientific tools.
  • Crawled-context relevancy threshold = 0.5
    External link content is kept only if LLM relevancy score exceeds 0.5 across six dimensions; threshold is a design choice that shapes the enriched text views used in evaluation.
  • Domain classifier model and prompt = gpt-4.1-mini few-shot
    GPT-4.1-mini with few-shot pydantic-ai structured output assigns SMD division; model/prompt choice is free relative to the claim that the corpus is correctly domain-classified.
  • Identifier filtering pipeline cutoffs = len>=5; drop top 1% frequency; stopword list
    Identifier datasets drop names <5 chars, top 1% frequency names, dictionary/stopword filters, then mask identifiers; these thresholds define the hard identifier-retrieval task.
axioms (4)
  • domain assumption NASA Science Mission Directorate five-division taxonomy is an appropriate and sufficient domain axis for scientific software retrieval evaluation.
    All corpus labels and many result breakdowns are defined on these five divisions (Section 3.3, Table 2); alternative taxonomies could regroup performance gaps.
  • domain assumption Subject-matter experts starting from known repositories and listing all familiar relevant tools produce valid multi-relevance ground truth for IR metrics.
    Section 4.1 explicitly uses repository-to-query annotation with average 2.3 GTs; metrics and domain comparisons depend on this labeling process.
  • standard math Standard IR metrics (MRR, Recall@k, NDCG@k) at fixed cutoffs are appropriate primary measures of scientific tool and snippet discovery quality.
    Sections 6.1–6.3 report all claims via these metrics in BEIR/MTEB-compatible form.
  • ad hoc to paper LLM cleaning of READMEs and relevancy filtering of crawled pages preserve scientific signal without systematically distorting retrieval comparisons across domains.
    Context enrichment and cleaned views are central to reported gains (Table 9); correctness of cleaning is assumed from process description (Appendix D), not human audit at scale.

pith-pipeline@v1.1.0-grok45 · 29445 in / 3364 out tokens · 33992 ms · 2026-07-12T01:26:28.399955+00:00 · methodology

0 comments
read the original abstract

Scientists increasingly rely on open-source tools to support their research workflows, yet discovering relevant software among over 600 million GitHub repositories remains challenging. Existing code search benchmarks focus on general software engineering tasks and fail to capture the domain-specific vocabulary and needs of scientific computing. We present a curated corpus of 5,264 high-quality, domain-classified scientific repositories spanning five NASA Science Mission Directorate divisions -- Earth Science, Astrophysics, Planetary Science, Heliophysics, and Biological & Physical Sciences -- enriched with cleaned READMEs, extracted topics, and additional context from crawled links. Building on this corpus, we introduce two novel information retrieval benchmarks: (1) a repository search benchmark with 219 expert-curated queries designed by domain scientists, and (2) a large-scale code snippet retrieval benchmark containing 117,950 code snippets and 119,720 queries across seven programming languages. Baseline evaluations on repository search reveal significant performance variation across scientific domains. Code snippet retrieval proves equally challenging, with substantial variation driven by differing documentation practices, coding standards, and programming language conventions across scientific communities. All datasets and benchmarks are publicly released on HuggingFace to support research on scientific tool discovery.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 4 canonical work pages

  1. [1]

    The astrophysics source code library.Astrophysics Source Code Library, 2012

    Alice Allen, Patrick Shanahan, et al. The astrophysics source code library.Astrophysics Source Code Library, 2012

    2012

  2. [2]

    The resource identification initiative: a cultural shift in publishing.Brain and behavior, 6(1):e00417, 2016

    Anita Bandrowski, Matthew Brush, Jeffrey S Grethe, Melissa A Haendel, David N Kennedy, Sean Hill, Patrick R Hof, Maryann E Martone, Maarten Pols, Serena Tan, et al. The resource identification initiative: a cultural shift in publishing.Brain and behavior, 6(1):e00417, 2016

    2016

  3. [3]

    Introducing the fair principles for research software.Scientific Data, 9(1):622, 2022

    Michelle Barker, Neil P Chue Hong, Daniel S Katz, Anna-Lena Lamprecht, Carlos Martinez- Ortiz, Fotis Psomopoulos, Jennifer Harrow, Leyla Jael Castro, Morane Gruenpeter, Paula Andrea Martinez, et al. Introducing the fair principles for research software.Scientific Data, 9(1):622, 2022

    2022

  4. [4]

    Scibert: A pretrained language model for scientific text

    Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3615–3620, 2019

    2019

  5. [5]

    Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain V

    Bishwaranjan Bhattacharjee, Aashka Trivedi, Masayasu Muraoka, Muthukumaran Ramasub- ramanian, Takuma Udagawa, Iksha Gurung, Nishan Pantha, Rong Zhang, Bharath Dandala, Rahul Ramachandran, Manil Maskey, Kaylin Bugbee, Michael M. Little, Elizabeth Fancher, Irina Gerasimov, Armin Mehrabian, Lauren Sanders, Sylvain V. Costes, Sergi Blanco-Cuaresma, Kelly Lock...

    2024

  6. [6]

    Repeatability in computer systems research

    Christian Collberg and Todd A Proebsting. Repeatability in computer systems research. Communications of the ACM, 59(3):62–69, 2016

    2016

  7. [7]

    Nine best practices for research software registries and repositories.PeerJ Computer Science, 8: e1023, 2022

    Daniel Garijo, Hervé Ménager, Lorraine Hwang, Ana Trisovic, Michael Hucka, Thomas Morrell, Alice Allen, Task Force on Best Practices for Software Registries, and SciCodes Consortium. Nine best practices for research software registries and repositories.PeerJ Computer Science, 8: e1023, 2022. doi: 10.7717/peerj-cs.1023

    work page doi:10.7717/peerj-cs.1023 2022

  8. [8]

    Bridging the gap: Enhancing prominence and provenance of nasa datasets in research publications.Data Science Journal, 23(1), 2024

    Ilya Gerasimov, Andrey Savtchenko, Jennifer Alfred, James Acker, Jianfu Wei, and Bijaya KC. Bridging the gap: Enhancing prominence and provenance of nasa datasets in research publications.Data Science Journal, 23(1), 2024. doi: 10.5334/dsj-2024-001

    work page doi:10.5334/dsj-2024-001 2024

  9. [9]

    Cosqa: 20,000+ web queries for code search and question answering

    Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. Cosqa: 20,000+ web queries for code search and question answering. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 5690–5700, 2021

    2021

  10. [10]

    Michael Hucka and Matthew J. Graham. Software search is not a science, even among scientists: A survey of how scientists and engineers find software.Journal of Systems and Software, 141: 171–191, 2018. doi: 10.1016/j.jss.2018.03.047

    work page doi:10.1016/j.jss.2018.03.047 2018

  11. [11]

    Codesearchnet challenge: Evaluating the state of semantic code search

    Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. Codesearchnet challenge: Evaluating the state of semantic code search. InarXiv preprint arXiv:1909.09436, 2019

    Pith/arXiv arXiv 1909

  12. [12]

    Tools and data services registry: a community effort to document bioinformatics resources.Nucleic acids research, 44(D1):D38–D47, 2016

    Jon Ison, Kristoffer Rapacki, Hervé Ménager, Matúš Kalaš, Emil Rydza, Piotr Chmura, Christian Anthon, Niall Beard, Karel Berber, Dan Bolser, et al. Tools and data services registry: a community effort to document bioinformatics resources.Nucleic acids research, 44(D1):D38–D47, 2016

    2016

  13. [13]

    Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Coir: A comprehensive benchmark for code information retrieval models

    Xiangyang Li et al. Coir: A comprehensive benchmark for code information retrieval models. arXiv preprint arXiv:2407.02883, 2024

    Pith/arXiv arXiv 2024

  15. [15]

    Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664, 2021

    Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. Codexglue: A machine learning benchmark dataset for code understanding and generation.arXiv preprint arXiv:2102.04664, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Mteb: Massive text embedding benchmark

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. Mteb: Massive text embedding benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014–2037, 2023

    2014

  17. [17]

    The vault: A comprehensive multilingual dataset for advancing code understanding and generation

    Dung Nguyen, Le Nam, Anh Dau, Anh Nguyen, Khanh Nghiem, Jin Guo, and Nghi Bui. The vault: A comprehensive multilingual dataset for advancing code understanding and generation. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 4763–4788, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023....

    work page doi:10.18653/v1/2023.f 2023

  18. [18]

    From open data to open science

    Rahul Ramachandran, Kaylin Bugbee, and Kevin Murphy. From open data to open science. Earth and Space Science, 8(5):e2020EA001562, 2021

    2021

  19. [19]

    Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code llama: Open foundation models for code.arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  20. [20]

    Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science.arXiv preprint arXiv:2211.09085, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Docling technical report

    Deep Search Team. Docling technical report. Technical report, 8 2024. URLhttps://arxiv. org/abs/2408.09869

    Pith/arXiv arXiv 2024

  22. [22]

    Not a NASA Division

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. 16 A Query Examples Table 12 provides additional example queries from each s...

    arXiv 2021

  23. [23]

    Authenticity: Write queries representing real information needs you or colleagues have encountered in your research

  24. [24]

    Answerability: Ensure the query can be answered by examining repository README files and descriptions

  25. [25]

    Specificity: Include domain-specific terminology where appropriate (e.g., instrument names, data formats, mission names)

  26. [26]

    What package does X?

    Completeness: Identify ALL relevant repositories from the corpus, not just the most obvious one. 10https://github.com/unclecode/crawl4ai 20 5.Diversity: Cover different query types: •Tool discovery (“What package does X?”) •Workflow questions (“How do I accomplish Y?”) •Data access (“How do I get data from Z?”) •Analysis methods (“What tools implement alg...

    2021

  27. [27]

    download WorldClim MAT/MAP tiles and extract data for site locations

  28. [28]

    combine 1 and 2 into a single dataframe

  29. [29]

    determine the OzFlux sites that fall within the ’dryland regions’ defined by United Nations Environment World Monitoring Centre

  30. [30]

    area": "Earth Science Division

    get boundaries for US, mexico, and Australia territories SPEI: code used to compute SPEI and extract meterological variables for flux tower locations. CRU data downloaded from: https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_4.04/cruts.2004151855.v4.04/ meterological variables downloaded included: precipitation (pre), 29 tmin (tmn), tmax (tmx), vapor pressu...

    Pith/arXiv arXiv doi:10.1038/s41597-020-0453-3 1991