{"id":"02a2f01c-f8c5-4de5-a352-68f387aa04d5","arxiv_id":"2611.13118","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Language model agents extract 44,332 substantive discussions of 13,616 galaxies from 156,006 arXiv astro-ph papers, creating the GalaxyWiki database.","lead":"This paper uses language model agents to scan 156,006 arXiv astro-ph papers and extract every passage where a galaxy is discussed in more than two sentences. The result is GalaxyWiki, a searchable database of 44,332 discussions covering 13,616 galaxies, each with quotes and links to the source papers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Counts do not reconcile: 43,546 located + 674 unresolved = 44,220, not the stated 44,332; the 112-row Abell invalidation is unaccounted in the headline totals.","rationale":"The reader's weakest assumption is recall; I read that as an acknowledged limitation and as scoped by the paper's own language about discussions in the extracted literature. The central per-record accuracy claim is well supported by the two random manual-review passes and by the quote-verification statistics, so I do not object to that claim. The unacknowledged soft spot is the arithmetic of the aggregate counts: the 112 invalidated Abell rows are missing from the total and unresolved reconciliation. These counts appear in the abstract and results, so they need to be settled before the paper is published as-is. Conditional acceptance preserves the scientific contribution while requiring the count reconciliation and a check of whether the 13,616 grouping changes.","tokens_in":10545,"tokens_out":18999,"duration_ms":151945,"concrete_test":"Re-run the final production script with the 112 invalidated Abell-pattern rows explicitly flagged, and recompute: total mention records, resolved and unresolved counts, coordinate-coverage percentage, and the 1-arcsecond neighbor-of-neighbor grouping. Then compare the released dataset's record count with the abstract's 44,332. If the final total is 44,220, update all headline counts and re-derive the 13,616 entries; if the final total is 44,332, set unresolved to 786 and check whether 98.2% remains correct. Report which case matches the public data release.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 2's coordinate accounting is internally inconsistent after the Abell-pattern invalidation described in Limitations. The extraction step reports 44,332 mentions. Programmatic NED resolution supplies 23,770 successes; agent steps supply 19,888 more, for 43,658 located by arithmetic. The text then says coordinates are available for 43,546 of 44,332 mentions (98.2%), exactly 112 fewer, and the residual is given as 674 unresolved. But 43,546 + 674 = 44,220, not 44,332. The missing 112 matches the rows invalidated as 'Galaxy A1'-type false positives. If those rows were deleted from the final dataset, the total mention count, coordinate percentage, and the 13,616 grouped entries all need recomputation. If the rows were retained without coordinates, the unresolved count should be 786. Since the paper's headline results are these aggregate counts, the published numbers cannot be taken at face value until this is reconciled. This is a reporting and data-accounting issue, distinct from the acknowledged recall limitation, and does not by itself impeach the per-record accuracy evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents GalaxyWiki, a database and web resource built by using GPT-5.5 agents in the Codex harness to identify galaxies discussed in more than two sentences across 156,006 arXiv astro-ph papers, extract quotations and summaries, and resolve sky coordinates through a combination of programmatic NED queries, agent-based NED queries, and internet searches. The authors report 44,332 extracted galaxy-discussion records, 85,314 quotes, 98.2% coordinate coverage, and 13,616 grouped galaxy entries. They validate the pipeline by comparing extracted quotes against source text (96.5% match after normalization) and by manually reviewing 120 records, finding correct galaxy identification and substantive discussion in 118 cases. The paper also describes GalaxyWiki's user features and discusses limitations including incomplete corpus coverage, guaranteed omission of galaxies due to the 30-galaxy-per-paper cap, and a known false-positive pattern in the Abell identifier resolution.","tokens_in":10910,"tokens_out":3797,"duration_ms":32702,"significance":"If the reported numbers are correct, GalaxyWiki would be a valuable new resource for astronomy, filling a genuine gap: existing tools link objects to papers but do not distinguish extended discussion from catalog appearances. The paper's main strengths are its concrete, reproducible pipeline; the public release of code, data, and the wiki; and the use of multiple verification steps, including character-level quote matching and manual review. The manual review evidence is credible and supports the central claim that the pipeline can identify individual galaxies and extract substantive discussion in the large majority of cases. The paper also appropriately acknowledges several limitations, such as the incomplete MINT-1T corpus, the 30-galaxy cap, and the possibility of missed mentions. The principal weakness is an unresolved internal inconsistency in the coordinate accounting that directly affects the headline totals, and a smaller but related inconsistency in the quote-matching categories. These issues do not undermine the per-record accuracy evidence, but they must be fixed before the aggregate claims can be taken at face value.","major_comments":[{"comment":"The coordinate accounting is internally inconsistent. The text states that programmatic NED resolution succeeded for 23,770 mentions and that agent steps found coordinates for 19,888 more, which sums to 43,658 located mentions. It then says 674 records remain unresolved, and immediately afterward that coordinates are available for 43,546 of 44,332 mentions (98.2%). The numbers do not reconcile: 43,546 + 674 = 44,220, not 44,332. The 112-row difference matches exactly the number of rows invalidated by the Abell-pattern false positives described in the Limitations. The authors must clarify whether those 112 rows were removed from the final dataset or retained without coordinates. If they were removed, then the total mention count, the 98.2% coordinate fraction, the unresolved count, and the 13,616 grouped entries all need recomputation. If they were retained without coordinates, the unresolved count should be 786 and the coordinate fraction should be recalculated accordingly. This is a load-bearing reporting issue because the abstract and results sections lead with these aggregate numbers.","section":"§2, 'Coordinate resolution for grouping'; Limitations","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The quote-matching categories do not sum consistently. The paper reports 66,362 character-for-character matches, 15,971 matches after normalization, 34 further matches after case/letter filtering, and 2,826 quotes that can be split into fragments matching after normalization. The sum of these four categories is 85,193, which is 121 fewer than the total of 85,314 quotes. The categories may be overlapping (for example, a quote could both match after normalization and be splittable into matching fragments), but the text does not say so. Please clarify whether the categories are mutually exclusive or nested, and provide the exact count of quotes that fall outside all categories.","section":"§3, 'Results'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper's operational definition of 'discussed in more than two sentences' is not stated precisely enough to be auditable. The extraction prompt asks the model for one to three quotes totaling three to eight sentences, and the manual review criterion is 'at least three distinct substantive sentences,' but the paper does not specify how the two-sentence threshold is enforced or whether a record is counted as a discussion by construction or by reviewer judgment. Since the abstract and title make this threshold a defining feature of the dataset, please state the exact instruction used and how compliance was checked beyond the manual sample.","section":"§2, 'Galaxy mention extraction'; §3, 'Results'","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The first sentence of the MINT-1T description cites Awadalla et al. (2024) twice; please consolidate into one citation.","section":"§1, first paragraph","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The count of 3,379 English Wikipedia articles using the galaxy infobox has no retrieval date or query details; please add the access date and the exact query or method used.","section":"§1, Introduction","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement that the model was instructed to extract at most 30 galaxies per paper is followed later by the observation that one paper yielded 250 galaxies despite this instruction. Please report how many papers exceeded the limit and how those cases were handled, since this bears on the completeness discussion.","section":"§2, 'Galaxy mention extraction'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The manual review sample of 120 records is small relative to 44,332 records. The authors should state whether the sample was drawn with stratification across agent runs, coordinate-resolution sources, or paper subfields, and provide a simple binomial confidence interval for the 118/120 success rate to help readers calibrate the uncertainty.","section":"§3, 'Results'","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Abell-pattern false positive is mentioned only in the Limitations and is not cross-referenced in the coordinate-resolution section where the pattern is implicitly used. Please describe the pattern, its invalidation, and its effect on the reported totals at the point where the totals are presented.","section":"Limitations","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nThe stress-test note is right, and it should be read before anything else. The Section 2 arithmetic does not close. Programmatic NED resolution gives 23,770 located mentions; the agent step gives 19,888 more; that totals 43,658. The paper then reports coordinates for 43,546 of 44,332 mentions (98.2%) -- exactly 112 fewer. The Limitations section explains the cause: an Abell-designation pattern matched local identifiers like \"Galaxy A1\", and 112 of 156 rows resolved with that pattern were invalidated. But 43,546 plus the 674 unresolved leaves 44,220, not 44,332. The invalidated rows were dropped from the numerator but not the denominator. Either the dataset total should be 44,220, with the coordinate percentage and the 13,616 galaxy grouping recomputed, or the unresolved count should be 786. Since the headline deliverable of this paper is precisely these aggregate numbers, this needs to be reconciled before the counts are taken at face value. It does not impeach the per-record accuracy evidence, which stands independently, but it is a genuine accounting bug, not a nitpick.\n\nWhat is actually new and good: the paper builds a real resource. It extracts extended discussions of individual galaxies, not just identifiers or properties like the prior work it cites (Shao et al., Wang et al., Li et al.), and it ships the code, the dataset, and a searchable wiki. The quote-fidelity check is unusually candid -- 96.5% exact or normalized match, with the messy remainder reported openly: agents combining non-contiguous excerpts, MINT-1T parsing corrupting LaTeX symbols, and truncated identifiers that can split one galaxy across entries. The manual review of 120 records is small relative to 44,332, and the authors do not oversell it; the key limitations (the 30-galaxy cap, incomplete corpus coverage, the 1-inch grouping choice) are stated plainly rather than buried.\n\nMinor caveats: the pipeline runs on proprietary models (GPT-5.5, Gemini), so the extraction itself is not reproducible even though the released dataset is; and the two \"already in use\" claims rest on arXiv preprints, one from the same group. Both are minor. The per-record evidence, the public code, and the candid limitations section show clear thinking and honest engagement.\n\nWho gets value: astronomers checking whether a specific galaxy has been discussed, and anyone building LLM-based literature-mining tools. It deserves a serious referee. The count reconciliation is a required revision, not a rejection.","headline":"GalaxyWiki is a genuinely useful dataset with unusually honest verification, but the headline counts do not reconcile: 43,546 located + 674 unresolved = 44,220, not 44,332, and the 112-row Abell-pattern invalidation is the missing piece.","tokens_in":11361,"tokens_out":4618,"duration_ms":36940},"pith_extraction":{"msc":[],"pacs":["95.80.+p","98.62.-g","95.75.-z"],"model":"deepseek/deepseek-v4-flash","headline":"Language model agents extract published discussions of 13,616 galaxies","keywords":["galaxy literature mining","language model agents","natural language processing for astronomy","GalaxyWiki","scientific knowledge extraction","arXiv astro-ph","entity linking"],"falsifier":"A manual review of a larger random sample of papers (e.g., 500 papers) comparing the agents' extracted galaxy discussions against a human reading of the same papers would reveal the true recall and precision of the extraction pipeline. If the false-negative rate is found to be high (e.g., >50%), the claim that the pipeline achieves its primary goal of identifying discussed galaxies would be weakened.","tokens_in":12063,"feed_emoji":"🌌","tokens_out":1501,"duration_ms":24112,"temperature":0.7,"pith_summary":"The paper uses language model agents to scan 156,006 arXiv astro-ph papers and identify every galaxy discussed in more than two sentences, extracting 44,332 discussions with 85,314 quotes from 18,641 papers. By resolving sky coordinates for 98.2% of those mentions, the authors group them into 13,616 distinct galaxies and assemble the results into GalaxyWiki, a searchable database where astronomers can immediately see what has been said about a galaxy and follow links to the source papers. A manual review of 120 records found that 118 correctly identified a galaxy and extracted substantive discussion. The paper argues that this pipeline makes a previously impractical project feasible and provides a complement to existing object databases that only link papers to objects without distinguishing genuine discussion from catalog appearances.","feed_headline":"Language model agents extract published discussions of 13,616 galaxies","feed_subtitle":"A pipeline scans 156,000 arXiv papers, extracts 44,332 discussions with quotes, and groups them into a searchable wiki.","key_machinery":"The pipeline rests on two agent passes of the GPT-5.5 language model in the Codex agent harness. In the first pass, agents read each paper, identify galaxies mentioned in more than two sentences, extract one to three quotes totaling three to eight sentences, generate a 20-to-40-word summary, assign up to five UAT topics, and extract sky coordinates when explicit. In the second pass, agents resolve coordinates for remaining mentions by searching the paper, querying NED, and using internet access as a fallback. The coordinate resolution is the central mechanism that enables grouping mentions into distinct galaxies.","core_discovery":"The paper demonstrates that a multi-stage pipeline combining a language model agent (GPT-5.5 in the Codex harness) for galaxy mention extraction, programmatic NED queries for coordinate resolution, and a second agent pass for resolving remaining coordinates can identify and extract substantive discussions of individual galaxies from a large corpus of astronomy papers. The key numbers are: 44,332 discussions extracted from 18,641 papers, grouped into 13,616 galaxies, with 98.2% of mentions having sky coordinates and 96.5% of quotes matching source text after normalization. Manual review of 120 records confirmed 118 correct identifications with substantive discussion. The authors claim this creates a unique dataset that captures scientific interpretations of galaxies, not just object-to-paper links.","pith_inferences":["The paper implicitly demonstrates that language model agents can perform a task that would be prohibitively expensive with human readers: scanning 156,000 papers and extracting structured, quote-backed records for 13,616 galaxies. This suggests that similar literature-mining pipelines could be built for other types of astronomical objects or for other scientific fields with arXiv-like preprint cor","The coordinate resolution step, which uses a combination of programmatic queries and agent-driven web searches, reveals a general strategy for linking textual mentions to canonical identifiers when the text itself does not provide coordinates. This hybrid approach could be applied to other entity-linking problems in scientific literature.","The paper's manual review of only 120 out of 44,332 records (0.27%) means that the true false-positive rate and the rate of missing galaxies are not well constrained. A larger-scale evaluation would be needed to understand the pipeline's limitations for use cases that require high recall, such as systematic literature reviews.","The fact that 69.2% of grouped entries have only a single mention suggests that the dataset is heavily biased toward galaxies that are discussed in only one paper, while a small number of well-studied galaxies have many discussions. This distribution could be used to study how scientific attention is distributed across galaxies."],"forward_implications":["Astronomers can immediately check whether a galaxy has been discussed in the extracted literature, read what was said, and find the source papers, without needing to inspect each linked paper individually.","The GalaxyWiki dataset can guide similarity and semantic searches for galaxies with related features, supporting anomaly detection and identification of galaxies of scientific interest.","The methodology can be extended beyond galaxies to stars, supernovae, exoplanets, and even theories of dark matter, and could support similar databases in other scientific domains like bioRxiv for genes or ChemRxiv for materials.","The database enables tracking which galaxies are discussed and cited most often, and how discussion patterns vary across subfields and years, providing a new tool for literature surveys in astronomy."],"supporting_citations":[{"why":"Describes the NASA/IPAC Extragalactic Database (NED) which the paper positions as the existing standard for galaxy literature links, motivating the need for a complementary resource that captures genuine discussion.","marker":"Mazzarella, 2020"},{"why":"Describes SIMBAD, another standard database for object identifiers and bibliographic references, which the paper uses as a comparison point and as a source for coordinate resolution.","marker":"Wenger et al., 2000"},{"why":"Provides the MINT-1T dataset from which the paper draws its corpus of 156,006 astro-ph papers; the corpus's incomplete coverage is a stated limitation.","marker":"Awadalla et al., 2024"},{"why":"Introduces PaperQA2 and WikiCrow, the language-agent approach for scientific literature synthesis that the paper adapts for galaxy discussion extraction.","marker":"Skarlinski et al., 2024"},{"why":"Documents the NED API that the paper uses for programmatic coordinate resolution of galaxy identifiers.","marker":"Mazzarella and NED Team, 2017"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The paper assumes that the language model agents correctly identify all galaxies discussed in more than two sentences in the papers, but the authors acknowledge that the agents likely miss some galaxies and that the limit of 30 galaxies per paper guarantees some omissions.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":8.7e-05,"raw_usage":{"cost":8.6814e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":8.6814e-05,"upstream_inference_prompt_cost":8.645e-05,"upstream_inference_completions_cost":3.64e-07},"total_tokens":952,"prompt_tokens":950,"completion_tokens":2,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":950,"tokens_out":2,"cost_source":"openrouter_usage","duration_ms":1264,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-11T17:52:21.798319+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"A manual review of a larger random sample of papers (e.g., 500 papers) comparing the agents' extracted galaxy discussions against a human reading of the same papers would reveal the true recall and precision of the extraction pipeline. If the false-negative rate is found to be high (e.g., >50%), the claim that the pipeline achieves its primary goal of identifying discussed galaxies would be weakened.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}