{"id":"cd2ab81c-9efd-40e4-ba79-8d92392d4119","arxiv_id":"2607.06128","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM-based multi-agent system (FORMA) verifies DESI spectral classifications with 95.5% agreement to expert adjudication by reconstructing expert reasoning into an auditable workflow.","lead":"The paper introduces FORMA, a multi-agent LLM workflow that verifies astronomical spectroscopic classifications by extracting spectral evidence, testing physical hypotheses, and auditing consistency. It matters because it offers a scalable, auditable alternative to human visual inspection for large sky surveys like DESI.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Headline 95.5% agreement comes from a single full-pipeline run; only Step 5 was stress-tested for stability (100 runs on 2 cases), leaving the reproducibility of the 331 definite predictions and their accuracy unverified across the LLM-based Steps 2–4.","rationale":"The paper presents a well-motivated and clearly described protocol with promising initial results. The counterfactual ablation tests (Fig. 3a, Table B1) provide genuine evidence that the Result Auditor responds to physical evidence rather than rubber-stamping hypotheses. The distinction between feature counting and credibility-based triage (Appendix A) is thoughtful and well-argued. However, the central quantitative claim rests on a single full-pipeline run, and the only stability test covers one step on two cases. For a system whose entire value proposition is reliability and reproducibility of verification, this is a significant gap. The reader correctly identified the credibility scoring as a weak point, but the concern is broader: the full pipeline's LLM-based Steps 2–4 are equally untested for stability, and their outputs feed directly into the Result Auditor's decisions. The verdict remains CONDITIONAL because the conceptual framework is sound and the preliminary evidence is encouraging, but the headline numbers need to be shown stable across multiple full-pipeline runs before the system can be considered a reliable verification layer. This is a concrete, addressable gap — not a fundamental flaw in the approach.","tokens_in":32524,"tokens_out":4050,"duration_ms":317214,"concrete_test":"Re-run the complete FORMA pipeline (all 6 steps, not just Step 5) on the full 1,149-spectrum evaluation set at least 5 times with independent LLM sampling. For each run, record: (1) the set of spectra receiving medium-or-higher credibility, (2) the total count of definite predictions, (3) the binary agreement rate with expert VI classes, and (4) the per-class agreement rates. If the definite-prediction count varies by more than ~10% across runs, or if the agreement rate shifts by more than ~2 percentage points, the headline 95.5% figure is not a stable property of the system but an artifact of a single sampling trajectory. Additionally, compute the Jaccard similarity of the medium-or-higher credibility sets across runs; if it falls below ~0.85, the triage boundary itself is unstable.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central quantitative claim — 331 definite predictions with 95.5% binary agreement — derives from a single execution of the full 6-step FORMA pipeline on 1,149 spectra. The only reproducibility evidence provided is 100 independent re-runs of the Result Auditor (Step 5) on 2 counterfactual spectra (Table B1). However, Steps 2–4 (Single-Hypothesis Evaluation, Feature Auditor, Hypothesis Synthesis) are also LLM-based and involve open-ended reasoning: each Step 2 agent assigns line statuses (LIKELY/MARGINAL/NOTFOUND/MASKED) and selects anchor lines; the Step 3 Feature Auditor makes KEEP/REMOVE/FLAG judgments on contested features; the Step 4 synthesis agent selects the best hypothesis and assigns an intermediate confidence level. These upstream LLM judgments directly determine the line catalogues and hypothesis reports that the Result Auditor receives. If any of Steps 2–4 exhibit run-to-run variability — which is expected from LLM sampling — the set of spectra receiving medium-or-higher credibility could shift, changing both the 331 definite predictions and the 95.5% agreement figure. The paper does not report the stability of these intermediate steps or of the full pipeline. The reader's concern about the Result Auditor's credibility calibration is a subset of this broader issue: even if Step 5 is stable in isolation (as the 100-run test suggests), instability in Steps 2–4 would propagate through the feature catalogues and hypothesis reports, undermining the reproducibility of the headline result. This matters because the paper's claim is not just that FORMA can verify spectra, but that it does so with a specific, quantified level of reliability.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The manuscript introduces FORMA (Formalized Observational Reasoning with Auditable Decisions), a multi-agent LLM-based protocol that formalizes expert spectroscopic verification as an executable workflow. FORMA decomposes visual inspection into six steps: deterministic feature extraction (CWT-based), parallel single-hypothesis evaluation by LLM agents, adversarial feature auditing, hypothesis synthesis, an independent Result Auditor that assigns a credibility rating, and structured report generation. Applied to 1,149 DESI EDR visual-inspection spectra in a zero-shot setting, FORMA produces 331 definite predictions at medium-or-higher credibility with 95.5% binary agreement with expert-adjudicated classes. The credibility score is shown to correlate with redshift residual concentration and to be not merely a proxy for SNR. Counterfactual evidence-ablation tests (100 runs on 2 cases) demonstrate that the Result Auditor rejects physically inconsistent interpretations. The paper also provides a thoughtful critique (Appendix A) of the DESI VI QUALITY flag's limitations in the low-SNR ELG regime.","tokens_in":33417,"tokens_out":1265,"duration_ms":200879,"significance":"The paper addresses a genuine and timely problem: scaling scientific verification for next-generation surveys. Its main conceptual contribution — framing verification as an executable, auditable protocol distinct from prediction or post-hoc interpretability — is well-motivated. Strengths include: (1) a clear architectural separation between deterministic numerical computation and LLM reasoning, which is a sound design principle; (2) the counterfactual ablation methodology (Appendix B) providing falsifiable evidence that the Result Auditor responds to physical evidence rather than pattern-matching; (3) the critique of VI QUALITY in Appendix A, which identifies a real limitation of the DESI inspection protocol; and (4) public code availability. The central claim that expert verification can be partially automated is supported by the DESI evaluation. However, the reproducibility of the headline quantitative result across the full LLM-based pipeline is not established, which is the primary concern detailed below.","major_comments":[{"comment":"The headline quantitative result — 331 definite predictions with 95.5% binary agreement — derives from a single execution of the full 6-step FORMA pipeline on 1,149 spectra. The only reproducibility evidence is 100 independent re-runs of the Result Auditor (Step 5) on 2 counterfactual spectra (Table B1, Appendix B). However, Steps 2–4 (Single-Hypothesis Evaluation, Feature Auditor, Hypothesis Synthesis) are also LLM-based and involve open-ended reasoning: Step 2 agents assign line statuses (LIKELY/MARGINAL/NOTFOUND/MASKED) and select anchor lines; Step 3 makes KEEP/REMOVE/FLAG judgments; Step 4 selects the best hypothesis and assigns intermediate confidence. These upstream judgments directly determine the line catalogues and hypothesis reports that the Result Auditor receives. If Steps 2–4 exhibit run-to-run variability — which is expected from LLM sampling — the set of spectra receiving","section":null}],"minor_comments":[{"comment":"The paper uses 'DeepSeek-V4-Pro [20]' as the base LLM but reference [20] is titled 'DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence' with no 'Pro' designation. Please reconcile the model name and cite it accurately.","section":null},{"comment":"Fig. 2a: the x-axis label is unclear. The text refers to 'low-or-higher' and 'medium-or-higher' thresholds, but the figure axis should explicitly state what quantity is being thresholded (e.g., credibility tier rank).","section":null},{"comment":"Section 'Dataset Composition': the stratification ratio is stated as 4:4:2 (high/medium/low) but it is unclear whether this refers to SNR tiers or VI QUALITY tiers. Please clarify.","section":null},{"comment":"The paper states that Redrock returns 'top-N fits' (Step 1 description) but does not specify the value of N used in the evaluation. This should be stated for reproducibility.","section":null},{"comment":"Appendix A provides a detailed and valuable critique of VI QUALITY limitations, but this analysis is confined to the appendix. Given that it motivates the entire credibility-score design, a brief pointer in the main text (Section 2) would help readers understand why a new scoring mechanism is needed.","section":null},{"comment":"The paper mentions 'a recent systematic evaluation of astronomical VLMs [21]' in the Discussion. Reference [21] (arXiv:2604.24589) appears to be a 2026 preprint; please verify the reference details.","section":null},{"comment":"Fig. 3b: the AGN example shows line labels with redshift-shifted wavelengths (e.g., [Ne V] 4242.4 Å) but the rest-frame wavelength of [Ne V] is 3426 Å. At z=0.238, the observed wavelength would be ~4240 Å, which is consistent, but the label format could be clearer about whether values are observed or rest-frame.","section":null},{"comment":"The term 'binary agreement' (Section 2) is used for a four-class problem (QSO/LRG/ELG/BGS). It would be clearer to state that this is overall accuracy across all classes, or clarify if a binary QSO-vs-galaxy split was used in the computation.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about full-pipeline reproducibility is, in my assessment, the single most important issue. The paper is conceptually strong and the counterfactual methodology is creative, but the central quantitative claim rests on a single run of a stochastic pipeline. If the authors can show that the 331/95.5% figure is stable across even a modest number of full-pipeline re-runs (e.g., 5–10), this would substantially strengthen the paper and likely warrant upgrade to minor revision. I also note that the paper sits at the boundary of astro-ph.CO and a methods/AI venue; the conceptual framing is broad enough for either, but the DESI application grounds it well for this journal."},"author_rebuttal":{"model":"glm-5.2","summary":"The referee raises a legitimate and important concern about the reproducibility of the headline quantitative result across the full LLM-based pipeline. We acknowledge this gap and will address it with additional full-pipeline re-runs in the revised manuscript.","responses":[{"response":"The referee is correct. The headline 95.5% agreement figure derives from a single full-pipeline execution, and the reproducibility evidence in Appendix B covers only Step 5 (Result Auditor) on two counterfactual spectra. Steps 2–4 are LLM-based and involve open-ended reasoning — line status assignment, anchor line selection, KEEP/REMOVE/FLAG judgments, and hypothesis synthesis — so run-to-run variability in these upstream steps could, in principle, change which spectra receive medium-or-higher credibility and thus alter the agreement rate. We acknowledge this as a genuine gap in the current manuscript. We note that several design choices are intended to reduce upstream variability: Step 1 (Visual Interpreter) is fully deterministic (CWT feature detection, continuum fitting, Redrock hypothesis generation); all wavelength-to-redshift conversions and numerical computations are handled by deterministic tools rather than LLM arithmetic; and the structured prompt protocol constrains each Step 2 agent to evaluate a single hypothesis against pre-computed evidence rather than conducting open-ended exploration. However, these mitigations do not substitute for empirical evidence of stability across repeated runs. We will address this by running the complete FORMA pipeline (Steps 1–6) independently N≥3 times on the full 1,149-spectrum evaluation set and reporting: (i) the variance in the number of definite predictions at the medium-or-higher credibility threshold, (ii) the variance in the binary agreement rate, and (iii) the spectrum-level stability of credibility assignments (the fraction of spectra that change credibility tier across runs). We will add these results as a new supplementary table and discuss them in the main text. If the variance is non-negligible, we will report a","revision_made":"no","referee_comment":"The headline quantitative result — 331 definite predictions with 95.5% binary agreement — derives from a single execution of the full 6-step FORMA pipeline on 1,149 spectra. The only reproducibility evidence is 100 independent re-runs of the Result Auditor (Step 5) on 2 counterfactual spectra (Table B1, Appendix B). However, Steps 2–4 (Single-Hypothesis Evaluation, Feature Auditor, Hypothesis Synthesis) are also LLM-based and involve open-ended reasoning... If Steps 2–4 exhibit run-to-run variability — which is expected from LLM sampling — the set of spectra receiving definite predictions and the resulting agreement rate could change across runs."}],"tokens_in":32194,"tokens_out":949,"duration_ms":53631,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The core contribution here is genuinely new: FORMA reframes spectroscopic verification not as classification but as an executable protocol that extracts evidence, generates physically constrained hypotheses, runs adversarial audits, and gates decisions on credibility. The architecture is well-motivated — deterministic numerical tools are cleanly separated from LLM reasoning, and the counterfactual ablation tests (100 runs on 2 cases, 2 KB conditions) show the Result Auditor rejects physically inconsistent interpretations robustly. The Appendix A critique of VI QUALITY is sharp and well-argued. Code is public on GitHub. This is real infrastructure work, not just another classifier paper.","headline":"FORMA formalizes expert spectroscopic verification as an executable, auditable protocol — promising concept, but headline numbers rest on a single pipeline run with limited stability evidence.","tokens_in":33362,"tokens_out":202,"would_cite":false,"duration_ms":80042,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":["95.75.Mn","95.85.Kr"],"model":"glm-5.2","headline":"Executable verification protocol matches expert astronomers 95.5%","keywords":[],"falsifier":"Run FORMA on a large independent sample of expert-adjudicated spectra where the ground truth is known with high confidence, and check whether the credibility score reliably separates correct from incorrect classifications. If the score fails to stratify accuracy beyond what signal-to-noise alone predicts, or if counterfactual evidence ablations fail to trigger downgrades in a substantial fraction of runs, the claim that the protocol tracks evidential sufficiency rather than data quality would be undermined.","tokens_in":32784,"feed_emoji":"🔭","tokens_out":965,"duration_ms":188304,"temperature":0.7,"pith_summary":"This paper introduces FORMA (Formalized Observational Reasoning with Auditable Decisions), a computational protocol that reconstructs the reasoning steps a human expert uses when verifying astronomical spectra — extracting evidence, generating physically constrained hypotheses, testing alternatives, and checking consistency — into an executable, auditable workflow. The central claim is that scientific verification, not just prediction, can be formalized as a machine-executable process. Applied to the DESI spectroscopic survey's expert-review catalogue, FORMA operates without dataset-specific training: it takes candidate redshifts from a template-fitting engine as hypotheses to be tested, not as accepted answers, and assigns a credibility score based on whether the physical evidence suffices to support the proposed interpretation. At a medium-or-higher credibility threshold, FORMA produces 331 definite predictions with 95.5% binary agreement with expert-adjudicated classifications. The credibility score correlates with redshift accuracy but is not a proxy for raw signal-to-noise ratio — low credibility reflects insufficient or ambiguous evidence as judged by the verification pipeline, not simply poor data quality. Counterfactual tests show that when diagnostic spectral evidence is removed or made physically inconsistent, FORMA downgrades or rejects the interpretation rather than rubber-stamping a template match.","feed_headline":"Executable verification protocol matches expert astronomers 95.5%","feed_subtitle":"FORMA turns expert spectral review into an auditable computational workflow, triaging which classifications are safe before they enter a科学al","key_machinery":"The FORMA protocol decomposes expert spectroscopic verification into four modules and six steps. The Visual Interpreter (Step 1) converts raw spectral data into computational features using wavelet-based detection and Chebyshev continuum fitting, and generates redshift hypotheses via a template-fitting engine (Redrock). The Hypothesis Analyst (Steps 2 and 4) assigns each candidate redshift to an independent LLM agent that evaluates predicted spectral lines against observed features using deterministic tools for Gaussian fitting and wavelength-to-redshift conversion, then synthesises a verdict across all hypotheses. The Analysis Auditor (Steps 3 and 5) performs two-stage adversarial review: a","core_discovery":"The paper demonstrates that the expert reasoning process underlying astronomical spectral verification — hypothesis generation under physical constraints, systematic testing of alternatives, and adversarial consistency checking — can be decomposed into discrete, executable computational steps whose outputs are auditable. The FORMA credibility score, assigned by an adversarial review agent without a hand-crafted scoring specification, tracks evidential sufficiency rather than data quality: it separates cases where the physical evidence genuinely supports a classification from cases where a template fit merely looks plausible. This is shown through three lines of evidence: (1) 95.5% agreement ","pith_inferences":[],"forward_implications":["If the verification-protocol approach generalises, next-generation surveys (Rubin, SKAO, Euclid, 4MOST, DESI II) could deploy automated verification layers that triage which candidate classifications are safe for scientific use before human review, reducing the expert bottleneck at scale.","The credibility-score mechanism — which gates acceptance on evidential sufficiency rather than model confidence — could become a standard output of survey pipelines, replacing or complementing binary quality flags that conflate data quality with classification reliability.","The counterfactual evidence-ablation methodology provides a template for testing whether any automated reasoning system bases its decisions on physically meaningful evidence rather than pattern matching, applicable beyond astronomy.","Accumulated verification records with full evidential provenance could serve as versioned knowledge repositories that preserve operational expertise across long-lived survey collaborations, transferring judgment that is otherwise tacit."],"fun_headline_variants":["FORMA encodes astronomer reasoning into auditable verification steps","331 spectral calls verified via executable expert reasoning at 95.5% agreement","FORMA credibility score separates evidence-backed spectra from plausible fits","Executable verification of DESI spectra matches expert review 95.5%","Adversarial audit agent scores spectral evidence without hand-crafted rules"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The credibility rating (HIGH/MEDIUM/LOW) is assigned entirely by an LLM agent using a knowledge base and spectral inspection, without a hand-crafted scoring specification. If the agent's internal calibration of what constitutes sufficient evidence is unstable or sensitive to prompt format, the credibility-gated triage mechanism could route cases incorrectly.","fun_headline_variants_meta":{"raw":{"variants":["FORMA encodes astronomer reasoning into auditable verification steps","331 spectral calls verified via executable expert reasoning at 95.5% agreement","FORMA credibility score separates evidence-backed spectra from plausible fits","Executable verification of DESI spectra matches expert review 95.5%","Adversarial audit agent scores spectral evidence without hand-crafted rules"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":647,"prompt_tokens":572,"completion_tokens":75,"prompt_tokens_details":null},"tokens_in":572,"tokens_out":75,"duration_ms":65224,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-08T15:43:51.332806+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"Run FORMA on a large independent sample of expert-adjudicated spectra where the ground truth is known with high confidence, and check whether the credibility score reliably separates correct from incorrect classifications. If the score fails to stratify accuracy beyond what signal-to-noise alone predicts, or if counterfactual evidence ablations fail to trigger downgrades in a substantial fraction of runs, the claim that the protocol tracks evidential sufficiency rather than data quality would be undermined.","supporting_citations":[],"review_version":1}