Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents

T0 review · 4 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that structuring an LLM's research-question formation through a fixed certificate of definitions, assumptions, mechanism, falsifiable hypothesis, decisive test, and failure rule makes the resulting questions substantially

desk verdict A sensible, honestly hedged framework for structuring LLM research questions, but the certificate-ablation evidence doesn't yet separate question quality from format cues. read the letter →

arxiv 2607.05682 v2 pith:6WNOMAHD submitted 2026-07-06 cs.AI

Yufeng Wang This is my paper · ORCID
classification cs.AI
keywords scientificdiscoveryagentslargelanguagemodelsresearchquestionformationauditabilityfalsifiabilitystructuredcertificatesLLMevaluationderivationconstraints
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that the first research question an LLM scientific-discovery agent proposes can be made inspectable by requiring the model to produce a structured Research Question Certificate alongside the question. The certificate records the definitions, assumptions, mechanism, tension, falsifiable hypothesis, decisive test, and failure-update rule that the question depends on. If correct, this would give scientists a concrete artifact to audit before spending resources on experiments, and it would mean that the act of forcing a derivation chain improves the judged quality of the question. In the paper's evaluation, certificate-bound questions score roughly 4.9/5 under two independent LLM judges, while removing the certificate drops below 1/5.

What carries the argument

The Research Question Certificate is a fixed seven-section template the LLM must fill in to emit a research question. It carries the argument by turning an opaque generation step into a visible derivation chain: each section is meant to be consistent with the previous ones, and any gap becomes inspectable by a reader. The ablation showing that scores collapse without the certificate indicates that this structured template, not the surrounding prompting, is the load-bearing component.

What would settle it

Have a panel of human domain experts rate the same 40 question packages with and without certificate formatting, stripped to plain prose, under a validated rubric for research-question quality; if human rankings do not separate certificate-derived from prompt-derived questions when formatting is controlled, the central claim fails. A more direct test would regress judge scores on the presence of each certificate section to see whether mere section labels predict score independent of content.

Watch

Extended reading notes

Core claim

The central claim is that an LLM asked to form a research question produces far more auditable and higher-judged questions when the generation is constrained to pass through a fixed Research Question Certificate — seven enumerated sections covering primitive definitions, assumptions, a mechanism model, a tension, a falsifiable hypothesis, a minimal decisive test, and a failure-update rule — than when the same question is generated from ordinary prompts. In the reported evaluation, certificate-bound generation scores 4.86/5 under a primary judge and 4.88/5 under a second judge in a one-repeat ablation, while removing the certificate collapses scores below 1/5 under both. The paper takes this

Load-bearing premise

The evaluation rests on the assumption that the two LLM judges' scores reflect the genuine scientific merit and audibility of the research questions, not just the visible presence of a structured certificate; the score collapse without the certificate makes that assumption the point of vulnerability.

Editorial extensions

If this is right

  • Auditing shifts earlier: a scientist can check assumptions, mechanism, and decisive test before running any experiment, rather than after seeing a plausible-sounding question.
  • The failure-update rule gives each rejected question an explicit revision path, turning question formation into an iterative loop rather than a one-shot generation.
  • Because two independent LLM judges preserve the ranking, the quality advantage is not tied to one judge's preferences.
  • The ablation result indicates that the certificate, not the prompt wording, is what produces the improvement.
  • The framework offers a concrete checklist for what 'auditable' means in AI-generated research questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The score collapse when the certificate is removed (below 1/5) could mean the judges are rewarding the visible structure rather than the underlying question; a human expert panel rating the same questions stripped of certificate formatting would tell whether the effect is real content quality or format detection.
  • The certificate's seven slots may constitute a minimal argument skeleton, so the same template could be repurposed for auditing other scientific outputs — experiment proposals, literature summaries, or generated theorems — wherever assumptions and falsifiers matter.
  • The ordering of sections (definitions before assumptions before mechanism) may itself be a reasoning chain; ablating individual sections or permuting them would reveal whether each slot contributes or only the full structure does.
  • The 'minimal decisive test' slot is where a question stands or falls: a certificate can pass all other fields yet fail if the decisive test cannot distinguish the proposed mechanism from a plausible alternative, making that slot the highest-value target for a fast screen.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces FirstResearch, a framework for LLM scientific-discovery agents that generates a structured Research Question Certificate containing primitive definitions, assumptions, a mechanism model, a tension/contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule. The authors claim that this explicit derivation structure improves the auditability and judged quality of LLM-proposed research questions. They report evaluations on ten LLM-agent research topics, comparing FirstResearch against prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2, using DeepSeek and Gemini judge models. They report system-level ranking preservation across judges, a large performance gap when the certificate is present versus absent in a one-repeat ablation, and they explicitly frame the results as preliminary due to the use of LLM judges rather than human experts. Code, prompts, outputs, and reproduction scripts are provided.

Significance. If the central claim is sound, the paper offers a lightweight, auditable mechanism for improving the inspectability of LLM-generated research questions, which is a real bottleneck in scientific-discovery agents. The authors deserve credit for releasing code, prompts, saved outputs, and reproduction scripts, and for including an independent judge model as a robustness check. However, the evidence base is thin: ten topics, a single-repeat ablation, and LLM judges with no human validation. The central construct — auditability — is a property for human scientists, and the validation relies on LLM judges whose scores may track certificate presence rather than question quality. The core evaluation-validity issue is load-bearing, and the paper's own limitation statement acknowledges the lack of human expert evaluation.

major comments (4)
  1. [Abstract — evaluation protocol] The central claim is about auditability, but the only outcome measure is LLM-judge scores. The abstract's own final sentence concedes that the evaluation uses LLM judges rather than human domain experts. Auditability is a property for human scientists; without any human audit or a validated proxy, the reported numbers (e.g., 4.90/5 vs <1/5) cannot be interpreted as evidence about auditability. Please add a human audit study, or at minimum validate the LLM-judge rubric against human judgments on a held-out set before treating the scores as a measure of the construct.
  2. [Abstract — ablation and judge blindness] The comparison is between FirstResearch, which always emits a certificate, and prompt-level baselines that do not. The judge therefore sees a structural signature of the intervention. The certificate-removal ablation dropping below 1/5 is exactly the pattern expected if the judge rewards template headings such as 'assumptions' and 'falsifiable hypothesis' regardless of content. This is a confound between artifact presence and judged quality. Please blind the judge to certificate presence (e.g., strip or shuffle the certificate headings, or judge plain-text questions) and include a sham-certificate control using the same template with vacuous content.
  3. [Abstract — circularity of rubric] If the judge's scoring criteria are the seven certificate components, then removing the certificate removes the checklist the judge is told to score. The abstract does not state the rubric, but the near-collapse of scores under certificate removal is consistent with a mechanical dependence. Specify the exact judge prompt and rubric, and show that scores are not simply a count of which headings are present. Without this, the reported effect sizes cannot be attributed to improved question quality or auditability.
  4. [Abstract — one-repeat ablation] The ablation is described as a 'one-repeat' checkpoint. With a single repetition there is no variance estimate, and the large gaps between conditions (e.g., 4.90/5 vs below 1/5) could be dominated by run-to-run noise. Report multiple seeds or repeats, confidence intervals, and topic-level breakdowns. This is load-bearing because the claim that the certificate is the strongest component rests entirely on this ablation.
minor comments (4)
  1. [Abstract — 'DeepSeek-blind-judge'] The phrase 'DeepSeek-blind-judge protocol' is undefined. What is the judge blind to? Clarify the procedure (e.g., blind to system identity, blind to the certificate, blind to topic?).
  2. [Abstract — Pearson agreement] The sentence 'Pearson agreement of 0.865 on average score' is ambiguous. Specify the two variables correlated (e.g., DeepSeek vs Gemini average scores over the 40 packages?) and the units.
  3. [Abstract — 'certificate-only'] The 'certificate-only' condition is unclear. Does it mean FirstResearch with all other components removed, or a standalone certificate without the framework's question formation process? Define the condition explicitly.
  4. [Abstract — reproducibility] The GitHub link is appreciated. Consider also archiving a versioned snapshot (e.g., Zenodo DOI) to ensure long-term reproducibility and to allow citation of a specific version.

Circularity Check

0 steps flagged · score 0.0 of 10

No demonstrated circularity: the abstract reveals a possible LLM-judge validity concern, but no definitional identity or fitted-input prediction is shown.

full rationale

The abstract describes FirstResearch as producing a Research Question Certificate with fixed components (definitions, assumptions, mechanism, tension, falsifiable hypothesis, decisive test, failure update rule) and reports that certificate-only outputs score ~4.9/5 while removing certificates drops below 1/5. For this to be circular under the stated rules, the judge's scoring rubric would have to be definitionally identical to the certificate components, so that removing the certificate mechanically removes the criteria being scored. The abstract does not disclose the judge rubric or the exact scoring instructions, so I cannot exhibit that specific reduction. The observed ablation is equally consistent with the paper's intended mechanism: judges find structured, transparent questions more auditable. The paper itself flags the key limitation: 'These results are preliminary and use LLM judges rather than human domain experts.' That is a construct-validity and correctness risk for the auditability claim, not a demonstrated circular step. There are no fitted parameters, no load-bearing self-citations, no imported uniqueness theorems, and no equations in the abstract that reduce to the paper's inputs. Under the requirement to quote a specific reduction before flagging circularity, this abstract-only review finds no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 1 invented entities

No numerical free parameters are reported in the abstract; the design choices (certificate fields, judge rubrics) are structural rather than numerical. The axioms listed are the load-bearing assumptions behind the evaluation, and the certificate itself is an invented artifact without independent evidence.

assumptions (3)
  • domain assumption LLM judges provide valid, unbiased measures of research-question quality.
    The entire evaluation rests on LLM judges; the abstract explicitly acknowledges this limitation, and judge biases could be confounded with the certificate format.
  • domain assumption The 10 topics and 40 baseline outputs are representative of scientific discovery tasks.
    Generalization of the claim depends on this sampling; no justification of topic diversity is given in the abstract.
  • ad hoc to paper The certificate's component list (definitions, assumptions, mechanism, tension, hypothesis, test, update rule) is sufficient and necessary for auditability.
    This is the paper's design assumption, tested only via a coarse ablation that removes the whole certificate rather than varying the components.
invented entities (1)
  • Research Question Certificate
    purpose: Structured output artifact intended to make LLM-proposed research questions inspectable before execution
    It has no external falsifiable handle; its usefulness is only evidenced by the paper's own LLM-judge evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents." pith.science (2026). https://pith.science/paper/6WNOMAHD

@misc{pith2026260705682,
  author       = {Pith},
  title        = {Pith review of: FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6WNOMAHD}},
  note         = {Machine review of arXiv:2607.05682}
}
read the original abstract

LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at https://github.com/louiswang524/FirstResearch.

Figures

Figures reproduced from arXiv: 2607.05682 by the authors.

Figure 1
Figure 1. FirstResearch pipeline and ablation boundaries for auditable scientific ideation. CertificateOnly is an internal ablation that keeps the derivation and certificate core but removes downstream review and meta￾update layers; NoCertificate bypasses the certificate/gate path. observation would reject it, and why the experiment isolates the relevant tension. FirstResearch operationalizes this view through the Research Qu… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Style Wins, Substance Loses: A Diagnosis of LLM-as-Judge in Idea Generation

    cs.CL 2026-08 conditional novelty 6.0 of 10

    LLM judges of scientific ideas are measurably swayed by writing style; a style-detecting module reduces but does not remove the bias.

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.