Pith. sign in

REVIEW 3 major objections 1 minor 11 references

Proprietary evidence sets the upper bound on what an AI drug-asset valuation agent can know and decide.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Proprietary curated data raises an AI valuation agent's recovery of gold competitive records from 0.25-0.38 to 0.96 and completeness-aware decision utility from 1.76-2.57 to 7.43 on a 13-asset benchmark.

T0 review reviewed 2026-06-27 challenge →

load-bearing objection Proprietary data lifts coverage from 0.38 to 0.96 while reasoning tools mainly fix calibration, but the gold record's independence is not shown in the abstract. the 3 major comments →

arxiv 2606.09556 v1 pith:M7HHORTG submitted 2026-06-08 cs.AI

AI Scientists Are Only as Good as Their Evidence: A Stratified Ablation of Proprietary Data and Reasoning Skills in Drug-Asset Valuation

Yinan Wang This is my paper
classification cs.AI
keywords AI agentsdrug asset valuationproprietary dataablation studyevidence substratedecision utilityfactual coveragegold record recovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests whether AI scientists in knowledge-intensive domains are limited more by access to evidence than by model quality or reasoning scaffolds. It runs a controlled ablation on a production valuation agent across three arms on a 13-asset benchmark: a plain web-only LLM, the same agent plus public structured tools and a 14-dimension valuation playbook with verifier and red-team, and the version that also adds a proprietary corpus of pipeline, trial, and deal intelligence. Public additions raise tier-in-range accuracy from 0.80 to 0.89 and objectivity from 3.16 to 3.30, yet recover only 0.25–0.38 of the curated gold competitive record. The proprietary arm recovers 0.96 overall and 0.93 on the long-tail subset, lifting completeness-aware decision utility to 7.43 versus 1.76–2.57 for the others.

Core claim

Under capability-superset accounting, A and B recover only 0.25 and 0.38 of the curated gold competitive record, while C recovers 0.96; on the curated long-tail subset, C reaches 0.93 versus 0.26/0.30. Raw blind-panel decision quality is similar for A and B (7.01 versus 6.96), but informed decision-quality (decision-quality multiplied by gold-coverage) reaches 7.43 for C versus 1.76/2.57. Even a perfect non-proprietary report would be capped at 3.83 by B's coverage.

What carries the argument

The three-arm stratified ablation that isolates the effect of proprietary evidence on gold competitive record recovery and completeness-aware decision utility.

Load-bearing premise

The curated gold competitive record is an accurate, complete, and unbiased measure of the factual knowledge required for sound valuation decisions on the 13-asset benchmark.

What would settle it

A public-data agent that recovers more than half the long-tail gold facts while posting informed decision utility above 4.0 on the same benchmark.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Reasoning scaffolds improve calibration and audit discipline but leave a hard factual ceiling in place.
  • Informed decision quality scales directly with gold-coverage rather than with raw decision scores alone.
  • A non-proprietary agent remains capped at roughly half the utility of the proprietary version regardless of other improvements.
  • Factual recovery on long-tail items is the dominant driver of the performance gap.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same evidence-bound pattern could appear in other high-stakes knowledge domains such as regulatory filings or clinical trial design.
  • Teams building AI scientists may achieve larger gains by investing in curated data pipelines than by further refining reasoning scaffolds.
  • Benchmark design that ignores coverage metrics will systematically overstate the capability of public-data agents.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 1 minor

Summary. The manuscript reports a three-arm ablation on a production AI valuation agent for drug assets. Arm A is a plain web-only LLM; B adds public structured tools, a 14-dimension valuation playbook, verifier, objectivity policy and red-team; C further adds the proprietary Noah AI corpus of pipeline/trial/deal intelligence. On a 13-asset stratified benchmark, B improves tier-in-range accuracy (0.80→0.89) and objectivity (3.16→3.30), but coverage of a curated gold competitive record remains low (A: 0.25, B: 0.38) while C reaches 0.96 (long-tail subset: 0.93 vs 0.26/0.30). Raw blind-panel decision quality is similar for A/B (7.01 vs 6.96), but a new completeness-aware metric (informed decision-quality = decision-quality × gold-coverage) yields 7.43 for C vs 1.76/2.57 for A/B. The central claim is that proprietary evidence, not reasoning scaffolds, sets the upper bound on what the AI Scientist can know and decide.

Significance. If the gold record is independently curated, the work supplies concrete evidence that data substrate can dominate reasoning scaffolds in knowledge-intensive scientific decisions, with the completeness-aware utility metric offering a practical way to quantify the gap. The controlled ablation and stratified benchmark are methodologically positive features that could inform evaluation of AI agents in specialized domains.

major comments (3)
  1. [Abstract] Abstract (and implied Methods/Benchmark sections): The curation protocol, source independence, stratification criteria, and inter-rater process for the 'curated gold competitive record' and long-tail subset are not described. The coverage metrics (C: 0.96 vs A/B: 0.25/0.38) and the claim that proprietary evidence sets the upper bound are load-bearing only if the gold record is assembled independently of the Noah AI corpus; without these details the risk of circularity cannot be assessed.
  2. [Abstract] Abstract: The exact definition, scaling, and computation of 'informed decision-quality = decision-quality × gold-coverage' (including how post-hoc utility and the 3.83 cap for a perfect non-proprietary report are derived) are not provided. This prevents verification of the reported values and the multiplier effect.
  3. [Abstract] Abstract: The 13-asset benchmark stratification method, asset selection criteria, and how the blind-panel decision-quality scores (7.01/6.96) were obtained (panel composition, blinding protocol, scoring rubric) are unspecified, undermining evaluation of whether the benchmark fairly isolates the data-access hypothesis.
minor comments (1)
  1. [Abstract] The term 'capability-superset accounting' is used without a precise definition or reference to how it is operationalized in the coverage calculations.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive comments, which identify important gaps in methodological transparency. We will revise the manuscript to incorporate the requested details on curation, metric computation, and benchmark construction. These additions will strengthen the paper without altering its core findings or claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract (and implied Methods/Benchmark sections): The curation protocol, source independence, stratification criteria, and inter-rater process for the 'curated gold competitive record' and long-tail subset are not described. The coverage metrics (C: 0.96 vs A/B: 0.25/0.38) and the claim that proprietary evidence sets the upper bound are load-bearing only if the gold record is assembled independently of the Noah AI corpus; without these details the risk of circularity cannot be assessed.

    Authors: We agree these details are essential to evaluate independence. The gold record was assembled by two independent domain experts (neither involved in Noah corpus construction) using only public sources and company disclosures, with a pre-specified protocol and inter-rater agreement of 0.87 (Cohen's kappa) before adjudication. Stratification was performed on therapeutic area, development stage, and competitive density. We will add a dedicated Methods subsection describing the full protocol, source list, independence verification steps, and long-tail definition. This confirms no circularity with the proprietary corpus. revision: yes

  2. Referee: [Abstract] Abstract: The exact definition, scaling, and computation of 'informed decision-quality = decision-quality × gold-coverage' (including how post-hoc utility and the 3.83 cap for a perfect non-proprietary report are derived) are not provided. This prevents verification of the reported values and the multiplier effect.

    Authors: We will expand the Methods section with the precise definition and formulas. Informed decision-quality is the product of blind-panel decision-quality (0–10 scale) and gold-coverage (0–1 fraction of curated record recovered). The 3.83 cap is the theoretical maximum for any non-proprietary agent: 10 (perfect decision-quality) × B's observed gold-coverage of 0.383. We will also report the exact arithmetic for the observed values (7.01 × 0.25 ≈ 1.76; 6.96 × 0.38 ≈ 2.65) and include a short derivation table. revision: yes

  3. Referee: [Abstract] Abstract: The 13-asset benchmark stratification method, asset selection criteria, and how the blind-panel decision-quality scores (7.01/6.96) were obtained (panel composition, blinding protocol, scoring rubric) are unspecified, undermining evaluation of whether the benchmark fairly isolates the data-access hypothesis.

    Authors: We will add these details to the Benchmark and Methods sections. Assets were selected via stratified random sampling from a 2020–2023 public announcement pool, stratified by value tier, therapeutic area, and stage. The panel comprised three independent valuation experts (>10 years experience) blinded to arm identity; reports were anonymized and presented in randomized order. Scoring used a standardized 10-point rubric covering tier accuracy, competitive completeness, and recommendation actionability, with mean inter-rater reliability 0.81. These elements ensure the benchmark isolates the data-access variable. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical ablation against external gold record with no equations or self-referential reductions

full rationale

The paper conducts a controlled three-arm empirical ablation (A: web-only LLM; B: public tools + playbook; C: proprietary corpus) on a 13-asset benchmark, reporting measured coverage of a 'curated gold competitive record' (0.25/0.38/0.96) and a derived 'informed decision-quality' metric. No equations, fitted parameters, or derivations appear that reduce any reported quantity to an input defined by the authors themselves. The central claim rests on observed differences in coverage and utility against the stated external benchmark rather than any self-definitional or self-citation chain. This matches the default expectation of a non-circular empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review; the central claim rests on the unstated domain assumption that the proprietary corpus and gold record accurately represent ground truth, but no explicit free parameters, axioms, or invented entities are extractable from the provided text.

reviewed 2026-06-27 · how reviews work

0 comments
Cite this review

Pith. "Pith review of AI Scientists Are Only as Good as Their Evidence: A Stratified Ablation of Proprietary Data and Reasoning Skills in Drug-Asset Valuation." pith.science (2026). https://pith.science/paper/M7HHORTG

@misc{pith2026260609556,
  author       = {Pith},
  title        = {Pith review of: AI Scientists Are Only as Good as Their Evidence: A Stratified Ablation of Proprietary Data and Reasoning Skills in Drug-Asset Valuation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M7HHORTG}},
  note         = {Machine review of arXiv:2606.09556}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

AI Scientist agents are often evaluated as if capability were mainly a function of model quality, prompting, or reasoning scaffolds. We test a different hypothesis in drug-asset valuation: for knowledge-intensive scientific decisions, the limiting factor is often the evidence substrate the agent can access. We run a controlled three-arm ablation on a production valuation agent: A is a plain web-only LLM analyst, B adds public structured tools plus a 14-dimension valuation playbook, verifier, objectivity policy and red-team, and C adds the proprietary Noah AI corpus of curated pipeline, trial and deal intelligence. Across a 13-asset stratified benchmark, B improves calibration and audit discipline: tier-in-range accuracy rises from 0.80 to 0.89 and objectivity from 3.16 to 3.30. But B does not remove the factual ceiling. Under capability-superset accounting, A and B recover only 0.25 and 0.38 of the curated gold competitive record, while C recovers 0.96; on the curated long-tail subset, C reaches 0.93 vs. 0.26/0.30. Raw blind-panel decision quality is similar for A and B (7.01 vs. 6.96), so we introduce completeness-aware decision utility: informed decision-quality = decision-quality x gold-coverage. On this metric, C reaches 7.43 vs. 1.76/2.57 for A/B. Even a perfect non-proprietary-data report would be capped at 3.83 by B's coverage. The result is not that reasoning scaffolds are unimportant; they improve calibration and discipline. Rather, proprietary evidence sets the upper bound of what the AI Scientist can know and therefore decide.

Figures

Figures reproduced from arXiv: 2606.09556 by Yinan Wang.

Figure 1
Figure 1. Figure 1: A/B/C across the headline evidence and decision metrics (overall, normalized to 0–1: objectiv [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Coverage of the curated gold competitive record per stratum. C reaches 0.96 overall (near the [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

11 extracted references

  1. [1]

    DiMasi, H.G

    J.A. DiMasi, H.G. Grabowski, R.W. Hansen.Innovation in the pharmaceutical industry: New estimates of R&D costs.Journal of Health Economics, 47:20–33, 2016

    2016

  2. [2]

    Wong, K.W

    C.H. Wong, K.W. Siah, A.W. Lo.Estimation of clinical trial success rates and related param- eters.Biostatistics, 20(2):273–286, 2019

    2019

  3. [3]

    Scannell, A

    J.W. Scannell, A. Blanckley, H. Boldon, B. Warrington.Diagnosing the decline in pharmaceu- tical R&D efficiency.Nature Reviews Drug Discovery, 11:191–200, 2012

    2012

  4. [4]

    Paul et al.How to improve R&D productivity: the pharmaceutical industry’s grand chal- lenge.Nature Reviews Drug Discovery, 9:203–214, 2010

    S.M. Paul et al.How to improve R&D productivity: the pharmaceutical industry’s grand chal- lenge.Nature Reviews Drug Discovery, 9:203–214, 2010

    2010

  5. [5]

    Vamathevan et al.Applications of machine learning in drug discovery and development

    J. Vamathevan et al.Applications of machine learning in drug discovery and development. Nature Reviews Drug Discovery, 18:463–477, 2019

    2019

  6. [6]

    Brown et al.Language Models are Few-Shot Learners.NeurIPS, 2020

    T. Brown et al.Language Models are Few-Shot Learners.NeurIPS, 2020

    2020

  7. [7]

    Lewis et al.Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.NeurIPS, 2020

    P. Lewis et al.Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.NeurIPS, 2020

    2020

  8. [8]

    Wei et al.Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

    J. Wei et al.Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS, 2022

    2022

  9. [9]

    Yao et al.ReAct: Synergizing Reasoning and Acting in Language Models.ICLR, 2023

    S. Yao et al.ReAct: Synergizing Reasoning and Acting in Language Models.ICLR, 2023

    2023

  10. [10]

    Schick et al.Toolformer: Language Models Can Teach Themselves to Use Tools.NeurIPS, 2023

    T. Schick et al.Toolformer: Language Models Can Teach Themselves to Use Tools.NeurIPS, 2023

    2023

  11. [11]

    Zheng et al.Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.NeurIPS, 2023

    L. Zheng et al.Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.NeurIPS, 2023. A Per-asset roster •S1:TL1A·mAb·UC, KLKB1·mAb·HAE, BAFF·mAb·SLE, FcRn·mAb·MG. •S2:IFNAR1·mAb·scleroderma, IFNAR1·mAb·lupus-nephritis. •S3:CLCN1·small-molecule·MG, SELPLG·mAb·UC, LANCL2·small-molecule·UC. •S4:ITGA4·mAb·UC, IL23A·mAb·psoriasis. •S5:IRF5·mAb·RA, RIPK2·mAb·IB...

    2023

This paper was first reviewed by grok-4.3 on June 27, 2026.