REVIEW 3 major objections 5 minor 35 references
Evaluating Agentic Harness Systems for Autonomous Computational Pathology
T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5
Pith's one-line read Current agentic systems can start pathology analysis workflows and write fluent reports, but almost never complete them end-to-end with traceable evidence and safe clinical claim boundaries.
desk verdict Solid multi-lane audit of agentic pathology workflows: 369 trajectories show partial capability and rare end-to-end completion, with scoring weights as the main caveat the authors already flag. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
ACP-Bench: a three-lane trajectory audit that separately scores automation ability (planning–action–diagnosis–reflection), recovered whole-slide-image-to-label diagnostic results, and clinical-workflow alignment via CWAS-3, R6 safety-boundary review, and pathologist report validation.
What would settle it
Run the same 41 tasks under independent full re-adjudication or prospective multi-reader review and test whether systems can raise formal end-to-end pass rates well above 10/369 while also preserving endpoint, evidence sufficiency, and claim-boundary language on pathologist review; if high workflow scores still leave unsafe or unsupported reports, or if independent scores collapse, the central capability claim fails.
Extended reading notes
Core claim
Evaluated agentic harness systems show emerging autonomous computational pathology capability but not reliable end-to-end autonomy: the overall expert-adjudicated workflow score was 61.53 percent, only 10 of 369 trajectories met the 75 percent formal pass threshold, and planning and diagnostic reporting outpaced tool execution, result binding, and reflection, while diagnostic accuracy and clinical-boundary alignment tracked different trajectory properties.
Load-bearing premise
The paper’s rule-based scores, pass threshold, and component weights faithfully measure clinically meaningful workflow autonomy rather than only compliance with this benchmark’s process checklist.
Editorial extensions
If this is right
- Claims of clinical autonomy for pathology agents must be deferred until systems preserve tools, outputs, metrics, and claim boundaries across multi-step runs.
- Agent comparisons in pathology should report separate workflow, prediction, and clinical-boundary profiles rather than a single leaderboard score.
- Near-term useful systems are auditable research-workflow assistants that keep intermediate files and bounded reports for expert review.
- Design priority shifts to stable result discovery, output verification, measurable repair loops, and hard claim-boundary enforcement.
- Benchmarks for medical agents should track where the evidence funnel narrows from plans and logs to normalized labels and probability metrics.
Reading between the lines
- The same execution-and-binding gap is likely to appear in other high-stakes agent settings where a fluent final note can hide missing intermediate artifacts.
- If result discovery and verification remain the bottlenecks, progress may come more from harness interfaces and memory than from larger base models alone.
- Risk-enriched pathologist review of report language could become a required companion layer for any agent benchmark that outputs clinical text.
- A useful next experiment is controlled file, tool, and endpoint perturbations to test whether systems recover with comparable rerun evidence rather than more narrative.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces ACP-Bench, a multi-lane benchmark for autonomous computational pathology (ACP), defined as converting high-level pathology goals into executable, traceable, and clinically bounded workflows. It evaluates 41 tasks (24 biomarker, 7 morphology, 10 prognosis) across 6 body systems and 9 endpoint families, using 9 models in 3 harness families (Claude Code, Codex, Open Code) for 369 complete trajectories. Each trajectory is scored on automation ability (expert-adjudicated planning/action/diagnosis/reflection; overall 61.53%, only 10/369 formal passes at 75%), diagnostic-result performance on a 139-trajectory/1,743-row normalized subset, and clinical-boundary alignment via CWAS-3 (mean 59.5%), R6 safety review (mean 47.8%), and pathologist validation of 90 risk-enriched reports. The central finding is carefully scoped: systems show emerging ACP capability, with planning and diagnostic reporting stronger than tool-bound execution, result discovery, and reflection, while the three evidence lanes measure non-substitutable properties and formal end-to-end completion remains rare.
Significance. If the descriptive audit holds, ACP-Bench is a timely and useful contribution for computational pathology and medical agent evaluation. It correctly shifts attention from representation quality or fluent reports to evidence-preserving workflow conversion, and it documents a concrete capability gap (goal interpretation ~83% vs result discovery ~39%; only 10 formal passes) with large denominators, threshold sensitivity, qualitative failure taxonomy, and pathologist claim-boundary review. Strengths include explicit multi-lane denominators, leakage controls that prevent report-only scoring, public release plans for data/code, and conservative language that avoids equating workflow audit with clinical autonomy. The work can serve as a reusable audit standard before stronger autonomy claims are made.
major comments (3)
- Methods, Expert-adjudicated workflow assessment and Independent reliability review: the headline formal-pass claim (10/369 at 75%) and stage/subcapability profiles rest substantially on 65,981 semantic checkpoint judgments. Independent reliability on the pre-specified 27-trajectory sample is only moderate (all-three exact agreement 59.7%; Krippendorff interval alpha 0.658; pairwise weighted kappa 0.560–0.672), with residual ambiguity concentrated in action, diagnosis, and reflection. For a load-bearing process-audit claim, the manuscript should either (i) expand independent re-adjudication beyond 27 trajectories for the most contested checkpoints, or (ii) more clearly demote semantic-stage conclusions relative to deterministic checks and report sensitivity of stage scores under alternative status mappings.
- Methods, Clinical workflow alignment and safety-boundary review (CWAS-3 and R6 equations) and Long-horizon tier construction (Lraw formula): CWAS-3 weights (0.35/0.35/0.30), R6 weights (0.25/0.30/0.20/0.15/0.10), long-horizon coefficients, and the 75% pass threshold are design-chosen and explicitly unvalidated externally. The paper already treats them as rule-based indices, but Results still present CWAS-3/R6 means, correlations, and long-horizon associations as primary clinical-boundary and complexity evidence. Please add weight/threshold sensitivity analyses (or an unweighted/equal-weight baseline) and keep any clinical-boundary ranking claims strictly relative to the stated audit definitions rather than as validated clinical scales.
- Results, Pathologist clinical validation (Table 1) and Limitations: the 90-report pathologist review is risk-enriched, single-reader, and not a prevalence sample, yet it is used to support the clinical-workflow lane’s claim that evidence insufficiency and overclaim are common. This is informative for mechanism, but the manuscript should more sharply separate reviewed-set counts from any implication about rate across all 369 trajectories, and state that multi-reader agreement was not measured for this layer. Without that tightening, the clinical-boundary conclusions risk over-generalization beyond the selected packets.
minor comments (5)
- Figures 1–6 pack many scores and denominators; several panels would benefit from explicit n captions and consistent mapping of short baseline codes (Hk/Sn/Op etc.) to full model names in every panel legend.
- Clarify early that diagnostic-result metrics (139 trajectories) and probability metrics (120) are narrower subsets, so automation, prediction, and clinical-boundary percentages are never compared as if they share a common denominator.
- Subcapability codes (P1, A1, D1, R1…) are introduced late; a compact table mapping codes to stage definitions would help readers of Figures 3–4.
- The arXiv date stamp and some references appear future-dated relative to typical publication timelines; verify bibliographic metadata for consistency before journal submission.
- Data/code availability promises progressive GitHub release; for reproducibility of the 369-trajectory audit, specify which score tables, checkpoint ledgers, and normalization scripts will be frozen with the camera-ready version.
Circularity Check
Empirical benchmark audit, not a derivation: scores are measurements under author-defined rubrics, not predictions forced by construction.
-
self definitional
[Methods: Expert-adjudicated workflow assessment; CWAS-3 and R6 formulas]
"Formal pass status was assigned when S(s)≥0.75. This threshold was chosen as a conservative workflow-completion rule... CWAS3(s)=0.35Us+0.35Ns+0.30Es... R6(s)=0.25Ds+0.30Os+0.20Cs+0.15Ts+0.10Rs... The long-horizon score is a rule-based index of task-design complexity; its component weights require external validation."
The headline numerical results (61.53% workflow score; 10/369 formal passes; CWAS-3 mean 59.5%; R6 mean 47.8%; long-horizon tier associations) are produced by author-chosen weights and cutpoints applied to the same trajectories. This is mild self-reference of measurement design, not a derivation that forces a scientific prediction: the paper reports complete-set descriptive audit scores under those rules and does not claim the weights are externally validated or that the pass count is independent of the threshold definition. Sensitivity across nearby thresholds and multi-lane denominators prevent the central qualitative claim from reducing fully to the definitions.
full rationale
ACP-Bench is a multi-lane empirical evaluation of 369 agent trajectories, not a first-principles derivation. The central claim (partial ACP capability; 61.53% workflow score; 10/369 formal passes; planning/diagnosis stronger than tool-bound execution/result discovery/reflection; non-substitutable automation, diagnostic-result, and clinical-boundary axes) is a descriptive complete-set audit under stated denominators and rules. Design-chosen elements—the 75% pass threshold, CWAS-3 weights (0.35/0.35/0.30), R6 weights, and the long-horizon Lraw formula—shape the numerical scores by construction, which is normal for a new benchmark and is openly framed as rule-based indices needing external validation. That is mild measurement self-reference, not circular derivation: the paper does not fit parameters then re-label them as predictions, import uniqueness from author-only theorems, or redefine the target in terms of the result. Threshold sensitivity (35/10/1 passes at 70/75/80%) and separate evidence lanes keep the qualitative finding from collapsing into a tautology. No load-bearing self-citation chain or fitted-input-as-prediction step is present. Score 1 reflects only the inherent self-reference of author-defined audit machinery, not a forced result.
Assumptions & free parameters
free parameters (5)
- formal workflow pass threshold
- CWAS-3 component weights
- R6 component weights
- long-horizon raw score coefficients
- checkpoint item weights and status mapping
assumptions (5)
- domain assumption A complete instrumented trajectory with planning/action/diagnosis/reflection artifacts is the right unit for judging ACP capability.
- domain assumption Retrospective WSI-to-label prediction supports computational prediction claims but not prospective pathologist adjudication or clinical diagnostic authority.
- ad hoc to paper Expert-adjudicated semantic checkpoints plus deterministic file/log checks can score workflow fidelity despite moderate inter-rater agreement.
- ad hoc to paper Design-chosen CWAS-3/R6/long-horizon indices are informative clinical-boundary and complexity measures even without external weight validation.
- domain assumption Harness-level behavior across Claude Code, Codex, and Open Code with nine models is representative enough to support a conservative capability snapshot.
invented entities (5)
-
ACP (Autonomous Computational Pathology)
-
ACP-Bench
-
CWAS-3 clinical workflow alignment score
-
R6 safety-boundary review index
-
Relative long-horizon tiers
Cite this review
Pith. "Pith review of Evaluating Agentic Harness Systems for Autonomous Computational Pathology." pith.science (2026). https://pith.science/paper/6UDJXG52
@misc{pith2026260702598,
author = {Pith},
title = {Pith review of: Evaluating Agentic Harness Systems for Autonomous Computational Pathology},
year = {2026},
howpublished = {\url{https://pith.science/paper/6UDJXG52}},
note = {Machine review of arXiv:2607.02598}
}
read the original abstract
Autonomous computational pathology (ACP) converts high-level pathology analysis goals into executable, traceable and clinically bounded workflows. Realizing this capability requires adapting general agentic harness systems to pathology-specific tasks, tools, evidence standards and clinical claim boundaries. We contribute ACP-Bench, a framework that adapts existing harness systems from computational pathology support toward ACP workflow capability. ACP-Bench evaluates 41 pathology workflow tasks, including 24 biomarker, 7 morphology and 10 prognosis tasks spanning 6 body-system groups and 9 endpoint families. The benchmark evaluates 9 models and 3 harness groups (Claude Code, Codex and Open Code), yielding 369 complete trajectories. ACP-Bench evaluates each trajectory across workflow execution, diagnostic performance and clinical-boundary alignment, combining expert-adjudicated process audits, diagnostic assessment and pathologist-validated safety review. Across evaluated systems, workflow initiation, task interpretation and diagnostic reporting were more mature than tool-bound execution, result binding and reflective workflow revision, and formal end-to-end completion remained rare. ACP-Bench provides a reusable standard for auditing whether agentic systems can operationalize pathology workflows before claims of reliable clinical autonomy.
Reference graph
Works this paper leans on
-
[1]
Nature616, 259–265 (2023)
Moor, M.et al.Foundation models for generalist medical artificial intelligence. Nature616, 259–265 (2023)
2023
-
[2]
Topol, E. J. High-performance medicine: the convergence of human and artificial intelligence.Nature Medicine25(2019)
2019
-
[3]
M.et al.Multimodal generative ai for medical image interpretation
Rao, V. M.et al.Multimodal generative ai for medical image interpretation. Nature639, 888–896 (2025)
2025
-
[4]
Sellergren, A.et al.Medgemma technical report.arXiv preprint arXiv:2507.05201 (2025)
arXiv 2025
-
[5]
Litjens, G.et al.A survey of deep learning in medical image analysis.Medical Image Analysis42, 60–88 (2017)
2017
-
[6]
Huang, Z.et al.A pathologist-ai collaboration framework for enhancing diag- nostic accuracies and efficiencies.Nature Biomedical Engineering9, 455–470 (2025)
2025
-
[7]
Faust, K.et al.Pharaoh: A collaborative crowdsourcing platform for phenotyping and regional analysis of histology.Nature Communications16, 742 (2025). 35
2025
-
[8]
Neidlinger, P.et al.Benchmarking foundation models as feature extractors for weakly supervised computational pathology.Nature biomedical engineering1–11 (2025)
2025
Show all 35 references
-
[9]
Zhao, Y.et al.Foundation models for computational pathology: advances and opportunities.Nature Reviews Bioengineering(2025)
2025
-
[10]
Truhn, D.et al.Large language models should be used as scientific reasoning engines, not knowledge databases.Nature Medicine(2023)
2023
-
[11]
Nature Methods1–4 (2026)
Zheng, Y.et al.Lazyslide: accessible and interoperable whole-slide image analysis. Nature Methods1–4 (2026)
2026
-
[12]
& Sun, J
Wang, Z., Danek, B., Yang, Z., Chen, Z. & Sun, J. Making large language mod- els reliable data science programming copilots for biomedical research.Nature Biomedical Engineering1–15 (2026)
2026
-
[13]
Bu, D.et al.Empowering ai data scientists using a multi-agent llm frame- work with self-evolving capabilities for autonomous, tool-aware biomedical data analyses.Nature Biomedical Engineering1–16 (2026)
2026
-
[14]
arXiv preprint arXiv:2407.02483(2024)
Li, B.et al.Mmedagent: Learning to use medical tools with multi-modal agent. arXiv preprint arXiv:2407.02483(2024)
2024 arXiv
-
[15]
Wang, H.et al.Spatialagent: An autonomous ai agent for spatial biology.bioRxiv (2025)
2025
-
[16]
Ferber, D.et al.Development and validation of an autonomous artificial intelli- gence agent for clinical decision-making in oncology.Nature cancer6, 1337–1349 (2025)
2025
-
[17]
Li, S.et al.A co-evolving agentic ai system for medical imaging analysis.arXiv preprint arXiv:2509.20279(2025)
2025
-
[18]
Huang, K.et al.Biomni: A general-purpose biomedical ai agent.bioRxiv(2025)
2025
-
[19]
J.et al.Nova: An agentic framework for automated histopathology analysis and discovery.arXiv preprint arXiv:2511.11324(2025)
Vaidya, A. J.et al.Nova: An agentic framework for automated histopathology analysis and discovery.arXiv preprint arXiv:2511.11324(2025)
2025
-
[20]
Lu, C.et al.Towards end-to-end automation of ai research.Nature651, 914–919 (2026)
2026
-
[21]
Zhang, L.et al.Molclaw: An autonomous agent with hierarchical skills for drug molecule evaluation, screening, and optimization.bioRxiv2026–04 (2026)
2026
-
[22]
L., Pak, J
Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E. & Zou, J. The virtual lab of ai agents designs new sars-cov-2 nanobodies.Nature1–8 (2025). 36
2025
-
[23]
A., MacKnight, R., Kline, B
Boiko, D. A., MacKnight, R., Kline, B. & Gomes, G. Autonomous chemical research with large language models.Nature624, 570–578 (2023)
2023
-
[24]
Wang, H.et al.Scientific discovery in the age of artificial intelligence.Nature 620, 47–60 (2023)
2023
-
[25]
arXiv preprint arXiv:2604.06132(2026)
Ye, B.et al.Claw-eval: Towards trustworthy evaluation of autonomous agents. arXiv preprint arXiv:2604.06132(2026)
2026 arXiv
-
[26]
Li, X.et al.Skillsbench: Benchmarking how well agent skills work across diverse tasks.arXiv preprint arXiv:2602.12670(2026)
2026 arXiv
-
[27]
Liu, X.et al.Agentbench: Evaluating llms as agents.arXiv preprint arXiv:2308.03688(2023)
2023 arXiv
-
[28]
International Conference on Learning Representations(2023)
Yao, S.et al.React: Synergizing reasoning and acting in language models. International Conference on Learning Representations(2023)
2023
-
[29]
Advances in Neural Information Processing Systems36, 68539–68551 (2023)
Schick, T.et al.Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems36, 68539–68551 (2023)
2023
-
[30]
Mialon, G.et al.Augmented language models: a survey.arXiv preprint arXiv:2302.07842(2023)
2023 arXiv
-
[31]
Xie, T.et al.Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments.Advances in Neural Information Processing Systems (2024)
2024
-
[32]
Trivedi, H.et al.Appworld: A controllable world of apps and people for bench- marking interactive coding agents.Association for Computational Linguistics (2024)
2024
-
[33]
E.et al.Swe-bench: Can language models resolve real-world github issues?International Conference on Learning Representations(2024)
Jimenez, C. E.et al.Swe-bench: Can language models resolve real-world github issues?International Conference on Learning Representations(2024)
2024
-
[34]
Luo, L.et al.A clinical environment simulator for dynamic ai evaluation.Nature medicine1–8 (2026)
2026
-
[35]
37 Appendix T able 2
Bareja, R.et al.Evaluating vision and pathology foundation models for com- putational pathology: a comprehensive benchmark study.medRxiv2025–05 (2025). 37 Appendix T able 2. Implementation details for checkpoint scoring and semantic adjudication.Automated semantic judging supp...
2025
Reviewed July 12, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.