Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis

Alexandre Strube; Dmitriy Kostunin; Elisa Jones; Sergo Golovachev; Valery Sotnikov; Vladimir Sotnikov

REVIEW 3 major objections 5 minor 1 cited by

Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis

T0 review · 3 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Domain-adapted LLM agents with sandboxed execution and iterative error feedback can generate usable code for CTAO control software and Gammapy data-analysis scripts.

desk verdict Short proceedings on two LLM agents for CTAO; the engineering pattern is sensible, but the reliability claims are unquantified and rest on internal success proxies, so treat this as a progress report rather than a benchmarked result. read the letter →

arxiv 2510.01299 v3 pith:RVPRFKUX submitted 2025-10-01 astro-ph.IM cs.AI

Dmitriy Kostunin , Elisa Jones , Vladimir Sotnikov , Valery Sotnikov , Sergo Golovachev , Alexandre Strube This is my paper · ORCID

classification astro-ph.IMcs.AI

keywords largelanguagemodelsagenticsystemsCherenkovTelescopeArrayACADAGammapygamma-rayastronomycodegenerationself-correction

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a large language model becomes practically usable for specialized astronomy software if it is wrapped in a validation-first agent loop: the agent generates code, runs it in a sandbox, checks the output, extracts a concise error summary, and feeds that back for the next attempt. Two agents are presented: one for the CTAO control software (ACADA), which synthesizes data-structure classes from interface documents, and one for Gammapy, which turns a natural-language analysis request into a runnable Python script. Early tests show that after a few self-correction iterations, both agents produce code that passes their automated checks, and on 'straightforward' Gammapy tasks the tested models reached a 100% success rate. A sympathetic reader would care because this is evidence that agentic scaffolding, not just raw model scale, can make LLMs reliable in a fast-changing, low-training-data scientific domain.

What carries the argument

The load-bearing mechanism is the validation-first agent loop, implemented differently in each agent. In CTAgent, input interface documents are normalized to text, several prompt-tuned LLM 'experts' propose code, and an orchestrator parses and structurally tests each proposal, sending error summaries back for revision. In Gammapygpt, the agent uses strict output formatting to obtain a single self-contained script, runs it in a sandbox that mirrors an analysis environment, and if the script crashes or yields values outside an 'acceptable range,' it condenses the log into a concise error message and re-prompts the model. The same idea — externally check the artifact, extract failure informatio

What would settle it

Run a Gammapygpt-generated spectral-analysis script on a real CTAO observation, compare its flux and spectral-index result to an independent expert analysis of the same data, and check whether the discrepancy is consistent with the quoted uncertainties; a systematic disagreement would show the sandbox's run-and-range check misses the point.

Watch

Extended reading notes

Core claim

The authors' central claim is that the two agents they built — CTAgent for ACADA control-software development and Gammapygpt for Gammapy analysis — can turn a high-level description into validated, working code by combining project-specific context with an automated 'generate, run, check, repair' loop. They report that CTAgent produced 'correct, ready-to-use code for most inputs after a few iterations of self-correction,' and that in initial benchmarking both LLMs they compared reached 100% success on straightforward Gammapy analysis tasks, with the more advanced model requiring fewer repair iterations. The paper frames the lesson as 'validation-first': trust is earned by executing the code

Load-bearing premise

The entire reliability claim rests on the assumption that 'runs in the sandbox and produces values in an acceptable range' is a faithful stand-in for scientifically correct, standards-compliant code — and the paper does not test that stand-in against expert review or real CTAO data.

Editorial extensions

If this is right

CTAO developers can offload boilerplate code generation for data models and configuration schemas to CTAgent, reducing manual effort in ACADA development.
Researchers can obtain working Gammapy scripts from natural-language prompts, lowering the barrier to gamma-ray data analysis.
The validation-first pattern—sandboxed execution plus iterative error feedback—can be applied to other fast-evolving scientific libraries where general LLMs are unreliable.
Replacing or mixing proprietary models with open-access ones remains feasible through the same prompt-and-validate interface, enabling privacy-preserving offline deployments.
The paper's initial benchmarks indicate that given enough repair iterations, even less capable models can converge to passing code, so model choice mainly affects iteration count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The success criterion used by Gammapygpt—script runs and values land in an acceptable range—does not by itself establish that the analysis is scientifically valid; the paper leaves that to future expert validation, so an external benchmark against reference analyses would be the natural next test.
The staged architecture (normalization, specialized generators, orchestrator-driven repair) reads like a general template for any domain with well-documented interfaces and deterministic tests, not just CTAO; one could port it to other observatories or even non-astronomy instrument control.
A testable extension suggested by the authors' comparison: if the benefit of more advanced models is mainly fewer repair iterations, then open-source models may become competitive when combined with richer error diagnostics or longer iteration budgets, which is directly measurable.
The 'acceptable range' check could be sharpened to validate physical units, spectral shapes, or statistical uncertainties; doing so would likely catch a class of silent failures the current proxy misses.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The thing to know: this is a workshop proceedings that bundles two earlier agent papers into a single narrative. If you need a one-stop citation for "LLM agents for CTAO control software and Gammapy analysis," this serves as a pointer. But the reliability claims are mostly anecdotal. The paper says agents reach 100% success on "straightforward" tasks and produce "correct, ready-to-use code for most inputs," without giving the task list, the number of runs, a baseline, or iteration counts. Those numbers are not yet evidence.

What is genuinely useful is the design pattern, and the paper explains it clearly: convert inputs to a unified text representation, have specialized LLM experts propose code, validate by parsing/structural tests or sandboxed execution, and feed errors back for self-correction. For CTAgent, generating data classes from interface control documents is a sensible bounded use case. For Gammapygpt, sandboxed execution with output-range checks is a reasonable filter. I also appreciate the commitment to open-access local models via Helmholtz Blablador, so they are not locked into one vendor.

The weak spot is exactly what the stress-test note flags. "Successful" means the script runs and returns output in an acceptable range, or the generated code parses and passes structural tests. That is not the same as "the analysis is scientifically correct" or "the code conforms to CTAO interface semantics." A script can fit the wrong model or silently drop systematics and still run to completion with in-range values. So the 100% success rate overclaims if read as domain correctness. The paper does not define "straightforward," and there is no expert review or external benchmark. That said, this is not fatal for a proceedings: the authors are reporting work in progress, and they cite the longer reports [3]–[5] where more detail presumably lives.

My other reservation is that most of what is described here is already in those references; the only new element is the o3-vs-gpt-5 note, which is too thin to stand as a distinct result.

Who is this for? Someone tracking AI-assisted software engineering in astro-IM, or new CTAO members wanting a quick orientation. For a journal, I would not desk-reject, but I would ask the authors to either include the evaluation protocol and results in an appendix or frame the claims as preliminary. A conference proceedings can get away without that; a journal article should not.

My recommendation: if you are refereeing this for a venue that expects more than work-in-progress, send it back with a request for quantitative evaluation and an explicit statement that internal success is a proxy, not proof. If the venue is a workshop, it is acceptable with the claims toned down.

Referee Report

3 major / 5 minor

Summary. The manuscript describes two domain-adapted LLM agents for the Cherenkov Telescope Array Observatory (CTAO) ecosystem. CTAgent ingests CTAO design artifacts such as interface control documents and synthesizes data-structure/configuration code for the ACADA control software, using parsing and structural tests for validation and an iteration loop for self-repair. Gammapygpt generates Gammapy analysis scripts from natural-language prompts, executes candidate scripts in a sandbox, and feeds error summaries back to the model. The authors report that Gammapygpt reached a 100% success rate on 'straightforward' Gammapy tasks with OpenAI o3 and GPT-5, and that CTAgent produced 'correct, ready-to-use code for most inputs after a few iterations of self-correction.' The conclusion asserts that domain-specific, validation-first agents can produce reliable outputs meeting CTAO standards.

Significance. The engineering direction is sensible and timely: constraining LLMs with project documentation, sandboxed execution, and iterative error feedback is a plausible way to make code-generation assistants useful in specialized scientific domains. The paper is clearly written and appropriately gives credit to the open-source Gammapy ecosystem and to open-access model platforms. However, the central claim of 'high reliability' rests on a handful of unquantified statements. The manuscript provides no task list, no dataset, no iteration counts, no error rates, no baseline comparison, and no expert or ground-truth validation of the generated code. The evaluation proxy for Gammapygpt is that a script runs in the sandbox and returns values in an 'acceptable range'; for CTAgent it is that code parses and passes structural tests. These are internal criteria and do not establish scientific correctness or standards conformance. The absence of a defined benchmark is load-bearing because the paper's contribution is precisely the claim of reliability. The companion papers cited as [3]-[5] may contain the missing details, but the present manuscript does not.

major comments (3)

[§3 (Gammapygpt benchmarking)] The central quantitative claim—'Initial benchmarking indicates that Gammapygpt can achieve high reliability... 100% success rate on straightforward analysis tasks'—is not supported by any experimental specification. The number of tasks, the definition of 'straightforward,' the acceptable output range, the maximum number of self-correction iterations, and the criterion for final success are all absent. A 100% success rate on an unspecified set is not a measurable claim. The authors should either report a concrete benchmark with a task list and explicit success criteria, or reduce the claim to 'success on a small internal smoke-test set' and defer stronger claims to the companion paper (arXiv:2509.26110).
[§2 (CTAgent validation)] The statement that CTAgent produced 'correct, ready-to-use code for most inputs after a few iterations of self-correction' is unquantified: no number of inputs, pass rate, or iteration distribution is reported. Moreover, validation is described only as parsing and structural tests. Those checks cannot verify that generated classes correctly implement the semantics of interface control documents or conform to ACADA coding standards. 'Correct' is therefore not established. Please specify the test suite, count of documents tested, pass rates, and whether any expert review or conformance check against ICD requirements was performed.
[§3 and §2 (evaluation validity)] The success criteria are internal proxies: for Gammapygpt, a script is successful if it runs to completion and produces values within an 'acceptable range'; for CTAgent, if it parses and passes structural tests. A script can satisfy both while fitting the wrong source, using the wrong energy range, or omitting systematic effects; a data class can parse while violating interface-control semantics. The load-bearing assumption that execution/parse success equals domain correctness is unvalidated. I recommend adding external ground truth: compare generated Gammapy scripts on a small fixed task set against outputs from official Gammapy tutorials or expert-written scripts, and have a domain expert review a sample of CTAgent-generated classes. If such validation already exists in the companion papers, summarize it in the main text.

minor comments (5)

[Introduction, line 1] 'strict reproducibility ,' has a stray space before the comma.
[§2 and §3] Agent names are typeset without spacing before the citation markers: 'CTAgent[4]is' and 'Gammapygpt[5]is' should read 'CTAgent [4] is' and 'Gammapygpt [5] is'.
[§4] 'Using the Helmholtz Blablador platform 23' has an unformatted footnote marker; place the superscript and format the URL properly.
[Figure 1 caption] There is an extra space before the period in 'the Gammapy library .'.
[§3] 'OpenAI’s o3 and gpt-5' should use consistent capitalization, e.g., 'o3 and GPT-5'.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central claims rest on the effectiveness of validation loops and on self-defined success metrics. There are no fitted numerical parameters, and no new physical entities are postulated. The paper's contribution is engineering, so the ledger records domain assumptions about agent behavior rather than math axioms.

assumptions (2)

domain assumption Iterative execution/parse feedback converges to correct code
Both agents rely on a refinement loop; the paper reports success but gives no convergence statistics or iteration counts (§2, §3).
domain assumption Running script plus acceptable output range equals correct analysis
Gammapygpt uses sandbox execution and output-range checks as the verification step (§3); this is an internal proxy, not scientific validation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis." pith.science (2026). https://pith.science/paper/RVPRFKUX

@misc{pith2026251001299,
  author       = {Pith},
  title        = {Pith review of: Domain-Specific Agents for Cherenkov Telescope Array Control Software and Gamma-Ray Data Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RVPRFKUX}},
  note         = {Machine review of arXiv:2510.01299}
}

read the original abstract

We present domain-adapted large language model agents designed to support Cherenkov Telescope Array operation and data analysis. The agents combine contextual knowledge with automated validation and iterative correction to produce more reliable outputs. This approach reduces manual effort, improves consistency, and helps accelerate operational and scientific workflows. The results demonstrate the potential of agentic systems as practical assistants in specialized research environments.

Figures

Figures reproduced from arXiv: 2510.01299 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Executable verification through formalized expert reasoning in astronomical spectroscopy
astro-ph.CO 2026-07 conditional novelty 7.0 of 10

An LLM-based multi-agent system (FORMA) verifies DESI spectral classifications with 95.5% agreement to expert adjudication by reconstructing expert reasoning into an auditable workflow.

Reference graph

Works this paper leans on

8 extracted references · 1 canonical work pages · cited by 1 Pith paper

[3]

Q. Wu , G. Bansal , J. Zhang , Y. Wu , B. Li , E. Zhu , L. Jiang , X. Zhang , S. Zhang , J. Liu , A. H. Awadallah , R. W. White et al., AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , arXiv e-prints arXiv:2308.08155 (2023), doi:10.48550/arXiv.2308.08155, 2308.08155

arXiv doi:10.48550/arxiv.2308.08155 2023
[5]

Kostunin, V

D. Kostunin, V. Sotnikov, S. Golovachev, A. Mehta, T. Holch and E. Jones, Agent-based code generation for the Gammapy framework , arXiv e-prints arXiv:2509.26110 (2025), doi:10.48550/arXiv.2509.26110, 2509.26110

doi:10.48550/arxiv.2509.26110 2025
[1]

Oya et al., The first release of the Cherenkov Telescope Array Observatory array control and data acquisition software , Proc

I. Oya et al., The first release of the Cherenkov Telescope Array Observatory array control and data acquisition software , Proc. SPIE Int. Soc. Opt. Eng. 13101, 131011D (2024), doi:10.1117/12.3017568

doi:10.1117/12.3017568 2024
[2]

Schindler and R

F. Schindler and R. Trappe, NiceGUI: Web-based user interfaces with Python. The nice way , https://github.com/zauberzeug/nicegui, doi:10.5281/zenodo.7785516, Version 2.24.2 (2025)

work page doi:10.5281/zenodo.7785516 2025
[4]

Kostunin , V

D. Kostunin , V. Sotnikov , S. Golovachev and A. Strube , AI Agents for Ground-Based Gamma Astronomy , arXiv e-prints arXiv:2503.00821 (2025), doi:10.48550/arXiv.2503.00821, 2503.00821

arXiv doi:10.48550/arxiv.2503.00821 2025
[6]

Agarwal , L

OpenAI , S. Agarwal , L. Ahmad , J. Ai , S. Altman , A. Applebaum , E. Arbus , R. K. Arora , Y. Bai , B. Baker , H. Bao , B. Barak et al., gpt-oss-120b & gpt-oss-20b Model Card , arXiv e-prints arXiv:2508.10925 (2025), doi:10.48550/arXiv.2508.10925, 2508.10925

arXiv doi:10.48550/arxiv.2508.10925 2025
[7]

, " * write output.state after.block =

ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 'af...
[8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.

[1] [3]

Q. Wu , G. Bansal , J. Zhang , Y. Wu , B. Li , E. Zhu , L. Jiang , X. Zhang , S. Zhang , J. Liu , A. H. Awadallah , R. W. White et al., AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation , arXiv e-prints arXiv:2308.08155 (2023), doi:10.48550/arXiv.2308.08155, 2308.08155

arXiv doi:10.48550/arxiv.2308.08155 2023

[2] [5]

Kostunin, V

D. Kostunin, V. Sotnikov, S. Golovachev, A. Mehta, T. Holch and E. Jones, Agent-based code generation for the Gammapy framework , arXiv e-prints arXiv:2509.26110 (2025), doi:10.48550/arXiv.2509.26110, 2509.26110

doi:10.48550/arxiv.2509.26110 2025

[3] [1]

Oya et al., The first release of the Cherenkov Telescope Array Observatory array control and data acquisition software , Proc

I. Oya et al., The first release of the Cherenkov Telescope Array Observatory array control and data acquisition software , Proc. SPIE Int. Soc. Opt. Eng. 13101, 131011D (2024), doi:10.1117/12.3017568

doi:10.1117/12.3017568 2024

[4] [2]

Schindler and R

F. Schindler and R. Trappe, NiceGUI: Web-based user interfaces with Python. The nice way , https://github.com/zauberzeug/nicegui, doi:10.5281/zenodo.7785516, Version 2.24.2 (2025)

work page doi:10.5281/zenodo.7785516 2025

[5] [4]

Kostunin , V

D. Kostunin , V. Sotnikov , S. Golovachev and A. Strube , AI Agents for Ground-Based Gamma Astronomy , arXiv e-prints arXiv:2503.00821 (2025), doi:10.48550/arXiv.2503.00821, 2503.00821

arXiv doi:10.48550/arxiv.2503.00821 2025

[6] [6]

Agarwal , L

OpenAI , S. Agarwal , L. Ahmad , J. Ai , S. Altman , A. Applebaum , E. Arbus , R. K. Arora , Y. Bai , B. Baker , H. Bao , B. Barak et al., gpt-oss-120b & gpt-oss-20b Model Card , arXiv e-prints arXiv:2508.10925 (2025), doi:10.48550/arXiv.2508.10925, 2508.10925

arXiv doi:10.48550/arxiv.2508.10925 2025

[7] [7]

, " * write output.state after.block =

ENTRY address archive author booktitle chapter doi edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type url volume year label INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 'af...

[8] [8]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...