REVIEW 2 major objections 5 minor 48 references
Partitioning CTAO events by predicted direction error and analyzing them with type-specific response functions improves sensitivity by about 25% and spatial resolving power by 25-50%.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-13 04:07 UTC pith:A52EMSZD
load-bearing objection Solid end-to-end MC demonstration that event-type partitioning gives CTAO ~25% sensitivity and 25–50% angular-resolution gains; the only real soft spot is the untested transfer to real data. the 2 major comments →
Enhancing the Cherenkov Telescope Array Observatory high-level performance through an event-type-based analysis
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
An event-type analysis that ranks CTAO simulated events by multi-layer-perceptron-predicted misdirection, builds type-specific instrument response functions, and analyzes the types jointly improves combined differential sensitivity by approximately 25% (performance-per-unit-time ~1.27 North / 1.22 South) and spatial resolving power by 25-50% relative to the conventional single-IRF analysis that discards lower-quality events.
What carries the argument
Misdirection-ranked event types: a multi-layer perceptron predicts the angular difference between true and reconstructed gamma-ray direction; thresholds on that continuous score partition the data into independent quality classes, each with its own instrument response functions that are later combined in a joint high-level fit.
Load-bearing premise
The Monte-Carlo simulations used to train the predictor and to compute the response functions faithfully reproduce the reconstruction-quality distributions and gamma-hadron separation that real CTAO telescopes will deliver.
What would settle it
Apply the identical misdirection-prediction and event-type pipeline to real data from the Large-Sized Telescope prototype (LST-1) and measure whether the combined sensitivity and angular-resolution gains remain at the levels reported on the simulations.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a proof-of-concept event-type analysis for CTAO using Prod5v0.1 Monte Carlo simulations. An MLP is trained (in 20 energy quantile bins) to predict logarithmic direction reconstruction error (misdirection) from image and array parameters; events are then partitioned into four types with fixed gamma-ray fractions (15/15/30/40 %). Type-specific IRFs are produced with pyirf after independent cut optimization, and joint high-level analyses are performed with Gammapy. Relative to the standard single-IRF pipeline the authors report a combined differential sensitivity improvement of ~25 % (PPUT ≈ 1.27 North / 1.22 South) and a 25–50 % gain in spatial resolving power, demonstrated via extension-detection and close-source-separation likelihood-ratio tests.
Significance. If the gains survive the transition to real CTAO data they would be scientifically important for source-confused fields (Galactic Plane Survey), morphology studies, IGMF halo searches and spectral-line dark-matter analyses. The work is transparent: the MLP architecture, training/test split, random-partition control, multiple fraction configurations, and public tools (EventDisplay, pyirf, Gammapy) are all documented; the event-type reconstruction code is released. These elements make the result reproducible within the simulation framework and constitute a clear methodological advance over the conventional single-IRF IACT analysis.
major comments (2)
- [§2.5.1 / Fig. 3] §2.5.1 and Fig. 3: the joint sensitivity (and therefore the headline ~25 % PPUT) is obtained exclusively with the Forward-folding / Cash-statistic estimator. That estimator is shown to return fluxes ~25 % higher than the standard Li & Ma + N_excess≥10 + Bkg-fraction cuts at the highest energies, and it cannot enforce those cuts. While the relative comparison is internally consistent, the abstract and §3.1 statements of “~25 % in sensitivity” should be explicitly qualified as relative under this particular estimator; an additional joint Li & Ma-style calculation (or a hybrid) would strengthen the claim.
- [§3.1.2 / Table 1] §3.1.2 / Table 1 and §4: the chosen four-type partition already reduces the MC statistics available for the rarest event types; the paper notes that five types become statistics-limited but does not propagate the resulting IRF uncertainties into the PPUT, AP or likelihood-ratio significances. A simple bootstrap or jackknife estimate of the PPUT variance would quantify whether the reported 25 % gain remains significant once finite-MC and MLP-training fluctuations are included.
minor comments (5)
- [§4] §4, second paragraph: typographical duplication “These results were were calculated”.
- [Introduction] Introduction: “multiwavelenght” → “multiwavelength”.
- [Fig. 2] Fig. 2 caption and main text: the energy bin shown in the right panel is stated as 0.50–0.79 TeV; confirm consistency with the 20 quantile bins used for training.
- [Appendix A] Appendix A / Fig. A.12: the colour scale for Garson ranks is inverted relative to the usual “high-importance = dark” convention; a short note would avoid misreading.
- [Data Availability] Data-availability statement: the Zenodo DOI for the event-type code is given, but a short README describing the exact Prod5v0.1 subset and EventDisplay version used would improve long-term reproducibility.
Circularity Check
No significant circularity: gains are measured empirically on held-out MC after independent quality partitioning, not forced by definition or self-citation.
full rationale
The paper's chain is: (1) train an MLP regressor on 25% of Prod5 MC to predict log-misdirection from image/array features (Sec. 2.3, Fig. 2); (2) apply energy/offset-binned thresholds on the predicted quantity to the held-out 75% so that gamma-ray fractions match chosen partitions (optimized via PPUT scan in Sec. 3.1.2); (3) compute independent type-wise IRFs with pyirf (Sec. 2.4, Figs. 7-8); (4) feed those IRFs into Gammapy joint analyses and measure sensitivity (forward-folding), angular performance, extension significance and source-separation significance against the single-IRF baseline (Secs. 2.5, 3.1-3.3). None of these steps reduces the reported ~25% PPUT or 25-50% resolving-power numbers to the training labels or to a fitted parameter by construction. A random-partition control (Sec. 3.1.2) yields PPUT consistent with 1, confirming the gain is not an artifact of the MC sample size or of the IRF machinery. Self-citations ([13],[14]) are earlier conference/arXiv versions of the same idea and are not load-bearing uniqueness theorems; the method is motivated by the external Fermi-LAT precedent. Using the same MC production for training and IRF evaluation is standard practice and does not embed the final performance metrics into the event-type definition. The work is therefore a self-contained empirical demonstration on simulations, not a circular derivation.
Axiom & Free-Parameter Ledger
free parameters (4)
- MLP architecture (2 hidden layers 36/6, tanh, max 2e4 iterations, tol 1e-5)
- Event-type gamma-ray fractions (15%/15%/30%/40%)
- Number of energy bins for MLP training (20 quantile bins)
- Quality-cut optimization criteria (5σ Li&Ma, ≥10 excess, ≥5% background)
axioms (3)
- domain assumption CORSIKA + sim_telarray + EventDisplay Prod5v0.1 simulations accurately model CTAO telescope response, shower development, and reconstruction biases for both gamma rays and hadronic background.
- ad hoc to paper Predicted logarithmic misdirection is a sufficient scalar proxy for overall reconstruction quality (direction and energy).
- domain assumption Joint likelihood analysis of independent event-type datasets with Gammapy correctly combines information without introducing bias from the correlation between angular and energy resolution.
read the original abstract
The analysis traditionally employed by Imaging Atmospheric Cherenkov Telescopes involves optimizing quality cuts to select a sub-sample of high-quality events. These events are used for the scientific interpretation of the data, employing a single set of Instrument Response Functions (IRFs). All selected events are treated equally and assumed to be well represented by these IRFs, while the rest are discarded. An alternative approach, successfully applied in experiments such as Fermi-LAT, is an event-type-based analysis. This method divides datasets into subsamples, each containing events of a given expected reconstruction quality. IRFs are computed for each subsample independently, improving the accuracy with which IRFs represent the reconstruction quality of each event. The high-level analysis of these subsamples is performed treating them as independent observations, each with their own set of IRFs, and analyzed jointly. In this work we present a proof-of-concept implementation of an event-type-based analysis for the future CTAO using simulated data. A neural network (specifically a multi-layer perceptron) is trained to predict the direction reconstruction error of each event, and the simulated dataset is divided into event types based on this predicted variable. We compute IRFs for each event type and compare them with those from the standard analysis (without event types). Finally, we simulate observations using these event-type-wise IRFs and analyze them with high-level analysis tools to test the performance of both approaches. This implementation demonstrates notable improvements: 25% to 50% boost in spatial resolving power and ~25% in sensitivity. This boost in performance will have strong implications in the scientific exploitation of the CTAO data, especially in crowded regions such as the Galactic Plane or searching for spectral signatures like Dark Matter annihilation lines.
Figures
Reference graph
Works this paper leans on
-
[1]
Zanin, et al., CTA – the World’s largest ground-based gamma-ray observatory, PoS ICRC2021 (2021) 005
R. Zanin, et al., CTA – the World’s largest ground-based gamma-ray observatory, PoS ICRC2021 (2021) 005. doi:10.22323/1.395.0005
-
[2]
Cherenkov Telescope Array Consortium, et al., Science with the Cherenkov Telescope Array, World Scientific, 2019. doi:10.1142/10986. 31 Figure A.12: Grid showing the ranking of importance calculated using Garson’s algorithm. Annotations in the grid represent the rank of that particular variable and energy bin, with the x axis ordered by the overall import...
doi:10.1142/10986 2019
-
[3]
Cherenkov Telescope Array Observatory and Cherenkov Telescope Ar- ray Consortium, Ctao instrument response functions - prod5 version v0.1, 2021.URL:https://doi.org/10.5281/zenodo.5499840.doi:10. 5281/zenodo.5499840
-
[4]
K. Bernlöhr, O. Gueta, G. Maier, A. Moralejo, Y. Suda, CTAO Simulation Telescope Models for CORSIKA and sim_telarray - prod5, 2022. URL:https://doi.org/10.5281/zenodo.6218687. doi:10.5281/zenodo.6218687
-
[5]
K. Bernlöhr, et al., Monte carlo design studies for the cherenkov tele- scope array, Astroparticle Physics 43 (2013) 171–188. URL:https: //doi.org/10.1016/j.astropartphys.2012.10.002. doi:10.1016/j. astropartphys.2012.10.002
-
[6]
T. Hassan, et al., Monte carlo performance studies for the site selection of the cherenkov telescope array, Astroparticle Physics 93 (2017) 76–85. doi:https://doi.org/10.1016/j.astropartphys.2017.05.001. 32
-
[7]
A. Acharyya, et al., Monte carlo studies for the optimisation of the cherenkov telescope array layout, Astroparticle Physics 111 (2019) 35–53. URL:https://doi.org/10.1016/j.astropartphys.2019.04
-
[8]
doi:10.1016/j.astropartphys.2019.04.001
-
[9]
Donath, et al., Gammapy: A python package for gamma-ray as- tronomy, A&A 678 (2023) A157
A. Donath, et al., Gammapy: A python package for gamma-ray as- tronomy, A&A 678 (2023) A157. URL:https://doi.org/10.1051/ 0004-6361/202346488. doi:10.1051/0004-6361/202346488
-
[10]
Knödlseder, et al., GammaLib and ctools
J. Knödlseder, et al., GammaLib and ctools. A software frame- work for the analysis of astronomical gamma-ray data, Astronomy & Astrophysics 593 (2016) A1. doi:10.1051/0004-6361/201628822. arXiv:1606.00393
-
[11]
G. D’Amico, T. Terzić, J. Strišković, M. Doro, M. Strzys, J. van Scherpenberg, Signal estimation in on/off measurements including event-by-event variables, Phys. Rev. D 103 (2021) 123001. URL: https://link.aps.org/doi/10.1103/PhysRevD.103.123001. doi:10. 1103/PhysRevD.103.123001
-
[12]
W. B. Atwood, A. A. Abdo, et al., The large area telescope on the fermi gamma-ray space telescope mission, The Astrophysical Journal 697 (2009) 1071. URL:https://dx.doi.org/10.1088/0004-637X/697/2/
-
[13]
doi:10.1088/0004-637X/697/2/1071
-
[14]
Atwood, et al., Pass 8: Toward the full realization of the fermi-lat scientific potential, 2013
W. Atwood, et al., Pass 8: Toward the full realization of the fermi-lat scientific potential, 2013. URL:https://arxiv.org/abs/1303.3514. arXiv:1303.3514
Pith/arXiv arXiv 2013
-
[15]
T. Hassan, O. Gueta, G. Maier, M. Nöthe, M. Peresano, I. Vovk, Per- formance of a proposed event-type based analysis for CTA, in: Proceed- ings of 37th International Cosmic Ray Conference — PoS(ICRC2021), Sissa Medialab, 2021, p. 752. URL:https://doi.org/10.22323/1. 395.0752. doi:10.22323/1.395.0752
doi:10.22323/1 2021
-
[16]
J. Bernete, O. Gueta, T. Hassan, M. Linhoff, G. Maier, A. Sinha, Perfor- mance update of an event-type based analysis for the Cherenkov Tele- scope Array, arXiv e-prints (2023) arXiv:2309.11375. doi:10.48550/ arXiv.2309.11375.arXiv:2309.11375. 33
Pith/arXiv arXiv 2023
-
[17]
H. Abdalla, et al., The h.e.s.s. galactic plane survey, Astronomy & Astrophysics 612 (2018) A1. URL:http://dx.doi.org/10.1051/ 0004-6361/201732098. doi:10.1051/0004-6361/201732098
-
[18]
H. Abdalla, et al., Sensitivity of the cherenkov telescope array for probing cosmology and fundamental physics with gamma-ray propa- gation, Journal of Cosmology and Astroparticle Physics 2021 (2021) 048–048. URL:https://doi.org/10.1088/1475-7516%2F2021%2F02% 2F048. doi:10.1088/1475-7516/2021/02/048
-
[19]
A. Acharyya, et al., Sensitivity of the cherenkov telescope array to a dark matter signal from the galactic centre, Journal of Cosmology and Astroparticle Physics 2021 (2021) 057. URL:https://doi.org/ 10.1088/1475-7516/2021/01/057. doi:10.1088/1475-7516/2021/01/ 057
-
[20]
K. Abe, et al., Prospects for dark matter observations in dwarf spheroidal galaxies with the cherenkov telescope array observa- tory, Monthly Notices of the Royal Astronomical Society 544 (2025) 2946–2986. URL:https://doi.org/10.1093/mnras/staf1798. doi:10.1093/mnras/staf1798
-
[21]
A. Acharyya, et al., Sensitivity of the cherenkov telescope array to tev photon emission from the large magellanic cloud, Monthly Notices of the Royal Astronomical Society 523 (2023) 5353–5387. URL:https: //doi.org/10.1093/mnras/stad1576. doi:10.1093/mnras/stad1576
-
[22]
S. Abe, et al., Dark matter line searches with the cherenkov telescope array, Journal of Cosmology and Astroparticle Physics 2024 (2024) 047. URL:http://dx.doi.org/10.1088/1475-7516/2024/07/047. doi:10. 1088/1475-7516/2024/07/047
-
[23]
D. Heck, J. Knapp, J. N. Capdevielle, G. Schatz, T. Thouw, COR- SIKA: a Monte Carlo code to simulate extensive air showers., Tech- nical Report FZKA 6019, Forschungszentrum Karlsruhe, 1998. URL: https://publikationen.bibliothek.kit.edu/270043064/3813660
arXiv 1998
-
[24]
Bernlöhr, Simulation of imaging atmospheric Cherenkov telescopes with CORSIKA and sim_ telarray, Astroparticle Physics 30 (2008) 149–
K. Bernlöhr, Simulation of imaging atmospheric Cherenkov telescopes with CORSIKA and sim_ telarray, Astroparticle Physics 30 (2008) 149–
2008
-
[25]
doi:10.1016/j.astropartphys.2008.07.009.arXiv:0808.2253. 34
Pith/arXiv arXiv doi:10.1016/j.astropartphys.2008.07.009.arxiv:0808.2253 2008
-
[26]
S. Ostapchenko, Monte carlo treatment of hadronic interac- tions in enhanced pomeron scheme: Qgsjet-ii model, Phys. Rev. D 83 (2011) 014018. URL:https://link.aps.org/doi/10.1103/ PhysRevD.83.014018. doi:10.1103/PhysRevD.83.014018
-
[27]
S. Bass, Microscopic models for ultrarelativistic heavy ion collisions, Progress in Particle and Nuclear Physics 41 (1998) 255–369. URL: http://dx.doi.org/10.1016/S0146-6410(98)00058-1. doi:10.1016/ s0146-6410(98)00058-1
-
[28]
M. Bleicher, et al., Relativistic hadron-hadron collisions in the ultra- relativistic quantum molecular dynamics model, Journal of Physics G: Nuclear and Particle Physics 25 (1999) 1859. URL:https://dx. doi.org/10.1088/0954-3899/25/9/308. doi:10.1088/0954-3899/25/ 9/308
-
[29]
G. Maier, J. Holder, Eventdisplay: An Analysis and Reconstruction Package for Ground-based Gamma-ray Astronomy, in: Proceedings, 35th International Cosmic Ray Conference, volume 301, 2017, p. 747. doi:10.22323/1.301.0747
-
[30]
G. Maier, et al., Eventdisplay: An Analysis and Reconstruction Package for Ground-based Gamma-ray Astronomy, 2024. doi:10.5281/zenodo. 11096726
doi:10.5281/zenodo 2024
-
[31]
Pedregosa, et al., Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830
F. Pedregosa, et al., Scikit-learn: Machine learning in Python, Journal of Machine Learning Research 12 (2011) 2825–2830
2011
-
[32]
G. D. Garson, Interpreting neural-network connection weights, Artificial Intelligence Expert 6 (1991) 46–51
1991
-
[33]
A. Goh, Back-propagation neural networks for modeling complex sys- tems, Artificial Intelligence in Engineering 9 (1995) 143–151. doi:https: //doi.org/10.1016/0954-1810(94)00011-S
-
[34]
G. Schwefer, R. Parsons, J. Hinton, A hybrid approach to event re- construction for atmospheric cherenkov telescopes combining machine learning and likelihood fitting, 2024. URL:https://arxiv.org/abs/ 2406.17502.arXiv:2406.17502. 35
Pith/arXiv arXiv 2024
-
[35]
T. P. Li, Y. Q. Ma, Analysis methods for results in gamma-ray as- tronomy., The Astrophysical Journal 272 (1983) 317–324. doi:10.1086/ 161295
1983
-
[36]
URL:https:// www.mdpi.com/2218-1997/7/10/374
C.Nigro, T.Hassan, L.Olivera-Nieto, Evolutionofdataformatsinvery- high-energy gamma-ray astronomy, Universe 7 (2021). URL:https:// www.mdpi.com/2218-1997/7/10/374. doi:10.3390/universe7100374
-
[37]
Deil, et al., Open high-level data formats and software for gamma- ray astronomy, in: AIP Conference Proceedings, volume 1792, AIP Publishing LLC, 2017, p
C. Deil, et al., Open high-level data formats and software for gamma- ray astronomy, in: AIP Conference Proceedings, volume 1792, AIP Publishing LLC, 2017, p. 070006
2017
-
[38]
M. Linhoff, et al., pyirf, 2025. URL:https://github.com/ cta-observatory/pyirf. doi:10.5281/zenodo.15488321
-
[39]
Aguasca-Cabot, et al., Gammapy: Python toolbox for gamma-ray astronomy, 2023
A. Aguasca-Cabot, et al., Gammapy: Python toolbox for gamma-ray astronomy, 2023. URL:https://doi.org/10.5281/zenodo.8033275. doi:10.5281/zenodo.8033275
-
[40]
Acero, et al., Gammapy: Python toolbox for gamma-ray as- tronomy, 2025
F. Acero, et al., Gammapy: Python toolbox for gamma-ray as- tronomy, 2025. URL:https://doi.org/10.5281/zenodo.17814297. doi:10.5281/zenodo.17814297
-
[41]
W. Cash, Parameter estimation in astronomy through application of the likelihood ratio., The Astrophysical Journal 228 (1979) 939–947. doi:10.1086/156922
doi:10.1086/156922 1979
-
[42]
F. Aharonian, et al., The crab nebula and pulsar between 500 gev and 80 tev: Observations with the hegra stereoscopic air cerenkov telescopes, The Astrophysical Journal 614 (2004) 897. URL:https://dx.doi.org/ 10.1086/423931. doi:10.1086/423931
-
[43]
S. Abe, et al., Prospects for a survey of the galactic plane with the cherenkov telescope array, Journal of Cosmology and Astropar- ticle Physics 2024 (2024) 081. URL:https://dx.doi.org/10.1088/ 1475-7516/2024/10/081. doi:10.1088/1475-7516/2024/10/081
-
[44]
Zanin, et al., MARS, The MAGIC Analysis and Reconstruction Software, in: International Cosmic Ray Conference, volume 33 ofInter- national Cosmic Ray Conference, 2013, p
R. Zanin, et al., MARS, The MAGIC Analysis and Reconstruction Software, in: International Cosmic Ray Conference, volume 33 ofInter- national Cosmic Ray Conference, 2013, p. 2937. 36
2013
-
[45]
D. Morcuende, et al., Performance of the large-sized telescope proto- type of the cherenkov telescope array, in: Proceedings of 38th Interna- tional Cosmic Ray Conference — PoS(ICRC2023), ICRC2023, Sissa Me- dialab, 2023, p.594.URL:http://dx.doi.org/10.22323/1.444.0594. doi:10.22323/1.444.0594
-
[46]
Bernete, O
J. Bernete, O. Gueta, T. Hassan, Iact event types, 2025. URL:https: //github.com/cta-observatory/iact_event_types
2025
-
[47]
A. M. Hillas, Cerenkov Light Images of EAS Produced by Primary Gamma Rays and by Nuclei, in: F. C. Jones (Ed.), 19th International Cosmic Ray Conference (ICRC19), Volume 3, volume 3 ofInternational Cosmic Ray Conference, 1985, p. 445
1985
-
[48]
H. Krawczynski, et al., Gamma hadron separation methods for the VERITAS array of four imaging atmospheric Cherenkov telescopes, As- troparticle Physics 25 (2006) 380–390. doi:10.1016/j.astropartphys. 2006.03.011.arXiv:astro-ph/0604508. 37
work page internal anchor Pith review Pith/arXiv arXiv doi:10.1016/j.astropartphys 2006
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.