Pith. sign in

REVIEW 3 major objections 6 minor 13 references

Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment

T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read A deep-learning event selection for the Intermediate Water Cherenkov Detector raises simulated electron-neutrino sample purity from 51.1% to 61.5% while improving efficiency from 69.5% to 78.2% compared to the likelihood-based fiTQun…

desk verdict A plausible but in-sample ML improvement for IWCD electron neutrino selection; the reported purity/efficiency gains are likely optimistic until validated on a fresh sample. read the letter →

arxiv 2411.09562 v1 pith:Q7EAG5X4 submitted 2024-11-14 hep-ex physics.ins-det

classification hep-exphysics.ins-det
keywords electronneutrinoCC0piselectionIntermediateWaterCherenkovDetectorHyper-KamiokandeconvolutionalneuralnetworkparticleidentificationfiTQunNCpi0backgroundpurityandefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that a modern convolutional neural network can select electron-neutrino charged-current events without pions more cleanly than the standard likelihood-based reconstruction used in water Cherenkov detectors. For the Intermediate Water Cherenkov Detector (IWCD) planned for the Hyper-Kamiokande program, the authors report that their machine-learning selection raises the purity of the simulated $\nu_e$CC0$\pi$ sample from 51.1% to 61.5% and the efficiency from 69.5% to 78.2% compared with the fiTQun likelihood algorithm. These gains matter because electron neutrinos make up only about 1.5% of the accelerator beam, so better rejection of backgrounds such as neutral-current $\pi^0$ production directly reduces the systematic uncertainties that limit measurements of CP violation in the neutrino sector. The paper presents the improvement as evidence that machine learning can outperform likelihood-based particle identification for this detector.

What carries the argument

The mechanism that carries the argument is the trained softmax output of an 18-layer residual convolutional network (ResNet-18). The network converts raw PMT hit patterns into four class probabilities $P(e)$, $P(\mu)$, $P(\gamma)$, and $P(\pi^0)$; the selection then applies manually chosen two-dimensional cuts on these probabilities versus reconstructed momentum, layered on the same fiducial-volume and kinematic cuts used in the fiTQun analysis. The baseline against which the claim is measured is fiTQun, a maximum-likelihood event reconstruction whose likelihood ratios are used for particle identification.

What would settle it

Train the identical network and apply the identical softmax cut thresholds to a freshly generated independent IWCD Monte Carlo sample, then recompute purity and efficiency; if the numbers fall materially below 61.5% and 78.2%, the reported gain is overfit to the tuning sample. A simpler check is to split the existing simulated beam sample in half, tune on one half and evaluate on the other.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a ResNet-18 convolutional network trained on four particle classes ($e^-$, $\mu^-$, $\gamma$, $\pi^0$) yields softmax probabilities that, after manual cuts tuned against reconstructed momentum, select $\nu_e$CC0$\pi$ events at higher purity and efficiency than fiTQun's likelihood ratios on the same simulated beam sample. The ML selection retained 2856 signal events versus 2535 for fiTQun, achieved 61.5% purity and 78.2% efficiency, and suppressed NC$\pi^0$ and NC$\gamma$ backgrounds more strongly, at the cost of admitting more $\nu_e$CC-other and $\bar{\nu}_e$ events. The authors conclude that with further development, more complex and automated machine-learning cuts should improve the sample beyond what is reported.

Load-bearing premise

The cuts producing the 61.5% purity and 78.2% efficiency were tuned by eye on the same simulated sample from which the final numbers are drawn, so the result assumes that sample is a faithful, unbiased stand-in for real IWCD data and that the tuning did not overfit its statistical fluctuations.

Editorial extensions

If this is right

  • A convolutional-network PID can serve as the $\nu_e$ event selection for IWCD, replacing likelihood-based cuts with a single pass through a trained classifier.
  • The higher purity and efficiency reduce the statistical and systematic footprint of the $\nu_e$ cross-section measurement, which is one of the inputs to the CP-violation sensitivity of the Hyper-Kamiokande program.
  • Because NC$\pi^0$ and NC$\gamma$ contamination is the hardest part of the likelihood selection, the ML result shows these backgrounds can be suppressed substantially without losing signal.
  • The authors expect further development of more complex and automated machine-learning cuts to push the sample's purity and efficiency beyond the reported values.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported numbers are in-sample estimates because the softmax cuts were manually tuned on the simulated beam sample used to evaluate them; an independent test sample would likely show a smaller (though possibly still positive) improvement.
  • The network was trained only on single-particle particle-gun events, not on full neutrino interaction topologies; retraining or fine-tuning on beam-like events with multiple rings could improve rejection of asymmetric $\pi^0$ decays further.
  • The same softmax-plus-cut strategy could transfer to Hyper-K's far detector, but the different PMT geometry and granularity make that transfer an open question.
  • Because the ML selection increases the $\nu_e$CC-other and $\bar{\nu}_e$ backgrounds, the net benefit for a cross-section measurement depends on how well those backgrounds are constrained independently.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This ICHEP 2024 proceedings paper compares two event-selection methods for electron-neutrino charged-current quasi-elastic (nu_e CC0pi) events in the Intermediate Water Cherenkov Detector (IWCD) of Hyper-Kamiokande, using simulated data from NEUT, WCSim, and fiTQun. The fiTQun-based analysis applies fiducial-volume, kinematic, and likelihood-ratio cuts and reports a purity of 51.1% and efficiency of 69.5%. The authors then train a ResNet-18 CNN within the WatChMaL framework on particle-gun simulated events and apply it to the same simulated beam sample; after manually tuning three softmax-probability discriminators, they report an improved purity of 61.5% and efficiency of 78.2%. The paper concludes that the ML-based selection outperforms fiTQun and expects further gains with automated cut optimization.

Significance. If the reported improvement is robust, it is a useful result for Hyper-Kamiokande's IWCD physics program: better nu_e event selection would reduce backgrounds for cross-section measurements and, ultimately, for CP-violation sensitivity. The paper provides concrete simulated event counts and a direct head-to-head comparison with an established likelihood-based reconstruction, which is valuable to the community. The main weakness is statistical: the ML cut thresholds are manually tuned on the same simulated beam sample used to quote the final purity and efficiency, so the headline numbers are in-sample estimates. A proper validation on a held-out sample, or cross-validation, is needed before the central claim can be accepted as stated.

major comments (3)
  1. [Section 3, Figure 4, Table 1] The central claim that the ResNet-18 selection achieves 61.5% purity and 78.2% efficiency is based on in-sample evaluation. The paper states that the three discriminators were manually tuned 'based on the distribution of signal and background in the histograms' (Figure 4), and that the same 'test dataset' is used for ML model evaluation. The final purity and efficiency in Table 1 are then quoted from that same sample after applying the tuned cuts. No independent validation split, cross-validation, or hold-out set is described. This is a load-bearing issue because manual threshold tuning can exploit statistical fluctuations in the simulated sample, making the reported improvement over fiTQun optimistic. Please evaluate the selected sample on events that were not used for tuning the ML cuts, and report the purity/efficiency with statistical uncertainties.
  2. [Sections 2.1 and 3, Table 1] The comparison between fiTQun and ML is not made on equal footing. The fiTQun cut lines are optimized using a predefined figure of merit, FOM = S/sqrt(S+B), whereas the ML softmax cuts are manually tuned to improve both purity and efficiency without an explicit, common objective. This asymmetry means that the ML result includes an additional hand-tuning step that can absorb favorable statistical fluctuations. A fairer comparison would either optimize both methods with the same FOM on a training sample and evaluate both on a separate test sample, or use a blinded analysis strategy.
  3. [Table 1] The central table is not self-explanatory as printed. The header appears to list more categories than can be unambiguously matched to the numeric entries, and the sum of the listed counts does not directly reproduce the denominators implied by the quoted purities (e.g., for the ML row, the listed categories sum to 5362 events, while the quoted purity of 61.5% corresponds to a denominator of about 4642 events). Please restate Table 1 with explicit column definitions, a clear 'total selected' row/column, and the exact formula used for the purity denominator, so that the results are reproducible.
minor comments (6)
  1. [Section 3] The terminology 'test dataset' is confusing: the same sample is used both for manually tuning the discriminators and for reporting the final numbers. Please use consistent terms such as training/validation/test, or describe the split explicitly.
  2. [Table 1] No statistical or systematic uncertainties are reported for the event counts, purity, or efficiency. At minimum, Poisson or binomial errors on the MC counts should be given, and the absence of detector/systematic uncertainties should be stated as a limitation.
  3. [Section 3] The input representation for the ResNet-18 model is not described: what exactly is fed to the network (mPMT hit images, charge/time channels, etc.)? Also, details of the train/validation split used during the 20-epoch training are omitted.
  4. [Section 2] The simulation chain is not fully referenced: NEUT and WCSim are mentioned in the text but no citations are provided. Please add the appropriate references.
  5. [Section 2.1] The statement that fiTQun processes 'at most 1 event per minute' is a quantitative runtime claim without supporting measurements or hardware specifications. If computational cost is part of the motivation, provide a benchmark on the same hardware for both methods.
  6. [General] Some figure references and caption details are incomplete in the extracted text (e.g., axis labels, color scales, and exact definitions of 'Total NC' versus 'NC pi0' and 'NC gamma'). Please ensure the final version includes clear, self-contained figure and table captions.

Circularity Check

1 steps flagged · score 4.0 of 10

Reported ML purity/efficiency are in-sample estimates because the three discriminators were manually tuned on the same beam sample used for Table 1; the comparison with fiTQun may be optimistic.

  1. fitted input called prediction [Section 3, Figure 4, Table 1]
    "Based on the distribution of signal and background in the histograms, we manually tuned three discriminators across P(μ), P(π0), and P(e) to improve both the purity and efficiency. After applying all these ML cuts sequentially along with the basic FV and kinematic variable cuts (as defined for the fiTQun-based analysis), the sample’s purity improved to 61.5%, with an increased efficiency of 78.2% (Table 1)."

    The histograms in Figure 4 are generated from the same test dataset that the paper says is used for ML model evaluation and that is counted in Table 1. The three discriminators are thus manually fitted to the very sample whose purity (61.5%) and efficiency (78.2%) are then reported as the result. This is an in-sample, post-tuning performance estimate: the thresholds are chosen by looking at signal and background distributions in that sample, so the quoted numbers are not an unbiased prediction for an unseen dataset. The fiTQun cuts were also optimized on the same sample using FOM = S/√(S+B), so the comparison is between two in-sample estimates; the ML side has an additional hand-tuning step that can absorb statistical fluctuations.

full rationale

The paper’s central comparison (ML 61.5%/78.2% vs fiTQun 51.1%/69.5%) is the only quantitative prediction in the paper. It is not a mathematical derivation, so there is no equation-level circularity. However, the ML thresholds are manually tuned on the same simulated beam sample whose event counts appear in Table 1, making the headline numbers trained-sample estimates. The underlying CNN itself was trained on independent particle-gun events (e−, μ−, γ, π0), and the ROC comparison in Figure 3 provides independent evidence for the ML approach; that is why this is partial circularity rather than a complete reduction to input. The paper presents no cross-validation, independent validation sample, or uncertainty on the efficiency and purity numbers. The fiTQun result is also in-sample (FOM optimized on the same sample), but the ML result includes an extra manual tuning stage, so the reported margin should not be taken at face value. Score 4 reflects one significant in-sample-fitting step while the core ML classifier retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest entirely on a Monte Carlo simulation chain (NEUT, WCSim, fiTQun) and a trained CNN. The main uncharged input is the manual tuning of selection cuts on the same simulated evaluation sample, which directly shapes the reported purity and efficiency.

free parameters (4)
  • ML softmax cut thresholds for P(e), P(mu), P(pi0) = not reported
    Manually tuned on the simulated beam dataset used for the final purity/efficiency (Section 3, Figure 4); directly set the reported 61.5%/78.2%.
  • fiTQun likelihood-ratio cut lines = not reported
    Optimized with FOM = S/sqrt(S+B) on the same simulated sample (Section 2.1); define the baseline 51.1%/69.5%.
  • ResNet-18 network weights = trained, not reported
    Trained on 12 million particle-gun simulated events (Section 3); the classifier's softmax outputs drive all ML cuts. Not a physics constant, but fitted to simulation data.
  • Fiducial volume and kinematic cut values = Dwall > 100 cm, Towall > 100 cm, momentum > 100 MeV/c
    Chosen by hand as analysis thresholds (Section 2.1), applied to both methods.
assumptions (4)
  • domain assumption NEUT neutrino interaction generator correctly models nu_e/nu_mu CC and NC processes for IWCD
    Used in Section 2.1 to generate the simulated beam sample; the central purity and efficiency numbers are entirely MC-based.
  • domain assumption WCSim detector simulation accurately reproduces the IWCD mPMT response
    Used in Section 2.1; all reconstructed events are simulated, so detector response assumptions are baked into both methods.
  • domain assumption The fiTQun likelihood function in Eq. (1) is a valid model of PMT hit probabilities for particle hypotheses
    Borrowed from prior T2K/Super-K work [7,8]; used as the baseline reconstruction and to define the fiducial and kinematic variables.
  • domain assumption The ResNet-18 architecture and WatChMaL framework are appropriate and correctly implemented
    Used in Section 3; based on references [11,12,13], with no implementation details or verification in this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment." pith.science (2026). https://pith.science/paper/Q7EAG5X4

@misc{pith2026241109562,
  author       = {Pith},
  title        = {Pith review of: Likelihood and Deep Learning Analysis of the electron neutrino event sample at Intermediate Water Cherenkov Detector (IWCD) of the Hyper-Kamiokande experiment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7EAG5X4}},
  note         = {Machine review of arXiv:2411.09562}
}
abstract

Hyper-Kamiokande (Hyper-K) is a next-generation long baseline neutrino experiment. One of its primary physics goals is to measure neutrino oscillation parameters precisely, including the Dirac CP violating phase. As conventional $\nu_{\mu}$ beam generates from the J-PARC neutrino baseline contains only 1.5$\%$ of $\nu_{e}$ interaction of total, it is challenging to measure $\nu_{e}/\bar{\nu}_{e}$ scattering cross-section on nuclei. To reduce these systematic uncertainties, IWCD will be built to study neutrino interaction rates with higher precision. Simulated data comprise $\nu_{e}CC0\pi$ as the main signal with NC$\pi^{0}$ and $\nu_{\mu}CC$ are major background events. To reduce the backgrounds initially, a log-likelihood-based reconstruction algorithm to select candidate events was used. However, this method sometimes struggles to distinguish $\pi^{0}$ events properly from electron-like events. Thus, a Machine Learning-based framework has been developed and implemented to enhance the purity and efficiency of $\nu_{e}$ events.

Figures

Figures reproduced from arXiv: 2411.09562 by the authors.

Figure 1
Figure 1. 2D histograms depicting the distribution of 𝜈𝑒𝐶𝐶0𝜋 signal in panels (a) and (c) and background events 𝜈𝜇𝐶𝐶 in panel (b), and 𝑁𝐶𝜋0 in panel (d). In each panel the log-likelihood ratios are shown with respect to either the re￾constructed lepton momentum or the reconstructed 𝜋 0 mass [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Comparison between fiTQun and ML performance in 𝛾/electron separation. Motivated by the performance of the ML framework in distinguishing particle gun events (𝑒 − , 𝜇 − , 𝛾, and 𝜋 0 ), we applied it to select 𝜈𝑒 events from a simulated beam profile. This IWCD simulated dataset contains single-ring CCQE (𝜈𝑒𝐶𝐶0𝜋, 𝜈𝜇𝐶𝐶), NC (NC 𝛾, NC𝜋 0 ), and other events such as more complex CC events with pions and/or multiple other… view at source ↗
Figure 4
Figure 4. 2D histograms depicting the distribution of 𝜈𝑒𝐶𝐶0𝜋 signal in panels (a) and (c), background events 𝜈𝜇𝐶𝐶 in panel (b), and NC𝜋 0 in panel (d). In each panel the softmax probability is shown with respect to the recon￾structed lepton momentum [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 11 canonical work pages

  1. [1]

    Atmospheric neutrino oscillation analysis with external constraints in super- kamiokande i-iv.Physical Review D, 97(7):072001, 2018

    K Abe, C Bronner, Y Haga, Y Hayato, M Ikeda, K Iyogi, J Kameda, Y Kato, Y Kishimoto, Ll Marti, et al. Atmospheric neutrino oscillation analysis with external constraints in super- kamiokande i-iv.Physical Review D, 97(7):072001, 2018

  2. [2]

    Hyper-Kamiokande Proto-Collaboration, :, K. Abe, Ke. Abe, H. Aihara, A. Aimi, R. Akutsu, C. Andreopoulos, I. Anghel, et al. Hyper-kamiokande design report, 2018. 5 IWCD electron neutrino event selection for Hyper-K experiment T. Mondal

  3. [3]

    Evidence for an oscillatory signature in atmospheric neutrino oscillations

    Y Ashie, J Hosaka, K Ishihara, Y Itow, J Kameda, Y Koshio, A Minamino, C Mitsuda, M Miura, S Moriyama, et al. Evidence for an oscillatory signature in atmospheric neutrino oscillations. Physical review letters, 93(10):101801, 2004

  4. [4]

    Cpviolationandneutrinooscillations

    HiroshiNunokawa,StephenParke,andJoséW.F.Valle. Cpviolationandneutrinooscillations. Progress in Particle and Nuclear Physics, 60(2):338–402, Apr 2008

  5. [5]

    An intermediate water cherenkov detector at j-parc

    Mark Scott. An intermediate water cherenkov detector at j-parc. InProceedings of the 10th InternationalWorkshoponNeutrino-NucleusInteractionsinFew-GeVRegion(NuInt15) ,page 010039, 2016

  6. [6]

    RB Patterson, EM Laird, Y Liu, PD Meyers, I Stancu, and HA Tanaka. The extended-track event reconstruction for miniboone.Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 608(1):206– 224, 2009

  7. [7]

    Improving the t2k oscillation analysis with fitqun: a new maximum-likelihood event reconstruction for super-kamiokande

    Andrew D Missert, T2K Collaboration, et al. Improving the t2k oscillation analysis with fitqun: a new maximum-likelihood event reconstruction for super-kamiokande. InJournal of Physics: Conference Series, volume 888, page 012066. IOP Publishing, 2017

  8. [8]

    Atmosphericneutrinooscillationanalysiswithimprovedeventreconstruction in super-kamiokande iv.Progress of Theoretical and Experimental Physics, 2019(5):053F01, 2019

    M Jiang, K Abe, C Bronner, Y Hayato, M Ikeda, K Iyogi, J Kameda, Y Kato, Y Kishimoto, LlMarti,etal. Atmosphericneutrinooscillationanalysiswithimprovedeventreconstruction in super-kamiokande iv.Progress of Theoretical and Experimental Physics, 2019(5):053F01, 2019

Show all 13 references
  1. [9]

    Deep learning.nature, 521(7553):436– 444, 2015

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning.nature, 521(7553):436– 444, 2015

  2. [10]

    AdvancesinsimulationandreconstructionforHyper-Kamiokande

    NickProuse. AdvancesinsimulationandreconstructionforHyper-Kamiokande. In Proceed- ings of 40th International Conference on High Energy physics — PoS(ICHEP2020), volume 390, page 919, 2021

  3. [11]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  4. [12]

    Watchmal: Github repository.https://github.com/WatChMaL/WatChMaL

    WatChMaL. Watchmal: Github repository.https://github.com/WatChMaL/WatChMaL. GitHub repository

  5. [13]

    Machine learning techniques to enhance event reconstruction in water cherenkov detectors

    Nicholas Prouse, Patrick de Perio, and Wojciech Fedorko. Machine learning techniques to enhance event reconstruction in water cherenkov detectors. InPhysical Sciences Forum, volume 8, page 63. MDPI, 2023. 6

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.