Pith. sign in

REVIEW 3 major objections 5 minor 20 references

GollumFit: An IceCube Open-Source Framework for Binned-Likelihood Neutrino Telescope Analyses

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Open-source fitter tames 38 nuisance parameters for neutrino telescopes

desk verdict GollumFit is a genuinely useful open-source software artifact whose main physics claim is under-validated by a single self-closure test on compressed MC; worth peer review, with revisions. read the letter →

arxiv 2506.04491 v2 pith:D4ZHFORD submitted 2025-06-04 hep-ex astro-ph.HEphysics.comp-phphysics.data-an

IceCube Collaboration This is my paper · ORCID
classification hep-exastro-ph.HEphysics.comp-phphysics.data-an
keywords binnedlikelihoodneutrinotelescopeIceCubenuisanceparameterseventreweightingautomaticdifferentiationFastMCL-BFGS-B
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents GollumFit, an open-source framework for binned-likelihood fits to neutrino telescope data. It argues that the framework solves the practical problem of fitting dozens of nuisance parameters simultaneously, and demonstrates that it recovers known input values and runs in minutes to hours on a typical analysis. The key is combining event-by-event reweighting with automatic differentiation and a Monte Carlo compression step called FastMC. If correct, this gives the neutrino community a reusable tool for diffuse-flux and sterile-neutrino analyses that previously required bespoke code.

What carries the argument

The load-bearing object is the per-event weight function $w_e(\vec{q}_e, \vec{Q}_e, \vec{\theta}, \vec{\eta})$ that maps model parameters to the Poisson expectation $\mu_i = \sum_{e \in i} w_e$ in each bin. GollumFit evaluates this weight by three parametrizations (analytical formulas, gradients, or splines), and the likelihood is a modified Poisson (effective) likelihood that accounts for Monte Carlo statistical uncertainty. The optimization uses L-BFGS-B through PhysTools with automatic differentiation for gradients. FastMC then merges events into 'meta events' by binning in true and reconstructed space with a scale factor $k$, preserving the total event number while the number of events shrinks, which speeds up each likelihood evaluation.

What would settle it

Run the same fit on the same data with two very different FastMC compression scales (e.g., $k=0.1$ and $k=0.5$) and compare the best-fit nuisance parameters: if the per-bin expectations of Eq. (6) are not preserved, the best-fit values will shift by more than the statistical uncertainty.

Watch

Extended reading notes

Core claim

GollumFit's central claim is that a binned-likelihood analysis can incorporate the full set of neutrino-telescope-specific model parameters (38 nuisance parameters covering flux normalization, atmospheric flux shape, cross-section uncertainties, detector efficiency, hole ice, and bulk ice) and still converge to the true parameter values in a feasible time. The paper shows a single fit recovering all nominal nuisance parameters from random initialization, linear scaling of likelihood-evaluation time with Monte Carlo size, and speedups from omitting parameter classes that depend on the weighting method, not simply the number of parameters. The framework bundles nominal IceCube Monte Carlo events as a reference dataset, so analyses can run out of the box.

Load-bearing premise

The compression parameter $k$ must be hand-chosen so that merging Monte Carlo events into meta events preserves each bin's expected count for every plausible value of the model parameters; the paper provides no automatic way to verify this.

Editorial extensions

If this is right

  • Fits over tens of nuisance parameters recover nominal values even when initialized randomly away from the truth.
  • Time per likelihood evaluation scales linearly with the number of Monte Carlo events, and FastMC reduces the event set from about 13 million to about 0.8 million at $k=0.25$, cutting per-evaluation time.
  • Fit time depends on the weighting method of the parameters, not just their count; omitting six cosmic-ray parameters can be slower than omitting four astrophysical parameters.
  • The framework is designed to be extendable to other neutrino telescopes and to joint multi-telescope analyses, provided new Monte Carlo and splines are supplied.
  • FastMC preserves the total expected event number and, when $k$ is chosen correctly, the per-bin expectations for all reasonable model parameters.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If FastMC compression is validated per analysis, the same reweighting-and-minimization machinery could be applied to other binned-likelihood experiments (e.g., cosmic-ray or gamma-ray observatories) whose systematic uncertainties are encoded as event-weight derivatives.
  • The modular weighter design suggests that adding new observables (such as azimuth or event morphology) only requires redefining the binning and weighters, so joint analyses across detectors become a matter of sharing splines and priors.
  • Because the paper only benchmarks parameter recovery on nominal Monte Carlo, a natural stress test is to inject a mismodeled detector response and see whether the 38-parameter fit absorbs it or biases the physics parameters.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript presents GollumFit, an open-source C++/Python framework for binned-likelihood analyses in neutrino telescope experiments, together with a set of 38 nuisance parameters spanning atmospheric and astrophysical fluxes, neutrino cross sections, and IceCube-specific detector response. The likelihood is evaluated by reweighting Monte Carlo events, and the FastMC procedure compresses the Monte Carlo set by merging nearby events into meta events. The performance section reports a single closure fit on FastMC-compressed pseudo-data and timing benchmarks on a single machine. The central claim is that GollumFit can accurately and quickly recover input nuisance parameter values in high-dimensional fits.

Significance. If the central claim were fully validated, GollumFit would be a valuable community resource for neutrino telescope analyses: the code is publicly available under LGPL with documentation, a reference Monte Carlo set is included, and the paper is candid about current limitations such as manual tuning of the compression scale and sensitivity to initialization. The likelihood formulation (Eq. 1) and the use of automatic differentiation are standard and sensible. However, the validation is currently too thin to support the claims of accuracy and time-efficiency: it rests on a single self-closure test at one compression level, and the key compression approximation (Eq. 6) is asserted rather than tested. The required additional work is well-scoped and does not require a change in the framework's architecture.

major comments (3)
  1. [Section 3.4, Eq. (6)] Equation (6) is load-bearing but not validated. The manuscript states that k must be chosen so that the per-bin expectation from meta events equals the full-MC expectation for all reasonable parameter values, but this equality is asserted rather than derived or tested. For the gradient and spline reweighters in Table 1, the per-event weight is a nonlinear function of the true and reconstructed quantities, so the sum of reweighted individual events is not generally equal to the count times the weight evaluated at the weighted-average meta-event quantities; the equality N = sum_e w_e = sum_e-bar w_e-bar holds only at the original weights, not under reweighting. Because Figures 3-5 and the closure test all use FastMC at k=0.25, an unvalidated Eq. (6) directly affects the central accuracy and speed claims. I request a validation study: compare per-bin expectations from FastMC with those from the full 13M-event Monte Carlo set for each parameter varied over, say, +/-2 prior widths, report the maximum and mean per-bin deviations, and show at least one fit with low compression or no compression for comparison.
  2. [Section 4, Figure 3] The single closure fit shown in Figure 3 cannot detect compression bias because both the pseudo-data and the fitted model are produced from the same FastMC-compressed set at k=0.25. The test verifies internal consistency of the fitting machinery but not the fidelity of Eq. (6). A stronger validation would use an ensemble of pseudo-experiments, report the distribution of best-fit pulls and their coverage, and include at least one fit in which the pseudo-data and/or the model are constructed from the full (uncompressed) Monte Carlo set. Without this, the claim that GollumFit can 'accurately' recover input nuisance parameters is not established.
  3. [Section 4, Figures 4 and 5] The timing benchmarks are single runs on a single machine and are not compared with any alternative implementation. Figure 4 shows a linear scaling of per-evaluation time with the number of compressed events, but no comparison with the uncompressed 13M-event set or with an existing binned-likelihood tool, so the claim of being 'time-efficient' is not quantified relative to the available alternatives. Figure 5 reports fit durations between about 1 and 3 hours without run-to-run spread or a stated convergence criterion. I request repeated runs with reported medians and spreads, and, if feasible, a timing comparison with an uncompressed Monte Carlo fit or with a standard reweighting implementation.
minor comments (5)
  1. [Abstract and Section 1] The abstract claims the framework 'uniquely incorporates' the model parameters necessary for neutrino telescopes, but no comparison with existing software is provided; I suggest either softening this wording or supporting it with a short survey of related frameworks.
  2. [Section 2.2] The sentence beginning 'IceCube of nuisance parameters are the efficiency...' appears to be missing words; it should read 'Examples of IceCube nuisance parameters are...'.
  3. [Section 3.2] The phrase 'PhysTools;uses the L-BFGS-B algorithm' contains a stray semicolon; it should read 'PhysTools uses the L-BFGS-B algorithm'.
  4. [Section 3.3] There are several typographical spacing issues, including 'GollumFitis' and 'aDataPaths', and 'respetively' should be 'respectively'.
  5. [Section 5 and Figure 4 caption] Section 5 contains 'Nontheless' which should be 'Nonetheless', and the Figure 4 caption should say 'as a function of the number of MC events' rather than 'as function'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: GollumFit is a software and validation paper whose closure test and FastMC approximation are self-consistency checks or stated limitations, not derivations that reduce to their own inputs.

full rationale

This paper presents a software framework for binned-likelihood neutrino telescope analyses; it does not claim a physics prediction or a first-principles derivation, so the central circularity patterns do not apply. The single validation fit in Fig. 3 is explicitly framed as 'to ensure that GollumFit can successfully recover values' by fitting an output from FastMC at nominal nuisance parameter values. Because the pseudo-data and the fitted model are generated from the same Monte Carlo expectation, the test checks internal consistency of the optimizer and gradient implementation, not an external prediction. The paper does not rename this closure test as a discovery, so this is not a fitted input called a prediction. The FastMC approximation in Eq. (6) asserts that meta-event expectations equal full-MC expectations for reasonable parameters; the paper itself states that 'the need to find an optimal value of k represents a limitation of the FastMC procedure' and that a robust way to determine it 'represents a possible avenue for future work.' An accuracy assumption that is acknowledged as unvalidated is a correctness or robustness risk, not a circular step. The paper's self-citations to prior IceCube work (e.g., Refs. [5], [6], [8], [13], [15]) supply Monte Carlo sets, detector models, and parameter prescriptions; these are external data and models, not unverified uniqueness theorems invoked to force the paper's conclusions. No step in the paper's argument reduces by construction to its own inputs. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the faithfulness of the reweighting parametrizations, the atmospheric flux and ice models from prior IceCube literature, and the validity of the example Monte Carlo set. These are inputs from previous work, not derived here. The FastMC compression adds one hand-chosen parameter, k.

free parameters (1)
  • FastMC compression scale k = 0.25 in the presented fits
    Controls the trade-off between Monte Carlo compression and accuracy; must be chosen by hand on a case-by-case basis (Sec. 3.4, Figs. 3-5 captions). The paper states this is a limitation and future work.
assumptions (4)
  • standard math The effective binned likelihood of Ref. [16] correctly describes the statistical uncertainties of data and Monte Carlo samples.
    Equations (1)-(3) use this likelihood without re-derivation.
  • domain assumption The gradient- and spline-based reweighting methods faithfully represent how event weights depend on each nuisance parameter.
    Section 3.1; the accuracy of fits depends on these parametrizations.
  • domain assumption The DAEMONFLUX atmospheric flux model and the SnowStorm ice model provide valid parametrizations of the relevant systematic uncertainties.
    Section 2.2; imported from Refs. [14,15].
  • domain assumption The LeptonInjector Monte Carlo sample supplied with the framework is representative of neutrino telescope events.
    Sections 1 and 3; the benchmarks and validation use this specific sample.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GollumFit: An IceCube Open-Source Framework for Binned-Likelihood Neutrino Telescope Analyses." pith.science (2026). https://pith.science/paper/D4ZHFORD

@misc{pith2026250604491,
  author       = {Pith},
  title        = {Pith review of: GollumFit: An IceCube Open-Source Framework for Binned-Likelihood Neutrino Telescope Analyses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D4ZHFORD}},
  note         = {Machine review of arXiv:2506.04491}
}
read the original abstract

We present GollumFit, a framework designed for performing binned-likelihood analyses on neutrino telescope data. GollumFit incorporates model parameters common to any neutrino telescope and also model parameters specific to the IceCube Neutrino Observatory. We provide a high-level overview of its key features and how the code is organized. We then discuss the performance of the fitting in a typical analysis scenario, highlighting the ability to fit over tens of nuisance parameters. We present some examples showing how to use the package for likelihood minimization tasks. This framework uniquely incorporates the particular model parameters necessary for neutrino telescopes, and solves an associated likelihood problem in a time-efficient manner.

Figures

Figures reproduced from arXiv: 2506.04491 by the authors.

Figure 1
Figure 1. Left: a plot of the binned Monte Carlo event distribution, binned in the space of reconstructed log energy (log10(𝐸∕GeV)) and the cosine of the reconstructed zenith angle (cos 𝜃). The color scale indicates the number 𝑁 of events in each bin. To generate this distribution, all nuisance parameters are held at their prior central value. Right: in contrast, a pull plot that compares another Monte Carlo event distributio… view at source ↗
Figure 2
Figure 2. Overview of GollumFit. This illustration shows the main components of the main GollumFit object, the necessary inputs, and the most useful functions for analysis purposes. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. The result of minimization for a single fit, comparing the best-fit and initial value of the nuisance parameters, using 𝜎, the number of prior standard deviations away from the prior value. The parameter astroPivot has a uniform prior, and no influence on the shape of the flux, and therefore little effect on the likelihood. Hence there is no preferred value to be recovered. A value of 𝑘 = 0.25 was used for the FastM… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Plot of time per likelihood evaluation as function of the number MC events looped over (corresponding to the size of the FastMC file). For reference, the total MC dataset, before the FastMC compression, consists of 13,061,947 events. 15 [PITH_FULL_IMAGE:figures/full_f…
Figure 5
Figure 5. Figure 5: Plot of the time taken for a fit (with identical seed nuisance parameters) to converge. Different bars are for different sets of nuisance parameters turned off. The number of remaining nuisance parameters fitted over is given in the parenthesis. This was performed usin…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 5 canonical work pages

  1. [1]

    Ahrens, et al., Icecube - the Next Generation Neutrino Telescope at the South Pole, Nucl

    J. Ahrens, et al., Icecube - the Next Generation Neutrino Telescope at the South Pole, Nucl. Phys. B Proc. Suppl. 118 (2003) 388–395. arXiv:astro-ph/0209556,doi:10.1016/S0920-5632(03)01337-9

    arXiv 2003

  2. [2]

    Achterberg, et al., First Year Performance of The IceCube Neutrino Telescope, Astropart

    A. Achterberg, et al., First Year Performance of The IceCube Neutrino Telescope, Astropart. Phys. 26 (2006) 155–173.arXiv:astro-ph/ 0604450,doi:10.1016/j.astropartphys.2006.06.007

    work page doi:10.1016/j.astropartphys.2006.06.007 2006

  3. [3]

    Adrian-Martinez, et al., Letter of intent for KM3NeT 2.0, J

    S. Adrian-Martinez, et al., Letter of intent for KM3NeT 2.0, J. Phys. G 43 (8) (2016) 084001.arXiv:1601.07459,doi:10.1088/ 0954-3899/43/8/084001

    arXiv 2016

  4. [4]

    Aiello, et al., Sensitivity of the KM3NeT/ARCA neutrino telescope to point-like neutrino sources, Astropart

    S. Aiello, et al., Sensitivity of the KM3NeT/ARCA neutrino telescope to point-like neutrino sources, Astropart. Phys. 111 (2019) 100–110. arXiv:1810.08499,doi:10.1016/j.astropartphys.2019.04.002

    arXiv 2019

  5. [5]

    Abbasi, et al., Improved Characterization of the Astrophysical Muon–neutrino Flux with 9.5 Years of IceCube Data, Astrophys

    R. Abbasi, et al., Improved Characterization of the Astrophysical Muon–neutrino Flux with 9.5 Years of IceCube Data, Astrophys. J. 928 (1) (2022) 50.arXiv:2111.10299,doi:10.3847/1538-4357/ac4d29

    arXiv 2022

  6. [6]

    Abbasi, et al., A search for an eV-scale sterile neutrino using improved high-energy𝜈𝜇 event reconstruction in IceCube (5 2024)

    R. Abbasi, et al., A search for an eV-scale sterile neutrino using improved high-energy𝜈𝜇 event reconstruction in IceCube (5 2024). arXiv:2405.08070

    arXiv 2024

  7. [7]

    Abbasi, et al., LeptonInjector and LeptonWeighter: A neutrino event generator and weighter for neutrino observatories, Comput

    R. Abbasi, et al., LeptonInjector and LeptonWeighter: A neutrino event generator and weighter for neutrino observatories, Comput. Phys. Commun. 266 (2021) 108018.arXiv:2012.10449,doi:10.1016/j.cpc.2021.108018

    work page arXiv 2021

  8. [8]

    Abbasi, et al., Methods and stability tests associated with the sterile neutrino search using improved high-energy𝜈𝜇 event reconstruction in IceCube (5 2024).arXiv:2405.08077

    R. Abbasi, et al., Methods and stability tests associated with the sterile neutrino search using improved high-energy𝜈𝜇 event reconstruction in IceCube (5 2024).arXiv:2405.08077

    work page arXiv 2024

Show all 20 references
  1. [9]

    M.G.Aartsen,etal.,SearchesforSterileNeutrinoswiththeIceCubeDetector,Phys.Rev.Lett.117(7)(2016)071801.arXiv:1605.01990, doi:10.1103/PhysRevLett.117.071801

    2016 arXiv

  2. [10]

    M. G. Aartsen, et al., eV-Scale Sterile Neutrino Search Using Eight Years of Atmospheric Muon Neutrino Data from the IceCube Neutrino Observatory, Phys. Rev. Lett. 125 (14) (2020) 141801.arXiv:2005.12942,doi:10.1103/PhysRevLett.125.141801. 12

    2020

  3. [11]

    Abbasi, et al., The IceCube high-energy starting event sample: Description and flux characterization with 7.5 years of data, Phys

    R. Abbasi, et al., The IceCube high-energy starting event sample: Description and flux characterization with 7.5 years of data, Phys. Rev. D 104 (2021) 022002.arXiv:2011.03545,doi:10.1103/PhysRevD.104.022002

    2021

  4. [12]

    Abbasi, et al., Observation of Seven Astrophysical Tau Neutrino Candidates with IceCube, Phys

    R. Abbasi, et al., Observation of Seven Astrophysical Tau Neutrino Candidates with IceCube, Phys. Rev. Lett. 132 (15) (2024) 151001. arXiv:2403.02516,doi:10.1103/PhysRevLett.132.151001

    2024

  5. [13]

    M. G. Aartsen, et al., Searching for eV-scale sterile neutrinos with eight years of atmospheric neutrinos at the IceCube Neutrino Telescope, Phys. Rev. D 102 (5) (2020) 052009.arXiv:2005.12943,doi:10.1103/PhysRevD.102.052009

    2020

  6. [14]

    J. P. Yañez, A. Fedynitch, Data-driven muon-calibrated neutrino flux, Phys. Rev. D 107 (12) (2023) 123037.arXiv:2303.00022,doi: 10.1103/PhysRevD.107.123037

    2023 arXiv

  7. [15]

    M. G. Aartsen, et al., Efficient propagation of systematic uncertainties from calibration to analysis with the SnowStorm method in IceCube, JCAP 10 (2019) 048.arXiv:1909.01530,doi:10.1088/1475-7516/2019/10/048

    2019

  8. [16]

    C. A. Argüelles, A. Schneider, T. Yuan, A binned likelihood for stochastic models, JHEP 06 (2019) 030.arXiv:1901.04645,doi: 10.1007/JHEP06(2019)030

    2019 arXiv

  9. [17]

    R.H.Byrd,P.Lu,J.Nocedal,C.Zhu,Alimitedmemoryalgorithmforboundconstrainedoptimization,SIAMJournalonScientificComputing 16 (5) (1995) 1190–1208.doi:10.1137/0916069

    1995 doi

  10. [18]

    C. Zhu, R. H. Byrd, P. Lu, J. Nocedal, Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale bound-constrained optimization, ACM Trans. Math. Softw. 23 (4) (1997) 550–560.doi:10.1145/279232.279236

    1997

  11. [19]

    Homola, et al., Cosmic Ray Extremely Distributed Observatory, Symmetry 12 (11) (2020) 1835.arXiv:2010.08351,doi:10.3390/ sym12111835

    P. Homola, et al., Cosmic Ray Extremely Distributed Observatory, Symmetry 12 (11) (2020) 1835.arXiv:2010.08351,doi:10.3390/ sym12111835

    2020 arXiv

  12. [20]

    G.D.Martinez,J.McKay,B.Farmer,P.Scott,E.Roebber,A.Putze,J.Conrad,ComparisonofstatisticalsamplingmethodswithScannerBit, the GAMBIT scanning module, Eur. Phys. J. C 77 (11) (2017) 761.arXiv:1705.07959,doi:10.1140/epjc/s10052-017-5274-y. 13 □1.0 □0.8 □0.6 □0.4 □0.2 0.0 cos(θ) 2.5...

    2017 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.