{"id":"82dc5b64-3e1a-4bb0-96a7-ed23cb15e2d8","arxiv_id":"1908.05181","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SkyLLH is a Python package that generalizes telescope-specific unbinned log-likelihood analyses into a modular, reusable framework for multi-messenger astronomy.","lead":"A new open-source Python framework named SkyLLH helps researchers build unbinned log-likelihood ratio tests for finding astrophysical neutrino and gamma-ray sources. If it works as described, it could standardize and speed up common multi-messenger astronomy analyses across different telescopes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No unit tests or validation are shown; a numerical bug in the core likelihood or parameter mapping would invalidate every analysis, so the central claim is not yet supported.","rationale":"The reader's weakest_assumption is exactly the premise I find most load-bearing: the paper's central claim is about a working, general tool, but no evidence is provided that the implemented code computes the likelihood correctly. The equations in Section 1 are standard, and the class structure in Sections 2 through 5 is plausible, but the paper stops short of demonstrating that the software realizes those equations. A math error in the paper would not tell us whether the code is buggy; conversely, even a correct code outline could contain an implementation slip. This is a software paper, so the code itself is the result, and validation is the natural substitute for a mathematical proof. I also considered the statistic of Eq. (1.4) as a possible issue, but it is background material and the paper explicitly uses trial generation for frequentist properties, so it is not what the tool's correctness rests on. The lack of a second detector module is a related weakness for the 'telescope-independent' phrasing, but it is secondary to numerical correctness: if the core math is wrong, even IceCube analyses fail; if the core math is right, adding a detector module can be tested separately. The concrete test I propose is minimal and decisive: comparing SkyLLH's likelihood against an independent, directly implemented Eq. (1.2) on a synthetic dataset would either reveal a numerical bug or clear the most serious doubt. Until such a check is reported, the conditional verdict is appropriate, and my reading does not move it.","tokens_in":5279,"tokens_out":6425,"duration_ms":70176,"concrete_test":"Run an independent numerical check of the core likelihood: construct a synthetic dataset of N events from a known two-component model (e.g., a Gaussian signal plus uniform background), evaluate the SkyLLH log-likelihood via the high-level analysis API for a given ns and source parameters, and compare it to a direct implementation of Eq. (1.2) in plain numpy/scipy that does not use SkyLLH. The check should include ns=0, where the result must equal sum_i log B(D_i) to machine precision. If the difference exceeds 1e-6 relative, the central numerical premise fails; if it passes, the concern about correctness is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2 states that core.PDF.get_prob returns a probability for each event, and Sections 4 and 5 present TrialDataManager and ModelParameterMapper as the mechanisms that make likelihood evaluation correct and efficient. The paper, however, contains no test suite, no validation against analytic likelihood values, and no benchmark. The central claim that users can build reliable unbinned log-likelihood-ratio analyses on this framework depends entirely on these routines being numerically correct. Even a single error -- for example, a wrong normalization in a signal PDF or an incorrect mapping when several local spectral indices share one global parameter -- would silently corrupt every fitted ns and test statistic. In addition, the abstract's statement that SkyLLH 'can be applied' to Fermi-LAT-type data is not backed by any second detector implementation in the paper; Section 6 lists only the IceCube i3 module and refers to other messengers as 'possible future applications'. The architecture may make such an extension straightforward, but the claimed telescope-independence is currently unexercised. This is not an accusation of a bug; it is an observation that the paper as written does not supply the evidence needed to accept that the tool performs as advertised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SkyLLH, an open-source Python framework for constructing and evaluating unbinned log-likelihood-ratio tests in multi-messenger astronomy. It states the standard likelihood-ratio formalism (Eqs. 1.1-1.4), describes a modular class architecture organized into core, physics, plotting, and detector-specific modules, and discusses source-hypothesis groups, data management via Dataset and DataFieldArray, trial-data handling through TrialDataManager, and flexible global-to-local parameter mapping via ModelParameterMapper. The central claim is that SkyLLH provides an easy-to-use, efficient, and telescope-independent way to define and run such analyses, currently used for IceCube and intended for Fermi-LAT and other experiments. The paper contains no tests, benchmarks, worked examples, or a second-detector implementation, so the advertised functionality rests entirely on design descriptions.","tokens_in":5466,"tokens_out":2976,"duration_ms":29881,"significance":"If the claims are substantiated, SkyLLH would be a useful contribution: a GPLv3-licensed, dependency-light (numpy, scipy, astropy) framework that maps mathematical objects directly onto classes, supports flexible shared or independent fit parameters, and could reduce duplicate analysis code across experiments. The statistical formulas are standard and appear correctly stated, and there is no circularity in the presentation. The main value would be as a community tool, but the paper as written does not provide the evidence needed to accept that the tool actually performs as advertised: no unit tests, no validation against analytic likelihood values, no runtime measurements, and no example analysis are included.","major_comments":[{"comment":"The central claim that users can build reliable unbinned log-likelihood-ratio analyses on SkyLLH is not supported by any verification evidence. The paper describes core.PDF.get_prob, TrialDataManager, and ModelParameterMapper as the mechanisms that make likelihood evaluation correct and efficient, but it shows no unit tests, no validation against known likelihood values, no comparison with existing codes, and no end-to-end analysis example. A numerical bug in PDF normalization or in the mapping of global to local parameters -- for instance, when several sources share one spectral index -- would silently corrupt fitted ns values and test statistics. I am not claiming that such a bug exists, only that the manuscript currently supplies no reason to trust that it does not.","section":"Sections 2, 4, and 5"},{"comment":"The paper asserts that storing data as one-dimensional numpy arrays in DataFieldArray reduces memory cache misses and thereby causes \"faster data access and shorter program execution times,\" but no benchmark or profiling data is presented. Since the abstract explicitly promises that SkyLLH executes likelihood functions \"efficiently ... with high performance,\" this performance claim is load-bearing and needs at least a simple timing comparison against structured arrays or an alternative implementation.","section":"Section 4, DataFieldArray"},{"comment":"The abstract states that SkyLLH \"can be applied\" to data from experiments such as IceCube and Fermi-LAT, but Section 6 lists only the IceCube i3 module as implemented and describes ANTARES/KM3NeT and Fermi-LAT as \"possible future applications.\" The claimed telescope independence is therefore unexercised: no second detector implementation, no interface demonstration, and no cross-experiment example are shown. This is not a fatal flaw in the architecture, but the generality claim should be reworded or supported by at least one additional detector specialization.","section":"Abstract and Section 6"}],"minor_comments":[{"comment":"In the text following Eq. (3.4), \"spectrial index\" should be \"spectral index,\" and \"choosen\" should be \"chosen.\"","section":"Section 3"},{"comment":"In the TrialDataManager paragraph, \"the trail data equals the experimental data\" should read \"the trial data,\" and \"reproduce-ability\" would be better as \"reproducibility.\"","section":"Section 4"},{"comment":"The phrase \"The general enfolded properties of a data set\" is unclear; if \"enfolded\" is intended to mean \"built-in\" or \"intrinsic,\" the wording should be revised.","section":"Section 4"},{"comment":"\"super-position\" should be \"superposition,\" and the sentence about the test statistic distribution would read more clearly if it explicitly stated that Eq. (1.4) applies when ns is fixed to zero under the null hypothesis while other parameters remain free.","section":"Section 1, Eq. (1.4)"},{"comment":"Since the paper is about open-source software, giving a version number, a DOI or specific commit reference, and the location of the test suite would materially improve reproducibility for readers who want to verify the claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a proceedings-style software description rather than a full software paper. The main gap is evidentiary: no tests, benchmarks, or second-detector demonstrations. The authors are likely capable of supplying these, and the architecture described is plausible, so rejection seems disproportionate. If the journal's scope includes tools with demonstrated validation, I would encourage the editor to request the missing artifacts and re-review. I would also suggest checking whether the repository actually contains the claimed i3 module and whether the code is importable by third parties, since the paper does not demonstrate either."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Martin,\n\nQuick read on 1908.05181. It is exactly what it says: a short software design report for SkyLLH, the IceCube group's Python framework for unbinned log-likelihood ratio tests. The statistics in Section 1 are standard and correctly stated — the two-component likelihood, the boundary-corrected test statistic distribution citing Cowan et al. No red flags there. What is genuinely new is the architecture: a plugin-style separation of flux models, datasets, trial data, and parameter mapping, so you can define a source stack, share a spectral index across sources or keep them free, and generate pseudo-data trials without rewriting the whole analysis. That is useful. The ModelParameterMapper and TrialDataManager ideas are sensible, and the paper is honest that only the IceCube i3 module exists so far.\n\nThe soft spot is exactly where the stress-test note points: there is no evidence in the paper that the implementation actually works as advertised. No unit tests, no validation against analytic likelihood values, no runtime comparisons, no second detector. The abstract says SkyLLH 'can be applied' to Fermi-LAT type data, but Section 6 lists that only as a possible future application. That is a mismatch between the abstract and the body. The claim isn't false — it's unsubstantiated. For a one-off conference proceedings that might be tolerable, but if this were a journal submission, a referee would need to see at least a minimal test suite and a benchmark on a real or synthetic dataset before the performance and reliability claims could be evaluated.\n\nThere is no circularity or invented entity here; the equations are borrowed correctly, and the code is on GitHub, so at least the claims are checkable in principle. My bottom line: this is a genuinely useful design document and the code base is worth knowing about, but the paper itself is not evidence of a validated tool. I'd send it to peer review — a serious referee can push for the missing validation — and I'd bring it up in reading group for people working on MMA likelihood methods. But I wouldn't cite it as evidence that SkyLLH is fast or correct; I'd cite it as the reference for the software's existence and design.\n\nRecommendation: engage with it as a design report, ask the authors for the test suite and benchmarks, and don't desk-reject.","headline":"SkyLLH is a cleanly written design report for a useful open-source likelihood framework, but the performance and generality claims are not backed by tests or benchmarks, so treat it as a design proposal rather than a validated tool.","tokens_in":5960,"tokens_out":2232,"would_cite":true,"duration_ms":22479,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkyLLH aims to make unbinned likelihood ratio tests a single telescope-independent Python framework.","keywords":["SkyLLH","unbinned log-likelihood","likelihood ratio test","multi-messenger astronomy","Python framework","neutrino astronomy","gamma-ray astronomy","event data analysis"],"falsifier":"Take a toy data set of, say, ten events with known signal and background densities, compute the log-likelihood and best-fit parameters by direct evaluation of equation (1.2), then run the same setup through SkyLLH; any nonzero difference in the log-likelihood value at the same parameters, or a different maximizer, would show the framework's evaluation is not correct as claimed.","tokens_in":5084,"feed_emoji":"🔭","tokens_out":8197,"duration_ms":77883,"temperature":0.7,"pith_summary":"This paper presents SkyLLH, a Python framework meant to let researchers define, evaluate, and test unbinned log-likelihood ratio statistics for any experiment that records events with celestial coordinates. The central claim is telescope independence: by tying every software class to a mathematical object in the likelihood formalism, the same code can serve neutrino and gamma-ray searches rather than requiring a separate implementation for each detector. If that claim holds, analysis teams can build a likelihood test by composing flux models, data sets, trial data, and fit parameters, then run pseudo-experiment trials to estimate sensitivity and discovery potential in a frequentist framework. The paper's value is therefore architectural: it offers a reusable, high-performance layer for the standard likelihood-ratio tests of multi-messenger astronomy.","feed_headline":"SkyLLH puts unbinned likelihood tests in one telescope-independent tool","feed_subtitle":"Neutrino and gamma-ray experiments could share one code base for their likelihood analyses instead of rewriting per detector.","key_machinery":"The load-bearing object is the modular Python class hierarchy that mirrors the likelihood-ratio formalism. The central identity is the two-component unbinned likelihood $L(n_s,\\vec\\theta_s|\\vec D)=\\prod_{i=1}^N[\\frac{n_s}{N}S(D_i|\\vec\\theta_s)+(1-\\frac{n_s}{N})B(D_i)]$, with the likelihood-ratio statistic of equation (1.1) as the target. SkyLLH carries the argument through a small set of replaceable components: a core.PDF interface whose get_prob returns event probabilities; a FactorizedFluxModel that separates spatial, energy, and time profiles; source hypothesis groups that let sources sharing one flux model share calculations; a columnar DataFieldArray; a TrialDataManager with dependency tracking so only affected fields are recomputed during maximization; and a ModelParameterMapper that binds global fit parameters to local model parameters. The class structure is what does the work: it turns every analytic ingredient of the LRT into a replaceable component, which is what gives the design its telescope independence.","core_discovery":"The central claim is that the mathematical structure of an unbinned two-component likelihood-ratio test can be mapped one-to-one onto an object-oriented class hierarchy. The framework starts from the likelihood $L(n_s,\\vec\\theta_s|\\vec D)=\\prod_{i=1}^N\\left[\\frac{n_s}{N}S(D_i|\\vec\\theta_s)+(1-\\frac{n_s}{N})B(D_i)\\right]$ and the log-likelihood-ratio test statistic of equation (1.1), then provides abstract classes for each ingredient: PDFs with a get_prob method, flux models with spatial, energy, and time profiles, source hypothesis groups that share computational work, a data field array that stores event fields column-wise, a trial-data manager that recalculates only fields whose dependencies changed, and a parameter mapper that lets one global fit parameter control local parameters in several sources. Because each piece is an abstract interface, a new experiment contributes only detector-specific implementations, and a new physics model contributes only a new flux profile. The paper reports active use for neutrino point-source searches and positions SkyLLH for future stacking and combined-messenger analyses.","pith_inferences":["A natural, testable extension is to pin SkyLLH's log-likelihood values on a small toy data set to brute-force computation and keep that pair as a regression test, converting the design claim into a verified numerical one.","The dependency-tracked trial-data manager suggests an optimization whose size the paper does not quantify: profiling how many fields are skipped during a typical fit would show how much of the claimed performance gain is real.","Because flux profiles are pluggable, the same framework could host transient or extended-source likelihoods by swapping only spatial and time profiles, opening it to time-domain multi-messenger alerts without touching the likelihood machinery."],"forward_implications":["A new experiment supplies only detector-specific file loaders and PDF implementations; all likelihood, trial, and parameter machinery is reused unchanged.","Stacking analyses of many point-like sources can share or split spectral indices per source through the parameter mapper, without rewriting the likelihood.","Pseudo-data trials for sensitivity and discovery potential become a routine framework service rather than a per-analysis custom script.","Combined multi-messenger fits, such as neutrino and gamma-ray events with separate signal PDFs but shared source hypotheses, can be assembled from the same modular pieces.","Column-wise data storage and dependency-tracked trial fields should reduce memory traffic and redundant recalculations compared with structured-row event records."],"supporting_citations":[{"why":"Establishes Wilks's theorem result that the likelihood-ratio test statistic approaches a chi-squared distribution away from parameter bounds, which justifies the test-statistic framework SkyLLH evaluates.","marker":"[1]"},{"why":"Supplies the boundary-corrected test-statistic distribution for a null hypothesis at $n_s=0$, namely a delta function plus a chi-squared distribution, used by SkyLLH's two-component likelihood model.","marker":"[2]"},{"why":"Describes the cubic-kilometre neutrino detector whose data handling motivates the detector-specific module of the framework.","marker":"[3]"}],"fun_headline_variants":["SkyLLH: One Python tool for log-likelihood in multi-messenger astronomy","SkyLLH turns unbinned likelihood tests into reusable Python classes","Telescope-independent log-likelihood: SkyLLH for multi-messenger","SkyLLH: Generalized likelihood analysis for neutrino and gamma-ray data","SkyLLH: one likelihood framework for all telescopes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire framework works only if the Python code computes the likelihood, trial data, and parameter mappings exactly as the mathematical equations say; the paper shows class designs and equations but no unit tests or benchmark outputs verifying that correctness.","fun_headline_variants_meta":{"raw":{"variants":["SkyLLH: One Python tool for log-likelihood in multi-messenger astronomy","SkyLLH turns unbinned likelihood tests into reusable Python classes","Telescope-independent log-likelihood: SkyLLH for multi-messenger","SkyLLH: Generalized likelihood analysis for neutrino and gamma-ray data","SkyLLH: one likelihood framework for all telescopes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000566,"raw_usage":{"total_tokens":2667,"prompt_tokens":916,"completion_tokens":1751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":1655}},"tokens_in":532,"tokens_out":1751,"duration_ms":12834,"temperature":1.0,"reasoning_tokens":1655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:20:32.331057+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a toy data set of, say, ten events with known signal and background densities, compute the log-likelihood and best-fit parameters by direct evaluation of equation (1.2), then run the same setup through SkyLLH; any nonzero difference in the log-likelihood value at the same parameters, or a different maximizer, would show the framework's evaluation is not correct as claimed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the boundary-corrected test-statistic distribution for a null hypothesis at $n_s=0$, namely a delta function plus a chi-squared distribution, used by SkyLLH's two-component likelihood model."},{"cited_title":"Cowan, K","cited_arxiv_id":null,"evidence_quote":"Describes the cubic-kilometre neutrino detector whose data handling motivates the detector-specific module of the framework."}],"review_version":1}