{"id":"020c6d00-5a24-49f4-9e33-0ef6a62d4f30","arxiv_id":"2607.12853","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A hash-augmented adaptive multilevel splitting Monte Carlo algorithm accurately estimates arbitrarily small two-sample permutation-test p-values, with a Python package for user-defined statistics.","lead":"The authors introduce a hash-augmented adaptive multilevel splitting Monte Carlo method that estimates very small p-values for two-sample permutation tests more accurately than plain sampling. Practitioners who need reliable tiny p-values for custom statistics or multiple-testing correction may find the approach and the accompanying hamstest package useful.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the abstract-only information limit already flagged by the reader.","rationale":"The review is constrained to the abstract. The central claim is a practical algorithmic contribution whose correctness rests on (i) the multilevel-splitting estimator remaining unbiased under the discrete support of a general permutation statistic and (ii) the hash-augmentation correctly eliminating the usual Monte-Carlo under-sampling of rare events. Both points are asserted for KS and Mann–Whitney and claimed to extend to user-defined statistics via the package. No equations, proofs, or numerical tables are present, so no concrete technical flaw (e.g., an incorrect normalization, an unaccounted-for dependence induced by hashing, or a coverage failure of the CIs) can be exhibited. The reader's identification of the generalization-to-arbitrary-statistics premise is therefore the only load-bearing concern that can be stated; it does not warrant a change of the already-UNVERDICTED status. A single verification step once the full artifacts appear is the appropriate next action.","tokens_in":2011,"tokens_out":476,"duration_ms":5164,"concrete_test":"Once the full manuscript and hamstest source become available, re-run the exact-vs-Monte-Carlo comparison of Section 4 (or equivalent) on at least one additional custom statistic whose distribution is known to be highly discrete (e.g., a two-sample median-difference or a contingency-table statistic with many ties); if the reported p-value point estimates or CI coverage deviate systematically from the exact values, the generalization claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption correctly isolates the only load-bearing premise that can be assessed from the abstract: that hash-augmentation plus adaptive multilevel splitting correctly handles discreteness of the test-statistic distribution for arbitrary user-defined statistics, not merely the two named examples (KS and Mann–Whitney) that were validated against an exact algorithm. Because the full text, theorems, experiments, and package source are unavailable, no deeper internal inconsistency, hidden bias, or failure of the confidence-interval construction can be verified or refuted. The abstract itself already acknowledges the discreteness issue and claims a remedy; without the paper body there is simply nothing further to stress-test.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The manuscript proposes a hash-augmented adaptive multilevel splitting Monte Carlo algorithm for estimating arbitrarily small p-values in two-sample permutation tests. It highlights pitfalls arising from discreteness of the test-statistic distribution (illustrated on Kolmogorov–Smirnov and Mann–Whitney U), claims that the proposed remedy yields accurate point estimates and valid confidence intervals by comparison with an exact algorithm, and supplies a Python package (hamstest) that accepts user-defined statistics.","tokens_in":2129,"tokens_out":581,"duration_ms":15366,"significance":"If the claims hold, the work would supply a practical, general-purpose tool for reliable small-p-value estimation in nonparametric two-sample settings where exact enumeration is infeasible—directly relevant to multiple-testing correction and custom test statistics. Explicit treatment of discreteness and release of an open-source package are concrete strengths that would increase utility to practitioners, provided the accuracy and CI validity claims are substantiated beyond the two named examples.","major_comments":[{"comment":"The abstract asserts accuracy of p-value estimates and validity of associated confidence intervals by comparison with an exact algorithm for the Kolmogorov–Smirnov and Mann–Whitney U tests. With only the abstract available, no theorems, error analyses, simulation tables, or proofs can be examined; the central empirical claim is therefore unverifiable from the material in hand and remains load-bearing for acceptance.","section":"Abstract"},{"comment":"The claim that the method extends to arbitrary user-defined statistics rests on the premise that hash augmentation correctly resolves discreteness of the test-statistic distribution without distorting the target tail probability or the confidence intervals. The abstract reports validation only for the two named tests; generalization is therefore unsupported by the available text and constitutes a load-bearing assumption that requires either theoretical guarantees or broader empirical evidence.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract alludes to free parameters of the adaptive multilevel splitting schedule and of the hash function but does not indicate whether sensitivity analyses, default recommendations, or diagnostics for users of hamstest are supplied.","section":"Abstract"},{"comment":"No references to prior multilevel-splitting or rare-event Monte Carlo literature appear in the abstract; the full manuscript should situate the hash-augmentation contribution relative to existing AMS variants.","section":"Abstract"}],"recommendation":"uncertain","confidential_remarks":"Only the abstract was supplied for this review. A definitive recommendation (accept / minor / major / reject) cannot be issued without the full methods, theoretical justification of the CIs, numerical results, and package documentation. I recommend obtaining the complete manuscript before an editorial decision."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The one thing to know is that this is a practical methods paper: hash-augmented adaptive multilevel splitting specialized to two-sample permutation p-values, aimed at getting accurate estimates of very small p-values (and CIs) when exact enumeration is impossible. They ship a Python package (hamstest) that takes a user-defined statistic. That combination is the real product.\n\nWhat is new is the packaging of AMS with a hash step to deal with the discrete support of permutation statistics, plus explicit attention to the discreteness pitfalls that break naive rare-event sampling. They check the method against an exact algorithm on KS and Mann–Whitney and claim both point estimates and confidence intervals look right. For people who need custom test statistics under multiple-testing regimes, that is useful engineering.\n\nThe soft spots are exactly what you would expect from an abstract-only read. We cannot see the theorems, the level schedule, the hash construction, or the simulation tables. The load-bearing claim is that the same discreteness handling works for arbitrary user statistics, not just the two classical ones they validated. Free parameters (AMS intermediate levels, hash details) are present; whether they introduce bias or only affect variance is not visible here. None of that is a red flag yet—it is simply un-audited.\n\nThis is for computational statisticians and people who write custom nonparametric tests and then hit the wall of tiny p-values. It is not a conceptual shift; it is tooling. I would send it to a serious referee rather than desk-reject: the problem is real, the approach is standard rare-event Monte Carlo applied carefully, and a package is promised. If the full paper has the comparisons and the code, it is worth the referee time. If the generalization section is thin, that is a revision item, not a reason to kill it.\n\nBottom line: engage if the full text and package materialize; treat the abstract claims as provisional until then.","headline":"Solid applied Monte Carlo tooling for small permutation p-values; abstract-only so the generalization claim is still un-audited.","tokens_in":2741,"tokens_out":487,"would_cite":false,"duration_ms":10720,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G10","65C05"],"pacs":[],"model":"grok-4.5","headline":"Hash-augmented adaptive multilevel splitting Monte Carlo accurately estimates arbitrarily small two-sample permutation p-values with valid confidence intervals.","keywords":["permutation test","Monte Carlo","multilevel splitting","p-value estimation","Kolmogorov–Smirnov","Mann–Whitney U","hash augmentation","hamstest"],"falsifier":"Run the algorithm on a custom discrete statistic whose exact permutation p-value is known by exhaustive enumeration; if the reported point estimate falls systematically outside the claimed confidence interval, or the empirical coverage of the intervals is far from nominal, the central claim fails.","tokens_in":2877,"feed_emoji":"🎲","tokens_out":748,"duration_ms":6424,"temperature":0.7,"pith_summary":"Two-sample permutation tests are popular because they need no distributional assumptions, yet exact p-values become intractable for complex or custom statistics and ordinary Monte Carlo sampling loses relative accuracy precisely when the p-value is small. The paper introduces a hash-augmented adaptive multilevel splitting Monte Carlo scheme that recovers accurate point estimates and valid confidence intervals for arbitrarily small p-values. Using the classical Kolmogorov–Smirnov and Mann–Whitney U statistics as concrete cases, it shows how discreteness of the permutation distribution can bias naïve estimators and how the proposed machinery removes that bias. Validation against an exact algorithm confirms both accuracy and interval coverage. The method is released as the open Python package hamstest so that any user-defined test statistic can be analysed under the same guarantees.","feed_headline":"Monte Carlo now nails tiny permutation p-values with confidence","feed_subtitle":"Hash-augmented multilevel splitting recovers accurate estimates and intervals for any two-sample statistic","key_machinery":"Hash-augmented adaptive multilevel splitting: successive levels of rare-event sampling are combined with a hash function that correctly accounts for the discrete support of the permutation distribution, thereby producing unbiased p-value estimates and properly calibrated confidence intervals.","core_discovery":"A hash-augmented adaptive multilevel splitting Monte Carlo algorithm yields accurate estimates of arbitrarily small p-values for two-sample permutation tests, together with valid confidence intervals, even when the test statistic is discrete; the procedure is validated on Kolmogorov–Smirnov and Mann–Whitney U statistics against an exact reference and is available for arbitrary user-defined statistics via the hamstest package.","pith_inferences":["The same multilevel-splitting-plus-hash idea could be transferred to one-sample, multi-sample or paired permutation tests with only modest redesign of the proposal kernel.","Because the method scales to extreme rarity, it may become the default engine for genome-wide or high-throughput screening pipelines that currently rely on asymptotic approximations.","Empirical coverage studies on a broader suite of discontinuous statistics would strengthen the claim of generality beyond KS and Mann–Whitney."],"forward_implications":["Arbitrarily small permutation p-values can be estimated without exhaustive enumeration, removing a common barrier to multiple-testing correction.","Any user-defined two-sample statistic can be analysed with the same Monte-Carlo guarantees via the hamstest package.","Discreteness-induced bias that previously plagued small-p Monte-Carlo estimates is systematically eliminated.","Confidence intervals supplied by the method can be used directly for rigorous significance statements."],"fun_headline_variants":["Hash-augmented multilevel splitting estimates tiny permutation p-values accurately","Adaptive AMS Monte Carlo yields precise small two-sample p-value estimates","Hash-augmented splitting recovers valid confidence intervals for discrete tests","Multilevel Monte Carlo for accurate arbitrary-statistic permutation p-values","hamstest enables reliable estimation of tiny two-sample permutation p-values"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"That the same hash-augmentation and multilevel-splitting corrections remain unbiased for completely general user-defined statistics, not only for the two classical tests that were checked against an exact algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Hash-augmented multilevel splitting estimates tiny permutation p-values accurately","Adaptive AMS Monte Carlo yields precise small two-sample p-value estimates","Hash-augmented splitting recovers valid confidence intervals for discrete tests","Multilevel Monte Carlo for accurate arbitrary-statistic permutation p-values","hamstest enables reliable estimation of tiny two-sample permutation p-values"]},"model":"grok-4.5","effort":"low","cost_usd":0.007842,"raw_usage":{"total_tokens":1861,"prompt_tokens":735,"num_sources_used":0,"completion_tokens":94,"cost_in_usd_ticks":78420000,"prompt_tokens_details":{"text_tokens":735,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1032,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":735,"tokens_out":94,"duration_ms":9293,"temperature":1.0,"reasoning_tokens":1032,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T02:50:34.577401+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the algorithm on a custom discrete statistic whose exact permutation p-value is known by exhaustive enumeration; if the reported point estimate falls systematically outside the claimed confidence interval, or the empirical coverage of the intervals is far from nominal, the central claim fails.","supporting_citations":[],"review_version":1}