{"id":"9f5aafaa-fc85-46fd-8748-ac45def6ee55","arxiv_id":"2608.06621","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A hypothesis is certified only if both a significance e-value and a mechanism e-value pass, using the data itself as the conformal null.","lead":"NxN E-valuation is a proposed algorithm that certifies machine-generated hypotheses by turning a large dataset itself into the null hypothesis, using two e-value tests and a diversity check. It aims to replace circular LLM self-verification and naive held-out testing for hypotheses that can be expressed as per-sample restrictions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Diversity threshold is label-calibrated and Section B.3 admits a non-constant confound can have D=0; the universal, automatic confound-rejection claim is unsupported.","rationale":"I read the paper as trying to certify machine-proposed per-unit hypotheses without a case-specific null, and to replace held-out testing for a class of LLM-generated hypotheses. The e-value core (Propositions 1–3, Theorem 1) is sound for a single pre-specified hypothesis and a fixed decision threshold: E_sig and E_mech are valid e-values under their exchangeability nulls, and the intersection–union bound with no multiplicity correction is correct for that narrow per-hypothesis statement. The empirical demonstration, however, is the load-bearing part of the broad claim, and it depends on the diversity threshold τ_div. The paper's own Section A says τ_div was calibrated using ground-truth labels ('midpoint between a known glocal and a known confound'). The confound row in Table 3 (diversity 0.28) is rejected only because the threshold is set below 0.28; a slightly higher unlabeled threshold certifies it as glocal. The paper also explicitly acknowledges in Section B.3 that D=0 does not imply a constant restriction: a non-constant confound can have D=0 if the outcomes in the sample do not exercise the differences. That is not a minor caveat; it is an identifiability failure in the exact regime Theorem 2 claims to resolve. The per-hypothesis Type-I guarantee in Theorem 1 remains true for the unit-specific branch, but it does not cover the glocal/confound routing, which is where the headline empirical win comes from. The reader's weakest_assumption identified the label-calibrated threshold; my concern extends it with the B.3 non-identifiability corner, so I marked agreement as partial. The right verdict remains rejection (no change from the reader) because neither the central 'automatic' claim nor the confound-rejection result is supported by a label-free procedure or a distribution-free guarantee.","tokens_in":13306,"tokens_out":11776,"duration_ms":108801,"concrete_test":"Run a controlled re-evaluation of the Section 5 generator with two modifications: (i) set τ_div by a pre-registered, label-free rule (e.g., τ_div=0.1 or the binomial bound of Section B.5) instead of the ground-truth midpoint, and (ii) add an adversarial structural confound whose per-unit restrictions differ only on items outside the observed outcome universe, so D=0 exactly. If the confound is certified as glocal, or if the reject/certify decision for the original confound changes with any non-label-calibrated τ, the universal/automatic claim is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that N×N E-valuation is a general, automatic, distribution-free replacement for held-out testing rests on the diversity routing in Section 3.5: when E_sig >> 1 and E_mech ≈ 1, the method rejects a structural confound only if diversity exceeds τ_div and certifies a unit-invariant hypothesis only if diversity ≤ τ_div. Two problems make this load-bearing. First, τ_div is not set by any data-only procedure. Section A calibrates it 'per world as the midpoint between a known glocal (diversity 0) and a known confound,' i.e., using ground-truth labels; Section 3.4 states it 'carries no Type-I guarantee.' In Table 3, the confound has diversity 0.28 and the glocal has 0.00, so the label-calibrated midpoint 0.14 is exactly what produces the 100% rejection; any threshold above 0.28 would certify the confound as glocal. Second, Theorem 2's separation is not distribution-free even with an oracle threshold. The theorem requires 'the observed outcomes distinguish distinct restrictions,' and Appendix B.3 explicitly concedes that a non-constant structural confound can have D=0 when the per-unit restrictions differ only on items absent from the observed outcome set, so it is read as glocal. This is an admitted non-identifiability corner, not a tuning detail. Together these mean the method's only defense against structural confounds has no label-free threshold and no finite-sample guarantee, so the 'universally better replacement' and 'automatic certifier' claims are unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes N×N E-valuation, an e-value-based certifier for per-unit hypotheses. For a hypothesis H with scope U, it builds an N×N cross-prediction matrix M[i][j] = r_i(y_j), aggregates per-unit significance e-values E_sig (diagonal vs. population base rate) and mechanism e-values E_mech (diagonal vs. in-scope row mean), and computes a diversity statistic D. The decision rule (Eq. 7) certifies a unit-specific hypothesis when both e-values exceed 1/α, certifies a glocal hypothesis when E_sig ≥ 1/α with small diversity, flags global-prevalence, and rejects otherwise. Propositions 1–3 establish exact e-value validity and dependence-robust aggregation; Theorem 1 gives an intersection–union error bound; Theorem 2 claims that D separates glocal effects from structural confounds. A synthetic recommendation world with planted ground truth reports 99% correct verdicts for the proposed method versus 50% for naive held-out validation.","tokens_in":13500,"tokens_out":7556,"duration_ms":67222,"significance":"If the claims held, the method would be a practical drop-in replacement for LLM circular verification and held-out testing for a useful class of per-unit hypotheses: it needs no case-specific null, uses only the data as a null, and the e-value formulation gives a natural way to accumulate evidence across independent banks. The formal e-value results (Propositions 1–3) are correct as stated for a fixed, pre-specified hypothesis, and the paper is unusually candid about the limits of D in Section B. The central weakness is that the method's empirical and conceptual force rests on the diversity routing threshold τ_div, which is calibrated using ground-truth labels in the experiments (Section A), carries no Type-I guarantee (Section 3.4), and is not given any label-free estimation procedure. Combined with the admitted non-identifiability in Section B.3, this undercuts the claim of a 'general, automatic, distribution-free certifier' and the abstract's 'universally better replacement' wording.","major_comments":[{"comment":"The routing threshold τ_div is load-bearing and is calibrated using ground-truth labels: Section A states 'τ_div is calibrated per world as the midpoint between a known glocal (diversity 0) and a known confound.' Because Eq. (7) routes a hypothesis with E_sig ≫ 1 and E_mech ≈ 1 to 'glocal' when diversity ≤ τ_div and to 'reject' when diversity > τ_div, the Table 3 results (e.g., confound diversity 0.28 rejected, seasonal diversity 0.00 certified) are partly a fitted outcome rather than an out-of-sample prediction. The paper gives no data-only procedure for setting τ_div, and Section 3.4 explicitly states that the threshold 'carries no Type-I guarantee.' A paper claiming to be a general, automatic, distribution-free certifier must either provide a principled, label-free calibration method or substantially reframe the experimental verdicts as an oracle-calibrated upper bound.","section":"Section A, Eq. (7)"},{"comment":"Theorem 2's separation of glocal from structural confound holds only 'provided the observed outcomes distinguish distinct restrictions,' and Section B.3 concedes a concrete failure: two per-unit restrictions differing only on items absent from the observed outcome set {y_j}_{j∈U} yield σ_j = 0 for every j, hence D = 0, and such a hypothesis is read as glocal. This is an admitted non-identifiability, not a tuning detail. It means the method cannot universally separate 'predicts for the right reason' from a structural confound, and the abstract's 'universally better replacement for at least LLM circular verification and held-out-data testing' overstates the scope. The authors should either relax the claim or quantify how often the distinguishing-outcome condition holds for realistic data-generating processes.","section":"Section B.3 / Theorem 2"},{"comment":"The reject branch for structural confounds rests entirely on the diversity statistic, which 'is not an e-value' and 'carries no Type-I guarantee' (Section 3.4). Consequently, false rejection of a genuine unit-specific hypothesis caused by a routing error is not controlled by any of the paper's theorems. If the procedure is presented as a certifier, the user needs a statement of what, if anything, the reject decision guarantees; as written, the decision rule is a heuristic overlay on valid e-value tests, and the headline '99% correct verdicts' conflates the valid e-value machinery with the heuristically calibrated routing.","section":"Section 3.4, Section 3.5"},{"comment":"The anytime-valid composition in Algorithm 2 multiplies per-bank e-values and relies on 'independent banks,' but the paper does not explain how this remains valid when hypotheses are proposed, refined, and re-tested after seeing earlier banks, which is exactly the 'multi-round, data-dependent testing' setting advertised in Section 1. Without a selection-effect correction or an explicit restriction to pre-registered hypotheses, the multi-round, anytime-valid claim is not supported by the stated assumptions.","section":"Section 3.7 / Algorithm 2"}],"minor_comments":[{"comment":"The column header 'decision ours base' is ambiguous; the paper should add a note that '✓' means correct verdict and '×' means wrong verdict, since the baseline column shows × for the fakes even though the text says the baseline 'certifies everything predictive.'","section":"Table 3"},{"comment":"The notation g(rj)_0 has a subscript '0' that is never defined; the paper should explicitly say that this is the population base-rate null baseline for the restriction r_j.","section":"Eq. (1)"},{"comment":"The experiments do not include sensitivity analyses for p_follow = 0.9, the matrix cap at |U| = 400, or the choice of τ_div; a small sensitivity table would help readers judge how robust the 'correct verdict' rates are to the oracle-calibrated threshold.","section":"Section A"},{"comment":"The OneRec Team technical report is cited with no author names; the entry should be completed or the citation format should be consistent with other technical reports.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound e-value core but the empirical and conceptual claims go beyond what the current text supports. The tau_div calibration is the main verification concern: for a venue that cares about out-of-sample claims, using ground-truth labels to set the key routing constant is not acceptable. I would ask the authors to supply a data-only calibration procedure or to explicitly relabel the experiments as oracle-calibrated, and to temper the 'universally better replacement' claim in the abstract and conclusion. The paper may be acceptable after these changes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is the N×N cross-prediction matrix: applying each unit's restriction to every other unit's outcome and reading off a conformal mechanism e-value from the diagonal-versus-row-mean contrast. That is a clean way to turn the dataset itself into a CRT null, and the theoretical core—Propositions 1–3 and Theorem 1—is correct for a single pre-specified hypothesis. The intersection–union argument is standard but properly applied. Credit where it is due: the authors know e-value theory and the conformal normalization is done right.\n\nThe soft spots are where the paper overreaches. The diversity statistic τ_div is the load-bearing separator between glocal effects and structural confounds, and it is calibrated in the experiments as the midpoint between a known glocal (D=0) and a known confound, i.e., using ground-truth labels. Section 3.4 admits it carries no Type-I guarantee. So the headline 100% rejection of the confound in Table 3 is partly a fitted value, not an out-of-sample prediction. There is no data-only procedure for setting τ_div, and Appendix B.3 concedes a non-constant confound can have D=0 when restrictions differ only on items absent from the observed outcomes—an identifiability corner, not a tuning detail.\n\nThe anytime-validity claim is also oversold. Algorithm 2 multiplies e-values across banks, but Theorem 1 covers a single hypothesis; no multiplicity control is given for a stream of data-dependent proposals. The 'universally better replacement' phrasing in the abstract is not supported by one synthetic benchmark against naive held-out testing.\n\nStill, the core construction is novel and the math for the narrow claim is solid. This is not a desk-reject; a serious referee could push the authors to either provide a principled threshold rule or scale the claims down to what the theory actually guarantees. I'd want to see the method work on a real-world case or two before believing it, but the paper deserves the reviewer time.","headline":"The N×N cross-prediction construction is genuinely new and the e-value theory is sound for a single hypothesis, but the universal certifier claim rests on a label-calibrated diversity threshold and an unproven anytime-validity extension.","tokens_in":14136,"tokens_out":1188,"would_cite":false,"duration_ms":12512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that any per-sample hypothesis can be certified by reading two e-values off an N×N cross-prediction matrix, with per-hypothesis error at most alpha and no multiplicity correction.","keywords":["e-values","hypothesis certification","conditional randomization test","conformal inference","anytime-valid inference","structural confound","LLM hallucination","cross-prediction matrix"],"falsifier":"Construct a synthetic world whose structural confound has per-unit restrictions that differ only on items never appearing in any in-scope held-out outcome, so every column of the cross-prediction matrix is constant and diversity equals zero; the algorithm will route the hypothesis to the glocal branch and certify it, contradicting the claim that it rejects structural confounds.","tokens_in":12985,"feed_emoji":"✅","tokens_out":8924,"duration_ms":73843,"temperature":0.7,"pith_summary":"The paper is trying to establish that a machine-proposed hypothesis about individual data units—such as an LLM's rule saying a phone-buyer buys a case—can be certified against data automatically, without building a bespoke null hypothesis for each proposal. Its method, N×N E-valuation, scores every unit's restriction against every unit's held-out outcome to form a cross-prediction matrix, then reads two e-values off that matrix: one for beating the population base rate and one for beating the within-scope average. It certifies a hypothesis only when both exceed $1/\\alpha$, and it proves that the per-hypothesis certification error is at most $\\alpha$ even as data and discovery rounds accumulate. In synthetic worlds with planted ground truth, the certifier assigned the correct verdict 99% of the time while naive held-out validation was fooled by all four planted fakes. If the proof and experiments hold, this gives a general, distribution-free alternative to LLM self-verification and held-out testing, with the dataset itself serving as the null.","feed_headline":"N×N matrix certifies hypotheses without hand-built nulls","feed_subtitle":"Two e-value ratios separate real per-unit effects from base rates and shared confounds.","key_machinery":"The central object is the $N\\times N$ cross-prediction matrix $M[i][j] = r_i(y_j)$, where $r_i = \\mathrm{apply}(x_i)$ is the restriction the hypothesis returns for unit $i$ and $y_j$ is unit $j$'s held-out outcome. The diagonal entries are the correct pairings; each row supplies the exchangeable pool used by the conformal mechanism e-value, and the restriction's own base rate over the test set supplies the significance null. Diversity, the average over columns of $\\sigma_j/\\mu_j$, is read off the same matrix and routes the verdict among unit-specific, glocal, global-prevalence, and reject. The matrix is what realizes a conditional randomization test without a hand-built null: different samples serve as null hypotheses for one another.","core_discovery":"The central claim is that certification reduces to a ratio comparison between a unit's restriction evaluated on its own held-out outcome and the same restriction evaluated on a pool of other units' outcomes. For each in-scope unit $j$, the significance e-value $E^{\\mathrm{sig}}_j = r_j(y_j)/g_0(r_j)$ compares the score on $j$'s own outcome to the restriction's base rate over the whole test set; the mechanism e-value $E^{\\mathrm{mech}}_j = M[j][j]/(\\text{row mean})$ compares the diagonal entry of the cross-prediction matrix to the row's off-diagonal average. The paper proves each is an exact e-value under its respective exchangeability null, that their mean remains an e-value under arbitrary dependence, and that certifying iff both exceed $1/\\alpha$ gives per-hypothesis error at most $\\alpha$ with no multiplicity correction. A third statistic, diversity, defined as the mean column coefficient of variation of the matrix, is claimed to separate a constant real rule (glocal) from a structural confound that looks personalized but is driven by a shared hidden factor. The empirical claim is that on eight planted rules the certifier's verdicts matched ground truth, rejecting both prevalence fakes, the structural confound, and a Simpson's-paradox rule that naive held-out testing certified.","pith_inferences":["Because the e-value proofs require only nonnegativity and exchangeability, the same matrix construction should extend to continuous score functions, not just the recall-style restrictions used in the experiments.","The diversity threshold is calibrated on ground-truth knowledge in the experiments, so a practical release of the method would need a calibration procedure for $\\tau_{\\mathrm{div}}$ that does not know which rules are real.","The formal object is any per-unit scoring function, so program synthesizers, subgroup miners, and other automated proposers are natural targets beyond the LLM recommendation framing.","A direct testable extension is to widen the probe set beyond observed outcomes, as the paper's appendix suggests, and measure how often the glocal-versus-confound separation improves."],"forward_implications":["LLM outputs that take the form of per-sample hypotheses can be certified directly against a training set, replacing self-verification loops that cannot catch the model's own errors.","Predictive success on held-out data is no longer enough: a rule that tracks only a base rate or a shared confound fails the mechanism e-value even when it predicts well.","Certificates compose across independent data banks, so a rare hypothesis can accumulate support until a target sample size is reached while preserving anytime validity.","The three-statistic verdict taxonomy gives one unified classification—unit-specific, glocal, global-prevalence, or reject—across very different failure modes.","No case-specific null construction or multiplicity correction is needed; the per-hypothesis certification error is bounded by $\\alpha$."],"supporting_citations":[{"why":"Supplies the conditional randomization test (CRT) construction that N×N E-valuation automates by using other samples as nulls.","marker":"Candès et al. (2018)"},{"why":"Provides the e-value definition and the conformal row-normalization identity used to form the mechanism e-value.","marker":"Vovk & Wang (2021)"},{"why":"Provides the anytime-valid e-value framework that makes the certificate composable across data banks and rounds.","marker":"Ramdas et al. (2023)"},{"why":"Supplies the safe-testing e-value validity results underlying the significance and mechanism e-values.","marker":"Grünwald et al. (2024)"},{"why":"Documents the failure of LLM self-correction that the paper positions its certifier to replace.","marker":"Huang et al. (2024)"},{"why":"Documents the spurious-correlation failures of held-out testing that the mechanism e-value is designed to catch.","marker":"Ye et al. (2024)"}],"fun_headline_variants":["Mutual nulls: N×N e-values certify any hypothesis","No hand-built nulls: NxN e-values certify every hypothesis","Cross-sample nulls from the training set certify hypotheses","E-value ratios certify real effects, reject shared confounds","Certify any hypothesis without a dedicated null"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's defense against a rule that looks personalized but is secretly driven by a shared hidden cause is a diversity threshold, and the paper only shows how to set that threshold when the experimenter already knows which rules are real and which are fake; nothing in the theory says how to choose it without that knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Mutual nulls: N×N e-values certify any hypothesis","No hand-built nulls: NxN e-values certify every hypothesis","Cross-sample nulls from the training set certify hypotheses","E-value ratios certify real effects, reject shared confounds","Certify any hypothesis without a dedicated null"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000936,"raw_usage":{"total_tokens":4039,"prompt_tokens":1016,"completion_tokens":3023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":2941}},"tokens_in":632,"tokens_out":3023,"duration_ms":21883,"temperature":1.0,"reasoning_tokens":2941,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:13:49.723134+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic world whose structural confound has per-unit restrictions that differ only on items never appearing in any in-scope held-out outcome, so every column of the cross-prediction matrix is constant and diversity equals zero; the algorithm will route the hypothesis to the glocal branch and certify it, contradicting the claim that it rejects structural confounds.","supporting_citations":[],"review_version":1}