{"id":"4096889f-8fc9-47dd-ad75-5c2558d1a22a","arxiv_id":"2606.19737","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Develops pseudo-labels from ordered p-value spacings to enable post-hoc calibration assessment of local FDR estimates in multiple testing without labels, revealing miscalibration of q-values in empirical literature.","lead":"The paper introduces a method to assess calibration of local false discovery rate estimates in multiple testing by constructing pseudo-labels from the spacings of ordered p-values. This enables stochastic calibration checks without observed ground truth and shows that q-values can be severely miscalibrated in published psychology and neuroscience studies.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Ordering-induced dependence may prevent pseudo-labels from having exact local FDR as conditional expectation given individual p-value","rationale":"The reader's weakest assumption is exactly the regression-target claim. The dependence issue is the most direct threat to that claim; confirming or refuting the equality via the proposed simulation would settle whether the calibration-assessment machinery is valid before the empirical survey can be trusted.","tokens_in":1644,"tokens_out":344,"duration_ms":38127,"concrete_test":"Under a known two-group mixture (e.g., 80% uniform null + 20% Beta(0.5,3) alternative, m=10^4), compute the true lfdr(p) analytically, generate the spacing-based pseudo-labels, and regress them on p (or bin by p); if the fitted curve deviates from the analytic lfdr by more than Monte-Carlo error, the claimed regression target does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The construction derives pseudo-labels from spacings of the full vector of ordered p-values. Local FDR is defined as the conditional probability given an individual test statistic (or p-value). Because each spacing depends on neighboring order statistics, the pseudo-label for hypothesis i is a function of the entire ordered list. Consequently E[pseudo-label_i | p_i] need not equal lfdr(p_i) even if the marginal construction appears to target it; the dependence is not removed by conditioning on a single p-value. The paper asserts this holds under misspecification, but the joint nature of the spacings makes the equality non-obvious without an explicit derivation showing that the conditional expectation factors correctly.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes a framework for assessing calibration of local false discovery rate (lfdr) estimates and related error measures like q-values in multiple testing without access to ground-truth labels. It constructs pseudo-labels from spacings of ordered p-values, asserts that these have the lfdr as their regression target (even under misspecification), and uses this to enable stochastic calibration checks. A large-scale empirical survey of psychology and neuroscience literature is presented as evidence that q-values can be severely miscalibrated.","tokens_in":1792,"tokens_out":439,"duration_ms":16459,"significance":"If the pseudo-label construction is valid, the work supplies a practical tool for post-hoc calibration assessment in settings where labels are unavailable, extending ideas from probabilistic forecasting to multiple testing. The empirical survey finding, if reproducible, would be notable for fields that rely on FDR-based error measures.","major_comments":[{"comment":"The central technical claim is that pseudo-labels derived from spacings of the ordered p-values have the local FDR as their regression target (i.e., E[pseudo-label_i | p_i] = lfdr(p_i)). The skeptic note correctly identifies that each spacing depends on the full vector of order statistics, so the conditional expectation given only p_i does not automatically factor; an explicit derivation establishing the equality (under the paper's assumptions or under misspecification) is required before the calibration-assessment machinery can be used.","section":"Abstract / core construction"},{"comment":"The empirical claim that q-values are 'severely miscalibrated' rests on the pseudo-label calibration assessment. Because the validity of that assessment hinges on the unresolved conditional-expectation step above, the survey results cannot yet be interpreted as evidence of miscalibration.","section":"Empirical survey"}],"minor_comments":[{"comment":"The abstract states the conceptual approach and the empirical finding but supplies no equations or proof sketches; readers cannot evaluate the construction from the given information.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their careful and constructive review. We address the two major comments point by point below. We agree that an explicit derivation of the key conditional-expectation property is needed for clarity and will supply it in revision.","responses":[{"response":"We accept that the manuscript would be strengthened by an explicit derivation. The pseudo-label construction uses normalized spacings of the ordered p-values; under the two-group model the joint distribution of the order statistics yields E[spacing-based pseudo-label_i | p_i] = lfdr(p_i) by direct computation of the conditional density. The same equality holds under misspecification because the target is defined as the regression function of the pseudo-label on p_i. We will add a self-contained appendix deriving the result from first principles and will reference it in the main text.","revision_made":"yes","referee_comment":"[Abstract / core construction] The central technical claim is that pseudo-labels derived from spacings of the ordered p-values have the local FDR as their regression target (i.e., E[pseudo-label_i | p_i] = lfdr(p_i)). The skeptic note correctly identifies that each spacing depends on the full vector of order statistics, so the conditional expectation given only p_i does not automatically factor; an explicit derivation establishing the equality (under the paper's assumptions or under misspecification) is required before the calibration-assessment machinery can be used."},{"response":"We agree that interpretability of the survey depends on the core claim. Once the requested derivation is supplied, the large-scale survey of published q-values can be read as evidence that many reported q-values deviate from the pseudo-label regression target. The survey itself requires no substantive change; we will simply add a forward reference to the new appendix when discussing the empirical findings.","revision_made":"partial","referee_comment":"[Empirical survey] The empirical claim that q-values are 'severely miscalibrated' rests on the pseudo-label calibration assessment. Because the validity of that assessment hinges on the unresolved conditional-expectation step above, the survey results cannot yet be interpreted as evidence of miscalibration."}],"tokens_in":1328,"tokens_out":466,"duration_ms":24226,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's core move is to turn the spacings of ordered p-values into pseudo-labels whose regression target is the local false discovery rate. This lets them borrow calibration diagnostics from forecasting even though the true null status is never observed. That is the actual novelty relative to standard empirical Bayes work on FDR. The large-scale survey of psychology and neuroscience papers is also useful on its face; showing that q-values can be badly off in real published work gives the method a concrete hook.\n\nThe stress-test note on ordering dependence lands. Each pseudo-label is a function of the entire ordered list, so it is not immediate that the conditional expectation given a single p_i equals lfdr(p_i). The abstract asserts this holds under misspecification, but without an explicit step showing the dependence factors out, the claim is hard to verify from what is given. The empirical finding on miscalibration is only as strong as the pseudo-label justification, so that part needs tightening.\n\nThe work is aimed at people who run large-scale multiple testing and want to audit their error measures when ground truth is unavailable. A reader already comfortable with local FDR and willing to work through the spacing construction will get the most out of it. The idea is coherent enough on its own terms to deserve a serious referee, even if the current write-up leaves the key equality under-specified.","headline":"The pseudo-label construction from p-value spacings is a neat idea for label-free calibration checks, but the dependence from ordering makes the exact lfdr target claim non-obvious and in need of a clean derivation.","tokens_in":2252,"tokens_out":360,"would_cite":false,"duration_ms":9175,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Pseudo-labels from p-value spacings allow calibration assessment of false discovery rate estimates without observing true labels.","keywords":["multiple testing","false discovery rate","calibration","q-value","p-value spacings","local false discovery rate","model misspecification","pseudo-labels"],"falsifier":"A simulation in which the data-generating process is fully known so that true local false discovery rates can be computed exactly, yet the pseudo-labels fail to regress to those rates in expectation.","tokens_in":2551,"feed_emoji":"","tokens_out":714,"duration_ms":26350,"temperature":0.7,"pith_summary":"Large-scale hypothesis testing produces probability claims about individual hypotheses, such as those based on the false discovery rate. Because the true null or alternative status is never observed, direct checks on whether these claims are calibrated as probabilities are impossible. The paper constructs pseudo-labels from the spacings between ordered p-values; these pseudo-labels have the local false discovery rate as their regression target. This step permits standard tools for calibration assessment and post-hoc adjustment to be used indirectly and stochastically. An empirical survey of published psychology and neuroscience papers shows that the popular q-value can be severely miscalibrated under this approach.","feed_headline":"P-value spacings create pseudo-labels for FDR calibration checks","feed_subtitle":"This enables verification that q-values act as accurate probability forecasts in large-scale testing without ever seeing the true nulls.","key_machinery":"Pseudo-labels derived from the spacings of ordered p-values, which serve as regression targets for the local false discovery rate.","core_discovery":"We study how such claims can be interpreted as approximately calibrated forecasts of the null hypothesis, yielding interpretable error probabilities even under model misspecification. Our approach draws conceptual inspiration from probabilistic forecasting but addresses a different challenge: unlike forecasting, where labels are eventually observed, in multiple testing the ground truth is never revealed, so calibration must be assessed stochastically and established indirectly. We address this challenge by constructing a set of pseudo-labels, derived from the spacings of ordered p-values, which have the local false discovery rate as their regression target. Our construction unlocks existing","pith_inferences":["The same spacing-based construction could be applied to other error measures besides the q-value.","Routine reporting in empirical fields might eventually include calibration diagnostics derived from these pseudo-labels.","The method opens a route to compare calibration properties across different multiple-testing procedures on the same data."],"forward_implications":["Standard calibration diagnostics from forecasting become applicable to multiple testing procedures.","Error probabilities attached to individual hypotheses remain meaningful even when the underlying model is misspecified.","Large empirical surveys of published work can detect miscalibration in common error measures such as the q-value.","Post-hoc recalibration adjustments can be performed on existing multiple-testing outputs."],"fun_headline_variants":["Spacings of ordered p-values yield pseudo-labels for FDR checks","Pseudo-labels from p-value spacings for stochastic FDR calibration","No ground truth needed for FDR calibration with p-value spacings","p-value spacings unlock calibration tools for multiple testing"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The pseudo-labels built from p-value spacings have the local false discovery rate as their regression target even when the model is misspecified.","fun_headline_variants_meta":{"raw":{"variants":["Spacings of ordered p-values yield pseudo-labels for FDR checks","Pseudo-labels from p-value spacings for stochastic FDR calibration","No ground truth needed for FDR calibration with p-value spacings","p-value spacings unlock calibration tools for multiple testing"]},"model":"grok-4.3","cost_usd":0.005446,"raw_usage":{"total_tokens":2610,"prompt_tokens":648,"num_sources_used":0,"completion_tokens":69,"cost_in_usd_ticks":54462000,"prompt_tokens_details":{"text_tokens":648,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1893,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":648,"tokens_out":69,"duration_ms":14926,"temperature":1.0,"reasoning_tokens":1893,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T16:45:04.792897+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A simulation in which the data-generating process is fully known so that true local false discovery rates can be computed exactly, yet the pseudo-labels fail to regress to those rates in expectation.","supporting_citations":[],"review_version":1}