{"id":"2fcea543-22f7-4ff8-938b-e7e7bca65cb1","arxiv_id":"2508.20616","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A credibility test for survey data in regression, SurVerify, uses a dimension-free number of validation samples by testing the survey model's population loss instead of the full distribution.","lead":"Surveys can be checked for regression credibility by comparing the fitted model's errors on a small clean sample against its training errors, without comparing the full data distributions. This yields a validation sample size that does not grow with the number of variables, which is useful when high-quality population samples are expensive.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 5 requires equal irreducible noise variance under D* and DS; Appendix C.1 Eq. (11) asserts this without justification, so the ACCEPT/REJECT boundary can shift by the variance mismatch.","rationale":"The reader's weakest_assumption identifies the same load-bearing gap: Lemma 14's proof needs equality of the irreducible noise variance between DS and D*, and Assumption 1 does not provide it. This is not a minor tightening issue; the mismatch enters additively into the test statistic and can produce false accepts or false rejects even when the functional distance is zero, so Theorem 5's two-sided guarantee is not valid under the stated assumptions. The paper does have real value: the dimension-agnostic sample complexity idea is original, the experiments are concrete, and the concentration-constant error in Eq. (26) (the stated tau gives 2*delta/3 rather than delta/3) is separately fixable. The equal-noise problem is also fixable, either by stating it as an explicit assumption or by estimating the noise variance from the D* samples, at a mild cost in the algorithm's design. For these reasons I agree with the CONDITIONAL verdict rather than recommending rejection: the framework and metric are plausible, but the main theorem needs a stated assumption or a modified procedure before the guarantee is established.","tokens_in":24004,"tokens_out":11170,"duration_ms":87525,"concrete_test":"Run a simulation with identical covariate distributions and identical regression function but different noise variances: D* draws x uniformly from [-1,1]^2 and y = theta* . x + eta_D* with sigma_{D*}^2 = 0.1; DS draws the same x marginal and the same theta*, but with eta_DS ~ N(0, 0.001). Use epsilon = 0.01, delta = 0.1, a survey S of size m = 10^6 from DS, and sample-to-test tau = ceil(2/(0.019)^2 log(3/0.1)). Run SurVerify 100 times. If it REJECTS with frequency above delta although FDD(DS,D*) = 0, the equal-noise assumption in Eq. (11) is load-bearing. Repeat with sigma_{DS}^2 = 0.5 to check the false-accept direction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SurVerify's guarantee is anchored on Lemma 4, which says E_{D*}[(f_S-y)^2] = FDD^2 + sigma_eta^2. To turn this into a test, the algorithm compares the validation loss gamma against the survey loss L_S, relying on Lemma 7/14 to certify |sigma_eta^2 - L_S| <= epsilon/10. The proof of Lemma 14 in Appendix C.1 uses Eq. (11): sigma_eta^2 = E_{DS}[(f*_S(x)-y)^2], i.e., the noise variance under the survey distribution DS equals the irreducible noise variance under the true distribution D*. Assumption 1 only postulates homoskedasticity and uncorrelatedness of the noise within each distribution; it does not say Var_{DS}(y - f_DS(x)) = Var_{D*}(y - f*(x)). If the survey is cleaner or noisier by Delta, then E_{D*}[(f_S-y)^2] approximates FDD^2 + sigma_{D*}^2, while L_S approximates sigma_{DS}^2. The final ACCEPT test, gamma_tau - tau*L_S <= 3*tau*epsilon, then carries an additive bias of tau*(sigma_{D*}^2 - sigma_{DS}^2). For example, with FDD = 0 and sigma_{DS}^2 < sigma_{D*}^2 - 4*epsilon, SurVerify will reject a perfectly credible survey with probability close to 1, contradicting part 1 of Theorem 5; the opposite mismatch causes false accepts. Since Theorem 5's conditions are silent on this equality, the central claim as stated is not established. The fix is to add equal noise variance as an explicit assumption or to estimate sigma_eta^2 from D* samples and adjust the comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a task-based approach to assessing whether a sample survey is credible for downstream regression analysis. It defines the Functional Distance of Distributions (FDD), the distance, measured under the true distribution, between the regression models that are optimal for the survey distribution and for the true distribution. The main algorithmic contribution is SurVerify, which fits a regression model to the survey, draws a small number of validation samples from the true distribution, and decides ACCEPT or REJECT by comparing validation loss against the survey's empirical loss. The headline claim is that the number of validation samples is O(log(1/delta)/epsilon^2), independent of the covariate dimension, with a 1-delta guarantee under an epsilon/5epsilon acceptance/rejection gap. The paper also proves a two-sided Rademacher-style generalization bound, gives survey-size lower bounds for Lasso, Ridge, and kernel classes, and presents a dimension-dependent lower bound for approaches that reconstruct the model. Experiments on synthetic data and ACS_Income illustrate the acceptance-rate behavior. The central idea is appealing, but the proof of the main theorem contains several load-bearing gaps that need to be addressed before the stated guarantees can be accepted.","tokens_in":24406,"tokens_out":12055,"duration_ms":117277,"significance":"If the main theorem were fully established, the paper would make a useful contribution: it offers a concrete way to certify survey credibility for regression without exponential-in-dimension distribution testing, and its validation sample complexity is genuinely dimension-free. The proposed two-sided generalization bound (Theorem 6) is also of independent interest, and the lower bound in Lemma 9 gives a useful contrast between model reconstruction and task-based testing. The experiments are careful in structure and support the qualitative accept/reject behavior claimed for the algorithm. However, the paper's central guarantee as stated is not established: the proof of Theorem 5 relies on incorrect Hoeffding constants, an unstated equality of irreducible noise variances across the survey and true distributions, and a mismatch between the FDD condition in the theorem statement and the empirical model actually used by the algorithm. These issues are load-bearing, but they appear fixable either by adding the missing assumptions or by restating the theorem in terms of the quantity the algorithm actually controls.","major_comments":[{"comment":"The Hoeffding-based concentration estimates in the proof of Theorem 5 use constants appropriate for [0,1]-valued variables, while the squared loss is bounded in [0,4] by Assumption 2. In Eq. (22), the threshold sqrt(t * 2 log(3τ/δ)) with [0,4]-valued summands gives exponent -log(3τ/δ)/4, not -log(3τ/δ), so the claimed bound delta/(3τ) does not follow. In Eq. (26), even under [0,1] boundedness the stated calculation gives 2 exp(-τ(1.9ε)^2/2) = 2δ/3 rather than delta/3; with [0,4] boundedness the discrepancy is larger. Consequently the union-bound argument does not establish the stated 1-delta guarantee. This is fixable by choosing tau with an additional factor of 8 (and adjusting the early-stopping threshold), which preserves the O(1/epsilon^2 log(1/delta)) dimension-free sample complexity, but the constants in the theorem and the algorithm description must be corrected.","section":"Appendix D, Eqs. (22) and (26)"},{"comment":"Lemma 14 and Lemma 7 identify the survey empirical loss L_S with the true irreducible noise variance sigma_eta^2 through Eq. (11), which asserts sigma_eta^2 = E_{D*}[(f*(x)-y)^2] = E_{DS}[(f*_S(x)-y)^2]. Assumption 1 only postulates homoskedasticity and non-correlation of the noise within each distribution; it does not state that the irreducible noise variance under DS equals that under D*. If the survey is noisier or cleaner than the population by Delta, the final ACCEPT test gamma_tau - tau L_S <= 3 tau epsilon carries an additive bias tau(sigma_{D*}^2 - sigma_{DS}^2), so a perfectly credible survey can be rejected with high probability, and a far-from-credible survey can be accepted. The theorem's conditions are silent on this equality, so Theorem 5 as stated is not established. The fix is to add equal noise variance across DS and D* as an explicit assumption, or to estimate sigma_eta^2 from the D* validation samples and adjust the comparison accordingly.","section":"Appendix C.1, Eq. (11)"},{"comment":"The theorem statement conditions on FDD_{D*}(DS,D*) = dist_{D*}(f_{DS}, f*), the distance between population-optimal models, but the algorithm uses fS, the empirical minimizer on the finite survey. In the proof, the condition used before Eq. (23) is 'dist^2_{D*}(fS,f*) <= epsilon', which is a different quantity from FDD^2. Nothing in Lemma 7 or in the survey-size conditions of Table 1 bounds dist_{D*}(fS, f_{DS}) under D*; uniform convergence on DS controls risk under DS, and DS and D* may have very different covariate supports. Thus FDD <= epsilon does not prevent the validation loss from being large when fS differs from f_{DS} on the support of D*, which would cause part 1 of Theorem 5 to fail as stated. The proof also assumes a unique minimizer f_{DS}, which Assumption 1 does not guarantee. The theorem should be restated in terms of dist_{D*}(fS, f*) (the quantity the algorithm actually tests), or the authors must prove a transfer bound from fS to f_{DS} under D* under explicit additional assumptions.","section":"Theorem 5, statement and proof in Appendix D"}],"minor_comments":[{"comment":"The proof of Theorem 5 contains the repeated headings 'Proof of 5' and 'Proof of 5'; these should be 'Proof of Part 1' and 'Proof of Part 2'.","section":"Appendix D"},{"comment":"The figures in Section 5 and the appendix state delta = 0.1, while the appendix tables say delta = 0.01 throughout; this discrepancy should be reconciled.","section":"Section 5 and Tables 3-6"},{"comment":"The text says that if the expected losses differ by epsilon, then FDD = epsilon by Lemma 4; under equal noise variance Lemma 4 gives FDD^2 equal to the loss difference, so FDD would be sqrt(epsilon), not epsilon.","section":"Lemma 9, discussion after the proof"},{"comment":"The phrase 'sample complexity of our algorithm is independent of the data's dimension' refers only to the validation samples from D*; the survey size in Table 1 for Ridge is Omega(d/epsilon^2) and for Lasso depends on d through log d. This distinction should be stated explicitly in the abstract and introduction.","section":"Abstract and Section 1"},{"comment":"The sentence 'how does the acceptance rate of SurVerify change as the the distance between the survey set S and the true distribution D*, and the tolerance parameter change?' contains a duplicated article and an ungrammatical structure.","section":"Section 5, RQ1"},{"comment":"Assumption 2 states f(x) <= 1, but for the kernel class with ||theta||_H <= 1 and K(x,x) <= r^2 one only has |f(x)| <= r; if r > 1 the boundedness condition is violated, and the kernel row of Table 1 should state the additional requirement r <= 1 or rescale the kernel.","section":"Assumption 2 and Table 1"},{"comment":"The reference list contains the typo 'NuerPS' for NeurIPS, and Section 1 contains 'odtained' for 'obtained'; Section 5 repeatedly uses '#samples-to-set' where '#samples-to-test' is meant.","section":"References and typos"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this: the paper has a new idea—judge survey credibility by whether the regression model it induces is close to the one you'd get from the population, rather than by distribution distance—and the sample complexity of the resulting test is dimension-free. That is worth engaging with. But the main theorem as written is not established: the proof uses the wrong Hoeffding range and silently assumes equal noise variance across the two distributions.\n\nThe task-based framing is the real contribution. The FDD metric and the 'validate the model, don't reconstruct it' algorithm are not in the cited distribution-testing or model-validation literature. The two-sided generalization bound in Theorem 6 is a useful extension of the standard one-sided Rademacher bound, and the assembly of the linear and kernel Rademacher bounds is clean. The experiments are honest—they run on a real dataset that violates the homoskedasticity assumption and report that, and the acceptance-rate curves match the intended behavior.\n\nNow the soft spots. First, the proof of Theorem 5 uses Hoeffding constants for [0,1]-valued variables when the squared loss actually lies in [0,4] (Appendix D, Eqs. 22 and 26). The stated sample complexity τ = O(1/ε² log(1/δ)) is off by a constant factor, so the guarantee doesn't follow from the proof. That's a constant-factor bug, fixable.\n\nSecond, and more serious, Lemma 14's proof (Appendix C.1, Eq. 11) assumes the irreducible noise variance under D_S equals that under D^*. Assumption 1's homoskedasticity doesn't give that. If the survey is cleaner or noisier, the ACCEPT/REJECT boundary shifts by the variance mismatch, and Theorem 5's guarantees can fail even when FDD is small. This is patchable: add equal noise variance as an explicit assumption, or estimate σ² from D^* samples and adjust the comparison.\n\nThere's also a notational gap: FDD is defined using the population-optimal models for each distribution, but the algorithm actually tests the empirical survey model f_S. The proof of Theorem 5 uses dist(f_S, f^*), which only equals the stated FDD if the survey is large enough that f_S ≈ f_{D_S}. The theorem's assumption and the quantity actually being tested are slightly different objects.\n\nNone of this kills the core idea, but the paper as submitted overclaims its theoretical guarantees. The dimension-free claim is conceptually solid and should hold up once the proof is repaired.\n\nI'd send this to peer review with a request for major revision. The audience is people working on high-dimensional distribution testing and survey validation. They should know the FDD notion and the algorithm, but should not quote the theorem in its current form.","headline":"The task-based FDD/survey-credibility idea is new and dimension-free, but Theorem 5's proof has a Hoeffding-range bug and an unstated equal-noise-variance assumption.","tokens_in":24909,"tokens_out":8041,"would_cite":false,"duration_ms":71486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Survey credibility for regression can be certified with a sample count that never depends on the data dimension.","keywords":["survey credibility","distribution testing","regression","sample complexity","Rademacher complexity","Functional Distance of Distributions","dimension-free","model validation"],"falsifier":"Fix a true distribution $D^*$ with noise variance $\\sigma_\\eta^2$ and construct a survey distribution $D_S$ whose conditional mean is the same function $f^*$ but whose noise variance is $\\sigma_\\eta^2/4$; then the squared FDD is zero, so Theorem 5 predicts ACCEPT with probability $1-\\delta$, yet the survey loss $\\hat L_S$ underestimates $\\sigma_\\eta^2$, which can push $\\hat\\gamma - \\tau \\hat L_S$ above $3\\tau\\epsilon$ and trigger REJECT. Observing rejections above $\\delta$ under this construction would show that equation (11), which asserts equality of noise variances across the two distributions, is load-bearing.","tokens_in":23839,"feed_emoji":"📊","tokens_out":6304,"duration_ms":59540,"temperature":0.7,"pith_summary":"This paper argues that deciding whether a survey dataset is credible for downstream regression does not require estimating how similar the survey's distribution is to the population's, since that approach costs exponentially many samples in the dimension. Instead, it proposes a task-specific metric, the Functional Distance of Distributions (FDD), which measures how far the best regression model on the survey is from the best model on the true distribution, evaluated on the true distribution. The central theorem states that a simple algorithm, SurVerify, can ACCEPT or REJECT a survey using only $O(\\epsilon^{-2}\\log(1/\\delta))$ samples drawn from the true population, a number independent of the number of covariates. The guarantee holds for Lasso, Ridge, and kernel regression when the survey is large enough and the regression noise is homoskedastic and uncorrelated. If true, this means credible surveys can be certified cheaply even in high-dimensional settings where classical distribution testing is infeasible.","feed_headline":"Survey credibility check escapes the dimension curse","feed_subtitle":"It compares losses to a small true-population sample, accepting or rejecting without reconstructing the model.","key_machinery":"The central object is the Functional Distance of Distributions (FDD), defined as the distributional $\\ell_2$-distance between the optimal model for the survey distribution and the optimal model for the true distribution, both evaluated under the true distribution. The identity that carries the argument is Lemma 4, which decomposes the true expected squared loss of the survey-fitted model as $\\text{FDD}^2 + \\sigma_\\eta^2$, so that credibility becomes a statement about loss rather than about distributions. SurVerify then estimates the true expected loss on a small fresh sample, compares it to the survey's own training loss, and uses an early-rejection rule plus a final threshold to accept or reject. The sample-size requirements come from a new two-sided generalization bound, proved with empirical Rademacher complexity and Talagrand's contraction lemma, which controls both over- and under-estimation of the expected loss.","core_discovery":"The paper's central claim is Theorem 5: given a survey drawn from an unknown distribution $D_S$, a regression model class, and i.i.d. sampling access to the true distribution $D^*$, SurVerify accepts with probability at least $1-\\delta$ when the squared Functional Distance of Distributions is at most $\\epsilon$, rejects with probability at least $1-\\delta$ when that squared distance exceeds $5\\epsilon$, and uses at most $\\lceil 2/(1.9\\epsilon)^2 \\log(3/\\delta)\\rceil$ samples from $D^*$. The efficiency comes from testing the model the survey produces rather than reconstructing the population model; the paper also proves that any algorithm that reconstructs the model first must spend $\\Omega(d\\lambda_{\\min}\\sigma_\\eta^2/\\epsilon^2)$ samples. A new two-sided generalization bound, obtained through Rademacher complexity, supplies the required survey size for Lasso, Ridge, and kernel classes. The FDD-variance decomposition, Lemma 4, is the load-bearing identity: the expected squared loss of the survey model under the true distribution equals the squared FDD plus the irreducible noise variance $\\sigma_\\eta^2$.","pith_inferences":["If the same two-sided concentration can be obtained for other losses, such as classification losses or heteroskedastic regression, SurVerify's compare-the-losses logic would extend to a wider family of task-specific credibility notions; the paper explicitly leaves these extensions open.","The dimension-free sample count suggests a practical validation protocol: maintain a small gold-standard sample from the population and reuse it to certify many different survey datasets for a fixed regression task, since the required sample size is constant for fixed tolerance and confidence.","The early-rejection stopping time itself carries quantitative information about how far the survey model is from the true model, so the algorithm could be adapted to return a constant-factor estimate of FDD rather than just a binary accept or reject.","The FDD equivalence classes point to a possible design principle: two survey distributions are interchangeable for a task if they induce the same optimal model under the chosen hypothesis class, which could inform stratified sampling design before data collection."],"forward_implications":["For Lasso, Ridge, and kernel regression, a survey of sufficient size can be certified using only $O(\\epsilon^{-2}\\log(1/\\delta))$ samples from the true population, with no dependence on the ambient dimension.","If the squared FDD is at most $\\epsilon$, SurVerify accepts with probability $1-\\delta$; if it exceeds $5\\epsilon$, SurVerify rejects with probability $1-\\delta$, leaving an explicit tolerance gap between $\\epsilon$ and $5\\epsilon$.","The early-rejection rule means that surveys that are far from credible are detected with fewer samples than the worst-case bound, matching the paper's experiments where average samples used drop rapidly as the model distance grows.","Because SurVerify does not reconstruct the regression model, it avoids the linear-in-dimension sample cost that any reconstruction-based testing method must pay, as established by the paper's lower bound.","The same fresh samples from $D^*$ can be reused for multiple projection-based credibility checks, since the paper notes that FDD is not robust to arbitrary projections and recommends running SurVerify on projected dimensions."],"supporting_citations":[{"why":"Supplies the Rademacher complexity definitions and textbook generalization bounds that Theorem 6 extends to a two-sided form.","marker":"[MRT18]"},{"why":"Provides the empirical Rademacher complexity bounds for bounded-weight linear hypothesis classes used in Lemma 15 to derive the survey-size requirements.","marker":"[AFM20]"},{"why":"Talagrand's contraction lemma bounds the Rademacher complexity of the squared-loss composition, a necessary step in the two-sided generalization bound.","marker":"[LT91]"},{"why":"Fano-type lower bound on parameter estimation used in Lemma 19 to prove that model-reconstruction approaches require samples growing linearly with dimension.","marker":"[DW13]"},{"why":"Data processing inequality invoked to argue SurVerify remains effective when only an approximation to the true population distribution is available.","marker":"[PW25]"}],"fun_headline_variants":["Dimension-free survey credibility via regression","Survey check skips model reconstruction","SurVerify: sample complexity ignores dimension","Credibility test without rebuilding the model"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof requires that the noise variance in the survey data equals the noise variance in the true population, so that the survey's training loss estimates the irreducible noise variance $\\sigma_\\eta^2$; if the survey is noisier or cleaner than the population, the accept and reject thresholds shift and the guarantee can fail even when the fitted models are close.","fun_headline_variants_meta":{"raw":{"variants":["Dimension-free survey credibility via regression","Survey check skips model reconstruction","SurVerify: sample complexity ignores dimension","Credibility test without rebuilding the model"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000771,"raw_usage":{"total_tokens":3425,"prompt_tokens":969,"completion_tokens":2456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":2407}},"tokens_in":585,"tokens_out":2456,"duration_ms":15909,"temperature":1.0,"reasoning_tokens":2407,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:43:42.919647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix a true distribution $D^*$ with noise variance $\\sigma_\\eta^2$ and construct a survey distribution $D_S$ whose conditional mean is the same function $f^*$ but whose noise variance is $\\sigma_\\eta^2/4$; then the squared FDD is zero, so Theorem 5 predicts ACCEPT with probability $1-\\delta$, yet the survey loss $\\hat L_S$ underestimates $\\sigma_\\eta^2$, which can push $\\hat\\gamma - \\tau \\hat L_S$ above $3\\tau\\epsilon$ and trigger REJECT. Observing rejections above $\\delta$ under this construction would show that equation (11), which asserts equality of noise variances across the two distributions, is load-bearing.","supporting_citations":[],"review_version":2}