{"id":"93f24422-a873-42fa-8820-3f42351cc10a","arxiv_id":"2506.11271","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A data-driven rule decides when to merge or keep separate datasets for linear prediction, with a high-probability guarantee under Gaussian linear models.","lead":"This paper proposes an algorithm that decides, for a collection of datasets, whether training one shared linear model on merged data beats training separate models. It gives a mathematical condition for when merging helps and tests the rule on synthetic and real data, including neural network features.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1's guarantee is one-sided: Lemma 1 proves only φδ≥ψδ ⇒ h≥g; when φδ<ψδ, Algorithm 1 defaults to 'no merge' without any bound on false-negative error, so the claimed 1−5δ minimum-OSE property does not follow.","rationale":"The reader identified the underspecified plug-in estimators for A0 and B0 in Algorithm 2, which is a real limitation. However, the most load-bearing concern is more fundamental and affects even Algorithm 1, the version with a stated high-probability guarantee. Proposition 1 claims that Algorithm 1 selects the better of 'merge' and 'separate' with probability at least 1−5δ. The proof route only gives a one-sided test: φδ is a lower bound for h(σ²), ψδ is an upper bound for g(β(1),β(2)), and the merge decision is certified only when φδ ≥ ψδ. When this condition fails, Algorithm 1 chooses 'no merge', but the lemmas supply no bound on the probability that merging was actually better. The paper's own synthetic results are consistent with nontrivial false-negative rates on the merge side, and the text explicitly concedes that Algorithm 2 has no high-probability guarantee. Because the central theoretical claim is not established as stated, the current manuscript should be rejected; a major revision could repair the argument by proving a two-sided decision rule or by reframing the claim as a one-sided reduction guarantee when a merge is selected. I do not see this as an attack on the authors: Theorem 1 is a clean exact identity, the high-probability bounds in Lemmas 5 and 6 may be usable, and the experimental results are suggestive, but the main proposition that carries the headline claim does not follow from the stated lemmas.","tokens_in":41008,"tokens_out":8480,"duration_ms":98823,"concrete_test":"Simulate the known-distribution setting of Proposition 1 with p=2, n1=n2=30, x∼N(0,I), σ²=1, β1=0, and β2 chosen so that h(σ²)>g(β1,β2), i.e., merging is the true optimum. Compute A0 and B0 exactly by Monte Carlo, set δ=0.05, and run Algorithm 1 on 10^4 independent dataset pairs. Measure the false-negative rate P(φδ≤ψδ) conditional on the true label 'merge'. If this rate is not bounded by 5δ (or if it is substantially larger than 0.25), Proposition 1 is refuted. As an analytical companion check, search Appendix B.1.2 for any high-probability upper bound on h and lower bound on g; absent such bounds, the no-merge branch of Algorithm 1 cannot be certified.","verdict_should_be":"REJECT","load_bearing_attack":"Proposition 1 asserts that with probability at least 1−5δ, Algorithm 1 outputs the decision rule with minimum out-of-sample error between merging and separate fits. The supporting lemmas establish only a one-sided implication. Lemma 5 provides φδ as a high-probability lower bound on h(σ²); Lemma 6 provides ψδ as a high-probability upper bound on g(β(1),β(2)). Lemma 1 therefore proves: if φδ ≥ ψδ, then h(σ²) ≥ g(β(1),β(2)) with probability at least 1−5δ. But Algorithm 1 outputs 'no merge' whenever φδ < ψδ. From h(σ²) ≥ φδ and g(β(1),β(2)) ≤ ψδ alone, the event φδ < ψδ gives no information about the order of h(σ²) and g(β(1),β(2)); the true h(σ²) can exceed g(β(1),β(2)) even when the data-dependent test fails. Hence the false-negative probability P(φδ < ψδ and h>g) is uncontrolled, and the output need not be the minimum-OSE decision with probability 1−5δ. A correct proof would need the converse direction (for example, φδ ≤ ψδ ⇒ h≤g, or a separate high-probability upper bound on h and lower bound on g). This gap affects even the 'provable' Algorithm 1, before the separate plug-in-estimator issue for A0/B0 raised by the reader, and it directly undermines the abstract's claim of a high-probability reduction in population loss.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies when two (or more) datasets should be merged for training a linear regression model, with an extension to classification and to neural network representations. For two datasets generated from linear models with Gaussian noise, Theorem 1 gives an exact equivalence: merging reduces the sum of out-of-sample errors if and only if h(σ²)>g(β(1),β(2)), where h is an affine function of the noise variance and g is a Mahalanobis distance between the two coefficient vectors. The paper then proposes Algorithm 1, which replaces the unknown population quantities by estimators and decides to merge when a computable lower bound on h exceeds a computable upper bound on g; Proposition 1 claims a 1−5δ high-probability guarantee for this decision. Because Algorithm 1 requires distribution-dependent constants, the paper introduces Algorithm 2, which tunes a confidence surrogate α by estimated success rate, and Algorithm 3 for greedily clustering multiple datasets. Experiments on synthetic data, real regression datasets, and representations from simple MLPs are reported.","tokens_in":41336,"tokens_out":7219,"duration_ms":82453,"significance":"The exact out-of-sample-error decomposition in Appendix B.1.1 (Lemma 2, Lemma 3, and Theorem 1) is a clean and useful contribution: it makes explicit the bias-variance trade-off in merging two linear-regression datasets and gives an interpretable condition in terms of the coefficient distance and noise variance. If the advertised high-probability decision guarantee were established, the paper would provide a principled alternative to direct empirical-loss comparison, and the code and experimental comparisons are valuable. However, the central probability guarantee is not currently established: Proposition 1 does not follow from the one-sided bounds in Lemma 1, and the practical Algorithm 2 relies on plug-in estimators for distribution-dependent constants that are never defined or controlled. The classification extension compares upper bounds rather than actual population losses, so the broad applicability claimed in the abstract is stronger than what the theorems support.","major_comments":[{"comment":"Proposition 1 asserts that with probability at least 1−5δ, Algorithm 1 outputs the decision rule with minimum out-of-sample error. The supporting Lemma 1 only proves the one-sided implication: if φδ(βhat(1),βhat(2),σhat(c)²) ≥ ψδ(βhat(1),βhat(2),σhat(c)²), then h(σ²) ≥ g(β(1),β(2)) with probability at least 1−5δ. Formally, Lemma 1 controls the false-merge event {φδ ≥ ψδ} ∩ {h < g} but gives no control over the false-no-merge event {φδ < ψδ} ∩ {h > g}. Since Algorithm 1 outputs 'no merge' whenever φδ < ψδ, the probability that the algorithm selects the suboptimal decision on the no-merge side is not bounded by 5δ. Therefore Proposition 1 does not follow from the stated lemmas. A correct proof would require a high-probability converse (for example, an upper bound on h and a lower bound on g), or the stated guarantee must be weakened to 'whenever Algorithm 1 merges, merging is correct with probability at least 1−5δ.' This issue is load-bearing because Proposition 1 is the theoretical basis for the paper's central claim of high-probability population-loss reduction.","section":"Section 2.2.2, Proposition 1 and Lemma 1"},{"comment":"Algorithm 2 is presented as the computationally feasible version of Algorithm 1, but it invokes Algorithm 1 as a subroutine. The functions φδ and ψδ used by Algorithm 1 depend on population-level constants: A0 appears in the definitions of Ã1(δ) and Ã2(δ) in Lemma 5, and B0 enters ψδ through the definition of g and the term Tδ. These constants are defined in Theorem 1 using population expectations such as Wk, E[(X(k)⊤X(k))^{-1}], and E[(X(c)⊤X(c))^{-1}]. No estimator for these constants is defined in Appendix B.1.2, and no error bound is provided for replacing them with sample quantities. The remark before Section 2.2.3 acknowledges that Algorithm 1 requires knowledge of the marginal distribution of x, but Algorithm 2 does not resolve this issue unless explicit plug-in estimators are supplied. Theorem 4 only proves consistency of the bootstrap estimator of the out-of-sample error; it does not control the error in the decision boundary φδ or ψδ. Thus the practical algorithm's theoretical support is incomplete.","section":"Section 2.2.3 and Appendix B.1.2"},{"comment":"Even setting aside the missing estimators for A0 and B0, the claimed consistency of Algorithm 2 is not established. The text states that Theorem 4 justifies the consistency of the approximation, but Theorem 4 is a pointwise consistency statement for dOSEM(βhat(k),Dout_k) as an estimator of OSE(βhat(k), P(k)). It does not imply that the success-rate maximization over α in Part 1 of Algorithm 2 converges, nor that the final decision Merge=1{proxy_acc>λ} converges to the oracle decision. In addition, the same out-of-sample bootstrap is used both to tune α and to compute the final proxy_acc, so the reported proxy accuracy is a training-selected quantity; no uniform consistency or selection-bias control is provided. This gap affects the practical algorithm that is featured in the numerical experiments and in the abstract's claim of a practical method with theoretical guarantees.","section":"Section 2.2.3 and Theorem 4 (Appendix B.1.4)"}],"minor_comments":[{"comment":"The statement of Lemma 6 contains a typographical error: D⊤D is written as {(X(1)⊤X(1))^{-1} + (X(2)⊤X(2))^{-2}}^{-1}, but the second term should be (X(2)⊤X(2))^{-1}.","section":"Appendix B.1.2, Lemma 6"},{"comment":"The restated Lemma 1 and Lemma 5 use the symbol f(σ²) in some places where the main text and Theorem 1 use h(σ²). The notation should be unified throughout.","section":"Appendix B, Lemma 1 and Lemma 5"},{"comment":"The pseudocode for the update is missing the indicator notation: 'correct_α ← correct_α +1 (ϕα−ψα)×OSE_dif>0' should be written as 'correct_α ← correct_α + 1{(ϕα−ψα)×OSE_dif>0}'.","section":"Algorithm 2, line 8"},{"comment":"The classification criterion in Theorem 9 compares upper bounds Φ and Ψ on a surrogate loss, not the actual cross-entropy population losses. The main text should state clearly that the classification result is an error-bound comparison rather than an exact population-loss reduction.","section":"Section 2.3 and Appendix B.2"},{"comment":"The abstract claims the algorithm 'reduces population loss with high probability' by leveraging 'data-driven estimators,' but the only stated high-probability result is Proposition 1 for Algorithm 1, which requires unknown distributional constants. The presentation should distinguish the guaranteed regime from the heuristic/plug-in regime.","section":"Abstract and Section 2.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core derivation in Theorem 1 and the appendices, but the headline theoretical claim is overstated. The one-sided nature of Lemma 1 is not a minor technicality: Proposition 1 as written is not implied by the lemmas, and this affects the abstract's central assertion. The practical Algorithm 2 also lacks estimators for the distribution-dependent constants A0 and B0, and Theorem 4 does not establish consistency of the decision rule. I believe these issues are addressable within the manuscript's scope by adding converse high-probability bounds and explicit plug-in estimators with convergence guarantees, but the current version should not be accepted as is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the exact out-of-sample-error decomposition in Theorem 1 is genuinely useful, and the high-probability merge test in Lemma 1 is a clean conservative procedure. But Proposition 1 overclaims. The lemmas only give a one-sided implication: if φδ ≥ ψδ, then h(σ²) ≥ g(β1,β2) with probability at least 1−5δ. They do not control the false-negative case. When φδ < ψδ, the algorithm defaults to no merge, and nothing in the proof rules out h(σ²) > g(β1,β2). So the claimed minimum-OSE guarantee does not follow. This is the headline issue, and it affects Algorithm 1, before you even get to the practical approximation.\n\nWhat is new and worth credit: I have not seen the exact equivalence between the OSE comparison and h(σ²) > g(β1,β2) in the related multi-task or data-integration literature. Lemma 2's decomposition of the combined-model OSE into variance and bias terms is exact, self-contained, and well worth citing. The experiments are extensive, code is available, and the application to penultimate-layer representations in neural networks is a sensible use of the linear theory.\n\nThe paper has a second soft spot, noted by the reader: Algorithm 2 depends on A0 and B0, which are population expectations, and the paper never defines an estimator for them or bounds the plug-in error. The authors honestly admit Algorithm 2 has no high-probability guarantee, but that admission leaves the abstract's central claim resting entirely on Algorithm 1, where the one-sided gap sits. The classification section is a bound-comparison criterion rather than a decision rule; it is a reasonable extension but does not carry the same weight as the regression results.\n\nWho is this for? People working on data-centric ML, multi-task learning, or linear probing who want a checkable condition for merging datasets. It deserves a serious referee even with the flaw, because the core derivation is solid and the practical algorithm shows real empirical gains. My recommendation: send it to review, but require the authors to either prove a two-sided version (for example, a high-probability upper bound on h and lower bound on g) or rewrite Proposition 1 and the abstract to claim only a conservative merge test. If they keep the current wording, the 1−5δ minimum-OSE claim should not appear in a published version.","headline":"Theorem 1 is a clean, citable decomposition of the merge-vs-separate OSE comparison, but Proposition 1 overclaims a two-sided guarantee the lemmas do not support.","tokens_in":704,"tokens_out":900,"would_cite":true,"duration_ms":41949,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J05","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that merging two linear-regression datasets beats fitting them separately exactly when a scaled noise-variance term exceeds a Mahalanobis distance between the coefficient vectors, and packages this condition into a…","keywords":["dataset merging","linear regression","out-of-sample error","oracle inequality","high-probability guarantee","collaborative prediction","multi-task learning","dataset selection"],"falsifier":"Simulate two Gaussian linear populations with known $\\sigma^2$, $\\beta^{(1)}$, and $\\beta^{(2)}$; compute $h(\\sigma^2)-g(\\beta^{(1)},\\beta^{(2)})$ exactly from the population moments, then draw many datasets and compare Algorithm 2's merge decision with the sign of the true out-of-sample-error difference. If the agreement rate fails to match the claimed success rate, or if a case appears where the sample plug-in of $A_0$ and $B_0$ reverses the inequality, the practical guarantee is falsified.","tokens_in":40788,"feed_emoji":"🧩","tokens_out":12503,"duration_ms":126497,"temperature":0.7,"pith_summary":"This paper asks when it is better to fit one model to two merged datasets rather than separate models to each dataset, and gives an exact answer for linear regression. The merge is beneficial exactly when a scaled version of the common noise variance exceeds a Mahalanobis distance between the two true coefficient vectors: $h(\\sigma^2)>g(\\beta^{(1)},\\beta^{(2)})$, a quadratic distance weighted by the covariate second-moment structure. The authors turn this oracle condition into a data-driven decision rule with a high-probability guarantee, and then into a practical approximation that tunes a confidence parameter and validates decisions on held-out data. The practical version is presented as an approximation whose success rate is estimated rather than guaranteed with high probability. Experiments on real regression data and on neural-network representations show large out-of-sample-error reductions from clustering datasets with the rule.","feed_headline":"Merge datasets when noise variance beats coefficient distance","feed_subtitle":"Exact rule: combining two regressions pays only when scaled noise outweighs the Mahalanobis gap between their true coefficient vectors.","key_machinery":"The load-bearing object is the exact out-of-sample-error decomposition of Lemma 2. For the merged least-squares estimator $\\hat{\\beta}^{(c)}$, the error on population $k$ equals $\\sigma^2+\\sigma^2\\operatorname{tr}(W_k E[(X^{(c)T}X^{(c)})^{-1}])+(\\beta^{(1)}-\\beta^{(2)})^T E[Z_{3-k}^T W_k Z_{3-k}](\\beta^{(1)}-\\beta^{(2)})$, where $W_k$ is the second-moment matrix of an out-of-sample covariate from population $k$ and $Z_k=(X^{(c)T}X^{(c)})^{-1}X^{(k)T}X^{(k)}$. This decomposition converts the merge decision into a direct comparison of a variance effect and a parameter-distance effect. The rest of the machinery packages that comparison for data: $\\phi_\\delta$ and $\\psi_\\delta$ are high-probability surrogates for the two sides of the oracle inequality, built from concentration bounds and anti-concentration bounds, and Algorithm 2 searches over the confidence parameter to maximize an estimated success rate.","core_discovery":"Under the paper's standing assumption that each dataset is generated by a linear model with Gaussian noise of the same variance $\\sigma^2$, the authors derive an explicit formula for the out-of-sample error of the least-squares estimator trained on the merged data. The formula splits that error into a noise term proportional to $\\sigma^2$ and a term quadratic in the coefficient difference $\\beta^{(1)}-\\beta^{(2)}$. Subtracting the two separate-fit errors gives Theorem 1: the merged model wins exactly when $h(\\sigma^2)>g(\\beta^{(1)},\\beta^{(2)})$, where $h(x)=A_0x$, $g(y,z)=\\|y-z\\|_{B_0}^2$, and $A_0$, $B_0$ are constants determined by the covariate distributions. The paper then replaces the unknown $\\sigma^2$, $\\beta^{(1)}$, and $\\beta^{(2)}$ by estimators and, using concentration and anti-concentration bounds, proves Lemma 1: with probability at least $1-5\\delta$, the computable condition $\\phi_\\delta(\\hat{\\beta}^{(1)},\\hat{\\beta}^{(2)},\\hat{\\sigma}^{(c)2}) \\ge \\psi_\\delta(\\hat{\\beta}^{(1)},\\hat{\\beta}^{(2)},\\hat{\\sigma}^{(c)2})$ certifies that merging reduces population loss. Algorithm 1 turns this into a decision rule whose output matches the better of the two out-of-sample losses with probability at least $1-5\\delta$.","pith_inferences":["A direct extension the paper leaves untested is whether the same criterion shape holds for ridge regression, where the variance term depends on the regularizer; the Lemma 2 decomposition suggests the merge rule would become a comparison between a regularized variance term and the same Mahalanobis parameter distance.","Because the practical algorithm never defines or bounds plug-in estimators for $A_0$ and $B_0$, a finite-sample guarantee would need explicit $\\hat{A}_0-A_0$ and $\\hat{B}_0-B_0$ error bounds; until then, Algorithm 2's success-rate estimate is the only validation.","The greedy clustering in Algorithm 3 assumes pairwise signals are enough; a testable extension is whether a three-way merge can be beneficial when no two-way merge is, which would require a different clustering rule.","The representation-space experiments suggest the oracle inequality might serve as a general task-similarity score for transfer learning, where the 'coefficient' is a linear probe on shared features; this connection is not developed in the paper."],"forward_implications":["For two linear-regression datasets, the optimal choice is known exactly from population quantities: merge if $h(\\sigma^2)>g(\\beta^{(1)},\\beta^{(2)})$, and keep separate fits otherwise.","Algorithm 1 makes this choice with probability at least $1-5\\delta$ from samples, so the merge decision does not require knowing the true coefficients or noise level.","Algorithm 2 makes the procedure operational by tuning a confidence parameter on a grid and validating decisions on held-out data, and Algorithm 3 extends it to many datasets with a greedy clustering loop.","On real regression datasets and on neural-network penultimate-layer representations, the algorithm reduces reported out-of-sample error by roughly 17.5% to 99.7% relative to fitting each data partition separately.","For classification with linear models, the parallel result says the combined model's error bound beats separate fits when the distance between the true task parameters is small enough, making the merge criterion one of parameter closeness rather than mere data size."],"supporting_citations":[{"why":"Concentration bound for quadratic forms of sub-Gaussian vectors; underpins the high-probability surrogates $\\phi_\\delta$ and $\\psi_\\delta$.","marker":"[Hsu et al., 2011]"},{"why":"Gaussian anti-concentration inequality used to lower-bound the noise-side quantity in Lemma 6.","marker":"[Lovett, 2010]"},{"why":"Bounded-difference inequality behind the generalization bounds used in the classification merge criterion.","marker":"[McDiarmid, 1989]"},{"why":"Rademacher-complexity risk bounds that the classification analysis invokes.","marker":"Bartlett and Mendelson [2002]"},{"why":"Bootstrap central limit theorem that gives consistency of the estimated out-of-sample errors in Algorithm 2.","marker":"[Gill, 1989]"},{"why":"Grid-search calibration method that Algorithm 2 adopts for choosing the confidence parameter.","marker":"Arlot and Bach [2009]"},{"why":"The regression multi-task algorithm used as the experimental baseline for out-of-sample error reduction.","marker":"[Murugesan et al., 2017]"}],"fun_headline_variants":["Merge datasets when noise variance beats coefficient distance","Combine regressions only if noise outweighs coefficient gap","Dataset fusion criterion: noise variance vs Mahalanobis gap","When to join datasets: noise variance versus coefficient shift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The practical algorithm assumes that a few distribution-dependent constants appearing in the threshold inequality can be estimated from data without changing the decision, but the paper never defines those estimators or bounds their error, so the high-probability guarantee is proved for the distribution-known version only.","fun_headline_variants_meta":{"raw":{"variants":["Merge datasets when noise variance beats coefficient distance","Combine regressions only if noise outweighs coefficient gap","Dataset fusion criterion: noise variance vs Mahalanobis gap","When to join datasets: noise variance versus coefficient shift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3114,"prompt_tokens":993,"completion_tokens":2121,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":2058}},"tokens_in":609,"tokens_out":2121,"duration_ms":15280,"temperature":1.0,"reasoning_tokens":2058,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:12:06.887703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate two Gaussian linear populations with known $\\sigma^2$, $\\beta^{(1)}$, and $\\beta^{(2)}$; compute $h(\\sigma^2)-g(\\beta^{(1)},\\beta^{(2)})$ exactly from the population moments, then draw many datasets and compare Algorithm 2's merge decision with the sign of the true out-of-sample-error difference. If the agreement rate fails to match the claimed success rate, or if a case appears where the sample plug-in of $A_0$ and $B_0$ reverses the inequality, the practical guarantee is falsified.","supporting_citations":[{"cited_title":"A T ail I nequality for Q uadratic F orms of S ubgaussian R andom V ectors","cited_arxiv_id":null,"evidence_quote":"Concentration bound for quadratic forms of sub-Gaussian vectors; underpins the high-probability surrogates $\\phi_\\delta$ and $\\psi_\\delta$."},{"cited_title":"An E lementary P roof of A nti- C oncentration of P olynomials in G aussian V ariables","cited_arxiv_id":null,"evidence_quote":"Gaussian anti-concentration inequality used to lower-bound the noise-side quantity in Lemma 6."},{"cited_title":"On the M ethod of B ounded D ifferences","cited_arxiv_id":null,"evidence_quote":"Bounded-difference inequality behind the generalization bounds used in the classification merge criterion."},{"cited_title":"Rademacher and G aussian C omplexities: R isk B ounds and S tructural R esults","cited_arxiv_id":null,"evidence_quote":"Rademacher-complexity risk bounds that the classification analysis invokes."},{"cited_title":"Non- and S emi- P arametric M aximum L ikelihood E stimators and the von M ises M ethod","cited_arxiv_id":null,"evidence_quote":"Bootstrap central limit theorem that gives consistency of the estimated out-of-sample errors in Algorithm 2."},{"cited_title":"Data-driven C alibration of L inear E stimators with M inimal P enalties","cited_arxiv_id":null,"evidence_quote":"Grid-search calibration method that Algorithm 2 adopts for choosing the confidence parameter."}],"review_version":1}