{"id":"da10cfeb-820f-41ad-81bd-cac1dfb35ddd","arxiv_id":"2506.06853","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CEMS augments regression training by sampling from a second-order, curvature-aware local model of the joint input-output manifold, and reports competitive in-distribution and out-of-distribution results on nine benchmarks.","lead":"This paper introduces CEMS, a regression data augmentation method that creates new input-output pairs by sampling from a curved, second-order approximation of the data manifold instead of a flat tangent plane. It targets continuous prediction tasks, where classification-style augmentation such as mixup often produces invalid input-output pairs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The curvature estimate is non-identifiable at the implemented neighbor counts: with k=B, Eq. 6 has d(d+3)/2 unknowns, and for the reported d=10/12 (Crimes, SkillCraft) the system is underdetermined; the paper's own Sec.","rationale":"Both the reader and I target Eq. 6. The paper's central assertion requires H to be a genuine local curvature descriptor. Eq. 6 has p = d + d(d+1)/2 unknowns per normal coordinate; with k fixed to the batch size, identifiability requires k > p. Algorithm 2 sets k = B, and Sec. 6 admits the system may be underdetermined but never quantifies the failure. Using the paper's own intrinsic dimension estimates, the condition is violated in at least two reported experiments: Crimes (d=10, B=16, p=65) and SkillCraft (d=12, B=16, p=90). In these cases the least-squares/pseudoinverse solution is a minimum-norm point in the nullspace of Ψ, so the recovered Hessian can be an artifact of the solver, not of the manifold. App. D compounds this by counting only d(d+1)/2 unknowns, omitting the d gradient terms, and then resorting to ridge regularization. This does not disprove CEMS as an augmentation heuristic, but it means the 'second-order representation' narrative is not currently supported by the reported evidence. The DTI table inconsistency (Table K reports CEMS Avg R = 5.110 while Table 2 reports 0.511) is a separate reproducibility defect and reinforces the need for a clean re-run, but it is not the main load-bearing issue. The concrete audit I propose would settle whether the curvature mechanism is real: compare the recovered Hessian against known/true curvature and against an H≡0 ablation. Since the reader already conditioned acceptance on neighbor counts tied to d, my read does not move the verdict; it sharpens the required condition.","tokens_in":26619,"tokens_out":8027,"duration_ms":92768,"concrete_test":"Audit Hessian identifiability on the paper's own constant-curvature hypersphere setup (App. H) and on Crimes/SkillCraft: for each dataset, build actual kNN batches, construct Ψ with the reported intrinsic dimension d and k=B, compute rank(Ψ) and the minimum-norm Hessian H_B. Then retrain with k = d(d+3)/2 + 10 (an overdetermined system) and, in a third arm, with the quadratic term forced to zero (H≡0), keeping all other hyperparameters fixed. If the overdetermined H differs materially from H_B, or if H≡0 reproduces CEMS's reported Tables 1-2 performance, then the claimed second-order advantage is not supported by the implementation as described.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CEMS outperforms first-order/FOMA methods because it samples from a genuine second-order model of the data manifold. This requires the Hessian H recovered from Eq. 6 to be determined by the data rather than by the solver. In App. A, Ψ ∈ R^{k×(d+d(d+1)/2)}: for each normal coordinate there are d gradient unknowns plus d(d+1)/2 Hessian unknowns, so the system is overdetermined only when k > d(d+3)/2. Algorithm 2 sets the neighborhood size to the batch size B, and the paper's Sec. 6 explicitly admits the linear system 'might be underdetermined' while claiming that O(d^2) neighbors are needed. The reported experiments likely violate this condition: from Table 6, Crimes has d=10 with B=16 (65 unknowns > 16), and SkillCraft has d=12 with B=16 (90 unknowns > 16); RCF, Poverty, and DTI are never audited. For these configurations, X = Ψ†G is the minimum-norm solution, so H can be dominated by the nullspace of Ψ and reflect the solver's regularization rather than true curvature. App. D even misstates the unknown count as d(d+1)/2, omitting the d gradient terms, before conceding that the system is underdetermined and switching to ridge regression. The authors flag the limitation but never report, per dataset, whether the O(d^2) neighbor condition is met or how much the recovered Hessian changes when it is enforced. Without that evidence, the observed gains in Tables 1-2 could be produced by the implicit smoothing of an underdetermined solve rather than by the second-order mechanism that motivates the method.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CEMS, a second-order manifold-based data augmentation method for regression. It extends FOMA by estimating the gradient and Hessian of a local embedding map g in tangent/normal coordinates (Eq. 3 and Eq. 6), sampling a perturbation η in the tangent space, and un-projecting via the second-order Taylor map. The authors provide a theoretical error bound (Theorem A.1), a differentiable PyTorch pipeline, and experiments on nine datasets covering in-distribution and out-of-distribution settings, comparing against ERM, Mixup, manifold Mixup, C-Mixup, ADA, and FOMA. The central claim is that CEMS delivers superior performance while adding minimal computational overhead, because it captures curvature better than first-order methods.","tokens_in":26961,"tokens_out":5054,"duration_ms":49928,"significance":"If the claims hold, CEMS would be a useful, domain-independent regression augmentation tool that improves on first-order manifold sampling with modest extra cost. The paper contributes a clean algorithmic formulation, a public code release, a differentiable implementation, and extensive ablations over intrinsic dimension, noise scale, and batch size. However, the empirical evidence for the central 'superior performance' claim is currently weakened by the use of imported baseline numbers, selection of the best input/manifold configuration, and the lack of significance testing. The largest concern is that the Hessian estimation, which is the defining second-order component, is underdetermined at the reported neighborhood sizes for several datasets; without addressing this, the mechanism behind the observed gains remains unclear.","major_comments":[{"comment":"The load-bearing claim that CEMS samples from a genuine second-order model requires the Hessian in Eq. 6 to be identifiable from the data. The matrix Ψ in Eq. 6 has d(d+3)/2 columns per normal coordinate (d gradient unknowns plus d(d+1)/2 Hessian unknowns), so the system is overdetermined only if the number of neighbors k exceeds d(d+3)/2. Algorithm 2 sets k equal to the batch size B, and from Table 6 and Table 10, Crimes has d=10 with B=16 (65 unknowns > 16) and SkillCraft has d=12 with B=16 (90 unknowns > 16). For these configurations the least-squares solution X = Ψ†G is a minimum-norm or ridge solution, so the recovered Hessian is not determined by the data and may reflect solver regularization rather than true curvature. The paper acknowledges this possibility in Sec. 6 but does not report, per dataset, whether k > d(d+3)/2 holds, nor does it provide an ablation that enforces the condition. This is central to the abstract's claim that CEMS 'leverages a second-order representation of the data manifold.' Please report for every dataset the intrinsic dimension and the number of unknowns, and provide results with an overdetermined system (e.g., k = 2d(d+3)/2) or a stability analysis of the Hessian estimate under the ridge regularization used in App. D.","section":"Sec. 4, Eq. 6; Sec. 6; App. D"},{"comment":"The comparison protocol is not controlled. The paper states in Sec. 5.2 that 'the results of all previous methods are reported as they appear in the corresponding original papers,' while CEMS results are produced by the authors' own runs with their own hyperparameter choices. This mixes implementations, seeds, and tuning protocols. Additionally, App. I says that CEMS is applied in either input space or latent space and 'we report the configuration with the best performance'; if this choice is made using the test set or is not applied equally to baselines, it inflates the apparent advantage. The abstract's claim of 'superior performance' is therefore not supported by a fair comparison. Please rerun all baselines under the same codebase, seeds, and hyperparameter selection procedure, or provide paired significance tests; if the input/manifold choice is made on a validation set, report both configurations.","section":"Sec. 5.2 and 5.3; App. I"},{"comment":"Many of the headline improvements are within the reported standard deviations, and some comparisons are ties or favor the baseline. For example, Airfoil RMSE is CEMS 1.455±0.119 vs FOMA 1.471±0.047; Exchange-Rate RMSE is CEMS 0.014±0.001 vs FOMA 0.013±0.000; Electricity RMSE is a tie at 0.058 for both; and RCF average RMSE is CEMS 0.146±0.002 vs C-Mixup 0.146±0.005. With only three seeds, these differences are not statistically meaningful. The paper needs significance testing (e.g., paired bootstrap, confidence intervals, or a statement of the number of trials needed to differentiate the methods) before the claim of 'superior performance in both in-distribution and out-of-distribution scenarios' can be accepted.","section":"Tables 11 and 12; Sec. 5.2-5.3"}],"minor_comments":[{"comment":"The complexity analysis states 'l = d × (d + 1)/2 variables' for the linear system, omitting the d gradient unknowns; the correct count is d + d(d+1)/2. This is not just a typo because it obscures the underdetermination discussed in Sec. 6.","section":"App. D"},{"comment":"There are typos such as 'Ridgr-Regreession' and 'o par'; please proofread the appendix.","section":"App. D"},{"comment":"The DTI average R for CEMS is reported as '5.110 ± 0.005' while Table 2 shows 0.511; this appears to be a decimal-point typo and should be corrected.","section":"Table 12"},{"comment":"The text says the estimators show 'strong agreement' and cites that all three return 10 for Crimes, but for NO2 the estimates are TwoNN=6, PH=1, MLE=4. Please clarify or correct this statement.","section":"App. G.2, Table 6"},{"comment":"The main text says the system is solved via 'differentiable least squares,' while App. D says an underdetermined system is 'solved used Ridgr-Regreession.' Please specify which solver is actually used in the released code and in the reported experiments.","section":"Sec. 4, Step 5"}],"recommendation":"major_revision","confidential_remarks":"The authors are from the same group as the FOMA baseline, so the FOMA comparison deserves extra care; however, the main issue is technical rather than motivational. The underdetermined Hessian problem and the uncontrolled baseline comparison are both fixable within the manuscript's scope, so I do not recommend rejection. Please also verify the Table 12 typo before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on regression data augmentation. The genuinely new piece is CEMS, a second-order (curvature-aware) version of the authors' own FOMA: it fits a local Taylor expansion with a Hessian term via CAML-style local coordinates and samples from that. That is a real, if incremental, extension, and the paper does several things well. The algorithm is clearly described, the code is public, the ablations on intrinsic dimension, noise scale, batch size, and per-point vs. shared basis are useful, and the authors are upfront in Section 6 that the linear system in Eq. 6 may be underdetermined. The toy sine example is illustrative. Credit where due: this is an honest, well-structured paper, not a hype job.\n\nThe soft spots, in proportion. First, the central mechanistic claim is not actually supported. The Hessian is recovered by solving G = ΨA with k neighbors. With k set to the batch size B, the system has d(d+3)/2 unknowns (d gradient terms plus d(d+1)/2 Hessian terms). For Crimes, d=10 and B=16, so 65 unknowns against 16 equations; SkillCraft has d=12 and B=16, 90 unknowns. The system is underdetermined in exactly the reported experiments, and the minimum-norm or ridge solution is then dominated by the solver's regularization, not by true curvature. The paper admits this, but it never reports, per dataset, whether k > d(d+3)/2 or how much the recovered Hessian changes when the condition is enforced. That means the observed gains over FOMA could come from the implicit smoothing of an underdetermined solve. This is the load-bearing weakness, and the stress-test note lands on it correctly.\n\nSecond, the experimental comparisons are weaker than the abstract claims. Baselines are imported from original papers rather than re-run in one codebase, the best of input-space vs. manifold-space is reported per dataset (App. I), there are no significance tests, and there are several near-ties with FOMA. The DTI row in Table K shows CEMS average R as 5.110 instead of 0.511, a typo that should have been caught. These are fixable but they currently inflate the impression of dominance.\n\nAll that said, the method is plausible and the failure mode is diagnosed by the authors themselves, even if not quantified. A serious referee should engage with it, but the revision bar is real: re-run baselines in one framework, report neighbor counts versus d(d+3)/2 per dataset, and ideally show the Hessian is identifiable or say plainly that the benefit is regularization rather than curvature. For a reader in the regression-DA niche, this is a useful incremental paper; for the broader community, the second-order story needs more evidence. I'd send it to review and push for those fixes.","headline":"A coherent second-order extension of FOMA with a real identifiability problem in the Hessian estimate and a performance claim that currently outruns the evidence.","tokens_in":27516,"tokens_out":1456,"would_cite":false,"duration_ms":19001,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that augmenting regression data by sampling from a locally curved manifold—not just its tangent space—keeps synthetic points on the data structure and improves in-distribution and out-of-distribution performance.","keywords":["data augmentation","regression","manifold learning","curvature","second-order approximation","tangent space sampling","out-of-distribution generalization","CEMS"],"falsifier":"On a synthetic manifold with analytically known curvature, set the intrinsic dimension $d$ high enough that the fixed batch size satisfies $b < d(d+1)/2$, then compare the Hessian recovered by CEMS with the true Hessian. If the recovered Hessian is far from the true one yet CEMS still improves over first-order sampling, the improvement is not coming from curvature; if the Hessian is wrong and CEMS matches first-order performance, the method's core mechanism is falsified.","tokens_in":26385,"feed_emoji":"📈","tokens_out":9951,"duration_ms":96913,"temperature":0.7,"pith_summary":"The paper argues that data augmentation for regression should be treated as a problem of sampling from a curved, low-dimensional data manifold, and that the first-order tangent-space approximation used by earlier manifold-based augmentation leaves error proportional to the squared sampling radius. CEMS therefore fits a second-order Taylor expansion of the manifold locally, estimates its gradient and Hessian from the neighborhood of each point, and uses that expansion to generate new input-output pairs. The paper claims this yields better in-distribution and out-of-distribution regression performance than mixing-based and first-order augmentation methods across four in-distribution and five out-of-distribution benchmarks, with run time comparable to first-order methods. The practical value would be a domain-independent, differentiable regularizer that follows curved structure instead of cutting across it.","feed_headline":"Curvature-aware manifold sampling wins regression augmentation tests","feed_subtitle":"CEMS adds local curvature to synthetic regression samples and tops or ties state-of-the-art on nine benchmarks.","key_machinery":"The load-bearing object is the second-order Taylor parameterization $g: T_u\\mathcal{M} \\to N_u\\mathcal{M}$ of the joint input-output manifold. At each point $z$, nearest neighbors provide a local frame through SVD; tangent coordinates $u$ and normal coordinates $g$ are separated, and the normal-coordinate map is expanded as $g(u_j) = u_j^\\top \\nabla g + \\frac{1}{2} u_j^\\top H u_j$. The paper forms the linear system $G = \\Psi X$ whose unknowns are the gradient and upper-triangular Hessian, solves it by differentiable least squares, and uses the resulting quadratic map to un-project sampled tangent points back to the ambient space. The machinery converts local curvature into a generative map rather than treating curvature only as a descriptive feature.","core_discovery":"CEMS is a second-order manifold sampling method. Treating each concatenated input-output pair as a point on a joint manifold, it builds a local orthonormal basis from the nearest neighbors via SVD, projects the neighborhood into tangent and normal coordinates, solves a least-squares system for the gradient and Hessian of the normal-coordinate map, and then draws Gaussian samples in the tangent space and maps them back through the quadratic expansion. The paper's central claim is that this curvature correction keeps synthetic points on or near the true data manifold where first-order sampling drifts, and that this geometric fidelity translates into consistently lower regression error, including under distribution shift. In the paper's own framing, the prior first-order method FOMA is a special case of CEMS obtained by dropping the second-order term and rescaling the normal coordinates. The provided error bounds give the mechanism: first-order approximations err as $O(\\|u-u_0\\|^2)$, while CEMS's second-order approximation errs as $O(\\|u-u_0\\|^3)$ under a Lipschitz-Hessian smoothness assumption.","pith_inferences":["The underdetermined linear system identified in the conclusion suggests a concrete extension: regularize the Hessian solve with ridge regression and check whether CEMS degrades gracefully as intrinsic dimension grows; the paper names this as future work rather than a tested claim.","The error-bound comparison predicts that CEMS's advantage over first-order sampling should grow with local curvature; the appendix's synthetic curvature experiment already shows that trend, which could be used to select between first- and second-order sampling per point.","The shared-neighborhood batch approximation makes the method sensitive to how batches are formed, and the appendix shows random, k-nearest-neighbor, and probability-weighted batch selection each win on different datasets, so neighborhood strategy is itself a tuning choice the authors leave open.","Nothing in the method limits it to regression; the joint manifold could in principle be built over inputs and discrete labels, though the standard manifold hypothesis is less natural for categorical label spaces."],"forward_implications":["If CEMS is correct, synthetic regression samples should stay closer to the data manifold than first-order samples in regions of high curvature, so augmentation expands the training distribution without adding off-manifold points.","On the in-distribution benchmark, CEMS would be a reliable drop-in augmentation for tabular and time-series regressors, taking best or second-best RMSE and MAPE on all four datasets.","On the out-of-distribution benchmark, CEMS would also improve robustness to domain shift, achieving the best result in 6 of 9 average and worst-domain metrics and an 8% relative improvement on the SkillCraft worst-domain error.","Because the augmentation pipeline is fully differentiable and domain-independent, the same sampling rule can run in input space or in a learned latent space without changing the algorithm.","The cost analysis implies the curvature correction is not a computational barrier: under the manifold hypothesis $d \\ll D$, per-batch complexity is $O(b^2 D)$, proportional to ambient dimension."],"supporting_citations":[{"why":"Supplies FOMA, the first-order tangent-space augmentation that CEMS generalizes and the direct baseline in most comparisons.","marker":"(Kaufman & Azencot, 2024b)"},{"why":"Supplies the curvature-aware manifold learning parameterization and the least-squares system from which CEMS estimates gradient and Hessian.","marker":"(Li, 2018)"},{"why":"Defines the in-distribution and out-of-distribution regression benchmarks, datasets, baselines, and training settings CEMS is evaluated on.","marker":"(Yao et al., 2022)"},{"why":"Provides the Riemannian geometry definitions of tangent and normal spaces and the Taylor expansion used to build the local model.","marker":"(Lee, 2012)"},{"why":"Provides the TwoNN estimator used to set the intrinsic dimension $d$, a key hyperparameter.","marker":"(Facco et al., 2017)"},{"why":"Source of the error-bound theorem that gives CEMS $O(\\|u-u_0\\|^3)$ sampling error versus $O(\\|u-u_0\\|^2)$ for first-order methods.","marker":"(Fowkes et al., 2013)"}],"fun_headline_variants":["Second-order manifold sampling wins regression tests","Curvature boost for regression data augmentation","CEMS: curvature-enhanced sampling for regression","Sampling with curvature beats regression SOTA","Improved regression via curved manifold sampling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recovered Hessian is assumed to be a meaningful local curvature estimate even though the number of neighbors is fixed at the batch size, which can be far below the $O(d^2)$ neighbors needed to make the least-squares system overdetermined.","fun_headline_variants_meta":{"raw":{"variants":["Second-order manifold sampling wins regression tests","Curvature boost for regression data augmentation","CEMS: curvature-enhanced sampling for regression","Sampling with curvature beats regression SOTA","Improved regression via curved manifold sampling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000348,"raw_usage":{"total_tokens":1915,"prompt_tokens":969,"completion_tokens":946,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":883}},"tokens_in":585,"tokens_out":946,"duration_ms":10182,"temperature":1.0,"reasoning_tokens":883,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:48:05.664459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic manifold with analytically known curvature, set the intrinsic dimension $d$ high enough that the fixed batch size satisfies $b < d(d+1)/2$, then compare the Hessian recovered by CEMS with the true Hessian. If the recovered Hessian is far from the true one yet CEMS still improves over first-order sampling, the improvement is not coming from curvature; if the Hessian is wrong and CEMS matches first-order performance, the method's core mechanism is falsified.","supporting_citations":[{"cited_title":"C-mixup: Improving generalization in regression","cited_arxiv_id":null,"evidence_quote":"Defines the in-distribution and out-of-distribution regression benchmarks, datasets, baselines, and training settings CEMS is evaluated on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Riemannian geometry definitions of tangent and normal spaces and the Taylor expansion used to build the local model."},{"cited_title":"Estimating the intrinsic dimension of datasets by a minimal neighborhood information","cited_arxiv_id":null,"evidence_quote":"Provides the TwoNN estimator used to set the intrinsic dimension $d$, a key hyperparameter."},{"cited_title":"M., Gould, N","cited_arxiv_id":null,"evidence_quote":"Source of the error-bound theorem that gives CEMS $O(\\|u-u_0\\|^3)$ sampling error versus $O(\\|u-u_0\\|^2)$ for first-order methods."}],"review_version":1}