Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings

Lucas Mattioli; Martin Gonzalez; Sabrina Chaouche; Youness Ait Hadichou

REVIEW 4 major objections 5 minor 20 references

Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Training tabular classifiers on uncurated text embeddings can silently collapse them into single-class prediction, and the collapse manufactures a spurious Accuracy-on-the-Line correlation that falsely reads as OOD predictability.

desk verdict A real and easily reproduced failure mode for LLM-embedding tabular pipelines, with the caveat that the headline prevalence number depends on hyperparameter grids inherited from raw-tabular training. read the letter →

arxiv 2506.17989 v1 pith:JAQKJF4G submitted 2025-06-22 cs.LG

Lucas Mattioli , Youness Ait Hadichou , Sabrina Chaouche , Martin Gonzalez This is my paper

classification cs.LG

keywords modelcollapsetextembeddingstabulardataout-of-distributiongeneralizationAccuracy-on-the-LinedistributionshiftcurationLLM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that using LLM text embeddings as a data-curation layer for tabular data can instead trigger model collapse: with the same hyperparameter settings that work on raw tabular columns, classifiers trained on 4096-dimensional row embeddings often predict a single class for every input. It reports this failure across four embedding models, seven classifier families, and 4,615 configurations, and introduces collapse-ratio metrics that measure how many shared hyperparameter settings degenerate. The deeper claim is that strong collapse manufactures a spurious Accuracy-on-the-Line correlation: the $R^2$ between in-distribution and out-of-distribution accuracy rises not because performance transfers but because collapsed models all pile onto the same class-ratio point. Anyone who uses a high Accuracy-on-the-Line $R^2$ to predict out-of-distribution accuracy from in-distribution accuracy would therefore be silently misled, which is why the paper insists that collapse severity must be reported alongside any such correlation. It also finds that leaderboard rankings of embedding models do not predict how well an embedding serves as a curation layer under tabular distribution shift.

What carries the argument

The central instrument is the bound-pair on-the-line plane. For each hyperparameter configuration $c$, the authors train one model on raw tabular data and a twin model on the LLM embeddings of the same rows, then plot both in the plane whose axes are in-distribution and out-of-distribution scores, binding each pair with a straight segment. Collapse is defined by counting predictions: a model is collapsed on a test set if it predicts only one class, and near-collapsed if its true-negative plus false-negative rates fall below a threshold $\varepsilon$. The collapse ratio $\mathrm{CR}^{S}_{HP}$ is the fraction of configurations in a hyperparameter set (chosen non-collapsing on raw tabular data) that become single-class predictors when trained on embeddings; the strong-collapse ratio $\mathrm{CR}_s$ and projection ratio $\mathrm{CR}_p$ count configurations collapsing on both test sets or on at least one. These ratios expose a spurious Accuracy-on-the-Line: when many embedding-trained models stack on the class-ratio point, the fitted $R^2$ rises while the non-collapsed remainder shows no linear trend.

What would settle it

Retrain every collapsing configuration on the same embeddings using a hyperparameter grid designed for high-dimensional inputs, such as lower learning rates, stronger regularization, weight decay, or embeddings projected to a matched dimension, and recompute the collapse ratios. If $\mathrm{CR}_s$ falls to near zero across all four encoders, the reported collapse is a grid-mismatch artifact rather than a property of text embeddings. A second check is to run the identical bound-pair analysis on random Gaussian features of dimension 4096; if those collapse at comparable rates, the failure is about dimensionality and optimization, not about the semantic content of embeddings.

Watch

Extended reading notes

Core claim

The central discovery is that model collapse is a characteristic failure mode of training on uncurated text embeddings, and that this collapse contaminates the standard diagnostic used to claim out-of-distribution predictability. When a hyperparameter configuration that produces a functioning classifier on raw tabular data is reused on LLM-embedded rows, the resulting model frequently predicts one class regardless of input, so its in-distribution and out-of-distribution accuracies both sit at the test set's positive-class ratio. Many such collapsed points overlap in the Accuracy-on-the-Line plane, mechanically raising the $R^2$ of the linear fit even though the surviving models follow no discernible line; the paper exhibits a case where collapsed models yield $R^2 \approx 0.38$ on accuracy and $0.87$ on macro-F1 while the non-collapsed points are scattered. The authors conclude that Accuracy-on-the-Line, and OOD performance prediction built on it, is only meaningful when collapse is measured and excluded, and they frame the collapse ratio as the diagnostic that should accompany any such claim.

Load-bearing premise

The load-bearing premise is that the hyperparameter configurations chosen for raw tabular models are a fair test of the embedding-based pipeline; if those same settings are simply a poor match for 4096-dimensional inputs, the observed single-class predictions could be an artifact of the grid rather than a property of uncurated text embeddings.

Editorial extensions

If this is right

Every reported Accuracy-on-the-Line $R^2$ for an embedding-trained model should be read alongside a collapse ratio; a high $R^2$ with a high strong-collapse ratio is evidence of degeneracy, not of transfer.
Pipelines that predict out-of-distribution accuracy from in-distribution accuracy should reject the prediction whenever models already collapse on the source domain, which the paper shows happens with text embeddings.
Embedding leaderboard rankings are not a proxy for data-curation quality under tabular shift, since the lowest-ranked of the four encoders tested was the most robust on this task.
Using a different encoder at test time than at training time is itself a distribution shift that provokes collapse, even for configurations that behave well on embeddings from the training encoder.
Test-time adaptation methods, including entropy minimization and a tabular-specific variant, do not restore a valid on-the-line relation for embedding-trained models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension: check whether the collapse ratio tracks the intrinsic dimension or spectral structure of the embedding space; if it does, the metric becomes a cheap, OOD-label-free way to pick between embedding models.
The same bound-pair collapse analysis could be applied to vision or audio features; if single-class collapse appears there too, the phenomenon is a general property of training on high-dimensional pretrained representations rather than something specific to text.
The near-collapse threshold $\varepsilon$ is chosen per experiment after the fact; a fixed, pre-registered threshold would make the collapse ratios usable as a transferable benchmark metric, as the paper's appendix leaves that choice open.
The spurious-ACL mechanism implies an audit rule for earlier work: any accuracy-on-the-line claim computed over a set where several models sit exactly at the class-ratio point deserves discounting, because those points raise $R^2$ by geometry rather than by semantics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a look. The paper documents something concrete: train a model on LLM text embeddings of a tabular dataset, using hyperparameter configurations that work fine on the raw table, and a large fraction of those models collapse to predicting a single class. That collapse then can inflate the R² of Accuracy-on-the-Line, giving a false signal that OOD performance is predictable. The authors show this across four LLMs and several US states on ACS Income, and they introduce simple collapse-ratio metrics that are easy to compute. The observation that high-dimensional or poorly scaled features can yield constant classifiers is not new, but the documentation for modern text embeddings on tabular data, and the connection to ACL inflation, is a useful addition to the LLM-embeddings-for-tabular literature.

The main soft spot is the evidence base. It is one binary dataset, one run per configuration, no error bars, and no code or data released. That alone would make me hesitant to over-generalize. The deeper issue is the shared hyperparameter grids: they come from raw-tabular pipelines, and 4096-dimensional embedding inputs are a different regime. A 16-unit MLP with a small learning rate on 4096-d inputs can underfit trivially. The stress-test note worries that the high collapse ratios in Section 4.1 could be a grid-mismatch artifact rather than an intrinsic property of text embeddings. That concern is legitimate, but the paper partially answers it in Section 4.2: the tuned configuration from Zeng et al. avoids collapse on the same LLM's embeddings but still collapses when transferred to other LLMs' embeddings. So collapse is not purely an artifact of a bad grid — it also appears with a good grid when the embedding source changes. What remains unresolved is how prevalent collapse would be under embedding-adapted hyperparameter search. The authors should run that control before claiming collapse is a consistent failure mode.

The spurious ACL claim is supported by visual examples and by the logic that a pile of overlapping collapsed points inflates R², but there is no formal comparison of R² before and after removing collapsed points. That would be easy to add. The epsilon for near-collapse is chosen per experiment; a sensitivity check would help.

Overall, the paper is honest about its scope and frames itself as an initial investigation. It deserves a serious referee. I would not desk-reject it. It needs a control with embedding-adapted hyperparameters, error bars or multiple runs, and code/data release. After that, the central caution — that uncurated text embeddings can silently produce degenerate models and misleading on-the-line correlations — is likely to hold, at least as a warning for practitioners.

Referee Report

4 major / 5 minor

Summary. The paper studies binary tabular classification from the ACS Income dataset, comparing models trained on raw tabular features with models trained on 4096-dimensional text embeddings produced by four off-the-shelf LLMs under identical hyperparameter configurations. It defines collapse metrics based on the fraction of configurations that yield single-class predictors, reports that such collapse is common in embedding-trained models, and argues that strong collapse can artificially inflate Accuracy-on-the-Line (ACL) R² values, producing a misleading signal of OOD predictability. It also compares MTEB rankings with downstream OOD performance and studies cross-LLM embedding transfer. The experiments cover seven model families, four encoders, and five target states, with additional test-time adaptation results.

Significance. If the central empirical claim is confirmed, the paper identifies a practically important failure mode: using general-purpose text embeddings as a data-curation layer for tabular data can produce degenerate single-class predictors, and the resulting ACL correlations are not trustworthy evidence of OOD generalization. The proposed collapse metrics are simple, interpretable, and potentially reusable, and the paper is honest that the study is an 'initial investigation' and 'far from being comprehensive' (Section 4, Conclusion). However, the evidence currently rests on one binary dataset, a single training state, one run per configuration, and hyperparameter grids inherited from raw-tabular pipelines; these limitations are load-bearing for the generality of the abstract's claims. The paper would be strengthened by a formal link between collapse ratios and R² inflation, by an embedding-adapted hyperparameter control, and by additional datasets or seeds.

major comments (4)

[Section 4.1, Table 6] The central comparison assumes that the hyperparameter grids designed for raw tabular models are a fair baseline for 4096-dimensional embeddings. The grids in Table 6 come from low-dimensional tabular pipelines (e.g., MLP hidden sizes 16-128, LR regularization on the raw feature scale), with no feature normalization or dimension-aware regularization for the embedding space. Under such a mismatch, trivial majority-class convergence is expected. This concern is not hypothetical: Section 4.2 shows that the single configuration c2, tuned on e5 embeddings, avoids collapse on all e5-encoded states but collapses when transferred to other LLMs' embeddings, demonstrating extreme sensitivity to the HP-input match. To support the claim that uncurated text embeddings inherently induce collapse, the authors should run a control with an embedding-adapted search (e.g., standardized embeddings, larger capacity, dimension-aware regularization) and report the resulting collapse ratios; if those ratios are near zero, the main claim is not established.
[Section 4.1, 'Strong Model collapse induces spurious ACL'] The claim that strong collapse induces spurious ACL is supported by visual inspection of Figure 2b rather than by a formal comparison. The paper states that the R² of the green points is 'significantly higher' and attributes this to overlapping collapsed points, but it does not quantify the effect. Please compute R² on the non-collapsed subset of configurations, or report the difference between R² with and without collapsed points across all LLM/model/state combinations, and relate that difference to the collapse ratios. Without such a test, the warning about spurious ACL remains an illustrative observation rather than an established property.
[Section 4.1, Section 3.4] The collapse metrics are defined only for 'well-defined' hyperparameter sets, i.e., sets that do not collapse on raw tabular data. However, Table 6 marks XGB and RF as not well-defined, and Section 4.1 still reports collapse patterns and ratios for these families (including the RF discussion and Figure 13). This is an internal inconsistency in the metric-based evidence. Either restrict the collapse-ratio analysis to the well-defined subsets for XGB and RF, or formally extend the definitions to cover non-well-defined baseline sets and state the implications for interpretation.
[Section 4.1, Experimental setup] The abstract and Section 4.1 claim that collapse is a 'consistent failure mode' across LLM-embedding training, but the evidence is from a single binary dataset (ACS Income), a single training state (CA), a single LLM per embedding modality, and one run per configuration with no error bars. The paper's own wording concedes an 'initial investigation' and prioritizes 'qualitative insights,' which is appropriate for a workshop-scale study but not sufficient for the generality claimed in the abstract. At minimum, add a second tabular dataset and a small number of seeds to quantify variance, or revise the claims to be explicitly scoped to the ACS Income setup.

minor comments (5)

[Section 3.2] Typo: 'pics' should be 'picks' in the sentence 'Our method pics a set of training Hyper-Parameter Configurations'.
[Section 3.4] Typo: 'de ratio of configurations' should be 'the ratio of configurations'.
[Appendix A.2] The near-collapse threshold epsilon is stated to 'vary from one experiment to the next' and is chosen 'with consideration to the overall shape' of the scatter. Since epsilon is a free parameter, please report sensitivity of the near-collapse ratios to epsilon or fix a single value across all experiments and justify it.
[Table 6] For methods with grids larger than 200, the paper says 200 configurations are randomly sampled, but no random seed is reported; the stability of the collapse ratios under this sampling should be discussed.
[Section 3.3] The sentence 'one should not be overlooked the fact that...' is grammatically awkward and should be rephrased for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical measurement using explicitly defined descriptive metrics, and no prediction reduces to its inputs by construction.

full rationale

The paper's central claims are empirical findings, not derivations. The collapse metrics in Section 3.4 are definitional ratios (e.g., CR_S_HP = 1/|HP| sum of indicator functions over strict single-class predictions), and the reported ACL R2 values are computed directly from measured ID/OOD accuracy coordinates. No fitted parameter is later renamed as a prediction, and no stated result is equivalent to an input by construction. The comparison with raw tabular models uses shared hyperparameter configurations as a controlled baseline; any concern that those grids are poorly matched to 4096-dimensional embeddings is a validity or correctness threat, not a circularity, because the paper does not claim to derive collapse from the grids and does not fit anything to force the collapse counts. The only experimenter-chosen quantity is the near-collapse epsilon, and the paper explicitly acknowledges that 'The value of ε varies from one experiment to the next' (Appendix A.2), while also reporting strict collapse counts that do not depend on epsilon. Citations to Zeng et al. (2024) supply the embedding procedure, hyperparameter grids, and the reference optimal MLP configuration; these are external prior works with no author overlap with the present paper, so they are independent evidence rather than self-citation. The paper also explicitly frames itself as an initial investigation 'picked to exemplify our findings rather than to propose exhaustive numerical experiments,' which is an acknowledged scope limitation, not circular reasoning. Overall, the manuscript is self-contained as an empirical study against external data and benchmarks, so no circular step is present.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new entities are introduced. The free parameters are limited to the near-collapse epsilon and the inherited hyperparameter grids. The main assumptions are the reliance on the prior embedding pipeline and the fairness of the shared HP baseline.

free parameters (1)

epsilon (near-collapse threshold) = 0.1 (varies per experiment)
Used to define near-collapse ratios; the paper states the value should be chosen with consideration of the scatter shape, so it is a hand-set threshold rather than a pre-registered constant.

assumptions (3)

domain assumption The embedding and serialization pipeline of Zeng et al. (2024) is correctly re-implemented and its hyperparameter grids are appropriate for both modalities.
The paper defers the Tab2Text serialization and the exact prompts to Zeng et al., so the results inherit any assumptions in that pipeline.
standard math Macro F1 over the two classes is a suitable companion metric to accuracy for imbalanced binary shifts.
The F1L plane relies on this choice; it is standard but not the only option.
ad hoc to paper The 'well-defined' HP sets (configs that do not collapse on raw tabular data) are a fair reference set for measuring embedding-induced collapse.
This filter excludes some configurations and therefore shapes the collapse ratios; it is a modeling choice specific to this paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings." pith.science (2026). https://pith.science/paper/JAQKJF4G

@misc{pith2026250617989,
  author       = {Pith},
  title        = {Pith review of: Data Curation Matters: Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JAQKJF4G}},
  note         = {Machine review of arXiv:2506.17989}
}

read the original abstract

Training models on uncurated Text Embeddings (TEs) derived from raw tabular data can lead to a severe failure mode known as model collapse, where predictions converge to a single class regardless of input. By comparing models trained with identical hyper-parameter configurations on both raw tabular data and their TE-derived counterparts, we find that collapse is a consistent failure mode in the latter setting. We introduce a set of metrics that capture the extent of model collapse, offering a new perspective on TE quality as a proxy for data curation. Our results reveal that TE alone does not effectively function as a curation layer - and that their quality significantly influences downstream learning. More insidiously, we observe that the presence of model collapse can yield artificially inflated and spurious Accuracy-on-the-Line correlation. These findings highlight the need for more nuanced curation and evaluation of embedding-based representations, particularly in out-of-distribution settings.

Figures

Figures reproduced from arXiv: 2506.17989 by the authors.

**Figure 1.** Illustration of our results. We train models with shared hyper-parameter configuration on raw tabular data and their LLMbased embedded counterparts. We exhibit model collapse to single class prediction both in ID & OOD test-sets and observe a rise of spurious correlation between ID-OOD accuracy (i.e. Accuracy-onthe-Line), as a consequence of model collapse severity. Liu et al., 2023; Gorishniy et al., 2021; Shwart… view at source ↗

**Figure 2.** Different patterns as exemplified by different combinations of LLMs, models and target states. All figures are reported in Subsection A.5 in the Appendix. All tested LLMs induce model collapse and, while some combinations of LLMs, model family and (source, target) states show no model collapse, there is no model family not exhibiting model collapse for all tested states for any tested LLM. Moreover, such collapses d… view at source ↗

**Figure 3.** (a) An example of a valid ACL and F1L; (b,c,d) Test-Time adaptation methods fail to strengthen ACL or F1L. are both pair-wise strong collapses as well as projection collapses so that CRs = 18, 23% and CRp = 43, 84%. Last, but not least, all figures for RF in the appendix show that the set HPRF is not well-defined w.r.t. CA. in the sense of this article. In other words, there are already models collapsing over tabula… view at source ↗

Figures from the paper (40 more)

**Figure 5.** Figure 5: Pattern behavior across different LLMs for MLP. The MTEB ranking of LLMs is not correlated with their OOD Generalization capabilities. To address Question 4, we train MLPs fc2 for the optimal reported configuration from (Zeng et al., 2024) over the 4 LLM embeddings of …

**Figure 6.** Figure 6: Model behavior across LLM Embeddings. (a) Results for fc1 ; b) (resp. c)) Accuracy (resp. F1-score) for fc2 . we conducted an exhaustive pairwise comparison across all LLMs, hyper-parameter configurations for all families, and all combinations of train and target domai…

**Figure 7.** Figure 7: Fraction Best across all states with ∆ = 0%. The results of FractionBest w.r.t. accuracy reveal striking differences in performance across encoders. All embedding based models outperform traditional methods like XGBoost (6%), while traditional methods like Logistic Reg…

**Figure 8.** Figure 8: Pattern behavior across different LLMs Embeddings for CA on the MLP HP Configuration Set. other states; for instance, for cfg-1, the MLP with a hidden size of 16 was trained for 200 epochs on the embeddings of CA with a lr = 0.01 and a dropout ratio of 0.1 then was tes…

**Figure 9.** Figure 9: Pattern behavior across different LLMs for (CA,AZ). ACL and F1L are valid in tabular training but do not transfer into valid ACL or F1L in LLM-based representation training. Let ε > 0. We define the positive ε-near collapse ratio of HP w.r.t. S is defined as PCRS HP(ε)…

**Figure 10.** Figure 10: Pattern behaviour across different LLMs for MLP. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]

**Figure 11.** Figure 11: Pattern behaviour across different LLMs for CVaR-DRO. 0.76 0.78 0.80 0.82 0.84 0.86 0.70 0.75 0.80 0.85 0.90 Acc.‑on‑the‑line: e5|lr (CA→PR) Std. Tr., R 2=0.006 Rep. Tr., R 2=0.793 GT: -50k y = x 0.74 0.76 0.78 0.80 0.82 0.84 0.50 0.55 0.60 0.65 0.70 F1‑on‑the‑line: e…

**Figure 12.** Figure 12: Pattern behaviour across different LLMs for LR. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]

**Figure 13.** Figure 13: Pattern behaviour across different LLMs for RF. 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 Acc.‑on‑the‑line: e5|svm (CA→PR) Std. Tr., R 2=0.036 Rep. Tr., R 2=0.470 GT: -50k y = x 0.79 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.20 0.30 0.…

**Figure 14.** Figure 14: Pattern behaviour across different LLMs for SVM. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

**Figure 15.** Figure 15: Pattern behaviour across different LLMs for GBM. 0.40 0.50 0.60 0.70 0.80 0.90 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90 Acc. on the line: e5|xgb (CA PR) Std. Tr., R 2=0.000 Rep. Tr., R 2=0.002 GT: -50k y = x 0.30 0.40 0.50 0.60 0.70 0.80 0.90 0.10 0.20 0.30 0.40 0…

**Figure 16.** Figure 16: Pattern behaviour across different LLMs for XGB. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png]

**Figure 17.** Figure 17: Pattern behaviour across different LLMs for MLP. 0.65 0.70 0.75 0.80 0.85 0.68 0.70 0.72 0.74 0.76 0.78 0.80 Acc.‑on‑the‑line: e5|cvar_dro (CA→AL) Std. Tr., R 2=0.005 Rep. Tr., R 2=0.948 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 F…

**Figure 18.** Figure 18: Pattern behaviour across different LLMs for CVaR-DRO. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_18.png]

**Figure 19.** Figure 19: Pattern behaviour across different LLMs for LR. 0.65 0.70 0.75 0.80 0.85 0.90 0.45 0.50 0.55 0.60 0.65 0.70 0.75 Acc. on the line: e5|rf (CA AL) Std. Tr., R 2=0.956 Rep. Tr., R 2=0.003 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.40 0.45 0.50 0.55 0.60 0.65 0.70 F1 …

**Figure 20.** Figure 20: Pattern behaviour across different LLMs for RF. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_20.png]

**Figure 21.** Figure 21: Pattern behaviour across different LLMs for SVM. 0.50 0.60 0.70 0.80 0.90 0.30 0.40 0.50 0.60 0.70 0.80 Acc. on the line: e5|gbm (CA AL) Std. Tr., R 2=0.942 Rep. Tr., R 2=0.047 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.30 0.40 0.50 0.60 0.70 F1 on the line: e5|gb…

**Figure 22.** Figure 22: Pattern behaviour across different LLMs for GBM. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_22.png]

**Figure 23.** Figure 23: Pattern behaviour across different LLMs for XGB. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_23.png]

**Figure 24.** Figure 24: Pattern behaviour across different LLMs for MLP. 0.65 0.70 0.75 0.80 0.85 0.65 0.68 0.70 0.72 0.75 0.78 0.80 Acc.‑on‑the‑line: e5|cvar_dro (CA→AZ) Std. Tr., R 2=0.026 Rep. Tr., R 2=0.986 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 F…

**Figure 25.** Figure 25: Pattern behaviour across different LLMs for CVaR-DRO. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_25.png]

**Figure 26.** Figure 26: Pattern behaviour across different LLMs for LR. 0.65 0.70 0.75 0.80 0.85 0.90 0.55 0.60 0.65 0.70 0.75 0.80 Acc. on the line: e5|rf (CA AZ) Std. Tr., R 2=0.964 Rep. Tr., R 2=0.093 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 F1 …

**Figure 27.** Figure 27: Pattern behaviour across different LLMs for RF. 27 [PITH_FULL_IMAGE:figures/full_fig_p027_27.png]

**Figure 28.** Figure 28: Pattern behaviour across different LLMs for SVM. 0.50 0.60 0.70 0.80 0.90 0.40 0.50 0.60 0.70 0.80 Acc. on the line: e5|gbm (CA AZ) Std. Tr., R 2=0.963 Rep. Tr., R 2=0.027 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.40 0.50 0.60 0.70 0.80 F1 on the line: e5|gbm (CA…

**Figure 29.** Figure 29: Pattern behaviour across different LLMs for GBM. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_29.png]

**Figure 30.** Figure 30: Pattern behaviour across different LLMs for XGB. 29 [PITH_FULL_IMAGE:figures/full_fig_p029_30.png]

**Figure 31.** Figure 31: Pattern behaviour across different LLMs for MLP. 0.65 0.70 0.75 0.80 0.85 0.60 0.63 0.65 0.68 0.70 0.73 0.75 0.78 Acc.‑on‑the‑line: e5|cvar_dro (CA→AK) Std. Tr., R 2=0.070 Rep. Tr., R 2=0.994 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0…

**Figure 32.** Figure 32: Pattern behaviour across different LLMs for CVaR-DRO. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_32.png]

**Figure 33.** Figure 33: Pattern behaviour across different LLMs for LR. 0.65 0.70 0.75 0.80 0.85 0.90 0.40 0.45 0.50 0.55 0.60 0.65 0.70 0.75 Acc. on the line: e5|rf (CA AK) Std. Tr., R 2=0.966 Rep. Tr., R 2=0.020 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.40 0.45 0.50 0.55 0.60 0.65 0.7…

**Figure 34.** Figure 34: Pattern behaviour across different LLMs for RF. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_34.png]

**Figure 35.** Figure 35: Pattern behaviour across different LLMs for SVM. 0.50 0.60 0.70 0.80 0.90 0.40 0.50 0.60 0.70 0.80 Acc. on the line: e5|gbm (CA AK) Std. Tr., R 2=0.947 Rep. Tr., R 2=0.002 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.40 0.50 0.60 0.70 F1 on the line: e5|gbm (CA AK) …

**Figure 36.** Figure 36: Pattern behaviour across different LLMs for GBM. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_36.png]

**Figure 37.** Figure 37: Pattern behaviour across different LLMs for XGB. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_37.png]

**Figure 38.** Figure 38: Pattern behaviour across different LLMs for MLP. 0.65 0.70 0.75 0.80 0.85 0.68 0.70 0.72 0.74 0.76 0.78 0.80 0.82 Acc.‑on‑the‑line: e5|cvar_dro (CA→AR) Std. Tr., R 2=0.019 Rep. Tr., R 2=0.770 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.45 0.50 0.55 0.60 0.65 0.70 0.75 F…

**Figure 39.** Figure 39: Pattern behaviour across different LLMs for CVaR-DRO. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_39.png]

**Figure 40.** Figure 40: Pattern behaviour across different LLMs for LR. 0.65 0.70 0.75 0.80 0.85 0.90 0.30 0.40 0.50 0.60 0.70 0.80 Acc. on the line: e5|rf (CA AR) Std. Tr., R 2=0.910 Rep. Tr., R 2=0.027 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.30 0.40 0.50 0.60 0.70 F1 on the line: e5…

**Figure 41.** Figure 41: Pattern behaviour across different LLMs for RF. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_41.png]

**Figure 42.** Figure 42: Pattern behaviour across different LLMs for SVM. 0.50 0.60 0.70 0.80 0.90 0.30 0.40 0.50 0.60 0.70 0.80 Acc. on the line: e5|gbm (CA AR) Std. Tr., R 2=0.884 Rep. Tr., R 2=0.019 GT: -50k y = x 0.40 0.50 0.60 0.70 0.80 0.90 0.30 0.40 0.50 0.60 0.70 F1 on the line: e5|gb…

**Figure 43.** Figure 43: Pattern behaviour across different LLMs for GBM. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_43.png]

**Figure 44.** Figure 44: Pattern behaviour across different LLMs for XGB. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_44.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 7 canonical work pages

[1]

and Fusi, N

Alvarez-Melis, D. and Fusi, N. Robustness and Gen- eralization via Algorithmic Stability.arXiv preprint arXiv:2002.08791,

arXiv 2002
[6]

The column ”Well-defined” refers to empirical observation that the corresponding HP sets do not collapse for the tabular modality in our setting

Sets of shared hyper-parameter combinations used in all experiments.⋄for methods with the total grid size above 200, we randomly sample 200 configurations for fair comparisons. The column ”Well-defined” refers to empirical observation that the corresponding HP sets do not collapse for the tabular modality in our setting. Model # of HPs Well-Defined Hyperp...

work page 1944
[7]

Mitigating Modality Collapse via Explicit Basis Reallocation.arXiv preprint arXiv:2505.22483,

Chaudhuri, S., Yang, Y ., Yan, J., Khan, S., Shah, M., and Liu, Z. Mitigating Modality Collapse via Explicit Basis Reallocation.arXiv preprint arXiv:2505.22483,

arXiv
[9]

Strong Model Collapse from Weak Synthetic Data.arXiv preprint arXiv:2410.04840,

Dohmatob, E., Tarres, P., and Thomas, V . Strong Model Collapse from Weak Synthetic Data.arXiv preprint arXiv:2410.04840,

arXiv
[10]

Collision Avoidance Detour for Multi-Agent Trajectory Forecasting

Gardner, J., Popovic, M., and Schmidt, M. TableShift: Quantifying Distribution Shift in Tabular Data.arXiv preprint arXiv:2306.11638,

work page Pith review arXiv
[13]

and Yamaguchi, S

Koyama, M. and Yamaguchi, S. When Is Invariance Use- ful In An Out-Of-Distribution Generalization Problem? arXiv preprint arXiv:2008.01883,

arXiv 2008
[14]

MTEB: Massive Text Embedding Benchmark

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. InPro- ceedings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pp. 2014–2037,

work page 2014
[15]

and Singh, S

Slack, D. and Singh, S. Tablet: Learning from Instructions for Tabular Data.arXiv preprint arXiv:2304.13188,

arXiv

Show all 20 references

[16]

Improving Text Embeddings with Large Language Models.arXiv preprint arXiv:2401.00368,

Wang, L., Yang, N., Huang, X., Yang, L., Majumder, R., and Wei, F. Improving Text Embeddings with Large Language Models.arXiv preprint arXiv:2401.00368,

arXiv
[17]

LLM Em- beddings Improve Test-time Adaptation to TabularY|X - Shifts

11 Model Collapse and Spurious Shift Performance Prediction from Training on Uncurated Text Embeddings Zeng, Y ., Liu, J., Lam, H., and Namkoong, H. LLM Em- beddings Improve Test-time Adaptation to TabularY|X - Shifts. InNeurIPS 2024 Third Table Representation Learning Workshop,

2024
[19]

and (Zeng et al., 2024). Table

2024
[2013]

Tab- transformer: Tabular Data Modeling Using Contextual Embeddings.arXiv preprint arXiv:2012.06678,

Huang, X., Khetan, A., Cvitkovic, M., and Karnin, Z. Tab- transformer: Tabular Data Modeling Using Contextual Embeddings.arXiv preprint arXiv:2012.06678,

2012 arXiv
[2016]

Linq-Embed-Mistral Technical Report.arXiv preprint arXiv:2412.03223,

Choi, C., Kim, J., Lee, S., Kwon, J., Gu, S., Kim, Y ., Cho, M., and yong Sohn, J. Linq-Embed-Mistral Technical Report.arXiv preprint arXiv:2412.03223,

arXiv
[2017]

Baek, C., Jiang, Y ., Raghunathan, A., and Kolter, J. Z. Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift.arXiv preprint arXiv:2206.13089,

arXiv
[2019]

Unifying Dis- tributionally Robust Optimization via Optimal Transport Theory.arXiv preprint arXiv:2308.05414,

Blanchet, J., Kuhn, D., Li, J., and Taskesen, B. Unifying Dis- tributionally Robust Optimization via Optimal Transport Theory.arXiv preprint arXiv:2308.05414,
[2021]

Invariant Risk Minimization.arXiv preprint arXiv:1907.02893,

Arjovsky, M., Bottou, L., Gulrajani, I., and Lopez-Paz, D. Invariant Risk Minimization.arXiv preprint arXiv:1907.02893,

1907 arXiv
[2022]

J., Bouchacourt, D., and Sagun, L

Bell, S. J., Bouchacourt, D., and Sagun, L. Reassessing the Validity of Spurious Correlations Benchmarks.arXiv preprint arXiv:2409.04188,

arXiv
[2023]

T., Namkoong, H., and Yadlowsky, S

Cai, T. T., Namkoong, H., and Yadlowsky, S. Diagnos- ing Model Performance Under Distribution Shift.arXiv preprint arXiv:2303.02011,

arXiv
[2024]

doi: 10.1038/s41586-024-07566-y. Hu, Z. and Hong, L. J. Kullback-Leibler Divergence Con- strained Distributionally Robust Optimization.Optimiza- tion Online, 1(2):9,

doi
[2025]

ToEdit: Token-Level Editing Prevents Non-Iterative Model Collapse on Synthetic Data.arXiv preprint arXiv:2412.14689,

Zhu, P., Dai, X., Yao, Z., Dou, D., and Yang, D. ToEdit: Token-Level Editing Prevents Non-Iterative Model Collapse on Synthetic Data.arXiv preprint arXiv:2412.14689,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

and Fusi, N

Alvarez-Melis, D. and Fusi, N. Robustness and Gen- eralization via Algorithmic Stability.arXiv preprint arXiv:2002.08791,

arXiv 2002

[2] [6]

The column ”Well-defined” refers to empirical observation that the corresponding HP sets do not collapse for the tabular modality in our setting

Sets of shared hyper-parameter combinations used in all experiments.⋄for methods with the total grid size above 200, we randomly sample 200 configurations for fair comparisons. The column ”Well-defined” refers to empirical observation that the corresponding HP sets do not collapse for the tabular modality in our setting. Model # of HPs Well-Defined Hyperp...

work page 1944

[3] [7]

Mitigating Modality Collapse via Explicit Basis Reallocation.arXiv preprint arXiv:2505.22483,

Chaudhuri, S., Yang, Y ., Yan, J., Khan, S., Shah, M., and Liu, Z. Mitigating Modality Collapse via Explicit Basis Reallocation.arXiv preprint arXiv:2505.22483,

arXiv

[4] [9]

Strong Model Collapse from Weak Synthetic Data.arXiv preprint arXiv:2410.04840,

Dohmatob, E., Tarres, P., and Thomas, V . Strong Model Collapse from Weak Synthetic Data.arXiv preprint arXiv:2410.04840,

arXiv

[5] [10]

Collision Avoidance Detour for Multi-Agent Trajectory Forecasting

Gardner, J., Popovic, M., and Schmidt, M. TableShift: Quantifying Distribution Shift in Tabular Data.arXiv preprint arXiv:2306.11638,

work page Pith review arXiv

[6] [13]

and Yamaguchi, S

Koyama, M. and Yamaguchi, S. When Is Invariance Use- ful In An Out-Of-Distribution Generalization Problem? arXiv preprint arXiv:2008.01883,

arXiv 2008

[7] [14]

MTEB: Massive Text Embedding Benchmark

Muennighoff, N., Tazi, N., Magne, L., and Reimers, N. MTEB: Massive Text Embedding Benchmark. InPro- ceedings of the 17th Conference of the European Chap- ter of the Association for Computational Linguistics, pp. 2014–2037,

work page 2014

[8] [15]

and Singh, S

Slack, D. and Singh, S. Tablet: Learning from Instructions for Tabular Data.arXiv preprint arXiv:2304.13188,

arXiv