{"id":"773deba1-98b7-44b0-a05c-d69563a14965","arxiv_id":"2506.17989","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Training on LLM text embeddings can cause tabular classifiers to collapse to single-class predictions, which spuriously inflates Accuracy-on-the-Line correlations.","lead":"This paper shows that training classifiers on text embeddings of tabular data often produces models that always predict the same class. Such degenerate models can make out-of-distribution accuracy look falsely predictable, so embedding-based pipelines need careful evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The shared hyperparameter grids (Table 6) are inherited from raw-tabular pipelines with no adaptation to 4096-d embeddings, so the reported single-class collapses could be a grid-mismatch artifact rather than an intrinsic property of text embeddings; Section 4.2's own tuned-config result suggests…","rationale":"I agree with the reader's weakest_assumption; it is the single most load-bearing point. Other weaknesses (single dataset, no code, wide epsilon) affect breadth and reproducibility but do not directly threaten the logic of the collapse claim. The HP mismatch, however, does: if it holds, both the collapse finding and the spurious-ACL finding are consequences of an unfair comparison. The paper deserves credit for the transfer experiment (Section 4.2) and for honestly scoping the study as qualitative, but those do not address the control. My recommended verdict is unchanged (conditional) because the reader already conditioned on exactly this concern; the proposed test would decide between 'artifact' and 'real effect.'","tokens_in":48945,"tokens_out":5888,"duration_ms":70143,"concrete_test":"Run a single control experiment per model family from Table 6 (CA training, PR and AL targets) with an embedding-adapted hyperparameter grid: z-score or unit-norm preprocessing of the 4096-d embeddings, larger MLP widths (256, 512, 1024), lower learning rates (1e-4 to 1e-3), and a broad L2/regularization sweep, selected by ID validation macro-F1 via random search of 300-500 configs per LLM. Then recompute the strict collapse ratio CR and the strong/projection ratios CRs/CRp on the same test sets. If CR drops below 5% for all (LLM, family) pairs while the original Table 6 grid gave >20%, the observed model collapse is an artifact of the shared grid and the central claim would be rejected; if CR remains high (e.g., >20%) for configs with good ID validation performance, the claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('collapse is a consistent failure mode' of TE-based models) rests on comparing raw-tabular and embedding-trained models under identical hyperparameter configurations. That comparison is fair only if the chosen grids are reasonable for both modalities. The grids in Table 6 come from Liu et al. (2023) and Zeng et al. (2024) for raw tabular data: MLP hidden units are 16-128 with input dimension 4096; LR and SVM use regularization ranges designed for low-dimensional tabular features; no feature normalization or dimension-aware regularization is described. Under such a mismatch, models may trivially converge to the majority class (e.g., a 4096-dim input passed through a 16-unit layer with a modest learning rate can underfit severely). The paper's own Section 4.2 demonstrates that a single MLP configuration (c2, tuned on e5 embeddings) avoids collapse on all e5-encoded states, and only collapses when transferred to other LLMs' embeddings. This shows collapse severity is highly sensitive to the HP-input match, and suggests that the high collapse ratios in Section 4.1 may be an artifact of using raw-tabular grids on embeddings. If an embedding-adapted grid, or even a modest random search on embeddings, produces collapse ratios near zero, the claim that uncurated TEs inherently induce collapse would be unsupported. Until that control is run, the central assertion is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies binary tabular classification from the ACS Income dataset, comparing models trained on raw tabular features with models trained on 4096-dimensional text embeddings produced by four off-the-shelf LLMs under identical hyperparameter configurations. It defines collapse metrics based on the fraction of configurations that yield single-class predictors, reports that such collapse is common in embedding-trained models, and argues that strong collapse can artificially inflate Accuracy-on-the-Line (ACL) R² values, producing a misleading signal of OOD predictability. It also compares MTEB rankings with downstream OOD performance and studies cross-LLM embedding transfer. The experiments cover seven model families, four encoders, and five target states, with additional test-time adaptation results.","tokens_in":49236,"tokens_out":4857,"duration_ms":60203,"significance":"If the central empirical claim is confirmed, the paper identifies a practically important failure mode: using general-purpose text embeddings as a data-curation layer for tabular data can produce degenerate single-class predictors, and the resulting ACL correlations are not trustworthy evidence of OOD generalization. The proposed collapse metrics are simple, interpretable, and potentially reusable, and the paper is honest that the study is an 'initial investigation' and 'far from being comprehensive' (Section 4, Conclusion). However, the evidence currently rests on one binary dataset, a single training state, one run per configuration, and hyperparameter grids inherited from raw-tabular pipelines; these limitations are load-bearing for the generality of the abstract's claims. The paper would be strengthened by a formal link between collapse ratios and R² inflation, by an embedding-adapted hyperparameter control, and by additional datasets or seeds.","major_comments":[{"comment":"The central comparison assumes that the hyperparameter grids designed for raw tabular models are a fair baseline for 4096-dimensional embeddings. The grids in Table 6 come from low-dimensional tabular pipelines (e.g., MLP hidden sizes 16-128, LR regularization on the raw feature scale), with no feature normalization or dimension-aware regularization for the embedding space. Under such a mismatch, trivial majority-class convergence is expected. This concern is not hypothetical: Section 4.2 shows that the single configuration c2, tuned on e5 embeddings, avoids collapse on all e5-encoded states but collapses when transferred to other LLMs' embeddings, demonstrating extreme sensitivity to the HP-input match. To support the claim that uncurated text embeddings inherently induce collapse, the authors should run a control with an embedding-adapted search (e.g., standardized embeddings, larger capacity, dimension-aware regularization) and report the resulting collapse ratios; if those ratios are near zero, the main claim is not established.","section":"Section 4.1, Table 6"},{"comment":"The claim that strong collapse induces spurious ACL is supported by visual inspection of Figure 2b rather than by a formal comparison. The paper states that the R² of the green points is 'significantly higher' and attributes this to overlapping collapsed points, but it does not quantify the effect. Please compute R² on the non-collapsed subset of configurations, or report the difference between R² with and without collapsed points across all LLM/model/state combinations, and relate that difference to the collapse ratios. Without such a test, the warning about spurious ACL remains an illustrative observation rather than an established property.","section":"Section 4.1, 'Strong Model collapse induces spurious ACL'"},{"comment":"The collapse metrics are defined only for 'well-defined' hyperparameter sets, i.e., sets that do not collapse on raw tabular data. However, Table 6 marks XGB and RF as not well-defined, and Section 4.1 still reports collapse patterns and ratios for these families (including the RF discussion and Figure 13). This is an internal inconsistency in the metric-based evidence. Either restrict the collapse-ratio analysis to the well-defined subsets for XGB and RF, or formally extend the definitions to cover non-well-defined baseline sets and state the implications for interpretation.","section":"Section 4.1, Section 3.4"},{"comment":"The abstract and Section 4.1 claim that collapse is a 'consistent failure mode' across LLM-embedding training, but the evidence is from a single binary dataset (ACS Income), a single training state (CA), a single LLM per embedding modality, and one run per configuration with no error bars. The paper's own wording concedes an 'initial investigation' and prioritizes 'qualitative insights,' which is appropriate for a workshop-scale study but not sufficient for the generality claimed in the abstract. At minimum, add a second tabular dataset and a small number of seeds to quantify variance, or revise the claims to be explicitly scoped to the ACS Income setup.","section":"Section 4.1, Experimental setup"}],"minor_comments":[{"comment":"Typo: 'pics' should be 'picks' in the sentence 'Our method pics a set of training Hyper-Parameter Configurations'.","section":"Section 3.2"},{"comment":"Typo: 'de ratio of configurations' should be 'the ratio of configurations'.","section":"Section 3.4"},{"comment":"The near-collapse threshold epsilon is stated to 'vary from one experiment to the next' and is chosen 'with consideration to the overall shape' of the scatter. Since epsilon is a free parameter, please report sensitivity of the near-collapse ratios to epsilon or fix a single value across all experiments and justify it.","section":"Appendix A.2"},{"comment":"For methods with grids larger than 200, the paper says 200 configurations are randomly sampled, but no random seed is reported; the stability of the collapse ratios under this sampling should be discussed.","section":"Table 6"},{"comment":"The sentence 'one should not be overlooked the fact that...' is grammatically awkward and should be rephrased for clarity.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a plausible workshop-level empirical study with a useful warning about embedding-trained tabular models. The main risk is that the shared-hyperparameter comparison is not a controlled experiment for the embedding modality, and the 'spurious ACL' claim currently lacks quantitative support. I do not see a fundamental circularity in the collapse metrics, but the epsilon threshold and the non-well-defined HP sets for XGB/RF need to be addressed. With the controls requested in the major comments, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The paper documents something concrete: train a model on LLM text embeddings of a tabular dataset, using hyperparameter configurations that work fine on the raw table, and a large fraction of those models collapse to predicting a single class. That collapse then can inflate the R² of Accuracy-on-the-Line, giving a false signal that OOD performance is predictable. The authors show this across four LLMs and several US states on ACS Income, and they introduce simple collapse-ratio metrics that are easy to compute. The observation that high-dimensional or poorly scaled features can yield constant classifiers is not new, but the documentation for modern text embeddings on tabular data, and the connection to ACL inflation, is a useful addition to the LLM-embeddings-for-tabular literature.\n\nThe main soft spot is the evidence base. It is one binary dataset, one run per configuration, no error bars, and no code or data released. That alone would make me hesitant to over-generalize. The deeper issue is the shared hyperparameter grids: they come from raw-tabular pipelines, and 4096-dimensional embedding inputs are a different regime. A 16-unit MLP with a small learning rate on 4096-d inputs can underfit trivially. The stress-test note worries that the high collapse ratios in Section 4.1 could be a grid-mismatch artifact rather than an intrinsic property of text embeddings. That concern is legitimate, but the paper partially answers it in Section 4.2: the tuned configuration from Zeng et al. avoids collapse on the same LLM's embeddings but still collapses when transferred to other LLMs' embeddings. So collapse is not purely an artifact of a bad grid — it also appears with a good grid when the embedding source changes. What remains unresolved is how prevalent collapse would be under embedding-adapted hyperparameter search. The authors should run that control before claiming collapse is a consistent failure mode.\n\nThe spurious ACL claim is supported by visual examples and by the logic that a pile of overlapping collapsed points inflates R², but there is no formal comparison of R² before and after removing collapsed points. That would be easy to add. The epsilon for near-collapse is chosen per experiment; a sensitivity check would help.\n\nOverall, the paper is honest about its scope and frames itself as an initial investigation. It deserves a serious referee. I would not desk-reject it. It needs a control with embedding-adapted hyperparameters, error bars or multiple runs, and code/data release. After that, the central caution — that uncurated text embeddings can silently produce degenerate models and misleading on-the-line correlations — is likely to hold, at least as a warning for practitioners.","headline":"A real and easily reproduced failure mode for LLM-embedding tabular pipelines, with the caveat that the headline prevalence number depends on hyperparameter grids inherited from raw-tabular training.","tokens_in":49750,"tokens_out":2243,"would_cite":true,"duration_ms":23500,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training tabular classifiers on uncurated text embeddings can silently collapse them into single-class prediction, and the collapse manufactures a spurious Accuracy-on-the-Line correlation that falsely reads as OOD predictability.","keywords":["model collapse","text embeddings","tabular data","out-of-distribution generalization","Accuracy-on-the-Line","distribution shift","data curation","LLM embeddings"],"falsifier":"Retrain every collapsing configuration on the same embeddings using a hyperparameter grid designed for high-dimensional inputs, such as lower learning rates, stronger regularization, weight decay, or embeddings projected to a matched dimension, and recompute the collapse ratios. If $\\mathrm{CR}_s$ falls to near zero across all four encoders, the reported collapse is a grid-mismatch artifact rather than a property of text embeddings. A second check is to run the identical bound-pair analysis on random Gaussian features of dimension 4096; if those collapse at comparable rates, the failure is about dimensionality and optimization, not about the semantic content of embeddings.","tokens_in":48725,"feed_emoji":"📉","tokens_out":9620,"duration_ms":92515,"temperature":0.7,"pith_summary":"The paper claims that using LLM text embeddings as a data-curation layer for tabular data can instead trigger model collapse: with the same hyperparameter settings that work on raw tabular columns, classifiers trained on 4096-dimensional row embeddings often predict a single class for every input. It reports this failure across four embedding models, seven classifier families, and 4,615 configurations, and introduces collapse-ratio metrics that measure how many shared hyperparameter settings degenerate. The deeper claim is that strong collapse manufactures a spurious Accuracy-on-the-Line correlation: the $R^2$ between in-distribution and out-of-distribution accuracy rises not because performance transfers but because collapsed models all pile onto the same class-ratio point. Anyone who uses a high Accuracy-on-the-Line $R^2$ to predict out-of-distribution accuracy from in-distribution accuracy would therefore be silently misled, which is why the paper insists that collapse severity must be reported alongside any such correlation. It also finds that leaderboard rankings of embedding models do not predict how well an embedding serves as a curation layer under tabular distribution shift.","feed_headline":"Training on text embeddings collapses classifiers to one class","feed_subtitle":"A high Accuracy-on-the-Line R-squared can signal collapse rather than transfer, and leaderboard rank won't save you.","key_machinery":"The central instrument is the bound-pair on-the-line plane. For each hyperparameter configuration $c$, the authors train one model on raw tabular data and a twin model on the LLM embeddings of the same rows, then plot both in the plane whose axes are in-distribution and out-of-distribution scores, binding each pair with a straight segment. Collapse is defined by counting predictions: a model is collapsed on a test set if it predicts only one class, and near-collapsed if its true-negative plus false-negative rates fall below a threshold $\\varepsilon$. The collapse ratio $\\mathrm{CR}^{S}_{HP}$ is the fraction of configurations in a hyperparameter set (chosen non-collapsing on raw tabular data) that become single-class predictors when trained on embeddings; the strong-collapse ratio $\\mathrm{CR}_s$ and projection ratio $\\mathrm{CR}_p$ count configurations collapsing on both test sets or on at least one. These ratios expose a spurious Accuracy-on-the-Line: when many embedding-trained models stack on the class-ratio point, the fitted $R^2$ rises while the non-collapsed remainder shows no linear trend.","core_discovery":"The central discovery is that model collapse is a characteristic failure mode of training on uncurated text embeddings, and that this collapse contaminates the standard diagnostic used to claim out-of-distribution predictability. When a hyperparameter configuration that produces a functioning classifier on raw tabular data is reused on LLM-embedded rows, the resulting model frequently predicts one class regardless of input, so its in-distribution and out-of-distribution accuracies both sit at the test set's positive-class ratio. Many such collapsed points overlap in the Accuracy-on-the-Line plane, mechanically raising the $R^2$ of the linear fit even though the surviving models follow no discernible line; the paper exhibits a case where collapsed models yield $R^2 \\approx 0.38$ on accuracy and $0.87$ on macro-F1 while the non-collapsed points are scattered. The authors conclude that Accuracy-on-the-Line, and OOD performance prediction built on it, is only meaningful when collapse is measured and excluded, and they frame the collapse ratio as the diagnostic that should accompany any such claim.","pith_inferences":["A testable extension: check whether the collapse ratio tracks the intrinsic dimension or spectral structure of the embedding space; if it does, the metric becomes a cheap, OOD-label-free way to pick between embedding models.","The same bound-pair collapse analysis could be applied to vision or audio features; if single-class collapse appears there too, the phenomenon is a general property of training on high-dimensional pretrained representations rather than something specific to text.","The near-collapse threshold $\\varepsilon$ is chosen per experiment after the fact; a fixed, pre-registered threshold would make the collapse ratios usable as a transferable benchmark metric, as the paper's appendix leaves that choice open.","The spurious-ACL mechanism implies an audit rule for earlier work: any accuracy-on-the-line claim computed over a set where several models sit exactly at the class-ratio point deserves discounting, because those points raise $R^2$ by geometry rather than by semantics."],"forward_implications":["Every reported Accuracy-on-the-Line $R^2$ for an embedding-trained model should be read alongside a collapse ratio; a high $R^2$ with a high strong-collapse ratio is evidence of degeneracy, not of transfer.","Pipelines that predict out-of-distribution accuracy from in-distribution accuracy should reject the prediction whenever models already collapse on the source domain, which the paper shows happens with text embeddings.","Embedding leaderboard rankings are not a proxy for data-curation quality under tabular shift, since the lowest-ranked of the four encoders tested was the most robust on this task.","Using a different encoder at test time than at training time is itself a distribution shift that provokes collapse, even for configurations that behave well on embeddings from the training encoder.","Test-time adaptation methods, including entropy minimization and a tabular-specific variant, do not restore a valid on-the-line relation for embedding-trained models."],"supporting_citations":[{"why":"Defines Accuracy-on-the-Line, the ID-OOD linear correlation whose R-squared the paper shows can be spuriously inflated by collapse.","marker":"Miller et al., 2021"},{"why":"Supplies the text-embedding-as-curation pipeline, the MLP configurations, and the experimental setup that this paper re-implements and extends.","marker":"Zeng et al., 2024"},{"why":"Provides the tabular-domain ACL baseline and the hyperparameter grids that the raw-data training reproduces.","marker":"Liu et al., 2023"},{"why":"Provides the leaderboard whose rankings the paper finds do not predict embedding quality for tabular shifts.","marker":"Muennighoff et al., 2023"},{"why":"Provides the ACS Income dataset used for the six-state binary classification experiments.","marker":"Ding et al., 2021"},{"why":"The claim that test-time adaptation strengthens Accuracy-on-the-Line, which the paper tests and finds does not hold for embedding-trained models.","marker":"Kim et al., 2024"},{"why":"Defines TENT, the entropy-minimization adaptation method the paper evaluates and finds ineffective at restoring ACL.","marker":"Wang et al., 2021"},{"why":"Defines FTAT, the tabular test-time adaptation method that fails to restore ACL and instead produces a spurious second line.","marker":"Zhou et al., 2025"}],"fun_headline_variants":["Uncurated embeddings induce model collapse, fake OOD fit","Model collapse on embeddings masks real OOD transfer signals","Text embedding training without curation yields spurious accuracy lines","Accuracy-on-the-Line can be an artifact of model collapse"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hyperparameter configurations chosen for raw tabular models are a fair test of the embedding-based pipeline; if those same settings are simply a poor match for 4096-dimensional inputs, the observed single-class predictions could be an artifact of the grid rather than a property of uncurated text embeddings.","fun_headline_variants_meta":{"raw":{"variants":["Uncurated embeddings induce model collapse, fake OOD fit","Model collapse on embeddings masks real OOD transfer signals","Text embedding training without curation yields spurious accuracy lines","Accuracy-on-the-Line can be an artifact of model collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1365,"prompt_tokens":922,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":376}},"tokens_in":538,"tokens_out":443,"duration_ms":5251,"temperature":1.0,"reasoning_tokens":376,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:57.078843+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain every collapsing configuration on the same embeddings using a hyperparameter grid designed for high-dimensional inputs, such as lower learning rates, stronger regularization, weight decay, or embeddings projected to a matched dimension, and recompute the collapse ratios. If $\\mathrm{CR}_s$ falls to near zero across all four encoders, the reported collapse is a grid-mismatch artifact rather than a property of text embeddings. A second check is to run the identical bound-pair analysis on random Gaussian features of dimension 4096; if those collapse at comparable rates, the failure is about dimensionality and optimization, not about the semantic content of embeddings.","supporting_citations":[],"review_version":1}