{"id":"83e39748-dcdb-4efd-8941-e04af17cd484","arxiv_id":"2509.06184","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Synthetic data for text embedders yields sparse, task-localized MTEB gains and cross-task trade-offs, not broad robust improvement.","lead":"This paper reproduces and publicly releases the synthetic training data behind a state-of-the-art text embedding model, then studies where that data actually helps. It finds that synthetic data improves benchmark scores through only a few tasks, and can hurt others, challenging the idea that more synthetic data broadly improves embedders.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-replicate influence t-tests lack a valid error term; seed variance is unmeasured, so the 'sparse' null claim is underdetermined.","rationale":"I agree with the reader's weakest_assumption and see it as the central issue. The paper's contribution—reproducing and releasing synthetic data plus a 16-run influence analysis—is valuable, and the headline trade-offs are large enough that they may survive replication. But the 'sparse' claim in the abstract and the explicit 'no statistically significant improvement' language in the Introduction depend entirely on valid p-values. The reported unpaired t-test has no defensible error term in a single-replicate factorial design, and no seed variance is reported anywhere (Appendix C gives training details but not seeds). The Limitations section addresses base-dataset interactions and model capability but not statistical power or seed sensitivity. A data-size confound is also present (short-short/long-long are ~20k examples vs ~100-150k for short-long/long-short), but the statistical issue is more load-bearing because it undermines the significance-based negative claim even before considering size. I therefore keep the reader's conditional verdict: the empirical artifacts and dataset release are useful, but the central interpretational claim should be conditional on a seeded replication or, at minimum, a paired analysis with an explicit error model.","tokens_in":18880,"tokens_out":12550,"duration_ms":163645,"concrete_test":"Run the 16 influence combinations with 3 independent random seeds (48 training runs, same data/hyperparameters). For each seed, compute the 8 paired differences for each synthetic category and estimate the main effect with a paired t-test / mixed-effects model with seed as random effect; also report the seed standard deviation for identical data subsets. If the seed σ exceeds ~0.5 MTEB points, or if the set of significant influences changes materially (e.g., long-short on classification loses significance, or previously null cells such as short-short become significant), the original sparse/localized conclusion is not robust. A minimal version: rerun only the 8 runs that vary long-short at 3 seeds to bound the main-effect error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is the statistical identification of the influence t-tests in §3 ('Estimating data influence'). The 16 training runs are one observation per cell of a 2^4 factorial. For a given category S, the 8 runs with S and the 8 without S are not eight independent samples from a noise distribution; they are the complete set of combinations of the other three factors. The unpaired two-sample t-test therefore uses variance driven by the systematic effects of those other categories (and their interactions) as if it were error. The design is naturally paired—each of the 8 other-factor combinations appears once with S and once without—and a paired t-test, or a factorial ANOVA that treats high-order interactions as residual, is the appropriate analysis; neither is reported. With one run per cell there is no pure error term unless higher-order interactions are assumed negligible, and no seed variance is reported. If training seeds were reused, there is no stochastic variation at all; if seeds varied, the single observation per subset cannot separate seed noise from interaction variance. Because the Introduction's headline claim is a null claim ('no statistically significant improvement on a majority of MTEB tasks'), a low-power or mis-specified test can manufacture 'sparse' even when real but small effects exist. The large trade-off effects may survive, but the formal support for the central negative claim is not identified by the reported procedure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the contribution of synthetic LLM-generated data to general-purpose text embedding models. The authors reproduce the synthetic data pipeline of Wang et al. (2024) using LLaMA-3.1-8B and LLaMA-3.1-70B, release the data and code, and show that adding this data to the Springer et al. (2024) public E5 mixture improves average MTEB scores for Mistral-v0.1-7B, Mistral-v0.2-7B, and Qwen2-1.5B. In Section 3, they train 16 models per base model/generator corresponding to all subsets of four synthetic categories (short-short, short-long, long-long, long-short) and define 'influence' as the difference in mean MTEB category/task performance between the eight models containing a category and the eight not containing it. They apply two-sided t-tests to these differences and conclude that synthetic-data benefits are sparse, localized, and sometimes traded off across tasks. The paper argues against the view that more diverse synthetic data is strictly better.","tokens_in":19207,"tokens_out":7975,"duration_ms":102787,"significance":"The main contribution is a public release of high-quality synthetic embedding training data and a careful replication of the overall MTEB improvement. If the Section 3 conclusions were statistically well-founded, the paper would be an important corrective: it would show that synthetic data's benefits are concentrated and that large gains in one task can come at the expense of another. The trade-off observations (e.g., long-short data increasing classification/clustering while decreasing other categories in Figure 3) are striking and likely robust given their magnitude. However, the paper's central negative claim—that benefits are 'sparse' and that most tasks show no statistically significant improvement—rests on a statistical procedure that is not identified: the t-tests use one observation per cell of a 2^4 factorial design, with no seed variance and no valid error term. The quantitative support for sparsity is therefore not established by the reported evidence. The reproducibility assets (data and code release) are a concrete strength.","major_comments":[{"comment":"The t-tests are not identified. The 16 training runs are the cells of a 2^4 factorial design, so the two groups of eight runs for a category S are not independent samples: the other three factors vary systematically within each group. The variance used by the unpaired two-sample t-test therefore includes the main effects and interactions of the other categories, not sampling noise. The design is naturally paired (each of the eight other-factor combinations appears once with S and once without), but no paired test or factorial ANOVA is reported. With one run per cell there is no pure error term unless higher-order interactions are assumed negligible, and no seed variance is given. The claims that short-short has no significant influence (Fig. 3) and that improvements are 'sparse' are therefore unsupported. Please state whether seeds are shared across runs, add replicates to estimate seed","section":"§3, 'Estimating data influence'"},{"comment":"No adjustment for multiple comparisons is reported. Each heatmap performs dozens of tests at p<0.05 (20 category-level cells in Fig. 3, and many more at the task level in the appendix figures); under the null, one expects several false positives at this threshold. The 'sparse' conclusion is stated as if the highlighted significant cells were the true set of effects. The authors should apply a correction (e.g., Benjamini-Hochberg) or provide exact p-values and effect sizes. This compounds the invalid error term described above.","section":"§3, Figures 3–9"}],"minor_comments":[{"comment":"'Appendix 3' appears to be a typo; the results are in Appendix E.","section":"Figure 3 caption; Table 1 caption"},{"comment":"'can led to' should be 'can lead to'.","section":"§2, first paragraph"},{"comment":"'standard evaluation protocol is which embeddings' should be 'in which embeddings'.","section":"Appendix D"},{"comment":"The caption says 'Colored cells indicate statistically significant influence' while also saying the color encodes normalized influence. Clarify whether all cells are colored or only significant cells, and provide a legend separating significance from magnitude.","section":"Figures 3 and 4 captions"},{"comment":"The text says the dataset, models, and code 'will be released upon publication,' which is inconsistent with the abstract and Section 2 claiming public release. Please clarify the availability timeline and repository status.","section":"Appendix C"},{"comment":"The term 'influence function' is used for a simple marginal effect of including a category. This is not the standard influence-function estimate; consider renaming it 'marginal effect' or 'ablation effect'.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the reproduction is clean and useful, the dataset release is a real community contribution, but the significance analysis behind the \"sparse / localized\" conclusion is not valid as run. The big trade-offs are probably real; the null claims are not established.\n\nWhat's actually new: a per-category influence decomposition of four synthetic data types (short-short, short-long, long-long, long-short) on MTEB categories and individual tasks, and a public synthetic dataset generated by LLaMA-3.1-8B/70B. The replication of Wang et al. is done across two base models and two generators, and the overall MTEB gains are consistent. That is good work. The appendix contains full influence tables for all settings, which is generous.\n\nThe soft spot is the t-test in Section 3. The authors train 16 models—one per subset of the four synthetic categories—and for each category compare the eight models with it against the eight without it using an unpaired two-sample t-test. That is the wrong error term. The eight \"with\" runs are not independent draws; they are the complete set of combinations of the other three categories, and the differences among them are driven largely by those other categories and their interactions. The natural analysis is a paired comparison (each of the eight other-category combinations appears once with and once without the target category) or a factorial ANOVA using high-order interactions as residual. With one run per cell there is no pure error term, and no seed variance is reported. So the p-values and the \"majority of tasks show no statistically significant improvement\" claim cannot be trusted. A low-power or mis-specified test can manufacture \"sparse\" even when real effects exist.\n\nThat said, the trade-offs are large. Effects like +6.65 on classification and -1.34 on STS for long-short data are not borderline; they are an order of magnitude larger than plausible training noise. Even if the significance machinery is wrong, I'd expect the trade-off pattern to hold. The authors' central descriptive claim is not load-bearing on the t-test; the inferential claim is.\n\nMinor points: the abstract and intro say the data is publicly released, but the appendix says it will be released upon publication. That matters because the paper's main artifact isn't yet verifiable. Also, Section 3 substitutes faster retrieval sets; that's reasonable but worth flagging.\n\nWho should read this: anyone working on synthetic data for embedders, and anyone using MTEB as a testbed. It deserves a serious referee; the right outcome is probably conditional acceptance with a mandatory re-analysis of the influence data, or a restriction of the claims to \"observed differences\" without significance language.","headline":"A clean, useful reproduction of the synthetic-data pipeline and a valuable dataset release, but the paper's 'sparse and localized' significance claim rests on a t-test that is not valid for its own 16-run factorial design.","tokens_in":19648,"tokens_out":2702,"would_cite":true,"duration_ms":32729,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Synthetic training data helps text embedders only in narrow spots, and often at the cost of other tasks.","keywords":["synthetic data","text embeddings","influence analysis","MTEB","contrastive learning","data curation","generalization","LLM-generated data"],"falsifier":"Retrain the same 16 configurations with multiple seeds (e.g., three or more per configuration) and recompute the influence estimates and t-tests; if the statistically significant gains and trade-offs disappear or flip sign across seeds, the paper's central claim fails. A complementary test is to evaluate the trained models on held-out tasks outside MTEB to check whether the gains are truly general or specific to the benchmark.","tokens_in":18831,"feed_emoji":"⚖️","tokens_out":3837,"duration_ms":44541,"temperature":0.7,"pith_summary":"The paper sets out to test the assumption that training text embedding models on LLM-generated synthetic data broadly improves their general-purpose quality. It reproduces and publicly releases the synthetic dataset behind Mistral-E5, then measures how each of four synthetic data categories influences downstream performance by training models on all 16 combinations of those categories. The result is that synthetic data's benefits are sparse and highly localized: most MTEB tasks show no statistically significant gain, and categories that help one task often degrade another. This matters because synthetic data is currently a dominant ingredient in state-of-the-art embedders, so understanding where it actually helps is central to building reliable general-purpose models.","feed_headline":"Synthetic data helps text embedders only in narrow spots","feed_subtitle":"Influence analysis of 16 training runs shows gains are sparse and often come with losses on other MTEB tasks.","key_machinery":"The load-bearing mechanism is an influence estimation based on exhaustive subset training. For four synthetic data categories (short-short, short-long, long-long, long-short), the authors train 16 models, one for every subset, all on top of the same public base dataset. The influence of a category is the difference in average MTEB performance between the eight runs that include it and the eight that exclude it, with a two-sided t-test marking which differences are statistically significant. This isolates each category's contribution and is what turns the aggregate improvement into a per-task, per-category map.","core_discovery":"On the paper's own terms, the central discovery is that adding synthetic data to a public training mixture does improve average MTEB scores, but the improvements are not spread across tasks. An influence analysis that compares models trained with and without each synthetic category finds, for example, that long-short synthetic data strongly improves classification and clustering while leaving or hurting other categories; short-long data improves reranking but degrades sentence similarity; and short-short data shows no statistically significant benefit on any MTEB category. The authors interpret this as evidence that current synthetic-data recipes do not make embedders more robust overall, an","pith_inferences":["The observed trade-offs may reflect a format mismatch: long-short synthetic examples teach classification/clustering-style mapping, while retrieval and STS require finer-grained similarity judgments, so gains in one skill can crowd out another.","Because each influence estimate comes from a single training run per configuration, seed variance could be masking or amplifying apparent effects; multi-seed replication would be the natural next check.","If leaderboard-driven training increasingly relies on synthetic data, models may become tuned to MTEB-like distributions; evaluating on held-out tasks outside MTEB would test whether sparse gains generalize.","The cost advantage of the 8B generator makes full factorial influence analysis practical at larger scale, e.g., including bitext and STS categories or higher-order interactions."],"forward_implications":["Practitioners cannot assume that adding more synthetic categories makes an embedder strictly better; category composition needs to be chosen with task priorities in mind.","Aggregate MTEB scores can mask localized gains and losses, so leaderboard-style evaluation should be complemented by per-task significance analysis.","Synthetic data from LLaMA-3.1-8B is a viable low-cost alternative to 70B and proprietary generation, with similar overall improvements at much lower cost.","Public release of the reproduced synthetic data and code lets other groups study interactions between synthetic and public data that this paper did not control for."],"supporting_citations":[{"why":"Supplies the synthetic-data generation pipeline and the Mistral-E5 recipe being reproduced.","marker":"Wang et al. (2024)"},{"why":"Defines MTEB, the benchmark used for all evaluation and the source of the per-task scores.","marker":"Muennighoff et al. (2023)"},{"why":"Provides the public E5 base dataset replication used as the non-synthetic training mixture.","marker":"Springer et al. (2024)"},{"why":"Serves as a comparison method for synthetic data generated from smaller fine-tuned models.","marker":"Chen et al. (2024)"},{"why":"Provides the faster MTEB retrieval datasets used in the influence analysis.","marker":"Enevoldsen et al. (2025)"},{"why":"Supplies the Mistral base models used for training.","marker":"Jiang et al. (2023)"},{"why":"Supplies the Qwen2 base model used for training.","marker":"Yang et al. (2024)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used throughout the experiments.","marker":"Hu et al. (2022)"}],"fun_headline_variants":["Synthetic data's boost for embedders is patchy and sometimes backfires","Why synthetic data won't make text embedders universally better","Synthetic training data: gains are rare, trade-offs are real","Embedders get little from synthetic data, and it can hurt other tasks","Synthetic data's benefits for embedders are isolated and costly"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The conclusions assume that the 16 training runs, one per subset, can be compared as if each were a clean sample of that data mixture; if training noise from different random seeds is large relative to the measured differences, the sparse-and-localized pattern could be an artifact of single runs.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic data's boost for embedders is patchy and sometimes backfires","Why synthetic data won't make text embedders universally better","Synthetic training data: gains are rare, trade-offs are real","Embedders get little from synthetic data, and it can hurt other tasks","Synthetic data's benefits for embedders are isolated and costly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000555,"raw_usage":{"total_tokens":2434,"prompt_tokens":651,"completion_tokens":1783,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":395,"completion_tokens_details":{"reasoning_tokens":1692}},"tokens_in":395,"tokens_out":1783,"duration_ms":15820,"temperature":1.0,"reasoning_tokens":1692,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T23:58:12.692197+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same 16 configurations with multiple seeds (e.g., three or more per configuration) and recompute the influence estimates and t-tests; if the statistically significant gains and trade-offs disappear or flip sign across seeds, the paper's central claim fails. A complementary test is to evaluate the trained models on held-out tasks outside MTEB to check whether the gains are truly general or specific to the benchmark.","supporting_citations":[],"review_version":1}