Pith. sign in

REVIEW 2 major objections 6 minor 16 references

Synthetic training data helps text embedders only in narrow spots, and often at the cost of other tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Synthetic data for text embedders yields sparse, task-localized MTEB gains and cross-task trade-offs, not broad robust improvement.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A clean, useful reproduction of the synthetic-data pipeline and a valuable dataset release, but the paper's 'sparse and localized' significance claim rests on a t-test that is not valid for its own 16-run factorial design. the 2 major comments →

arxiv 2509.06184 v1 pith:BS62WIYD submitted 2025-09-07 cs.CL

Understanding the Influence of Synthetic Data for Text Embedders

Jacob Mitchell Springer , Vaibhav Adlakha , Siva Reddy , Aditi Raghunathan , Marius Mosbach This is my paper
classification cs.CL
keywords synthetic datatext embeddingsinfluence analysisMTEBcontrastive learningdata curationgeneralizationLLM-generated data
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to test the assumption that training text embedding models on LLM-generated synthetic data broadly improves their general-purpose quality. It reproduces and publicly releases the synthetic dataset behind Mistral-E5, then measures how each of four synthetic data categories influences downstream performance by training models on all 16 combinations of those categories. The result is that synthetic data's benefits are sparse and highly localized: most MTEB tasks show no statistically significant gain, and categories that help one task often degrade another. This matters because synthetic data is currently a dominant ingredient in state-of-the-art embedders, so understanding where it actually helps is central to building reliable general-purpose models.

Core claim

On the paper's own terms, the central discovery is that adding synthetic data to a public training mixture does improve average MTEB scores, but the improvements are not spread across tasks. An influence analysis that compares models trained with and without each synthetic category finds, for example, that long-short synthetic data strongly improves classification and clustering while leaving or hurting other categories; short-long data improves reranking but degrades sentence similarity; and short-short data shows no statistically significant benefit on any MTEB category. The authors interpret this as evidence that current synthetic-data recipes do not make embedders more robust overall, an

What carries the argument

The load-bearing mechanism is an influence estimation based on exhaustive subset training. For four synthetic data categories (short-short, short-long, long-long, long-short), the authors train 16 models, one for every subset, all on top of the same public base dataset. The influence of a category is the difference in average MTEB performance between the eight runs that include it and the eight that exclude it, with a two-sided t-test marking which differences are statistically significant. This isolates each category's contribution and is what turns the aggregate improvement into a per-task, per-category map.

Load-bearing premise

The conclusions assume that the 16 training runs, one per subset, can be compared as if each were a clean sample of that data mixture; if training noise from different random seeds is large relative to the measured differences, the sparse-and-localized pattern could be an artifact of single runs.

What would settle it

Retrain the same 16 configurations with multiple seeds (e.g., three or more per configuration) and recompute the influence estimates and t-tests; if the statistically significant gains and trade-offs disappear or flip sign across seeds, the paper's central claim fails. A complementary test is to evaluate the trained models on held-out tasks outside MTEB to check whether the gains are truly general or specific to the benchmark.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners cannot assume that adding more synthetic categories makes an embedder strictly better; category composition needs to be chosen with task priorities in mind.
  • Aggregate MTEB scores can mask localized gains and losses, so leaderboard-style evaluation should be complemented by per-task significance analysis.
  • Synthetic data from LLaMA-3.1-8B is a viable low-cost alternative to 70B and proprietary generation, with similar overall improvements at much lower cost.
  • Public release of the reproduced synthetic data and code lets other groups study interactions between synthetic and public data that this paper did not control for.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The observed trade-offs may reflect a format mismatch: long-short synthetic examples teach classification/clustering-style mapping, while retrieval and STS require finer-grained similarity judgments, so gains in one skill can crowd out another.
  • Because each influence estimate comes from a single training run per configuration, seed variance could be masking or amplifying apparent effects; multi-seed replication would be the natural next check.
  • If leaderboard-driven training increasingly relies on synthetic data, models may become tuned to MTEB-like distributions; evaluating on held-out tasks outside MTEB would test whether sparse gains generalize.
  • The cost advantage of the 8B generator makes full factorial influence analysis practical at larger scale, e.g., including bitext and STS categories or higher-order interactions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper studies the contribution of synthetic LLM-generated data to general-purpose text embedding models. The authors reproduce the synthetic data pipeline of Wang et al. (2024) using LLaMA-3.1-8B and LLaMA-3.1-70B, release the data and code, and show that adding this data to the Springer et al. (2024) public E5 mixture improves average MTEB scores for Mistral-v0.1-7B, Mistral-v0.2-7B, and Qwen2-1.5B. In Section 3, they train 16 models per base model/generator corresponding to all subsets of four synthetic categories (short-short, short-long, long-long, long-short) and define 'influence' as the difference in mean MTEB category/task performance between the eight models containing a category and the eight not containing it. They apply two-sided t-tests to these differences and conclude that synthetic-data benefits are sparse, localized, and sometimes traded off across tasks. The paper argues against the view that more diverse synthetic data is strictly better.

Significance. The main contribution is a public release of high-quality synthetic embedding training data and a careful replication of the overall MTEB improvement. If the Section 3 conclusions were statistically well-founded, the paper would be an important corrective: it would show that synthetic data's benefits are concentrated and that large gains in one task can come at the expense of another. The trade-off observations (e.g., long-short data increasing classification/clustering while decreasing other categories in Figure 3) are striking and likely robust given their magnitude. However, the paper's central negative claim—that benefits are 'sparse' and that most tasks show no statistically significant improvement—rests on a statistical procedure that is not identified: the t-tests use one observation per cell of a 2^4 factorial design, with no seed variance and no valid error term. The quantitative support for sparsity is therefore not established by the reported evidence. The reproducibility assets (data and code release) are a concrete strength.

major comments (2)
  1. [§3, 'Estimating data influence'] The t-tests are not identified. The 16 training runs are the cells of a 2^4 factorial design, so the two groups of eight runs for a category S are not independent samples: the other three factors vary systematically within each group. The variance used by the unpaired two-sample t-test therefore includes the main effects and interactions of the other categories, not sampling noise. The design is naturally paired (each of the eight other-factor combinations appears once with S and once without), but no paired test or factorial ANOVA is reported. With one run per cell there is no pure error term unless higher-order interactions are assumed negligible, and no seed variance is given. The claims that short-short has no significant influence (Fig. 3) and that improvements are 'sparse' are therefore unsupported. Please state whether seeds are shared across runs, add replicates to estimate seed
  2. [§3, Figures 3–9] No adjustment for multiple comparisons is reported. Each heatmap performs dozens of tests at p<0.05 (20 category-level cells in Fig. 3, and many more at the task level in the appendix figures); under the null, one expects several false positives at this threshold. The 'sparse' conclusion is stated as if the highlighted significant cells were the true set of effects. The authors should apply a correction (e.g., Benjamini-Hochberg) or provide exact p-values and effect sizes. This compounds the invalid error term described above.
minor comments (6)
  1. [Figure 3 caption; Table 1 caption] 'Appendix 3' appears to be a typo; the results are in Appendix E.
  2. [§2, first paragraph] 'can led to' should be 'can lead to'.
  3. [Appendix D] 'standard evaluation protocol is which embeddings' should be 'in which embeddings'.
  4. [Figures 3 and 4 captions] The caption says 'Colored cells indicate statistically significant influence' while also saying the color encodes normalized influence. Clarify whether all cells are colored or only significant cells, and provide a legend separating significance from magnitude.
  5. [Appendix C] The text says the dataset, models, and code 'will be released upon publication,' which is inconsistent with the abstract and Section 2 claiming public release. Please clarify the availability timeline and repository status.
  6. [§3] The term 'influence function' is used for a simple marginal effect of including a category. This is not the standard influence-function estimate; consider renaming it 'marginal effect' or 'ablation effect'.

Circularity Check

0 steps flagged

No significant circularity: influence is measured on held-out MTEB, not derived from the synthetic-data inputs; self-citations are non-load-bearing tools.

full rationale

The paper's central claims are empirical. The influence estimates in §3 are computed by training 16 models on all subsets of four synthetic categories and comparing held-out MTEB performance between groups with and without each category. The MTEB benchmark is external to the training/generation pipeline, so the measured influence is not defined in terms of the outcome it is said to predict. No parameter is fit to MTEB scores to produce the 'sparse/localized' conclusion; the conclusion is a reading of the resulting group-mean differences and t-tests. The self-citations (Springer et al. 2024 for the public E5 replication, BehnamGhader et al. for architecture choices) cite public datasets/methods and are used as tools, not as authority for the paper's main empirical claims. The Limitations section acknowledges uncontrolled base-dataset interactions and generalizability to stronger generators, which are scope limitations rather than circular reductions. The main methodological risk—using an unpaired t-test with one run per 2^4 combination and no seed variance—would undermine the statistical support for the null claims, but that is a validity concern, not a circularity: the reported estimates are not equal to their inputs by construction. Therefore no circular step is identified.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper's central empirical claims rest on the representativeness of its synthetic data reproduction, the validity of MTEB as an evaluation, the base dataset used, and the choice of which synthetic categories to include in the influence analysis. None of these are fitted; they are choices that bound the scope of the conclusions.

axioms (4)
  • domain assumption Synthetic data generated by Llama-3.1-8B and 70B is representative of the proprietary GPT-based synthetic data used by Wang et al. for the purpose of the conclusions.
    The paper generalizes from its reproduction to 'synthetic data' broadly; the authors flag this as a limitation in the Limitations section.
  • domain assumption MTEB, including the faster retrieval subsets, is a valid proxy for general-purpose embedding quality.
    Evaluation throughout relies on MTEB; faster retrieval sets are justified by citation to Enevoldsen et al. (2025) in Appendix D.
  • domain assumption The public E5 dataset replication from Springer et al. (2024) is a faithful base set for isolating the effect of adding synthetic data.
    All training runs use this base set; the paper notes potential interactions with synthetic data as a limitation in the Limitations section.
  • domain assumption The four synthetic categories (short-short, short-long, long-long, long-short) capture the effective variation in synthetic data relevant to the influence analysis.
    Bitext and STS categories are excluded from Section 3's influence analysis; the paper does not justify this exclusion explicitly.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding the Influence of Synthetic Data for Text Embedders." pith.science (2026). https://pith.science/paper/BS62WIYD

@misc{pith2026250906184,
  author       = {Pith},
  title        = {Pith review of: Understanding the Influence of Synthetic Data for Text Embedders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BS62WIYD}},
  note         = {Machine review of arXiv:2509.06184}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Recent progress in developing general purpose text embedders has been driven by training on ever-growing corpora of synthetic LLM-generated data. Nonetheless, no publicly available synthetic dataset exists, posing a barrier to studying its role for generalization. To address this issue, we first reproduce and publicly release the synthetic data proposed by Wang et al. (Mistral-E5). Our synthetic data is high quality and leads to consistent improvements in performance. Next, we critically examine where exactly synthetic data improves model generalization. Our analysis reveals that benefits from synthetic data are sparse and highly localized to individual datasets. Moreover, we observe trade-offs between the performance on different categories and data that benefits one task, degrades performance on another. Our findings highlight the limitations of current synthetic data approaches for building general-purpose embedders and challenge the notion that training on synthetic data leads to more robust embedding models across tasks.

Figures

Figures reproduced from arXiv: 2509.06184 by Aditi Raghunathan, Jacob Mitchell Springer, Marius Mosbach, Siva Reddy, Vaibhav Adlakha.

Figure 1
Figure 1. Figure 1: Performance on MTEB across time. Start￾ing with Wang et al. (2024), models trained on syn￾thetic data have led to considerable improvements on the MTEB leaderboard. We exclude more recent models that were trained on in-domain data. by leveraging GPT-4 to produce synthetic data that expands existing training datasets for embeddings to new tasks (see [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Improvement in MTEB score when adding synthetic data to the training mixture. Across all set￾tings, our results are consistent with Wang et al. (2024), showing that training on synthetic data leads to higher MTEB performance. datasets along with the details of our evaluation, see Appendix D. Synthetic data leads to improvements on MTEB. We compare our synthetic data to Wang et al. (2024), and Chen et al. (… view at source ↗
Figure 4
Figure 4. Figure 4: Influence of different training data categories [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of cross-influence results for different models and synthetic datasets. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Detailed breakdown of Mistral-v0.2-7B influence on various tasks with LLaMA-3.1-8B synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Detailed breakdown of Mistral-v0.2-7B influence on various tasks with LLaMA-3.1-70B synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Detailed breakdown of Qwen2-1.5B influence on various tasks with LLaMA-3.1-8B synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Detailed breakdown of Qwen2-1.5B influence on various tasks with LLaMA-3.1-70B synthetic data. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 11 canonical work pages · 1 internal anchor

  1. [5]

    Preprint, arXiv:2407.21783

    The llama 3 herd of models. Preprint, arXiv:2407.21783. Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz

    Pith/arXiv arXiv

  2. [6]

    In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–2037, Dubrovnik, Croatia

    MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho

    work page 2014

  3. [7]

    (2022) were among the first to demon- strate that a powerful decoder -only LLM can be transformed into a high -quality text encoder

    C Training details Wang et al. (2022) were among the first to demon- strate that a powerful decoder -only LLM can be transformed into a high -quality text encoder. To obtain text embedding, they appended an [EOS] token to each input and constructed an embedding form its last layer representation. They fine-tuned on sentence-pair data using a contrastive l...

    work page 2022

  4. [9]

    In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423, Singapore

    Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423, Singapore. Association for Com- putational Linguistics. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff

    work page 2023

  5. [10]

    Preprint, arXiv:2407.10671

    Qwen2 techni- cal report. Preprint, arXiv:2407.10671. A Background Let V be a finite vocabulary of tokens. A se- quence s ∈ V∗ is a finite concatenation of to- kens s = (s1, . . . , s∣s∣) where each si ∈ V. Our goal is to train an embedder ϕ ∶ V∗ → Rd, pa- rameterized by θ ∈ Rp, which maps sequences to embeddings. The ideal embedder accurately esti- mates...

    Pith/arXiv arXiv 2023

  6. [12]

    They per- form parameter-efficient fine-tuning via LoRA (Hu et al., 2022), using a batch size of

    on 1.8 million sentence pairs. They per- form parameter-efficient fine-tuning via LoRA (Hu et al., 2022), using a batch size of

    work page 2022

  7. [14]

    We use the AdamW optimizer with a learning rate of4e−4, linear learn- ing rate warm-up for the first 100 steps, and weight decay with 0.1 coefficient afterwards

    We use a maxi- mum sequence length of 512 tokens for fair compar- ison to previous approaches. We use the AdamW optimizer with a learning rate of4e−4, linear learn- ing rate warm-up for the first 100 steps, and weight decay with 0.1 coefficient afterwards. We train all models for one epoch. Training Mistral-7B on public + synthetic data (∼2M samples) take...

    work page 2020

  8. [15]

    In MTEB, every task is reformulated as an em- bedding task where the only requirement is that the model produces a vector (embedding) for each text input

    include thousands of queries with a 6M document corpus. In MTEB, every task is reformulated as an em- bedding task where the only requirement is that the model produces a vector (embedding) for each text input. For example, in classification, MTEB uses the embeddings as fixed features and trains a lightweight linear classifier (typically logistic re- gres...

    work page 2025

  9. [16]

    query":

    In general, we observe similar trends from the main paper. Cls. Clust. P. Clas. Rera. Retr . STS Sum. Mean # of datasets → 12 11 3 4 15 10 1 56 Qwen2-1.5B Public data only 70.2 42.3 83.1 51.6 48.2 76.2 31.4 58.3 Qwen2-1.5B LLaMA-3.1-8B 76.6 47.0 86.2 55.5 54.1 80.2 30.8 63.4 Qwen2-1.5B LLaMA-3.1-70B 76.5 47.2 86.9 55.0 54.4 79.3 29.7 63.3 Mis.-v0.1-7B Pub...

    work page 2018

  10. [2018]

    arXiv preprint arXiv:1611.09268

    Ms marco: A human gener- ated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy

    Pith/arXiv arXiv

  11. [2019]

    arXiv preprint arXiv:1904.08375

    Document expansion by query prediction. arXiv preprint arXiv:1904.08375. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe

    Pith/arXiv arXiv 1904

  12. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online

    Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping

    work page 2020

  13. [2021]

    In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 671–688, Punta Cana, Dominican Republic

    TSDAE: Using transformer-based sequential denois- ing auto-encoderfor unsupervised sentence embed- ding learning. In Findings of the Association for Computational Linguistics: EMNLP 2021 , pages 671–688, Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang...

    work page 2021

  14. [2023]

    Preprint, arXiv:2310.06825

    Mistral 7b. Preprint, arXiv:2310.06825. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih

    Pith/arXiv arXiv

  15. [2024]

    Little Giants: Synthesizing High-Quality Embedding Data at Scale

    Little giants: Synthesizing high-quality embedding data at scale. arXiv preprint arXiv:2410.18634. Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming-Wei Chang

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [2048]

    Our training procedure largely follows Wang et al., but we make minor modifications inspired by subsequent work

    The entire fine-tuning process takes roughly 18 hours on 32 V100 GPUs. Our training procedure largely follows Wang et al., but we make minor modifications inspired by subsequent work. First, we use a more re- cent base model from the same model family (Mistral-7B-Instruct-v0.2). Next, following BehnamGhader et al. (2024) and Muennighoff et al. (2024), we ...

    work page 2024

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.