REVIEW 3 major objections 5 minor 11 references
The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks
T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read With 75 real examples, GPT-4o-generated texts match a full training set
desk verdict A practical synthetic-data recipe for applied text classification, with a plausible 75-example threshold that still needs a leak-proof CV re-run before the headline numbers can be believed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The synthetic imputation generation loop is the load-bearing mechanism: for each synthetic text, five original examples are drawn at random with replacement and inserted into a fixed prompt that instructs GPT-4o to write a new paragraph or speech that is different in content and sentence structure but retains the category's underlying meaning. The random draws are meant to keep the generated texts varied enough to avoid overfitting, while the five examples anchor the texts semantically so the classifier can learn the category. These synthetic texts are then pooled with the original examples and used to fine-tune a RoBERTa classifier, and the resulting F1 is compared against models trained on original data alone and on SSMBA-augmented data.
What would settle it
Run the same 50/75/100-example protocol with explicit cross-validation folds and keep the five prompt examples for each synthetic text out of the training folds; if the reported F1 parity at 75 examples drops materially or disappears, the result depends on leakage rather than on the generated texts themselves.
Extended reading notes
Core claim
The central claim is that a generative LLM can serve as a data imputer for underrepresented categories in supervised text classification. The author shows that prompting GPT-4o with five examples randomly drawn with replacement from the available original texts, together with an instruction to vary content and structure while preserving the category's tone, produces synthetic texts that let a fine-tuned RoBERTa classifier match the F1 performance of a model trained on the full original dataset once at least 75 original examples are available. At 50 original examples, the synthetic-imputed model overfits by only about 2 to 4 percent in a consistent upward direction, so the researcher can report a penalized performance score. This performance beats both including no synthetic data and the SSMBA augmentation baseline, which tends to generate texts so similar to the originals that the classifier's scores are inflated.
Load-bearing premise
The central claim depends on the assumption that a random draw of five original examples gives GPT-4o enough of the category's meaning to write texts that are both faithful and varied, and that in cross-validation none of those five prompt examples is allowed to sit in the training folds for a synthetic text in the validation folds.
Editorial extensions
If this is right
- Applied researchers with at least 75 original texts per rare category can augment up to roughly 200 examples per category and expect classifier performance equivalent to a fully labeled dataset.
- At 50 original examples, overfitting is small and consistently positive, so researchers can report a predictable penalized F1 rather than accept the large performance drop from training on scarce original data alone.
- Synthetic imputation outperforms SSMBA in these tests because its generated texts are more varied, reducing the inflated performance that comes from near-duplicate training examples.
- Projects that would otherwise be abandoned or that would merge rare categories into broader, less precise categories can retain the original categories without incurring the usual measurement bias.
- Because category-level F1 improves with the added synthetic texts, overall model F1 also improves even when the rare category is a minority of the data.
Reading between the lines
- The 75-example threshold is probably task-dependent: harder or more heterogeneous categories may need more originals, and easier ones fewer, because the five randomly chosen examples must capture the category's full meaning.
- A stronger test would hold the five prompt examples out of the training folds during cross-validation; the paper does not state whether its CV assignment does this, so a replication with explicit fold separation would clarify how much of the reported parity comes from synthetic text quality rather than prompt leakage.
- Because the prompt procedure is model-agnostic in principle, the same protocol should transfer to open-weight generative models, though weaker generators may produce less varied or less faithful imputations and require prompt adjustments.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes 'synthetic imputation': generating synthetic texts with GPT-4o, prompted with a general instruction and five random original examples drawn with replacement, to augment underrepresented categories in supervised text classification. It tests the approach on two applications (political nostalgia sentences and international speeches), fine-tuning RoBERTa-large with 10-times repeated 10-fold cross-validation and comparing against SSMBA and against no-augmentation baselines. The paper claims that with at least 75 original examples, the augmented training set matches full-sample performance, and that at 50 examples the overfitting is small, predictable, and in a constant direction, allowing researchers to report penalized performance scores.
Significance. If the central empirical claims survive scrutiny, the paper offers a practical and easy-to-implement augmentation recipe for social-science text classification with rare categories, with the advantage that the generator is a general-purpose LLM rather than a task-specific model. The manuscript is clearly written and provides full prompt text, dataset details, and replication materials, and it benchmarks against an established augmentation method (SSMBA). The main strength is the concrete workflow and the explicit comparison; the main weakness is that the evaluation protocol for synthetic texts in cross-validation is unspecified, which directly affects every reported performance number and the headline parity and overfitting claims.
major comments (3)
- [Generating Data for the Synthetic Imputation Procedure / Analysis and Results] The manuscript never states how synthetic texts are assigned to the 10-fold CV splits or whether the five prompt examples are restricted to the training portion of each fold. The generation section says the five examples are 'drawn at random each time from the sub-sample of original speeches,' and the Analysis section only says 'I run 10-times repeated 10-fold cross-validation.' If the synthetic corpus is pooled with the original texts before splitting, then synthetic texts derived from originals can appear in validation folds while their source originals appear in training folds, and validation-fold originals can appear as prompt examples for synthetic texts in training folds. Both pathways leak label information and would inflate the reported F1 values, including the 75-original parity result (0.829 vs. 0.822) and the 50-original overfitting estimates (3.5% and 3.1%). The authors must specify the fold-assignment protocol; if synthetic texts were generated per training fold with prompt examples drawn only from that fold's training data, this must be stated explicitly. If not, the experiments need to be re-run under that leakage-safe protocol.
- [Analysis and Results (Figures 2 and 3)] The parity claim at 75 original examples is based on an informal 'within one standard deviation' comparison. This is not a valid test of whether two CV mean F1 scores are equal: the reported standard deviations are across the 10 repeated CV runs, and the difference of 0.007 in the nostalgia case is not tested against the standard error of the difference. The authors should report a paired test across the 10 repetitions (e.g., a paired t-test or signed-rank test, with appropriate caution about non-independence) or a confidence interval for the difference between the synthetic-imputation model and the full-sample model at each original-sample size. Without this, the central claim that performance is 'on par' is not statistically supported.
- [Conclusion and Abstract] The threshold claim 'with 75 original examples or more' is presented as a general result, but it is estimated from only two datasets, and the paper does not test values between 75 and 100 in the nostalgia task or between 75 and 150 in the speeches task. The paper's own discussion of batch sizes implies the required number is task-dependent. The abstract and conclusion should either present the 75-example result as an empirical observation from these two applications, with explicit caveats about task-dependence, or provide additional evidence (e.g., subsampling at intermediate sizes on both datasets) before making a general threshold claim.
minor comments (5)
- [Analysis and Results] The Analysis section contains an internal inconsistency: it says 'Each set of three models cover the different data structures with 50, 100, and 150 original observations,' but the experiments for the nostalgia dataset use 50, 75, and 100 original observations, as stated both in the generation section and in the next sentence of the same paragraph. Please correct the 150 to 75.
- [Throughout] The phrases '(citation omitted)' appear twice (in the discussion of alternative generative LLMs and in the few-shot learning paragraph); these should be replaced with actual references or removed.
- [Analysis and Results] Use consistent terminology: '10-time repeated 10-fold cross-validation' should be '10-times repeated 10-fold cross-validation' or '10 repetitions of 10-fold cross-validation.'
- [Figures 2 and 3] Figures 2 and 3 report mean F1 scores but do not show uncertainty; the text mentions standard deviations, and the figures would benefit from error bars or shaded bands so the reader can assess the claimed parity visually.
- [Generalization section] The Generalization section argues that the approach will transfer because RoBERTa and GPT-4 are general models; this is a plausibility argument, and the paper would be stronger if it explicitly acknowledged that external validity beyond the two applications is an open question.
Circularity Check
No circular derivation: the 75-example parity claim is an empirical benchmark result, not a quantity constructed from its own inputs; the CV protocol ambiguity is a validity concern, not demonstrated circularity.
full rationale
This paper is an empirical evaluation, not a derivation. The central claim—that synthetic imputation with at least 75 original examples matches full-sample F1—is a measured outcome on two external benchmarks (Müller and Proksch nostalgia data; Hawkins et al. Global Populism Database), compared against SSMBA and no-augmentation baselines. No parameter is fitted to the target F1 values, and no uniqueness or identifiability theorem is imported. The self-citations to Timoneda and Vallejo Vera set hyperparameters, batch conventions, and model-selection details, but they do not carry the parity claim. The synthetic texts are generated by a stated rule (five randomly drawn original examples plus a prompt), and the paper never claims to derive a result from that rule by construction. The one genuine concern is an unstated cross-validation protocol: the paper says 'the five examples are drawn at random each time from the sub-sample of original speeches and passed inside the prompt' and later only that 'I run 10-times repeated 10-fold cross-validation' without stating whether synthetic texts or their prompt-source originals were confined to training folds. If pooling occurred before splitting, validation F1s could be inflated by leakage. However, this would be a methodological validity flaw, not a circular reduction; the text does not exhibit a specific equation or construction by which an output equals its input. The paper also contains an internal inconsistency—the Analysis section refers to nostalgia designs with '50, 100, and 150' original observations while the data-generation section and figures use 50/75/100—further underscoring that the protocol is under-specified, but again this is not circularity. The stated limitations (footnotes 3, 11, 31) acknowledge overfitting direction, below-50 behavior, and downstream measurement error, and they do not hide a circular step. Under the rule that circularity must be exhibited by quotation and reduction, no such step is present.
Assumptions & free parameters
free parameters (2)
- k (number of prompt examples) =
5
- target category size after imputation =
151 (nostalgia), 218 (speeches)
assumptions (4)
- domain assumption GPT-4o produces synthetic texts that preserve the category's substantive meaning while varying content and sentence structure enough to avoid overfitting.
- domain assumption The five randomly drawn examples adequately represent the category's distribution.
- domain assumption Cross-validation does not leak the prompt examples into the training folds for synthetic texts in the validation folds.
- domain assumption RoBERTa-large hyperparameters from prior work (learning rates, epochs, batch size) transfer to these tasks.
Cite this review
Pith. "Pith review of The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks." pith.science (2026). https://pith.science/paper/NN7QIP7V
@misc{pith2026250415160,
author = {Pith},
title = {Pith review of: The Synthetic Imputation Approach: Generating Optimal Synthetic Texts For Underrepresented Categories In Supervised Classification Tasks},
year = {2026},
howpublished = {\url{https://pith.science/paper/NN7QIP7V}},
note = {Machine review of arXiv:2504.15160}
}
read the original abstract
Encoder-decoder Large Language Models (LLMs), such as BERT and RoBERTa, require that all categories in an annotation task be sufficiently represented in the training data for optimal performance. However, it is often difficult to find sufficient examples for all categories in a task when building a high-quality training set. In this article, I describe this problem and propose a solution, the synthetic imputation approach. Leveraging a generative LLM (GPT-4o), this approach generates synthetic texts based on careful prompting and five original examples drawn randomly with replacement from the sample. This approach ensures that new synthetic texts are sufficiently different from the original texts to reduce overfitting, but retain the underlying substantive meaning of the examples to maximize out-of-sample performance. With 75 original examples or more, synthetic imputation's performance is on par with a full sample of original texts, and overfitting remains low, predictable and correctable with 50 original samples. The synthetic imputation approach provides a novel role for generative LLMs in research and allows applied researchers to balance their datasets for best performance.
Figures
Reference graph
Works this paper leans on
-
[4]
SSMBA: Self-supervised man- ifold based data augmentation for improving out-of-domain robustness
“SSMBA: Self-supervised man- ifold based data augmentation for improving out-of-domain robustness.” arXiv preprint arXiv:2009.10195 . Nie, Yuyang, Yuanhe Tian, Xiang Wan, Yan Song and Bo Dai
arXiv 2009
-
[5]
Named entity recognition for social media texts with semantic augmentation
“Named entity recognition for social media texts with semantic augmentation.” arXiv preprint arXiv:2010.15458 . Ornstein, Joseph T, Elise N Blasingame and Jake S Truscott
arXiv 2010
-
[8]
Text Augmentation in a Multi-Task View
“Text augmentation in a multi-task view.” arXiv preprint arXiv:2101.05469 . Wei, Jason and Kai Zou
-
[9]
Eda: Easy data augmentation techniques for boosting performance on text classification tasks
“Eda: Easy data augmentation techniques for boosting performance on text classification tasks.” arXiv preprint arXiv:1901.11196 . Wettig, Alexander, Tianyu Gao, Zexuan Zhong and Danqi Chen
arXiv 1901
-
[2019]
Roberta: A robustly optimized bert pretraining approach
“Roberta: A robustly optimized bert pretraining approach.” arXiv preprint arXiv:1907.11692 . M¨ uller, Stefan and Sven-Oliver Proksch
arXiv 1907
-
[2020]
Data boost: Text data augmentation through reinforcement learning guided conditional gener- ation
“Data boost: Text data augmentation through reinforcement learning guided conditional gener- ation.” arXiv preprint arXiv:2012.02952 . Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov
arXiv 2012
-
[2021]
A survey of data augmentation approaches for NLP
“A survey of data augmentation approaches for NLP.” arXiv preprint arXiv:2105.03075 . Gao, Andrew
-
[2022]
Should you mask 15% in masked language modeling?
“Should you mask 15% in masked language modeling?” arXiv preprint arXiv:2202.08005 . White, Jules, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf El- nashar, Jesse Spencer-Smith and Douglas C Schmidt
Show all 11 references
-
[2023]
A prompt pattern catalog to enhance prompt engineering with chatgpt
“A prompt pattern catalog to enhance prompt engineering with chatgpt.” arXiv preprint arXiv:2302.11382 . Zhou, Wenxuan, Bill Yuchen Lin and Xiang Ren
-
[2024]
Identifying the sources of ideological bias in GPT models through linguistic variation in output
“Identifying the sources of ideological bias in GPT models through linguistic variation in output.” arXiv preprint arXiv:2409.06043 . Wei, Jason, Chengyu Huang, Shiqi Xu and Soroush Vosoughi
-
[2025]
Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks
“Memory Is All You Need: Testing How Model Memory Affects LLM Performance in Annotation Tasks.” arXiv preprint arXiv:2503.04874 . Timoneda, Joan C. and Sebasti´ an Vallejo Vera. 2025 a. “Behind the mask: Random and se- lective masking in transformer models applied to specializ...
2025 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.