{"id":"a4bf70b4-7143-4ee4-a893-7d8fafdf080d","arxiv_id":"2412.03679","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"AgoraBench shows that an LM's ability to solve problems does not predict its ability to generate useful synthetic training data.","lead":"This paper introduces AgoraBench, a benchmark that compares six language models as generators of synthetic training data across math, code, and instruction-following. It finds that problem-solving strength does not predict data-generation quality, and that cheaper models can sometimes be better.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PCA \"93.4% variance explained\" appears to be variance of intrinsic metrics, not of PGR; the central claim rests on this conflation.","rationale":"The reader's weakest assumption concerned PGR stability under different student models, seeds, and data volumes. My concern is complementary: even if PGR were stable, the predictive analysis in §5.2 overstates its evidence because the 93.4% figure describes variance of the intrinsic feature matrix, not variance of PGR; the actual regression R^2 is 0.325 in-sample. The reader did flag the ambiguity of the PCA variance in the rationale and asked for out-of-sample validation, but the weakest_assumption field identified a different issue. The benchmark, PGR table, and model-level comparisons remain a solid contribution, so the conditional verdict is appropriate; the required conditions should include correcting the 93.4% claim and reporting out-of-sample predictive performance.","tokens_in":22195,"tokens_out":3216,"duration_ms":32819,"concrete_test":"Recompute the PCA-prediction pipeline in §5.2: (1) Regress standardized PGR on the top-5 PC scores and report the R^2; verify whether it equals 0.325 (Figure 6) rather than 93.4%. (2) Run leave-one-out cross-validation for this regression and report out-of-sample R^2. If the out-of-sample R^2 is near zero or negative, the claim that intrinsic features are better predictors of PGR is not established. (3) Relabel Figure 5 to distinguish \"cumulative variance of intrinsic features explained by PCs\" from \"variance of PGR explained by PCs\" and correct any use of 93.4% in the abstract if it refers to the former.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that intrinsic data features \"collectively serve as better indicators\" of data generation ability rests on Figure 5 and the abstract: \"top-5 principal components extracted from intrinsic measurements can explain 93.4% of the variance in the PGR values.\" However, PCA was fit on the intrinsic feature matrix (nine features), and the percentages 39.2%, 30.4%, 11.9%, 7.0%, and 4.9% sum to 93.4% of the variance of those standardized features, not of PGR. The actual regression of PGR on the same top-5 principal components, shown in Figure 6, yields R^2 = 0.325. Thus the 93.4% figure is not the explained variance of PGR; the real in-sample explanatory power is about one-third, over n = 54 observations with 5 predictors, and no out-of-sample validation is reported. Taking the abstract literally, the statement is not supported by the reported analysis. Since the conclusion that intrinsic features are better indicators than problem-solving ability depends on this quantitative evidence, the conflation is load-bearing, even though the AgoraBench benchmark itself remains useful.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AgoraBench, a benchmark for comparing LMs as synthetic data generators under controlled conditions. It fixes meta-prompts, seed datasets, data volume, and student model (Llama-3.1-8B), and measures teacher quality by Performance Gap Recovered (PGR), the relative improvement of a student SFT model over the base model toward Llama-3.1-8B-Instruct. Across three domains (math, code, instruction following) and three generation methods (instance generation, response generation, quality enhancement), six generators produce 10K instances each, yielding 99 student models. The paper reports (i) generator-specific strengths (e.g., GPT-4o in instance generation, Claude-3.5-Sonnet in quality enhancement), (ii) weak correlation between problem-solving ability and PGR, (iii) an intrinsic-feature PCA analysis claiming top-5 components explain 93.4% of variance in PGR, with a linear regression R²=0.325, and (iv) experiments on cost, scaling, and meta-prompt format.","tokens_in":22392,"tokens_out":6638,"duration_ms":63815,"significance":"AgoraBench addresses a real gap: previous synthetic-data work varies too many factors to compare generators. The fully specified protocol, public code/checkpoints/data, 1.26M generated instances, and 99 trained student models are valuable resources. The PGR metric is a reasonable extrinsic measure, and the robustness checks on meta-prompts and data volume are useful. The headline non-correlation claim, if supported, would matter for model selection. However, the current quantitative evidence for the intrinsic-feature conclusion is weaker than presented, and the ranking claims lack variance estimates; the benchmark itself remains a useful contribution.","major_comments":[{"comment":"The claim that 'top-5 principal components ... can explain 93.4% of the variance in the PGR values' conflates two different quantities. The percentages in Figure 5 (39.2, 30.4, 11.9, 7.0, and 4.9) sum to 93.4% of the variance of the nine standardized intrinsic features, not of PGR. The actual regression of PGR on those components, reported in Figure 6, gives R²=0.325 on the same 54 observations. The abstract and Section 5.2 should be corrected to say that the PCs explain 93.4% of the intrinsic-feature variance; the explained variance of PGR is roughly one-third. Because the paper's conclusion that intrinsic features are better indicators than problem-solving ability rests on this evidence, the conflation is load-bearing.","section":"Section 5.2 / Figure 5 / Abstract"},{"comment":"The regression in Figure 6 is an in-sample fit: the regression coefficients are estimated on the same PGR values that Figure 6 then 'predicts,' with five predictors on n=54 observations and no out-of-sample or cross-validated assessment. The word 'predict' in the text therefore overstates what is shown; the adjusted R², cross-validated R², or a permutation test is needed before claiming intrinsic metrics are better predictors. Table 4 is also presented as contribution 'to the PGR results,' but the numbers are normalized average loading magnitudes across the principal components, which do not measure each feature's contribution to PGR; this should be relabeled or replaced with a regression-based contribution analysis.","section":"Section 5.2 / Figure 6 / Table 4"},{"comment":"The central ranking comparisons are point estimates from a single training seed (seed 42), a single student model (Llama-3.1-8B), and one data volume (10K instances). Several differences that drive the narrative are small relative to the likely noise (e.g., math instance generation 20.6 vs 16.1; quality enhancement averages 17.9 vs 5.5). Without multiple seeds, confidence intervals, or a bootstrap over benchmarks, Table 2 does not establish that the observed ordering is stable. The limitations section acknowledges the single base model and 10K scale but not the single seed. Either add variance estimates or soften the ranking claims to 'point estimates under one configuration.' This also affects the non-correlation analysis in Figure 4, whose R² values cannot be separated from seed noise.","section":"Section 4 / Table 2 / Appendix E"}],"minor_comments":[{"comment":"The caption mixes R and R² and the reported p-values appear inconsistent with the sample sizes; please report regression statistics (slope, SE, F, adjusted R²) and ensure axes match the text.","section":"Figure 4"},{"comment":"The sentence 'and average performance on AGORA BENCH are listed in Table 3. and average performance on AGORA BENCH for all six LMs' is garbled and should be rewritten.","section":"Section 4, paragraph after Table 3"},{"comment":"The row labeled 'Instruction Difficulty (Perplexity)' is misleading because the text defines the feature as response perplexity conditioned on the instruction; correct the row label to avoid confusing it with instruction difficulty.","section":"Table 8"},{"comment":"The horizontal axis is unlabeled and has no tick values, making the scaling comparison difficult to verify; adding the actual data volumes and cost ratios would strengthen the presentation.","section":"Figure 7"},{"comment":"PGR is undefined if the reference model and base model have equal scores on a benchmark; a sentence noting degenerate cases would be helpful for practitioners applying the metric.","section":"Section 2, Equation (2)"}],"recommendation":"major_revision","confidential_remarks":"The benchmark itself is a worthwhile contribution and the main statistical issue is correctable; I would not reject the paper on the basis of the 93.4% conflation if the authors re-run the analysis with cross-validation and rephrase the claims. The single-seed issue is more awkward because re-running 99 student models with multiple seeds is expensive, but at minimum the paper should report uncertainty or explicitly reframe the ranking claims as point estimates. The positioning relative to Xu et al. (2024c) is handled fairly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. AgoraBench is a real contribution: it standardizes nine settings (three domains times three generation methods), controls meta-prompts, seed data, instance counts, and the student model, and the PGR metric sensibly measures how much of a reference model's post-training gap a synthetic dataset recovers. The paper also includes careful scaling and meta-prompt robustness checks, which is more than most benchmark papers do. But the central analytical claim leans on a misread number: the abstract says top-5 principal components from intrinsic measurements explain 93.4% of variance in PGR, when that 93.4% is actually the cumulative variance of the intrinsic feature matrix. The regression of PGR on those same five components gives R^2 = 0.325, in-sample, on 54 points with five predictors. That is a meaningful difference, and it is the basis for saying intrinsic features are better indicators than problem-solving ability.\n\nWhat is new and good: this is the first systematic comparison of LMs as generators across instance generation, response generation, and quality enhancement in one controlled framework. The observation that weaker solvers can beat stronger generators in some settings is interesting and worth following up, though with only six generators it is a pattern, not a law. The cost analysis is practical and honest. Contemporaneous work by Xu et al. (2024c) only covered response generation, so the benchmark occupies a distinct slot. Code, data, and checkpoints are public, which makes the whole thing more credible.\n\nSoft spots, in proportion: beyond the PCA wording, the experiments use a single training seed, a single student model (Llama-3.1-8B), and no confidence intervals. Many PGR differences are small (20.6 vs. 16.1 in math instance generation), so fine-grained rankings should not be over-read. The authors explicitly acknowledge the base-model and 10K-instance limitations, and they did run partial scaling experiments to 50K, which deserves credit. Still, the paper's strongest conceptual claim about prediction requires out-of-sample validation and ideally multiple seeds before it is presented as a finding.\n\nBottom line: practitioners choosing data generators will get real value from AgoraBench, and the benchmark itself is citable. The paper deserves a serious referee, but the referee should push for corrected PCA reporting and multi-seed evidence. I would bring it to reading group, mostly to argue about the analysis section.","headline":"A genuinely useful benchmark with a misstated PCA claim; the headline finding about intrinsic indicators needs rework before it can be trusted at full strength.","tokens_in":22971,"tokens_out":1919,"would_cite":true,"duration_ms":21187,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgoraBench, a controlled benchmark that trains 99 student models on synthetic data from six language models, finds that data-generation ability does not track problem-solving ability and that intrinsic data-quality signals predict it…","keywords":["synthetic data generation","language model evaluation","data generation ability","Performance Gap Recovered","instruction difficulty","response quality","data diversity","model selection"],"falsifier":"Take the same six generators and the same nine settings, but train the student model on 50,000 instances per setting and with a second base model (e.g., Llama-3.1-70B-Base or Qwen2.5-7B), then recompute PGR and the rank correlation with problem-solving scores. If the generator ordering changes substantially, or if the top-five principal components of intrinsic metrics no longer explain most PGR variance, the paper's central claims are specific to its 10K-instance, Llama-3.1-8B regime rather than general properties of data-generation ability.","tokens_in":21983,"feed_emoji":"🤖","tokens_out":7386,"duration_ms":64414,"temperature":0.7,"pith_summary":"The paper builds a standardized arena for ranking language models (LMs) as generators of synthetic post-training data. It creates nine settings (three domains—math, code, instruction-following—times three generation methods: instance generation, response generation, and quality enhancement), holds every variable except the generator LM fixed, and trains Llama-3.1-8B on 10,000 instances produced by each of six generator LMs. It then measures each generator by Performance Gap Recovered (PGR), the fraction of the gap between the base model and Llama-3.1-8B-Instruct that the generated data closes. The central claim is that a model's problem-solving score does not predict its data-generation score, and that a combination of intrinsic data features—instruction difficulty, response quality, and response perplexity—explains far more of the variation in student-model improvement. The paper argues this matters because practitioners choosing an LM for synthetic data currently lack a standardized way to compare data-generation ability.","feed_headline":"Better problem-solvers aren't better data generators","feed_subtitle":"A new benchmark trained 99 student models to rank six LMs as synthetic-data teachers.","key_machinery":"The central machinery is the PGR metric together with the controlled benchmark design. PGR is defined as $(\\text{score}_B(S_{D_G}) - \\text{score}_B(S_\\emptyset)) / (\\text{score}_B(S_{ref}) - \\text{score}_B(S_\\emptyset)) \\times 100$, where $S_\\emptyset$ is the base Llama-3.1-8B model, $S_{D_G}$ is that model supervised-fine-tuned on the generated data, and $S_{ref}$ is Llama-3.1-8B-Instruct; it reports how much of the reference post-training improvement the synthetic data recovers. The benchmark holds the meta-prompt, seed data, data volume, and student model fixed across generators, isolating the generator as the only variable. A second piece of machinery is the intrinsic-metric analysis: each generated dataset is scored on instruction difficulty, response quality (via two LLM-as-judge models and a reward model), response perplexity under the student model, and diversity, and principal component analysis is used to find low-dimensional combinations of these features that track PGR.","core_discovery":"On the paper's own terms, the discovery is that data generation ability is a distinct capability from problem solving. Across 1.26 million synthesized training instances and 99 trained student models, AgoraBench shows that no single LM dominates every data-generation setting: GPT-4o is strongest at generating new instances, Claude-3.5-Sonnet at enhancing existing ones, and cheaper or weaker models sometimes beat their stronger counterparts (e.g., Llama-3.1-8B-Instruct outperforms GPT-4o on code instance generation). Regression of benchmark problem-solving scores against PGR gives weak or non-significant correlations, while the top five principal components of intrinsic metrics (instruction difficulty, response quality from two judges and a reward model, response perplexity, and diversity) account for 93.4% of the variance in PGR. A linear regression on those components reaches statistical significance with $R^2 = 0.325$, against $R^2 < 0.1$ for problem-solving scores, leading the authors to conclude that intrinsic data-quality features collectively are the better indicator of an LM's data-generation ability.","pith_inferences":["Because PGR is measured with a single student base model (Llama-3.1-8B), the ordering of generators might change if the student were, say, a 70B or multilingual model; the paper's own limitations note this. A natural next experiment is to re-run the benchmark with a second student model and see whether the top-5 principal-component predictor transfers.","The 93.4% variance explained is computed over 54 setting-level measurements, not over individual instances, so it reflects how well intrinsic features separate settings rather than how accurately any single dataset's downstream gain can be forecast; a practitioner should expect larger error at the individual-dataset level.","The finding that weaker generators can outperform stronger ones hints that data-generation skill is trainable independently of problem-solving skill; one could test this by fine-tuning a small model specifically on high-PGR generated data and checking whether its PGR rises without changing its benchmark scores.","The meta-prompt comparison (unoptimized vs. optimized vs. JSON) implies that prompt engineering can shift PGR by about 4% on average, so future benchmarks should report the exact meta-prompt alongside any ranking."],"forward_implications":["Practitioners can choose data generators by matching method to model: GPT-4o for generating novel instances, Claude-3.5-Sonnet for refining existing data, and cheap models when scaling volume, since 50K instances from GPT-4o-mini can beat 10K from GPT-4o at lower cost.","Problem-solving leaderboards should not be used as a proxy for data-generation ability, because the regression shows the two are nearly uncorrelated.","Intrinsic data features could serve as a low-cost screening signal: measuring instruction difficulty, response quality, and perplexity of a candidate generator's output predicts a meaningful share of downstream student improvement without training a student model.","Output format matters: free-form meta-prompts outperform JSON-format prompts by about 4.5% in PGR, so structuring the generation format can silently change rankings.","Because no single model wins all nine settings, benchmarks like this one create a case for specialized data-generation models tuned for particular domains and methods."],"supporting_citations":[{"why":"Introduces instance generation via Self-Instruct, which the instance-generation setting follows.","marker":"Wang et al. (2023)"},{"why":"WizardLM's Evol-Instruct defines the quality-enhancement style of rewriting instructions to be more complex.","marker":"Xu et al. (2024a)"},{"why":"Magpie provides the instruction-only seed data and the design for the response-generation setting.","marker":"Xu et al. (2024b)"},{"why":"WebInstruct supplies the low-quality web instances used as seed data for quality enhancement in math and instruction-following.","marker":"Yue et al. (2024)"},{"why":"Provides the Llama-3.1 models used both as data generators and as the student/reference pair for PGR.","marker":"Dubey et al. (2024)"},{"why":"Documents the extensive post-training of Llama-3.1-8B-Instruct, the reference against which PGR measures recovered performance.","marker":"MetaAI (2024)"},{"why":"Supplies the PCA-based method the authors adopt for predicting performance from observable features.","marker":"Ruan et al. (2024)"},{"why":"Prometheus-2 is one of the LLM judges used to score response quality and instruction difficulty.","marker":"Kim et al. (2024)"},{"why":"Cited for the finding that structured output formats reduce LM output quality, which the meta-prompt experiments confirm.","marker":"Tam et al. (2024)"}],"fun_headline_variants":["Smarter models don't always make better data generators","Data quality, not model size, predicts synthetic data success","GPT-4o writes new problems, Claude-3.5 enhances existing ones","Cheaper LMs sometimes beat stronger ones at data generation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking rests on treating the PGR score computed with a single student model (Llama-3.1-8B), a single random seed, and 10,000 generated instances per setting as a stable measure of data-generation ability; if a different student model, more training data, or another seed reorders the generators, the headline conclusions about problem-solving vs. data-generation ability would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["Smarter models don't always make better data generators","Data quality, not model size, predicts synthetic data success","GPT-4o writes new problems, Claude-3.5 enhances existing ones","Cheaper LMs sometimes beat stronger ones at data generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000953,"raw_usage":{"total_tokens":4077,"prompt_tokens":971,"completion_tokens":3106,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":3035}},"tokens_in":587,"tokens_out":3106,"duration_ms":23838,"temperature":1.0,"reasoning_tokens":3035,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:12:05.216807+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same six generators and the same nine settings, but train the student model on 50,000 instances per setting and with a second base model (e.g., Llama-3.1-70B-Base or Qwen2.5-7B), then recompute PGR and the rank correlation with problem-solving scores. If the generator ordering changes substantially, or if the top-five principal components of intrinsic metrics no longer explain most PGR variance, the paper's central claims are specific to its 10K-instance, Llama-3.1-8B regime rather than general properties of data-generation ability.","supporting_citations":[{"cited_title":"A., Khashabi, D., and Hajishirzi, H","cited_arxiv_id":null,"evidence_quote":"Introduces instance generation via Self-Instruct, which the instance-generation setting follows."},{"cited_title":"MA mmo TH 2: Scaling instructions from the web","cited_arxiv_id":null,"evidence_quote":"WebInstruct supplies the low-quality web instances used as seed data for quality enhancement in math and instruction-following."},{"cited_title":"Introducing meta llama 3: The most capable openly available llm to date","cited_arxiv_id":null,"evidence_quote":"Documents the extensive post-training of Llama-3.1-8B-Instruct, the reference against which PGR measures recovered performance."},{"cited_title":"J., and Hashimoto, T","cited_arxiv_id":null,"evidence_quote":"Supplies the PCA-based method the authors adopt for predicting performance from observable features."},{"cited_title":"Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., and Seo, M","cited_arxiv_id":null,"evidence_quote":"Prometheus-2 is one of the LLM judges used to score response quality and instruction difficulty."}],"review_version":1}