Pith. sign in

REVIEW 4 major objections 6 minor 64 references

Data Swarms: Optimizable Generation of Synthetic Evaluation Data

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Particle swarm optimization over data-generator LLM weights turns benchmark construction into a measurable search problem: the paper claims the optimized generators beat eight baselines in 15 of 16 dataset-objective settings, with an…

desk verdict A genuinely new optimization angle on synthetic eval data, but the headline margins are inflated by best-of-20 selection on the reported metric; still worth a referee's time. read the letter →

arxiv 2506.00741 v2 pith:WZYDCLPA submitted 2025-05-31 cs.CL

Shangbin Feng , Yike Wang , Weijia Shi , Yulia Tsvetkov This is my paper
classification cs.CL
keywords syntheticevaluationdataparticleswarmoptimizationLLMgenerationadversarialco-evolutionobjectivesweight-spacesearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes treating the generation of synthetic evaluation data as an optimization problem: instead of hand-crafting prompts or heuristics, a swarm of question-writing language models is moved through weight space by particle swarm optimization to maximize quantitative evaluation objectives such as producing harder, more separating, more novel, more consistent, or more user-like questions. The authors claim that the resulting Data Swarms procedure outperforms eight data-generation baselines in 15 of 16 dataset-objective settings, and that an adversarial extension in which test-taker models co-evolve with the question generators improves fine-tuning by 8.2% on average over static or one-sided generated data. A sympathetic reader would care because it reframes benchmark construction as a measurable, automatable search problem rather than an artisanal, per-dataset engineering task.

What carries the argument

The carrying mechanism is particle swarm optimization applied to LLM checkpoints in weight space. Each data generator is a particle whose weights are updated by a velocity that combines inertia, attraction toward the personal best and global best checkpoints, and repulsion away from the global worst checkpoint. The idea is that interpolation between fine-tuned checkpoints yields new valid data generators with different question-writing behavior; the evaluation objective supplies the scalar fitness that guides the swarm. The adversarial variant alternates a particle swarm optimization step for the data generators with a particle swarm optimization step for the test-taker models, producing a sliding-window joint dataset on which both sides compete.

What would settle it

Use a held-out, human-verified validity check on the final generated questions and recompute the five objectives after counting invalid questions as failures instead of discarding them; if Data Swarms' advantage over the eight baselines shrinks or disappears under that accounting, the reported gains come substantially from filtering artifacts rather than from the weight-space search itself.

Watch

Extended reading notes

Core claim

The central claim is that the weight space of a fine-tuned LLM data generator is a searchable terrain in which particle swarm optimization can find new data generators that optimize evaluation desiderata. Starting from a seed dataset, the method clusters examples, trains several generator checkpoints, and then iteratively updates their weights toward the best-performing checkpoints and away from the worst-performing checkpoint, using the evaluation objective as the fitness signal. The paper reports that this yields evaluation data that is more difficult, more separating, more novel, and more consistent than data from the dev set, held-out set, initial generators, weight-averaged generators, Self-Instruct, and three objective-guided baselines across four domains, and that the adversarial variant improves held-out test performance of fine-tuned models by 8.2% on average. It also claims generalization to unseen test takers and successful optimization of weighted combinations of objectives.

Load-bearing premise

The search assumes that the region of weight space between and around the fine-tuned question-writing checkpoints is smooth enough that stepping from one checkpoint toward another produces new models that still write valid questions; the paper notes that 10-15% of generated math problems are invalid and are filtered by the same judge used for scoring, so this smoothness holds only approximately.

Editorial extensions

If this is right

  • Benchmark construction can be formulated as a quantitative optimization problem, so a new evaluation goal only requires defining the objective and rerunning the swarm search.
  • Evaluation data optimized against one pool of test-taker models transfers to other open and closed models, meaning benchmark builders do not need to know the final evaluated model in advance.
  • Adversarial Swarms turns synthetic data into a training signal as well as an evaluation signal: models improve by learning from the harder questions the swarm produces.
  • Weighted combinations of objectives can be optimized, so evaluation suites can balance hardness, separation, novelty, and consistency according to user priorities.
  • Because the method works on tasks with natural ground truths, such as graph reasoning, it does not depend on LLM-as-a-judge in domains where ground truth is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test not run in the paper: replace the particle swarm velocity update with random convex interpolations of the same checkpoints at equal compute; if random interpolation matches the objective gains, the swarm guidance is not the binding ingredient.
  • The observed 10-15% invalid-generation rate suggests an upper bound on how far the weight-space continuity assumption can be stretched; for larger generators or more open-ended domains the filter may need to become part of the objective rather than a post-hoc discard.
  • Since the personalized objective is independent of any test-taker model, the method could be adapted to optimize for user-representativeness of evaluation data, not just difficulty, which would matter for instruction-following and safety evaluation.
  • Adversarial Swarms points toward a red-team versus blue-team dynamic in benchmark construction: data and models can be co-evolved indefinitely, so evaluation data may become a moving target rather than a static resource.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes Data Swarms, an algorithm that starts from a seed dataset, trains multiple data generators via self-instruct on clusters, and then applies particle swarm optimization over the generator weights to maximize one of five evaluation objectives (difficult, separate, novel, consistent, personalized). An extension, Adversarial Swarms, alternates PSO steps for the data generators and for a pool of test-taker models. Experiments on Alpaca, GSM8K, TruthfulQA, and WikiDYK compare Data Swarms with eight baselines and report that it is best in 15 of 16 dataset-objective settings; additional analyses cover the personalized objective, natural ground-truth graph problems, generalization to unseen test takers, and compositions of objectives.

Significance. If the reported comparisons survive correction for selection, the paper offers a useful general framework for optimizing data generators toward quantitative evaluation desiderata, and the adversarial extension is a plausible mechanism for producing training data that improves held-out performance. The paper is explicit about its objectives, includes a human-annotation check of the LLM-as-a-judge, and includes a natural-ground-truth setting that avoids judge confounding. However, the core comparative claim is currently supported by an uncalibrated estimate of the objective: up to 20 random hyperparameter configurations are selected on the same noisy objective f that is later reported, and significance is provided for only one of five objectives. The paper also leaves the validity of weight-space interpolation as a data generator largely asserted rather than demonstrated. These issues are fixable in revision but are central to the quantitative claims.

major comments (4)
  1. [Section 3 / Appendix C] The main comparative claim (Section 4, '15 of the 16 dataset and objective settings') rests on selecting the best of up to 20 random hyperparameter configurations using the same evaluation objective f that is later reported (Appendix C: 'We run up to 20 runs by randomly choosing over these hyperparameter search settings and report the best-found data generator on evaluation objective f'; Section 3: 'running grid search over other hyperparameters and report the best-found data generator based on evaluation objective f'). Because f is estimated from only 200 generated items per iteration and scored by an LLM judge, taking the best of roughly 20 noisy evaluations yields an upward-biased value. The objective-guided baselines also select on f (Autobencher tries 10 subsets; TaskElicit and PromptBreeder retain the best), but with fewer candidates or less capacity, so the comparison is not apples-to-apples. Please report selection-corrected results, e.g., the mean or median across the random configurations, or evaluate the best configuration on a separate holdout set of generated items, and state the selection procedure explicitly in the main text.
  2. [Table 1 / Appendix C Table 5] Significance is reported only for the difficult objective (the stars in Table 5), while the abstract and Section 4 claim consistent gains on separate, novel, and consistent objectives as well. No significance tests are reported for those objectives, so the '15 of 16' claim and the '8.2% on average' improvement for Adversarial Swarms (Section 4, Figure 3) are not supported by inferential statistics. For Adversarial Swarms, Figure 3 also reports held-out test accuracy without error bars or tests across seeds. Please provide significance tests or confidence intervals for every reported objective and for the fine-tuning comparison.
  3. [Section 2.3, Algorithm 1, line 12 / Appendix A] The PSO update moves generator weights along a weighted difference of checkpoints; the paper cites Eilertsen et al. [8] for continuity of the weight neighborhood, but that reference concerns classifier weight spaces, not the validity or quality of language-model data generators. Appendix A concedes that 10-15% of generated math problems are invalid and that these are filtered by the same LLM judge used for scoring. Because the objective is computed after filtering, the reported improvements could partly reflect changes in the filtering rate rather than in data quality. Please (i) measure the invalid-problem rate along the PSO trajectory, (ii) validate with human judges on a sample of the final generated data, and (iii) compare against random-walk or interpolation baselines to demonstrate that the PSO update direction is necessary for the gains.
  4. [Section 2.2 / Table 1] Because Data Swarms explicitly optimizes the reported f, higher f on the same test-taker pool is partly by construction. This does not make the method uninteresting, but it weakens the interpretation of Table 1 as evidence of evaluation-quality improvement. The paper should separate optimization success from generalization by reporting (a) changes in each objective when optimizing a different objective (e.g., does optimizing difficult also improve separate or novel on held-out test takers?), and (b) comparisons on model-agnostic objectives such as personalized, where the objective does not depend on the test takers. Table 2's generalization to unseen test takers is a useful step but is reported only for difficult.
minor comments (6)
  1. [Section 3 / Appendix C] Section 3 says 'running grid search over other hyperparameters' while Appendix C says 'up to 20 runs by randomly choosing over these hyperparameter search settings'; please reconcile the two statements and specify the exact selection protocol.
  2. [Section 2.2] The definition of fpersonalized is a sum over generated items rather than an average; as written, its scale depends on the number of generated items. Please normalize it so that it is comparable across different generation budgets.
  3. [Figure 1] Figure 1 contains rendering artifacts (unicode arrows, missing objective labels) and is difficult to parse; please redraw it with standard math notation and clear captions.
  4. [Throughout] There are several typos: 'evaluaiton' (Appendix C), 'datset' (Algorithm 2), 'unfornuately' (Table 4 caption), 'genearl-purpose' and 'diffierent' (Appendix C), and 'velocty' (Section 2.3).
  5. [Appendix C, Table 5] The significance notation in Table 5 should be explained in the main text; currently it appears only in the appendix and only for the difficult objective.
  6. [Table 1] The consistent-objective values are all near 0.99, making raw differences hard to interpret; the footnote in Table 1 is helpful, but the paper should also report the underlying standard deviations or an effect-size measure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported objectives are the optimizer's targets and the load-bearing generalization claims are evaluated on unseen models and held-out data.

full rationale

Data Swarms is an empirical optimization method rather than a derivation from first principles. The five evaluation objectives are explicitly defined as optimization targets (§2.2) and the paper reports the objective values achieved by the returned global-best generator. Reporting higher values on the optimized objective is the expected behavior of an optimizer; the substantive claims are the comparisons against objective-guided baselines on the same objectives and, more importantly, the generalization experiments in Table 2, which evaluate f_difficult on three sets of unseen test takers M′ not used during optimization. The weight-space continuity assumption is supported by an external citation to Eilertsen et al. [8] and by the empirical filtering of invalid generations, not by a self-citation chain. The best-of-20 hyperparameter selection on the noisy objective (Appendix C) is a selection-bias and multiple-comparisons limitation that could inflate the reported margins, but it is not a circular reduction: the reported quantity is the algorithm's optimization value, not a prediction derived from a fit. The self-citations to Model Swarms [9,10] are motivational and not load-bearing. No equation or fitted parameter is reused as an independent prediction, so the derivation chain is not circular.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

The central claim rests on several domain assumptions about clustering, weight-space continuity, Self-Instruct generation quality, and judge reliability, plus hand-set hyperparameters. The most important free parameters are the PSO coefficients and the number of random search runs, because the paper selects the best configuration on the reported objective, turning the headline numbers into maxima rather than typical performance. No new physical entities are introduced.

free parameters (6)
  • PSO coefficients phi_v, phi_p, phi_g, phi_w, lambda = Best of grid search, not reported
    Appendix C: grid over phi_v in {0.1,0.2,0.3}, phi_p 0.1-0.5, phi_g 0.2-0.6, phi_w 0.01-0.1, lambda 0.5-1.0; best found generator selected on evaluation objective f, so the reported objective values are selected maxima.
  • Number of data generators / clusters N = 4
    Section 3 and baseline description use 4 initial generators (clusters); N is chosen by hand and not ablated.
  • Self-instruct in-context sample size k = 5
    Section 3: 'sample 10k in-context learning instances with k=5'; affects generator quality and is hand-set.
  • Fine-tuning epochs, learning rate, batch size = 5 epochs, lr 2e-4, batch 32
    Section 3 defaults for initial data generator training; not optimized per dataset.
  • PSO patience and max iterations (c, K) = c=5, K=30
    Section 3 defaults; hand-set and not ablated.
  • Number of random hyperparameter search runs = up to 20
    Appendix C: 'run up to 20 runs by randomly choosing over these hyperparameter search settings and report the best-found data generator on evaluation objective f'; this is selection on the reported metric.
assumptions (6)
  • domain assumption K-means clusters over average-pooled embeddings correspond to meaningful sub-evaluations of a dataset.
    Invoked in Section 2.1 when partitioning D into N subsets; if clustering does not track capability differences, the initial swarm's diversity is not useful.
  • domain assumption Weight-space neighborhoods of fine-tuned generators are locally continuous enough that PSO interpolation between checkpoints yields valid generators.
    Invoked in Section 2.3 via the velocity update and xi <- xi + lambda vi, citing Eilertsen et al. [8]; the paper itself observes 10-15% invalid math problems, so this holds only approximately.
  • domain assumption Self-Instruct fine-tuning on cluster samples produces generators capable of producing new in-distribution questions.
    Section 2.1 trains initial generators with Self-Instruct; no distributional validity check beyond later filtering.
  • domain assumption LLM-as-a-judge gives reliable quality scores for generated open-ended questions.
    Appendix C reports 92% agreement with human majority on 100 examples; the judge is also used to filter invalid questions, coupling validity and scoring.
  • domain assumption The five evaluation objectives (difficult, separate, novel, consistent, personalized) are appropriate quantitative desiderata for evaluation data.
    Section 2.2 defines these as the objective suite; downstream claims about better evaluation data depend on this normative choice.
  • domain assumption Fine-tuning Gemma-9B on four Tulu-v2 SFT domains yields a heterogeneous test-taker pool representative of LLM capabilities.
    Section 3 describes the four test takers; generalization experiments to other models test this assumption indirectly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Data Swarms: Optimizable Generation of Synthetic Evaluation Data." pith.science (2026). https://pith.science/paper/WZYDCLPA

@misc{pith2026250600741,
  author       = {Pith},
  title        = {Pith review of: Data Swarms: Optimizable Generation of Synthetic Evaluation Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WZYDCLPA}},
  note         = {Machine review of arXiv:2506.00741}
}
read the original abstract

We propose Data Swarms, an algorithm to optimize the generation of synthetic evaluation data and advance quantitative desiderata of LLM evaluation. We first train a swarm of initial data generators using existing data, and define various evaluation objectives to reflect the desired properties of evaluation (e.g., generate more difficult problems for the evaluated models) and quantitatively evaluate data generators. We then employ particle swarm optimization to optimize the swarm of data generators, where they collaboratively search through the model parameter space to find new generators that advance these objectives. We further extend it to Adversarial Swarms, where the data generator swarm generates harder data while the test taker model swarm learns from such data, co-evolving dynamically for better data and models simultaneously. Extensive experiments demonstrate that Data Swarms outperforms eight data generation baselines across five evaluation objectives, while Adversarial Swarms produce more robust learning of synthetic data and stronger generalization. Further analysis reveals that Data Swarms successfully optimizes compositions of multiple evaluation objectives and generalizes to new off-the-shelf LLMs, unseen at optimization time.

Figures

Figures reproduced from arXiv: 2506.00741 by the authors.

Figure 1
Figure 1. Overview of DATA SWARMS. (Left) Given a data generator, we sample evaluation problems, evaluate test taker LLMs on them, and calculate objectives. (§2.2) (Center) We optimize multiple data generators with particle swarm optimization to adapt them to evaluation objectives. Specifically, models are impacted by the personal/global best/worst checkpoints to update velocity and explore model weight space for optimization… view at source ↗
Figure 2
Figure 2. Performance on the window set and held-out set in [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance on the test set when fine-tuning on different data settings and using ADVERSARIAL SWARMS: ours outperforms fine-tuning by offering co-evolution of data genera￾tors and test takers in competition. Adversarial Swarms Data generators and test taker models compete on the sliding window subset of data (§2.4): data generators seek to generate harder questions (improving fdifficult) as the window set, and test … view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Results for the personal￾ized objective fpersonalized on Alpaca. DATA SWARMS outperforms baselines by 8.4%, flexibly adapting to real-world user instructions. We compare the best-found test taker model in ADVER￾SARIAL SWARMS with other ways of leveraging synthetic data…
Figure 5
Figure 5. Figure 5: Evaluating DATA SWARMS and baselines with composite evaluation objectives, weighted averages of difficult, separate, and novel objectives with one being dominant and having the largest weight. DATA SWARMS outperforms baselines in ten of the twelve dataset and objective…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

64 extracted references · 35 canonical work pages

  1. [8]

    Clas- sifying the classifier: dissecting the weight space of neural networks

    Gabriel Eilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Clas- sifying the classifier: dissecting the weight space of neural networks. InECAI 2020, pages 1119–1126. IOS Press, 2020

    work page 2020

  2. [1]

    Kgquiz: Evaluating the generalization of encoded knowledge in large language models

    Yuyang Bai, Shangbin Feng, Vidhisha Balachandran, Zhaoxuan Tan, Shiqi Lou, Tianxing He, and Yulia Tsvetkov. Kgquiz: Evaluating the generalization of encoded knowledge in large language models. InProceedings of the ACM Web Conference 2024, pages 2226–2237, 2024

    2024

  3. [2]

    Au- toeval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008, 2024

    Pierre Boyeau, Anastasios N Angelopoulos, Nir Yosef, Jitendra Malik, and Michael I Jordan. Au- toeval done right: Using synthetic data for model evaluation.arXiv preprint arXiv:2403.07008, 2024

    arXiv 2024

  4. [3]

    Adaptively evaluating models with task elicitation.arXiv preprint arXiv:2503.01986, 2025

    Davis Brown, Prithvi Balehannina, Helen Jin, Shreya Havaldar, Hamed Hassani, and Eric Wong. Adaptively evaluating models with task elicitation.arXiv preprint arXiv:2503.01986, 2025

    arXiv 2025

  5. [4]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [5]

    Knowledge crosswords: Geometric knowledge reasoning with large language models

    Wenxuan Ding, Shangbin Feng, Yuhan Liu, Zhaoxuan Tan, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge crosswords: Geometric knowledge reasoning with large language models. InFindings of the Association for Computational Linguistics ACL 2024, pages 2609–2636, 2024

    work page 2024

  7. [6]

    Self-boosting large language models with synthetic preference data.arXiv preprint arXiv:2410.06961, 2024

    Qingxiu Dong, Li Dong, Xingxing Zhang, Zhifang Sui, and Furu Wei. Self-boosting large language models with synthetic preference data.arXiv preprint arXiv:2410.06961, 2024

    arXiv 2024

  8. [7]

    Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36, 2024

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback.Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 64 references
  1. [9]

    Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems.arXiv preprint arXiv:2502.04510, 2025

    Shangbin Feng, Zifeng Wang, Palash Goyal, Yike Wang, Weijia Shi, Huang Xia, Hamid Palangi, Luke Zettlemoyer, Yulia Tsvetkov, Chen-Yu Lee, et al. Heterogeneous swarms: Jointly optimizing model roles and weights for multi-llm systems.arXiv preprint arXiv:2502.04510, 2025

    2025

  2. [10]

    Model swarms: Collaborative search to adapt LLM experts via swarm intelligence

    Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Collaborative search to adapt LLM experts via swarm intelligence. InICML, 2025

    2025

  3. [11]

    Promptbreeder: Self-referential self-improvement via prompt evolution

    Chrisantha Fernando, Dylan Sunil Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-referential self-improvement via prompt evolution. In Forty-first International Conference on Machine Learning, 2024

    2024

  4. [12]

    Open llm leaderboard v2

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_ llm_leaderboard, 2024

    2024

  5. [13]

    Time travel in llms: Tracing data contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Time travel in llms: Tracing data contamination in large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  6. [14]

    Automated evaluation of retrieval-augmented language models with task-specific exam generation

    Gauthier Guinet, Behrooz Omidvar-Tehrani, Anoop Deoras, and Laurent Callot. Automated evaluation of retrieval-augmented language models with task-specific exam generation. In Forty-first International Conference on Machine Learning, 2024

    2024

  7. [15]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021. 10

    2021

  8. [16]

    Datagen: Unified synthetic dataset generation via large language models

    Yue Huang, Siyuan Wu, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Chaowei Xiao, Jianfeng Gao, Lichao Sun, et al. Datagen: Unified synthetic dataset generation via large language models. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  9. [17]

    Camels in a changing climate: Enhancing lm adaptation with tulu 2.arXiv preprint arXiv:2311.10702, 2023

    Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing lm adaptation with tulu 2.arXiv preprint arXiv:2311.10702, 2023

    2023 arXiv

  10. [18]

    Wildteaming at scale: From in- the-wild jailbreaks to (adversarially) safer language models.Advances in Neural Information Processing Systems, 37:47094–47165, 2024

    Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. Wildteaming at scale: From in- the-wild jailbreaks to (adversarially) safer language models.Advances in Neural Information Pro...

    2024

  11. [19]

    Teaching language models to hallucinate less with synthetic tasks

    Erik Jones, Hamid Palangi, Clarisse Simões Ribeiro, Varun Chandrasekaran, Subhabrata Mukherjee, Arindam Mitra, Ahmed Hassan Awadallah, and Ece Kamar. Teaching language models to hallucinate less with synthetic tasks. InThe Twelfth International Conference on Learning Represent...

    2024

  12. [20]

    Au- tonomous evaluation of llms for truth maintenance and reasoning tasks

    Rushang Karia, Daniel Richard Bramblett, Daksh Dobhal, and Siddharth Srivastava. Au- tonomous evaluation of llms for truth maintenance and reasoning tasks. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  13. [21]

    Realtime qa: What’s the answer right now? Advances in neural information processing systems, 36:49025–49043, 2023

    Jungo Kasai, Keisuke Sakaguchi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, Kentaro Inui, et al. Realtime qa: What’s the answer right now? Advances in neural information processing systems, 36:49025–49043, 2023

    2023

  14. [22]

    Particle swarm optimization

    James Kennedy and Russell Eberhart. Particle swarm optimization. InProceedings of ICNN’95- international conference on neural networks, volume 4, pages 1942–1948. ieee, 1995

    1942

  15. [23]

    Openassistant conversations-democratizing large language model alignment.Advances in Neural Information Processing Systems, 36:47669–47681, 2023

    Andreas Köpf, Yannic Kilcher, Dimitri V on Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, et al. Openassistant conversations-democratizing large language model alignment.Advances in Neural Information Proc...

    2023

  16. [24]

    Eliciting language model behaviors with investigator agents.arXiv preprint arXiv:2502.01236, 2025

    Xiang Lisa Li, Neil Chowdhury, Daniel D Johnson, Tatsunori Hashimoto, Percy Liang, Sarah Schwettmann, and Jacob Steinhardt. Eliciting language model behaviors with investigator agents.arXiv preprint arXiv:2502.01236, 2025

    2025 arXiv

  17. [25]

    Autobencher: Towards declarative benchmark construction

    Xiang Lisa Li, Farzaan Kaiyom, Evan Zheran Liu, Yifan Mai, Percy Liang, and Tatsunori Hashimoto. Autobencher: Towards declarative benchmark construction. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  18. [26]

    Gen- dataagent: On-the-fly dataset augmentation with synthetic data

    Zhiteng Li, Lele Chen, Jerone Andrews, Yunhao Ba, Yulun Zhang, and Alice Xiang. Gen- dataagent: On-the-fly dataset augmentation with synthetic data. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  19. [27]

    Hemm: Holistic evaluation of multimodal foundation models

    Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Russ Salakhut- dinov, and Louis-Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks ...

    2024

  20. [28]

    Holistic evaluation of language models.Transactions on Machine Learning Research, 2022

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic evaluation of language models.Transactions on Machine Learning Research, 2022

    2022

  21. [29]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pages 3214–3252, 2022

    2022

  22. [30]

    Best practices and lessons learned on synthetic data

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data. InFirst Conference on Language Modeling, 2024. 11

    2024

  23. [31]

    Fictitious synthetic data can improve llm factuality via prerequisite learning.arXiv preprint arXiv:2410.19290, 2024

    Yujian Liu, Shiyu Chang, Tommi Jaakkola, and Yang Zhang. Fictitious synthetic data can improve llm factuality via prerequisite learning.arXiv preprint arXiv:2410.19290, 2024

    2024 arXiv

  24. [32]

    Adaptive labeling for efficient out-of-distribution model evaluation.Advances in Neural Information Processing Systems, 37:70981–71003, 2024

    Daksh Mittal, Yuanzhe Ma, Shalmali Joshi, and Hongseok Namkoong. Adaptive labeling for efficient out-of-distribution model evaluation.Advances in Neural Information Processing Systems, 37:70981–71003, 2024

    2024

  25. [33]

    Unearthing skill-level insights for understanding trade-offs of foundation models.arXiv preprint arXiv:2410.13826, 2024

    Mazda Moayeri, Vidhisha Balachandran, Varun Chandrasekaran, Safoora Yousefi, Thomas Fel, Soheil Feizi, Besmira Nushi, Neel Joshi, and Vibhav Vineet. Unearthing skill-level insights for understanding trade-offs of foundation models.arXiv preprint arXiv:2410.13826, 2024

    arXiv

  26. [34]

    Enhancing reason- ing capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    Terufumi Morishita, Gaku Morio, Atsuki Yamaguchi, and Yasuhiro Sogawa. Enhancing reason- ing capabilities of llms via principled synthetic logic corpus.Advances in Neural Information Processing Systems, 37:73572–73604, 2024

    2024

  27. [35]

    Caps: Collaborative and private synthetic data generation from distributed sources

    Sikha Pentyala, Mayana Pereira, and Martine De Cock. Caps: Collaborative and private synthetic data generation from distributed sources. InInternational Conference on Machine Learning, pages 40397–40413. PMLR, 2024

    2024

  28. [36]

    Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, et al. Humanity’s last exam.arXiv preprint arXiv:2501.14249, 2025

    2025 arXiv

  29. [37]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  30. [38]

    Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175, 2024

    Melanie Sclar, Jane Yu, Maryam Fazel-Zarandi, Yulia Tsvetkov, Yonatan Bisk, Yejin Choi, and Asli Celikyilmaz. Explore theory of mind: Program-guided adversarial data generation for theory of mind reasoning.arXiv preprint arXiv:2412.12175, 2024

    2024 arXiv

  31. [39]

    Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold.Advances in Neural Information Processing Systems, 37:43000–43031, 2024

    Amrith Setlur, Saurabh Garg, Xinyang Geng, Naman Garg, Virginia Smith, and Aviral Kumar. Rl on incorrect synthetic data scales the efficiency of llm math reasoning by eight-fold.Advances in Neural Information Processing Systems, 37:43000–43031, 2024

    2024

  32. [40]

    Is your benchmark truly adversarial? advscore: Evaluating human-grounded adversarialness.arXiv preprint arXiv:2406.16342, 2024

    Yoo Yeon Sung, Maharshi Gor, Eve Fleisig, Ishani Mondal, and Jordan Lee Boyd-Graber. Is your benchmark truly adversarial? advscore: Evaluating human-grounded adversarialness.arXiv preprint arXiv:2406.16342, 2024

    2024 arXiv

  33. [41]

    Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size.arXiv preprint arXiv:2408.00118, 2024

    2024 arXiv

  34. [42]

    Measuring general intelligence with generated games, 2025

    Vivek Verma, David Huang, William Chen, Dan Klein, and Nicholas Tomlin. Measuring general intelligence with generated games, 2025

    2025

  35. [43]

    Cuts: Customizable tabular synthetic data generation

    Mark Vero, Mislav Balunovic, and Martin Vechev. Cuts: Customizable tabular synthetic data generation. InInternational Conference on Machine Learning, pages 49408–49433. PMLR, 2024

    2024

  36. [44]

    The power of llm-generated synthetic data for stance detection in online political discussions.arXiv preprint arXiv:2406.12480, 2024

    Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, and Stefan Harmeling. The power of llm-generated synthetic data for stance detection in online political discussions.arXiv preprint arXiv:2406.12480, 2024

    2024 arXiv

  37. [45]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019

    2019

  38. [46]

    Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023

    Heng Wang, Shangbin Feng, Tianxing He, Zhaoxuan Tan, Xiaochuang Han, and Yulia Tsvetkov. Can language models solve graph problems in natural language?Advances in Neural Information Processing Systems, 36:30840–30861, 2023. 12

    2023

  39. [47]

    Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization

    Yidong Wang, Zhuohao Yu, Wenjin Yao, Zhengran Zeng, Linyi Yang, Cunxiang Wang, Hao Chen, Chaoya Jiang, Rui Xie, Jindong Wang, et al. Pandalm: An automatic evaluation benchmark for llm instruction tuning optimization. InThe Twelfth International Conference on Learning Represent...

    2024

  40. [48]

    Self-instruct: Aligning language models with self-generated in- structions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated in- structions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics...

    2023

  41. [49]

    Pre-training with synthetic data helps offline reinforcement learning

    Zecheng Wang, Che Wang, Zixuan Dong, and Keith W Ross. Pre-training with synthetic data helps offline reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  42. [50]

    Rocketeval: Efficient automated llm evaluation via grading checklist

    Tianjun Wei, Wei Wen, Ruizhi Qiao, Xing Sun, and Jianghong Ma. Rocketeval: Efficient automated llm evaluation via grading checklist. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  43. [51]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  44. [52]

    Differentially private synthetic data via foundation model apis 2: Text

    Chulin Xie, Zinan Lin, Arturs Backurs, Sivakanth Gopi, Da Yu, Huseyin A Inan, Harsha Nori, Haotian Jiang, Huishuai Zhang, Yin Tat Lee, et al. Differentially private synthetic data via foundation model apis 2: Text. InInternational Conference on Machine Learning, pages 54531–54...

    2024

  45. [53]

    Automating dataset updates towards reliable and timely evaluation of large language models

    Jiahao Ying, Yixin Cao, Yushi Bai, Qianru Sun, Bo Wang, Wei Tang, Zhaojun Ding, Yizhe Yang, Xuanjing Huang, and YAN Shuicheng. Automating dataset updates towards reliable and timely evaluation of large language models. InThe Thirty-eight Conference on Neural Information Proces...

    2024

  46. [54]

    xfinder: Large language models as automated evaluators for reliable evaluation

    Qingchen Yu, Zifan Zheng, Shichao Song, Feiyu Xiong, Bo Tang, Ding Chen, et al. xfinder: Large language models as automated evaluators for reliable evaluation. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  47. [55]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    2019

  48. [56]

    Evaltree: Profiling language model weaknesses via hierarchical capability trees.arXiv preprint arXiv:2503.08893, 2025

    Zhiyuan Zeng, Yizhong Wang, Hannaneh Hajishirzi, and Pang Wei Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees.arXiv preprint arXiv:2503.08893, 2025

    2025 arXiv

  49. [57]

    Bidirectional lms are better knowledge memorizers? a benchmark for real-world knowledge injection.arXiv preprint arXiv:2505.12306, 2025

    Yuwei Zhang, Wenhao Yu, Shangbin Feng, Yifan Zhu, Letian Peng, Jayanth Srinivasa, Gaowen Liu, and Jingbo Shang. Bidirectional lms are better knowledge memorizers? a benchmark for real-world knowledge injection.arXiv preprint arXiv:2505.12306, 2025

    2025 arXiv

  50. [58]

    Wildchat: 1m chatgpt interaction logs in the wild

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  51. [59]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  52. [60]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 13

    2023

  53. [61]

    Sotopia: Interactive evaluation for social intelligence in language agents

    Xuhui Zhou, Hao Zhu, Leena Mathur, Ruohong Zhang, Haofei Yu, Zhengyang Qi, Louis- Philippe Morency, Yonatan Bisk, Daniel Fried, Graham Neubig, et al. Sotopia: Interactive evaluation for social intelligence in language agents. InThe Twelfth International Conference on Learning ...

    2024

  54. [62]

    Dyval: Dynamic evaluation of large language models for reasoning tasks

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Dynamic evaluation of large language models for reasoning tasks. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  55. [63]

    Dynamic evaluation of large language models by meta probing agents

    Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. InProceedings of the 41st International Conference on Machine Learning, pages 62599–62617, 2024. 14 A Limitations DATASWARMSpresents an approach...

    2024

  56. [64]

    Top Secret

    **Convert to a percentage** * 0.2083 * 100% = 20.83% **Answer:** The cup stayed full approximately **20.83%** of the time.✗ 18 Example 4: math Question:For a long-distance swim contest, the total distance is 1000m. Mr. James swims at a speed of 30 m per hour, while Mr. Mark sw...

    1953

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.