Pith. sign in

REVIEW 4 major objections 4 minor 89 references

Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Closed-loop data refinement lifts fine-tuned LLM accuracy 7.15% on average

desk verdict Middo is a plausible integration of known data-optimization signals, but the 7.15% headline is the best tuned iteration and the 'constant dataset size' claim is contradicted by the appendix. read the letter →

arxiv 2508.21589 v5 pith:6C554KTE submitted 2025-08-29 cs.CL cs.AI

Zinan Tang , Xin Gao , Qizhi Pei , Zhuoshi Pan , Mengzhang Cai , Jiang Wu , Conghui He , Lijun Wu This is my paper
classification cs.CLcs.AI
keywords dataoptimizationsupervisedfine-tuningclosed-looplearningself-alignmentscoringselectionsynthesisdynamiccurriculumLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the data used for supervised fine-tuning can be improved dynamically by letting the model inspect its own training set and rewrite weak samples, rather than filtering once with fixed rules. The proposed Middo pipeline runs three diagnostic signals — loss patterns, embedding-cluster sparsity, and self-alignment scores — to flag overly hard, underrepresented, and low-quality samples, then simplifies, extends, or rewrites those samples while preserving their semantic intent. On eight benchmarks across two 7B-scale base models, the authors report an average accuracy improvement of 7.15% on one base model and 4.75% on another over three iterations, with dataset size staying close to the original. The practical interest is that better fine-tuned models can come from the same data budget, and the loop adapts automatically as the model improves. The load-bearing premise is that the model's own judgments of quality are reliable enough to drive rewriting.

What carries the argument

The mechanism is the self-referential closed loop: a diagnostic stage that reads three model-produced signals — loss pattern for complexity, embedding-cluster sparsity for diversity, and self-alignment scores for quality — and an optimization stage that rewrites the flagged samples using the model's own generation capacity. The selection thresholds are dynamic (τ = μ + mσ), so as the model improves, the amount and composition of selected data shifts automatically. The loop is what carries the argument: each retrained model generates the signals that select the next round of data.

What would settle it

Run Middo on a dataset where the correct answers are written in a style the base model rarely produces. If the model's self-alignment scores systematically downgrade those correct samples, the quality module is rewarding stylistic familiarity rather than quality; the framework's gains should disappear or reverse when the self-scores are replaced by human or external-judge scores on the same samples.

Watch

Extended reading notes

Core claim

The central claim is that supervised fine-tuning data should not be treated as a fixed resource: a closed loop in which the model scores its own training samples, selects the weak ones, rewrites them, and retrains can improve downstream accuracy while roughly preserving dataset size. The paper identifies three axes of weakness using signals already available from the model: high pre- and post-training loss marks overly complex samples, low neighborhood cosine similarity in the embedding space marks sparse or underrepresented regions, and low self-alignment scores on factuality, clarity, and completeness mark low-quality samples. Each flagged subset is transformed — complex items are decompos

Load-bearing premise

The quality module assumes that the fine-tuned model can score its own training examples reliably on factuality, clarity, and completeness, and that those self-scores are an objective measure of data quality; if the model rewards its own current preferences, the loop can amplify them rather than fix them.

Editorial extensions

If this is right

  • Fine-tuning gains can be obtained without increasing dataset size, so the method preserves the compute and memory footprint of SFT.
  • Applying the loop to lower-quality seed data yields the largest gains; higher-quality datasets reach their peak in fewer iterations.
  • The optimized models solve substantially more hard test questions on MATH and GPQA, not merely easier average tasks.
  • Each of the three modules earns its place: ablations show removing any one of loss patterns, diversity extension, or self-alignment scores lowers average performance across iterations.
  • The approach transfers across different 7B-scale base models, suggesting the loop is architecture-agnostic within that scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the self-alignment scores mainly reflect the model's current stylistic preferences, the quality module could reinforce those preferences instead of correcting factual errors; comparing self-scores against an external judge on a held-out quality set would isolate this effect.
  • The dynamic threshold rule effectively creates an automatic curriculum — difficult samples are simplified early, subtler refinements remain later — which could be tested directly by comparing performance with versus without threshold adaptation across iterations.
  • The diversity extension preferentially adds examples near cluster boundaries; a targeted test would feed the augmented samples to a downstream classifier to check whether they reduce errors on rare or out-of-distribution inputs.
  • The reported gains might be partly attributable to retraining multiple times on slightly altered data rather than to the specific rewrites; an ablation rerunning the base training for the same number of epochs without data changes would quantify this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Middo, an iterative closed-loop data optimization framework for supervised fine-tuning of LLMs. Middo uses three model-derived signals to identify suboptimal training samples: loss patterns (complexity), embedding cluster dynamics (diversity), and self-alignment scores (quality). Selected samples are then either simplified, augmented, or rewritten by a synthesis LLM, and the resulting dataset is used to fine-tune the model for the next iteration. Experiments on Alpaca, Alpaca-4o-mini, and WizardLM with LLaMA-3.1-8B and Mistral-7B-v0.3 report average accuracy gains of 7.15% and 4.75% over the initial datasets, alongside ablations, sensitivity analyses, and comparisons to several data selection and augmentation baselines. The paper claims that these gains are achieved 'while maintaining the original dataset scale'.

Significance. If the reported gains are robust, Middo would be a useful contribution to the growing literature on model-aware, iterative data curation for LLM fine-tuning. The framework is clearly presented, the ablation study is informative, and the public release of datasets, models, and code is a concrete strength. The main claims are, however, currently supported by an evaluation protocol that is selection-heavy: threshold multipliers, the neighbor count, and the number of iterations are tuned on the same benchmarks used for the headline results, and the main table reports the best-performing iteration per dataset/model. The 'maintaining original dataset scale' claim is also directly contradicted by the paper's own Appendix Table 6. The underlying idea is plausible and the results are suggestive, but additional experiments and honest reporting are needed before the headline claims can be accepted.

major comments (4)
  1. [Appendix B (Tables 5, 7; Figure 7) and Section 4.2 (Table 1)] Hyperparameter selection leaks evaluation signal. The threshold multiplier m is chosen from a sensitivity analysis on the same Alpaca benchmarks (Table 7: 'performance peaks when the selected data proportion falls within this empirically determined optimal range'), k is chosen from Table 5, and the iteration count is chosen from Figure 7. Table 1 then reports the best iteration per dataset/model, e.g., Alpaca-4o-mini/LLaMA peaks at iter2 while Wizard/LLaMA peaks at iter1. With no seeds or error bars, the 7.15% headline is a selected maximum, not an estimate under a fixed protocol. Please report all iterations under a fixed rule, add at least 3-5 seeds with variance, or use a held-out tuning split.
  2. [Abstract, Section 4.2, Appendix Table 6] The claim of 'maintaining the original dataset scale' is contradicted by Table 6: dataset sizes grow from 52,002 to 57,636 (Alpaca/LLaMA), 52,002 to 65,324 (Alpaca-4o-mini/LLaMA), and 70,000 to 89,178 (Wizard/Mistral). The diversity module adds examples rather than replacing them, so total size increases by roughly 10-25%. Either control for data size (e.g., compare against adding the same number of random/unaltered samples) or revise the claim in the abstract, introduction, and conclusion.
  3. [Section 4.2, Table 1] The assertion that Middo 'consistently enhances model performance across all benchmarks' is not supported by the table. Counterexamples include Alpaca/LLaMA MMLU dropping from 50.13 (iter1) to 41.82 (iter2), Alpaca-4o-mini/LLaMA GPQA decreasing monotonically 24.24→21.72→19.70, and Mistral/Alpaca-4o-mini average at iter3 (32.86) falling below the initial value (34.56). The claim should be softened to an average improvement with explicit per-benchmark caveats.
  4. [Section 3.4, Appendix D, Table 3] The quality module relies on the fine-tuned model's own self-alignment scores as ground truth for data quality. Since the ablation in Table 3 shows that removing this module is often the largest performance drop, the method's effectiveness depends critically on these scores being valid. The paper does not validate the self-scores against an external judge (e.g., GPT-4 or human ratings) on a subset. Please add such a validation experiment, or explicitly discuss and test the risk that self-scoring reinforces the model's existing preferences rather than improving data quality.
minor comments (4)
  1. [Appendix B.3, Table 6] The m values used in the main experiments (e.g., Alpaca initial row: m=1, m=-1, m=-1.5) do not appear in the sensitivity table (Table 7), which lists combinations such as m=0, -1, -1.5 and m=1, -2, -1.5. This makes it difficult to see how the stated 'targeting the 15% mark' rule maps to the actual selected data sizes.
  2. [Appendix B.2, Figure 7] Figure 7 shows average score versus iteration with no error bars or multiple runs. The statement that performance 'peaks at three iterations' is based on a single trajectory and should be described as such.
  3. [Table 1, Mistral/Alpaca-4o-mini iter1] The value 41.32 appears twice (for both HumanEval and Hellaswag), which is likely a typo or copy-paste error. Please verify the reported numbers.
  4. [Title and running header] The title contains a typo: 'Fine-T uning' should be 'Fine-Tuning' (also 'T uning' in the running header).

Circularity Check

3 steps flagged · score 5.0 of 10

Headline 7.15% gain is partly selected from on-benchmark tuning of thresholds and iteration count; the self-alignment 'quality' improvement is self-referential.

  1. self definitional [Section 3.4 (Quality Selection) and Section 5.3 (Data Analysis, Quality)]
    "Specifically, for each instruction-response pair (Xi, Yi) in D, the model generate scores Sπ ins.(Xi) for instruction and Sπ res.(Xi, Yi) for instruction-response pair based on three key metrics π from AlignBench [37]: Clarity, Completeness, and Factuality. The final quality score S(X i, Yi) is obtained by averaging these scores. These samples with scores below a similar dynamic threshold are identified as low-quality... As the self-alignment scores increase, it reflects that the refined data is not only more accurate but also more consistent with the model’s internal standards and expectation"

    The 'quality' axis is defined by the model's own self-alignment scores. Low-scoring samples are selected and rewritten, then the same model is fine-tuned on the revised data and asked to score the data again. The average self-alignment score increase in Figure 6 is therefore partly guaranteed by the optimization loop: the data are being trained and then judged against the very rubric that drove the selection/rewriting. Presenting this increase as evidence that 'low-quality samples' were transformed into 'high-quality learning material' is circular as a quality validation. The external benchmarks are independent, but the internal quality-improvement claim reduces to the optimization target.

  2. fitted input called prediction [Appendix B.3 (The Impact of Thresholds) and Section 4.2 (Main Results)]
    "To determine appropriate m values for each module, we conducted a sensitivity analysis, presented in Table 7. This table shows how different combinations of m for complexity, diversity, and quality impact the total percentage of unique data selected for refinement and the resulting average model performance on benchmarks... performance peaks when the selected data proportion falls within this empirically determined optimal range. For instance, the combination yielding 14.88% selected data achieved the best average score of 43.23... The actual data sizes selected in each iteration for the exper"

    The m threshold multipliers that control which samples Middo refines were chosen by maximizing average performance on the same Alpaca benchmark suite later used for the headline claim of a 7.15% average accuracy increase. The main Table 1 numbers are generated with parameters fit to the evaluation set itself, so the reported gain is a selected optimum rather than an independent prediction. The sensitivity analysis explicitly reports 'the best average score of 43.23' on the same benchmark family used in Table 1, making the headline partly a fitted result.

1 more flagged steps
  1. fitted input called prediction [Appendix B.2 (The Impact of Iterations) and Section 4.2 (Main Results)]
    "As shown in Figure 7, we tested the number of iterations on the Alpaca dataset and found that the model’s performance significantly declined after the third iteration. Therefore, we chose to optimize each dataset for three iterations."

    The main paper reports the 7.15% gain 'over three iterations' on the Alpaca dataset, but the number of iterations was selected by inspecting the performance curve on that same Alpaca evaluation and stopping at the observed peak (iteration 3). Thus the headline improvement is the maximum of the tested trajectory, not a pre-registered or held-out result. The same iteration count is then applied to other datasets and models without independent validation, so the reported magnitude is partly an artifact of choosing the best-looking iteration count on the benchmark being reported.

full rationale

This paper is an empirical method paper, not a mathematical derivation, so there is no equation-level derivation chain to be circular. The most serious circularity-like issue is the self-referential quality signal: the model scores its own data on clarity/completeness/factuality, low-scoring data are rewritten, and then the same model's rising self-alignment scores are presented as evidence of quality improvement; that internal claim reduces to the optimization target. Additionally, the headline 7.15% average gain is not a clean held-out prediction: the threshold multipliers m and the iteration count (three) were selected by sensitivity analysis on the same Alpaca benchmark suite used to report the main results. This is selection-on-the-evaluation-set, which makes the headline partly a fitted maximum rather than an independent finding. I did not count the Appendix Table 6 dataset-size growth (52k to 57.6k/65.3k/89.2k) as circularity; that is a factual consistency/correctness issue about the 'maintaining the original dataset scale' claim. No load-bearing self-citation chain is present, and the external benchmark evaluations give the method some independent content, so the overall circularity score is moderate rather than severe.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The method rests on several modeling assumptions about what loss trajectory, embedding sparsity, and self-scores mean, and on the choice of multiple thresholds tuned to the evaluation set. The central effectiveness claim depends on these assumptions holding simultaneously.

free parameters (6)
  • threshold multiplier m for complexity selection = e.g., 1.0 for Alpaca/LLaMA iter1, 0.5 for Alpaca/Mistral iter1
    Selected by sensitivity analysis (Table 7) to target 10-20% of data selected and to maximize average benchmark score; values differ per dataset and model (Table 6).
  • threshold multiplier m for diversity selection = -1.0 to -2.5 across datasets
    Chosen in the same sensitivity analysis as the complexity threshold; controls how sparse a region must be before samples are selected for augmentation.
  • threshold multiplier m for quality selection = -0.5 to -4 across datasets
    Chosen in the same sensitivity analysis; controls the self-alignment score cutoff for low-quality samples to be rewritten.
  • number of neighbors k = 2
    Selected from Table 5 because k=2 gave the best average score in the sensitivity analysis.
  • number of iteration rounds = 3
    Selected from Figure 7 because performance declined after the third iteration on Alpaca.
  • fine-tuning loss thresholds tau_pre and tau_post = derived from mean + m * std
    These thresholds are not fixed constants; they are re-derived each iteration from the model's loss distribution, with the multiplier m tuned as above.
assumptions (5)
  • domain assumption The fine-tuned model's self-alignment scores on factuality, clarity, and completeness are valid measures of training data quality.
    Section 3.4 and Appendix D define D_low using these scores. If the model cannot judge quality, the quality module selects the wrong samples.
  • domain assumption Loss before and after training above mean-plus-sigma thresholds identify samples that are too complex for the current model.
    Section 3.2 and Appendix B.3 define D_hard. This conflates model fit with intrinsic complexity.
  • domain assumption Low average cosine similarity to k-nearest neighbors in the last hidden layer indicates under-represented regions worth augmenting.
    Section 3.3 defines D_sparse. There is no external validation that these points correspond to missing concepts.
  • domain assumption LLM rewriting with GPT-4o-mini preserves the learning value of the original sample while making it easier or higher quality.
    Sections 3.2-3.4 and Appendix E prompts rely on this. If simplification changes content, gains may come from distribution shift rather than co-evolution.
  • domain assumption One epoch of fine-tuning per iteration is enough to update model capability without overfitting.
    Section 4.1 states this is done 'to mitigate the risk of overfitting'; it is not validated against multiple epochs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning." pith.science (2026). https://pith.science/paper/6C554KTE

@misc{pith2026250821589,
  author       = {Pith},
  title        = {Pith review of: Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-Tuning via Closed-Loop Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6C554KTE}},
  note         = {Machine review of arXiv:2508.21589}
}
read the original abstract

Supervised Fine-Tuning (SFT) Large Language Models (LLM) fundamentally rely on high-quality training data. While data selection and data synthesis are two common strategies to improve data quality, existing approaches often face limitations in static dataset curation that fail to adapt to evolving model capabilities. In this paper, we introduce Middo, a self-evolving Model-informed dynamic data optimization framework that uses model-aware data selection and context-preserving data refinement. Unlike conventional one-off filtering/synthesis methods, our framework establishes a closed-loop optimization system: (1) A self-referential diagnostic module proactively identifies suboptimal samples through tri-axial model signals - loss patterns (complexity), embedding cluster dynamics (diversity), and self-alignment scores (quality); (2) An adaptive optimization engine then transforms suboptimal samples into pedagogically valuable training points while preserving semantic integrity; (3) This optimization process continuously evolves with model capability through dynamic learning principles. Experiments on multiple benchmarks demonstrate that our Middo consistently enhances the quality of seed data and boosts LLM's performance with improving accuracy by 7.15% on average while maintaining the original dataset scale. This work establishes a new paradigm for sustainable LLM training through dynamic human-AI co-evolution of data and models. Our datasets, models, and code are publicly available at https://github.com/Word2VecT/Middo.

Figures

Figures reproduced from arXiv: 2508.21589 by the authors.

Figure 1
Figure 1. Comparison of different dataset and dif [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The Middo pipeline: a closed-loop, iterative dynamic optimization framework for LLM [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Performance comparison of Middo on the Alpaca dataset with varying refined data sizes. The x-axis represents the number and percentage of data selected for refinement, while the y-axis shows the average accuracy across three iterations. To ensure fairness, we guarantee that the data after refinement is the same. Quality. The self-alignment score trajectories across different iterations are presented in Fig￾ure 6. Th… view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: t-SNE visualization of the Alpaca dataset before and af￾ter applying Middo. The orig￾inal dataset is shown in light blue, while the augmented data is in dark blue. The dark blue points tend to occupy the sparsely populated regions of the light blue point distribution. …
Figure 7
Figure 7. Figure 7: Performance trends on the Alpaca dataset [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Self-alignment score evolution across iterations. The x-axis represents the self-alignment [PITH_FULL_IMAGE:figures/full_fig_p021_8.png]
Figure 9
Figure 9. Figure 9: Self-Alignment instruction score example. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png]
Figure 10
Figure 10. Figure 10: Self-Alignment response score example. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Complexity optimization example. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Quality optimization example. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Diversity extension example. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

89 extracted references · 53 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Program synthesis with large language models,

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models,

  3. [3]

    Synthetic and natural noise both break neural machine translation

    Yonatan Belinkov and Yonatan Bisk. Synthetic and natural noise both break neural machine translation. InIn- ternational Conference on Learning Representations, 2018. URLhttps://openreview.net/forum?id=BJ8vJebC-

    2018

  4. [4]

    Instruction mining: Instruction data selection for tuning large language models

    Yihan Cao, Yanbin Kang, Chi Wang, and Lichao Sun. Instruction mining: Instruction data selection for tuning large language models. InFirst Conference on Language Modeling, 2024. URL https://openreview. net/forum?id=wF6k0aWjAu

    2024

  5. [5]

    Alpagasus: Training a better alpaca with fewer data

    Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. Alpagasus: Training a better alpaca with fewer data. InThe 10 Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-T uning via Closed-Loop Learning Twelfth International Conference on Learning ...

    2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  7. [7]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URLhttps://arxiv.org/abs/2110.14168

    arXiv 2021

  8. [8]

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andre F. T. Martins, Fabrizio Esposito, Vera L´ucia Raposo, Sofia Morgado, and Michael Desa. Saullm-7b: A pioneering large language model for law, 2024. URLhttps://arxiv.org/abs/2403.03883

    arXiv 2024

Show all 89 references
  1. [9]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  2. [10]

    Auggpt: Leveraging chatgpt for text data augmentation.IEEE Transactions on Big Data, 2025

    Haixing Dai, Zhengliang Liu, Wenxiong Liao, Xiaoke Huang, Yihan Cao, Zihao Wu, Lin Zhao, Shaochen Xu, Fang Zeng, Wei Liu, et al. Auggpt: Leveraging chatgpt for text data augmentation.IEEE Transactions on Big Data, 2025

    2025

  3. [11]

    Mods: Model-oriented data selection for instruction tuning, 2023

    Qianlong Du, Chengqing Zong, and Jiajun Zhang. Mods: Model-oriented data selection for instruction tuning, 2023. URLhttps://arxiv.org/abs/2311.15653

    2023 arXiv

  4. [12]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  5. [13]

    Understanding back-translation at scale

    Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 489–500, 2018

    2018

  6. [14]

    A strategic coordination framework of small LMs matches large LMs in data synthesis

    Xin Gao, Qizhi Pei, Zinan Tang, Yu Li, Honglin Lin, Jiang Wu, Lijun Wu, and Conghui He. A strategic coordination framework of small LMs matches large LMs in data synthesis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar, editors,Proceedings of t...

    2025

  7. [15]

    Surveying the effects of quality, diversity, and complexity in synthetic data from large language models.arXiv preprint arXiv:2412.02980, 2024

    Alex Havrilla, Andrew Dai, Laura O’Mahony, Koen Oostermeijer, Vera Zisler, Alon Albalak, Fabrizio Milo, Sharath Chandra Raparthy, Kanishk Gandhi, Baber Abbasi, et al. Surveying the effects of quality, diversity, and complexity in synthetic data from large language models.arXiv...

    2024 arXiv

  8. [16]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations,

  9. [17]

    Measuring mathematical problem solving with the MATH dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. InThirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (...

    2021

  10. [18]

    URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

  11. [19]

    Boosting llm via learning from data iteratively and selectively, 2024

    Qi Jia, Siyu Ren, Ziheng Qin, Fuzhao Xue, Jinjie Ni, and Yang You. Boosting llm via learning from data iteratively and selectively, 2024. URLhttps://arxiv.org/abs/2412.17365

    2024 arXiv

  12. [20]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 11 Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-T...

    2024 arXiv

  13. [21]

    Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, low-quality model

    Jaehun Jung, Peter West, Liwei Jiang, Faeze Brahman, Ximing Lu, Jillian Fisher, Taylor Sorensen, and Yejin Choi. Impossible distillation for paraphrasing and summarization: How to make high-quality lemonade out of small, low-quality model. InProceedings of the 2024 Conference ...

    2024

  14. [22]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L´elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas...

    2023 arXiv

  15. [23]

    Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks

    Po-Nien Kung, Fan Yin, Di Wu, Kai-Wei Chang, and Nanyun Peng. Active instruction tuning: Improving cross-task generalization by training on prompt sensitive tasks. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1813–1829, 2023

    2023

  16. [24]

    Dataenvgym: Data generation agents in teacher environments with student feedback

    Zaid Khan, Elias Stengel-Eskin, Jaemin Cho, and Mohit Bansal. Dataenvgym: Data generation agents in teacher environments with student feedback. InThe Thirteenth International Conference on Learning Representa- tions, 2025. URLhttps://openreview.net/forum?id=00SnKBGTsz

    2025

  17. [25]

    The bigscience roots corpus: A 1.6 tb composite multilingual dataset.Advances in Neural Information Processing Systems, 35:31809–31826, 2022

    Hugo Laurenc ¸on, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo Gonz ´alez Ponferrada, Huu Nguyen, et al. The bigscience roots corpus: A 1.6 tb composite multilingual dataset.Advances in Neu...

    2022

  18. [26]

    Efficient memory management for large language model serving with pagedattention

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonza- lez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. InProceedings of the 29th symposium on operating systems principles, ...

    2023

  19. [27]

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models.Transactions on Machine Learning Research, 2025

    Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang, Shaohan Huang, Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, Yuxian Gu, Xin Cheng, Xun Wang, Si-Qing Chen, Li Dong, Wei Lu, Zhifang Sui, Benyou Wang, Wai Lam, and Furu Wei. Synthetic data (a...

    2025

  20. [28]

    Llm2llm: Boosting llms with novel iterative data enhancement

    Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anu- manchipalli, Michael Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting llms with novel iterative data enhancement. InFindings of the Association for Computational Linguistics ...

    2024

  21. [29]

    Superfiltering: Weak-to-strong data filtering for fast instruction-tuning

    Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfiltering: Weak-to-strong data filtering for fast instruction-tuning. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024

  22. [30]

    Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning

    Ming Li, Lichang Chen, Jiuhai Chen, Shwai He, Jiuxiang Gu, and Tianyi Zhou. Selective reflection-tuning: Student-selected data recycling for llm instruction-tuning. InFindings of the Association for Computational Linguistics ACL 2024, pages 16189–16211, 2024

    2024

  23. [31]

    Self-alignment with instruction backtranslation

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason E Weston, and Mike Lewis. Self-alignment with instruction backtranslation. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?id=1oijHJBRsT

    2024

  24. [32]

    From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning

    Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang, Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting llm performance with self-guided data selection for instruction tuning. InProceedings of the 2024 Conference of the North Americ...

    2024

  25. [33]

    Learning from committee: Reasoning distillation from a mixture of teachers with peer-review, 2024

    Zhuochun Li, Yuelyu Ji, Rui Meng, and Daqing He. Learning from committee: Reasoning distillation from a mixture of teachers with peer-review, 2024. URLhttps://arxiv.org/abs/2410.03663

    2024 arXiv

  26. [34]

    One-shot learning as instruction data prospector for large language models

    Yunshui Li, Binyuan Hui, Xiaobo Xia, Jiaxi Yang, Min Yang, Lei Zhang, Shuzheng Si, Ling-Hao Chen, Junhao Liu, Tongliang Liu, et al. One-shot learning as instruction data prospector for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Comp...

    2024

  27. [35]

    Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on synthetic data. InFirst Conference on Language Modeling, 2024. URLhttps://openreview.net/forum?id...

    2024

  28. [36]

    I-SHEEP: Self-alignment of LLM from scratch through an iterative self-enhancement paradigm

    Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Xeron Du, Jiawei Guo, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, Stephen Huang, and Jiajun Zhang. I-SHEEP: Self-alignment of LLM from scratch through an iterative self-enhancement paradigm. InScaling Self-Improving Founda...

    2025

  29. [37]

    Alignbench: Benchmarking chinese alignment of large language models

    Xiao Liu, Xuanyu Lei, Shengyuan Wang, Yue Huang, Andrew Feng, Bosi Wen, Jiale Cheng, Pei Ke, Yifan Xu, Weng Lam Tam, et al. Alignbench: Benchmarking chinese alignment of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Lingu...

    2024

  30. [38]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. InThe Twelfth International Conference on Learning Representations, 2024. URLhttps://openreview.net/forum?i...

    2024

  31. [39]

    Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement

    Maosongcao Maosongcao, Taolin Zhang, Mo Li, Chuyu Zhang, Yunxin Liu, Conghui He, Haodong Duan, Songyang Zhang, and Kai Chen. Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement. InProceedings of the 63rd Annual Meeting of the Association for Compu...

    2025

  32. [40]

    #instag: Instruction tagging for analyzing supervised fine-tuning of large language models

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models. InThe Twelfth International Conference on Learning Representations, 2024. URL http...

    2024

  33. [41]

    Llamaduo: Llmops pipeline for seamless migration from service llms to small-scale local llms, 2024

    Chansung Park, Juyong Jiang, Fan Wang, Sayak Paul, and Jing Tang. Llamaduo: Llmops pipeline for seamless migration from service llms to small-scale local llms, 2024. URLhttps://arxiv.org/abs/2408.13467

    2024 arXiv

  34. [42]

    Orca: Progressive learning from complex explanation traces of gpt-4, 2023

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4, 2023. URL https://arxiv. org/abs/2306.02707

    2023 arXiv

  35. [43]

    Instruction tuning with gpt-4

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4. arXiv preprint arXiv:2304.03277, 2023

    2023 arXiv

  36. [44]

    Datadreamer: A tool for synthetic data generation and reproducible llm workflows

    Ajay Patel, Colin Raffel, and Chris Callison-Burch. Datadreamer: A tool for synthetic data generation and reproducible llm workflows. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3781–3799, 2024

    2024

  37. [45]

    Gpqa: A graduate-level google-proof q&a benchmark

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling, 2024

    2024

  38. [46]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  39. [47]

    On efficient training of large-scale deep learning models.ACM Computing Surveys, 57(3):1–36, 2024

    Li Shen, Yan Sun, Zhiyuan Yu, Liang Ding, Xinmei Tian, and Dacheng Tao. On efficient training of large-scale deep learning models.ACM Computing Surveys, 57(3):1–36, 2024

    2024

  40. [48]

    GitHub - gururise/AlpacaDataCleaned: Alpaca dataset from Stanford, cleaned and curated — github.com.https://github.com/gururise/AlpacaDataCleaned, 2023

    Gene Ruebsamen. GitHub - gururise/AlpacaDataCleaned: Alpaca dataset from Stanford, cleaned and curated — github.com.https://github.com/gururise/AlpacaDataCleaned, 2023

    2023

  41. [49]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford alpaca, 2023

    2023

  42. [50]

    Cox, and Akash Srivastava

    Shivchander Sudalairaj, Abhishek Bhandwaldar, Aldo Pareja, Kai Xu, David D. Cox, and Akash Srivastava. Lab: Large-scale alignment for chatbots, 2024. URLhttps://arxiv.org/abs/2403.01081. 13 Middo: Model-Informed Dynamic Data Optimization for Enhanced LLM Fine-T uning via Close...

    2024 arXiv

  43. [51]

    Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models

    Ruida Wang, Wangchunshu Zhou, and Mrinmaya Sachan. Let’s synthesize step by step: Iterative dataset synthesis with large language models by extrapolating errors from small models. InFindings of the Association for Computational Linguistics: EMNLP 2023, pages 11817–11831, 2023

    2023

  44. [52]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    2008

  45. [53]

    Eda: Easy data augmentation techniques for boosting performance on text classifica- tion tasks

    Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classifica- tion tasks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces...

    2019

  46. [54]

    Self-instruct: Aligning language models with self-generated instructions

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (...

    2023

  47. [55]

    WizardLM: Empowering large pre-trained language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. WizardLM: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024. UR...

    2024

  48. [56]

    Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations

    John Wieting and Kevin Gimpel. Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, 2018

    2018

  49. [57]

    Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing

    Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing. InThe Thirteenth International Conference on Learning Representations, 2025. URL http...

    2025

  50. [58]

    Rethinking the instruction quality: Lift is what you need, 2023

    Yang Xu, Yongqiang Yao, Yufan Huang, Mengnan Qi, Maoquan Wang, Bin Gu, and Neel Sundaresan. Rethinking the instruction quality: Lift is what you need, 2023. URLhttps://arxiv.org/abs/2312.11508

    2023 arXiv

  51. [59]

    Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019

    2019

  52. [60]

    Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning

    Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. InProceedings of the 62nd Annual Meeting of the Association for Computat...

    2024

  53. [61]

    Long is more for alignment: a simple but tough-to-beat baseline for instruction fine-tuning

    Hao Zhao, Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. Long is more for alignment: a simple but tough-to-beat baseline for instruction fine-tuning. InProceedings of the 41st International Conference on Machine Learning, pages 60674–60703, 2024

    2024

  54. [62]

    Automatic instruction evolving for large language models

    Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, and Weizhu Chen. Automatic instruction evolving for large language models. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6998–7018, 2024

    2024

  55. [63]

    Llamafactory: Unified efficient fine-tuning of 100+ language models

    Yaowei Zheng, Richong Zhang, Junhao Zhang, YeYanhan YeYanhan, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages...

    2024

  56. [64]

    Tree-instruct: A preliminary study of the intrinsic relationship between complexity and alignment

    Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Minghao Li, Fei Huang, Nevin L Zhang, and Yongbin Li. Tree-instruct: A preliminary study of the intrinsic relationship between complexity and alignment. InProceedings of the 2024 Joint International Conference on Computational L...

    2024

  57. [65]

    Dataset quantization

    Daquan Zhou, Kai Wang, Jianyang Gu, Xiangyu Peng, Dongze Lian, Yifan Zhang, Yang You, and Jiashi Feng. Dataset quantization. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17205–17216, 2023

    2023

  58. [66]

    Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36: 55006–55021, 2023

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36: 55006–55021, 2023

    2023

  59. [67]

    Instruction-following evaluation for large language models, 2023

    Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models, 2023. URL https://arxiv.org/abs/ 2311.07911. 15 Middo: Model-Informed Dynamic Data Optimization for Enhanced L...

    2023 arXiv

  60. [68]

    Davir: Data selection via implicit reward for large language models

    Haotian Zhou, Tingkai Liu, Qianli Ma, Yufeng Zhang, Jianbo Yuan, Pengfei Liu, Yang You, and Hongxia Yang. Davir: Data selection via implicit reward for large language models. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    2025

  61. [70]

    Figure 9: Self-Alignment instruction score example

    The prompt is clear and understandable, but it could be more concise. Figure 9: Self-Alignment instruction score example. Self-Alignment Response Score Prompt Template We would like to request your feedback on the{dimension}of the prompt displayed below. Prompt: What are the t...

  62. [71]

    Simplify the language used in the prompt

  63. [72]

    Break down the instructions into simpler steps

  64. [73]

    Provide examples to clarify the concept of mode

  65. [74]

    Use smaller numbers or more relatable contexts

  66. [75]

    Step 2 #Plan#:

    Ask the question in a more direct manner. Step 2 #Plan#:

  67. [76]

    Change complex terminology like “mode” to a more descriptive explanation

  68. [77]

    Use a familiar context, like favorite fruits or animals

  69. [78]

    Clearly outline the steps needed to find the most common number, potentially using a smaller set

  70. [79]

    Offer a simple example to illustrate the mode calculation

  71. [80]

    Step 3 #Rewritten Prompt#: Find the most frequent number in this list: 3, 7, 2, 3, 5, 7

    Keep the structure straightforward and direct. Step 3 #Rewritten Prompt#: Find the most frequent number in this list: 3, 7, 2, 3, 5, 7. What number appears the most?’ Step 4 #Final Rewritten Prompt#: Find the number that appears the most in this list: 3, 7, 2, 3, 5, 7. What is...

  72. [81]

    Clarify the input requirements to make them more explicit

  73. [82]

    Use structured language with clear expectations for the output

  74. [83]

    Specify the context in which the job description will be used

  75. [84]

    Provide examples of desirable skills and experiences for the product manager role

  76. [85]

    Step 2 #Plan#:

    Ensure the language is professional and appropriate for a job description. Step 2 #Plan#:

  77. [86]

    Add a clear request for a job description that outlines key responsibilities and qualifications for a product manager

  78. [87]

    Specify the context, such as whether it’s for a tech company, a startup, or a different industry

  79. [88]

    Suggest including attributes and skills relevant to the product manager role, such as project management, communication, and analytics

  80. [89]

    Step 3 #Rewritten Prompt#:

    Format the request to clearly indicate the sections that should be included in the job description (e.g., summary, responsibilities, qualifications). Step 3 #Rewritten Prompt#: . . . Step 4 #Final Rewritten Prompt#: . . . Optimized High Quality Data Please create a detailed jo...

  81. [2021]

    URLhttps://arxiv.org/abs/2108.07732

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.