Pith. sign in

REVIEW 5 major objections 5 minor 117 references

HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

T0 review · 5 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Web text becomes 237k HSS instruction pairs; teacher-forced answers lift a fine-tuned 8B model past 14 rival datasets.

desk verdict Solid method for HSS data synthesis, but the human-preference SOTA is built on LLM judges and needs independent validation. read the letter →

arxiv 2607.27379 v1 pith:SEOM6O5I submitted 2026-07-29 cs.CL

Ru Peng , Tianyu Zhao , Xijun Gu , Zhiting Fan , Haokai Xu , Jinyang Zhang , Yawen Zeng , Yihong Zhuang
show 4 more authors
Kexin Yang Junyang Lin Dayiheng Liu Junbo Zhao
This is my paper · ORCID
classification cs.CL
keywords datasynthesisinstructiontuninghumanitiesandsocialsciencesteacher-forcedansweringbacktranslationQ&Aalignmentwebcorporafine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that high-quality, diverse instruction-tuning data for open-ended humanities and social-sciences tasks can be synthesized automatically from web corpora, rather than collected from humans or distilled from a frontier model. It builds a three-stage pipeline that filters and refines seed documents, backtranslates them into persona- and requirement-specific instructions, and anchors LLM answers to the seed documents during generation. If the central claim is right, the approach removes a major data bottleneck for HSS domains and transfers across model architectures and sizes, with a fine-tuned Qwen3-8B approaching the official instruction-tuned release.

What carries the argument

The load-bearing machinery is the three-stage pipeline. Stage one builds clean seed documents from web text through source sampling, heuristic filtering, LLM domain classification into 14 HSS fields, 12-rubric quality rating (readability, applicability, human-touch), LLM text refinement, and a refinement judge. Stage two is multi-attribute instruction backtranslation: the LLM writes instructions that specify domain, genre, length, core content, structure, narrative voice, and a persona ('what' and 'who'), then a Q&A alignment check confirms the reverse instruction can faithfully reproduce the seed. Stage three is TeachForceA, which feeds the seed document to the answering model as a semantic

What would settle it

Take a random sample of 1,000 HSS-Synth instruction–answer pairs and have independent human experts (not the paper's raters) rate them against the same 12 rubrics and a Q&A faithfulness check; or re-run the whole pipeline using a different model family (e.g., a non-Qwen judge) for quality rating and Q&A alignment and then compare downstream fine-tuning performance. If the quality advantage shrinks or vanishes, or if the human experts find systematic misalignment, the pipeline's internal LLM-based validation is circular and the headline gains are artifacts of self-preference.

Watch

Extended reading notes

Core claim

The central discovery is teacher-forced answering (TeachForceA): when generating an answer, the synthesis model is given the seed document alongside the instruction as a semantic anchor. This reduces hallucination, preserves the original tone and style, and improves completeness and readability, producing answers that score higher on the paper's 12-rubric quality scale and lead to better fine-tuned models than either the raw seed document or a purely reverse-generated answer. Combined with multi-attribute instruction backtranslation (which specifies both the task requirements and the persona) and a Q&A alignment check, the pipeline yields 237k instruction–answer pairs from Qwen3-30B-A3B that

Load-bearing premise

The pipeline's quality controls—domain classification, rubric rating, text refinement, refinement judging, Q&A alignment, and teacher-forced answering—are all performed by the same LLM family (Qwen3-30B-A3B) that generates the data, and the rubric's validity rests on only 400+ seed texts rated by human annotators plus a single GPT-4.1 comparison; if these judgments are biased or self-referential, the dataset may be optimized to the synthesizer's taste rather than to true HSS

Editorial extensions

If this is right

  • Fine-tuning a base model on HSS-Synth outperforms 14 leading instruction-data baselines across 16 benchmarks, coming closest to the official instruction-tuned Qwen3-8B.
  • Teacher-forced answers are the best answer type; raw seed documents are unsuitable as answers because their embedding distribution diverges sharply from reverse-generated answers.
  • The gains transfer across model families and sizes: Qwen2.5-14B and Llama3.1-8B both improve markedly on HSS-Synth data.
  • There is no detected 'performance seesaw': human-preference gains do not come at the cost of knowledge benchmarks, and some knowledge capabilities improve slightly.
  • Different capabilities saturate at different data scales: knowledge-based skills converge around 10% of the 237k samples, human-preference skills around 20%, and instruction following around 33%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same teacher-forced anchoring idea could apply to other open-ended synthetic-data pipelines (e.g., medicine, law, advice-giving), where grounding answers in a source document may reduce hallucination and style drift without human annotation.
  • Because the same Qwen model family both generates and judges the data, the reported 'quality' may partly reflect the synthesis model's own preferences; a direct test would be to re-run the pipeline with an independent judge model or human raters and see whether downstream gains survive.
  • The observed saturation pattern suggests that SFT injects little new factual knowledge; this implies that for knowledge-heavy HSS applications, larger gains would come from synthesizing pre-training data rather than instruction data.
  • The paper's reuse metrics (n-gram overlap, LCS, copy ratio) offer a cheap, general test for grounding in any synthetic QA pipeline; a low copy ratio with high quality could indicate genuine abstraction rather than copying.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces HSS-Synth, a three-stage pipeline for synthesizing instruction-tuning data in 14 humanities and social science domains. Stage 1 constructs seed documents from SlimPajama via source sampling, heuristic filtering, LLM domain classification, a 12-rubric quality rating, and LLM text refinement with a refinement judge. Stage 2 performs multi-attribute instruction backtranslation (requirements plus persona) with a Q&A alignment check. Stage 3 proposes teacher-forced answering (TeachForceA), where the seed document is fed alongside the instruction as a semantic anchor. The authors fine-tune Qwen3-8B-Base on the resulting ~230k samples and report improvements over 14 baselines across 16 benchmarks, claiming a new SOTA that approaches the official Qwen3-8B while avoiding a 'performance seesaw' between human preference and knowledge capabilities. They also report ablations, cross-model transfer to Llama3.1-8B and Qwen2.5-14B, scaling analyses, and a rubric validation study.

Significance. If the central claims hold, the work is a substantial step for open-ended, subject-centric data synthesis: it defines a principled HSS taxonomy, offers a reproducible pipeline, ships public code, and provides broad benchmark coverage with a decontamination check. The cross-architecture transfer and component ablations are also useful. However, the paper's signature claims are currently supported by evidence that is partly circular and partly indirect. The human-preference benchmarks are judged by Claude LLMs rather than humans; the quality-control stack is run entirely by the same model family (Qwen3-30B-A3B) that generates the data; rubric validation is limited to 400+ seed texts, not final model outputs; and no significance tests, multiple seeds, or length-controlled evaluations are reported. These gaps directly affect the headline 'SOTA' and 'no performance seesaw' claims, so the manuscript requires substantive additional validation before the claims can be accepted.

major comments (5)
  1. [§4.1 / Table 1] The 'human preference' category is not measured with humans. WritingBench, CreativeWriting-v3, Judgemark, BuzzBench, and EQ-Bench3 are all scored by Claude-3.5/3.7-Sonnet (the paper states this in §4.1). The claim of 'improving human preference' is therefore a claim about LLM-judge preference. No human preference study on the final model outputs is reported. Please add (a) a human evaluation of the fine-tuned models' outputs, or (b) at minimum a length-controlled LLM-judge evaluation (e.g., following Dubois et al., 2024, already cited), and (c) variance or significance measures across multiple seeds, since Table 1 reports point estimates with no error bars.
  2. [§3.1–§3.3 / §4.6] The pipeline uses Qwen3-30B-A3B for domain classification, quality rating, text refinement, refinement judging, Q&A alignment checking, and teacher-forced answering. The rubric validation in §4.6 is only on 400+ seed documents with human raters and a GPT-4.1 comparison; it does not validate the quality of the final instruction–answer pairs. This creates a circularity risk: the dataset may be optimized toward what the synthesis model (and its relatives) reward, rather than toward genuine HSS quality. Please validate the final dataset and the trained models with independent judges, ideally including human ratings of a sample of final (instruction, answer) pairs.
  3. [Abstract / §4.1 / Appendix C] The abstract claims 237k samples, §4.1 says 230k instruction–answer pairs were used for fine-tuning, and Appendix C reports 237,340 total samples. This discrepancy is not explained. If 230k is the training-set size, the abstract and appendix should be reconciled; if the dataset is larger than the training split, the filtering/selection procedure should be described. As written, the reader cannot determine what was actually trained on.
  4. [Appendix E / Table 10] The case study contradicts the instruction-fidelity claim. The reverse instruction explicitly requires 'approximately 413 tokens (around 2130 characters)', but the reverse answer is 433 tokens and the teacher-forced answer is 644 tokens, with the latter praised as satisfying the length constraint. This suggests the LLM judge used for rubric scores rewards length/detail over instruction fidelity, and it directly undermines the claimed advantage of TeachForceA. Please provide a length-controlled evaluation of answer quality and report the distribution of output lengths relative to instruction-specified lengths.
  5. [§4.2 / Figure 2] The 'no performance seesaw' and 'no cross-skill trade-offs' conclusions are based on comparing averages over heterogeneous benchmarks with different metrics and no confidence intervals. The scaling curves in Figure 2 are also point estimates. Without error bars, significance tests, or multiple seeds, the absence of a trade-off is not established. Please report per-seed variation or bootstrap confidence intervals for the key comparisons in Table 1 and Figure 2.
minor comments (5)
  1. [Table 3] Typo: 'Evlo-instruct' should be 'Evol-Instruct'. Also, Table 1 contains a run-together entry '51.2937.9053.36' in the SynthQuestions row; please fix the formatting.
  2. [Figure 5 / Figure 6] Captions: Figure 5's caption says 'model trained on seed documents' and Figure 6 says 'model trained on reverse answers', but the final sentence of Figure 6's caption reads 'The model trained on seed answers clearly finds its answer pattern easy to learn', which appears to be a copy-paste error (should be 'reverse answers').
  3. [§3.1] Minor nomenclature: 'Slimpajama' should be 'SlimPajama'. In Table 1, baseline rows list 'Qwen-30B-A3B' for WRAP and LongForm while the text uses 'Qwen3-30B-A3B'; please harmonize.
  4. [Abstract / §3.3] The abstract phrase 'teacher-forced Answering that fed seed documents during response' is grammatically unclear; suggest 'during response generation' or similar. Also, the term 'Teacher-Forced Answering' is introduced in §1, §3.3, and again in Table 1; please define it once in a glossary-like manner.
  5. [References] The Cosmopedia-v2 reference lists only 'Loubna Ben, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, Leandro von Werra' and the year; if this is the Cosmopedia technical report, please give the full author list or a citable handle. Some other entries (e.g., 'Tombench') also mix arXiv identifiers with incomplete author lists.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central SOTA claim is measured on external benchmarks; the same-model quality controls are a limitation, not a definitional reduction.

full rationale

The paper's central claim is that Qwen3-8B-Base fine-tuned on HSS-Synth outperforms 14 baselines on 16 benchmarks. This is an empirical result measured on external benchmarks (WritingBench, CreativeWriting-v3, Judgemark, MMLU, HellaSwag, etc.), not a quantity derived from the pipeline's own filters. The pipeline does use Qwen3-30B-A3B for domain classification, quality rating, text refinement judging, Q&A alignment checks, and teacher-forced answering, and the paper's own Limitations section concedes: 'HSS-Synth’s three-stage pipeline depends on the synthesis model and may introduce LLM bias, a common limitation of current data synthesis methods.' That is a recognized validity limitation, but it is not a circular derivation: none of the benchmark scores is defined in terms of these ratings, and the rubric validation in §4.6 includes independent human raters on 400+ seed texts ('Twenty uninvolved raters independently scored each item on a five-point scale... The synthesis model matched majority-vote human labels in 86% of cases'). The 'human preference' benchmarks are scored by Claude LLM judges rather than humans, and the comparison of answer types in §4.5 partly uses the same rubric family; these are measurement-validity caveats, not cases where an output equals an input by construction. No self-citation carries the argument, and no fitted parameter is relabeled as a prediction. The central comparison against 14 external baselines is self-contained and falsifiable, so the honest finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on hand-set quality thresholds and on the assumption that LLM-based judgments are reliable for open-ended HSS content. The paper partially validates the rubric with human raters, but the full pipeline is self-referential: the synthesis model also acts as judge. No new physical or mathematical entities are introduced.

free parameters (3)
  • Quality rubric thresholds = readability≥5, applicability≥4, human-touch≥3, total score≥45
    Hand-set in §3.1/Appendix A to balance seed-document quality against volume; directly controls which web documents become training seeds.
  • Source sampling rate for C4/CommonCrawl = 10%
    Appendix A: random 10% sample from crawler-based sources; affects domain composition and content diversity.
  • Rubric tier weights = readability weight 0.5, applicability 1.0, human-touch 1.5
    Appendix A: chosen to define the 0–60 rubric score used in quality bands and seed selection.
assumptions (5)
  • domain assumption The 14-field QS-derived taxonomy is an adequate decomposition of the humanities and social sciences.
    Adopted in §1 and §3.1; the Limitations section itself notes anthropology and religious studies are missing, so the taxonomy is a modeling choice, not a fixed ground truth.
  • domain assumption Qwen3-30B-A3B can reliably classify, rate, refine, judge, and align HSS documents.
    Used throughout §3.1–§3.3; validation in §4.6 covers only 400+ seed texts and one proprietary model comparison, not the full 230k pipeline.
  • domain assumption The 16 chosen benchmarks are valid measures of the target HSS and general LLM capabilities.
    Section 4.1: several 'human-preference' benchmarks are scored by Claude models rather than human raters, so the labels are only as valid as the judge models.
  • domain assumption A 13-gram decontamination check is sufficient to rule out train/test overlap.
    Section 4.1 asserts zero overlap between synthesized data and benchmark test sets using 13-gram analysis; longer or paraphrased overlap is not examined.
  • domain assumption Single-run fine-tuning with fixed hyperparameters is a fair comparison across all baselines.
    Section 4.1/Appendix D uses one training run per method with shared hyperparameters; no multiple seeds or variance estimates are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs." pith.science (2026). https://pith.science/paper/SEOM6O5I

@misc{pith2026260727379,
  author       = {Pith},
  title        = {Pith review of: HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SEOM6O5I}},
  note         = {Machine review of arXiv:2607.27379}
}
read the original abstract

High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.

Figures

Figures reproduced from arXiv: 2607.27379 by the authors.

Figure 1
Figure 1. Overview of the HSS-Synth pipeline: (1) Seed document construction: sample, filter, classify, rate, and refine web texts under judge into clean HSS seeds; (2) Question-Answer Pair Refinement—backtranslate instructions from seed with multiple attributes, specify the “what” and “who,” and enforce Q&A alignment checks to yield diverse, faithful Q&A pairs; (3) Teacher-forced Answering—anchor answers on seeds to better r… view at source ↗
Figure 2
Figure 2. Impact of HSS-Synth data scale on 8 LLM ca [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. t-SNE visualization (left) and Vendi scores [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: t-SNE plot of sentence embeddings for reverse [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Inference token likelihood heatmaps for models trained on [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: Inference token likelihood heatmaps for models trained on [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Prompt template for domain classification to classify documents into HSS domains. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png]
Figure 8
Figure 8. Figure 8: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part I). [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part II). [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part III). [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Prompt template for text refinement to polish seed documents. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Prompt template for a refinement judge that determines whether text refinement is successful. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png]
Figure 13
Figure 13. Figure 13: Prompt template for multi-attribute instruction backtranslation, which backtranslates instructions from [PITH_FULL_IMAGE:figures/full_fig_p025_13.png]
Figure 14
Figure 14. Figure 14: Prompt template for question-answer alignment check used to verify that the reverse instructions [PITH_FULL_IMAGE:figures/full_fig_p026_14.png]
Figure 15
Figure 15. Figure 15: Prompt template for teacher-forced answering, which produces answers anchored to seed documents that [PITH_FULL_IMAGE:figures/full_fig_p027_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

117 extracted references · 4 canonical work pages

  1. [1]

    Transactions on Machine Learning Research , issn=

    Emergent Abilities of Large Language Models , author=. Transactions on Machine Learning Research , issn=

  2. [2]

    The Thirteenth International Conference on Learning Representations , year=

    DataMan: Data Manager for Pre-training Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Scaling data-constrained language models , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    First Conference on Language Modeling , year=

    Best Practices and Lessons Learned on Synthetic Data , author=. First Conference on Language Modeling , year=

  5. [5]

    Awesome-LLM-in-Social-Science , howpublished =

  6. [6]

    Proceedings of the National Academy of Sciences , volume =

    Fabrizio Gilardi and Meysam Alizadeh and Maël Kubli , title =. Proceedings of the National Academy of Sciences , volume =. 2023 , doi =

    2023

  7. [7]

    and Khashabi, Daniel and Hajishirzi, Hannaneh

    Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh. Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.754

    doi:10.18653/v1/2023.acl-long.754 2023

  8. [8]

    Can Xu and Qingfeng Sun and Kai Zheng and Xiubo Geng and Pu Zhao and Jiazhan Feng and Chongyang Tao and Qingwei Lin and Daxin Jiang , booktitle=. Wizard

Show all 117 references
  1. [9]

    arXiv preprint arXiv:2412.08905 , year=

    Phi-4 technical report , author=. arXiv preprint arXiv:2412.08905 , year=

    arXiv

  2. [10]

    Behavior research methods , volume=

    MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment , author=. Behavior research methods , volume=. 2010 , publisher=

    2010

  3. [11]

    Transactions on machine learning research , year=

    The Vendi Score: A Diversity Evaluation Metric for Machine Learning , author=. Transactions on machine learning research , year=

  4. [12]

    Biochemia medica , volume=

    Interrater reliability: the kappa statistic , author=. Biochemia medica , volume=. 2012 , publisher=

    2012

  5. [13]

    The Twelfth International Conference on Learning Representations , year=

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models , author=. The Twelfth International Conference on Learning Representations , year=

  6. [14]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    arXiv

  7. [15]

    arXiv preprint arXiv:2305.17126 , year=

    Large language models as tool makers , author=. arXiv preprint arXiv:2305.17126 , year=

    arXiv

  8. [16]

    Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=

    Tabula: Harnessing language models for tabular data synthesis , author=. Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=. 2025 , organization=

    2025

  9. [17]

    arXiv preprint arXiv:2401.17268 , year=

    Weaver: Foundation models for creative writing , author=. arXiv preprint arXiv:2401.17268 , year=

    arXiv

  10. [18]

    arXiv preprint arXiv:2504.14375 , year=

    Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Prompts , author=. arXiv preprint arXiv:2504.14375 , year=

    arXiv

  11. [19]

    Advances in Neural Information Processing Systems , year=

    MAmmoTH2: Scaling Instructions from the Web , author=. Advances in Neural Information Processing Systems , year=

  12. [20]

    ACM Transactions on Intelligent Systems and Technology , volume=

    A comprehensive overview of large language models , author=. ACM Transactions on Intelligent Systems and Technology , volume=

  13. [21]

    arXiv preprint arXiv:2210.11610 , year=

    Large language models can self-improve , author=. arXiv preprint arXiv:2210.11610 , year=

    arXiv

  14. [22]

    arXiv preprint arXiv:2504.15573 , year=

    Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction , author=. arXiv preprint arXiv:2504.15573 , year=

    arXiv

  15. [23]

    Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan , title =

  16. [24]

    arXiv preprint arXiv:2112.11446 , year=

    Scaling language models: Methods, analysis & insights from training gopher , author=. arXiv preprint arXiv:2112.11446 , year=

    arXiv

  17. [25]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  18. [26]

    zip: Compressing text classification models , author=

    Fasttext. zip: Compressing text classification models , author=. arXiv preprint arXiv:1612.03651 , year=

    arXiv

  19. [27]

    Advances in Neural Information Processing Systems , volume=

    The fineweb datasets: Decanting the web for the finest text data at scale , author=. Advances in Neural Information Processing Systems , volume=

  20. [28]

    Proceedings

    On the resemblance and containment of documents , author=. Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171) , pages=. 1997 , organization=

    1997

  21. [29]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Efficient Attentions for Long Document Summarization , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  22. [30]

    D o G -Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping

    Chen, Yongrui and Jiang, Haiyun and Huang, Xinting and Shi, Shuming and Qi, Guilin. D o G -Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computat...

    2024 doi

  23. [31]

    arXiv preprint arXiv:2504.12563 , year=

    MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation , author=. arXiv preprint arXiv:2504.12563 , year=

  24. [32]

    arXiv preprint arXiv:2304.08460 , year=

    Longform: Effective instruction tuning with reverse instructions , author=. arXiv preprint arXiv:2304.08460 , year=

    arXiv

  25. [33]

    The Twelfth International Conference on Learning Representations , year=

    Self-Alignment with Instruction Backtranslation , author=. The Twelfth International Conference on Learning Representations , year=

  26. [34]

    arXiv preprint arXiv:2406.20094 , year=

    Scaling synthetic data creation with 1,000,000,000 personas , author=. arXiv preprint arXiv:2406.20094 , year=

    arXiv

  27. [35]

    arXiv preprint arXiv:2402.13064 , year=

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models , author=. arXiv preprint arXiv:2402.13064 , year=

    arXiv

  28. [36]

    arXiv preprint arXiv:2402.18334 , year=

    Learning to generate instruction tuning datasets for zero-shot task adaptation , author=. arXiv preprint arXiv:2402.18334 , year=

    arXiv

  29. [37]

    Advances in neural information processing systems , volume=

    Professor forcing: A new algorithm for training recurrent networks , author=. Advances in neural information processing systems , volume=

  30. [38]

    WildChat: 1M Chat

    Wenting Zhao and Xiang Ren and Jack Hessel and Claire Cardie and Yejin Choi and Yuntian Deng , booktitle=. WildChat: 1M Chat. 2024 , url=

    2024

  31. [39]

    2023 , publisher =

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants , author =. 2023 , publisher =

    2023

  32. [40]

    arXiv preprint arXiv:2502.17328 , year=

    Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization , author=. arXiv preprint arXiv:2502.17328 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2404.04475 , year=

    Length-controlled alpacaeval: A simple way to debias automatic evaluators , author=. arXiv preprint arXiv:2404.04475 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2507.05750 , year=

    DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities , author=. arXiv preprint arXiv:2507.05750 , year=

    arXiv

  35. [43]

    Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

    Maini, Pratyush and Seto, Skyler and Bai, Richard and Grangier, David and Zhang, Yizhe and Jaitly, Navdeep. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V...

    2024 doi

  36. [44]

    arXiv preprint arXiv:2406.08464 , year=

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing , author=. arXiv preprint arXiv:2406.08464 , year=

    arXiv

  37. [45]

    arXiv preprint arXiv:2412.02595 , year=

    Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset , author=. arXiv preprint arXiv:2412.02595 , year=

    arXiv

  38. [46]

    arXiv preprint arXiv:2502.04235 , year=

    Reformulation for Pretraining Data Augmentation , author=. arXiv preprint arXiv:2502.04235 , year=

    arXiv

  39. [47]

    arXiv preprint arXiv:2308.03281 , year=

    Towards general text embeddings with multi-stage contrastive learning , author=. arXiv preprint arXiv:2308.03281 , year=

    arXiv

  40. [48]

    Ben, Loubna and Lozhkov, Anton and Penedo, Guilherme and Wolf, Thomas and von Werra, Leandro , title =

  41. [49]

    arXiv preprint arXiv:2506.03968 , year=

    From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding , author=. arXiv preprint arXiv:2506.03968 , year=

    arXiv

  42. [50]

    arXiv preprint arXiv:2408.07055 , year=

    Longwriter: Unleashing 10,000+ word generation from long context llms , author=. arXiv preprint arXiv:2408.07055 , year=

    arXiv

  43. [51]

    arXiv preprint arXiv:2308.06259 , year=

    Self-alignment with instruction backtranslation , author=. arXiv preprint arXiv:2308.06259 , year=

    arXiv

  44. [52]

    arXiv preprint arXiv:2501.12273 , year=

    Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement , author=. arXiv preprint arXiv:2501.12273 , year=

    arXiv

  45. [53]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  46. [54]

    Qwen2.5: A Party of Foundation Models , url =

  47. [55]

    arXiv preprint arXiv:2312.06281 , year=

    Eq-bench: An emotional intelligence benchmark for large language models , author=. arXiv preprint arXiv:2312.06281 , year=

    arXiv

  48. [56]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    2009 arXiv

  49. [57]

    arXiv preprint arXiv:1604.01696 , year=

    A corpus and evaluation framework for deeper understanding of commonsense stories , author=. arXiv preprint arXiv:1604.01696 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    arXiv

  51. [59]

    arXiv preprint arXiv:2307.08689 , year=

    Collie: Systematic construction of constrained text generation tasks , author=. arXiv preprint arXiv:2307.08689 , year=

    arXiv

  52. [60]

    Transactions of the Association for Computational Linguistics , volume=

    Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  53. [61]

    Paech , title =

    Samuel J. Paech , title =. 2025 , howpublished =

    2025

  54. [62]

    arXiv preprint arXiv:2503.05244 , year=

    Writingbench: A comprehensive benchmark for generative writing , author=. arXiv preprint arXiv:2503.05244 , year=

  55. [63]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    1905 arXiv

  56. [64]

    arXiv preprint arXiv:2201.03533 , year=

    Scrolls: Standardized comparison over long language sequences , author=. arXiv preprint arXiv:2201.03533 , year=

    arXiv

  57. [65]

    arXiv preprint arXiv:1904.09728 , year=

    Socialiqa: Commonsense reasoning about social interactions , author=. arXiv preprint arXiv:1904.09728 , year=

    1904 arXiv

  58. [66]

    arXiv preprint arXiv:1808.08745 , year=

    Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization , author=. arXiv preprint arXiv:1808.08745 , year=

    arXiv

  59. [67]

    Transactions of the Association for Computational Linguistics , volume=

    The narrativeqa reading comprehension challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2018 , publisher=

    2018

  60. [68]

    arXiv preprint arXiv:2310.00746 , year=

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models , author=. arXiv preprint arXiv:2310.00746 , year=

    arXiv

  61. [69]

    arXiv preprint arXiv:2402.15052 , year=

    Tombench: Benchmarking theory of mind in large language models , author=. arXiv preprint arXiv:2402.15052 , year=

    arXiv

  62. [70]

    arXiv preprint arXiv:2305.07759 , year=

    Tinystories: How small can language models be and still speak coherent english? , author=. arXiv preprint arXiv:2305.07759 , year=

    arXiv

  63. [71]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 technical report , author=. arXiv preprint arXiv:2407.10671 , year=

    arXiv

  64. [72]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    arXiv

  65. [73]

    5-math technical report: Toward mathematical expert model via self-improvement , author=

    Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement , author=. arXiv preprint arXiv:2409.12122 , year=

    arXiv

  66. [74]

    arXiv preprint arXiv:2306.11644 , year=

    Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

    arXiv

  67. [75]

    arXiv preprint arXiv:2309.05463 , year=

    Textbooks are all you need ii: phi-1.5 technical report , author=. arXiv preprint arXiv:2309.05463 , year=

    arXiv

  68. [76]

    The Claude 3 Model Family: Opus, Sonnet, Haiku , url =

    Anthropic , year =. The Claude 3 Model Family: Opus, Sonnet, Haiku , url =

  69. [77]

    Alpacaeval: An automatic evaluator of instruction-following models , author=

  70. [78]

    From live data to high-quality benchmarks: The arena-hard pipeline , author=

  71. [79]

    Gonzalez and Ion Stoica , booktitle=

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , booktitle=. Judging. 2023 , url=

    2023

  72. [81]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    arXiv

  73. [82]

    arXiv preprint arXiv:2303.08774 , year=

    Gpt-4 technical report , author=. arXiv preprint arXiv:2303.08774 , year=

    arXiv

  74. [83]

    L lama F actory: Unified Efficient Fine-Tuning of 100+ Language Models

    Zheng, Yaowei and Zhang, Richong and Zhang, Junhao and Ye, Yanhan and Luo, Zheyan. L lama F actory: Unified Efficient Fine-Tuning of 100+ Language Models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)....

    2024 doi

  75. [84]

    Advances in neural information processing systems , volume=

    Redpajama: an open dataset for training large language models , author=. Advances in neural information processing systems , volume=

  76. [85]

    Hashimoto , title =

    Rohan Taori and Ishaan Gulrajani and Tianyi Zhang and Yann Dubois and Xuechen Li and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto , title =. GitHub repository , howpublished =. 2023 , publisher =

    2023

  77. [86]

    arXiv preprint arXiv:2305.14233 , year=

    Enhancing Chat Language Models by Scaling High-quality Instructional Conversations , author=. arXiv preprint arXiv:2305.14233 , year=

    arXiv

  78. [87]

    2024 , eprint=

    Better Alignment with Instruction Back-and-Forth Translation , author=. 2024 , eprint=

    2024

  79. [88]

    arXiv preprint arXiv:2506.05176 , year=

    Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models , author=. arXiv preprint arXiv:2506.05176 , year=

    arXiv

  80. [89]

    Applied Sciences , volume=

    What disease does this patient have? a large-scale open domain question answering dataset from medical exams , author=. Applied Sciences , volume=. 2021 , publisher=

    2021

  81. [90]

    The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

    FinBen: An Holistic Financial Benchmark for Large Language Models , author=. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track , year=

  82. [91]

    2018 , eprint=

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. 2018 , eprint=

    2018

  83. [92]

    arXiv preprint arXiv:1907.10641 , year=

    WinoGrande: An Adversarial Winograd Schema Challenge at Scale , author=. arXiv preprint arXiv:1907.10641 , year=

    1907 arXiv

  84. [93]

    arXiv preprint arXiv:2110.14168 , year=

    Training Verifiers to Solve Math Word Problems , author=. arXiv preprint arXiv:2110.14168 , year=

    arXiv

  85. [94]

    NeurIPS , year=

    Measuring Mathematical Problem Solving With the MATH Dataset , author=. NeurIPS , year=

  86. [95]

    OpenAI , year=

    Chatgpt: Optimizing language models for dialogue , author=. OpenAI , year=

  87. [96]

    2021 , eprint=

    Evaluating Large Language Models Trained on Code , author=. 2021 , eprint=

    2021

  88. [97]

    Learning from Task Descriptions

    Weller, Orion and Lourie, Nicholas and Gardner, Matt and Peters, Matthew E. Learning from Task Descriptions. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.105

    2020 doi

  89. [98]

    arXiv preprint arXiv:2110.08207 , year=

    Multitask prompted training enables zero-shot task generalization , author=. arXiv preprint arXiv:2110.08207 , year=

    arXiv

  90. [99]

    Cross-Task Generalization via Natural Language Crowdsourcing Instructions

    Mishra, Swaroop and Khashabi, Daniel and Baral, Chitta and Hajishirzi, Hannaneh. Cross-Task Generalization via Natural Language Crowdsourcing Instructions. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022. do...

    2022 doi

  91. [100]

    International Conference on Learning Representations , year=

    Finetuned Language Models are Zero-Shot Learners , author=. International Conference on Learning Representations , year=

  92. [101]

    FollowBench:

    Yuxin Jiang and Yufei Wang and Xingshan Zeng and Wanjun Zhong and Liangyou Li and Fei Mi and Lifeng Shang and Xin Jiang and Qun Liu and Wei Wang , editor =. FollowBench:. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa...

    2024 doi

  93. [102]

    Super- N atural I nstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks

    Wang, Yizhong and Mishra, Swaroop and Alipoormolabashi, Pegah and Kordi, Yeganeh and Mirzaei, Amirreza and Naik, Atharva and Ashok, Arjun and Dhanasekaran, Arut Selvan and Arunkumar, Anjana and Stap, David and Pathak, Eshaan and Karamanolakis, Giannis and Lai, Haizhi and Puroh...

    2022

  94. [103]

    2023 , url=

    Chunting Zhou and Pengfei Liu and Puxin Xu and Srini Iyer and Jiao Sun and Yuning Mao and Xuezhe Ma and Avia Efrat and Ping Yu and LILI YU and Susan Zhang and Gargi Ghosh and Mike Lewis and Luke Zettlemoyer and Omer Levy , booktitle=. 2023 , url=

    2023

  95. [104]

    Lion: Adversarial Distillation of Proprietary Large Language Models

    Jiang, Yuxin and Chan, Chunkit and Chen, Mingyang and Wang, Wei. Lion: Adversarial Distillation of Proprietary Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.189

    2023 doi

  96. [105]

    2023 , url =

    Mike Conover and Matt Hayes and Ankit Mathur and Jianwei Xie and Jun Wan and Sam Shah and Ali Ghodsi and Patrick Wendell and Matei Zaharia and Reynold Xin , title =. 2023 , url =

    2023

  97. [106]

    Together Computer , title =

  98. [107]

    The Twelfth International Conference on Learning Representations , year=

    OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text , author=. The Twelfth International Conference on Learning Representations , year=

  99. [108]

    doi:10.57967/hf/3488 , publisher =

    Xiaofeng Shi and Lulu Zhao and Hua Zhou and Donglin Hao , title =. doi:10.57967/hf/3488 , publisher =

    doi

  100. [109]

    , author=

    Visualizing data using t-SNE. , author=. Journal of machine learning research , volume=

  101. [110]

    5 Technical Report , author=

    Qwen2. 5 Technical Report , author=. arXiv preprint arXiv:2412.15115 , year=

    arXiv

  102. [111]

    The Thirteenth International Conference on Learning Representations , year=

    Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization , author=. The Thirteenth International Conference on Learning Representations , year=

  103. [112]

    When Evolution Strategy Meets Language Models Tuning

    Huang, Bo and Jiang, Yuxin and Chen, Mingyang and Wang, Yi and Chen, Hongyang and Wang, Wei. When Evolution Strategy Meets Language Models Tuning. Proceedings of the 31st International Conference on Computational Linguistics. 2025

    2025

  104. [113]

    From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning

    Li, Ming and Zhang, Yong and Li, Zhitao and Chen, Jiuhai and Chen, Lichang and Cheng, Ning and Wang, Jianzong and Zhou, Tianyi and Xiao, Jing. From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning. Proceedings of the 2024 Con...

    2024 doi

  105. [114]

    Learning to Edit: Aligning LLMs with Knowledge Editing , booktitle =

    Yuxin Jiang and Yufei Wang and Chuhan Wu and Wanjun Zhong and Xingshan Zeng and Jiahui Gao and Liangyou Li and Xin Jiang and Lifeng Shang and Ruiming Tang and Qun Liu and Wei Wang , editor =. Learning to Edit: Aligning LLMs with Knowledge Editing , booktitle =. 2024 , url =. d...

    2024 doi

  106. [115]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , booktitle =. 202...

    2020

  107. [116]

    Advances in Neural Information Processing Systems , year=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , year=

  108. [117]

    Advances in neural information processing systems , pages=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , pages=. 2022 , url=

    2022

  109. [118]

    Opencompass: A universal evaluation platform for foundation models , author=

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.