Pith. sign in

REVIEW 5 major objections 5 minor 117 references

Web text becomes 237k HSS instruction pairs; teacher-forced answers lift a fine-tuned 8B model past 14 rival datasets.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 08:25 UTC pith:SEOM6O5I

load-bearing objection Solid method for HSS data synthesis, but the human-preference SOTA is built on LLM judges and needs independent validation. the 5 major comments →

arxiv 2607.27379 v1 pith:SEOM6O5I submitted 2026-07-29 cs.CL

HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

Ru Peng , Tianyu Zhao , Xijun Gu , Zhiting Fan , Haokai Xu , Jinyang Zhang , Yawen Zeng , Yihong Zhuang
show 4 more authors
Kexin Yang Junyang Lin Dayiheng Liu Junbo Zhao
This is my paper
classification cs.CL
keywords data synthesisinstruction tuninghumanities and social sciencesteacher-forced answeringbacktranslationQ&A alignmentweb corporafine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that high-quality, diverse instruction-tuning data for open-ended humanities and social-sciences tasks can be synthesized automatically from web corpora, rather than collected from humans or distilled from a frontier model. It builds a three-stage pipeline that filters and refines seed documents, backtranslates them into persona- and requirement-specific instructions, and anchors LLM answers to the seed documents during generation. If the central claim is right, the approach removes a major data bottleneck for HSS domains and transfers across model architectures and sizes, with a fine-tuned Qwen3-8B approaching the official instruction-tuned release.

Core claim

The central discovery is teacher-forced answering (TeachForceA): when generating an answer, the synthesis model is given the seed document alongside the instruction as a semantic anchor. This reduces hallucination, preserves the original tone and style, and improves completeness and readability, producing answers that score higher on the paper's 12-rubric quality scale and lead to better fine-tuned models than either the raw seed document or a purely reverse-generated answer. Combined with multi-attribute instruction backtranslation (which specifies both the task requirements and the persona) and a Q&A alignment check, the pipeline yields 237k instruction–answer pairs from Qwen3-30B-A3B that

What carries the argument

The load-bearing machinery is the three-stage pipeline. Stage one builds clean seed documents from web text through source sampling, heuristic filtering, LLM domain classification into 14 HSS fields, 12-rubric quality rating (readability, applicability, human-touch), LLM text refinement, and a refinement judge. Stage two is multi-attribute instruction backtranslation: the LLM writes instructions that specify domain, genre, length, core content, structure, narrative voice, and a persona ('what' and 'who'), then a Q&A alignment check confirms the reverse instruction can faithfully reproduce the seed. Stage three is TeachForceA, which feeds the seed document to the answering model as a semantic

Load-bearing premise

The pipeline's quality controls—domain classification, rubric rating, text refinement, refinement judging, Q&A alignment, and teacher-forced answering—are all performed by the same LLM family (Qwen3-30B-A3B) that generates the data, and the rubric's validity rests on only 400+ seed texts rated by human annotators plus a single GPT-4.1 comparison; if these judgments are biased or self-referential, the dataset may be optimized to the synthesizer's taste rather than to true HSS

What would settle it

Take a random sample of 1,000 HSS-Synth instruction–answer pairs and have independent human experts (not the paper's raters) rate them against the same 12 rubrics and a Q&A faithfulness check; or re-run the whole pipeline using a different model family (e.g., a non-Qwen judge) for quality rating and Q&A alignment and then compare downstream fine-tuning performance. If the quality advantage shrinks or vanishes, or if the human experts find systematic misalignment, the pipeline's internal LLM-based validation is circular and the headline gains are artifacts of self-preference.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Fine-tuning a base model on HSS-Synth outperforms 14 leading instruction-data baselines across 16 benchmarks, coming closest to the official instruction-tuned Qwen3-8B.
  • Teacher-forced answers are the best answer type; raw seed documents are unsuitable as answers because their embedding distribution diverges sharply from reverse-generated answers.
  • The gains transfer across model families and sizes: Qwen2.5-14B and Llama3.1-8B both improve markedly on HSS-Synth data.
  • There is no detected 'performance seesaw': human-preference gains do not come at the cost of knowledge benchmarks, and some knowledge capabilities improve slightly.
  • Different capabilities saturate at different data scales: knowledge-based skills converge around 10% of the 237k samples, human-preference skills around 20%, and instruction following around 33%.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same teacher-forced anchoring idea could apply to other open-ended synthetic-data pipelines (e.g., medicine, law, advice-giving), where grounding answers in a source document may reduce hallucination and style drift without human annotation.
  • Because the same Qwen model family both generates and judges the data, the reported 'quality' may partly reflect the synthesis model's own preferences; a direct test would be to re-run the pipeline with an independent judge model or human raters and see whether downstream gains survive.
  • The observed saturation pattern suggests that SFT injects little new factual knowledge; this implies that for knowledge-heavy HSS applications, larger gains would come from synthesizing pre-training data rather than instruction data.
  • The paper's reuse metrics (n-gram overlap, LCS, copy ratio) offer a cheap, general test for grounding in any synthetic QA pipeline; a low copy ratio with high quality could indicate genuine abstraction rather than copying.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces HSS-Synth, a three-stage pipeline for synthesizing instruction-tuning data in 14 humanities and social science domains. Stage 1 constructs seed documents from SlimPajama via source sampling, heuristic filtering, LLM domain classification, a 12-rubric quality rating, and LLM text refinement with a refinement judge. Stage 2 performs multi-attribute instruction backtranslation (requirements plus persona) with a Q&A alignment check. Stage 3 proposes teacher-forced answering (TeachForceA), where the seed document is fed alongside the instruction as a semantic anchor. The authors fine-tune Qwen3-8B-Base on the resulting ~230k samples and report improvements over 14 baselines across 16 benchmarks, claiming a new SOTA that approaches the official Qwen3-8B while avoiding a 'performance seesaw' between human preference and knowledge capabilities. They also report ablations, cross-model transfer to Llama3.1-8B and Qwen2.5-14B, scaling analyses, and a rubric validation study.

Significance. If the central claims hold, the work is a substantial step for open-ended, subject-centric data synthesis: it defines a principled HSS taxonomy, offers a reproducible pipeline, ships public code, and provides broad benchmark coverage with a decontamination check. The cross-architecture transfer and component ablations are also useful. However, the paper's signature claims are currently supported by evidence that is partly circular and partly indirect. The human-preference benchmarks are judged by Claude LLMs rather than humans; the quality-control stack is run entirely by the same model family (Qwen3-30B-A3B) that generates the data; rubric validation is limited to 400+ seed texts, not final model outputs; and no significance tests, multiple seeds, or length-controlled evaluations are reported. These gaps directly affect the headline 'SOTA' and 'no performance seesaw' claims, so the manuscript requires substantive additional validation before the claims can be accepted.

major comments (5)
  1. [§4.1 / Table 1] The 'human preference' category is not measured with humans. WritingBench, CreativeWriting-v3, Judgemark, BuzzBench, and EQ-Bench3 are all scored by Claude-3.5/3.7-Sonnet (the paper states this in §4.1). The claim of 'improving human preference' is therefore a claim about LLM-judge preference. No human preference study on the final model outputs is reported. Please add (a) a human evaluation of the fine-tuned models' outputs, or (b) at minimum a length-controlled LLM-judge evaluation (e.g., following Dubois et al., 2024, already cited), and (c) variance or significance measures across multiple seeds, since Table 1 reports point estimates with no error bars.
  2. [§3.1–§3.3 / §4.6] The pipeline uses Qwen3-30B-A3B for domain classification, quality rating, text refinement, refinement judging, Q&A alignment checking, and teacher-forced answering. The rubric validation in §4.6 is only on 400+ seed documents with human raters and a GPT-4.1 comparison; it does not validate the quality of the final instruction–answer pairs. This creates a circularity risk: the dataset may be optimized toward what the synthesis model (and its relatives) reward, rather than toward genuine HSS quality. Please validate the final dataset and the trained models with independent judges, ideally including human ratings of a sample of final (instruction, answer) pairs.
  3. [Abstract / §4.1 / Appendix C] The abstract claims 237k samples, §4.1 says 230k instruction–answer pairs were used for fine-tuning, and Appendix C reports 237,340 total samples. This discrepancy is not explained. If 230k is the training-set size, the abstract and appendix should be reconciled; if the dataset is larger than the training split, the filtering/selection procedure should be described. As written, the reader cannot determine what was actually trained on.
  4. [Appendix E / Table 10] The case study contradicts the instruction-fidelity claim. The reverse instruction explicitly requires 'approximately 413 tokens (around 2130 characters)', but the reverse answer is 433 tokens and the teacher-forced answer is 644 tokens, with the latter praised as satisfying the length constraint. This suggests the LLM judge used for rubric scores rewards length/detail over instruction fidelity, and it directly undermines the claimed advantage of TeachForceA. Please provide a length-controlled evaluation of answer quality and report the distribution of output lengths relative to instruction-specified lengths.
  5. [§4.2 / Figure 2] The 'no performance seesaw' and 'no cross-skill trade-offs' conclusions are based on comparing averages over heterogeneous benchmarks with different metrics and no confidence intervals. The scaling curves in Figure 2 are also point estimates. Without error bars, significance tests, or multiple seeds, the absence of a trade-off is not established. Please report per-seed variation or bootstrap confidence intervals for the key comparisons in Table 1 and Figure 2.
minor comments (5)
  1. [Table 3] Typo: 'Evlo-instruct' should be 'Evol-Instruct'. Also, Table 1 contains a run-together entry '51.2937.9053.36' in the SynthQuestions row; please fix the formatting.
  2. [Figure 5 / Figure 6] Captions: Figure 5's caption says 'model trained on seed documents' and Figure 6 says 'model trained on reverse answers', but the final sentence of Figure 6's caption reads 'The model trained on seed answers clearly finds its answer pattern easy to learn', which appears to be a copy-paste error (should be 'reverse answers').
  3. [§3.1] Minor nomenclature: 'Slimpajama' should be 'SlimPajama'. In Table 1, baseline rows list 'Qwen-30B-A3B' for WRAP and LongForm while the text uses 'Qwen3-30B-A3B'; please harmonize.
  4. [Abstract / §3.3] The abstract phrase 'teacher-forced Answering that fed seed documents during response' is grammatically unclear; suggest 'during response generation' or similar. Also, the term 'Teacher-Forced Answering' is introduced in §1, §3.3, and again in Table 1; please define it once in a glossary-like manner.
  5. [References] The Cosmopedia-v2 reference lists only 'Loubna Ben, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, Leandro von Werra' and the year; if this is the Cosmopedia technical report, please give the full author list or a citable handle. Some other entries (e.g., 'Tombench') also mix arXiv identifiers with incomplete author lists.

Circularity Check

0 steps flagged

No circularity: the central SOTA claim is measured on external benchmarks; the same-model quality controls are a limitation, not a definitional reduction.

full rationale

The paper's central claim is that Qwen3-8B-Base fine-tuned on HSS-Synth outperforms 14 baselines on 16 benchmarks. This is an empirical result measured on external benchmarks (WritingBench, CreativeWriting-v3, Judgemark, MMLU, HellaSwag, etc.), not a quantity derived from the pipeline's own filters. The pipeline does use Qwen3-30B-A3B for domain classification, quality rating, text refinement judging, Q&A alignment checks, and teacher-forced answering, and the paper's own Limitations section concedes: 'HSS-Synth’s three-stage pipeline depends on the synthesis model and may introduce LLM bias, a common limitation of current data synthesis methods.' That is a recognized validity limitation, but it is not a circular derivation: none of the benchmark scores is defined in terms of these ratings, and the rubric validation in §4.6 includes independent human raters on 400+ seed texts ('Twenty uninvolved raters independently scored each item on a five-point scale... The synthesis model matched majority-vote human labels in 86% of cases'). The 'human preference' benchmarks are scored by Claude LLM judges rather than humans, and the comparison of answer types in §4.5 partly uses the same rubric family; these are measurement-validity caveats, not cases where an output equals an input by construction. No self-citation carries the argument, and no fitted parameter is relabeled as a prediction. The central comparison against 14 external baselines is self-contained and falsifiable, so the honest finding is no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central claim rests on hand-set quality thresholds and on the assumption that LLM-based judgments are reliable for open-ended HSS content. The paper partially validates the rubric with human raters, but the full pipeline is self-referential: the synthesis model also acts as judge. No new physical or mathematical entities are introduced.

free parameters (3)
  • Quality rubric thresholds = readability≥5, applicability≥4, human-touch≥3, total score≥45
    Hand-set in §3.1/Appendix A to balance seed-document quality against volume; directly controls which web documents become training seeds.
  • Source sampling rate for C4/CommonCrawl = 10%
    Appendix A: random 10% sample from crawler-based sources; affects domain composition and content diversity.
  • Rubric tier weights = readability weight 0.5, applicability 1.0, human-touch 1.5
    Appendix A: chosen to define the 0–60 rubric score used in quality bands and seed selection.
axioms (5)
  • domain assumption The 14-field QS-derived taxonomy is an adequate decomposition of the humanities and social sciences.
    Adopted in §1 and §3.1; the Limitations section itself notes anthropology and religious studies are missing, so the taxonomy is a modeling choice, not a fixed ground truth.
  • domain assumption Qwen3-30B-A3B can reliably classify, rate, refine, judge, and align HSS documents.
    Used throughout §3.1–§3.3; validation in §4.6 covers only 400+ seed texts and one proprietary model comparison, not the full 230k pipeline.
  • domain assumption The 16 chosen benchmarks are valid measures of the target HSS and general LLM capabilities.
    Section 4.1: several 'human-preference' benchmarks are scored by Claude models rather than human raters, so the labels are only as valid as the judge models.
  • domain assumption A 13-gram decontamination check is sufficient to rule out train/test overlap.
    Section 4.1 asserts zero overlap between synthesized data and benchmark test sets using 13-gram analysis; longer or paraphrased overlap is not examined.
  • domain assumption Single-run fine-tuning with fixed hyperparameters is a fair comparison across all baselines.
    Section 4.1/Appendix D uses one training run per method with shared hyperparameters; no multiple seeds or variance estimates are reported.

pith-pipeline@v1.3.0-daily-deepseek · 29614 in / 8896 out tokens · 86608 ms · 2026-08-01T08:25:26.283270+00:00 · methodology

0 comments
read the original abstract

High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. Data synthesis is a viable alternative and succeeds on closed tasks, yet the humanities and social sciences (HSS) are overlooked, and their open-ended nature makes synthesis challenging. Moving beyond prior capability-centric, fragmented attempts, we adopt a subject-centric paradigm, define the first HSS domain system covering 14 mainstream fields, and introduce HSS-Synth, the first data synthesis pipeline for HSS. HSS-Synth comprises: (1) constructing seed documents from web corpora via multi-step filtering and text refinement evaluated by a judge; (2) specifying "requirements + persona" to backtranslate seed documents into diverse yet faithful instructions with a strict Q&A alignment check; and (3) breaking LLM response limits via teacher-forced Answering that feeds seed documents during response generation to anchor semantics, reduce hallucinations, and preserve tone and integrity. HSS-Synth yields 237k high-quality, diverse instruction-tuning samples that outperform 14 leading baselines on 16 benchmarks. The fine-tuned Qwen3-8B-Base sets a new SOTA and approaches the official Qwen3-8B, improving both human preference and knowledge capabilities without performance seesaws. Extensive experiments demonstrate HSS-Synth's robustness and transferability. Our code is publicly available at https://github.com/pengr/HSS-Synth.

Figures

Figures reproduced from arXiv: 2607.27379 by Dayiheng Liu, Haokai Xu, Jinyang Zhang, Junbo Zhao, Junyang Lin, Kexin Yang, Ru Peng, Tianyu Zhao, Xijun Gu, Yawen Zeng, Yihong Zhuang, Zhiting Fan.

Figure 1
Figure 1. Figure 1: Overview of the HSS-Synth pipeline: (1) Seed document construction: sample, filter, classify, rate, and refine web texts under judge into clean HSS seeds; (2) Question-Answer Pair Refinement—backtranslate instructions from seed with multiple attributes, specify the “what” and “who,” and enforce Q&A alignment checks to yield diverse, faithful Q&A pairs; (3) Teacher-forced Answering—anchor answers on seeds t… view at source ↗
Figure 2
Figure 2. Figure 2: Impact of HSS-Synth data scale on 8 LLM ca [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: t-SNE visualization (left) and Vendi scores [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: t-SNE plot of sentence embeddings for reverse [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Inference token likelihood heatmaps for models trained on [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Inference token likelihood heatmaps for models trained on [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt template for domain classification to classify documents into HSS domains. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part I). [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part II). [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Prompt template for quality rating that assesses HSS documents using expert-crafted rubrics (Part III). [PITH_FULL_IMAGE:figures/full_fig_p022_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Prompt template for text refinement to polish seed documents. [PITH_FULL_IMAGE:figures/full_fig_p023_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt template for a refinement judge that determines whether text refinement is successful. [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Prompt template for multi-attribute instruction backtranslation, which backtranslates instructions from [PITH_FULL_IMAGE:figures/full_fig_p025_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Prompt template for question-answer alignment check used to verify that the reverse instructions [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Prompt template for teacher-forced answering, which produces answers anchored to seed documents that [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

117 extracted references · 4 canonical work pages

  1. [1]

    Transactions on Machine Learning Research , issn=

    Emergent Abilities of Large Language Models , author=. Transactions on Machine Learning Research , issn=

  2. [2]

    The Thirteenth International Conference on Learning Representations , year=

    DataMan: Data Manager for Pre-training Large Language Models , author=. The Thirteenth International Conference on Learning Representations , year=

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Scaling data-constrained language models , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    First Conference on Language Modeling , year=

    Best Practices and Lessons Learned on Synthetic Data , author=. First Conference on Language Modeling , year=

  5. [5]

    Awesome-LLM-in-Social-Science , howpublished =

  6. [6]

    Proceedings of the National Academy of Sciences , volume =

    Fabrizio Gilardi and Meysam Alizadeh and Maël Kubli , title =. Proceedings of the National Academy of Sciences , volume =. 2023 , doi =

    2023

  7. [7]

    and Khashabi, Daniel and Hajishirzi, Hannaneh

    Wang, Yizhong and Kordi, Yeganeh and Mishra, Swaroop and Liu, Alisa and Smith, Noah A. and Khashabi, Daniel and Hajishirzi, Hannaneh. Self-Instruct: Aligning Language Models with Self-Generated Instructions. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023. doi:10.18653/v1/2023.acl-long.754

    doi:10.18653/v1/2023.acl-long.754 2023

  8. [8]

    Can Xu and Qingfeng Sun and Kai Zheng and Xiubo Geng and Pu Zhao and Jiazhan Feng and Chongyang Tao and Qingwei Lin and Daxin Jiang , booktitle=. Wizard

  9. [9]

    arXiv preprint arXiv:2412.08905 , year=

    Phi-4 technical report , author=. arXiv preprint arXiv:2412.08905 , year=

    Pith/arXiv arXiv

  10. [10]

    Behavior research methods , volume=

    MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment , author=. Behavior research methods , volume=. 2010 , publisher=

    2010

  11. [11]

    Transactions on machine learning research , year=

    The Vendi Score: A Diversity Evaluation Metric for Machine Learning , author=. Transactions on machine learning research , year=

  12. [12]

    Biochemia medica , volume=

    Interrater reliability: the kappa statistic , author=. Biochemia medica , volume=. 2012 , publisher=

    2012

  13. [13]

    The Twelfth International Conference on Learning Representations , year=

    MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models , author=. The Twelfth International Conference on Learning Representations , year=

  14. [14]

    arXiv preprint arXiv:2108.07732 , year=

    Program synthesis with large language models , author=. arXiv preprint arXiv:2108.07732 , year=

    Pith/arXiv arXiv

  15. [15]

    arXiv preprint arXiv:2305.17126 , year=

    Large language models as tool makers , author=. arXiv preprint arXiv:2305.17126 , year=

    Pith/arXiv arXiv

  16. [16]

    Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=

    Tabula: Harnessing language models for tabular data synthesis , author=. Pacific-Asia Conference on Knowledge Discovery and Data Mining , pages=. 2025 , organization=

    2025

  17. [17]

    arXiv preprint arXiv:2401.17268 , year=

    Weaver: Foundation models for creative writing , author=. arXiv preprint arXiv:2401.17268 , year=

    Pith/arXiv arXiv

  18. [18]

    arXiv preprint arXiv:2504.14375 , year=

    Bottom-Up Synthesis of Knowledge-Grounded Task-Oriented Dialogues with Iteratively Self-Refined Prompts , author=. arXiv preprint arXiv:2504.14375 , year=

    Pith/arXiv arXiv

  19. [19]

    Advances in Neural Information Processing Systems , year=

    MAmmoTH2: Scaling Instructions from the Web , author=. Advances in Neural Information Processing Systems , year=

  20. [20]

    ACM Transactions on Intelligent Systems and Technology , volume=

    A comprehensive overview of large language models , author=. ACM Transactions on Intelligent Systems and Technology , volume=

  21. [21]

    arXiv preprint arXiv:2210.11610 , year=

    Large language models can self-improve , author=. arXiv preprint arXiv:2210.11610 , year=

    Pith/arXiv arXiv

  22. [22]

    arXiv preprint arXiv:2504.15573 , year=

    Instruction-Tuning Data Synthesis from Scratch via Web Reconstruction , author=. arXiv preprint arXiv:2504.15573 , year=

    Pith/arXiv arXiv

  23. [23]

    Soboleva, Daria and Al-Khateeb, Faisal and Myers, Robert and Steeves, Jacob R and Hestness, Joel and Dey, Nolan , title =

  24. [24]

    arXiv preprint arXiv:2112.11446 , year=

    Scaling language models: Methods, analysis & insights from training gopher , author=. arXiv preprint arXiv:2112.11446 , year=

    Pith/arXiv arXiv

  25. [25]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  26. [26]

    zip: Compressing text classification models , author=

    Fasttext. zip: Compressing text classification models , author=. arXiv preprint arXiv:1612.03651 , year=

    Pith/arXiv arXiv

  27. [27]

    Advances in Neural Information Processing Systems , volume=

    The fineweb datasets: Decanting the web for the finest text data at scale , author=. Advances in Neural Information Processing Systems , volume=

  28. [28]

    Proceedings

    On the resemblance and containment of documents , author=. Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171) , pages=. 1997 , organization=

    1997

  29. [29]

    Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    Efficient Attentions for Long Document Summarization , author=. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages=

    2021

  30. [30]

    D o G -Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping

    Chen, Yongrui and Jiang, Haiyun and Huang, Xinting and Shi, Shuming and Qi, Guilin. D o G -Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). 2024. do...

    work page doi:10.18653/v1/2024.naacl-long.230 2024

  31. [31]

    arXiv preprint arXiv:2504.12563 , year=

    MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation , author=. arXiv preprint arXiv:2504.12563 , year=

    arXiv

  32. [32]

    arXiv preprint arXiv:2304.08460 , year=

    Longform: Effective instruction tuning with reverse instructions , author=. arXiv preprint arXiv:2304.08460 , year=

    Pith/arXiv arXiv

  33. [33]

    The Twelfth International Conference on Learning Representations , year=

    Self-Alignment with Instruction Backtranslation , author=. The Twelfth International Conference on Learning Representations , year=

  34. [34]

    arXiv preprint arXiv:2406.20094 , year=

    Scaling synthetic data creation with 1,000,000,000 personas , author=. arXiv preprint arXiv:2406.20094 , year=

    Pith/arXiv arXiv

  35. [35]

    arXiv preprint arXiv:2402.13064 , year=

    Synthetic data (almost) from scratch: Generalized instruction tuning for language models , author=. arXiv preprint arXiv:2402.13064 , year=

    Pith/arXiv arXiv

  36. [36]

    arXiv preprint arXiv:2402.18334 , year=

    Learning to generate instruction tuning datasets for zero-shot task adaptation , author=. arXiv preprint arXiv:2402.18334 , year=

    Pith/arXiv arXiv

  37. [37]

    Advances in neural information processing systems , volume=

    Professor forcing: A new algorithm for training recurrent networks , author=. Advances in neural information processing systems , volume=

  38. [38]

    WildChat: 1M Chat

    Wenting Zhao and Xiang Ren and Jack Hessel and Claire Cardie and Yejin Choi and Yuntian Deng , booktitle=. WildChat: 1M Chat. 2024 , url=

    2024

  39. [39]

    2023 , publisher =

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants , author =. 2023 , publisher =

    2023

  40. [40]

    arXiv preprint arXiv:2502.17328 , year=

    Mutual Reinforcement of LLM Dialogue Synthesis and Summarization Capabilities for Few-Shot Dialogue Summarization , author=. arXiv preprint arXiv:2502.17328 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2404.04475 , year=

    Length-controlled alpacaeval: A simple way to debias automatic evaluators , author=. arXiv preprint arXiv:2404.04475 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2507.05750 , year=

    DocTalk: Scalable Graph-based Dialogue Synthesis for Enhancing LLM Conversational Capabilities , author=. arXiv preprint arXiv:2507.05750 , year=

    Pith/arXiv arXiv

  43. [43]

    Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling

    Maini, Pratyush and Seto, Skyler and Bai, Richard and Grangier, David and Zhang, Yizhe and Jaitly, Navdeep. Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024. doi:10.18653/v1/2024.acl-long.757

    doi:10.18653/v1/2024.acl-long.757 2024

  44. [44]

    arXiv preprint arXiv:2406.08464 , year=

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing , author=. arXiv preprint arXiv:2406.08464 , year=

    Pith/arXiv arXiv

  45. [45]

    arXiv preprint arXiv:2412.02595 , year=

    Nemotron-CC: Transforming Common Crawl into a refined long-horizon pretraining dataset , author=. arXiv preprint arXiv:2412.02595 , year=

    Pith/arXiv arXiv

  46. [46]

    arXiv preprint arXiv:2502.04235 , year=

    Reformulation for Pretraining Data Augmentation , author=. arXiv preprint arXiv:2502.04235 , year=

    Pith/arXiv arXiv

  47. [47]

    arXiv preprint arXiv:2308.03281 , year=

    Towards general text embeddings with multi-stage contrastive learning , author=. arXiv preprint arXiv:2308.03281 , year=

    Pith/arXiv arXiv

  48. [48]

    Ben, Loubna and Lozhkov, Anton and Penedo, Guilherme and Wolf, Thomas and von Werra, Leandro , title =

  49. [49]

    arXiv preprint arXiv:2506.03968 , year=

    From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding , author=. arXiv preprint arXiv:2506.03968 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2408.07055 , year=

    Longwriter: Unleashing 10,000+ word generation from long context llms , author=. arXiv preprint arXiv:2408.07055 , year=

    Pith/arXiv arXiv

  51. [51]

    arXiv preprint arXiv:2308.06259 , year=

    Self-alignment with instruction backtranslation , author=. arXiv preprint arXiv:2308.06259 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv preprint arXiv:2501.12273 , year=

    Condor: Enhance llm alignment with knowledge-driven data synthesis and refinement , author=. arXiv preprint arXiv:2501.12273 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    Pith/arXiv arXiv

  54. [54]

    Qwen2.5: A Party of Foundation Models , url =

  55. [55]

    arXiv preprint arXiv:2312.06281 , year=

    Eq-bench: An emotional intelligence benchmark for large language models , author=. arXiv preprint arXiv:2312.06281 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv preprint arXiv:2009.03300 , year=

    Measuring massive multitask language understanding , author=. arXiv preprint arXiv:2009.03300 , year=

    Pith/arXiv arXiv 2009

  57. [57]

    arXiv preprint arXiv:1604.01696 , year=

    A corpus and evaluation framework for deeper understanding of commonsense stories , author=. arXiv preprint arXiv:1604.01696 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2311.07911 , year=

    Instruction-following evaluation for large language models , author=. arXiv preprint arXiv:2311.07911 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2307.08689 , year=

    Collie: Systematic construction of constrained text generation tasks , author=. arXiv preprint arXiv:2307.08689 , year=

    Pith/arXiv arXiv

  60. [60]

    Transactions of the Association for Computational Linguistics , volume=

    Coqa: A conversational question answering challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2019 , publisher=

    2019

  61. [61]

    Paech , title =

    Samuel J. Paech , title =. 2025 , howpublished =

    2025

  62. [62]

    arXiv preprint arXiv:2503.05244 , year=

    Writingbench: A comprehensive benchmark for generative writing , author=. arXiv preprint arXiv:2503.05244 , year=

    arXiv

  63. [63]

    arXiv preprint arXiv:1905.07830 , year=

    Hellaswag: Can a machine really finish your sentence? , author=. arXiv preprint arXiv:1905.07830 , year=

    Pith/arXiv arXiv 1905

  64. [64]

    arXiv preprint arXiv:2201.03533 , year=

    Scrolls: Standardized comparison over long language sequences , author=. arXiv preprint arXiv:2201.03533 , year=

    Pith/arXiv arXiv

  65. [65]

    arXiv preprint arXiv:1904.09728 , year=

    Socialiqa: Commonsense reasoning about social interactions , author=. arXiv preprint arXiv:1904.09728 , year=

    Pith/arXiv arXiv 1904

  66. [66]

    arXiv preprint arXiv:1808.08745 , year=

    Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization , author=. arXiv preprint arXiv:1808.08745 , year=

    Pith/arXiv arXiv

  67. [67]

    Transactions of the Association for Computational Linguistics , volume=

    The narrativeqa reading comprehension challenge , author=. Transactions of the Association for Computational Linguistics , volume=. 2018 , publisher=

    2018

  68. [68]

    arXiv preprint arXiv:2310.00746 , year=

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models , author=. arXiv preprint arXiv:2310.00746 , year=

    Pith/arXiv arXiv

  69. [69]

    arXiv preprint arXiv:2402.15052 , year=

    Tombench: Benchmarking theory of mind in large language models , author=. arXiv preprint arXiv:2402.15052 , year=

    Pith/arXiv arXiv

  70. [70]

    arXiv preprint arXiv:2305.07759 , year=

    Tinystories: How small can language models be and still speak coherent english? , author=. arXiv preprint arXiv:2305.07759 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2407.10671 , year=

    Qwen2 technical report , author=. arXiv preprint arXiv:2407.10671 , year=

    Pith/arXiv arXiv

  72. [72]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    Pith/arXiv arXiv

  73. [73]

    5-math technical report: Toward mathematical expert model via self-improvement , author=

    Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement , author=. arXiv preprint arXiv:2409.12122 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2306.11644 , year=

    Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2309.05463 , year=

    Textbooks are all you need ii: phi-1.5 technical report , author=. arXiv preprint arXiv:2309.05463 , year=

    Pith/arXiv arXiv

  76. [76]

    The Claude 3 Model Family: Opus, Sonnet, Haiku , url =

    Anthropic , year =. The Claude 3 Model Family: Opus, Sonnet, Haiku , url =

  77. [77]

    Alpacaeval: An automatic evaluator of instruction-following models , author=

  78. [78]

    From live data to high-quality benchmarks: The arena-hard pipeline , author=

  79. [79]

    Gonzalez and Ion Stoica , booktitle=

    Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica , booktitle=. Judging. 2023 , url=

    2023

  80. [81]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

Showing first 80 references.