Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Synthesizing instructions by first imagining who would ask them and why, then generating them from web documents, yields a 1M dataset that fine-tunes LLaMA-3-8B to the best Alpaca Eval 2.0 win rate among models trained only on open data.

desk verdict Attributed grounding is a genuinely new recipe for scalable instruction synthesis and the Alpaca Eval gains look real, but the Arena Hard headline is partly a selection artifact from using the same rubric for filtering and evaluation. read the letter →

arxiv 2506.03968 v1 pith:2Q73UB45 submitted 2025-06-04 cs.CL

Chiwei Zhu , Benfeng Xu , Xiaorui Wang , Zhendong Mao This is my paper
classification cs.CL
keywords syntheticinstructiondataattributedgroundingtuningLLMalignmentdiversityweb-scalesynthesissupervisedfine-tuningArenaHardrubric
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models learn to follow instructions from example pairs, but human-written examples are scarce and existing synthetic generation methods produce narrow or shallow prompts. This paper tries to break that bottleneck with attributed grounding: first attribute real user instructions to a document, a user, and a motivation, then reverse the process and generate new instructions from arbitrary web documents by imagining who would ask them and why. The resulting 1-million-instruction dataset, SynthQuestions, fine-tunes LLaMA-3-8B to a 19.15 win rate on Alpaca Eval 2.0, the best among models trained on open-source data, and to 15.4 on Arena Hard, where only the 10M-instruction MAmmoTH2 baseline does better. The authors describe this as comparable to models trained with ten times more data plus preference training. A sympathetic reader would care because the method turns the open web into a scalable source of alignment data, with measured improvements continuing as the corpus grows.

What carries the argument

The load-bearing object is the attributed-grounding loop expressed in Equations (1)-(3). Equation (1) builds the attributed seed set $RQ_\alpha = \{(i, d, u, m)\}$ by linking each curated real instruction $i$ to a retrieved web document $d$, a user $u$, and a motivation $m$. Equation (2) reverses the direction: given a fresh web document $d'$, LLaMA generates a grounded situation (user $u'$, motivation $m'$) using attributed samples from $RQ_\alpha$ as in-context demonstrations, and Equation (3) then asks the model to speak as that user and produce the instruction $i'$ most likely to arise in that situation. The same seven-dimension rubric drawn from Arena Hard — specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application — scored by LLaMA-3-70B-Instruct, selects the 29K seed instructions (only full-score 7 entries are kept), discards generated instructions scoring below 3, and ranks the final dataset after BERTopic topic modeling keeps the highest-scoring instructions per topic.

What would settle it

Have human annotators score a random sample of SynthQuestions instructions with the same seven dimensions and measure their agreement with LLaMA-3-70B-Instruct; if agreement is low, or if a model trained on unfiltered SynthQuestions beats the rubric-filtered model in a blind pairwise human-preference evaluation, the reported gains are an artifact of the judge rather than of instruction quality.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that the missing ingredient in synthetic instruction data is situatedness: instructions that align models efficiently are typically grounded in a document (the background knowledge), a user (who asks), and a motivation (why they ask), and synthetic instructions inherit that grounding only when the generating model is forced to construct all three. The paper implements this as a two-stage loop — top-down attribution of 29K curated human instructions (REALQUESTIONS) into document-user-motivation triples, then bottom-up synthesis that starts from web documents, generates a situation, and then an instruction — and shows that LLaMA-3-8B fine-tuned on the resulting 1M SynthQuestions outperforms all open-data-trained baselines on Alpaca Eval 2.0 with a 19.15 win rate, and trails only MAmmoTH2 on Arena Hard (15.4 versus 16.6), where MAmmoTH2 used 10M data plus additional math and code fine-tuning. The paper further claims the data scales: performance improves monotonically as the training subset grows from 200K toward 1M, transfers to Qwen2.5-7B and Qwen2.5-14B with large margins on MATH, and supports preference optimization, with a SynthQuestions-plus-DPO model reaching a 33.81 Alpaca Eval win rate that surpasses the LLaMA-3-70B-Instruct generator itself.

Load-bearing premise

The load-bearing premise is that the seven-dimension rubric — specificity, domain knowledge, complexity, problem-solving, creativity, technical accuracy, and real-world application — as scored by the single judge model LLaMA-3-70B-Instruct, is a trustworthy measure of instruction quality, because the same scoring selects the seed set, filters generated instructions, and is used by the Arena Hard benchmark itself, with no human-agreement check reported.

Editorial extensions

If this is right

  • If the scaling curve holds, adding more web corpora to the attribution loop should keep improving instruction-following performance, since document volume is effectively unlimited.
  • The recipe transfers across base models: Qwen2.5-7B and Qwen2.5-14B trained on SynthQuestions subsets beat the same models trained on OpenHermes, GenQA, and MAmmoTH2 subsets on most benchmarks, with particularly large margins on MATH.
  • The data is compatible with preference optimization: applying DPO on 100K sampled instructions raises the Alpaca Eval win rate from 19.15 to 33.81, surpassing the LLaMA-3-70B-Instruct data generator.
  • Choosing the document corpus is a direct control over which capabilities improve: mixing math and code documents (PILE, MathPILE) into FineWeb raises GSM8K from 50.34 to 58.30 without hurting Alpaca Eval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the attribution loop could be iterated — instructions synthesized in one pass could themselves be attributed to documents, users, and motivations and serve as demonstrations for a second pass, a self-amplifying diversity loop the authors do not test.
  • Beyond the paper: because mixing math and code documents shifts gains specifically to GSM8K, document selection acts as a difficulty dial, and deliberately curating web corpora by cognitive-demand profile could steer which capabilities alignment data improves.
  • Beyond the paper: since one judge model supplies the scores that select the seed set, filter generations, and rank the final dataset, part of the reported gain may be self-consistent optimization against that judge; a blind human-preference comparison between filtered and unfiltered models would separate genuine alignment gains from rubric conformity.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a framework for synthesizing large-scale instruction data using "attributed grounding," in which a set of high-quality human instructions (REALQUESTIONS) is first attributed to documents, users, and motivations, and then new instructions are generated from web documents by reversing this process with LLM prompts. The authors construct a 1M-instruction dataset (SynthQuestions), filter it using a seven-dimension rubric based on Arena Hard scored by LLaMA-3-70B-Instruct, and fine-tune LLaMA-3-8B and Qwen2.5 models. They report strong performance on Alpaca Eval 2.0 and Arena Hard, as well as on closed-form benchmarks, with ablations showing the benefit of attributed grounding and the inclusion of math/code documents. The paper also includes a DPO extension and scaling experiments.

Significance. If the results are robust, the attributed-grounding pipeline is a valuable and scalable method for producing diverse, complex instruction data, potentially reducing reliance on expensive human annotation. The paper's empirical core is well structured: the ablation in Table 5 supports the importance of grounding (+~5 Alpaca Eval points) and math/code documents (+~8 GSM8K points), and the cross-model evaluations on LLaMA-3-8B and Qwen2.5 demonstrate generalizability. The release of the dataset and code is a further positive. However, the headline Arena Hard results are compromised by a selection-bias concern: the same rubric is used to filter both the seed and synthetic data and to evaluate the final model. The Alpaca Eval 2.0 results, judged by GPT-4-1106 under a different protocol, are the cleaner evidence for the core claim.

major comments (4)
  1. [Sections 3, 4.2, and 6.1] The Arena Hard result in Table 1 is potentially a selection artifact because the same seven-dimension rubric that filters REALQUESTIONS (score 7/7 in Section 3) and selects synthetic instructions (score >=3 in Section 4.2, then highest-score-per-topic selection) is also the foundation of the Arena Hard benchmark used in Section 6.1. The paper provides no human-agreement study for this rubric, and Appendix I does not describe what the claimed 'modification' of Arena Hard consists of; the criteria in Appendix I are an absolute scoring rubric, not the pairwise comparison protocol of the original Arena Hard. To make the headline claim credible, the authors should either validate the rubric against human judgments or present Alpaca Eval 2.0 as the primary evidence and explicitly flag Arena Hard as potentially optimistic due to rubric overlap.
  2. [Section 1 and Table 1] The statement that SynthQuestions-trained models are "comparable with models trained with 10 times more data and preference training" is not supported by the reported numbers. LLaMA-3-8B-Instruct (which uses >10M data) outperforms SynthQuestions on both Arena Hard (20.6 vs 15.4) and Alpaca Eval 2.0 (22.57 vs 19.15). While the DPO variant in Table 3 reaches AE 33.81 and AR 24.8, that improvement comes after preference optimization and is not what the introduction claims. The claim should be rephrased to be bounded by the actual comparison group, e.g., 'among models trained exclusively on open-source data.'
  3. [Section 6.2 and abstract] The scaling experiment does not support the abstract's claim of improvements 'with more web corpora.' The experiment draws random subsets from the already-filtered 1M SynthQuestions, so it demonstrates scaling with instruction count, not with an expanding set of web documents. To substantiate the claim, the authors should hold the synthesis pipeline fixed and vary the underlying document corpus (e.g., additional FineWeb subsets), observing how model performance changes as more documents are used.
  4. [Section 4.2] The rationale for the score threshold 3 is unclear and inconsistent with the stated criteria. The text says that 'Specificity, Problem-Solving and Technical Accuracy are three more fundamental requirements,' but the threshold is applied to the total score across all seven dimensions, not to those three dimensions specifically. This ambiguity affects reproducibility of the dataset construction. Please clarify the scoring rule and provide an ablation or sensitivity analysis of the threshold's effect on downstream performance, or at least a distribution plot showing how scores correlate with quality.
minor comments (6)
  1. [Abstract and Section 1] The word 'sufficientlydiversified' is missing a space in the abstract; also 'diversified' and 'complicated' appear in an odd concatenation. Please fix the typography.
  2. [Section 6.1] The word 'repectively' should be 'respectively' in the first paragraph of Section 6.1.
  3. [Tables 1 and 2] Several model names are misspelled: 'Proprietay' should be 'Proprietary', 'LLaMA-3-8B-Insturct' should be 'LLaMA-3-8B-Instruct', and 'Qwen-2-7B-Insturct' should be 'Qwen-2-7B-Instruct'. Also, the table notes should clarify whether the unreported MAmmoTH2 Alpaca Eval 2.0 result is intentionally omitted.
  4. [Section 5.2 and Figure 6] The t-SNE plots are referenced as 'Figure ??' in the text; the figure numbers should be fixed. The violin plot in Figure 6 lacks axis labels and a description of what the y-axis range represents (0-7), which makes the score distribution comparison difficult to interpret.
  5. [Appendix I] The criteria listed in Appendix I are the same seven items used in the filtering prompt in Appendix J.3. The paper says these criteria are 'modified from Arena Hard,' but the modification is never described. This is important because the original Arena Hard uses a pairwise-comparison protocol; the absolute 7-dimensional scoring used here is substantially different and should be documented exactly.
  6. [Section 5.4] The safety analysis reports that 4.32% of data points have potential risks, but then states that 'for all the other categories, the potentially risky data points are less than 0.2%.' This is misleading because the 'Specialized Advice' category alone is 3.60%, and the sum of all other categories is well under 1%; please report the percentages consistently and clarify whether 'less than 0.2%' refers to each category individually.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation-level circularity: the core benchmark claims rest on external GPT-4 pairwise evals and closed-form tests, while the only self-referential step is the complexity comparison, which scores the dataset with the same rubric used to select it.

  1. self definitional [Section 4.2 (Synthesizing); invoked again in Section 5.3 (Complexity)]
    "To assure the diversity of the dataset, we conduct topic modeling with BERTopic [15] following [20] and construct a 1M-size final dataset by including instructions with the highest scores in each topic."

    The 'scores' in Section 4.2 are assigned with the Section 3 filter, which modifies Arena Hard criteria to score instructions on seven dimensions. Section 5.3 then reports that 'the score distribution of SYNTHQUESTIONS is notably condensed to the higher end' after scoring instructions 'with criteria from Arena Hard.' Because the final 1M subset was deliberately built by taking the highest-scoring instructions per topic on that exact rubric, the high complexity score is an expected consequence of the selection rule rather than an independent property of the synthesizing framework.

full rationale

The paper's main derivation is not circular. SynthQuestions is built by filtering human instructions with an Arena-Hard-derived rubric, attributing them to documents/users/motivations, reversing the process on web documents to synthesize new instructions, and then filtering by rubric score and topic diversity. The headline claims are tested by fine-tuning LLaMA-3-8B and measuring Alpaca Eval 2.0 and Arena Hard win rates, plus closed-form benchmarks. Arena Hard's win rate is a pairwise GPT-4 evaluation over a fixed benchmark prompt set, not a re-application of the Section 3 seven-dimension score to the trained model; Alpaca Eval 2.0 uses a different judge and prompt set. The paper even removes instructions semantically similar to the evaluation benchmarks. Thus the central benchmark predictions are empirical, external, and not forced by construction. No load-bearing self-citations or uniqueness arguments appear. The only self-referential move is the Section 5.3 complexity comparison, which uses the same Arena-Hard-derived score as the data filter, making the high-complexity violin plot partly an artifact of selection. That is a methodological caveat, not a collapse of the derivation chain. The Arena Hard result sharing a rubric with the filter is a validity risk worth a human-agreement study, but it is not a definitional equivalence: training on rubric-conforming prompts and then winning GPT-4 pairwise comparisons could still fail or succeed. Score 2 reflects one minor self-referential auxiliary claim while the core reasoning is self-contained.

Assumptions & free parameters 4 free parameters · 5 assumptions · 1 invented entities

Everything load-bearing in the pipeline is either an asserted domain assumption (situated cognition, ZPD, rubric validity, web-corpus coverage) or a hand-chosen threshold (rubric cutoffs 7 and 3, dedup 0.85, per-topic cap at 1M). No independent empirical entities are introduced; the attribution triple is a prompt-driven construct without external validation. The cost of the ledger is concentrated in rubric validity: if the Arena Hard rubric is a weak proxy for instruction quality, both the seed and synthetic selection are misdirected and the Arena Hard evaluation partially measures the curation itself.

free parameters (4)
  • Instruction quality threshold (synthetic) = Arena Hard rubric score >= 3
    Generated instructions below 3 on the seven-dimension rubric are discarded; the threshold is justified post hoc: 'we notice that Specificity, Problem-Solving and Technical Accuracy are three more fundamental requirements' (Section 4.2).
  • Seed quality cutoff = full rubric score 7 (29K instructions)
    REALQUESTIONS keeps only instructions scoring 7/7, which determines the seed distribution and demonstrations for all later synthesis (Section 3).
  • Deduplication similarity threshold = 0.85 embedding similarity
    Instructions whose embedding similarity exceeds 0.85 are clustered and reduced to one representative (Section 3).
  • Per-topic dataset cap = 1M instructions, highest rubric score per BERTopic topic
    The final dataset keeps the highest-scoring instructions per topic up to 1M; the topic balance and cap are hand-chosen (Section 4.2).
assumptions (5)
  • domain assumption Situated cognition premise: language understanding and instructions are grounded in activity, context, and culture, so instructions grounded in documents, users, and motivations are better for alignment.
    Motivates the entire attributed-grounding framework (Introduction, Section 4); supported only by the single ablation in Table 5, not by an independent measurement of groundedness.
  • domain assumption Vygotsky's Zone of Proximal Development transfers to LLM alignment: tasks just beyond the model's current capability maximize learning.
    Invoked in the Introduction to justify filtering for complex and challenging instructions (the source of the score-7 and score-3 cutoffs).
  • domain assumption The seven-dimension Arena Hard rubric, scored by LLaMA-3-70B-Instruct, is a valid proxy for instruction quality.
    Every curation stage selects on this rubric (Section 3 seeds at score 7; Section 4.2 generated data at score >= 3 and per-topic top scores), and the same rubric's dimensions underlie the Arena Hard evaluation benchmark (Section 6.1). No human-agreement study is reported (Appendix I).
  • domain assumption The document sources (FineWeb, PILE, MathPILE) and Google top-1 retrieval cover the real-world distribution of user instructions.
    Bottom-up synthesis starts from these documents (Section 4.2); no coverage analysis against a held-out human instruction set is provided.
  • ad hoc to paper LLM-generated user and motivation annotations (Equation 2) are valid grounding for instruction synthesis.
    Users and motivations are produced by LLaMA-3-8B-Instruct conditioned on demonstrations; there is no human validation of the attributed situations, only the indirect Table 5 ablation.
invented entities (1)
  • Attribution triple (document, user, motivation)
    purpose: Intermediate representation that grounds real instructions in the top-down step and conditions bottom-up synthesis (Equations 1-3).
    A methodological construct rather than an empirical entity; its validity is measured only indirectly through downstream benchmark scores and the Table 5 ablation, with no human evaluation of generated situations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding." pith.science (2026). https://pith.science/paper/2Q73UB45

@misc{pith2026250603968,
  author       = {Pith},
  title        = {Pith review of: From Real to Synthetic: Synthesizing Millions of Diversified and Complicated User Instructions with Attributed Grounding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2Q73UB45}},
  note         = {Machine review of arXiv:2506.03968}
}
read the original abstract

The pursuit of diverse, complex, and large-scale instruction data is crucial for automatically aligning large language models (LLMs). While there are methods capable of generating synthetic instructions at scale, they either suffer from limited grounding sources, leading to a narrow distribution, or rely on trivial extensions that fail to produce meaningful trajectories in terms of complexity. In contrast, instructions that benefit efficient alignment are typically crafted with cognitive insights and grounded in real-world use cases. In this paper, we synthesize such instructions using attributed grounding, which involves 1) a top-down attribution process that grounds a selective set of real instructions to situated users, and 2) a bottom-up synthesis process that leverages web documents to first generate a situation, then a meaningful instruction. This framework allows us to harvest diverse and complex instructions at scale, utilizing the vast range of web documents. Specifically, we construct a dataset of 1 million instructions, called SynthQuestions, and demonstrate that models trained on it achieve leading performance on several common benchmarks, with improvements that continually scale with more web corpora. Data, models and codes will be available at https://github.com/Ignoramus0817/SynthQuestions.

Figures

Figures reproduced from arXiv: 2506.03968 by the authors.

Figure 1
Figure 1. Human instructions can be attributed to documents, users and motivations. Conversely, in￾structions can also be synthesized from them. Alignment training [35] has become an essential technique for instruction-following large language models [27, 26, 39, 9, 14], which aims to align language models behaviors with human when given certain instructions through training on instruction￾response pairs. Researchers have bee… view at source ↗
Figure 2
Figure 2. Overview of our synthesizing framework. Various linguistic and social studies have pointed out that language understanding is based on world knowledge, which is situated, being in part a product of the activity, context, and culture in which it is developed and used [2, 11, 1]. Given this idea, we believe it is critical to assure the generated instructions are grounded to real world. In this work, we propose a synth… view at source ↗
Figure 3
Figure 3. A random sample of web document, the corresponding grounding situation and generated instruction. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (6 more)
Figure 9
Figure 9. Figure 9: Data Cleaning and Deduplication. There is massive noise in the raw conversation data and a variety of measures are applied to reduce it: 3 [PITH_FULL_IMAGE:figures/full_fig_p003_9.png]
Figure 4
Figure 4. Figure 4: Scores distribution of human instructions, based [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Left: t-SNE plot of SYNTHQUESTIONS along with MAmmoTH2, SocraticChat, GenQA and OpenHermes. SYNTHQUESTIONS covers more area than other datasets. Right: Comparison of diversity (vendi score) of synthesized datasets. Documents. We use FineWeb [28] as the main source for …
Figure 6
Figure 6. Figure 6: Comparison of complexity (Arena Hard Score) of all synthetic data. Additionally, we provide Vendi Score as a supplemental quan￾titative metric of data diversity. Vendi Score [12] is a diversity evaluation metric designed for machine learning dataset, which increases li…
Figure 7
Figure 7. Figure 7: Performance of models trained with subsets in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: presents the full visualization of dataset diversity and complexity of different synthesized datasets [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs

    cs.CL 2026-07 conditional novelty 6.0 of 10

    HSS-Synth generates 230k instruction-tuning samples for 14 humanities/social-science fields and reports state-of-the-art fine-tuning results on 16 benchmarks.

Reference graph

Works this paper leans on

65 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    Experience grounds language, 2020

    Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language, 2020

    work page 2020

  2. [2]

    Situated cognition and the culture of learning.Educational Researcher, 18(1):32–42, 1989

    John Seely Brown, Allan Collins, and Paul Duguid. Situated cognition and the culture of learning.Educational Researcher, 18(1):32–42, 1989

    work page 1989

  3. [3]

    Genqa: Generating millions of instructions from a handful of prompts, 2024

    Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. Genqa: Generating millions of instructions from a handful of prompts, 2024

    work page 2024

  4. [4]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023

    2023

  5. [5]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot arena: An open platform for evaluating llms by human preference, 2024

    2024

  6. [6]

    Chi, 9 Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping H...

    work page 2022

  7. [7]

    Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023

    2023

  8. [8]

    Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations.arXiv preprint arXiv:2305.14233, 2023

    arXiv 2023

Show all 65 references
  1. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zha...

    2024

  2. [10]

    Hashimoto

    Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024

    2024

  3. [11]

    Perspective taking as egocentric anchoring and adjustment.Journal of personality and social psychology, 87:327–39, 09 2004

    Nicholas Epley, Boaz Keysar, Leaf Van Boven, and Thomas Gilovich. Perspective taking as egocentric anchoring and adjustment.Journal of personality and social psychology, 87:327–39, 09 2004

    2004

  4. [12]

    The vendi score: A diversity evaluation metric for machine learning, 2023

    Dan Friedman and Adji Bousso Dieng. The vendi score: A diversity evaluation metric for machine learning, 2023

    2023

  5. [13]

    The pile: An 800gb dataset of diverse text for language modeling, 2020

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020

    2020

  6. [14]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools, 2024

    Team GLM, :, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zh...

    2024

  7. [15]

    Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022

    Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure.arXiv preprint arXiv:2203.05794, 2022

    2022 arXiv

  8. [16]

    Rae, Oriol Vinyals, and Laurent Sifre

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osin...

    2022

  9. [17]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020

    2020

  10. [18]

    Platolm: Teaching llms in multi-round dialogue via a user simulator, 2024

    Chuyi Kong, Yaxin Fan, Xiang Wan, Feng Jiang, and Benyou Wang. Platolm: Teaching llms in multi-round dialogue via a user simulator, 2024

    2024

  11. [19]

    Openassistant conversations – democratizing large language model alignment, 2023

    Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and A...

    2023

  12. [20]

    Gonzalez, and Ion Stoica

    Tianle Li*, Wei-Lin Chiang*, Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024

    2024

  13. [21]

    Self-alignment with instruction backtranslation.arXiv preprint arXiv:2308.06259, 2023

    Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation.arXiv preprint arXiv:2308.06259, 2023

    2023 arXiv

  14. [22]

    #instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023

    Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. #instag: Instruction tagging for analyzing supervised fine-tuning of large language models, 2023

    2023

  15. [23]

    Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment.Behavior research methods, 42(2):381–392, 2010

    Philip M McCarthy and Scott Jarvis. Mtld, vocd-d, and hd-d: A validation study of sophisticated approaches to lexical diversity assessment.Behavior research methods, 42(2):381–392, 2010

    2010

  16. [24]

    Cross-task generalization via natural language crowdsourcing instructions, 2022

    Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions, 2022

    2022

  17. [25]

    Instruction in the wild: A user-based instruction dataset.https://github.com/XueFuzhao/InstructionWild, 2023

    Jinjie Ni, Fuzhao Xue, Kabir Jain, Mahir Hitesh Shah, Zangwei Zheng, and Yang You. Instruction in the wild: A user-based instruction dataset.https://github.com/XueFuzhao/InstructionWild, 2023

    2023

  18. [26]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024

  19. [27]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  20. [28]

    The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    Guilherme Penedo, Hynek Kydlíˇcek, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Le- andro V on Werra, and Thomas Wolf. The fineweb datasets: Decanting the web for the finest text data at scale, 2024

    2024

  21. [29]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023

    2023

  22. [30]

    Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist llm assistants, 2023

    2023

  23. [31]

    Harvard university press, 1978

    Lev Semenovich Vygotsky.Mind in society: The development of higher psychological processes, volume 86. Harvard university press, 1978

    1978

  24. [32]

    Interpretable preferences via multi- objective reward modeling and mixture-of-experts

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi- objective reward modeling and mixture-of-experts. InEMNLP, 2024

    2024

  25. [33]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions, 2023

    2023

  26. [34]

    Generative ai for math: Part i – mathpile: A billion-token-scale pretraining corpus for math, 2023

    Zengzhi Wang, Rui Xia, and Pengfei Liu. Generative ai for math: Part i – mathpile: A billion-token-scale pretraining corpus for math, 2023

    2023

  27. [35]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y . Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V . Le. Finetuned language models are zero-shot learners, 2022

    2022

  28. [36]

    Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. Learning from task descriptions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1361–1375, Online, November 2020. Association for Computational Linguistics

    2020

  29. [37]

    Expert- prompting: Instructing large language models to be distinguished experts, 2023

    Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, Yongdong Zhang, and Zhendong Mao. Expert- prompting: Instructing large language models to be distinguished experts, 2023

    2023

  30. [38]

    Wizardlm: Empowering large language models to follow complex instructions, 2023

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023

    2023

  31. [39]

    Qwen2 technical report, 2024

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024

  32. [40]

    Mammoth2: Scaling instructions from the web, 2024

    Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web, 2024

    2024

  33. [41]

    Wildchat: 1m chatgpt interaction logs in the wild, 2024

    Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: 1m chatgpt interaction logs in the wild, 2024

    2024

  34. [42]

    Xing, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2024

    2024

  35. [43]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. 13 A Dataset and Model Licenses Her...

    2023

  36. [44]

    Specificity: Does the instruction ask for a specific output?

  37. [45]

    Domain Knowledge: Does the instruction cover one or more specific domains?

  38. [46]

    Complexity: Does the instruction have multiple levels of reasoning, components, or variables?

  39. [47]

    Problem-Solving: Does the instruction directly involve the AI to demonstrate active problem-solving skills?

  40. [48]

    Creativity: Does the instruction involve a level of creativity in approaching the problem?

  41. [49]

    Technical Accuracy: Does the instruction require technical accuracy in the response?

  42. [50]

    Real-world Application: Does the instruction relate to real-world applications? J Prompts and Demonstrations J.1 Prompt and Demonstrations Used in Attributing Step Prompts Used in Attributing [SYSTEM PROMPT] Given a document and a query to an AI assistant

  43. [51]

    You should link the document and the user query with a practical scene , considering user identity and motivation

  44. [52]

    buy and hold

    Decompose the query regarding ability , knowledge , output and extra information : - Ability : The fundamental skills or capabilities required to address the problem . - Knowledge : The relevant domain or subject matter related to the query . - Output : The expected type of re...

  45. [53]

    Describe this scenario in one paragraph , detailing the logical steps from the document ’ s content to a query directed at an AI assistant

    Envision a real - world scenario based on the provided document . Describe this scenario in one paragraph , detailing the logical steps from the document ’ s content to a query directed at an AI assistant

  46. [54]

    - Knowledge : The relevant domain or subject matter related to the query

    Then list the compositions of a query that could emerge from this scenario , including : - Ability : The fundamental skills or capabilities required to address the problem . - Knowledge : The relevant domain or subject matter related to the query . - Output : The expected type...

  47. [55]

    thought

    Finally formulate a user query based on the scenario and query compositions you have identified . Ensure : - Maximize the ability that is needed to solve the query . Avoid simple copying or extracting tasks . - The query should be practical , complex and requires advanced skil...

  48. [56]

    Specificity : Does the prompt ask for a specific output ?

  49. [57]

    Domain Knowledge : Does the prompt cover one or more specific domains ?

  50. [58]

    Complexity : Does the prompt have multiple levels of reasoning , compositions , or variables ?

  51. [59]

    Problem - Solving : Does the prompt directly involve the AI to demonstrate active problem - solving skills ?

  52. [60]

    Creativity : Does the prompt involve a level of creativity in approaching the problem ?

  53. [61]

    Technical Accuracy : Does the prompt require technical accuracy in the response ? 23

  54. [62]

    Real - world Application : Does the prompt relate to real - world applications ? ## Rules

  55. [63]

    First analyze the prompt according to each aspect and then assign it with a score

    You should evaluate based on each aspects of the criteria independently . First analyze the prompt according to each aspect and then assign it with a score

  56. [64]

    Otherwise you should score it as 0

    If a prompt satisfies one aspect , you should score it as 1. Otherwise you should score it as 0

  57. [65]

    specificity

    Output your results with JSON dictionary format . ## Output Sample { " specificity ": {" analysis ": " analysis about specificity " , " score ": n } , " dom ain_know ledge ": {" analysis ": " analysis about domain knowledge " , " score ": n } , " complexity ": {" analysis ": "...

    2013

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.