Pith. sign in

REVIEW 3 major objections 5 minor 92 references

Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that a modular, model-agnostic pipeline can synthesize verifiable long-context datasets for SFT, DPO, GRPO, and evaluation, relieving the scarcity of high-quality long-context training data.

desk verdict A coherent engineering writeup with no experiments and circular 'verifiable' grounding; not a research contribution as it stands. read the letter →

arxiv 2509.01185 v2 pith:Z4POVTN3 submitted 2025-09-01 cs.CL cs.AI

Seganrasan Subramanian , Abhigya Verma This is my paper
classification cs.CLcs.AI
keywords syntheticdatagenerationlong-contextlanguagemodelsprompt-baseddirectpreferenceoptimizationgrouprelativepolicyLLMevaluationverifiableinstruction-responsedata-centricdevelopment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that the shortage of high-quality long-context data for large language models can be met by a modular synthetic-data framework that does not depend on real documents. The pipeline combines multi-field templated prompts, recursive scenario enrichment, metadata conditioning, and a two-layer validation stack of deterministic rule checks plus a separate LLM-based judge. It targets four output paradigms—multi-turn chat, document-grounded instruction-response triplets, schema-constrained verifiable tasks, and chain-of-thought reasoning—and claims each maps onto a post-training objective such as SFT, DPO, GRPO, reward modeling, or evaluation. Building on prior recipes that used synthetic data for long-context alignment, the paper argues that practitioners gain scalable, controllable, purpose-aligned dataset creation without manual curation, which is the stated bottleneck for advancing long-context capabilities.

What carries the argument

The carrying mechanism is the document-first generation loop. A multi-field prompt template (business scenario, text generation guidance, country, tone) seeds a scenario; a complexification step layers in stakeholders, regulatory constraints, and edge cases; and the generate_user_content template synthesizes a long document that becomes the ground truth context for all downstream tasks. Verifiability rests on two cooperating components: schema-constrained outputs validated by deterministic rule checks, and an LLM-based judge, kept separate from the generator, that emits structured scores along eight judgment axes. The same loop is instantiated four ways—chat, document-grounded, verifiable in

What would settle it

Run the pipeline on a scenario whose synthetic document contains invented statistics, fake citations, and wrong dates, then inspect the final dataset: if the rule-based validators and the LLM judge accept responses that faithfully repeat the fabrications, the claimed verifiability is settled to mean internal consistency only. A second test: generate many samples from one scenario with noise injection enabled and measure pairwise n-gram overlap; strong convergence toward a shared template would falsify the diversity claim.

Watch

Extended reading notes

Core claim

On its own terms, the paper's proposal is that verifiability in synthetic long-context data can be engineered rather than inherited. The pipeline first synthesizes a long document from a complexified scenario and then treats that document as the sole source of truth: an instruction is derived from it, a response is generated against it, and the response is checked with rule-based validators (JSON parseability, type and word-count limits, exact-match entity checks) and an LLM-based judge scoring along eight axes, from factual grounding and instruction compliance to tone fidelity and safety. This same loop is parameterized into four generators—multi-turn chat, document-grounded pairs, verifiab

Load-bearing premise

The framework's verifiability means a response agrees with a synthetic document the generator wrote itself, not with external reality, so factual errors baked into that document—which the authors concede can happen with weaker base models—pass every validation check.

Editorial extensions

If this is right

  • Long-context alignment data for SFT, DPO, and GRPO can be produced by configuring templates rather than curating corpora, cutting the manual cost that bottlenecks long-context model development.
  • Schema-constrained JSON outputs make rewards machine-assignable, so preference and policy-gradient training signals can be computed automatically and at scale.
  • A model-agnostic design means the same templates serve any base model or domain; adapting to a new task becomes a change of prompt and metadata rather than a new dataset effort.
  • Controlled noise, tone conditioning, and geographic grounding push outputs away from template collapse and geographic bias, helping both training robustness and evaluation fairness.
  • Judge scores double as supervision signals, usable beyond filtering for reward modeling and hallucination detection in long-context settings.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because validation checks responses against the synthetic document rather than the world, the framework could double as a probe for context-faithfulness: inject contradictions across document sections and test whether 'grounded' responses catch them.
  • The schema-derivation prompts effectively convert any instruction-response pair into a machine-checkable reward function, so the technique could transfer to reward-model training for open-ended generation tasks.
  • The multi-assistant handoff toggles (chat_awareness, solution_status) define a controllable experiment for measuring how much history long-context models actually use: degrade the receiving assistant's memory and quantify the performance drop.
  • Generator and judge from the same model family may share blind spots; swapping in an independently trained judge of a different family would reveal how much of the reported verifiability depends on the evaluator's own preferences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper describes a modular framework for generating synthetic long-context data for LLM training and evaluation. It proposes four generation paradigms (multi-turn chat, document-grounded tasks, verifiable instruction-schema tasks, and long-context reasoning), together with an LLM-based judge pipeline and a set of prompt templates. The authors frame their contribution around three research questions and claim that the framework supports SFT, DPO, GRPO, reward modeling, and evaluation. However, the manuscript contains no experiments, baselines, or quantitative evaluation; the only concrete artifact is a single illustrative JSON record in Listing 1. The conclusion states that 'our results demonstrate' design elements influence quality and downstream utility, but no results are presented.

Significance. If the framework were validated, it could offer a useful engineering contribution to the growing area of synthetic long-context data generation. The paper is clearly written and includes a number of thoughtful design considerations, such as geographic grounding, controlled noise injection, schema-based validation, and separation of generation from evaluation. These design choices are plausible and worth studying. However, the significance cannot be assessed from the current manuscript: no empirical evidence supports the claimed scalability, controllability, or downstream utility, and the verifiability guarantee is defined against internally generated documents rather than external ground truth. As it stands, the paper is a technical proposal rather than a demonstrated research contribution.

major comments (3)
  1. [§4 (Conclusion) and §1 (RQ1–RQ3)] The central claim is asserted, not demonstrated. The conclusion states that 'our results demonstrate that these design elements significantly influence both generation quality and downstream model utility' and that 'empirical analysis and illustrative examples show' fidelity and generalization, but the manuscript contains no experiments, baselines, ablations, or quantitative evaluation. The only supporting artifact is the single example in Listing 1, which has no comparison or analysis. RQ1–RQ3 are answered only by describing the framework. This is a load-bearing gap: without empirical evidence, the framework's claimed benefits for SFT/DPO/GRPO and evaluation are unsupported.
  2. [§3.2 Step 3 and §3.6 (Factual Grounding)] The verifiability guarantee is circular. Step 3 explicitly calls the LLM-generated source document 'ground truth context,' and §3.6 defines Factual Grounding as checking whether the response is 'entailed by input context (e.g., document, prior dialogue).' Thus the judge and rule-based checks can only establish internal consistency with a synthetic artifact, not factual correctness relative to the external world. If the generated document contains factual errors — as §5 concedes is possible for weaker models — the validation loop will pass them. The abstract's promise of 'verifiable' datasets is therefore misleading unless verifiability is explicitly redefined as internal consistency, or additional external ground-truth checks are added.
  3. [§3.6 (LLM-Based Judge Pipeline)] The LLM judge is the terminal quality gate, but no evidence is provided for its accuracy, calibration, or agreement with human judgments. The deployment benefits claim that all generated samples are 'validated for factual correctness, coherence, tone alignment, and output structure' is an unsubstantiated guarantee. Since the judge is itself an LLM and is used without validation, the framework's quality control rests on an untested assumption. A concrete evaluation of judge reliability (e.g., correlation with human judgments, false-positive/false-negative rates) is needed before this can be claimed as a verification mechanism.
minor comments (5)
  1. [Appendix references] The text repeatedly refers to 'Appendix 5' (e.g., §3.1 Step 1, Step 2, Step 3), but the appendix is not structured into numbered appendices. It contains Listings 1–14. Please update the cross-references to listing numbers.
  2. [Figure 3] The figure title 'End 2 End Pipeline' contains a typo; use 'End-to-End Pipeline.'
  3. [§3.5] The text states 'The Faker2 library is a widely used Python tool...' — presumably 'Faker' is intended. Also, the discussion of name-induced bias, while interesting, is not connected to any experimental evidence in this paper.
  4. [Prompt templates (Appendix)] Several prompt templates contain minor formatting and instruction errors: Listing 6 has two numbered item 7s, and Listing 7 says 'Dont use U.S cities like Austin, Texas, Denver in the response.' These are presentation issues but should be cleaned up.
  5. [References] Reference [9] has an incomplete arXiv identifier ('arXiv:2025'), [8] is a blog post without author or stable identifier, and [11] is a course rather than a primary research paper. Please provide complete and accurate references.

Circularity Check

1 steps flagged · score 4.0 of 10

Synthetic source documents are declared 'ground truth', so the judge's 'factual grounding' checks only self-consistency; the paper's utility claims are unsupported by experiments but not the main circularity.

  1. self definitional [Section 3.2, Step 3; Section 3.6, Table 2]
    "Given the enriched scenario, the generate_user_content template (Listing 7) synthesizes a document-scale passage (the “source document”) that serves as ground truth context. ... Factual Grounding: Verify that response content is entailed by input context (e.g., document, prior dialogue)"

    The pipeline first generates the source document, then labels it 'ground truth context.' The LLM judge's 'Factual Grounding' axis verifies only that responses are entailed by that same generated document, and rule-based validators check string/number matches against it. Thus the 'verifiability' of the dataset is defined relative to an artifact the pipeline itself produced. If the generated source document contains factual errors, the validation loop will still pass, as the paper's own limitation ('Weaker models may produce incoherent or factually incorrect samples') concedes. The claimed grounding therefore reduces to self-consistency with an internally generated referent, not to any external ground truth. This is a self-definitional loop: the truth criterion is the framework's own output.

full rationale

The paper is a framework/positioning paper with no empirical benchmark, so there are no fitted parameters or equations to reduce. The main circularity candidate is the internal ground-truth loop: Section 3.2 declares LLM-generated documents to be 'ground truth context,' and Section 3.6 defines factual grounding as entailment by that same context. This makes 'verifiability' a property of consistency with the pipeline's own output, not of correspondence to external facts. That is a genuine definitional circularity, though it is partial: the framework could still produce useful internal-consistency training data, and the paper explicitly acknowledges model-dependency and the need for human oversight. No load-bearing self-citation chain or imported uniqueness theorem appears; the related-work citations are not used to justify the core design. The unsupported claim in the conclusion that 'our results demonstrate' significant downstream utility is a lack-of-evidence issue, not a circularity issue. Overall, the central 'verifiability' claim is weakened by construction, meriting a moderate circularity score of 4 rather than a higher score reserved for fully fit-derived results or self-citation-forced conclusions.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The work introduces no new physical or formal entities. It relies on assumptions about the reliability of LLM-generated documents and LLM-based judges, and on the benign effect of controlled noise. No numbers are fitted to data: the framework's configurable variables (N, K, n, token budgets, word limits) are user-selected controls, not fitted parameters.

assumptions (4)
  • domain assumption LLM-generated synthetic documents are treated as the authoritative ground truth for task grounding and verifiability.
    Section 3.2 Step 3 says the synthesized document 'serves as ground truth context,' but the document is itself produced by an LLM, so grounding is not independently verifiable.
  • domain assumption The LLM-based judge reliably evaluates factual grounding, coherence, and compliance.
    Section 3.6 uses a separate LLM as a judge and assumes its scores are valid indicators of quality, without calibration against human judgments or external benchmarks.
  • domain assumption Prompt-injected controlled noise (synonym replacement, restructuring) preserves semantic fidelity while increasing diversity.
    Sections 3.1 and 3.5 assume these transformations do not degrade content quality; no experiments test this.
  • domain assumption The framework is model-agnostic and scales across base LLMs and domains.
    Stated in the abstract and Section 3, but only one example with QwQ-32b is shown, and no cross-model evaluation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation." pith.science (2026). https://pith.science/paper/Z4POVTN3

@misc{pith2026250901185,
  author       = {Pith},
  title        = {Pith review of: Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z4POVTN3}},
  note         = {Machine review of arXiv:2509.01185}
}
read the original abstract

The ability of large language models (LLMs) to process and reason over long textual inputs is critical for a wide range of real-world applications. However, progress in this area is significantly constrained by the absence of high-quality, diverse, and verifiable long-context datasets suitable for both training and evaluation. This work introduces a modular, extensible framework for synthetic long-context data generation via prompt-based interaction with LLMs. The framework supports multiple training and alignment objectives, including Supervised Fine-Tuning (SFT), Direct Preference Optimization (DPO), and Group Relative Policy Optimization (GRPO). It encompasses four core generation paradigms: multi-turn conversational dialogues, document-grounded input-output pairs, verifiable instruction-response tasks, and long-context reasoning examples. Through templated prompting, a model-agnostic architecture, and metadata-enriched outputs, the proposed approach facilitates scalable, controllable, and purpose-aligned dataset creation for advancing long-context capabilities in LLMs.

Figures

Figures reproduced from arXiv: 2509.01185 by the authors.

Figure 1
Figure 1. Overview of the long-context data generation framework. Prompt templates and metadata are used to guide [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The LLM lifecycle highlighting the centrality of data generation in downstream training quality. Our [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. End-to-end pipeline for long-context data generation. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Recursive multi-turn conversation generation. A scenario is sampled and initialized to produce an initial con [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

92 extracted references · 73 canonical work pages

  1. [1]

    NeurIPS (2022)

    Ouyang, L., et al.: Training language models to follow instructions with human feedback. NeurIPS (2022)

    2022

  2. [2]

    arXiv preprint arXiv:1909.08593 (2019)

    Ziegler, D., et al.: Fine-Tuning Language Models from Human Preferences. arXiv preprint arXiv:1909.08593 (2019)

    arXiv 1909

  3. [3]

    ICLR (2023)

    Rafailov, R., et al.: Direct Preference Optimization: Your Language Model is Secretly a Reward Model. ICLR (2023)

    2023

  4. [4]

    OpenAI Cookbook, https://github.com/openai/openai-cookbook

  5. [5]

    Li et al., WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale, arXiv:2502.16684

    arXiv

  6. [6]

    Bai et al., LongAlign: A Recipe for Long Context Alignment of Large Language Models, Findings of EMNLP 2024

    2024

  7. [7]

    Wang et al., Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models, arXiv:2402.02244

    arXiv

  8. [8]

    Synthetic Data Generation for Contexts Up to 1 Million Tokens Using Short-Context Models, Gradient AI Blog, 2025

    2025

Show all 92 references
  1. [9]

    Few-shot_LLM_Synthetic_Data_with_Distribution_Matching, arXiv:2025

    2025

  2. [10]

    et al.: SoftSRV prompting for synthetic data generation, arXiv:2410.16534

    DeSalvo, G. et al.: SoftSRV prompting for synthetic data generation, arXiv:2410.16534

    arXiv

  3. [11]

    HuggingFace LLM Course: Introduction to Reinforcement Learning and its Role in LLMs (2024)

    2024

  4. [12]

    LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

    Bai, Yushi, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. arXiv preprint arXiv:2308.14508, 2023

    2023 arXiv

  5. [13]

    S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model

    Lei, Wenqiang, et al. S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Model. Proceedings of NAACL, 2024

    2024

  6. [14]

    LONGATTN: Selecting Long-Context Training Data via Token-Level Attention

    Wu, Yifan, et al. LONGATTN: Selecting Long-Context Training Data via Token-Level Attention. arXiv preprint arXiv:2502.16860, 2024

    2024 arXiv

  7. [15]

    Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

    Wang, X., et al. Loong: Benchmarking Long-Context LLMs with Extended Multi-Doc QA. EMNLP 2024

    2024

  8. [16]

    LongSkywork: A Training Recipe for Efficiently Extending Context Length of Language Models

    LongSkywork Team. LongSkywork: A Training Recipe for Efficiently Extending Context Length of Language Models. arXiv preprint arXiv:2406.00605, 2023

    2023 arXiv

  9. [17]

    LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization

    Chen, G., et al. LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization. arXiv preprint arXiv:2502.13922, 2025

    2025 arXiv

  10. [18]

    AgoraBench: Evaluating Language Models as Synthetic Data Generators

    Kim, S., et al. AgoraBench: Evaluating Language Models as Synthetic Data Generators. arXiv preprint arXiv:2412.03679, 2024

    2024 arXiv

  11. [19]

    Luo et al.: Red-Teaming for Inducing Societal Bias in Large Language Models (2024)

    2024

  12. [20]

    Vijayaraghavan et al.: DECASTE: Unveiling Caste Stereotypes in Large Language Models through Multi- Dimensional Bias Analysis (2025)

    2025

  13. [21]

    id": "4c178f2359e81d042484d98c617bd43107a81cefb65fa8799ef67b1da4452ecf

    Salinas et al.: What’s in a Name? Auditing Large Language Models for Race and Gender Bias (2025) Appendix: Implementation Notes Sample Verifiable Long-Context Chat Record The following example illustrates a document-grounded, instruction-following conversation generated using ...

    2025

  14. [25]

    Complex Scenario

    **Mild Redundancy** – Introduce an occasional **extra descriptive phrase** or clarification to add variation. Strictly follow these instructions: - Ensure the generated scenario is contextually relevant, detailed, and realistic. - The scenario must take place outside the U.S. ...

    2025

  15. [29]

    ### **Language and Formatting Rules:** - The output **must be in English only**

    **Mild Redundancy** – Introduce an occasional **extra descriptive phrase** or clarification to add variation. ### **Language and Formatting Rules:** - The output **must be in English only**. - **Do not include any non-English words, phrases, characters, or scripts.** - **Stric...

    2025

  16. [30]

    The user first interacts with assistant-1 (L1 support)

  17. [31]

    After a few exchanges, assistant-1 hands off the conversation to assistant-2 (L2 support)

  18. [32]

    When the handoff happens: * The user may re-explain the issue OR * Assistant-2 may continue seamlessly, assuming they have chat history

  19. [33]

    Naming Convention: The first name provided is always the user

    The issue may either be resolved or remain unsolved after assistant-2’s response. Naming Convention: The first name provided is always the user. The second name provided is always the assistant-1. The third name provided is always the assistant-2. Parameters:

  20. [34]

    Country : {{country}}

  21. [35]

    Scenario: {{conversation_scenario}}

  22. [36]

    User Tone: {{user_tone}} (may include unorganized information, confused, abusive words, spelling mistakes, informal language, and noise such as emails, URLs, or irrelevant text),→

  23. [37]

    The assistant should ask only necessary and precise questions when seeking information.,→

    Assistant Tone: Always formal, polite, and patient. The assistant should ask only necessary and precise questions when seeking information.,→

  24. [38]

    Assistant-2 awareness of chat : {{chat_awareness}} (True: Assistant-2 has full context; False - The user must re-explain the issue),→

  25. [39]

    Assistant-2 solved the issue :{{solution_status}} (True - Assistant-2 provides a convincing solution; False - The issue remains unresolved.),→ Realism Instructions:

  26. [40]

    The dialogue should feel completely natural and human-like—no one should suspect it's machine-generated

  27. [41]

    Use realistic phrasing, contractions, and informal structures where appropriate

  28. [42]

    The user should not sound robotic—they may hesitate, backtrack, or provide unnecessary details

  29. [43]

    The user can use abusive or threatening words to force the assistant to get what he/she wanted

  30. [44]

    The assistant should be professional but sound human, not overly scripted

  31. [45]

    um," "you know

    Add minor pauses, filler words (e.g., "um," "you know"), and corrections where necessary for authenticity

  32. [46]

    Ensure the flow of conversation makes sense—responses should be logical and adaptive

  33. [47]

    **Each generated conversation must have at least one unique element** (e.g., misunderstanding, humor, unexpected turn)

  34. [48]

    {{user_name}}

    **Vary the user's style across generations**—sometimes clear, sometimes disorganized, sometimes emotional. Output Format (Only the conversation, nothing else): "{{user_name}}": "<user conversation>", "{{assistant_1_name}}": "<assistant-1 conversation>" "{{user_name}}": "<user ...

    2025

  35. [52]

    Example of a Complex Instruction: "Summarize the given conversation from both the user’s and the assistant’s perspectives

    The instruction must explicitly ask for the output in a proper JSON format. Example of a Complex Instruction: "Summarize the given conversation from both the user’s and the assistant’s perspectives. The user’s summary should be exactly 50 words, while the assistant’s summary s...

  36. [54]

    It may include word limits, format constraints, or multiple perspectives when applicable

  37. [57]

    Do not include any explanation, metadata, or additional text

    Output only the generated instruction. Do not include any explanation, metadata, or additional text. Listing 5: Sample prompt for Creating Conversation Instruction You are an advanced AI capable of processing complex instructions with high accuracy. Given a conversation and a ...

  38. [64]

    If the user conversation contain abusive or inappropriate words, do not use them in the response

  39. [65]

    No cross-lingual languages particularly Chinese or Japanese are not allowed

    Output must be in English. No cross-lingual languages particularly Chinese or Japanese are not allowed. Listing 6: Sample prompt for Creating Conversation Response 20 A PREPRINT - S EPTEMBER 25, 2025 {{final_scenario}} #### **Country:** {{country}} Strictly follow these instru...

    2025

  40. [66]

    **Synonym Substitutions** – Replace at least **five key words** with appropriate synonyms while preserving meaning

  41. [67]

    **Sentence Restructuring** – Modify the structure of at least **two sentences** while keeping intent intact

  42. [68]

    **Reordering Phrases** – Slightly alter the order of key phrases without changing the scenario’s meaning

  43. [69]

    **Mild Redundancy** – Introduce an occasional **extra descriptive phrase** or clarification to add variation. ### Additional Style Variation: Each time, apply one of the following subtle stylistic variations: - A slightly **formal tone** - A **conversational** and engaging ton...

  44. [70]

    The instruction should be challenging and require multiple constraints, such as different word limits, perspectives, or formatting styles.,→

  45. [71]

    The task should be logically complex, requiring the AI to process information in a structured manner

  46. [72]

    The instruction should not be generic; it must demand deep analysis and precise formatting

  47. [73]

    Example of a Complex Instruction : "Summarize the given conversation from both the user’s and the assistant’s perspectives

    The instruction must explicitly ask for the output in a proper JSON format. Example of a Complex Instruction : "Summarize the given conversation from both the user’s and the assistant’s perspectives. The user’s summary should be exactly 50 words, while the assistant’s summary ...

  48. [74]

    The generated instruction must be logically sound and highly detailed

  49. [75]

    It can include word limits, format constraints, or multiple perspectives when applicable

  50. [76]

    Ensure the instruction challenges the AI to produce a nuanced response

  51. [77]

    The instruction should not be generic; it should require deep analysis or structured output

  52. [78]

    Do not include any explanation, metadata, or additional text

    Output only the generated instruction. Do not include any explanation, metadata, or additional text. Listing 8: Sample prompt for Creating Long context Document Instruction 21 A PREPRINT - S EPTEMBER 25, 2025 You are an advanced AI capable of processing complex instructions wi...

    2025

  53. [79]

    Carefully analyze both the conversation and the instructions before generating a response

  54. [80]

    Ensure that every condition, constraint, and formatting rule mentioned in the instructions is fully met

  55. [81]

    If the instructions specify a particular format (e.g., JSON, XML, bullet points, etc.), the output must strictly follow it.,→

  56. [82]

    Maintain accuracy, coherence, and completeness in the generated response

  57. [83]

    Do not omit or alter any part of the instructions—ensure 100% compliance

  58. [84]

    Generate only the final response—do not include explanations, processing notes, or metadata

  59. [85]

    Output must be in English

  60. [86]

    ,→ ,→ ,→ ,→ ,→ ,→ ,→ Response JSON : {

    No cross-lingual languages particularly Chinese or Japanese are not allowed. Listing 9: Sample prompt for Creating Long context Document Response Given the following instructions and response JSON, generate a JSON schema that defines the structure of the response. The schema s...

    2025

  61. [87]

    is_metadata

    Add "is_metadata": true

  62. [88]

    type": "STRING

    Set "type": "STRING"

  63. [89]

    language

    Set "language": "en"

  64. [90]

    num_words

    Define "num_words" as a list with the lower and upper word limits extracted from the input JSON

  65. [91]

    For list fields:

    If only a lower or upper bound is specified, use 99999 as the max bound or 0 as the min bound accordingly. For list fields:

  66. [93]

    type": "LIST

    Define "type": "LIST"

  67. [94]

    For lists with nested dictionaries (item_type_details example):

    Convert list items into structured objects with corresponding metadata properties. For lists with nested dictionaries (item_type_details example):

  68. [95]

    is_metadata

    Maintain the "is_metadata": true property for the list itself

  69. [96]

    item_type

    Introduce "item_type" as a dictionary where keys represent nested dictionary fields

  70. [97]

    num_words

    Each nested dictionary field should follow metadata rules similar to string fields, defining "num_words" if applicable. For integer, float, boolean, and date fields:

  71. [98]

    is_metadata

    Set "is_metadata": true

  72. [99]

    Nested objects:

    Assign the correct "type" based on the available data types (INT, FLOAT, BOOL, or DATE). Nested objects:

  73. [100]

    Maintain hierarchy while ensuring each field has the appropriate metadata

  74. [101]

    is_metadata

    If an object is not inherently metadata, add "is_metadata": false. Example 1: input JSON structure : { "user_summary": "<string> <under 75 words>", "assistant_summary": "<string> <150-200 words>", "additional_details": { { "title": '<string> <10 words>' }, { "author": '<string...

    2025

  75. [102]

    Preserve the keys from the input JSON

  76. [103]

    Convert each key into a metadata object, indicating its type and constraints

  77. [104]

    is_metadata

    Specify "is_metadata": true for individual fields and "is_metadata": false for objects containing multiple properties

  78. [105]

    Include "type" to define whether the value is a STRING, LIST, INT, or DATE

  79. [106]

    language

    Specify "language": "en" for all STRING types

  80. [107]

    num_words

    Define "num_words": [min, max] for STRING fields with word count constraints

  81. [108]

    type": "LIST

    Represent lists with "type": "LIST" and define metadata for their items

  82. [109]

    type": "INT

    Ensure numerical fields such as item_rating are assigned "type": "INT"

  83. [110]

    type": "DATE

    Ensure date fields such as sell_date are assigned "type": "DATE"

  84. [111]

    Input JSON structure : {{input}} Output only JSON schema

    Handle nested structures correctly while preserving hierarchy. Input JSON structure : {{input}} Output only JSON schema. Do not output any other information. Listing 14: Sample prompt for Formatting Long context Conversation’s Verifiable Instruction-Schema - Part 4 26

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.