Pith. sign in

REVIEW 5 major objections 6 minor 160 references

UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following

T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-part constraint recipe—common constraints from two articles, revision of only the trivially satisfied ones, and scoring on the summary—closes the copy-and-paste loophole that lets LLMs fake instruction following.

desk verdict A useful new benchmark and method for constraint synthesis, with a real but fixable weakness in the summary-based 'core satisfaction' metric. read the letter →

arxiv 2608.09154 v1 pith:B34NL534 submitted 2026-08-10 cs.CL

Jeet Sharma , Balpreet Kaur , Jeremiah Hong , Hamed Zamani , Haw-Shiuan Chang This is my paper · ORCID
classification cs.CL
keywords instructionfollowingconstraintsynthesisback-translationcopy-and-pasteshortcutsuperficialsummarization-basedevaluationLLMbenchmarknaturalness
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Back-translation—generating test constraints by asking a model to extract them from a reference article—is undermined by a copy-and-paste shortcut: the generator copies specific text into a constraint, and the tested LLM copies it back out, earning credit without real compliance. UNSPECIFIC is the paper's attempt to close that loophole with three coordinated changes. Constraints are written against two similar reference articles at once, so they express general properties that no single article can leak; only the constraints a constraint-blind base article already satisfies are replaced with harder ones; and satisfaction is scored on the full response and again on its 25%-length summary, so a constraint that only appears in the details no longer counts. The paper shows this recipe produces harder and more natural constraints (GPT-5 Mini's satisfaction drops from about 90% to 78%, and human-judged naturalness against human-written constraints improves by 30 percentage points) and reveals that many satisfied constraints are followed only superficially.

What carries the argument

The load-bearing mechanism is the pairing of three operations that each close one escape hatch. Common-constraint synthesis (Section 2.1) presents the generator with two retrieved similar articles and demands constraints satisfied by both, removing the specific names, entities, and plot objects that enable copy-paste. Evaluation-based revision (Section 2.2) generates a base article blind to the constraints, runs the satisfaction judge on it, and hardens only the constraints the base article already satisfies—so roughly half the common constraints get replaced instead of all 39, preserving naturalness. Core satisfaction (Section 2.3) summarises the response with GPT-4.1 Mini to 25% of its length and re-runs the judge, defining a constraint as superficially satisfied if it passes on the full text but fails on the summary. The summary-length percentage is the tunable dial that sets how deep 'deep integration' must be.

What would settle it

Take the generated articles whose constraints pass on the full text but fail on the 25% summary, and have human annotators read the full articles and judge, constraint by constraint, whether each flipped constraint was superficial decoration or genuinely integrated into the narrative; if many flipped constraints look genuinely integrated, the metric overstates superficiality. A cheaper companion check is to re-run the benchmark with summaries at 50%, 25%, and 10% of original length: if model rankings shift with the ratio, the metric is partly measuring summary-length sensitivity rather than integration depth.

Watch

Extended reading notes

Core claim

The paper's central claim is that the copy-and-paste loophole is not an accident of individual models but a structural flaw of single-reference constraint synthesis, and that three design choices undo it. Common-constraint synthesis presents the generator with a reference article and a second similar article retrieved by sentence embeddings, requiring every constraint to be satisfied by both; proper nouns and unique plot objects disappear from the constraint list because they exist only in one article. Evaluation-based revision generates a base article from the main task alone, then replaces only the constraints that base article trivially satisfies—about 50–60% of common constraints versus 20–30% of single-article ones—with harder constraints, preserving the naturalness that full-list revision destroys. Core satisfaction summarises the response to 25% of its length with an LLM and re-checks every constraint, counting a constraint as superficially followed when it passes on the full article but fails on the summary. On this combination the paper reports benchmark results across news, story, and blog domains showing lower satisfaction rates, a 30-point improvement in naturalness relative to human-written constraints, reduced similarity between constraints and both references and responses, and wide differences in original-versus-summary satisfaction gaps across models—smaller GPT models and larger Claude and Llama models copying more.

Load-bearing premise

The core-satisfaction metric assumes that a 25%-length summary preserves the article's core narrative and strips away only peripheral detail, so a constraint that fails after summarisation must have been followed superficially—an assumption the paper never validates with human judges.

Editorial extensions

If this is right

  • Benchmarks built this way are harder without being weirder: satisfaction rates drop for every tested model while human-judged naturalness rises and coherence holds steady.
  • The original-versus-core gap separates shallow compliance from real integration: models that lean on copying show large gaps, and the paper shows that editing tasks and news-domain facts widen the gap.
  • Similarity-based copy detection is not a reliable reward: models with comparable constraint-response similarity can differ sharply in core satisfaction, so the summary metric is the usable signal.
  • Domain composition changes what a benchmark measures: only 28.4% of satisfied constraints in news survive summarisation versus 63.1% in blogs, so aggregate leaderboards can hide superficial following in fact-dense domains.
  • Model families differ in how extra parameters translate into instruction following: some families convert scale into genuine core satisfaction (the GPT series) while others spend it on surface copying (Claude and Llama families).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 25% summary length is a dial the authors did not turn: measuring core satisfaction at 50%, 25%, and 10% of original length would yield a depth profile of each model's integration, separating length sensitivity from true superficiality.
  • Common-constraint synthesis transfers beyond prose: generating requirements from two similar code snippets or two similar technical documents would push models toward behavioural specifications instead of regurgitated identifiers, and the article-similarity knob gives benchmark builders a continuous difficulty control.
  • If the summary metric holds up, it becomes a cheap training reward: fine-tuning to maximise satisfaction on summaries would pressure models to weave constraints into the narrative spine—the authors state explicitly that they have not tested this use.
  • The lack of human validation of the summarisation assumption is the point to probe next: comparing core-satisfaction verdicts against annotators reading full articles would tell whether the metric measures depth or merely summary sensitivity.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper targets the copy-and-paste shortcut in back-translated instruction-following benchmarks. It proposes UNSPECIFIC with three components: (i) generating constraints common to two similar reference articles rather than one, (ii) revising only constraints that a constraint-blind base article already satisfies, and (iii) measuring satisfaction after summarizing the generated article to 25% of its length, interpreting failures after summarization as superficial following. The authors construct benchmarks in news, story, and blog domains and report that UNSPECIFIC constraints are more challenging and natural than single-article back-translation, reduce constraint-reference similarity, and expose differences in superficial satisfaction across LLMs. They release code and data.

Significance. If validated, the framework offers a useful recipe for building constraint-satisfaction benchmarks that resist verbatim copying, which is a recognized weakness of back-translation. The paper's strengths include the multi-domain benchmark, multiple baseline comparisons, transparent prompts in the appendix, public release, and honest limitation statements. However, the central metric, core satisfaction after 25% summarization, lacks direct validation, and the naturalness claim rests on a small two-annotator study with low agreement. The difficulty increase is partly a selection effect of the revision procedure. These issues make the current evidence suggestive rather than conclusive, though the direction is promising and the required additional analyses appear feasible within the paper's scope.

major comments (5)
  1. [§2.3, Tables 6–11, Appendix A.1] The core satisfaction metric assumes that a 25% summary preserves all genuinely integrated constraints and that constraints lost in summarization were satisfied only superficially. Neither assumption is tested. Appendix A.1's short-article control addresses the length-truncation hypothesis for one model, but it does not validate the summarizer as a core-narrative extractor, nor does it rule out a superficial phrase surviving summarization. Moreover, §4.2's own explanation that news has more facts, which tend to live in details, suggests the metric may partly measure domain-specific summarization loss rather than superficiality. Please provide human validation: for a sample of constraints, have annotators judge whether the summary removed genuinely core content and whether constraints that survive or fail after summarization are correctly classified.
  2. [§2.2, Table 1] Because the Eval revision replaces exactly the constraints that the constraint-blind base article satisfies, the observed drop in Org and Core from Common+None to Common+Eval is in part the direct result of selecting harder-by-construction constraints. The paper should separate the selection effect from intrinsic difficulty, for example by reporting what fraction of the revised constraints would also be satisfied by a constraint-blind base article, or by comparing satisfaction on a held-out set of constraints matched for baseline satisfiability. Without this, the "more challenging" headline overstates the novelty of the difficulty gain.
  3. [§2.2 and Limitations] The revision step abandons the reference grounding that guaranteed the original constraint set was jointly satisfiable. After replacing "too easy" constraints with new ones, there is no check that the 39 constraints are simultaneously satisfiable; the paper itself acknowledges that contradictions may exist. If some instructions contain an unsatisfiable subset, the lower satisfaction rates reflect infeasibility, not required depth. Please report the proportion of instructions with detectable contradictions, for example by prompting a strong LLM to produce a satisfying article or by checking pairwise consistency, and either exclude or analyze those cases.
  4. [§3.1.3, Table 1] Naturalness is a headline claim, but it rests on two expert annotators with Cohen's kappa 0.310 and no confidence intervals, and only in the news domain. The abstract presents the naturalness result without the domain restriction. The study should be expanded with more annotators and per-condition agreement, and the win-rate numbers should be reported with uncertainty intervals. The claim should be restricted to news if story and blog naturalness are not human-evaluated.
  5. [§3.1 and §4.1] Only 25 instructions per domain are used, and the method depends on several free parameters: the 25% summary ratio, the retrieval similarity threshold of 0.85, the 39-constraint count, and the ~500-word article length. No sensitivity analysis is given for any of these, even though the summary ratio directly controls the core rate and the retrieval threshold controls constraint generality. Please add a compact sensitivity analysis around the default values, or provide a justification for why the conclusions are stable to those choices.
minor comments (6)
  1. [Abstract] The phrase "LLM win-rate gap improves by 30%" should say "the human-versus-LLM preference gap in the news domain improves by 30 percentage points"; the current phrasing is ambiguous.
  2. [§2.1] The retrieval criterion "highest similarity less than 0.85" does not specify the lower bound or whether the selected article is the top-1 below the threshold; please clarify the retrieval rule.
  3. [Tables 3, 6] Many comparative claims are made across differences that may be within the reported standard errors; the paper should state explicitly which differences are statistically meaningful or provide a consistent significance test.
  4. [Tables 14–16] The appendix examples of the copy-paste problem are very helpful; consider moving one into the main text to illustrate the problem earlier and make the motivation more concrete.
  5. [Throughout] There are minor formatting and terminology issues: Table 6 cells such as "Claude Sonnet 4.589.6" are missing spacing, and the embedding model is called "sentence BERT" in §2.1 but "all-mpnet-base-v2" in §3.1.1; unify the terminology.
  6. [§3.1.3 and Appendix D.1] The paper reports only pairwise preference results in Table 1; it should also report the 1–5 naturalness scores collected in the MTurk task, since those scores would provide a less coarse view of the naturalness comparison.

Circularity Check

3 steps flagged · score 4.0 of 10

Partial circularity: constraint-reference similarity drop is built into the synthesis prompt and the 'superficial' finding is definitional; final held-out benchmark retains independent content.

  1. self definitional [Section 2.1, Table 4, Appendix Template C.3]
    "we first retrieve another similar document B using sentence BERT (Reimers and Gurevych, 2019) and ask the LLM to generate the common constraints, all of which are satisfied by article A and by article B simultaneously. The procedure ensures that the synthesized constraint set could be satisfied by multiple articles and reduce the information leaking of individual article. ... Each constraint must be satisfied by BOTH Blog A and Blog B. ... Avoid proper nouns in your constraints."

    The prompt itself requires each constraint to hold for both documents and to avoid proper nouns, so constraints carrying single-article surface details are filtered out at synthesis time. The drop in constraint-to-reference similarity in Table 4 is therefore an input constraint of the method, not an independently measured effect. The paper presents this drop as support for reduced copying ('The Common also reliably reduces similarities between the reference article(s) and the constraints in Table 4'), which is circular support; the response-side similarity and core rates on held-out LLMs are the non-circular part.

  2. fitted input called prediction [Section 2.2, Table 12, Appendix Template C.12]
    "To identify the redundant constraints, we first ask GPT-4.1 Mini to generate a base article just according to only the main task without showing any constraints. Next, we ask GPT-4.1 Mini to replace the constraints that are satisfied by this base article with harder ones ... replace the constraints that are already satisfied (marked 'Yes') with NEW, HARDER constraints that are NOT satisfied by the base content."

    The 'too easy' set is defined by satisfaction of the constraint-agnostic base article, and the replacement prompt requires new constraints to be unsatisfied by that same base article. Consequently, the Eval condition is constructed to remove constraints that at least one model satisfies without seeing them; the reported drop in Org/Core (e.g., 95.1 to 78.2 for GPT-5 Mini) partly follows from this selection. Since final difficulty is measured on different LLMs, the result is not fully forced, but the direction of the difficulty increase is baked into the revision criterion.

1 more flagged steps
  1. self definitional [Section 2.3, Abstract, Contribution list]
    "To examine whether the LLMs follow the constraints using its core narrative, we use GPT-4.1 Mini to summarize the response before evaluating the constraint satisfaction rate. ... We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article)."

    The metric equates the unobservable 'core narrative' with whatever a 25% summary retains and labels a constraint 'superficially satisfied' exactly when it passes on the full response but fails on the summary. The headline 'large portion of constraints are satisfied superficially' is a restatement of this definition (Org minus Core), not an independent discovery. The short-article control in Appendix A.1 only rules out length truncation; it does not validate that the summary preserves all genuinely core constraint-relevant content, so the definitional equivalence carries an untested assumption.

full rationale

UNSPECIFIC contains three interlocking components, and two of them have by-construction elements that the paper reports as empirical support. First, 'common constraints' are generated under a prompt that requires each constraint to be satisfied by both documents and forbids proper nouns, so the reduced similarity between the constraints and any single reference article (Table 4) is a prompt property, not an independent finding about copy-paste reduction. Second, the evaluation-based revision selects constraints for replacement exactly when a constraint-agnostic base article satisfies them, and the replacement prompt requires the new constraints not to be satisfied by that base article; the direction of the resulting satisfaction drop is therefore partly forced, although the final measurements on GPT-5 Mini and Llama-3 8B are held-out and give independent evidence about the magnitude. Third, the core-satisfaction metric defines superficial following as failure after 25% summarization, so the 'large portion of constraints are satisfied superficially' headline is a restatement of the operationalization; Appendix A.1's short-article control addresses length but not the assumption that summaries preserve all core constraint-relevant content. These are partial circularities. They do not make the benchmark empty: Tables 6-11 measure held-out LLMs with independent judges, the naturalness results come from human comparisons, and no load-bearing argument reduces to a self-citation. Score 4 reflects substantial by-construction support for some claims while the central benchmark retains independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central results depend on proxy assumptions that are not calibrated to human ground truth: base-article satisfaction implies triviality, summary satisfaction implies deep integration, and LLM judges plus sentence-BERT similarity measure satisfaction and copying. These proxies are the main source of uncertainty in the paper's conclusions.

free parameters (4)
  • Summary length ratio for core satisfaction metric = 25% of original length
    Chosen by the authors as the default depth of summarization; the paper says the percentage could be tuned, meaning it is a hand-set hyperparameter of the new metric.
  • Similarity threshold for retrieving the second article = 0.85 maximum similarity
    Used in Section 2.1 to exclude near-duplicate article pairs; no analysis of sensitivity to this threshold is provided.
  • Number of constraints per instruction = 39
    Taken from Atmakuru et al. (2024) rather than derived; affects difficulty and cost.
  • Article length = about 500 words
    Generation and editing tasks target about 500 words; used throughout the benchmark.
assumptions (4)
  • domain assumption A constraint that is satisfied by a base article generated without seeing the constraints is trivially satisfiable and a good candidate for replacement with a harder constraint.
    Core to Section 2.2 Evaluation-based Constraint Revision; if the base article satisfies a constraint due to genre conventions rather than simplicity, the revision may remove natural constraints.
  • domain assumption Satisfaction of a constraint after summarizing the response to 25% of its length indicates the constraint is integrated into the core narrative; details lost in summarization are superficial.
    Defines the new core satisfaction metric in Section 2.3; no human validation of the summary's fidelity is provided.
  • domain assumption LLM judges (Claude and GPT variants) can reliably determine whether a constraint is satisfied by an article or a summary.
    Used throughout Sections 3 and 4; the paper compares three judges but does not measure judge accuracy against human ground truth for constraint satisfaction.
  • domain assumption Sentence-BERT similarity between a constraint and the closest sentence in a response or reference measures copy-and-paste behavior.
    Used in Section 3.1.2 and Tables 4 and 5; the paper itself notes similarity metrics are unreliable and affected by text style in Section 3.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following." pith.science (2026). https://pith.science/paper/B34NL534

@misc{pith2026260809154,
  author       = {Pith},
  title        = {Pith review of: UNSPECIFIC: General Constraint Synthesis for Breaking Copy-and-Paste Shortcut in LLM Instruction Following},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B34NL534}},
  note         = {Machine review of arXiv:2608.09154}
}
read the original abstract

Large language models (LLMs) are increasingly expected to follow long lists of constraints in complex instructions, and synthesizing instructions from a reference document (i.e., back-translation) is a widely used method to measure/enhance LLMs' ability to follow complex instructions. However, this method introduces a critical loophole: the constraint synthesis model copies text from the reference as a very specific constraint and the evaluated LLM trivially satisfies the constraint by copying its text in the response. To address these issues, we propose UNSPECIFIC, a novel framework that synthesizes constraints common to two similar reference articles to reduce copy-pasting, selectively hardens only trivially satisfied constraints to balance difficulty and naturalness, and evaluates satisfaction on both the generated article and its summary to penalize superficial instruction following. Consequently, we built the UNSPECIFIC benchmark on news, story, and blog domains to analyze the copy-pasting behavior of LLMs. Our results show that our synthesized constraints are not only more challenging (e.g., the satisfaction rate of GPT-5 Mini drops from 90% to 78%) and natural (LLM win-rate gap improves by 30%) from a human perspective but also mitigate the copy-pasting. We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article). The code and datasets are released at https://github.com/JeetDSharma/UNSPECIFIC.

Figures

Figures reproduced from arXiv: 2608.09154 by the authors.

Figure 2
Figure 2. Comparison of our three main novel components in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. MTurk task template: user-written constraint [PITH_FULL_IMAGE:figures/full_fig_p026_3.png] view at source ↗
Figure 4
Figure 4. MTurk task template: naturalness & prefer [PITH_FULL_IMAGE:figures/full_fig_p027_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

160 extracted references · 78 canonical work pages

  1. [1]

    The main task is to write a blog about strategies for successful remote working

    Identify the main task of the blog in one sentence. - For example: "The main task is to write a blog about strategies for successful remote working." - Phrase the main task as an instruction

  2. [2]

    The blog should include practical tips

    Generate a set of 39 free-form constraints that you think might have been given to the LLM to generate the blog. - DO NOT REPEAT CONSTRAINTS. - Constraints must be atomic (a single indivisible condition). If a constraint can be broken into smaller constraints, do so. - Avoid proper nouns in your constraints. - Constraints should drive at least a few sente...

  3. [3]

    arXiv preprint

    Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research. arXiv preprint. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao- hua Dong, Ran Lin, and Ruohui Huang. 2024. Conifer: Improving complex constrained instruction- following ability of large language models.arXiv preprint arXiv:2404.02823. Minzheng Wang, Longze Chen, Fu...

    arXiv 2024

  4. [4]

    Explain how the removal of commuting affects time use and daily rhythm

  5. [5]

    Suggest strategies for maintaining healthy eating while at home

  6. [6]

    Emphasize reducing environmental distractions in that space

  7. [7]

    Require the setting of defined working hours

  8. [8]

    Explain the risks of isolation if connection practices are neglected

Show all 160 references
  1. [9]

    Warn about the risk of burnout without intentional self-care

  2. [10]

    Stress the value of adhering consistently to those hours

  3. [11]

    Show how workspace ergonomics ( chair, desk) influence long-term health

  4. [12]

    Conclude with a call to action urging readers to adopt concrete changes immediately

  5. [13]

    Argue for the necessity of regular breaks during the workday

  6. [14]

    Encourage informal online gatherings to maintain rapport

  7. [15]

    Recommend mindfulness or meditation as stress-management tools

  8. [16]

    Explain how balanced nutrition influences concentration and resilience

  9. [17]

    Emphasize communicating work schedules to others in the household

  10. [18]

    Identify productivity as a central theme in remote work discussions

  11. [19]

    Link exercise directly to improved cognitive performance and focus

  12. [20]

    Establish remote work as a global trend that has transformed professional life

  13. [21]

    Connect emotional well-being to overall job performance and satisfaction

  14. [22]

    Recommend light physical movement or stretching during pauses

  15. [23]

    Integrate workspace, scheduling, health, and social practices into a unified remote-work strategy

  16. [24]

    Stress the importance of sleep in sustaining energy and productivity

  17. [25]

    Recommend scheduled video calls to replicate face-to-face connection

  18. [26]

    Recommend creating a physically separate space for work at home

  19. [27]

    Stress the need for essential tools and equipment to be easily accessible

  20. [28]

    Introduce one structured time- management method, such as work intervals

  21. [29]

    Show how mental health practices support long-term work sustainability

  22. [30]

    Show how shared rituals (e.g., virtual coffee breaks) strengthen belonging

  23. [31]

    Highlight the role of hydration and snacks in sustaining energy across breaks

  24. [32]

    Suggest environmental cues (like decor or layout) that reinforce the sense of a work zone

  25. [33]

    Describe the importance of adequate lighting for focus and energy

  26. [34]

    Suggest instant messaging as a tool for quick, ongoing collaboration

  27. [35]

    Highlight how a clear boundary between workspace and leisure areas aids focus

  28. [36]

    Show how enforcing those boundaries prevents interruptions

  29. [37]

    Highlight well-being as equally important alongside productivity

  30. [38]

    Warn about the risk of personal time erosion without such boundaries

  31. [39]

    Contrast the flexibility of remote work with the new challenges it creates

  32. [43]

    Recommend establishing a routine for daily physical exercise

  33. [44]

    Explain how breaks counteract mental fatigue and sustain performance

  34. [45]

    Now use the same approach for the next input blog

    Stress that remote work requires deliberate maintenance of social contact. Now use the same approach for the next input blog. Template C.2. Constraint Generation - News You are a writing expert. I am going to give you a news article as an input. You can assume that a large lan...

  35. [46]

    The main task is to write a news article about 43rd Annual Greek Experience Festival June 7-9 In Danbury

    Identify the main task of the news article in one sentence. - For example: "The main task is to write a news article about 43rd Annual Greek Experience Festival June 7-9 In Danbury." - Phrase the main task as an instruction

  36. [47]

    The story should center on a disappearance

    Generate a set of 39 free-form constraints that you think might have been given to the LLM to generate the news article. - DO NOT REPEAT CONSTRAINTS. - Constraints must be atomic (a single indivisible condition). If a constraint can be broken into smaller constraints, do so. -...

  37. [48]

    Food and Drink, Live Music, & Folk Dancing! Come enjoy our warm hospitality, learn more about our beautiful Byzantine Church, and experience our Orthodox Christian traditions

    <constraint 39> Here is a worked example to guide you: Input: Share event 43rd Annual Greek Festival Assumption Greek Orthodox Church in Danbury will host its 43rd Annual Greek Festival celebrating the rich culture of our Greek heritage June 7, 8 and 9 on the Church grounds - ...

  38. [49]

    The story should center on an upcoming cultural celebration

  39. [50]

    The event should be hosted by a community or religious organization

  40. [51]

    The organization should have a long- standing tradition of holding this event annually

  41. [52]

    The event should celebrate a specific cultural or ethnic heritage

  42. [53]

    The story should indicate that the celebration spans multiple consecutive days

  43. [54]

    The event's location should be mentioned with reference to a recognizable area or landmark

  44. [55]

    The story should state that the event is held on the organization's premises

  45. [56]

    The festival should promote awareness of traditional customs and community values

  46. [57]

    The story should highlight the preservation of cultural identity as a theme

  47. [58]

    The event should emphasize the warm hospitality of the hosting community

  48. [59]

    The narrative should invite the public to experience the featured culture firsthand

  49. [60]

    Religious or spiritual elements should be included to connect the event to its heritage

  50. [61]

    The article should present the event as both entertaining and educational

  51. [62]

    The story should mention a variety of attractions or activities

  52. [63]

    Traditional food and drink should be featured prominently

  53. [64]

    The festival should include live performances for entertainment

  54. [65]

    Folk dancing should be identified as a key cultural activity

  55. [66]

    Authentic costumes should be highlighted as part of the performances

  56. [67]

    Live music should be described as enhancing the festive atmosphere

  57. [68]

    Artisan crafts should be included as part of the offerings

  58. [69]

    Visitors should have opportunities to learn about cultural traditions

  59. [70]

    The tone should convey warmth, joy, and community spirit

  60. [71]

    The story should encourage public participation and inclusivity

  61. [72]

    The article should express pride in the cultural heritage being celebrated

  62. [73]

    The narrative should appeal to the senses through vivid descriptions of the experience

  63. [74]

    The story should state that admission is free

  64. [75]

    Practical details for attendees should be provided

  65. [76]

    Directions or reference points should be given for ease of navigation

  66. [77]

    The article should mention additional parking or transportation options

  67. [78]

    Accessibility and convenience should be emphasized to attract attendees

  68. [79]

    The story should begin with a clear announcement of the event and its host

  69. [80]

    A section should describe the cultural or historical significance of the event

  70. [81]

    The middle part should list the main attractions and entertainment

  71. [82]

    The closing section should include logistical information for visitors

  72. [83]

    The tone throughout should remain promotional and welcoming

  73. [84]

    The structure should progress from general overview to detailed information

  74. [85]

    The story should use present or future tense for immediacy

  75. [86]

    The language should be simple, lively, and community-oriented

  76. [87]

    Now use the same approach for the next input news article

    The narrative should maintain an informative tone typical of local event announcements. Now use the same approach for the next input news article. Template C.3. Common Constraint Generation - Blog You are a writing expert. You will be given two blogs (Blog A and Blog B) as inp...

  77. [88]

    Write a blog about strategies for successful remote working

    Identify a common main task that applies to BOTH blogs. - The main task must be implied by BOTH Blog A and Blog B - every detail in the task must be directly present in both blogs. - The more similar the blogs, the more specific the main task can be. - The more dissimilar the ...

  78. [89]

    Include a conclusion

    Generate a set of 39 free-form constraints that apply to BOTH blogs. - Each constraint must be satisfied by BOTH Blog A and Blog B. - DO NOT REPEAT CONSTRAINTS. - Constraints must be atomic (a single indivisible condition). If a constraint can be broken into smaller constraint...

  79. [90]

    End with a summary or concluding thought

  80. [91]

    Open with an introduction that establishes the topic

  81. [92]

    Include specific examples to illustrate points

  82. [93]

    Maintain a clear and organized structure

  83. [94]

    Use descriptive language to engage the reader. ... (remaining constraints would be similarly general) Example 2 (Very Similar Blogs): Blog A: A blog about remote work productivity, discussing workspace setup, time management with Pomodoro, and work-life boundaries. Blog B: A b...

  84. [95]

    Conclude with a call to action urging readers to implement changes

  85. [96]

    Establish remote work as a significant trend in professional life

  86. [97]

    Recommend creating a dedicated workspace at home

  87. [98]

    Emphasize reducing distractions in the work environment

  88. [99]

    Argue for the importance of regular breaks during work

  89. [100]

    Recommend a structured time- management approach

  90. [101]

    Stress setting clear boundaries between work and personal time

  91. [102]

    Suggest communicating work schedules to household members. ... (remaining constraints would be similarly specific) Now apply this approach to the following two blogs: Blog A: {blog1} Blog B: {blog2} Output Format: Main Task: [one sentence instruction] Constraints:

  92. [103]

    Common Constraint Generation - News You are a writing expert

    [constraint] Template C.4. Common Constraint Generation - News You are a writing expert. You will be given two news articles (Article A and Article B) as input. You can assume that a large language model (LLM) generated each news article. Your task has two parts:

  93. [104]

    Report on recent developments related to changes in workplace policies

    Identify a common main task that applies to BOTH news articles. - The main task must be implied by BOTH news article A and news article B - every detail in the task must be directly present in both the news articles. - The more similar the news articles, the more specific the ...

  94. [105]

    Include a conclusion

    Generate a set of 39 free-form constraints that apply to BOTH articles. - Each constraint must be satisfied by article A and article B. - DO NOT REPEAT CONSTRAINTS. - Constraints must be atomic (a single indivisible condition). If a constraint can be broken into smaller constr...

  95. [106]

    End with information that looks ahead to future developments

  96. [107]

    Open with a clear lead summarizing the core issue

  97. [108]

    Provide factual details supported by evidence

  98. [109]

    Maintain a neutral and objective reporting tone

  99. [110]

    Include contextual background to help readers understand the issue. ... (remaining constraints would be similarly general) Example 2 (Very Similar Articles): Article A: A news article reporting on the rise of remote work, covering recent survey findings, employer policy change...

  100. [111]

    Conclude by discussing how remote work policies may evolve in the near future

  101. [112]

    Open with a lead that summarizes the growing prevalence of remote work

  102. [113]

    Attribute productivity claims to surveys, studies, or expert commentary

  103. [114]

    Include perspectives from workers experiencing remote work firsthand

  104. [115]

    Report on employer or organizational policy changes related to remote work

  105. [116]

    Explain challenges related to managing work-life boundaries

  106. [117]

    Provide background on how remote work practices have changed over time

  107. [118]

    Maintain a neutral, factual, and non- prescriptive tone throughout. ... (remaining constraints would be similarly specific) Now apply this approach to the following two news articles: Article A: {blog1} Article B: {blog2} Output Format: Main Task: [one sentence instruction] Co...

  108. [119]

    I will give you a task description, and you need to generate { content_type} content that fulfills the task

    [constraint] Template C.5.Base Generation You are a creative writing expert. I will give you a task description, and you need to generate { content_type} content that fulfills the task. The content should be: - Well-structured and coherent - Engaging and creative - Of appropri...

  109. [120]

    Base {content_type} content

  110. [121]

    Restrict the length of output to 500 words

    A list of 39 constraints Your job is to revise and expand the base content to satisfy the constraints while maintaining coherence, quality, and natural flow. Restrict the length of output to 500 words. Instructions: - Keep the core ideas from the base content - Integrate const...

  111. [122]

    Restrict the length of output to 500 words

    A list of constraints to satisfy Your job is to generate a complete, high-quality {content_type} that fulfills the task and satisfies as many constraints as possible while maintaining natural flow and coherence. Restrict the length of output to 500 words. Instructions: - Gener...

  112. [125]

    Output EXACTLY one line per constraint, in numeric order, followed by ONE final line

  113. [126]

    Do NOT output any extra text, headers, explanations, summaries, or blank lines

  114. [127]

    <exact sentence or excerpt from the content that proves the constraint>

    Each constraint line MUST follow one of these exact formats: i. Yes - "<exact sentence or excerpt from the content that proves the constraint>" OR i. No - <one short sentence explaining why the constraint is not satisfied>

  115. [128]

    less than N words

    Mark "Yes" ONLY if the constraint is explicitly and completely satisfied. - Partial satisfaction = No - Inference, implication, or interpretation = No - If uncertain, mark No EVALUATION RULES: - Semantic constraints must be clearly stated in the content. - If you must explain ...

  116. [129]

    Content (a story or blog)

  117. [130]

    CRITICAL INDEX RULE (MANDATORY): Each output line i MUST evaluate ONLY constraint i

    A numbered list of constraints Your task: Evaluate each constraint independently and determine whether it is fully satisfied by the content. CRITICAL INDEX RULE (MANDATORY): Each output line i MUST evaluate ONLY constraint i. Do not merge, skip, combine, or split constraints. ...

  118. [131]

    Output EXACTLY one line per constraint, in numeric order

  119. [132]

    Each line MUST be EXACTLY one of these two formats, with NOTHING else: i. Yes i. No

  120. [133]

    Do NOT output quotes, reasons, explanations, headers, summaries, or blank lines

  121. [134]

    less than N words

    Mark "Yes" ONLY if the constraint is explicitly and completely satisfied. - Partial satisfaction = No - Inference, implication, or interpretation = No - If uncertain, mark No EVALUATION RULES: - Semantic constraints must be clearly stated in the content. - Do not stretch meani...

  122. [136]

    Only Output the revised constraints, with no preamble or explanation

    A base story/blog written for that task Your job is to create a REVISED set of 39 constraints by replacing constraints that are TOO EASY with NEW, HARDER constraints that would be more difficult for a writer to satisfy. Only Output the revised constraints, with no preamble or ...

  123. [138]

    Identify constraints that seem easy or generic and replace them with harder, more specific ones

  124. [139]

    Keep constraints that are already sufficiently difficult

  125. [144]

    Constraint Replacement - Eval You are a writing expert

    [constraint] --- Input: Main Task: {main_task} Original Constraints: {original_constraints} Base Content: {base_content} Output: Template C.12. Constraint Replacement - Eval You are a writing expert. You are given:

  126. [145]

    A set of 39 constraints for that task

  127. [146]

    A base story/blog written for that task

  128. [147]

    Only Output the revised constraints, with no preamble or explanation

    Evaluation results showing which constraints are already satisfied by the base content Your job is to create a REVISED set of 39 constraints by replacing the constraints that are already satisfied (marked "Yes") with NEW, HARDER constraints that are NOT satisfied by the base c...

  129. [148]

    Keep the main task EXACTLY the same

  130. [149]

    Keep all constraints marked "No" ( not satisfied) unchanged

  131. [150]

    Replace all constraints marked "Yes" (satisfied) with new, harder constraints

  132. [151]

    Ensure you still have exactly 39 constraints total

  133. [152]

    Number the revised constraints 1-39

  134. [153]

    Randomize the order (don't put all new constraints at the end) Output Format: Main Task: [same as original] Revised Constraints:

  135. [154]

    [constraint - either kept from original or new replacement]

  136. [155]

    You will be given two pieces of content (Content A and Content B) and you must compare them on two metrics:

    [constraint] --- Input: Main Task: {main_task} Original Constraints: {original_constraints} Base Content: {base_content} Evaluation Results (Yes = satisfied, No = not satisfied): {satisfaction_results} Output: Template C.13.Pairwise Quality You are an expert writing evaluator....

  137. [156]

    **Coherence**: Which content has better logical flow and cohesion?

  138. [157]

    You won't get away with this,

    **Likability**: Which content is more enjoyable and engaging to read ? For each metric: - First, identify concrete issues for both contents - Then provide a score out of 5 for both A and B - Specify which content you prefer (A or B) - Give a brief one-line reasoning for your p...

  139. [159]

    Constraint

    Naturalness Scoring: Independently score each constraint from 1 to 5 based on how likely you would include it in a natural, high-quality response to their respective prompt. Score Scale Explanation 1 Never: You would almost never include this constraint; it feels very unnatura...

  140. [160]

    Write a constraint that you personally think fits the prompt well

    Write Your Own Constraint: Because people may judge constraints differently, we want to understand what you consider an ideal constraint for each writing prompt. Write a constraint that you personally think fits the prompt well. You may use your own constraint as a reference w...

  141. [161]

    Naturalness Scoring: Independently score each constraint on a scale from 1 to 5, based on how likely you would be to include it in a natural, high-quality response to its corresponding prompt

  142. [162]

    Score Scale Explanation 1 – Never: You would almost never include this constraint; it feels very unnatural or inappropriate

    Preference Selection: For each constraint pair, choose whether Constraint A or Constraint B you would prefer to see included in a written response to its respective prompt. Score Scale Explanation 1 – Never: You would almost never include this constraint; it feels very unnatur...

  143. [163]

    Use your constraint you've written as a soft comparison

    Score each constraint from 1 (never) to 5 (always) based on how likely you would include it in a written response to their prompt. Use your constraint you've written as a soft comparison

  144. [164]

    Task Instructions

    Select the constraint you prefer to be in a written response responding to their corresponding writing prompt. For more details please read the "Task Instructions" section above. Prompts Prompt A ${task_A} Prompt B ${task_B} Your Constraint A Write your own constraint for A......

  145. [165]

    Preference Selection: For each prompt, choose the constraint (A or B) that you would prefer to see included in a written response to their respective prompt

  146. [166]

    Score Scale Explanation 1 – Never: You would almost never include this constraint; it feels very unnatural or inappropriate

    Naturalness Scoring: Independently score each constraint from 1 to 5 based on how likely you would include it in a natural, high-quality response to their respective prompt. Score Scale Explanation 1 – Never: You would almost never include this constraint; it feels very unnatu...

  147. [167]

    Select which constraint (A or B) you think is more natural to include in a response to the prompt

  148. [168]

    Task Instructions

    Score each constraint from 1 (never) to 5 (always) based on how likely you would include it in a natural, high-quality response. For more details, please review the blue "Task Instructions" section above

  149. [169]

    2 – Rarely: You would include it only in unusual or forced situations

    Prompt ${prompt_1} Constraints Constraint A: ${constraintB_1} Constraint B: ${constraintA_1} Which constraint is more natural for this prompt? Constraint A Constraint B Equal Constraint A: ${constraintB_1} Naturalness score: 1-5 Constraint B: ${constraintA_1} Naturalness score...

  150. [2024]

    Chau Minh Pham, Simeng Sun, and Mohit Iyyer

    Llm evaluators recognize and favor their own generations.Advances in Neural Information Pro- cessing Systems, 37:68772–68802. Chau Minh Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following in long- form text generation. InFindings of the Associ...

    2024

  151. [2025]

    arXiv preprint arXiv:2503.06573

    Wildifeval: Instruction following in the wild. arXiv preprint arXiv:2503.06573. Gabrielle Kaili-May Liu, Bowen Shi, Avi Caciularu, Idan Szpektor, and Arman Cohan. 2025. Mdcure: A scalable pipeline for multi-document instruction- following. InProceedings of the 63rd Annual Meet...

    2025 arXiv

  152. [2026]

    The demonstrations were organized by theAlaska Association of Student Government

    Muldimif: A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. InFindings of the Association for Computational Linguistics: ACL 2026, pages 2078–2104. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Y...

    2026

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.