Pith. sign in

REVIEW 4 major objections 4 minor 15 references

This paper claims that instruction-stacking collapse is structured, reproducible, and partly recoverable by a single training-free prompt rewrite whose benefit concentrates on weaker models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:23 UTC pith:RTOKY3YB

load-bearing objection A solid benchmark and conflict topology with an honest scorecard, but the 'capability-graded' compiler benefit is not separated from headroom—the headline should be read as 'models with more room gain more.' the 4 major comments →

arxiv 2608.02639 v1 pith:RTOKY3YB submitted 2026-07-31 cs.SE cs.AI

Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation

Atul Anand , Sourav Chattaraj This is my paper
classification cs.SE cs.AI
keywords instruction followingprompt compilationconstraint stackingLLM evaluationpairwise conflictscapability dependencebenchmark
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that instruction-following collapse—the silent failure of models to obey many simultaneous constraints—is real, nonlinear, and structured, and that a cheap training-free fix can recover a meaningful share of the lost compliance on exactly the models that lose most. It builds a benchmark of 22 deterministic verifier-checked instructions, stacked randomly from one to twenty at a time, and shows follow rate falls from about 96% to as low as 20%, driven by reproducible pairwise conflicts; the "output JSON" rule alone is jointly unsatisfiable with nine others. The fix is an "instruction compiler": one LLM call rewrites the stack into a grouped, precedence-annotated checklist, computed once and reused across queries. The compiler recovers +11.0 percentage points for the weakest target and +3.3 for the middle one, while leaving the strongest essentially unchanged; controls attribute the gain to the semantic rewrite, not tokens, reordering, or headroom. If right, production teams deploying smaller models get an amortized, no-per-query-cost mitigation, and prompt rewriting is reframed as capability-dependent rather than uniformly beneficial.

Core claim

Central claim: stacking instructions causes a structured, reproducible collapse in compliance, and one amortized LLM rewrite recovers part of the loss in a capability-graded way. The benchmark shows follow rate falling from about 0.96 at one instruction to 0.20-0.60 at twenty across three models. The 231 pairwise interactions share a topology (cross-model Spearman about +0.23 to +0.28), with "output JSON" the dominant conflict hub. The compiler recovers +11.0 pp for the weakest target, +3.3 for the middle, -1.2 (not robust) for the strongest; ablations rule out tokens and reordering, and a within-family ladder gives Spearman -0.85. Pre-registered misses: Gemini's decay is linear; d >= 0.5 wa

What carries the argument

The instruction compiler: a single zero-shot LLM call that transforms a raw stack of instructions into a numbered, category-grouped checklist with overlapping rules merged and a precedence note inserted wherever two conflict (reasoning first, format last, ending with a self-check line). Because it is computed once per instruction set and reused across queries, it has zero per-query cost. On the diagnostic side, the load-bearing tool is the pairwise interaction statistic observed(A and B) minus P(A)P(B) over all 231 instruction pairs, with Benjamini-Hochberg correction, which separates the 15 logically impossible pairs from behavioral conflicts and lets the paper claim the collapse is structu

Load-bearing premise

The load-bearing premise is that a model's raw stacked-following rate on the same benchmark measures its capability; the paper concedes this proxy makes the strength-recovery link partly circular, so if the ordering is really driven by measurement headroom, the headline claim weakens to "models with more room to improve gain more."

What would settle it

Run the same compiler comparison on a held-out instruction set where the strongest model's raw follow rate is artificially low (for example, a stack heavy in its worst category, format and length). If recovery rises to the weak-model level, the capability reading fails; if recovery follows raw rate rather than model family, the headroom explanation is correct.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Stacking collapse can be treated as predictable: a sigmoid curve fits the degradation for two of three models, and the conflict topology is shared across models and tasks.
  • The "output JSON" instruction is a systematic liability: it is jointly unsatisfiable with nine other instructions, so prompt authors should expect cascading silent failures when it is combined with markdown or structural rules.
  • A single amortized rewrite is a cost-effective fix for smaller deployed models: +11.0 pp for the weakest target, +3.3 pp for the middle one, with no per-query cost.
  • No prompt-only compiler can fully fix the problem: 15 of the 231 pairs are logically impossible to satisfy simultaneously, setting a structural ceiling on recovery.
  • Collapse is universal across advice, math, and code, but the per-model weakness ordering is task-specific; "which model is weakest" cannot be answered without specifying the task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the conflict topology could be used as a pre-deployment checker—scan a system prompt for known impossible pairs and report omissions to the caller instead of letting them fail silently.
  • Beyond the paper: a verifier-guided compiler that drops the lower-priority side of an impossible pair would close part of the residual gap; the paper's compiler never removes constraints, by design.
  • Beyond the paper: the claim that benefit tracks capability rather than headroom remains tentative because the ladder is single-seed; a multi-seed run across stack sizes would settle it.
  • Beyond the paper: the benchmark's per-instruction degradation curves could be turned into a "stack budget"—an empirical cap on how many format and length instructions one system prompt should carry.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper introduces Stacking-Collapse, a benchmark of 24 verifier-backed atomic instructions sampled uniformly into prompts of size 1–20, and uses it to measure instruction-following collapse on three models (Claude Sonnet 4.6, GPT-5-mini, Gemini 2.5 Flash). The authors report that follow rate falls from ~0.96 at stack=1 to 0.60/0.43/0.20 at stack=20, that the decline is non-linear and partly sigmoidal, and that a pairwise interaction analysis of all 231 pairs identifies roughly 12% of satisfiable pairs as behavioral conflicts, with “output JSON” as the dominant conflict hub. They then evaluate a training-free instruction compiler and report that it recovers up to +11.0 pp of follow rate for the weakest target while leaving the strongest target essentially unchanged, interpreting the benefit as capability-graded. The paper is explicitly pre-registered and transparently reports that two of three pre-registered predictions failed. Code, cached responses, verifiers, and analysis scripts are released.

Significance. If the capability-graded recovery claim were cleanly established, this would be a practically important result: a one-time, amortized prompt rewrite that helps the smaller models most deployed at scale. The benchmark and the collapse/conflict findings are a solid contribution in their own right: the 24 verifier-checked instructions, the separation of designed-in impossibilities from genuine behavioral conflicts, and the cross-model/cross-task reproducibility of the interaction landscape are concrete and reusable. The execution has notable strengths: cluster-bootstrap inference, same-baseline controls for token count and reordering, an honest pre-registration scorecard, and fully released artifacts. However, I do not think the headline capability-graded interpretation is established. Because target strength is operationalized as the raw follow rate on the same benchmark where recovery is measured, the ordering is vulnerable to a mechanical headroom/floor effect, and the within-family ladder is not monotone in the claimed direction. The collapse and conflict findings are independent of this weakness, but the mitigation's central interpretation needs substantial additional work before

major comments (4)
  1. [§4.1, §5.2, §5.3, Table 6] The “capability-graded” claim is confounded with baseline headroom. Target strength is operationalized in §4.1 as the model's raw stacked-following rate on the same benchmark, and recovery in §5.2 is measured as the change in that same rate. A lower raw rate mechanically leaves more absolute room for improvement under any more effective intervention. The same-baseline controls (expanded_raw, reorder_only) rule out extra tokens and reordering for a fixed model, but they do not rule out the cross-model baseline-deficit confound, because compiled is a qualitatively different and stronger intervention. The ladder in Figure 5/Table 7 shows the expected inverted-U signature (GPT-5-nano +6.0 vs GPT-5-mini +11.8; Sonnet −6.7 vs Opus −2.0). Section 6 limitation (i) concedes the link is “partly circular.” A concrete fix is to grade recovery against an external capability index or to include a head
  2. [§5.4, Table 7, Figure 5] The within-family ladder does not support the monotone claim that “recovery falls as the target strengthens.” Reading Table 7 by family: Anthropic Haiku +9.0, Sonnet −6.7, Opus −2.0; OpenAI nano +6.0, mini +11.8, GPT-5.5 +4.0; Google Flash-Lite +7.0, Flash +5.0, Pro −1.0. Only the Google family is monotone in the claimed direction. The pooled Spearman (−0.85) is therefore largely a cross-family correlation with raw rate, not a within-family capability ordering. In addition, the sentence “the top rung of every family gains the least” is contradicted by the Anthropic column, where the middle rung (Sonnet) has the smallest recovery (−6.7), not the top rung (Opus, −2.0). Finally, the Sonnet anchor result differs between Table 5 (pooled stacks 8–20: −1.2 pp) and Table 7 (stack=20 only: −6.7 pp); this discrepancy needs explanation.
  3. [§5.1, §5.2] No audit is reported that the compiler actually preserves the instruction set. The compiler prompt instructs the model to “Do NOT add or remove constraints. Only restructure and clarify,” but LLMs frequently violate such meta-instructions. If a compiled prompt drops or merges constraints, then the target response is graded against fewer instructions, inflating the follow rate independently of any real improvement in compliance. Since the benchmark ships deterministic verifiers, an audit of compiled prompts (e.g., verifying that each atomic instruction is still present, or at minimum reporting the number of constraints in compiled vs raw prompts) is necessary to attribute the recovery in Table 5 to rewriting rather than deletion. Without this check, the magnitude of the recovery estimate is not interpretable.
  4. [§5.3, “Not leakage”] The leakage control is not decisive. The argument is that if the compiler merely fed Sonnet its own reasoning, the same-model case would gain most; instead it gains least. But the same-model target is Sonnet 4.6, which is also the strongest model and, under the paper's own strength proxy, the one with the least headroom. The comparison therefore cannot separate compiler-identity effects from capability/headroom effects. A proper leakage control would require, for example, a strong model compiling for itself versus another strong model compiling for the same target, or a weaker compiler model. As it stands, the “not leakage” claim is underdetermined by the experimental design.
minor comments (4)
  1. [§4.2, §H] The reported 95% CI for the behavioral conflict rate, [4.6, 11.6], does not contain the point estimate “~12%.” Since this appears twice, it is likely a typo (e.g., the upper endpoint should be 12.6), but it should be reconciled.
  2. [§5.3, Table 6] The numbers in the “Not extra tokens / not reordering” paragraph do not match Table 6. For GPT-5-mini, compiled vs expanded_raw is 0.322−0.212 = +11.0 pp and vs reorder_only is +10.8 pp, not “+10.0/+12.1 pp”; for Sonnet the values are +7.4 and +5.9 pp, not “+6.2/+4.1 pp.” Please specify the stack range and which comparison is being reported, and align the text with the table.
  3. [§3, Table 9] The contributions call the verifiers an “audited verifier suite,” but the audit is incomplete: R1, R3, and S2 are spot-checked only, and the paper states a full 100-label-per-instruction audit is owed. This is acknowledged in Section 6, but the contribution wording should be softened accordingly.
  4. [Figure 5] The x-axis label “capability proxy” overstates the construct. It is the benchmark's own raw stacked-following rate, which is exactly the quantity whose interpretation is at issue. Consider relabeling the axis as “raw stacked-following rate at stack=20” and leaving the capability interpretation to the analysis.

Circularity Check

1 steps flagged

The measured collapse and conflict topology are self-contained; only the headline capability-grading reduces partly to the paper's own within-benchmark proxy, which it concedes is 'partly circular.'

specific steps
  1. self definitional [Section 4.1 (operationalization of target strength); Section 5.2 / Table 5 (recovery measured on the same rate); Section 6 Limitation (i)]
    "We operationalize target strength as a model's raw stacked-following rate (Sonnet > Gemini > GPT-5-mini), the axis Section 5 grades the compiler against. ... Limitations.(i)Construct:verifiers capture the letter, not the intent, of an instruction; “target strength” is a within-benchmark proxy, making the strength–recovery link partly circular (broken by the cross-model and same-baseline-control evidence)."

    The independent variable in the capability claim ('target strength') and the dependent variable ('recovery', compiled minus raw) are defined on the same measured quantity: the benchmark's raw stacked-following rate. A low raw rate mechanically creates more absolute room for improvement, so ordering models by that proxy and then reporting larger gains for the low-proxy models is, to a substantial degree, an artifact of the shared baseline rather than an independent capability measurement. The paper's own Limitation (i) admits this. The expanded_raw/reorder_only controls address extra tokens and reordering, not this proxy coupling, and the within-family ladder's inverted-U ('recovery peaks for weak-but-not-floored targets, with a floor effect at the bottom') is the expected headroom/floor si

full rationale

The collapse curves (Section 4.1), the pairwise conflict topology (Section 4.2), and the per-condition recovery numbers (Section 5.2, Table 5) are measured, cached, and externally checkable; they are not fitted inputs called predictions, and there is no self-citation chain or imported uniqueness theorem. The only load-bearing circularity is the interpretation of recovery as capability-graded: Section 4.1 defines 'target strength' as the raw stacked-following rate on the same benchmark, and Section 5 measures recovery as the change in that same rate. The reported ordering 'weak targets gain more' is therefore partly a restatement of 'lower baselines have larger absolute headroom,' not an independent capability axis. The paper itself labels this 'partly circular' in Limitation (i). The same-baseline controls rule out token-count and reordering confounds but not this shared-baseline coupling, and the within-family ladder's bottom inverted-U (nano +6.0 vs mini +11.8) is the signature of a headroom/floor effect. Because the numeric recovery results remain valid and the ladder adds some independent tier information, the circularity is partial rather than total; the collapse and conflict findings are untouched.

Axiom & Free-Parameter Ledger

2 free parameters · 7 axioms · 0 invented entities

The ledger is light because the paper's results are measurements, not derivations. Two hand-fitted components matter: the sigmoid characterization of the collapse curve (Section 4.1) and the verifier leniency policies calibrated in the pilot audit (Section 3, flipping 8.0% of verdicts). The axioms are the empirical premises the measurements sit on: verifier fidelity within audit tolerance, the within-benchmark capability proxy, temperature-0 reproducibility, the independence counterfactual for pairwise conflicts, and uniform unscreened stacking as the evaluation target. The paper itself flags two of these (Limitations i and ii). No invented entities: the instruction compiler is a prompt template, not a postulated object.

free parameters (2)
  • Sigmoid shape parameters (lower and upper asymptotes, midpoint, slope) per model = 4 parameters per model fit to 7 stack-size points; sigmoid wins AIC in 93% (Sonnet) and 100% (GPT-5-mini) of refits; Gem
    Section 4.1. Descriptive fit of the collapse curve with a model-selection bootstrap. This is a characterized shape, not a prediction; the paper does not misuse it as a prediction.
  • Verifier leniency policies (case-insensitive headers, any list marker, code-fence stripping) = Policies chosen in the pilot audit; they flipped 8.0% of verdicts compared with strict parsing
    Section 3. These hand-chosen policies calibrate every pass/fail verdict in the paper. A human audit supports 18/19 verifiers at 100% agreement, so the calibration is validated but not exhaustive (3 verifiers spot-checked only).
axioms (7)
  • domain assumption The 22 deterministic verifiers correctly implement the instruction set within the tolerance established by the abbreviated audit (~500 labels, 3 verifiers spot-checked only).
    Section 3 and Appendix C report 18/19 verifiers agreeing with humans at 100% and C4 at 80%, and Section 6 limitation (ii) concedes the audit is abbreviated. Every absolute follow rate and conflict count in Sections 4 and 5 depends on this.
  • domain assumption 'Target strength', operationalized as raw stacked-following rate on this same benchmark, is a valid capability axis for grading the compiler, and the strength-recovery ordering is not a measurement-headroom artifact.
    Section 4.1 defines the operationalization; Section 5.4 reads Spearman -0.85 as capability-dependence. Section 6 limitation (i) calls the link 'partly circular'. The ladder's inverted-U (GPT-5-nano +6.0 below GPT-5-mini +11.8, Opus -2.0) is the signature of a headroom effect.
  • domain assumption Temperature-0 API responses are deterministic enough that caching by (model, system, user) hash supports the claim of exact reproduction without new API calls.
    Section 3: 'All responses are cached by (model,system,user) hash for exact reproducibility.' Provider-side determinism is asserted, not demonstrated, and no cached artifacts are locatable from the manuscript.
  • domain assumption The pairwise interaction null, observed(A AND B) - P(A)P(B) with P from singleton baselines, is an unbiased counterfactual for behavioral conflicts.
    Section 4.2 and Appendix H. Independence and the singleton baseline are assumed; position and context effects are averaged over randomization but not explicitly tested. The reported CI [4.6, 11.6] also fails to bracket the 12% point estimate.
  • domain assumption Uniform random stacking without feasibility screening is the right evaluation target, because real production prompts are not screened for satisfiability.
    Section 3: stacks are 'sampled uniformly at random (not screened for feasibility, since real prompts are not)'. This raises the ecological validity of the collapse curve but makes high stack sizes include the 15 impossible pairs.
  • domain assumption Excluding the two LLM-judge instructions (C2, C3) does not materially skew the collapse or conflict measurements.
    Appendix A leaves the Content category with only 2 of 4 active instructions, so category coverage is imbalanced (2/4 vs 4/4 elsewhere) and the content category is under-represented in random stacks.
  • standard math Resampling whole stacks approximates the sampling distribution under within-stack correlation introduced by shared instruction draws and compiled prompts.
    Appendix H: 3,000 cluster-bootstrap reps, with the naive t-test reported as overstating significance. Standard resampling assumption, applied correctly.

pith-pipeline@v1.3.0-alltime-deepseek · 12831 in / 28965 out tokens · 265594 ms · 2026-08-05T04:23:01.296033+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation." pith.science (2026). https://pith.science/paper/RTOKY3YB

@misc{pith2026260802639,
  author       = {Pith},
  title        = {Pith review of: Instruction Stacking Collapse: A Benchmark and the Capability-Dependent Value of Prompt Compilation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RTOKY3YB}},
  note         = {Machine review of arXiv:2608.02639}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Production prompts rarely carry a single instruction. One system message may require valid JSON, a word limit, three citations, and a fixed tone at the same time. We study how instruction-following degrades as such constraints accumulate. We introduce a benchmark that stacks 24 verifier-checked instructions, one to twenty at a time, and evaluate three production-tier LLMs (Claude Sonnet 4.6, GPT-5-mini, Gemini 2.5 Flash). Instruction-following degrades non-linearly: the follow rate falls from ~96% to as low as 20%, driven by a structured and reproducible set of pairwise conflicts. A single "output JSON" constraint, for example, is jointly unsatisfiable with nine others. We then evaluate a training-free remedy: an instruction compiler that rewrites the stacked prompt in a single LLM call and is reused across queries. Its benefit is capability-graded. It recovers up to +11 points of follow rate for weaker models, which are also the models most often deployed at scale, while leaving stronger models, which already internalise the same structure, essentially unchanged. Cluster-robust tests, same-baseline controls, and a within-family scaling ladder attribute the gain to the rewrite itself rather than to additional tokens, reordering, or measurement headroom. We release the benchmark, verifiers, and cached runs for full reproduction.

Figures

Figures reproduced from arXiv: 2608.02639 by Atul Anand, Sourav Chattaraj.

Figure 1
Figure 1. Figure 1: Overview. Stacking many instructions into one prompt collapses instruction-following (top). A training-free instruction compiler rewrites the stack in one LLM call; its benefit is capability-graded: large for weak models, neutral-to-negative for strong ones (bottom). artifact of fitting four parameters to seven points. Conflict (RQ2). The cause is interference between instruc￾tions. Testing all 231 pairs a… view at source ↗
Figure 2
Figure 2. Figure 2: Instruction-following rate vs. stack size (advice). Each point is a mean over 30 random stacks × 10 items (∼300 trials). Follow rate falls from ∼0.96 to 0.20–0.60; GPT-5-mini collapses sharpest. from cache with no new calls. Three verifiers (R1, R3, S2) are spot-checked only; a full 100-label-per-instruction vali￾dation is owed (Section 6). Details and the agreement table are in Section C. Grid. Stacks are… view at source ↗
Figure 3
Figure 3. Figure 3: Pairwise interaction for the 231 pairs (Sonnet), ordered by category. Blue = conflict (joint rate below independent base￾line); red = synergy; dots are BH-significant at FDR 0.05. F1 (JSON) is the dominant conflict row; C4 (named sources) anchors the synergies. is non-sigmoidal: its decline begins immediately and is best described as linear-to-exponential (sigmoid in < 2% of refits). Categories degrade une… view at source ↗
Figure 4
Figure 4. Figure 4: Recovery (compiled−raw) by stack size per target; dashed line is the pre-registered +10 pp threshold. GPT-5-mini clears it from stack=12; Sonnet goes negative [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Recovery at stack=20 vs. raw rate, one line per family. Filled markers are the production-tier anchors; open markers the added rungs. Recovery falls as the target strengthens; a floor effect appears at the very bottom. 5.3. The mechanism is the LLM rewrite Four alternatives, all ruled out ( [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean follow rate by category, pooled across models. Lexical survives; format and length collapse hardest [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Per-instruction follow rate by stack size. L3, S1, F4, F1 fall fastest; S4, X2, X4 stay high. significance by Benjamini–Hochberg at FDR 0.05. The feasible-only conflict rate is ∼12% with a pair-resampling 95% CI of [4.6, 11.6]. Cross-model Spearman of all 231 interactions: +0.27, +0.28, +0.23 (all p < 10−3 ). Ladder. Spearman between raw stacked-following rate and recovery across the nine ladder targets is… view at source ↗
Figure 8
Figure 8. Figure 8: Per-model conflict heatmaps. F1 is the dominant conflict hub in every panel; GPT-5-mini (right) is the sparsest despite collapsing hardest in aggregate [PITH_FULL_IMAGE:figures/full_fig_p011_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Degradation on advice / math / code. The fact of collapse generalizes; the per-model ordering does not. 8 12 16 20 Stack size (instructions) 0.0 0.2 0.4 0.6 0.8 1.0 Instruction-following rate Claude Sonnet 4.6 (same-model) 8 12 16 20 Stack size (instructions) GPT-5-mini (cross-model) 8 12 16 20 Stack size (instructions) Gemini 2.5 Flash (cross-model) Raw stack Compiled (ours) Expanded raw Reorder only Foll… view at source ↗
Figure 10
Figure 10. Figure 10: Follow rate by condition × stack size per target. compiled separates from controls on the cross-model targets; on Sonnet raw is best. Claude Sonnet 4.6 (same-model)GPT-5-mini (cross-model) Gemini 2.5 Flash (cross-model) 0.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Instruction-following rate at stack=20 0.63 0.20 0.45 0.56 0.32 0.50 0.49 0.21 0.48 0.50 0.21 0.46 Ablation at stack=20: only LLM rewriting helps the we… view at source ↗
Figure 11
Figure 11. Figure 11: Stack=20 conditions per target. Only the LLM rewrite (compiled) separates from raw; token-padding and reordering do not. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 9 canonical work pages · 1 internal anchor

  1. [1]

    LLM Self-Correction with

    Ferraz, Thomas Palmeira and Mehta, Kartik and Lin, Yu-Hsiang and Chang, Haw-Shiuan and Oraby, Shereen and Liu, Sijia and Subramanian, Vivek and Chung, Tagyoung and Bansal, Mohit and Peng, Nanyun , journal =. LLM Self-Correction with. 2024 , url =

    work page 2024

  2. [2]

    2025 , url =

    Guo, Zhengkang and Liu, Wenhao and Xie, Mingchen and Xu, Jingwen and Huang, Zisu and Tian, Muzhao and Xu, Jianhan and Shen, Yuanzhe and Qian, Qi and Wu, Muling and Wang, Xiaohua and Lv, Changze and Wang, He-Da and Yao, Hu and Zheng, Xiaoqing and Huang, Xuanjing , journal =. 2025 , url =

    work page 2025

  3. [4]

    2024 , url =

    Jiang, Yuxin and Wang, Yufei and Zeng, Xingshan and Zhong, Wanjun and Li, Liangyou and Mi, Fei and Shang, Lifeng and Jiang, Xin and Liu, Qun and Wang, Wei , booktitle =. 2024 , url =

    work page 2024

  4. [5]

    2024 , url =

    Kong, Weize and Hombaiah, Spurthi and Zhang, Mingyang and Mei, Qiaozhu and Bendersky, Michael , booktitle =. 2024 , url =

    work page 2024

  5. [6]

    Transactions on Machine Learning Research (TMLR) , year =

    Inverse Scaling: When Bigger Isn't Better , author =. Transactions on Machine Learning Research (TMLR) , year =

    work page

  6. [8]

    2026 , note =

    Ye, Junjie and Huang, Caishuang and Chen, Zhuohan and Fu, Wenjie and Yang, Chenyuan and Yang, Leyi and Wu, Yilong and Wang, Peng and Zhou, Meng and Yang, Xiaolong and Gui, Tao and Zhang, Qi and Shi, Zhongchao and Fan, Jianping and Huang, Xuanjing , booktitle =. 2026 , note =

    work page 2026

  7. [10]

    LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints

    Ferraz, T. P., Mehta, K., Lin, Y.-H., Chang, H.-S., Oraby, S., Liu, S., Subramanian, V., Chung, T., Bansal, M., and Peng, N. Llm self-correction with DeCRIM : Decompose, critique, and refine for enhanced following of instructions with multiple constraints. arXiv preprint arXiv:2410.06458, 2024. URL https://arxiv.org/abs/2410.06458

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  8. [11]

    RECAST : Expanding the boundaries of LLMs ' complex instruction following with multi-constraint data

    Guo, Z., Liu, W., Xie, M., Xu, J., Huang, Z., Tian, M., Xu, J., Shen, Y., Qian, Q., Wu, M., Wang, X., Lv, C., Wang, H.-D., Yao, H., Zheng, X., and Huang, X. RECAST : Expanding the boundaries of LLMs ' complex instruction following with multi-constraint data. arXiv preprint arXiv:2505.19030, 2025. URL https://arxiv.org/abs/2505.19030

    arXiv 2025

  9. [12]

    Hakim, M. D. A. Brevity constraints reverse performance hierarchies in language models. arXiv preprint arXiv:2604.00025, 2026. URL https://arxiv.org/abs/2604.00025

    arXiv 2026

  10. [13]

    FollowBench : A multi-level fine-grained constraints following benchmark for large language models

    Jiang, Y., Wang, Y., Zeng, X., Zhong, W., Li, L., Mi, F., Shang, L., Jiang, X., Liu, Q., and Wang, W. FollowBench : A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024. URL https://aclanthology.org/2024.acl-long.257/

    work page 2024

  11. [14]

    PRewrite : Prompt rewriting with reinforcement learning

    Kong, W., Hombaiah, S., Zhang, M., Mei, Q., and Bendersky, M. PRewrite : Prompt rewriting with reinforcement learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024, Short Papers), 2024. URL https://aclanthology.org/2024.acl-short.54/

    work page 2024

  12. [15]

    McKenzie, I. R., Lyzhov, A., Pieler, M., Parrish, A., Mueller, A., Prabhu, A., McLean, E., Kirtland, A., Ross, A., Liu, A., Gritsevskiy, A., Wurgaft, D., Kauffman, D., Recchia, G., Liu, J., Cavanagh, J., Weiss, M., Huang, S., The Floating Droid , Tseng, T., Korbak, T., Shen, X., Zhang, Y., Zhou, Z., Kim, N., Bowman, S. R., and Perez, E. Inverse scaling: W...

    Pith/arXiv arXiv 2023

  13. [16]

    On the paradoxical interference between instruction-following and task solving

    Qi, Y., Peng, H., Shi, X., Xin, A., Wang, X., Xu, B., Hou, L., and Li, J. On the paradoxical interference between instruction-following and task solving. arXiv preprint arXiv:2601.22047, 2026. URL https://arxiv.org/abs/2601.22047

    arXiv 2026

  14. [17]

    MulDimIF : A multi-dimensional constraint framework for evaluating and improving instruction following in large language models

    Ye, J., Huang, C., Chen, Z., Fu, W., Yang, C., Yang, L., Wu, Y., Wang, P., Zhou, M., Yang, X., Gui, T., Zhang, Q., Shi, Z., Fan, J., and Huang, X. MulDimIF : A multi-dimensional constraint framework for evaluating and improving instruction following in large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Li...

    Pith/arXiv arXiv 2026

  15. [18]

    Instruction-following evaluation for large language models

    Zhou, J., Lu, T., Mishra, S., Brahma, S., Basu, S., Luan, Y., Zhou, D., and Hou, L. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023. URL https://arxiv.org/abs/2311.07911

    Pith/arXiv arXiv 2023