{"id":"2f9cbddd-5a9e-46f8-bdb7-4d6e8154f48e","arxiv_id":"2506.01265","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"LongGuide automatically learns task-specific quality and length guidelines from small training sets, significantly improving LLM long-form generation.","lead":"This paper introduces LongGuide, a method that automatically writes task-specific guidelines helping large language models produce better summaries, simplifications, translations, and other long-form outputs. The authors show that adding such guidelines improves model performance on several generation benchmarks, and they provide a theoretical argument for why demonstrations alone are insufficient.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Step 5's guideline selection is validated on the same 50-sample Dtrain used to build the OCG and MG; if this selection overfits Dtrain, the claimed automatic gains may not generalize.","rationale":"The reader's weakest assumption identifies the most load-bearing risk: LongGuide's Step 5 is a selection procedure, and its only validation signal is ROUGE-L on the same 50-sample Dtrain that also supplies the OCG length statistics and MG quality profile. This is not a speculative concern; the paper itself flags the high-variance failure mode in Limitations, and E.8 documents one instance where the Dtrain-selected guideline (MG-OCG) is not the best on the test set (MG for IWSLT17 en-ja with ChatGPT few-shot). That one discrepancy is not fatal by itself, but it demonstrates that the selection step can be wrong, and the paper does not quantify how often this happens across different Dtrain draws. I considered other candidate concerns: the theoretical proof (Remark B.1) only addresses the input-language distribution rather than the generated-output distribution, which the authors acknowledge, and the 'over 5%' abstract claim is actually supported as a mean over the seven tasks per model and setting. Neither is as central as the selection-overfitting risk, because the empirical value of the MG and OCG components could survive those issues, whereas the 'automatic' selection claim depends directly on Step 5's generalization. The proposed resampling test would settle whether the selection is robust or whether the reported numbers are tied to one favorable 50-sample draw. Since the reader already assigned CONDITIONAL and this concern falls within that condition, no verdict change is needed.","tokens_in":41974,"tokens_out":5615,"duration_ms":68258,"concrete_test":"Re-run LongGuide on SAMSum, SWiPE, and IWSLT17 en-ja using 10 different random 50-sample Dtrain draws. For each draw, record the Step-5-selected guideline and the resulting test ROUGE-L and GPT-4o-Judge scores; also run a 40/10 split where guidelines are built on 40 samples and selected on the held-out 10. If the selected guideline flips across draws (e.g., OCG vs. no-guideline on SWiPE) or if test gains shrink by more than the reported 95% CI when selection is done out-of-sample, then Step 5 is overfitting Dtrain and the headline 'over 5%' improvement should be reported conditionally on the specific Dtrain draw.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LongGuide automatically selects the best guideline combination rests on Step 5, which chooses among {none, MG, OCG, MG-OCG} by ROUGE-L on the same 50-sample Dtrain used to estimate the OCG min/max/average sentence and token statistics and to calibrate the MG score profile. This creates an overfitting risk: an OCG built from Dtrain's length statistics will trivially score well on Dtrain even when the test distribution has different statistics. The paper's Limitations section explicitly concedes that task-level average statistics can be ineffective for high-variance tasks and that Step 5 may respond by choosing no guideline. Consistent with this, E.8 reports a concrete discrepancy: for IWSLT17 en-ja with ChatGPT few-shot, Dtrain selection chooses MG-OCG while the best test-set guideline is MG. Because the same 50 samples are used both to construct and to select the guidelines, the reported test improvements are not protected against selection overfitting. If the Step 5 choice is unstable across resampled Dtrain draws, the 'automatic' part of the headline claim is weaker than presented, even if the individual MG and OCG components are useful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that in-context learning (ICL) demonstrations are insufficient for long-form generation because they fail to reliably transfer task-specific language and format properties. To address this, the authors propose LongGuide, which learns two kinds of guidelines from at most 50 training samples: Metric Guidelines (MGs) that steer the model toward self-evaluated linguistic qualities, and Output Constraint Guidelines (OCGs) that impose token- and sentence-level length bounds. LongGuide then selects among {no guideline, MG, OCG, MG-OCG} by validation performance on the same 50 samples. The empirical study covers seven generation tasks plus AlpacaEval2 with Mistral-7B-it and ChatGPT, reporting average ROUGE-L / GPT-4o-Judge gains over 5% in zero- and few-shot settings, along with human evaluation, ablations, transfer to non-instruct models, and cost comparisons.","tokens_in":42276,"tokens_out":4207,"duration_ms":47446,"significance":"If the claimed effects hold, LongGuide is a useful practical contribution: it offers a cheap, task-level procedure that improves strong LLMs on long-form generation, and the transfer experiments suggest an attractive workflow where a weak open model learns guidelines that later improve a strong closed model. The evaluation is broad, includes human judgments, and the component ablations are informative. The central effect is plausible—enforcing output length and format statistics from a small training set directly addresses a well-known failure mode of long-form generation. However, the theoretical support is limited to the marginal language distribution P(X) rather than the generation distribution P(Y|X), the test sets are small (200 samples per task), and the automatic selection step has a documented suboptimal case.","major_comments":[{"comment":"Step 5 of Algorithm 1 selects the guideline configuration on the same 50-sample Dtrain used to estimate the OCG statistics and MG score profiles, so the selection is not protected against overfitting to Dtrain. The paper itself documents a concrete failure in Apx. E.8: for IWSLT17 en-ja with ChatGPT few-shot, Dtrain selects MG-OCG (Table 14) while the best configuration on the test set is MG (Table 4), with ROUGE-L 38.43 for the selected MG-OCG versus 39.45 for MG. This weakens the abstract's claim that LongGuide 'automatically selects the best combination of guidelines.' Please add a stability check (e.g., bootstrap over Dtrain draws or multiple random Dtrain subsamples) and either qualify the selection claim or show that the selection is not systematically biased.","section":"§3, Step 5; Apx. E.8 (Tables 14 and 4)"},{"comment":"The theoretical result is about the marginal input distribution: Remark B.1 proves PM(X|Df) ≠ PT(X) under PM ≠ PT. It does not directly address the output distribution PM(Y|X) or arg max_y PM(Y|X), which is what generation quality depends on. The abstract and §2 nevertheless claim that ICL demonstrations are 'theoretically' insufficient to teach 'the task's language and format distributions for generation.' The Limitations section explicitly concedes this gap, but the main text should be revised so that the theoretical contribution is not overstated. I suggest rephrasing the claim to say the theory supports the language-distribution mismatch, while the generation-distribution statement remains a hypothesis supported empirically.","section":"Apx. B, Remark B.1 and §2"}],"minor_comments":[{"comment":"The 'over 5%' improvement is an average across seven tasks and settings; several individual cells are far below 5% (e.g., SWiPE zero-shot ChatGPT shows 0.00 ROUGE-L gain in Table 3, and Mistral few-shot CNN shows 1.63). Please qualify the abstract statement as an average and, if space permits, report the per-task range.","section":"Abstract and Table 3"},{"comment":"There is a typo: 'ansMistral' should be 'and Mistral'. Also, the name 'Suzzanne' in Figure 1 is inconsistent with 'Suzanne' in the text and other examples.","section":"§2, first paragraph"},{"comment":"The string 'MGâ˘A¸SOCG' appears to be an encoding artifact; it should likely read 'MG-OCG'. Please check all occurrences of this artifact in the appendix.","section":"Algorithm 1 header"},{"comment":"The cost comparison states LongGuide is 'approximately at least 3.75 times more cost-efficient' than prompt optimizers; Table 20 shows it is 3.75× cheaper than adv-ICL but 18.75× cheaper than APO. Please make the comparison explicit per baseline.","section":"§G.3, Table 20"},{"comment":"The 'Gen. Gui. on Demos' baseline is only briefly defined; please clarify how the general guideline prompt is elicited from demonstrations, since this is the main comparison for the 'learned from demonstrations' claim.","section":"§5.2, Figure 5"},{"comment":"The human evaluation is based on 50 samples and only 5 MG metrics; while useful, the small scale should be acknowledged in the main text (it is currently only implicit in the setup).","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution, but the automatic-selection component needs strengthening. The documented suboptimal selection in E.8 and the absence of a stability analysis are the main reasons I cannot recommend accept at this stage. The theory is honestly limited and is already acknowledged, but the abstract overstates it. I would encourage the editor to request a bootstrap or resampling analysis of Step 5 and a qualified abstract, after which the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: LongGuide is a practical, well-tested recipe for improving long-form generation with two automatically generated guideline streams. The paper deserves a serious referee, but the headline 'automatic' claim is a bit stronger than the evidence, and the theory is closer to a sanity check than a theorem.\n\nWhat's new: the combination of metric guidelines (MG), output-constraint guidelines (OCG), and a small-validation selection step is genuinely new. The empirical coverage is broad — seven long-form tasks, two models, human evals, ablations of each step — and the main finding holds up: explicit constraints on length and format, along with self-evaluated metric definitions, consistently shift outputs toward ground-truth length/format and improve ROUGE-L and judge scores. The property-transfer experiments in Section 2 are a nice, clean demonstration that ICL demonstrations alone do not reliably transfer length or format properties.\n\nWhere I'd push back: the stress-test about Step 5 is real, but minor. Selecting among {none, MG, OCG, MG-OCG} using ROUGE-L on the same 50 training samples that produced the OCG statistics does create an overfitting risk, and the paper's own E.8 reports one case where selection on Dtrain differs from the test-best (IWSLT17 en-ja, ChatGPT few-shot). That's 1 of 28 settings, so the automatic selection is mostly working. Still, the paper should quantify this: a small resampling/bootstrap of Dtrain would show how stable the Step 5 choice is, and the authors should release code/data so others can check. The abstract's 'over 5%' is an average; several settings, especially SWiPE zero-shot with ChatGPT, show no gain. That should be stated as an average, not a universal improvement.\n\nThe theory (Remark B.1) is weaker than the paper's framing. It addresses P(X), not the conditional output distribution, and under the stated assumptions it is close to a tautology. The authors acknowledge this in the limitations. Since the empirical work stands on its own, this is a presentation issue, not a fatal flaw.\n\nBottom line: send it out. This is a useful contribution that will help people working on prompting and controllable generation. With code, a stability analysis of Step 5, and an honest 'average gain' claim, it would be a solid paper.","headline":"LongGuide is a practical, well-tested pipeline for improving long-form generation, but the automatic-selection and 'over 5%' claims need honest qualification; it deserves serious refereeing.","tokens_in":42780,"tokens_out":2884,"would_cite":true,"duration_ms":30947,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In-context learning demonstrations alone cannot teach LLMs the format and language of long-form outputs; LongGuide's task-derived guidelines improve both open- and closed-source models by over 5%.","keywords":["long-form generation","in-context learning","prompt guidelines","LLM alignment","text property transfer","output constraints","self-evaluation","zero-shot and few-shot prompting"],"falsifier":"Scale the Section 2 property-maintenance experiment to large demonstration sets: choose one long-form task, fix demonstrations whose outputs share a single sentence count and token count, and measure what fraction of few-shot responses match those counts with 3, 10, 50, and 200 demonstrations; if the fraction approaches 100%, the claim that demonstrations cannot transfer format properties is false. Alternatively, construct a task with bimodal output lengths; LongGuide's selection step should choose no guideline there, and if the fixed min/max/average window still wins, the average-statistics assumption fails.","tokens_in":1919,"feed_emoji":"📏","tokens_out":2081,"duration_ms":109899,"temperature":0.7,"pith_summary":"The paper argues that few-shot examples—in-context learning demonstrations—cannot by themselves teach a large language model the length, format, and phrasing its long-form answers should have, because a model that does not already know the task's distribution will not reliably copy those properties from examples. To fix this, it proposes LongGuide, which reads a small set of training outputs, extracts the task's characteristic metrics (like brevity or clarity) and its sentence and token statistics, and writes these into explicit textual guidelines. The model is then prompted with the best of four combinations of these guidelines, chosen automatically. The authors report that this lifts performance by over 5% on average across seven long-form generation tasks, in both zero- and few-shot settings, and that it works for open- and closed-source models alike. If correct, this shows that explicit, sample-derived instructions are a cheap and effective complement to demonstrations for aligning LLM outputs.","feed_headline":"Explicit rules beat demonstrations for long-form LLM output","feed_subtitle":"LongGuide turns 50 training samples into task-specific guidelines that improve both open and closed LLMs.","key_machinery":"LongGuide is a guideline-generation algorithm that turns at most 50 training samples into two parallel instruction streams: Metric Guidelines (MG), natural-language descriptions of the expected quality on self-evaluated metrics selected from a pool of 27, and Output Constraint Guidelines (OCG), a single sentence specifying the minimum, maximum, and average number of sentences and tokens observed in the training outputs. The load-bearing mechanism is the automatic selection step (Step 5), which evaluates the model with MG only, OCG only, both, or neither on the training set and picks whichever maximizes ROUGE-L, so the method can adapt to a model's existing strengths and weaknesses.","core_discovery":"The paper's central claim is that ICL demonstrations are insufficient for long-form generation because of what it calls the text property transfer (PT) problem: when the language model's probability distribution over text differs from the task's true distribution, demonstrations cannot make the model consistently preserve the task's language and format properties, such as sentence count, token count, and informativeness. It proves a related remark and supports it with experiments showing that even when every demonstration shares the same token count and sentence count, fewer than 5% of generated responses match those counts. The proposed remedy, LongGuide, derives two kinds of guidelines from at most 50 training examples: Metric Guidelines (MG), which tell the model to optimize self-evaluated quality metrics, and Output Constraint Guidelines (OCG), which impose minimum, maximum, and average sentence and token counts. The best of four guideline configurations is chosen by measuring ROUGE-L on the training set. The paper reports that this raises performance by over 5% on average across seven long-form generation tasks for both ChatGPT and Mistral, in both zero- and few-shot settings.","pith_inferences":["A natural extension the paper leaves implicit: the same two-stream guideline recipe should transfer to other surface-constrained generation tasks where demonstrations alone underperform, such as code generation with length or style requirements; the OCG part in particular is model-independent and costs almost nothing to compute.","The paper's own failure case (high-variance outputs such as SWiPE) points to a concrete improvement: replacing the single average-based OCG with input-conditioned length predictions could extend the gains to tasks where a fixed min/max window cannot fit.","Because LongGuide's selection step uses ROUGE-L on the 50-sample train set as a proxy for test quality, the method inherits the usual risk of proxy-based selection; a testable extension would be to run the same selection with a learned reward model or a stronger evaluator and compare downstream performance.","The reported transfer of guidelines from a weak open model to a strong closed model suggests that guideline discovery could be separated from inference: teams with limited API budgets could amortize one-time guideline generation on cheap models and then apply the same text to expensive ones, but the paper only tests this in one direction, so the reverse pattern remains open."],"forward_implications":["LongGuide works even when its training set is just the 3–5 demonstrations used for ICL, so the method applies to tasks that lack a dedicated training split.","Guidelines learned by a weaker open-source model transfer to a stronger closed-source model, allowing cheap guideline discovery for expensive proprietary models.","LongGuide combines with automatic prompt optimizers: further optimizing the learned guidelines improves performance, especially when the detailed MG-OCG guideline is used.","The Output Constraint Guideline is especially effective on summarization, translation, and table-to-text tasks, and ablations show that removing either the token or the sentence constraint hurts performance.","The Jensen–Shannon divergence between generated and ground-truth distributions on the selected metrics correlates with ROUGE-L, linking the mechanism (better property transfer) to the reported quality gains."],"supporting_citations":[{"why":"Provides the implicit Bayesian inference account of ICL that assumes the model matches the task distribution; LongGuide's Remark B.1 contradicts this assumption, making it the theoretical target of the text property transfer argument.","marker":"Xie et al. (2021)"},{"why":"Latent-variable-model explanation of ICL that likewise assumes PM = PT, used as the foil for the claim that demonstrations cannot recover the task distribution.","marker":"Wang et al. (2024)"},{"why":"Source of 12 of the 27 metrics in LongGuide's metric pool, grounding the Metric Guideline's self-evaluation vocabulary.","marker":"Fu et al. (2023)"},{"why":"Self-consistency, used to score ground-truth outputs when computing metric scores in Step 2 and to measure property maintenance in Section 2.","marker":"Wang et al. (2022a)"},{"why":"Controllable generation work that motivates the Output Constraint Guideline's sentence- and token-level constraints.","marker":"Fan et al. (2018a)"},{"why":"Supplies the premise that LLMs behave as optimizers able to act on textual guidelines, which the prompt-based MG/OCG mechanism relies on.","marker":"Yang et al. (2024)"},{"why":"APO, the strongest prompt-optimization baseline LongGuide is compared against across the seven long-form tasks.","marker":"Pryzant et al. (2023)"},{"why":"SAMSum, the dataset used for the motivating example and the Section 2 experiments showing that ICL fails to preserve format properties.","marker":"Gliwa et al. (2019)"}],"fun_headline_variants":["Guidelines outperform demonstrations for long-form LLMs","LongGuide: 50 samples yield guidelines that boost LLMs","ICL insufficient for long-form; attribute guidelines fix it","Task rules beat demos for long-form generation"],"cache_read_input_tokens":44928,"weakest_assumption_plain":"The load-bearing premise is that a 50-sample training set faithfully represents the output length and style distribution of the whole task, so guidelines tuned on those samples will help at test time.","fun_headline_variants_meta":{"raw":{"variants":["Guidelines outperform demonstrations for long-form LLMs","LongGuide: 50 samples yield guidelines that boost LLMs","ICL insufficient for long-form; attribute guidelines fix it","Task rules beat demos for long-form generation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000522,"raw_usage":{"total_tokens":2548,"prompt_tokens":988,"completion_tokens":1560,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1496}},"tokens_in":604,"tokens_out":1560,"duration_ms":13762,"temperature":1.0,"reasoning_tokens":1496,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:46:04.356249+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scale the Section 2 property-maintenance experiment to large demonstration sets: choose one long-form task, fix demonstrations whose outputs share a single sentence count and token count, and measure what fraction of few-shot responses match those counts with 3, 10, 50, and 200 demonstrations; if the fraction approaches 100%, the claim that demonstrations cannot transfer format properties is false. Alternatively, construct a task with bimodal output lengths; LongGuide's selection step should choose no guideline there, and if the fixed min/max/average window still wins, the average-statistics assumption fails.","supporting_citations":[],"review_version":1}