{"id":"50466263-fa6a-4963-ae8d-f1f54b909ea7","arxiv_id":"2502.04498","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A fully verifiable format-following dataset and a progressive SFT-plus-DPO self-improvement pipeline improve 7B LLMs' format control, with mixed out-of-domain transfer.","lead":"This paper builds a verifiable format-following dataset (VFF) where Python functions check whether outputs obey constraints, and uses it to train small 7B language models with self-generated data. The method improves format control on the dataset itself and partially transfers to an existing out-of-domain benchmark, though gains are inconsistent across models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VFF in-domain gains may reflect train/test leakage: same Alpaca question pool used for both splits, and out-of-domain IFEval transfer is inconsistent across the three trained models.","rationale":"The reader's weakest_assumption concerns training-data quality; that is a reasonable concern but partially mitigated by the quality assessment in Table 8. The more load-bearing issue is the validity of the in-domain evaluation itself. The paper's strongest quantitative evidence is the trained LLaMA-3-8B level-3 VFF score (38.36 vs GPT-4 35.31), but this metric is vulnerable to train/test overlap at the question level because VFF is constructed by instantiating constraints on a fixed Alpaca question pool (Section 3.1). If the same questions appear in both splits, the model can rely on memorized content rather than on learning to format arbitrary content, which would not generalize. The inconsistent out-of-domain transfer (LLaMA-3 IFEval unchanged, Mistral slightly worse) is consistent with this overfitting story. A concrete check of split disjointness and a retraining experiment would settle whether the VFF gains are real. This concern is distinct from, but complements, the reader's content-quality point, and it more directly tests the advertised 'substantially improves' claim.","tokens_in":16205,"tokens_out":7541,"duration_ms":79430,"concrete_test":"Check the released VFF splits for question overlap: compute the set of Alpaca instruction IDs in the 10k training and 7k test examples for each level. If any ID appears in both, report the overlap rate and retrain the three models on a split where training and test questions are fully disjoint (e.g., 26k/26k split of Alpaca). If the large VFF gains disappear or shrink substantially, the in-domain results are artifacts of leakage. Additionally, report per-constraint-type IFEval breakdowns to see whether improvements are concentrated in VFF-overlapping constraint types (JSON, word count) rather than all types.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on large VFF improvements, but the evaluation protocol may be contaminated. Section 4.1 states that for each level, 10k training and 7k test samples are curated from VFF. The dataset is constructed by pairing every Alpaca question with up to 3 randomly instantiated meta constraints (Section 3.1, Figure 3). The paper does not state that the training and test splits are disjoint at the question level. With 52k questions and only ~60 meta constraints, it is statistically likely that many Alpaca prompts appear in both splits (with different constraint combinations). If so, a model can memorize content for a question during SFT/DPO and at test only needs to re-format a memorized response to satisfy the new constraints, inflating VFF scores. This would explain why trained LLaMA-3-8B appears to beat GPT-4 on level-3 VFF, while its out-of-domain IFEval score is essentially unchanged (68.22→68.50 prompt, 77.14→77.24 instruction) and Mistral's IFEval prompt-level drops (40.85→37.50). The only clear out-of-domain gain is LLaMA-2-7B. Thus the headline claim of substantial improvement of fine-grained format following is not robustly supported unless train/test question disjointness is established and out-of-domain transfer is more consistent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VFF, a fully verifiable format-following dataset constructed by pairing Alpaca questions with up to three Python-checkable meta constraints, and a progressive self-improvement training pipeline (response sampling, verifier-based annotation, SFT and DPO, repeated from level 1 to level 3) to improve the format control of 7B-level LLMs. Experiments on three 7B models (Mistral-7B, LLaMA-2-7B, LLaMA-3-8B) report large in-domain VFF gains, mixed out-of-domain IFEval results, and some decline on InfoBench; the trained LLaMA-3-8B is claimed to outperform GPT-4 on level-3 VFF. The paper also compares Python-based verification with LLM judges, reporting that Python is more accurate, faster, and cheaper.","tokens_in":16566,"tokens_out":5978,"duration_ms":57114,"significance":"If the central claims hold, the paper makes two useful contributions: a deterministic, verifiable format-following benchmark that removes LLM-judge bias and cost, and a self-improvement recipe that lets small open-source models improve format adherence using only self-generated data. The dataset release and the Python-versus-LLM-judge comparison (Table 6) are concrete, reproducible strengths. However, the evidence for the training method's general effectiveness is currently weakened by potential train/test leakage in VFF, absence of any uncertainty quantification, and inconsistent out-of-domain transfer across the three trained models. The stated limitation that general instruction-following performance can slightly degrade is acknowledged but not resolved. These issues make the headline claims disproportionate to the current evidence, though the underlying approach remains plausible.","major_comments":[{"comment":"The construction of VFF may allow the same Alpaca question to appear in both training and test splits. Section 3.1 creates instructions by pairing each question from the 52k Alpaca set with up to 3 randomly selected meta constraints, and Section 4.1 curates 10k training and 7k test samples per level from this pool, but the paper never states that the splits are disjoint at the question level. With only about 60 meta constraints, it is statistically likely that many questions occur in both splits with different constraint combinations. If so, a model can memorize the content during training and only need to reformat it at test time, inflating VFF scores. The authors should report the degree of question-level overlap between their training and test sets, or construct the splits at the question level. Without this, the in-domain VFF improvements—including the claim that trained LLaMA-3-8B surpasses GPT-4 on level-3—are not trustworthy.","section":"Sections 3.1 and 4.1"},{"comment":"All main results are single runs without variance, confidence intervals, or significance tests. Since IFEval and InfoBench each contain roughly 500 test samples, the observed differences (e.g., LLaMA-3-8B IFEval instruction 77.14 to 77.24, Mistral prompt-level 40.85 to 37.50) are likely within noise. The paper should report standard deviations across multiple runs or bootstrap confidence intervals, and apply a paired significance test for key comparisons such as the claimed superiority of trained LLaMA-3-8B over GPT-4 on VFF level-3. Without uncertainty quantification, the mixed out-of-domain results cannot be distinguished from sampling noise.","section":"Table 2"},{"comment":"The training-data annotation step marks every response with I(y)=1 as the preferred response y_w with no check of content quality, relevance, or factual correctness. Table 8 assesses quality only on 200 responses per setting, and the observed InfoBench declines for Mistral and LLaMA-3-8B (Table 2) suggest that format-only preferences can degrade general instruction quality. The authors should quantify the content quality of the self-generated preferred responses at scale (e.g., with automatic metrics or an LLM-based relevance filter) and either filter the training data or explicitly document the trade-off. The current reliance on a single verifier pass as the sole preference signal is a core design choice that is not independently validated.","section":"Section 3.2, Training Data Annotation"},{"comment":"The in-domain VFF benchmark shares the same verifier functions that label the training data, making the large in-domain gains partly circular: the model is directly optimized on the exact functions used for evaluation. The independent out-of-domain transfer to IFEval is inconsistent: LLaMA-2-7B improves substantially (31.42 to 40.48 prompt-level; 44.96 to 54.08 instruction-level), Mistral-7B declines (40.85 to 37.50 prompt-level), and LLaMA-3-8B is essentially unchanged (68.22 to 68.50). The paper's conclusion that the method demonstrates effectiveness in format following is therefore clearly supported only for one of the three models on out-of-domain data. The authors should either temper the generalization claim or provide stronger out-of-domain evidence with uncertainty quantification.","section":"Section 4.2 and Table 2"}],"minor_comments":[{"comment":"Typo: \"filing the variables\" should be \"filling the variables.\"","section":"Section 3.1"},{"comment":"The footnote marker for the results taken from Sun et al. (2024) is missing in the table; the caption contains an incomplete reference \"Results with are from Sun et al. (2024).\"","section":"Table 2"},{"comment":"The variable placeholder is inconsistently rendered as \"[[V AR1]]\" with a space; it should be \"[[VAR1]]\" throughout.","section":"Figures 1 and 3"},{"comment":"The description of the one-shot demonstration is unclear; please specify explicitly that the demonstration is a wrong response generated by the base model and how it is incorporated into the sampling prompt.","section":"Section 5.2 and Table 4"},{"comment":"Typo: \"preferece learning\" should be \"preference learning.\"","section":"Equation (2) and following text"},{"comment":"The statement that the training method \"may slightly harm some general instruction following performance\" is important; this trade-off should be quantified in the main results (e.g., with error bars on InfoBench) rather than left only as a limitation.","section":"Limitations section"}],"recommendation":"major_revision","confidential_remarks":"The paper's strengths are the verifiable dataset and the compelling cost/accuracy comparison of Python versus LLM judges (Table 6). However, the central effectiveness claim is not yet robust: the potential train/test leakage in VFF, the complete lack of uncertainty quantification, and the inconsistent out-of-domain transfer across models leave the headline result (trained LLaMA-3-8B beating GPT-4 on VFF level-3) ungrounded. The authors should be asked to provide question-level disjointness analysis, statistical testing, and a more balanced discussion of the content-quality trade-off. With these additions the paper could be suitable, but in its current form the evidence is insufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the VFF dataset is a real asset, and the progressive SFT+DPO loop is a sensible way to use Python verifiers for training. But the headline gains are mostly in-domain, and the paper doesn't yet rule out split leakage or show consistent transfer across base models. It deserves a serious review, not a desk reject.\n\nWhat's new: a large, fully verifiable format-following dataset (about 60 meta constraints instantiated over Alpaca prompts) with Python functions instead of LLM judges. That's genuinely useful: Tables 6 and 7 make a fair point that LLM judgment is slow, costly, and inconsistent. The progressive training idea—start with single constraints, sample, verify, SFT+DPO, repeat—is clean and cheap. The strongest evidence is LLaMA-2-7B going from 31.42 to 40.48 on IFEval prompt-level, an independent benchmark. That alone shows the method can work for a weak model.\n\nSoft spots: first, the train/test split. Section 4.1 says 10k/7k samples are curated per level from VFF, but it doesn't say whether an Alpaca question is ever reused across the splits. With 52k questions and random constraint pairing, overlap is likely. If exact (question, constraint) pairs show up in both, that's leakage and would directly inflate the in-domain numbers. Even if only questions overlap, the model could memorize a response and then just wrap it in the new format; that's less damaging but still worth checking. The authors need to report whether they deduplicated or enforced a question-level split.\n\nSecond, the out-of-domain transfer is inconsistent. LLaMA-3-8B barely moves on IFEval (68.22 to 68.50 prompt), Mistral actually drops on prompt-level IFEval, and both drop on InfoBench. Only LLaMA-2 improves clearly. That tempers the claim that the method improves small LLMs generally; it may be helping models that were weaker at format control to begin with.\n\nThird, the preference pairs choose any format-passing response as the winner, regardless of content quality. The authors' own human/GPT-4o assessment on 200 samples suggests quality doesn't collapse, but it's a small sample and the concern is real that SFT/DPO might reinforce shallow but well-formatted text. Finally, no comparison to constrained decoding (e.g., outlines, xgrammar), which is the natural alternative for structured outputs. The paper mentions it but doesn't benchmark it.\n\nThese are fixable with a bit more analysis. The dataset and the core idea are a solid incremental contribution. I'd send it to review, asking for split-disjointness checks, variance estimates, and a constrained-decoding baseline.","headline":"Useful verifiable format dataset and a plausible self-improvement pipeline, but the in-domain gains need a train/test-disjointness check and the out-of-domain transfer is inconsistent across base models; worth reviewing with requested clarifications.","tokens_in":17046,"tokens_out":3710,"would_cite":true,"duration_ms":38430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fully verifiable format-following dataset lets 7B-parameter LLMs self-train with Python checkers, and the resulting model beats GPT-4 on the hardest format-control level.","keywords":["format following","verifiable evaluation","self-improvement","direct preference optimization","synthetic data","instruction following","LLM evaluation","progressive training"],"falsifier":"Run the same progressive pipeline but swap the preferred responses for deliberately nonsensical yet format-correct text, such as JSON objects filled with random tokens, while keeping the verifier labels unchanged; if the trained models still improve on VFF and IFEval, the learning signal is purely structural, but if human-evaluated relevance collapses, the format-pass-as-quality assumption is broken.","tokens_in":15960,"feed_emoji":"✅","tokens_out":5463,"duration_ms":52858,"temperature":0.7,"pith_summary":"This paper tries to show that fine-grained format control can be pulled out of general instruction following and handled by a closed loop: a small LLM generates responses, a Python function verifies the format, and the verified responses are used to retrain the LLM. The authors build VFF, a fully verifiable format-following dataset whose constraints come with exact Python checkers, then run three rounds of self-training on 7B-parameter models. Their trained LLaMA-3-8B model reaches 38.36 percent on the hardest, three-constraint level of VFF, above GPT-4's 35.31 percent, and the gains transfer to the out-of-domain IFEval benchmark. The practical point is that automatic checks can replace GPT-4-style judges for format adherence, making evaluation cheaper, faster, and free of judge inconsistency.","feed_headline":"Self-taught 8B model beats GPT-4 on format control","feed_subtitle":"A Python-checked dataset lets small LLMs train on their own outputs, no GPT-4 judges needed.","key_machinery":"The load-bearing object is VFF, a dataset of about 60 human-curated meta constraints, each carrying an executable Python boolean verifier. An instruction is formed by concatenating an Alpaca question with one to three instantiated constraints, so the correctness label is the product $I(y)=\\prod_{k=1}^{c}F_k(y)$ of the per-constraint verifier outputs. That exact binary signal makes the training loop fully automatic: it labels self-generated responses, builds SFT and DPO data, and enables progressive level-by-level training. A secondary mechanism is the one-shot wrong demonstration used during response sampling, which raises the share of verifier-passing samples from 55 to 79 percent at level 1 and thereby supplies enough positive examples for the next training round.","core_discovery":"The central claim is that format-following is a separately learnable skill that small open LLMs can acquire without human-written or LLM-generated training data. The VFF dataset defines each format constraint as a triple: a natural-language constraint with fill-in variables, candidate values for those variables, and a Python function that returns true exactly when a response satisfies the constraint. The training loop samples several responses per instruction, marks every response that passes all active checkers as preferred and every non-passing response as dispreferred, then applies SFT on the preferred responses and DPO on the pairs. Training starts with one constraint, then two, then three, re-sampling and re-annotating at each level. The paper reports that this progression raises format-following accuracy on all three tested 7B models and that the best trained model exceeds GPT-4 on the hardest VFF level.","pith_inferences":["Editorial: A natural testable extension is to use the Python verifier as a reward in reinforcement learning; the paper notes this itself, and it could push hard-level accuracy above what DPO alone achieves.","Editorial: The 60-meta-constraint pool is small and domain-specific, but the same loop should work for any constraint expressible as a decoder-verifiable property such as regexes, schema validators, or unit tests, broadening VFF into a general verifiable-instruction trainer.","Editorial: Because the verifier checks form, not meaning, the pipeline risks rewarding fluent but vacuous answers; adding a lightweight relevance filter would be a cheap robustness test.","Editorial: Since constraints are paired randomly with questions, some instructions conflict; if conflict rates rise with constraint count, there may be an upper bound on how many stacked constraints can be learned before the model is trained on contradictions."],"forward_implications":["Training with verifier-labelled self-generated data can lift 7B models' format-following accuracy close to or above frontier-model levels on the same benchmark.","Because the verifier is deterministic, evaluation and data labelling scale without API cost and without the inconsistency of LLM judges.","Progressive training from one to three constraints is needed to get usable positive samples at harder levels; skipping levels leaves the hardest accuracy near baseline.","The trained models transfer part of the gain to out-of-domain format following on IFEval, suggesting the skill is not purely memorized.","The method may slightly reduce general instruction-following performance on InfoBench, indicating a trade-off that needs to be managed."],"supporting_citations":[{"why":"Supplies the Alpaca question set that is paired with constraints to instantiate VFF instructions.","marker":"(Taori et al., 2023)"},{"why":"Provides IFEval, the out-of-domain verifiable format-following benchmark used to test transfer.","marker":"(Zhou et al., 2023b)"},{"why":"FollowBench, the prior multi-level constraint benchmark that combined LLM and Python evaluation and is extended by VFF.","marker":"(Jiang et al., 2023b)"},{"why":"DPO, the preference-optimization objective trained on verifier-built preference pairs.","marker":"(Rafailov et al., 2024)"},{"why":"InfoBench, the GPT-4-judged general instruction-following benchmark used to measure side effects.","marker":"(Qin et al., 2024)"},{"why":"Conifer, whose reported scores are used to compare against a complex-constrained instruction-following approach.","marker":"(Sun et al., 2024)"}],"fun_headline_variants":["Small LLM beats GPT-4 on format control with self-training","Python-verified dataset trains small LLMs to master formats","Verifiable format control: small LLMs surpass GPT-4","Self-taught Python-checked training gives small LLM GPT-4-level format control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method treats any response that passes the Python format check as a good response to train on, even though nothing verifies that the content is relevant, coherent, or factually right.","fun_headline_variants_meta":{"raw":{"variants":["Small LLM beats GPT-4 on format control with self-training","Python-verified dataset trains small LLMs to master formats","Verifiable format control: small LLMs surpass GPT-4","Self-taught Python-checked training gives small LLM GPT-4-level format control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000499,"raw_usage":{"total_tokens":2680,"prompt_tokens":916,"completion_tokens":1764,"prompt_tokens_details":{"cached_tokens":896},"prompt_cache_hit_tokens":896,"prompt_cache_miss_tokens":20,"completion_tokens_details":{"reasoning_tokens":1688}},"tokens_in":20,"tokens_out":1764,"duration_ms":40338,"temperature":1.0,"reasoning_tokens":1688,"cache_read_input_tokens":896,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T22:30:58.540488+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same progressive pipeline but swap the preferred responses for deliberately nonsensical yet format-correct text, such as JSON objects filled with random tokens, while keeping the verifier labels unchanged; if the trained models still improve on VFF and IFEval, the learning signal is purely structural, but if human-evaluated relevance collapses, the format-pass-as-quality assumption is broken.","supporting_citations":[],"review_version":1}