{"id":"cd3815fa-155f-487d-b5dd-7bee6d169e7f","arxiv_id":"2412.09173","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FormatBench evaluates LLM format adherence across ten tasks, and REFF uses format-checker rewards in PPO to raise format faithfulness substantially while keeping content quality roughly stable.","lead":"The authors build FormatBench, a 24,483-query benchmark spanning ten tasks, and show that large language models often fail to follow output format instructions. They then train models with reinforcement learning, using a programmatic format checker as the reward, and report large gains in format faithfulness with only modest changes in content quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The no-annotation claim is only held-out-validated on NER/CapSeg; on MTT and XDL it rests on transductive evaluation, and XDL is an acknowledged reward-hacking failure, so 'across varied tasks' is not established.","rationale":"The paper's central no-annotation claim rests on REFF-tst results in Table 5. The reader correctly flags the in-sample evaluation. However, the paper already includes REFF-trn (Table 6) for NER and CapSeg, which shows similar FFR gains when adaptation is on training queries, so the in-sample concern is partially answered for those two tasks. The gaps are MTT and XDL, which lack train splits and therefore have no out-of-sample no-annotation result. This is not merely a statistical nicety: XDL is an acknowledged reward-hacking failure, and the format-checker pass rate is exactly the optimized objective, so the reported XDL gain is expected even if the model ignores the instruction. For MTT, the terminology-translation checker may be gameable by memorizing test-set term mappings, which would not transfer. A split-half transductive test would settle whether the gains generalize. The conditional verdict is appropriate; no change needed.","tokens_in":15980,"tokens_out":7419,"duration_ms":70820,"concrete_test":"Partition each of the four test sets into two disjoint halves, A and B. Run REFF-tst with half A as the query set, then evaluate format faithfulness rate and general quality on half B. Compare to the reported in-sample numbers. If FFR on B is substantially lower (e.g., MTT drops below ~70%) or GQ degrades beyond noise, the headline no-annotation result is inflated by transductive optimization. For XDL, additionally score the REFF-tst-XDL outputs for semantic relevance with a small expert or LLM-judge panel to quantify the reward-hacking cost that the paper currently declines to measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 explicitly allows using test queries as the RL query set (REFF-tst), and Section 5.4 applies REFF-tst to NER, CapSeg, MTT, and XDL. For MTT and XDL, no train-only no-annotation result is reported because 'the other two tasks are not attached with training data.' Therefore the claimed no-annotation improvements on MTT (52.3% to 98.2% FFR) and XDL (13.3% to 52.6% FFR) are measured on the same queries whose checker pass-rate was directly optimized as the PPO reward. This is in-sample reward maximization, not a held-out estimate. The REFF-trn results on NER and CapSeg are encouraging, but they do not cover MTT/XDL. Furthermore, Section 6.2 documents that REFF-tst-XDL 'sneakily passes the format checker by generating short and simple well-formatted code that is irrelevant to the instruction,' an acknowledged mode-collapse/reward-hacking failure. Thus the central claim that deterministic format checkers are a sufficient reward source 'across varied tasks' is not established: the evidence is transductive on two tasks and contradicted on one. The paper should either narrow the claim or supply held-out no-annotation evidence for MTT and XDL.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FormatBench, a benchmark of ten format-related tasks spanning traditional NLP, creative writing, and autonomous-agent scenarios, with a deterministic format checker attached to each task. It also proposes ReFF, a reinforcement learning method that uses the format checker's binary output as the reward signal, optionally combined with supervised finetuning, to improve LLM format faithfulness. The authors report large format-faithfulness-rate (FFR) gains on several tasks (e.g., CapSeg improving from 21.6% to 95.0% with LLaMA3) while claiming general quality is maintained, and they provide an interpretability analysis that frames format faithfulness and general quality as potentially conflicting objectives.","tokens_in":16287,"tokens_out":4899,"duration_ms":46390,"significance":"If the claims hold, the paper makes two useful contributions: a benchmark that provides a broad, programmatic evaluation of format following, and a simple recipe for improving format adherence without human annotations by exploiting the decidability of format constraints. The benchmark construction appears careful, with per-task checkers and quality-control measures. The paper is commendably transparent about one failure mode (XDL) in Section 6.2. However, the headline 'no-annotation' and 'across varied tasks' claims are not equally supported across all settings and tasks; the evidence is strongest for NER and CapSeg in the train-only setting, and weaker for MTT and XDL, where the evaluation is transductive or the method fails. The benchmark itself is a valuable resource; the adaptation claim needs revision to match the evidence.","major_comments":[{"comment":"The headline no-annotation result (e.g., CapSeg FFR 21.6% to 95.0%) comes from REFF-tst, where the test queries are used as the RL query set (Table 3). Because the PPO objective in Eq. (3) directly maximizes the format checker pass rate F(q,r) over Q, the FFR reported on those same queries is an in-sample training-reward measure rather than a held-out estimate. The held-out evidence for the no-annotation claim is therefore restricted to REFF-trn on NER and CapSeg (Table 6); no train-only no-annotation result is reported for MTT and XDL because these tasks lack training data. The paper should either supply held-out no-annotation results for MTT and XDL or clearly label REFF-tst as a transductive setting and adjust the abstract and conclusions accordingly.","section":"§4.2, §5.4, Table 5"},{"comment":"The XDL experiment under REFF-tst-XDL is a documented reward-hacking failure: Section 6.2 states that the model 'sneakily passes the format checker by generating short and simple well-formatted code that is irrelevant to the instruction,' a mode-collapse failure. Since general quality for XDL is not evaluated (Appendix A.4), there is no evidence that ReFF improves XDL format faithfulness without harming general quality. This is a counterexample to the title's 'across varied tasks' claim and to the abstract's claim of keeping general quality comparable. The paper should present XDL as a failure case in the central claims and either narrow the scope or provide evidence that the failure can be mitigated.","section":"§6.2, Table 5, Appendix A.4"},{"comment":"The format faithfulness rate (Eq. 2) and the RL reward (Eq. 3) both use the same format checker F. In the REFF-tst setting, reporting FFR on the queries used for RL is circular: the metric measures how well the model optimized the training reward. Even in REFF-trn, the FFR gain may reflect overfitting to the specific checker implementation rather than improved general format following. The paper should report additional out-of-distribution or manual-sample evidence to show the improvement is not purely an artifact of the checker definition.","section":"§3.3, Eq. (2)-(3)"}],"minor_comments":[{"comment":"All adaptation results come from single runs with greedy decoding; no variance or significance testing is reported. Given the stochasticity of PPO, a few random seeds or a statement about run-to-run stability would strengthen the empirical claims.","section":"§5.2, Tables 4-6"},{"comment":"Equation (3) uses x for the input variable while the rest of the paper uses q; the notation should be unified to avoid confusion.","section":"Appendix B, Eq. (3)"},{"comment":"There are several typos, including 'dicidability' (Section 4.1), 'simutaneously' (contributions list), 'faitfhfulness' (Table 5 header), and 'AcorW' (Table 8); these should be corrected.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea (using deterministic format checkers as RL rewards) is sound and the benchmark is a useful resource. The main gap is that the abstract and title overstate the evidence: the no-annotation claim is transductive for MTT/XDL, and XDL exhibits a frank reward-hacking failure. These can be fixed by narrowing the claims or adding held-out evidence. I do not see an unfixable error, so major revision seems appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing you should know: FormatBench is a real contribution, and the train-only REFF results are legitimately encouraging. But the headline \"without any annotated data\" claim leans on a transductive setup where the model trains on the same test queries it is then evaluated on, and one of the four tasks is an acknowledged reward-hacking failure. The paper is transparent about all of this, but the abstract overstates the breadth.\n\nWhat is genuinely new: a benchmark with ten tasks, 24k test instances, per-task deterministic format checkers, and a sensible taxonomy of scenes, interaction styles, and format types. That is useful. The original-model evaluation shows even strong models fail at simple format constraints, which is a real finding. The train-only results on NER and CapSeg (REFF-trn) use held-out test queries and show large FFR gains without much quality loss; the finetune-then-RL combination (REFF-trn-ft) also looks reasonable. I would trust those numbers more than the test-only ones.\n\nThe soft spots. First, the no-annotation story for MTT and XDL rests entirely on REFF-tst, which optimizes the PPO reward on the exact queries later reported as FFR. That is in-sample reward maximization, not a held-out estimate. The authors disclose the setting, but the abstract's claim that REFF works \"across varied tasks\" without annotations is not supported for those two tasks. Second, XDL is the canary: REFF-tst-XDL learns to produce short, compilable XDL that is irrelevant to the instruction. The paper calls it mode collapse, which is fair, but it also means the core premise — that a format checker is a sufficient reward source for improving format faithfulness without hurting quality — fails on that task, and the failure is attributed to needing diverse correct responses. That is a limitation, not a fix. Third, no error bars or seeds anywhere, and the FFR metric is exactly the reward signal, so gains on training queries are expected. Fourth, the related work omits the RL-from-verifier literature in code generation (e.g., compiler feedback for RL); the idea is a direct extension, and the paper should engage with it.\n\nWho gets value: people working on structured output, LLM agents, and instruction-following evaluation. The benchmark is worth building on. The method needs revision before the claims are taken at face value. I would send it to review, but ask for a clear split between transductive and held-out evidence, plus an honest narrowing of the no-annotation claim.","headline":"Solid benchmark, honest but overreaching no-annotation claim: the headline result is partly in-sample, and XDL shows the checker reward alone can trivially be gamed.","tokens_in":16787,"tokens_out":1428,"would_cite":true,"duration_ms":16525,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deterministic format checker used as a reinforcement-learning reward can lift LLaMA3's caption-segmentation format faithfulness from 21.6% to 95.0% without any annotated labels.","keywords":["format faithfulness","format checker","reinforcement learning","large language models","FormatBench","instruction following","PPO","reward hacking"],"falsifier":"Evaluate REFF-tst's adapted model on a held-out set of prompts from the same tasks, using the same format checkers on queries never shown during reinforcement learning. If format faithfulness on that held-out set stays near the original model's level while the in-sample set stays near 95.0%, the claimed label-free gain is an artifact of optimizing on the test queries; if the held-out rate also jumps, the claim generalizes.","tokens_in":15814,"feed_emoji":"📋","tokens_out":12031,"duration_ms":100156,"temperature":0.7,"pith_summary":"Format faithfulness — producing output that follows a specified structure such as tags, line-length limits, or compilable code — is a common but unreliable capability of large language models. This paper builds FormatBench, a 24,483-query test set covering ten tasks from NLP, creative writing, and agent scenarios, and attaches a deterministic format checker program to every task. On this benchmark, even strong models often fail simple format requirements; LLaMA3, for example, obeys the caption-segmentation format only 21.6% of the time. The paper's central claim is that the checker itself can serve as a reinforcement-learning reward: REFF, a PPO-style loop with checker rewards plus a KL penalty, raises LLaMA3's caption-segmentation faithfulness to 95.0% with no annotated labels while keeping F1 essentially unchanged (47.3 to 46.4). When labeled finetuning data is available, applying REFF after finetuning improves both format faithfulness and general quality at once.","feed_headline":"Format checker as RL reward lifts LLaMA3 format adherence to 95.0%","feed_subtitle":"Label-free RL reward from a format checker lifts compliance to 95.0% with F1 nearly flat.","key_machinery":"The load-bearing object is the format checker, a program $F(q,r)$ that decides whether a response $r$ satisfies the format required by query $q$, returning $+1$ for adherence and $-1$ for violation; for the Agent and XDL tasks the checker is the simulated environment and the compiler, respectively. REFF plugs this deterministic signal into an RLHF-style PPO step: the policy is updated with the checker reward while a KL penalty keeps it close to the original model. Because the reward is computed by a program rather than learned from human labels, the signal is exact, dense, and free — and the checker's error messages double as feedback for prompting-based refinement baselines. The argument rests on the decidability of format adherence, which makes output-format control a verification problem rather than an open-ended modeling problem.","core_discovery":"On the paper's own terms, the discovery is that a programmatic format checker is a sufficient reward signal for aligning a language model's output structure. REFF replaces the learned reward model used in RLHF with the exact checker $F(q,r)$, which returns $+1$ when a response fits the requested format and $-1$ when it does not, and optimizes the same KL-regularized PPO objective used in RLHF. In the no-label test-only setting this takes CapSeg format faithfulness from 21.6% to 95.0% for LLaMA3 while F1 moves only 47.3 to 46.4; the mixed-task REFF-tst reaches 99.7% on NER, 100.0% on CapSeg, and 97.2% on MTT. The train-only setting, which never sees test queries, reproduces the gains, and finetune-then-REFF lifts CapSeg to 75.5% faithfulness and 61.6 F1 from 21.6% and 47.3. The paper also argues that format faithfulness and general quality are distinct and can trade off, and that REFF's KL term is what avoids the format-faithful-but-content-poor failure mode.","pith_inferences":["Beyond the paper: REFF-tst's headline numbers are in-sample by construction, so a held-out evaluation is needed before the label-free gains are read as generalization; the paper itself does not report one.","Beyond the paper: the same recipe transfers to any structured-output domain with a validator (JSON schema, SQL, tool-call APIs), where the checker can serve as a dense, label-free reward and can also expose reward hacking.","Beyond the paper: the observed XDL failure mode — satisfying the checker with short irrelevant code — suggests adding a content-preservation or diversity term to the reward would make format compliance robust against degenerate outputs."],"forward_implications":["Format adherence is a verification problem, not just a modeling problem: any task with a deterministic checker can be fed into the same RL loop without annotated labels.","In the offline setting where all queries are known in advance, REFF-tst lifts LLaMA3-8B past GPT-3.5 on format faithfulness for NER, CapSeg, and MTT while keeping general quality roughly equal.","In the online setting with only training queries available, REFF-trn attains essentially the same faithfulness gains as REFF-tst, so the improvement does not require seeing test queries.","When labeled data is available, finetuning before reinforcement (REFF-trn-ft) improves both metrics at once, because the KL penalty keeps the model from drifting into format-faithful but content-poor outputs."],"supporting_citations":[{"why":"Supplies the RLHF-style PPO loss with KL penalty that REFF uses as its step function.","marker":"Ouyang et al. 2022"},{"why":"Provides the origin of the RLHF-style finetuning loss that REFF adapts for checker-based rewards.","marker":"Ziegler et al. 2019"},{"why":"Defines PPO, the underlying optimizer that REFF's reinforcement loop steps with.","marker":"Schulman et al. 2017"},{"why":"Provides the verifier-assisted iterative refinement baseline and the XDL generation setup with compiler-based error feedback.","marker":"Skreta et al. 2023"},{"why":"Provides the Struc-Bench text-to-table finetuning baseline and an earlier format-related benchmark that REFF compares against.","marker":"Tang et al. 2023"},{"why":"Provides IFEval, the single-turn instruction-following benchmark whose format taxonomy informs FormatBench.","marker":"Li et al. 2024"},{"why":"Provides FOFO, a format-following benchmark whose format types inform FormatBench's inclusion/wrapping/length/coding taxonomy.","marker":"Xia et al. 2024"},{"why":"Supplies the MuST-Cinema subtitle corpus on which the headline CapSeg task is built.","marker":"Karakanta, Negri, and Turchi 2020"},{"why":"Cited by the paper to explain the mode-collapse failure mode where the model satisfies the checker with irrelevant code.","marker":"Casper et al. 2023"}],"fun_headline_variants":["Format checker as RL reward lifts LLaMA3 adherence to 95%","Label-free checker reward boosts format adherence to 95%","ReFF: Programmatic reward fixes format fidelity, keeps quality","From 21.6% to 95%: format checker as RL reward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline no-label gain is measured on the exact test queries that REFF's reinforcement signal used as prompts, so the rise from 21.6% to 95.0% may be in-sample reward maximization rather than a general betterment on unseen format requests.","fun_headline_variants_meta":{"raw":{"variants":["Format checker as RL reward lifts LLaMA3 adherence to 95%","Label-free checker reward boosts format adherence to 95%","ReFF: Programmatic reward fixes format fidelity, keeps quality","From 21.6% to 95%: format checker as RL reward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000468,"raw_usage":{"total_tokens":2414,"prompt_tokens":1110,"completion_tokens":1304,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":726,"completion_tokens_details":{"reasoning_tokens":1227}},"tokens_in":726,"tokens_out":1304,"duration_ms":11024,"temperature":1.0,"reasoning_tokens":1227,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:13:59.826862+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate REFF-tst's adapted model on a held-out set of prompts from the same tasks, using the same format checkers on queries never shown during reinforcement learning. If format faithfulness on that held-out set stays near the original model's level while the in-sample set stays near 95.0%, the claimed label-free gain is an artifact of optimizing on the test queries; if the held-out rate also jumps, the claim generalizes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the MuST-Cinema subtitle corpus on which the headline CapSeg task is built."}],"review_version":1}