{"id":"187642a5-57b1-4115-a4fe-fab9f6e9578d","arxiv_id":"2505.14157","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Prior prompt choices during reinforcement fine-tuning steer model behavior, and a null-example prompt matches or beats reasoning prompts on average.","lead":"This paper tests whether the instruction text prepended to questions during reinforcement fine-tuning changes what language models learn to do. It finds that different instructions produce different behaviors, and that an instruction asking for examples performs as well as the standard step-by-step reasoning prompt.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pPE treatment bundles instruction wording with the format-reward tag set (Sec. 3.2), so the observed post-RFT behavioral differences cannot yet be attributed to prompt semantics rather than to the 0.5 tag-matching reward.","rationale":"The reader's weakest assumption identifies exactly the gap I consider most load-bearing: the design does not separate the semantic content of the prior prompt from the format reward that changes with each pPE condition. If the format reward alone is sufficient to produce the distinct tag-structured behaviors, then the paper's framing as 'prior prompt engineering' would be misleading, and the central claim would reduce to a statement about reward design. This is more fundamental than the single-run evaluation issue, because the qualitative claim of distinct internalized styles is the paper's core contribution, and that claim requires causal attribution to the prompt wording. The paper does provide useful evidence that pPE broadly matters—all pPE variants except knowledge outperform the No PP baseline on average, and the qualitative examples show visually different response styles. Those observations are not thrown into doubt by the confound, but they cannot pinpoint the causal ingredient. The crossover test I propose directly addresses the attribution: by crossing instruction and tag/format, the test distinguishes whether the behavior follows the wording or the format constraint. If the behavior follows the wording, the paper's interpretation holds; if it follows the tag/format, the authors would need to reframe the contribution as an interaction between prompt design and reward shaping. I therefore keep the reader's CONDITIONAL verdict: the paper is valuable and the concern is addressable, but the central causal claim is not yet established as stated.","tokens_in":29202,"tokens_out":12896,"duration_ms":121451,"concrete_test":"Run a 2x2 crossover with Qwen2.5-7B under the Section 3.2 GRPO setup: (A) instruction with as the format-reward tag, and (B) instruction with as the format-reward tag. Apply the Section D.4.2 behavior classifiers to the same evaluation suites. If (A) matches the original pPE profile and (B) matches pPE, instruction wording is causal; if (A) resembles and (B) resembles , the tag/format reward is the driver. Report the two crossed profiles alongside the original two profiles.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim requires that the semantic content of the prior prompt—the instruction text—causes the distinct post-RFT behaviors. But the experimental design varies the prior prompt and the format reward jointly: Section 3.2 defines a format reward worth 0.5 that requires exactly one pair of the expected XML tags (, , , , ), and Section D.2 states this reward is 'crucial' for structured outputs. Thus the treatment 'pPE approach' is a bundle of (i) instruction wording, (ii) tag names, and (iii) the format-reward specification. The No PP baseline removes both the prompt and the format reward, so it cannot separate these components. No condition holds the instruction fixed while varying the tag/format, or fixes the tag/format while varying the instruction. Consequently, the behavioral differences in Figures 6-7 could be driven by the RL pressure to emit a specific tag token (whose name primes content), rather than by the model internalizing the instruction. The paper's own finding that the pPE model produces the fewest 'examples' behaviors (Table 9) shows that the behavior is not a simple reading of the tag, but it does not establish which bundled component is causal. Because the novelty claim (pPE as an underexplored axis) depends on the prompt, and not merely on reward design, this attribution gap is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces prior prompt engineering (pPE) for reinforcement fine-tuning, adapting five inference-time prompting strategies (chain-of-thought, plan-and-solve, program-of-thought, generated knowledge, and null-shot) into prior prompts with corresponding XML tags. It trains Qwen2.5-7B with GRPO on a math dataset under each pPE, compares to iPE-prompted baselines and a No PP RFT baseline, and evaluates on AIME, AMC, MATH-500, HumanEval+, and GPQA-Diamond. The paper reports that all pPE models improve over iPE on average, that the null-example pPE achieves the largest average gain and best GPQA performance, and that different pPE approaches induce distinct behavioral styles as measured by an LLM-based classifier. Generalization experiments on Qwen2.5-3B, Llama 3.1-8B, and Qwen2.5-Coder-7B are also reported.","tokens_in":29497,"tokens_out":10148,"duration_ms":90129,"significance":"If the causal attribution is established, the paper identifies a practical and understudied axis for RFT, with a counterintuitive result (null-example prior outperforms reasoning prior) and implications for steering model behavior and test-time compute. The paper is transparent: it ships the training script, hyperparameters, full prior-prompt templates, and qualitative examples, and it reports a negative result (iPE trends do not transfer to pPE) that is useful. However, the central claim currently rests on a confounded design and unreplicated small-benchmark differences, so the significance is conditional on additional control experiments.","major_comments":[{"comment":"The experimental design varies the prior-prompt instruction and the format reward jointly. Section 3.2 defines a format reward of 0.5 for exactly one pair of the expected XML tag (, , , , or ), and Section D.2 states this reward is 'crucial' for structured outputs. Each pPE condition therefore differs from the others both in instruction wording and in the specific tag that is rewarded, and the No PP baseline removes both components. The behavioral differences in Figures 6 and 7 could consequently be caused by RL pressure to emit a particular tag token rather than by internalization of the instruction. The reward-hacking collapse in Section 4.3 (Qwen2.5-3B with pPE producing 9-token responses that collect only the format reward; Table 6 and Figures 25–26) shows the format reward alone can shape output structure. Please add control conditions that fix the format reward and tag structure while varying only the instruction wording, or fix the instruction while varying the tag and reward.","section":"3.2, D.2"},{"comment":"All evaluations are single runs with a fixed random seed (Section D.4.1), and the benchmarks are small: AIME has 30 problems, AMC has 83, and GPQA-Diamond has 198. The headline differences are within plausible noise: the Examples pPE average (47.60) exceeds Think (46.99) by 0.61 points; AIME is tied at 20.00 for Think, Plan, and Examples; and the GPQA difference (30.81 vs. 28.28) corresponds to about 5 questions out of 198. Without repeated seeds or bootstrap confidence intervals, the superiority of the null-example pPE over the reasoning pPE, and the ranking of pPE variants generally, are not established. Please report multiple seeds or statistical uncertainty estimates for the main Qwen2.5-7B comparison.","section":"D.4.1"},{"comment":"The classification prompts for the four fundamental behaviors are swapped in Section D.4.2. Figure 18, captioned 'Subgoal Settings Classification Prompt,' asks the model to detect backward chaining and gives a backward-chaining example, while Figure 19, captioned 'Backward Chaining Classification Prompt,' asks for explicit subgoal setting and gives a subgoal example. As a result, the subgoal-setting and backward-chaining counts in Figure 6 and Table 7 measure the opposite constructs. This undermines the qualitative evidence for distinct behavioral styles and must be corrected and the classification re-run.","section":"D.4.2, Figures 18–19"},{"comment":"The adapted LLM-based behavior classifier (gpt-4.1-mini) is used to quantify behavior differences that are a central claim of the paper, but no accuracy, agreement, or calibration check is reported for the new prompts. Given the prompt-swap error above and the very small counts in the generalization tables (e.g., Table 8), the reliability of the behavior counts is uncertain. Please validate the classifier on a hand-labeled sample or report inter-annotator agreement.","section":"D.4.2"}],"minor_comments":[{"comment":"The abstract and introduction state that 'all pPE-trained models surpass their iPE-prompted counterparts,' but Table 1 shows this holds only for average performance; per-benchmark exceptions exist (e.g., HE+ Think pPE 70.10 vs. iPE 75.00; GPQA Knowledge pPE 21.72 vs. iPE 24.24). Please qualify the claim.","section":"Abstract, Introduction"},{"comment":"The abstract states that null-example pPE achieves 'the highest improvement on AIME2024,' but Table 1 shows Think, Plan, and Examples all reach 20.00 on AIME, and the No PP baseline reaches 26.67. The claim should be restricted to average performance and GPQA.","section":"Abstract"},{"comment":"The reward-level comparison in Figure 4 between No PP and pPE is confounded by different reward scales: No PP uses an accuracy reward maxed at 1.0, while pPE uses accuracy 0.5 plus format 0.5. Normalize or rescale before interpreting the lower reward level of No PP.","section":"Figure 4"},{"comment":"Section 3.2 contains several language errors, including 'math-only training datasets provides' and 'which inconclusive in implementation standards of the reward function'; these should be copy-edited.","section":"3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting question and its appendices are unusually transparent (training script, hyperparameters, full prompts, qualitative examples). The main revision must add control conditions that separate prompt wording from the format reward; without them the central attribution is not supported. The swapped classification prompts in Figures 18–19 indicate that the appendix needs careful proofreading. If the confound is addressed, the paper could be a good fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of the Taveekitworachai et al. paper. The genuinely new thing is treating the prior prompt as a systematic axis in RFT and comparing five iPE-style prompts as training-time conditions. That comparison is missing from the literature. The finding that the null-example () prompt is competitive with —and beats it on GPQA and average—is a real empirical surprise, and the behavioral classification shows the five models look different, not just perform differently. The No PP baseline is a nice control for dataset effects, and the generalization runs, including the reward-hacking failures on small models, are honest and informative.\n\nThe main problem is that the treatment is a bundle. The pPE condition varies the instruction text, the tag name, and the format reward (0.5 for exactly one matching tag pair) all at once. No condition fixes one and varies the other. So the observed behavioral differences could be driven by the RL pressure to emit a specific tag, not by the model internalizing the wording. The paper's own finding that the model produces the fewest 'examples' behaviors cuts against a simple tag-priming story, but it does not tell us whether the instruction or the reward is the active ingredient. Since the paper's novelty claim is about prompt design, this attribution gap is load-bearing, not cosmetic.\n\nThe quantitative case is also thinner than the abstract suggests. Everything is a single run with no error bars. On AIME, three pPE variants tie at 20/30; the average differences among think/plan/code/examples are around one point. Those margins would not survive adding a couple of seeds. The generalization to 3B and Llama mostly fails, which is fine, but it undercuts the phrase 'powerful axis' in the abstract.\n\nWhat the paper does well: the qualitative examples are convincing that the models differ in style in a way that matches the prompt, and the behavior classification is adapted carefully. The limitations section is candid. The training script and hyperparameters are given, so the runs are reproducible in principle. It's a solid first empirical map, not a definitive result.\n\nWorth a serious referee. I'd want the revision to add seeds/error bars, run a 2x2 isolating prompt wording from format reward (or at least a fixed-tag varied-instruction condition), and tone down the abstract. I would not cite it in my own work yet, but I'd bring it to reading group.","headline":"A genuinely new empirical axis for RFT—systematic prior prompt comparison with a surprising null-example result—but the design bundles prompt wording with format reward, so the causal claim outruns the evidence.","tokens_in":30030,"tokens_out":2776,"would_cite":false,"duration_ms":40643,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-07T15:38:42.505617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}