{"id":"b36142e6-7209-407d-a5d6-6746d92e0ed5","arxiv_id":"2607.05339","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"TREK uses verified teacher proposals to expand a student model's exploration support before standard GRPO refinement, improving performance on hard math and agentic tasks.","lead":"TREK is a training method that helps AI models solve harder problems by first showing them verified solutions from a teacher or from the model itself with extra context, then resuming normal reinforcement learning. It improves math and agentic task performance, especially on the hardest tasks where standard RL stalls.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"The 2× rollout-budget GRPO baseline is described in Appendix B but its results are never reported, leaving the central thesis—that the bottleneck is missing support rather than insufficient sampling—directly untested.","rationale":"The reader correctly identified the 2× rollout baseline gap as condition (1) in their rationale, but their weakest_assumption field focused on the dS reachability metric instead. I believe the 2× baseline is the more load-bearing concern because it tests the paper's central thesis directly, whereas dS is a design choice within the method that could be imperfect without invalidating the approach. The reader's verdict of CONDITIONAL with HIGH confidence is appropriate—the experimental gains are consistent and the per-task-type breakdown (Table 3) is genuinely supportive, showing gains concentrated on hardest tasks. The OPD ablation (Tables 1-2) also provides useful evidence for the forward-KL design choice. However, the confidence should be tempered by the missing 2× baseline: the paper apparently ran this configuration (it is specified in detail) but did not report results, which is a notable omission for a control that directly tests the central claim. If the 2× baseline were reported and showed TREK maintaining a clear advantage, the paper would be substantially strengthened and could move toward ACCEPT. If 2× GRPO closes most of the gap, the verdict should shift toward REJECT or major revision. The dS sensitivity concern is valid but lower-stakes: a sensitivity analysis over (α, β) would be nice but is unlikely to overturn results given that the method ranks rather than thresholds. No code or data is shipped, which limits reproducibility, but the method description is sufficiently detailed to reimplement. The self-context variant's failure-lesson memory (Appendix C) is hand-curated with only 7 of ~40 rules shown, which is a minor transparency concern but not load-bearing for the central claim since the DeepSeek-V4 variant does not depend on it.","tokens_in":18389,"tokens_out":2291,"duration_ms":115661,"concrete_test":"Run the 2× rollout-budget GRPO baseline (K=32 unaided rollouts per prompt, no teacher proposals, all other hyperparameters matched per Table 5) on AIME 2024, AIME 2025, and ALFWorld for Qwen3-8B and Qwen2.5-7B-Instruct respectively. Compare against TREK (DeepSeek-V4) and direct GRPO. If 2× GRPO recovers more than ~60% of the TREK gain on AIME (i.e., reaches ≥39.1 on AIME 2025 or ≥49.9 on AIME 2024 for the 8B model), the missing-support thesis is substantially weakened and the contribution reduces to a more expensive way to get what extra sampling provides.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's core claim is that GRPO stalls on hard prompts because the student's on-policy support does not cover reward-bearing solution modes, and that 'larger rollout groups or sharper relative advantages may still search within a narrow region of the student's current support' (§1). This is the load-bearing premise: if simply doubling the number of unaided rollouts per prompt closes most of the gap, the problem is not missing support but insufficient sampling budget—a much simpler fix requiring no teacher at all. Appendix B (Table 5) explicitly defines the 2× baseline: '32 unaided samples per prompt; no teacher proposals.' Yet no table in the paper reports its AIME or ALFWorld results. This is not a peripheral ablation; it is the single most important control for the paper's thesis. The reported gains (e.g., +3.4 on AIME 2025 for Qwen3-8B, +7.0 on ALFWorld) are attributed to support expansion via teacher proposals, but without the 2x baseline we cannot distinguish 'teacher proposals expanded the support' from 'more samples per prompt would have achieved the same.' The dS reachability metric concern raised by the reader is real but secondary: even if dS is imperfect, the method could still work. The 2x baseline tests whether the entire approach is necessary at all.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes TREK, a staged training procedure that addresses the exploration bottleneck in GRPO on hard prompts. TREK identifies prompts where the student has low unaided pass rate, queries a proposal source (external teacher or same-model with extra context) for verified solutions, selects the top-r student-proximal trajectories by a trimmed length-normalized NLL metric, applies a short forward-KL consolidation phase, and then returns to standard GRPO. Experiments span mathematical reasoning (AIME 2024/2025 across Qwen3-1.7B/8B/14B) and agentic tasks (ALFWorld, ScienceWorld with Qwen2.5-7B-Instruct), with an OPD ablation and a self-context variant that requires no external teacher. The gains are consistent and concentrate on the hardest task types, supporting the central thesis.","tokens_in":18629,"tokens_out":1833,"duration_ms":80662,"significance":"The framing of distillation as exploration support expansion rather than imitation or credit shaping is a useful conceptual contribution. The output-only proposal interface (black-box teacher compatibility) is practically valuable. The self-context variant, which uses failure-lesson memory to generate proposals from the same model, is a notable demonstration that the method does not require an external teacher. The per-task-type ALFWorld breakdown (Table 3) provides falsifiable evidence that gains concentrate where unaided exploration is weakest. The 10-seed aggregation for agentic tasks and the OPD ablation across both domains strengthen the empirical contribution.","major_comments":[{"comment":"The 2× rollout-budget GRPO baseline is defined in Appendix B (Table 5: '32 unaided samples per prompt; no teacher proposals') but its results are never reported in any table or figure. This is the single most important control for the paper's central thesis. The paper claims in §1 that 'larger rollout groups or sharper relative advantages may still search within a narrow region of the student's current support,' but this claim is asserted rather than tested. If doubling the rollout budget closes most of the gap on AIME or ALFWorld, the problem is insufficient sampling rather than missing support, and the teacher-proposal mechanism would be unnecessary. The reported gains (e.g., +3.4 on AIME 2025 for Qwen3-8B, +7.0 on ALFWorld) cannot be distinguished from 'more samples per prompt would achieve the same' without this baseline. This must be reported, at minimum on AIME 2025 and ALFWorld,to","section":null},{"comment":"§2.2, Eq. (5): The trimmed length-normalized NLL dS is the core selection mechanism, and the paper acknowledges it is 'sensitive to verbosity and surface form' (§5). The trim parameters (α=0.10, β=0.02) are heuristic with no sensitivity analysis. While the reader's concern about dS validity is noted, the more pressing issue is that no ablation over r, α, β, or τ_low is reported. Since these are the free parameters governing which proposals are retained, at least a brief sensitivity analysis (e.g., varying r from 1 to 4, or varying τ_low) would strengthen the claim that the method is robust to these choices rather than tuned to the reported benchmarks.","section":null}],"minor_comments":[{"comment":"Table 1: The OPD (self-context) row for Qwen3-1.7B on AIME 2025 reports 16.0, which is below the direct GRPO baseline of 17.5. This regression is not discussed; a brief note would help.","section":null},{"comment":"Table 3, Pick & Place row: TREK shows −3.0 relative to GRPO. While the paper notes 'little remaining headroom,' a regression on any task type deserves explicit discussion of whether this is within noise or a genuine trade-off.","section":null},{"comment":"§3.2: The statement 'OPD trails TREK (self-context) by 1.1–1.6 points on AIME 2024' is slightly imprecise; for Qwen3-1.7B the gap is 1.1 (22.4 vs 21.3), but for Qwen3-14B it is 1.6 (50.2 vs 48.6). The range is correct but the phrasing could clarify these are per-scale ranges.","section":null},{"comment":"Figure 1 captions: The y-axis labels in (a) and (b) show 'Accuracy (%)' but the values are avg@16, which is a pass-rate-based metric. Clarifying that avg@16 is the average per-problem pass rate would avoid confusion for readers unfamiliar with the convention.","section":null},{"comment":"Appendix C: The failure-lesson memory contains 'roughly forty rules' but only seven are shown. Including the full list, perhaps in a supplementary file, would aid reproducibility of the self-context variant.","section":null},{"comment":"§3.1: The learning rate for math (1×10⁻⁷, Table 5) is unusually low for GRPO post-training. A brief justification would help readers replicate the setup.","section":null},{"comment":"The paper would benefit from a brief note on computational overhead: how many additional teacher queries and forward-KL steps does TREK add per training round compared to vanilla GRPO?","section":null}],"recommendation":"major_revision","confidential_remarks":"The missing 2× baseline is the dominant issue. The paper defines it in Appendix B but omits results, which suggests either it was run and did not support the thesis, or it was not run at all. Either way, the authors must address this in revision. If the 2× baseline was run and the gap persists, reporting it would substantially strengthen the paper; if it was not run, it should be. The dS sensitivity concern is real but secondary—the method appears to work empirically, and a brief ablation would suffice. The self-context variant is the most novel and practically useful contribution and should be emphasized more in the framing."},"author_rebuttal":{"model":"glm-5.2","summary":"We thank the referee for a careful and constructive report. The referee correctly identifies two gaps in our experimental evidence: the missing 2× rollout-budget GRPO baseline results and the absence of sensitivity analysis over the selection hyperparameters. We agree with both points and will address them in the revision. Our point-by-point response follows.","responses":[{"response":"The referee is correct that this baseline is the most important control for our central thesis, and we acknowledge that defining it in Appendix B without reporting results is a significant omission. We will run the 2× rollout-budget GRPO baseline (32 unaided samples per prompt, no teacher proposals, all other hyperparameters unchanged) on both AIME 2025 (Qwen3-8B) and ALFWorld (Qwen2.5-7B-Instruct) and report the results in the revised manuscript. We will also add it on AIME 2024 if compute permits. We agree that without this comparison, the reader cannot distinguish 'missing support' from 'insufficient sampling.' Our conceptual argument for why doubling rollouts should not close the gap is that the student's failure on hard prompts is not primarily a sampling-efficiency problem but a support-coverage problem: if the policy assigns negligible probability to the correct solution mode, drawing twice as many samples from the same narrow distribution does not systematically expand the set of reachable modes. However, we recognize this is an empirical claim that must be tested rather than asserted, and we will present the data honestly regardless of outcome. If the 2× baseline closes a substantial portion of the gap, we will revise our framing accordingly and discuss what the remaining gap (if any) implies about the boundary between sampling insufficiency and support deficiency.","revision_made":"yes","referee_comment":"The 2× rollout-budget GRPO baseline is defined in Appendix B (Table 5: '32 unaided samples per prompt; no teacher proposals') but its results are never reported in any table or figure. This is the single most important control for the paper's central thesis. The paper claims in §1 that 'larger rollout groups or sharper relative advantages may still search within a narrow region of the student's current support,' but this claim is asserted rather than tested. If doubling the rollout budget closes most of the gap on AIME or ALFWorld, the problem is insufficient sampling rather than missing support, and the teacher-proposal mechanism would be unnecessary. The reported gains cannot be distinguished from 'more samples per prompt would achieve the same' without this baseline. This must be reported, at minimum on AIME 2025 and ALFWorld."},{"response":"We agree that a sensitivity analysis over the selection hyperparameters is needed and absent from the current manuscript. We will add the following experiments in the revision: (1) varying r ∈ {1, 2, 3, 4} on AIME 2025 (Qwen3-8B) and ALFWorld, holding all other parameters fixed; (2) varying τ_low around the current value of 1/8 (e.g., 1/16, 1/8, 1/4) on the same benchmarks; and (3) varying the trim parameters (α, β) over at least two alternative settings (e.g., no trimming, and symmetric trimming at 0.05/0.05) to test whether the specific trim fractions matter. We will report these in a compact table or figure in the appendix. Our expectation is that the method is most sensitive to r (too small reduces multi-mode coverage; too large risks consolidating trajectories far from the student's support) and relatively insensitive to the exact trim fractions, but we will present the data and let the reader judge. We acknowledge that the current lack of this analysis is a legitimate weakness; the trim parameters were chosen on principled grounds (removing boilerplate tokens at the low end and rare-token outliers at the high end) but were not empirically validated against alternatives, and we will correct this.","revision_made":"yes","referee_comment":"§2.2, Eq. (5): The trimmed length-normalized NLL dS is the core selection mechanism, and the paper acknowledges it is 'sensitive to verbosity and surface form' (§5). The trim parameters (α=0.10, β=0.02) are heuristic with no sensitivity analysis. No ablation over r, α, β, or τ_low is reported. At least a brief sensitivity analysis (e.g., varying r from 1 to 4, or varying τ_low) would strengthen the claim that the method is robust to these choices rather than tuned to the reported benchmarks."}],"tokens_in":18066,"tokens_out":1256,"duration_ms":86889,"standing_objections":[]},"desk_editor":{"model":"glm-5.2","letter":"The headline: TREK is a well-executed combination of known ideas that produces consistent gains across math and agentic benchmarks. The central problem it addresses—GRPO stalling on hard prompts because the student never samples reward-bearing trajectories—is real, and the framing of distillation as exploration support expansion rather than imitation is a useful conceptual contribution. But the paper has one significant gap: the 2× rollout-budget GRPO baseline is defined in Appendix B (Table 5) but its results are never reported anywhere. This is the single most important control for the paper's thesis, and its absence is the main reason I can't give an unqualified positive read.","headline":"Solid method with consistent gains, but the missing 2× rollout baseline is a real gap that needs closing before publication.","tokens_in":19234,"tokens_out":196,"would_cite":false,"duration_ms":69217,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"TREK uses teacher solutions to expand where a student model can explore, then lets reinforcement learning take over","keywords":["GRPO","exploration support expansion","forward KL","distillation","reinforcement learning from verifiable rewards","reachability","prompt routing","language model reasoning"],"falsifier":"If forward-KL consolidation of top-r verified trajectories provides no advantage over simply adding more rollout budget to GRPO on hard prompts, or if the reachability ranking by trimmed NLL performs no better than random selection among verified proposals, the core mechanism claim would be undermined.","tokens_in":18451,"feed_emoji":"🧭","tokens_out":1319,"duration_ms":127513,"temperature":0.7,"pith_summary":"The paper argues that when GRPO-style reinforcement learning stalls on hard reasoning prompts, the bottleneck is not reward sparsity but missing exploration coverage: the student never samples reward-bearing trajectories in the first place. TREK addresses this by treating distillation as exploration support expansion rather than imitation. The method identifies prompts where the unaided student has very low pass rate, queries a proposal source (an external teacher, or the same model given extra inference-time context) for verified solutions, keeps only the top-r solutions ranked by a trimmed length-normalized negative log-likelihood metric that approximates how close each trajectory is to the student's current support, applies a short forward-KL update to pull those verified solution modes into the student's sampling distribution, and then returns to ordinary on-policy GRPO. The central mechanism is the separation of prompt-level proposal availability (which prompts are hard enough to warrant teacher help) from trajectory-level reachability (which verified solutions the student can actually internalize). The paper shows consistent gains across math reasoning (AIME 2024/2025) and agentic tasks (ALFWorld, ScienceWorld), with the largest improvements concentrated on the hardest task types where unaided exploration is weakest.","feed_headline":"Distillation as exploration: teacher solutions expand where RL can search","feed_subtitle":"TREK routes hard prompts to verified teacher trajectories, consolidates the closest ones via forward KL, then returns to on-policy RL — with","key_machinery":"trimmed length-normalized NLL (dS) as a reachability proxy for ranking verified teacher trajectories by proximity to the student's current policy support","core_discovery":"The paper's central claim is that forward-KL consolidation of verified, student-proximal teacher trajectories is a more effective support-expansion mechanism than on-policy distillation on the same trajectories, because forward KL directly penalizes the student for assigning low probability to verified proposal modes while OPD-style supervision only reshapes credit on trajectories the student already samples. This is supported by an ablation where replacing forward-KL consolidation with OPD-style supervision weakens performance by 1.1-2.9 points across AIME benchmarks and 3-5 points on agentic tasks. The paper also demonstrates that the proposal source need not be an external teacher: a self","pith_inferences":["The trimmed NLL reachability metric is essentially a heuristic proxy for learnability; a more principled measure of whether a trajectory will produce stable gradients after consolidation could improve selection quality, especially for trajectories that are close in surface form but structurally distant in reasoning pattern.","The staged schedule (mine, propose, consolidate, refine) resembles curriculum learning but operates on the student's support geometry rather than example difficulty; this suggests a connection to mode-covering vs. mode-seeking divergence choices in distributional alignment, where forward KL is mode-covering and reverse KL is mode-seeking.","If the self-context variant's failure-lesson memory is itself learnable online during GRPO, the proposal source could improve in tandem with the student, potentially reducing the gap between self-context and external-teacher variants over training."],"forward_implications":["If TREK's mechanism is correct, any on-policy RL method that depends on the student already sampling useful trajectories (DAPO, GSPO, PPO-based variants) could benefit from the same staged proposal-selection-then-refinement recipe, since the exploration bottleneck is structural rather than GRPO-specific.","The self-context variant suggests that inference-time reasoning enhancements (reflection, search, failure lessons) can be folded back into the deployment model's unaided policy through training, rather than requiring permanent test-time overhead.","The finding that gains concentrate on hardest task types implies that support-expansion methods should be evaluated on difficulty-stratified benchmarks rather than aggregate scores, which can mask where the mechanism actually operates.","The verified-only constraint is conservative; relaxing it to include near-miss teacher trajectories could broaden exploration further if the student has sufficient capacity to internalize them without instability."],"fun_headline_variants":["Forward KL distillation expands student support for RL on hard prompts","TREK uses forward-KL distillation to broaden RL exploration support","Distillation for exploration: Forward KL expands RL search space","Verified teacher trajectories pull RL into hard-prompt solution modes","Forward-KL consolidation of teacher solutions expands RL exploration"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The method depends on the trimmed length-normalized NLL being a good proxy for whether a verified teacher trajectory is genuinely internalizable by the student. This metric is sensitive to verbosity and surface form, and the trim parameters are heuristic; if it misranks trajectories, the consolidation phase could inject unstable gradients or waste capacity on trajectories that look close but are not actually learnable.","fun_headline_variants_meta":{"raw":{"variants":["Forward KL distillation expands student support for RL on hard prompts","TREK uses forward-KL distillation to broaden RL exploration support","Distillation for exploration: Forward KL expands RL search space","Verified teacher trajectories pull RL into hard-prompt solution modes","Forward-KL consolidation of teacher solutions expands RL exploration"]},"model":"glm-5.2","effort":"high","cost_usd":0.0,"raw_usage":{"total_tokens":1190,"prompt_tokens":672,"completion_tokens":518,"prompt_tokens_details":null},"tokens_in":672,"tokens_out":518,"duration_ms":24753,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-07T16:14:00.952199+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If forward-KL consolidation of top-r verified trajectories provides no advantage over simply adding more rollout budget to GRPO on hard prompts, or if the reachability ranking by trimmed NLL performs no better than random selection among verified proposals, the core mechanism claim would be undermined.","supporting_citations":[],"review_version":1}