{"id":"07d9a5e8-a50c-42fd-9277-342b7558f1f4","arxiv_id":"2608.07068","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reconstructing the exact model invocation state for each sampled action, instead of flattening the whole interaction, lets teacher distillation guide compact-memory agents without state mismatch.","lead":"MemOPD is a training method that makes on-policy distillation valid for long-horizon agents that rewrite their memory between steps, by reconstructing the exact state each sampled action was generated in. It improves answer F1 over plain PPO by up to 416% relative at the longest tested horizon and shows that state alignment alone accounts for a 7% gain over teacher scoring on a flattened transcript.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 7.0% matched-control effect—the cleanest evidence that state alignment matters—rests on one unnamed teacher and no significance test; its portability is the main load-bearing uncertainty.","rationale":"I read the paper in good faith. The formalisation in §3.3 and the RCE criterion in Eq. 5 are crisp, and Table 3's numerical-floor match is strong internal evidence that MemOPD does what it says. The criticism is not that the method is internally inconsistent; it is that the only direct measurement of the value of state alignment is one matched pair of models. The teacher identity is essential because Eq. 7 assumes a shared vocabulary; without naming the teacher, the reader cannot even check whether this assumption is satisfied. A second teacher run would settle whether the 7.0% is a robust effect. This agrees with the reader's weakest assumption, so the conditional verdict should stand unchanged.","tokens_in":15221,"tokens_out":8973,"duration_ms":87398,"concrete_test":"Require the authors to name the frozen 7B teacher and rerun the matched Q2 control (PPO vs persistent-history OPD vs MemOPD) with a second teacher of a different size or model family, using a shared tokenizer if available or an explicit vocabulary-projection layer if not. Keep the same five seeds, initialization, data order, and evaluation protocol. Report the paired F1/EM differences, standard errors, and a paired test (e.g., bootstrap or t-test on seed-level scores). If the 7.0% MemOPD-over-persistent-history F1 advantage is not reproduced within error, or if it is not distinguishable from zero, the state-alignment benefit is setup-specific rather than a general property of OPD for compact-memory agents.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's conceptual claim (state alignment is necessary for valid OPD under context rewriting) is well supported by the RCE audit: §4.4 shows reconstructed packing reproduces independent-invocation logits at p99 |Δlog p| ≈ 3e-5, whereas persistent-history scoring changes top predictions at 651 positions and falsely clips 13.29% of ratios. The load-bearing uncertainty is the empirical payoff of that alignment. The matched control in §4.5 is the only direct comparison: persistent-history OPD improves F1 over PPO by 5.6%, and MemOPD adds another 7.0%. This effect is reported without standard errors or a significance test across the five seeds, and it uses a single student (Qwen2.5-3B) plus a frozen 7B teacher that is never named in §3.4 or §4.1. Moreover, Eq. 7 explicitly requires the teacher and student to share a tokenizer and vocabulary V; if the unnamed teacher is not same-family, that condition is nontrivial and possibly unmet. If the 7.0% alignment advantage does not survive a different teacher or a proper paired test, the central empirical claim shrinks to a reproducibility artifact. The 416.2% headline should not be the basis for the paper's value; it is a relative gain over a very low absolute PPO baseline. The conceptual method may still be right, but the evidence for its practical benefit is currently one-pair and unquantified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that on-policy distillation for agents with compact memory is valid only if the teacher scores each sampled action under the exact model-invocation state (tokens, positions, causal visibility, prediction position) in which the action was generated, not under a flattened persistent transcript. MemOPD records each invocation, separates the sampled occurrence of a response from later context copies, reconstructs positions and visibility, and packs the invocations under a shared stable prefix for batched training. The training objective is PPO on the sampled-action mask plus a full-vocabulary reverse-KL teacher term. The paper's main evidence is (i) a rollout-context-equivalence (RCE) audit showing packed logits match independent-invocation logits at p99 |Δlog p| ≈ 3.4e-5, whereas persistent history changes 651 top predictions and falsely clips 13.29% of ratios; (ii) a matched Q2 control where persistent-teacher OPD improves F1 over PPO by 5.6% and MemOPD adds 7.0%; (iii) end-to-end gains on Q2/Q8/Q16 (F1 up to 416.2% relative to PPO) and Wiki-RAG (F1 +7.4%), plus up to 1.63× actor speedup from packing.","tokens_in":15438,"tokens_out":5007,"duration_ms":43758,"significance":"If the empirical claims hold, the paper makes a useful conceptual contribution: it distinguishes action provenance from training-state validity and provides a concrete compiler-style reconstruction plus a verification criterion (RCE) that is checked against independent execution. The matched Q2 control and the RCE audit are well designed and give the paper internal credibility. The method is also practically relevant because it makes dense teacher supervision usable under memory rewriting while preserving task-level PPO, and the public code and machine-auditable RCE are strengths. The main uncertainty is not the mechanism but the breadth of the empirical support: the alignment advantage is demonstrated with a single unnamed teacher and without uncertainty quantification, and the transfer to longer horizons rests on the same Q2-trained teacher signal.","major_comments":[{"comment":"The teacher is only described as \"the frozen 7B teacher\" in the models and training paragraph, but its architecture and tokenizer are not given, although Eq. (7) requires the teacher and student to share vocabulary V. If the unnamed 7B teacher is not from the Qwen2.5 family, the shared-vocabulary condition is nontrivial and may be violated, which would make the reverse-KL term ill-defined or inconsistent across positions. Please name the teacher, confirm the tokenizer and vocabulary match, and ideally report a sensitivity check with at least one different teacher family or size; this is load-bearing because the portability of the alignment benefit is the main empirical claim.","section":"Section 4.1 and Section 3.4, Eq. (7)"},{"comment":"The matched-control claim that state alignment \"improves F1 by 7.0%\" is reported as a point improvement with no standard errors, per-seed values, or paired significance test. Since this 7.0% is the cleanest evidence that the alignment mechanism, rather than teacher guidance in general, drives the gains, the paper should report the five-seed distribution and a paired test, or at least confidence intervals; without this, the improvement could be within seed noise. The same applies to the 5.6% persistent-teacher gain over PPO in the same figure.","section":"Section 4.5, Figure 3a"},{"comment":"The transfer claims for Q8/Q16 and Wiki-RAG are presented as evidence that the alignment benefit generalizes, but the teacher is selected on Q2 only and the absolute PPO baseline at Q8/Q16 is near floor (F1 0.655 and 0.549), making the 283.2% and 416.2% relative gains inflated and not directly informative about state alignment. Please provide at least one longer-horizon matched control or a per-horizon teacher-state audit, and report absolute improvements or effect sizes rather than only relative percentages; without such evidence, the central empirical payoff of alignment at longer horizons remains under-supported.","section":"Sections 4.2 and 4.3"}],"minor_comments":[{"comment":"Report the hyperparameters α, β, λ, entropy coefficient, GAE parameters, and PPO clipping ϵ; the text mentions λ = 0.02 but gives no values for the other coefficients in Eqs. (8) and (11).","section":"Section 4.1"},{"comment":"The persistent-history row reports p99 |Δlog p| = 1.7738 while the introduction states 1.774; please align the precision or explain the difference.","section":"Section 4.4 and Table 3"},{"comment":"Add error bars or per-seed scatter points to the bar chart; the displayed values (0.660, 0.689, 0.758, 0.813, 0.859, 0.919) have no indication of variance, which is especially important for the 7.0% alignment comparison.","section":"Figure 3a"},{"comment":"The caption text \"Peak context is in 102 tokens\" should read \"10^2 tokens\"; similarly, Table 2's \"105\" should be \"10^5\" to avoid ambiguity.","section":"Table 1 and Table 2 captions"}],"recommendation":"major_revision","confidential_remarks":"The paper's core mechanism is sound and the RCE audit is a genuine strength; there is no circularity issue because the teacher is an external frozen model. The main risk is the breadth of the empirical evidence for the alignment benefit: one unnamed teacher, no significance test on the 7.0% matched-control effect, and only relative percentages for the long-horizon transfer. These are fixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has one genuine contribution: it identifies that on-policy distillation for compact-memory agents requires the teacher to score each action under the exact state where the student sampled it, and context rewriting breaks that equivalence. The proposed fix—record each invocation, restore token positions and causal visibility, and pack reconstructed calls under a shared prefix—is simple and mechanically sound. The rollout context equivalence audit is the best part: reconstructed packing matches independent-invocation logits at p99 |Δlog p| ≈ 3.4e-5, while persistent-history scoring shifts top predictions at 651 positions and falsely clips 13.29% of ratios. That is concrete evidence the problem is real.\n\nThe matched Q2 control is the right experiment, and I agree with the conditional reading. Persistent-history teacher scoring improves F1 over PPO by about 5.6%, and state-aligned scoring adds another 7.0%. But that 7.0% is the load-bearing empirical result, and it currently rests on one 7B teacher the paper never names, one 3B student, and no significance test across the five seeds. A paired test and a second teacher/tokenizer family would settle it. The 416.2% headline is real arithmetic but the baseline is very low (Q16 F1 ≈ 0.55), so it should not carry the paper's weight.\n\nThe other soft spots are reporting gaps: no commit hash, datasets not released, and Eq. 7 assumes teacher and student share a tokenizer/vocabulary—the paper should state whether that assumption holds for the actual teacher. None of these are fatal; they are exactly what a referee should ask for.\n\nI'd send this to review. The conceptual claim is precise, the audit is reproducible, and the method is likely to be useful to anyone training long-horizon agents with compact memory.","headline":"A genuine state-alignment fix for on-policy distillation in memory-rewriting agents; the RCE audit is strong, but the 7% matched-control gain needs a named teacher and a significance test.","tokens_in":16048,"tokens_out":2905,"would_cite":true,"duration_ms":24374,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on-policy distillation for compact-memory agents is valid only when the teacher scores each sampled action under the exact invocation state that produced it, and that MemOPD's reconstruction-and-packing framework…","keywords":["on-policy distillation","memory state alignment","compact memory","long-horizon agents","proximal policy optimization","knowledge distillation","context rewriting","rollout context equivalence"],"falsifier":"Train the same 3B student with the same PPO and OPD coefficients but with a 1B teacher, or a teacher with a different tokenizer, on Q2 and evaluate Q16; if the state-alignment advantage over persistent-history scoring disappears or reverses, the identified mechanism is not the cause.","tokens_in":14954,"feed_emoji":"🧠","tokens_out":7444,"duration_ms":57565,"temperature":0.7,"pith_summary":"This paper is trying to establish that on-policy distillation for agents with compact memory is only valid if the teacher scores each student action under the exact state in which the action was sampled. Compact memory rewrites context between model calls, so flattening the interaction into one history changes token positions and causal visibility; an action can remain on-policy by provenance but be scored under a state the student never visited. MemOPD records every invocation, separates sampled responses from later context copies, and packs reconstructed calls with a shared task prefix. The paper verifies with rollout context equivalence that packed training reproduces independent invocation logits, and reports F1 gains over PPO that grow with horizon, up to 416.2% on Q16.","feed_headline":"State-aligned distillation lifts F1 up to 416% over PPO","feed_subtitle":"Restoring each call's positions and visibility lets a teacher score actions under the true rollout state.","key_machinery":"The central mechanism is memory state alignment. A memory state is the complete tokenized input to one model call together with token positions, causal visibility, and the prediction position for each sampled action token; identical decoded text can correspond to different memory states. The reconstruction compiler records exact token IDs from rollout, restores each call's original positions and visibility, duplicates a retained response once as a sampled action and once as later context, and packs reconstructed calls under one shared stable prefix $q$ with attention blocked between private blocks. The action mask $m^{\\mathrm{act}}$ marks which physical positions are policy decisions. Rollout context equivalence (RCE) checks that packed full-vocabulary logits match independent invocation logits within a numerical tolerance $\\epsilon_{\\mathrm{num}}$.","core_discovery":"The central claim is that standard teacher distillation, which assumes the autoregressive prefix used to generate an action remains unchanged when the action is scored, breaks as soon as memory rewriting changes that prefix. MemOPD treats the realized memory state $\\sigma_t = (z_t, \\rho_t, \\mathcal{V}_t, \\gamma_t)$ as the object that must be preserved, and reconstructs it for every invocation before any objective is applied. In an audit of native MEM1 trajectories, persistent-history reconstruction changed the teacher's top prediction at 651 sampled action positions and falsely triggered PPO clipping for 13.29% of actions, while reconstructed packing matched independent invocations at the numerical floor. With the same reward and data, teacher guidance on aligned states adds 7.0% F1 over persistent-history teacher scoring in the matched Q2 control, and the full MemOPD-3B system improves F1 over PPO by up to 416.2%.","pith_inferences":["The state-alignment principle should apply to any auxiliary objective evaluated at action positions, not only teacher distillation; the paper's audit metrics give a cheap diagnostic for detecting misaligned training states in other memory-agent pipelines.","The reported reductions in dependency and inference time on Wiki-RAG suggest that state-aligned teacher guidance may push the policy toward more economical memory formats, though the paper reports these efficiency gains without analyzing the mechanism.","A natural extension is to apply MemOPD to agents whose context update is noisy or non-deterministic, or where the environment itself rewrites observations; the paper tests deterministic memory updaters, and whether RCE still holds under stochastic rewriting remains an open question."],"forward_implications":["Any agent that rewrites context between invocations must reconstruct the exact invocation state before applying teacher, value, or reference objectives; provenance alone does not make a rollout batch valid.","The same packing strategy can be reused across memory topologies: RCE holds for full-response retention, suffix retention, summary replacement, sliding windows, retrieval refresh, and native MEM1, so the interface is not tied to one memory format.","Because the teacher supervises every sampled response token with full-vocabulary reverse KL while PPO retains the final task reward, a student can receive dense local guidance without losing the global task objective.","Measured efficiency: packing shared computation yields up to a 1.63x speedup in actor computation during training, with no change in the optimized states."],"supporting_citations":[{"why":"Defines the MEM1 compact-memory protocol that MemOPD instantiates and whose trajectories reveal the state-mismatch problem.","marker":"Zhou et al. 2025"},{"why":"Formulates on-policy distillation as querying a teacher on student-generated sequences, the method that MemOPD extends.","marker":"Agarwal et al. 2024"},{"why":"Supplies PPO, the task-level objective that MemOPD combines with teacher guidance.","marker":"Schulman et al. 2017"},{"why":"Provides the reverse-KL distillation objective used for the full-vocabulary teacher supervision.","marker":"Gu et al. 2024"},{"why":"Provides a streamlined reverse-KL distillation objective used for the full-vocabulary teacher supervision.","marker":"Ko et al. 2024"},{"why":"DAgger's principle of querying the expert on learner-visited states motivates the state-alignment requirement.","marker":"Ross, Gordon, and Bagnell 2011"},{"why":"Identifies Qwen2.5 3B as the student model in all training experiments.","marker":"Yang et al. 2025"},{"why":"Provides the single-objective Wiki-RAG benchmark used to test transfer from multi-objective retrieval.","marker":"Jin et al. 2025"}],"fun_headline_variants":["MemOPD: state-aligned distillation lifts F1 up to 416% over PPO","Aligned memory states give 416% F1 lift over PPO distillation","Restoring invocation state aligns teacher scoring, lifting F1 416%","Memory-state alignment boosts F1 up to 416% over standard PPO","State-aligned teacher scoring: up to 416% F1 gain over PPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole gain rests on the assumption that the frozen 7B teacher's dense full-vocabulary supervision on the aligned 3B student states transfers beyond the Q2 training horizon to Q8/Q16 and Wiki-RAG, and this is tested on two benchmarks with one teacher, so a different teacher or tokenizer could weaken or reverse the advantage.","fun_headline_variants_meta":{"raw":{"variants":["MemOPD: state-aligned distillation lifts F1 up to 416% over PPO","Aligned memory states give 416% F1 lift over PPO distillation","Restoring invocation state aligns teacher scoring, lifting F1 416%","Memory-state alignment boosts F1 up to 416% over standard PPO","State-aligned teacher scoring: up to 416% F1 gain over PPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1978,"prompt_tokens":1064,"completion_tokens":914,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":680,"completion_tokens_details":{"reasoning_tokens":808}},"tokens_in":680,"tokens_out":914,"duration_ms":7585,"temperature":1.0,"reasoning_tokens":808,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:22:40.381967+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 3B student with the same PPO and OPD coefficients but with a 1B teacher, or a teacher with a different tokenizer, on Q2 and evaluate Q16; if the state-alignment advantage over persistent-history scoring disappears or reverses, the identified mechanism is not the cause.","supporting_citations":[],"review_version":1}