{"id":"39848c1f-fdfc-49c7-a83c-92dc8e186700","arxiv_id":"2606.06967","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"GenPO++ achieves exact Jacobian-free likelihood ratio computation for generative flow policies by embedding history states as auxiliary memory in a high-order reversible ODE solver.","lead":"The paper presents GenPO++, which uses history states in a reversible ODE solver to compute exact likelihood ratios for flow-based generative policies in reinforcement learning without Jacobians or dummy actions. This approach aims to enable unbiased and efficient on-policy optimization for complex continuous control tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"History states as auxiliary memory may fail to deliver exact inversion without approximation error or effective dimension change","rationale":"The reader’s weakest assumption is precisely the load-bearing step for both exact inversion and the fixed-coefficient log-det claim; confirming or refuting that assumption via the concrete numerical check would directly settle whether the central technical contribution holds.","tokens_in":1715,"tokens_out":354,"duration_ms":23205,"concrete_test":"Take the paper’s reversible solver definition, implement forward and inverse passes on a linear ODE dx/dt = Ax with known closed-form solution and volume change; run with history length equal to the method order, recover the initial noise from the final action alone, and check whether the computed log-density matches the analytic value to machine precision (if the recovered noise deviates or the log-det differs from the fixed-coefficient prediction, the exactness claim fails).","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The strongest claim requires that a high-order reversible ODE solver using history states as auxiliary memory produces an exactly invertible generative map whose log-determinant depends only on fixed solver coefficients. In multistep or high-order reversible integrators, the history buffer is part of the extended state; recovering the original noise therefore requires either (a) treating the full history as additional output dimensions (altering the policy measure) or (b) discarding or approximating the history on inversion, which introduces truncation or accumulation error. Neither case is obviously compatible with “preserving the original action dimension” and “exact” likelihood ratios. The abstract supplies no derivation showing how the history is managed so that the map remains a bijection on the original action space with a determinant independent of the learned vector field.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript introduces GenPO++, a reversible generative policy optimization framework for on-policy RL. It claims that embedding history states as auxiliary memory in a high-order reversible ODE solver produces an exactly invertible generative policy map whose log-determinant depends only on fixed solver coefficients. This enables exact, Jacobian-free likelihood-ratio evaluation while preserving the original action dimension, avoiding both surrogate bias and dummy-action augmentation. The method is evaluated on large-scale simulated control, fine-tuning, and real-world robotic manipulation tasks, where it reports competitive or superior performance and improved stability.","tokens_in":1852,"tokens_out":507,"duration_ms":16916,"significance":"If the exact-inversion claim holds, the work would remove a central obstacle to deploying expressive flow-based generative policies in likelihood-based on-policy RL, offering unbiased updates without enlarging the action space. The reported gains in stability and efficiency on robotic tasks would then constitute a practical advance over existing flow-RL baselines.","major_comments":[{"comment":"Abstract and §3 (reversible ODE solver description): the claim that history states yield exact inversion 'without changing the original action dimension' is load-bearing for the Jacobian-free likelihood result, yet the text supplies no derivation showing how the extended state of a multistep integrator is inverted while keeping the map a bijection strictly on the original action space and ensuring the log-determinant remains independent of the learned vector field.","section":"Abstract / §3"},{"comment":"§3.2 (likelihood-ratio computation): the assertion that the log-determinant 'is determined only by fixed solver coefficients' must be accompanied by an explicit expression or proof that this quantity does not depend on the learned dynamics; without it, the Jacobian-free property cannot be verified and the exactness claim remains unsubstantiated.","section":"§3.2"}],"minor_comments":[{"comment":"The abstract is dense; a short paragraph or diagram clarifying the role of the history buffer versus the action variables would improve readability for readers unfamiliar with reversible integrators.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core technical novelty rests on a construction whose correctness is difficult to assess from the provided abstract alone; the full manuscript must contain the missing derivation for the inversion step. If that derivation is absent or incomplete, the paper would be better suited to a workshop rather than the target journal."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the careful and constructive review. The two major comments correctly identify that the manuscript lacks explicit derivations for the inversion of the multistep integrator and for the independence of the log-determinant. We will revise the paper to supply these.","responses":[{"response":"We agree the derivation is missing. In the revised manuscript we will add a detailed derivation (new subsection in §3 and appendix) showing that the auxiliary history states permit exact reversal of the multistep integrator while the overall transport map remains a bijection strictly on the original action coordinates; the volume scaling is shown to factor separately from the learned vector field.","revision_made":"yes","referee_comment":"[Abstract / §3] Abstract and §3 (reversible ODE solver description): the claim that history states yield exact inversion 'without changing the original action dimension' is load-bearing for the Jacobian-free likelihood result, yet the text supplies no derivation showing how the extended state of a multistep integrator is inverted while keeping the map a bijection strictly on the original action space and ensuring the log-determinant remains independent of the learned vector field."},{"response":"We agree an explicit expression and proof are required. The revision will include the closed-form expression for the log-determinant (derived from the fixed Butcher tableau or multistep coefficients) together with the short proof that it is independent of the learned dynamics f, obtained by showing that the Jacobian of the composite flow map separates into a constant factor and a term whose determinant is unity under the reversible construction.","revision_made":"yes","referee_comment":"[§3.2] §3.2 (likelihood-ratio computation): the assertion that the log-determinant 'is determined only by fixed solver coefficients' must be accompanied by an explicit expression or proof that this quantity does not depend on the learned dynamics; without it, the Jacobian-free property cannot be verified and the exactness claim remains unsubstantiated."}],"tokens_in":1388,"tokens_out":431,"duration_ms":15387,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Hi,\n\nThe main takeaway is that the paper introduces a construction for flow-based policies in on-policy RL that embeds history states as auxiliary memory in a high-order reversible ODE solver. This is meant to deliver exact inversion of the generative map while keeping the original action dimension and making the log-determinant depend only on fixed solver coefficients, avoiding both surrogate bias and dummy-action overhead.\n\nWhat it does well is target a concrete bottleneck: existing flow RL either biases the updates or inflates the policy space. The abstract frames the fix clearly and reports results on large-scale simulated control, fine-tuning, and real robotic manipulation, where the method matches or exceeds current on-policy baselines with gains in stability and speed. That practical scope is useful.\n\nThe soft spot is the central technical claim. The stress-test note is right to flag that history buffers in multistep reversible integrators become part of an extended state. Recovering the original noise exactly, without truncation error or an effective change in dimension, is not automatic and requires a derivation showing the map remains a bijection on the action space alone. The abstract gives no equations or proof sketch, so it is impossible to tell whether the fixed-coefficient log-det property actually survives the history handling. If the full paper has a clean argument for this, the contribution strengthens; if not, the advantage over prior work is smaller than stated.\n\nThis is for people working on expressive policy representations in continuous-control RL. Readers already using normalizing flows or related generative models would see the most direct value.\n\nIt deserves peer review because the problem is real and the proposed direction is distinct from the baselines it cites. The experiments look broad enough to be informative once the math is verified.","headline":"GenPO++ claims exact Jacobian-free likelihoods for flow policies via history-state memory in reversible ODE solvers, but the inversion exactness is the part that needs checking.","tokens_in":2330,"tokens_out":426,"would_cite":false,"duration_ms":17619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"GenPO++ makes flow-based generative policies usable for exact on-policy RL by computing likelihood ratios from fixed solver coefficients alone.","keywords":["generative policies","flow-based policies","reinforcement learning","likelihood ratio","on-policy learning","reversible ODE","continuous control"],"falsifier":"Run the same policy forward and backward through the solver on a held-out trajectory; if the recovered initial noise differs from the true noise by more than floating-point tolerance, or if the computed log-determinant changes when solver coefficients are altered, the exact-inversion claim fails.","tokens_in":2609,"feed_emoji":"🔄","tokens_out":670,"duration_ms":10649,"temperature":0.7,"pith_summary":"The paper shows that a high-order reversible ODE solver can treat past states as auxiliary memory to invert a generative policy map exactly, without enlarging the action space. This inversion yields a log-determinant that depends only on the solver's fixed coefficients, so the probability ratio of any executed action can be evaluated directly and without Jacobians. Because the method keeps the original action dimension and avoids surrogate approximations, it removes both the bias of earlier flow-RL surrogates and the computational overhead of dummy-action tricks. The resulting algorithm is evaluated on simulated control, fine-tuning, and real-robot tasks and matches or exceeds standard on-policy baselines while using less compute per update.","feed_headline":"Reversible solver gives exact likelihood ratios for flow policies without extra dimensions","feed_subtitle":"History states let the method recover the true action density from fixed coefficients alone, removing bias and dummy-action overhead in on-p","key_machinery":"High-order reversible ODE solver that stores history states as auxiliary memory to invert the generative transport map exactly while leaving the action dimension unchanged.","core_discovery":"A reversible generative policy optimization framework that uses history states as auxiliary memory in a high-order reversible ODE solver yields exact inversion without changing the original action dimension; the resulting generative policy map has a log-determinant determined only by fixed solver coefficients, enabling exact and Jacobian-free likelihood-ratio computation.","pith_inferences":["The approach could be combined with any black-box high-order reversible integrator that admits an explicit inverse, not only the one used in the experiments.","Because the log-determinant is independent of the learned vector field, the method may transfer to other transport-map families whose Jacobians are otherwise intractable.","In settings where action dimension is already large, the constant memory cost of the history buffer may still be cheaper than the quadratic cost of dummy-action augmentation."],"forward_implications":["On-policy updates for flow policies become unbiased because the true action density ratio is recovered.","No extra dummy dimensions are needed, so memory and compute scale with the original action size.","Training stability improves because the likelihood ratio is exact rather than approximated.","The same solver coefficients can be reused across environments without retuning the density term."],"fun_headline_variants":["Reversible ODE uses history for exact Jacobian-free likelihood ratios","Fixed coefficients determine log-determinants in GenPO++ flow policies","GenPO++ recovers exact action densities via reversible high-order solver","No dummy actions needed for exact likelihood ratios in generative policies"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"History states stored by the reversible ODE solver allow exact inversion of the generative policy without approximation error or change to the action space.","fun_headline_variants_meta":{"raw":{"variants":["Reversible ODE uses history for exact Jacobian-free likelihood ratios","Fixed coefficients determine log-determinants in GenPO++ flow policies","GenPO++ recovers exact action densities via reversible high-order solver","No dummy actions needed for exact likelihood ratios in generative policies"]},"model":"grok-4.3","cost_usd":0.004804,"raw_usage":{"total_tokens":2356,"prompt_tokens":654,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":48037000,"prompt_tokens_details":{"text_tokens":654,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1634,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":654,"tokens_out":68,"duration_ms":6324,"temperature":1.0,"reasoning_tokens":1634,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T22:45:32.509337+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Run the same policy forward and backward through the solver on a held-out trajectory; if the recovered initial noise differs from the true noise by more than floating-point tolerance, or if the computed log-determinant changes when solver coefficients are altered, the exact-inversion claim fails.","supporting_citations":[],"review_version":1}