{"id":"f27c4ab5-82d6-49c8-b2e0-c3e6af17659c","arxiv_id":"2502.02538","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FQL trains a one-step policy to maximize Q-values while distilling a flow-matching behavioral cloning policy, outperforming many offline RL baselines.","lead":"Flow Q-Learning trains an offline reinforcement learning agent by combining a flow-based behavioral cloning model with a separate one-step policy that maximizes predicted rewards. The method avoids the unstable backpropagation normally needed to guide diffusion or flow policies, and reports strong results across 73 OGBench and D4RL tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The one-step distillation gap in Eq. (7) is the load-bearing risk: if the L2 coupling to the flow policy is loose, the behavioral regularizer may not constrain OOD actions, although the empirical results do not yet disprove the claim.","rationale":"The Reader identifies the one-step approximation as the weakest assumption, and I agree. The paper's empirical evidence is strong and carefully controlled: FAWAC, FBRAC, and IFQL share the same codebase as FQL, and the comparisons in Figure 5 isolate the policy-extraction scheme. However, the theoretical framing in Eq. (8) is only an upper bound relative to the flow policy, not to the dataset, and no quantitative check of distillation fidelity is provided. The adroit results in Table 3 provide a partial warning: FQL underperforms ReBRAC on the high-dimensional pen tasks, which is exactly the regime where one-step expressivity should matter most. This is a genuine risk to the central claim, but it is not enough to overturn an empirical accept: the overall 73-task results still support the headline claim, and the proposed diagnostic would settle whether the gap actually lands. Therefore I keep the Reader's verdict unchanged rather than moving to conditional acceptance, while flagging that a single fidelity measurement would materially increase confidence.","tokens_in":32166,"tokens_out":9651,"duration_ms":111357,"concrete_test":"On cube-double-play and pen-human-v1, train FQL and every 100K steps compute (a) the held-out distillation loss E_{s,z}||μω(s,z)-μθ(s,z)||², and (b) the negative log-likelihood of actions sampled from μω and from μθ under a state-conditioned action-density model fit to the dataset. If μω's actions are not significantly less likely than μθ's, the one-step approximation gap is benign; if they are significantly less likely and this worsens during training, the Eq. (9) regularizer is not enforcing dataset support and the adroit underperformance is mechanistically explained. This directly tests the slackness of Eq. (8)'s upper bound under the Q-gradient.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism is Eq. (9): the one-step policy μω maximizes Q while being pulled toward μθ by α||μω(s,z)-μθ(s,z)||². The paper's own Remark (Eq. 8) shows this loss upper-bounds W2²(πω, πθ), the distance to the learned flow policy, not to the dataset. The support guarantee therefore requires two unquantified approximations to hold simultaneously: (i) the flow BC policy μθ itself generates approximately the dataset action distribution with only 10 Euler steps, and (ii) the single MLP μω can reproduce μθ's z→a map well enough that the Q-gradient term does not exploit residual error and push actions off-support. Prior distillation results cited (Liu et al. 2023; Frans et al. 2025) target image/representation generation with tailored objectives and much larger capacity, not L2 regression to an ODE endpoint in 8–24 dimensions under a concurrent Q-loss. The paper reports no measurement of the distillation gap or of OOD actions, so the behavioral-regularizer interpretation in Table 1 remains a loose bound that could be slack exactly in the regimes where FQL claims its advantage. Consistent with this concern, Table 3 shows FQL is substantially weaker than ReBRAC on the 24-D adroit pen tasks (e.g., pen-human: 53 vs 103), the highest-dimensional action spaces in the benchmark. This is not an observed failure of the entire method; it is the least-secure condition for the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes flow Q-learning (FQL), an offline RL method that combines a flow-matching behavioral-cloning policy with an expressive one-step policy. The flow policy is trained only by behavioral cloning, and the one-step policy is trained to maximize the Q-function while being regularized by an L2 distillation loss to the flow policy, thereby avoiding backpropagation through time during value maximization. The paper reports strong empirical results across 73 OGBench and D4RL tasks in both offline and offline-to-online settings, along with controlled comparisons against several flow- and diffusion-based baselines and extensive ablations.","tokens_in":32413,"tokens_out":5206,"duration_ms":51685,"significance":"If the empirical results hold up, FQL offers a simple and computationally efficient way to leverage expressive flow policies in offline RL, with one-step inference and no recursive backpropagation. The paper's main strengths are its careful experimental methodology: fixed evaluation epochs, per-environment hyperparameter tuning, controlled implementations of flow baselines on the same codebase, multiple seeds, and public code. The contribution is primarily empirical, and the central claim depends on the behavioral regularizer being effective in practice; the paper does not provide a theoretical guarantee, but that is not unusual for a methods paper of this type.","major_comments":[{"comment":"The behavioral-regularizer interpretation of FQL rests on an unquantified chain of approximations. The distillation loss in Eq. (7) and the Wasserstein bound in Eq. (8) measure distance to the learned flow policy πθ, not to the dataset policy. For the regularizer to be meaningful, the paper must assume that (i) the flow BC policy with 10 Euler steps accurately approximates the dataset action distribution, and (ii) the one-step policy μω reproduces μθ closely enough that the Q-gradient term does not exploit residual error and push actions out of the data support. The paper reports no measurement of either the flow approximation error or the distillation gap, nor any diagnostic of out-of-distribution actions during training. This is a load-bearing gap because the method's stated rationale (Table 1 and Section 1) is that it enforces an accurate behavioral constraint via flow expressivity. I recommend adding a small empirical analysis that reports these quantities on representative tasks, e.g., average ||μω(s,z)−μθ(s,z)||² and the fraction of sampled actions falling outside the dataset support, to validate the mechanism.","section":"Section 3, Eq. (7)–(9) and the Remark"},{"comment":"The D4RL adroit results, and particularly the pen tasks, are the clearest counterexample to the claim of 'best or near-best performance on most tasks.' FQL obtains 53±6 on pen-human-v1 and 74±11 on pen-cloned-v1, while ReBRAC obtains 103 on both; these are 24-dimensional action spaces, the highest in the benchmark. This is exactly the regime where the one-step distillation and the flow approximation are least secure, and it aligns with the unquantified approximation concern above. The paper should either provide an analysis of why the advantage disappears at high dimensionality or explicitly temper the claim by acknowledging this limitation.","section":"Table 3, D4RL adroit rows"},{"comment":"The claim that 'FQL achieves the best fine-tuning performance compared to both previous offline RL approaches ... and methods specifically designed for online fine-tuning (Cal-QL and RLPD)' is not supported by Table 4. For example, on puzzle-4x4-play-singletask-v0, RLPD reaches 100±1 while FQL reaches 38±52; on humanoidmaze-medium-navigate-singletask-v0, IFQL reaches 82±20 while FQL reaches 22±12. The aggregated curves in Figure 6 may look favorable on average, but the per-task results show that FQL is not uniformly best. The claim should be qualified to reflect the variability, and the paper should discuss why FQL underperforms on these specific tasks.","section":"Section 5.2, offline-to-online claim and Table 4"}],"minor_comments":[{"comment":"In the second category, 'Reparameterized policy gradient,' the text says the naive approach 'requires backpropagation through time (Equation (9))'; Equation (9) is the FQL actor loss, and the naive approach is actually given by Equation (6). Please correct the cross-reference.","section":"Section 4.1"},{"comment":"The Limitations appendix lists the ODE-solving cost and the lack of a built-in exploration mechanism, but it does not mention the potential failure mode identified by the reviewer: the one-step policy may drift out of the dataset support because the distillation loss only bounds distance to the flow policy. Adding a brief discussion of this assumption would make the limitations more complete.","section":"Appendix A"},{"comment":"In the figure showing policy extraction comparisons, the numbers 16, 29, 30, 44 are printed above bars but are not described in the caption; please add error bars or at least state the aggregation method (mean over the 50 tasks) and the standard error, so the reader can judge the significance of the differences.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning venue and the empirical study is extensive and well controlled. The main concern is that the central mechanism—behavioral regularization through one-step distillation—is not directly validated, and the adroit results and some offline-to-online results do not fully support the unqualified performance claims. These issues are addressable with additional analysis and more careful wording, so I do not see grounds for rejection. I would recommend major revision rather than minor revision because the missing diagnostics are directly relevant to the paper's core claim, not merely presentational issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper is that FQL is a real and useful contribution, not just another diffusion/flow RL variant. The idea is simple: train a flow policy with BC only, then train a separate one-step policy to maximize Q while being pulled toward the flow policy by an L2 distillation loss. That sidesteps backpropagation through time entirely, keeps test-time inference cheap, and it works. The empirical showing is large and mostly careful: 73 tasks, fixed evaluation epochs, multiple seeds, controlled flow baselines that share the same architecture, and honest ablations. I believe the central claim, that one-step guidance is a better policy extraction scheme than weighted regression, BPTT, or rejection sampling for flow policies.\n\nThe soft spots are real but not fatal. The biggest is the theory-adjacent gap: Eq. (8) correctly shows the distillation loss upper-bounds W2 between the one-step policy and the BC flow policy, not between the policy and the dataset. So the behavioral regularizer is only as good as the flow policy's own fidelity to the data, plus the one-step model's ability to reproduce the flow. The paper never measures either gap, and the adroit pen-human numbers (53 vs 103 for ReBRAC) are a hint that the one-step distillation starts to strain in 24-D action spaces. I would not call this a load-bearing flaw, but the paper oversells the Wasserstein connection slightly; it should at least report OOD-action statistics or distillation error.\n\nOther concerns are minor and standard for this area: per-environment tuning of alpha, some baseline numbers taken from prior work, and the benchmark OGBench coming from the same group. None of that undercuts the main result. The paper is honest about its limitations, and the code is promised to be open.\n\nWho gets value: anyone doing offline RL with expressive policies, especially practitioners who want a simple, fast, effective method. This deserves a serious referee. I would accept it with revisions that tighten the framing of the regularizer and add a small amount of evidence on the distillation gap.","headline":"A genuinely simple and effective flow-policy offline RL method whose empirical case is strong, but whose behavioral-regularizer guarantee is looser than the paper suggests.","tokens_in":33006,"tokens_out":1069,"would_cite":true,"duration_ms":12733,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Flow Q-learning trains a one-step policy by distilling from a behavior-cloned flow policy, and reports best or near-best results on 73 offline RL tasks.","keywords":["offline reinforcement learning","flow matching","policy extraction","one-step distillation","behavioral regularization","Wasserstein distance","OGBench","D4RL"],"falsifier":"A concrete check: on a multimodal task (e.g., cube-double), record the average distillation error between $\\mu_\\omega$ and the Euler-solved flow policy, and the fraction of chosen actions that fall outside the dataset's action support. If Q values rise while true return stays flat as the distillation gap grows, the behavioral regularizer is not holding and the central performance claim would not survive.","tokens_in":31899,"feed_emoji":"🌀","tokens_out":6865,"duration_ms":59790,"temperature":0.7,"pith_summary":"This paper proposes flow Q-learning (FQL), an offline reinforcement learning method that keeps the expressive action distributions of flow-matching policies without paying the cost of steering an iterative generative model with value gradients. Rather than guiding the flow policy directly, FQL trains it only by behavioral cloning and trains a separate one-step policy to maximize Q-values while distilling from the flow policy. The paper's experiments indicate that this one-step guidance extraction scheme outperforms weighted regression, backpropagation-through-time, and rejection sampling, and that FQL achieves best or near-best scores on 73 OGBench and D4RL tasks spanning robotic locomotion, manipulation, and pixel observations. If correct, the method shows that expressive flow policies can be used for offline RL with a simple, fast, one-step actor at test time.","feed_headline":"One-step guidance wins on 73 offline RL tasks","feed_subtitle":"Cloning a flow policy, then distilling a Q-seeking one-step actor, beats prior extraction schemes.","key_machinery":"The load-bearing object is one-step guidance: a decomposition of the actor into a BC-only flow policy defined by a velocity field $v_\\theta(t, s, x)$ integrated with Euler steps, and a one-step model $\\mu_\\omega(s, z)$ that directly maps Gaussian noise to actions. The one-step model absorbs the flow policy by regression (distillation) while an added Q-loss term steers it toward high-value actions; because $z$ is reparameterizable, no backpropagation through time is needed. The identity that carries the argument is that the distillation loss is an upper bound on the squared 2-Wasserstein distance between the one-step policy and the flow policy, turning behavioral cloning into a metric-aware regularization.","core_discovery":"The central claim is that an expressive flow-matching policy can be converted into a performant offline RL actor without ever backpropagating through the flow's iterative ODE solver. FQL trains the flow policy $\\mu_\\theta(s, z)$ solely with behavioral cloning, then trains a one-step network $\\mu_\\omega(s, z)$ to maximize Q-values through reparameterized gradients while minimizing a distillation loss to the flow policy's output. This distillation loss is shown to upper-bound the squared 2-Wasserstein distance between the one-step policy and the behavioral flow policy, so the behavioral regularizer is metric-aware in action space. Empirically, the paper finds that this one-step guidance scheme outperforms weighted regression, BPTT-based reparameterized gradients, and rejection sampling, and that FQL attains best or near-best performance on most of 73 OGBench and D4RL tasks, including pixel-based manipulation.","pith_inferences":["Beyond the paper: the Wasserstein upper-bound reading suggests $\\alpha$ could be set from the action-space geometry or replaced by a learned ground metric, which the paper does not explore.","Beyond the paper: the same split of cloning the modal policy and distilling the actor could be applied to other iterative samplers, such as consistency models or rectified flows, provided the one-step approximation is trained to match the solver's trajectory.","Beyond the paper: a testable monitoring rule follows from the tug-of-war picture: track the distillation gap and the out-of-support action rate during training; a widening gap under rising Q values would flag that the behavioral constraint is failing before evaluation."],"forward_implications":["Offline RL with expressive policies no longer requires backpropagation through time or iterative action generation at test time.","The one-step guidance scheme transfers to diffusion policies by converting the diffusion SDE to an ODE, as the paper notes.","FQL fine-tunes online by simply appending online transitions and continuing the same objective, without balanced replay or exploration bonuses.","The ablation against FAWAC, FBRAC, and IFQL shows policy extraction is the decisive design choice, not the generative model family."],"supporting_citations":[{"why":"Defines the flow-matching objective used to train the BC flow policy.","marker":"Lipman et al., 2023"},{"why":"Provides rectified-flow distillation results invoked to justify one-step approximation of the flow policy.","marker":"Liu et al., 2023"},{"why":"Shows one-step shortcut models can approximate multi-step generative models, supporting the distillation expressivity assumption.","marker":"Frans et al., 2025"},{"why":"Supplies TD3+BC, the behavior-regularized actor-critic baseline that FQL builds on.","marker":"Fujimoto & Gu, 2021"},{"why":"Supplies AWAC and the KL-regularized actor-critic template FQL contrasts with its Wasserstein regularizer.","marker":"Nair et al., 2020"},{"why":"Establishes Diffusion-QL, the naive BPTT-based approach that motivates FQL's one-step guidance.","marker":"Wang et al., 2023"},{"why":"Consistency-AC is the closest distillation-based baseline, also using BPTT, which FQL compares against.","marker":"Ding & Jin, 2024"},{"why":"OGBench is the main benchmark suite providing the 50 state-based and 5 pixel-based tasks.","marker":"Park et al., 2025"},{"why":"D4RL provides the antmaze and adroit tasks used for the additional 18-task evaluation.","marker":"Fu et al., 2020"},{"why":"Introduces the behavior-regularized actor-critic framework that FQL's actor-critic losses generalize.","marker":"Wu et al., 2019"}],"fun_headline_variants":["Flow Q-Learning: One-step actor wins on 73 RL tasks","Clone the flow, distill the Q: FQL for offline RL","One-step guidance from flow policy: FQL's edge","FQL: Avoid recursive backprop, keep flow expressivity","Metric-aware distillation: FQL's one-step actor"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a single-step network can approximate the full iterative flow policy closely enough that Q-gradient updates on the one-step policy do not push actions outside the dataset's action support.","fun_headline_variants_meta":{"raw":{"variants":["Flow Q-Learning: One-step actor wins on 73 RL tasks","Clone the flow, distill the Q: FQL for offline RL","One-step guidance from flow policy: FQL's edge","FQL: Avoid recursive backprop, keep flow expressivity","Metric-aware distillation: FQL's one-step actor"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000626,"raw_usage":{"total_tokens":2850,"prompt_tokens":855,"completion_tokens":1995,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":1909}},"tokens_in":471,"tokens_out":1995,"duration_ms":15654,"temperature":1.0,"reasoning_tokens":1909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T11:49:20.534030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: on a multimodal task (e.g., cube-double), record the average distillation error between $\\mu_\\omega$ and the Euler-solved flow policy, and the fraction of chosen actions that fall outside the dataset's action support. If Q values rise while true return stays flat as the distillation gap grows, the behavioral regularizer is not holding and the central performance claim would not survive.","supporting_citations":[{"cited_title":"-singletask","cited_arxiv_id":null,"evidence_quote":"OGBench is the main benchmark suite providing the 50 state-based and 5 pixel-based tasks."}],"review_version":1}