REVIEW 3 major objections 3 minor 14 cited by
Flow Q-Learning
T0 review · 3 major / 3 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read Flow Q-learning trains a one-step policy by distilling from a behavior-cloned flow policy, and reports best or near-best results on 73 offline RL tasks.
desk verdict A genuinely simple and effective flow-policy offline RL method whose empirical case is strong, but whose behavioral-regularizer guarantee is looser than the paper suggests. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is one-step guidance: a decomposition of the actor into a BC-only flow policy defined by a velocity field $v_\theta(t, s, x)$ integrated with Euler steps, and a one-step model $\mu_\omega(s, z)$ that directly maps Gaussian noise to actions. The one-step model absorbs the flow policy by regression (distillation) while an added Q-loss term steers it toward high-value actions; because $z$ is reparameterizable, no backpropagation through time is needed. The identity that carries the argument is that the distillation loss is an upper bound on the squared 2-Wasserstein distance between the one-step policy and the flow policy, turning behavioral cloning into a metric-aware regularization.
What would settle it
A concrete check: on a multimodal task (e.g., cube-double), record the average distillation error between $\mu_\omega$ and the Euler-solved flow policy, and the fraction of chosen actions that fall outside the dataset's action support. If Q values rise while true return stays flat as the distillation gap grows, the behavioral regularizer is not holding and the central performance claim would not survive.
Extended reading notes
Core claim
The central claim is that an expressive flow-matching policy can be converted into a performant offline RL actor without ever backpropagating through the flow's iterative ODE solver. FQL trains the flow policy $\mu_\theta(s, z)$ solely with behavioral cloning, then trains a one-step network $\mu_\omega(s, z)$ to maximize Q-values through reparameterized gradients while minimizing a distillation loss to the flow policy's output. This distillation loss is shown to upper-bound the squared 2-Wasserstein distance between the one-step policy and the behavioral flow policy, so the behavioral regularizer is metric-aware in action space. Empirically, the paper finds that this one-step guidance scheme outperforms weighted regression, BPTT-based reparameterized gradients, and rejection sampling, and that FQL attains best or near-best performance on most of 73 OGBench and D4RL tasks, including pixel-based manipulation.
Load-bearing premise
The load-bearing premise is that a single-step network can approximate the full iterative flow policy closely enough that Q-gradient updates on the one-step policy do not push actions outside the dataset's action support.
Editorial extensions
If this is right
- Offline RL with expressive policies no longer requires backpropagation through time or iterative action generation at test time.
- The one-step guidance scheme transfers to diffusion policies by converting the diffusion SDE to an ODE, as the paper notes.
- FQL fine-tunes online by simply appending online transitions and continuing the same objective, without balanced replay or exploration bonuses.
- The ablation against FAWAC, FBRAC, and IFQL shows policy extraction is the decisive design choice, not the generative model family.
Reading between the lines
- Beyond the paper: the Wasserstein upper-bound reading suggests $\alpha$ could be set from the action-space geometry or replaced by a learned ground metric, which the paper does not explore.
- Beyond the paper: the same split of cloning the modal policy and distilling the actor could be applied to other iterative samplers, such as consistency models or rectified flows, provided the one-step approximation is trained to match the solver's trajectory.
- Beyond the paper: a testable monitoring rule follows from the tug-of-war picture: track the distillation gap and the out-of-support action rate during training; a widening gap under rising Q values would flag that the behavioral constraint is failing before evaluation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes flow Q-learning (FQL), an offline RL method that combines a flow-matching behavioral-cloning policy with an expressive one-step policy. The flow policy is trained only by behavioral cloning, and the one-step policy is trained to maximize the Q-function while being regularized by an L2 distillation loss to the flow policy, thereby avoiding backpropagation through time during value maximization. The paper reports strong empirical results across 73 OGBench and D4RL tasks in both offline and offline-to-online settings, along with controlled comparisons against several flow- and diffusion-based baselines and extensive ablations.
Significance. If the empirical results hold up, FQL offers a simple and computationally efficient way to leverage expressive flow policies in offline RL, with one-step inference and no recursive backpropagation. The paper's main strengths are its careful experimental methodology: fixed evaluation epochs, per-environment hyperparameter tuning, controlled implementations of flow baselines on the same codebase, multiple seeds, and public code. The contribution is primarily empirical, and the central claim depends on the behavioral regularizer being effective in practice; the paper does not provide a theoretical guarantee, but that is not unusual for a methods paper of this type.
major comments (3)
- [Section 3, Eq. (7)–(9) and the Remark] The behavioral-regularizer interpretation of FQL rests on an unquantified chain of approximations. The distillation loss in Eq. (7) and the Wasserstein bound in Eq. (8) measure distance to the learned flow policy πθ, not to the dataset policy. For the regularizer to be meaningful, the paper must assume that (i) the flow BC policy with 10 Euler steps accurately approximates the dataset action distribution, and (ii) the one-step policy μω reproduces μθ closely enough that the Q-gradient term does not exploit residual error and push actions out of the data support. The paper reports no measurement of either the flow approximation error or the distillation gap, nor any diagnostic of out-of-distribution actions during training. This is a load-bearing gap because the method's stated rationale (Table 1 and Section 1) is that it enforces an accurate behavioral constraint via flow expressivity. I recommend adding a small empirical analysis that reports these quantities on representative tasks, e.g., average ||μω(s,z)−μθ(s,z)||² and the fraction of sampled actions falling outside the dataset support, to validate the mechanism.
- [Table 3, D4RL adroit rows] The D4RL adroit results, and particularly the pen tasks, are the clearest counterexample to the claim of 'best or near-best performance on most tasks.' FQL obtains 53±6 on pen-human-v1 and 74±11 on pen-cloned-v1, while ReBRAC obtains 103 on both; these are 24-dimensional action spaces, the highest in the benchmark. This is exactly the regime where the one-step distillation and the flow approximation are least secure, and it aligns with the unquantified approximation concern above. The paper should either provide an analysis of why the advantage disappears at high dimensionality or explicitly temper the claim by acknowledging this limitation.
- [Section 5.2, offline-to-online claim and Table 4] The claim that 'FQL achieves the best fine-tuning performance compared to both previous offline RL approaches ... and methods specifically designed for online fine-tuning (Cal-QL and RLPD)' is not supported by Table 4. For example, on puzzle-4x4-play-singletask-v0, RLPD reaches 100±1 while FQL reaches 38±52; on humanoidmaze-medium-navigate-singletask-v0, IFQL reaches 82±20 while FQL reaches 22±12. The aggregated curves in Figure 6 may look favorable on average, but the per-task results show that FQL is not uniformly best. The claim should be qualified to reflect the variability, and the paper should discuss why FQL underperforms on these specific tasks.
minor comments (3)
- [Section 4.1] In the second category, 'Reparameterized policy gradient,' the text says the naive approach 'requires backpropagation through time (Equation (9))'; Equation (9) is the FQL actor loss, and the naive approach is actually given by Equation (6). Please correct the cross-reference.
- [Appendix A] The Limitations appendix lists the ODE-solving cost and the lack of a built-in exploration mechanism, but it does not mention the potential failure mode identified by the reviewer: the one-step policy may drift out of the dataset support because the distillation loss only bounds distance to the flow policy. Adding a brief discussion of this assumption would make the limitations more complete.
- [Figure 5] In the figure showing policy extraction comparisons, the numbers 16, 29, 30, 44 are printed above bars but are not described in the caption; please add error bars or at least state the aggregation method (mean over the 50 tasks) and the standard error, so the reader can judge the significance of the differences.
Circularity Check
No significant circularity: FQL's one-step guidance is an independent training objective, and its self-citations to OGBench and prior policy-extraction studies are not load-bearing.
full rationale
The paper's central claim is an empirical one: FQL, trained with the actor objective in Eq. (9) (Q-maximization plus an L2 distillation loss toward a BC-trained flow policy), performs well on 73 OGBench and D4RL tasks. This claim is not derived from its own inputs: the equations define a training procedure, and the performance numbers come from public benchmark tasks with fixed evaluation protocols. The controlled comparison in Figure 5 and Table 2 among FAWAC, FBRAC, IFQL, and FQL shares the same flow-matching codebase and architectures and differs only in the policy-extraction scheme, so the result is not forced by construction. Self-citations exist: Park et al. (2024a) is cited to justify reparameterized policy gradient as an effective extraction scheme, and Park et al. (2025) introduced OGBench. Neither citation is load-bearing as a substitute for evidence: the paper re-validates the extraction comparison empirically (Figure 5) and evaluates on reproducible benchmark tasks against a wide set of external and re-implemented baselines. The Wasserstein remark (Eq. 8) is an upper-bound inequality relating LDistill to W2^2(pi_omega, pi_theta), not an equality that identifies the regularizer with the data distribution by definition; Table 1's labeling of FQL as W2^2 is an interpretation of the bound, not a circular redefinition. The acknowledged limitations in Appendix A (ODE-solving cost during training, lack of built-in exploration for online fine-tuning, absence of real-world evaluation) are risks rather than evidence of circularity. The distillation-gap concern raised in the reader analysis is an unquantified approximation risk about how tightly the one-step model tracks the BC flow policy under a concurrent Q-loss; it does not make any prediction equivalent to a fitted input. Overall, the derivation chain is self-contained and the empirical claims stand on external, reproducible benchmarks, so the circularity burden is minimal.
Assumptions & free parameters
free parameters (4)
- BC coefficient alpha =
per-task values, e.g., 3 to 30000 (Tables 6/7)
- target value aggregation =
mean(q1,q2) default; min(q1,q2) for adroit and antmaze-large/giant
- discount factor gamma =
0.99 default; 0.995 for antmaze-giant, humanoidmaze, antsoccer
- Euler flow steps M =
10
assumptions (4)
- domain assumption Flow matching training objective converges to the behavioral distribution.
- domain assumption Behavior-regularized actor-critic provides a valid objective for offline RL.
- domain assumption One-step distillation can preserve enough expressivity of the full flow model.
- domain assumption OGBench and D4RL benchmarks are representative and the evaluations are fair.
Cite this review
Pith. "Pith review of Flow Q-Learning." pith.science (2026). https://pith.science/paper/KYVGZYBU
@misc{pith2026250202538,
author = {Pith},
title = {Pith review of: Flow Q-Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/KYVGZYBU}},
note = {Machine review of arXiv:2502.02538}
}
read the original abstract
We present flow Q-learning (FQL), a simple and performant offline reinforcement learning (RL) method that leverages an expressive flow-matching policy to model arbitrarily complex action distributions in data. Training a flow policy with RL is a tricky problem, due to the iterative nature of the action generation process. We address this challenge by training an expressive one-step policy with RL, rather than directly guiding an iterative flow policy to maximize values. This way, we can completely avoid unstable recursive backpropagation, eliminate costly iterative action generation at test time, yet still mostly maintain expressivity. We experimentally show that FQL leads to strong performance across 73 challenging state- and pixel-based OGBench and D4RL tasks in offline RL and offline-to-online RL. Project page: https://seohong.me/projects/fql/
Figures
Figures from the paper (7 more)
Forward citations
Cited by 14 Pith papers
-
Source-Lifted Flow Matching for Intervenable Multimodal Imitation
Orthogonal Source Lifting makes flow-matching imitation intervenable by selecting only the source endpoint, changing future routes in 91.1% of matched-prefix tests while keeping free-deployment performance strong.
-
NFTR: From Provable Mode-Averaging to Geodesic Subgoal Selection in Offline Goal-Conditioned RL
Normalizing-flow subgoal policies plus triangle-slack reweighting provably avoid Gaussian mode-averaging and filter lucky transitions in offline hierarchical GCRL.
-
Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search
Bilinear contrastive critics remain good compatibility rankers but are unsafe to maximize for action selection; cosine bounding does not fix value decalibration, while Bellman TD-Q does.
-
Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning?
On six robot-manipulation tasks, offline Q-pretraining does not accelerate online RL fine-tuning from a pretrained policy, while seeding the replay buffer with rollouts from an ensemble of policies (IPE) improves fina...
-
From Prior to Pro: Efficient Skill Mastery via Distribution Contractive RL Finetuning
Residual off-policy RL with selective BC regularization and value-guided sampling contracts a pretrained generative robot policy around successful actions, reaching high success on hard long-horizon tasks from pixels ...
-
FM-IRL: Flow-Matching for Reward Modeling and Policy Regularization in Reinforcement Learning
An online imitation-learning method uses a flow-matching teacher's class-conditional loss as a reward and a regularizer to train a simple MLP policy, beating cloning and adversarial-imitation baselines on five of six tasks.
-
Latent Policy Barrier: Learning Robust Visuomotor Policies by Staying In-Distribution
Latent Policy Barrier improves behavior-cloned visuomotor policies by using a latent dynamics model trained on expert and rollout data to guide actions back toward in-distribution expert states.
-
Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
A single multi-task RL agent using a large regularized critic, categorical value loss, and task embeddings achieves state-of-the-art results across 283 tasks and transfers efficiently to new tasks.
-
Decision Flow Policy Optimization
Decision Flow frames the gradual action generation of flow-based policies as a flow MDP and updates the flow policy with flow-level value functions, reporting state-of-the-art results on several D4RL tasks.
-
Offline RL with Hierarchical Action Chunking
HiQC, a hierarchy of latent subgoal planning and chunked action execution, achieves the best OGBench aggregate score (53%) and an O(sqrt(T/k)) error bound under a bootstrap-chain model.
-
Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agents
Simplicial embeddings — group-wise softmax feature layers — improve sample efficiency and final performance of FastTD3, FastSAC, and PPO across continuous- and discrete-control benchmarks at no meaningful runtime cost.
-
Value Flows
Value Flows fits the full return distribution in RL with a flow-matching critic and reweights its learning objective by estimated return variance; the central theoretical guarantee does not follow from the stated equations.
-
Flow-Based Policy for Online Reinforcement Learning
FlowRL learns online RL policies as flow-matching models regularized by a Wasserstein-2 constraint toward behavior-optimal replay-buffer actions.
-
Extremum Flow Matching for Offline Goal Conditioned Reinforcement Learning
Extremum Flow Matching estimates distributional support bounds from a uniform source and uses them as return conditions for offline goal-conditioned robot policies.
Reference graph
Works this paper leans on
-
[5]
Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor-critic method with diffusion policies. ArXiv, abs/2304.10573,
-
[6]
Dif- fcps: Diffusion model based constrained policy search for offline reinforcement learning
He, L., Shen, L., Zhang, L., Tan, J., and Wang, X. Dif- fcps: Diffusion model based constrained policy search for offline reinforcement learning. ArXiv, abs/2310.05333,
-
[7]
Aligniql: Policy alignment in implicit q-learning through constrained opti- mization
He, L., Shen, L., Tan, J., and Wang, X. Aligniql: Policy alignment in implicit q-learning through constrained opti- mization. ArXiv, abs/2405.18187,
-
[8]
Hendrycks, D. and Gimpel, K. Gaussian error linear units (gelus). ArXiv, abs/1606.08415,
-
[10]
Lipman, Y ., Havasi, M., Holderrieth, P., Shaul, N., Le, M., Karrer, B., Chen, R. T. Q., Lopez-Paz, D., Ben-Hamu, H., and Gat, I. Flow matching guide and code. ArXiv, abs/2412.06264,
-
[11]
Mark, M. S., Gao, T., Sampaio, G. G., Srirama, M. K., Sharma, A., Finn, C., and Kumar, A. Policy agnostic rl: Offline rl and online rl fine-tuning of any class and backbone. ArXiv, abs/2412.06685,
-
[12]
The results are averaged over 8 seeds (4 seeds for pixel-based tasks), and we report standard deviations after “±” in tables and 95% bootstrap confidence intervals as shaded areas in plots. In tables, we denote values at or above 95% of the best performance in bold, following OGBench (Park et al., 2025). Results without standard deviations or confidence i...
work page 2023
-
[14]
B., Kumar, A., Zhang, G., and Levine, S
Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off- policy reinforcement learning. ArXiv, abs/1910.00177,
arXiv 1910
Show all 25 references
-
[15]
Behavior regularized offline reinforcement learning
Wu, Y ., Tucker, G., and Nachum, O. Behavior regularized offline reinforcement learning. ArXiv, abs/1911.11361,
1911 arXiv
-
[16]
Policy representation via diffusion probability model for reinforcement learning
Yang, L., Huang, Z., Lei, F., Zhong, Y ., Yang, Y ., Fang, C., Wen, S., Zhou, B., and Lin, Z. Policy representation via diffusion probability model for reinforcement learning. ArXiv, abs/2305.13122,
-
[17]
Zheng, Q., Le, M., Shaul, N., Lipman, Y ., Grover, A., and Chen, R. T. Guided flows for generative modeling and decision making. ArXiv, abs/2311.13443,
-
[18]
built-in
since flow matching happens in the relatively low-dimensionalaction space (as opposed to image generation), we believe this may further be improved by incorporating a more advanced one-step distillation method, such as shortcut models (Frans et al., 2025). Another limitation i...
2021
-
[20]
We ablate several components of FQL and study how they affect performance
Ablation studies. We ablate several components of FQL and study how they affect performance. The results are averaged over 8 seeds. Hyperparameters. We refer to Tables 5 to 7 for the complete list of hyperparameters. C. Ablation Study In this section, we ablate several compone...
2024
-
[22]
-singletask
on top of OGBench’s reference implementa- tions (Park et al., 2025). We provide our full implementation and exact commands to reproduce the main results of FQL at https://github.com/seohongpark/fql. E.1. Environments, Tasks, and Datasets OGBench (Park et al., 2025). OGBench is...
2025
-
[23]
open the drawer
Manipulation tasks usually involve more than one subtasks (e.g., “open the drawer”, “turn the first button’s color blue”, etc.), and rewards are bounded by −ntask and 0, where ntask is the number of subtasks, up to 16 in the set of environments we use. The episode ends when th...
2020
-
[24]
IDQL (Hansen-Estruch et al., 2023)
We use the default values for the other hyperparameters (e.g., noise standard deviation, noise clipping threshold, etc.), and normalize Q values only in the actor loss, following the official implementation (Tarasov et al., 2023b). IDQL (Hansen-Estruch et al., 2023). We use th...
2024
-
[25]
We use the official implementation of Cal-QL
Cal-QL (Nakamoto et al., 2023). We use the official implementation of Cal-QL. For the CQL regularizer coefficient α, we consider {0.003, 0.01, 0.03, 0.1, 0.3, 1, 3, 10} as well as its Lagrange dual variant with target action gapsβ of {0.2, 0.5, 0.8}. We use individually tuned ...
2023
-
[2012]
Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble
Lee, S., Seo, Y ., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learn- ing (CoRL), 2021b. Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutor...
2005 arXiv
-
[2013]
Accelerating online reinforcement learning with offline datasets.ArXiv, abs/2006.09359,
Nair, A., Dalal, M., Gupta, A., and Levine, S. Accelerating online reinforcement learning with offline datasets.ArXiv, abs/2006.09359,
2006 arXiv
-
[2017]
R., and Hinton, G
Ba, J., Kiros, J. R., and Hinton, G. E. Layer normalization. ArXiv, abs/1607.06450,
-
[2018]
and apply a random-shift augmentation with a probability of 0.5, following the official implementation of Park et al. (2025). In addition, we use frame stacking with three images, which we find to be important on some pixel-based tasks, such as cube and puzzle. Training and ev...
2025
-
[2019]
Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. ArXiv, abs/1312.5602,
-
[2023]
π0: A vision-language-action flow model for general robot control
9 Flow Q-Learning Black, K., Brown, N., Driess, D., Esmail, A., Equi, M., Finn, C., Fusai, N., Groom, L., Hausman, K., Ichter, B., et al. π0: A vision-language-action flow model for general robot control. ArXiv, abs/2410.24164,
-
[2024]
K., Zhang, Q., Kang, Y ., Lin, Z., and Liu, Y
Ding, Z., Jin, C., Liu, D., Zheng, H., Singh, K. K., Zhang, Q., Kang, Y ., Lin, Z., and Liu, Y . Dollar: Few-step video generation via distillation and latent reward optimization. ArXiv, abs/2412.15689, 2024b. Ding, Z., Zhang, A., Tian, Y ., and Zheng, Q. Diffusion world model...
-
[2025]
D4rl: Datasets for deep data-driven reinforcement learning
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. ArXiv, abs/2004.07219,
2004 arXiv
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.