RECORD
Quinoa: a Q-function You Infer Normalized Over Actions
Machine Learning · Submitted 5 Nov 2019
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. | contribution 0.99 | 0.93 | 0.36 | asserted 1.00 |
| s1 | We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. | method 0.91 | 0.82 | 0.09 | asserted 1.00 |
| s2 | Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. | method 0.96 | 0.83 | 0.10 | asserted 0.97 |
| s3 | We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. | contribution 0.40 | 0.85 | 0.22 | asserted 1.00 |
| s4 | We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact. | result 0.99 | 0.96 | 0.43 | asserted 1.00 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
We present an algorithm for learning an approximate action-value soft Q-function in the relative entropy regularised reinforcement learning setting, for which an optimal improved policy can be recovered in closed form. We use recent advances in normalising flows for parametrising the policy together with a learned value-function; and show how this combination can be used to implicitly represent Q-values of an arbitrary policy in continuous action space. Using simple temporal difference learning on the Q-values then leads to a unified objective for policy and value learning. We show how this approach considerably simplifies standard Actor-Critic off-policy algorithms, removing the need for a policy optimisation step. We perform experiments on a range of established reinforcement learning benchmarks, demonstrating that our approach allows for complex, multimodal policy distributions in continuous action spaces, while keeping the process of sampling from the policy both fast and exact.
Human review
References
How the literature uses it
3 arXiv papers cite this in 4 sentences
Papers citing it · 2 sentences without a confident label
Uses its method or tool 1 paper
-
To handle policy-induced covariate shift without necessitating an extra critic network, we propose using the reparameterization trick introduced by [ref] to regress future returns (i.e. $Q$-values) in terms of the log of policy ratios, allowing us to “read-off" the corresponding soft optimal policy, replacing the usual two-step procedure of fitting a critic followed by an explicit policy optimization step with a single step procedure.
Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF · Introduction · 0.96
What it cites · 9 sentences without a confident label
Uses the method or tool of 2 papers
-
We can choose to parameterise $V^s_π(s)$ as a deep neural network, and $π(a|s)$ as a density modelled by a normalising flow [ref] .
Variational Inference with Normalizing Flows · Quinoa · 0.95
-
In the following, we chose to use a Real NVP architecture [ref] for our policy, as we can both sample and infer the probability density function efficiently[We note that to the best of our knowledge there is no formal proof that Real NVP's are universal density function approximators, nor any counterexamples of why they would not be.
Density estimation using Real NVP · Quinoa · 0.96
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.