Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning

Guy Tennenholtz; Shie Mannor

RECORD

Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning

Guy Tennenholtz, Shie Mannor

Machine Learning · Submitted 22 Feb 2021

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim We leverage our method for uncertainty estimation in a pessimistic model-based framework, showing a significant improvement upon contemporary model-based offline approaches on continuous control and autonomous driving benchmarks.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2102.11327 (version 2)

Pith IDRMEFIHRV, Pith's permanent record of this paper and its review

CitationGuy Tennenholtz, Shie Mannor. "Uncertainty Estimation Using Riemannian Model Dynamics for Offline Reinforcement Learning." arXiv:2102.11327 (2021). https://pith.science/paper/RMEFIHRV

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,812 input tokens.

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.05 · human participants 0.03 · controlled comparison 0.67 · randomized assignment 0.01 · states quantitative result 0.05 · compares with prior results 0.93 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Model-based offline reinforcement learning approaches generally rely on bounds of model error. Estimating these bounds is usually achieved through uncertainty estimation methods. In this work, we combine parametric and nonparametric methods for uncertainty estimation through a novel latent space based metric. In particular, we build upon recent advances in Riemannian geometry of generative models to construct a pullback metric of an encoder-decoder based forward model. Our proposed metric measures both the quality of out-of-distribution samples as well as the discrepancy of examples in the data. We leverage our method for uncertainty estimation in a pessimistic model-based framework, showing a significant improvement upon contemporary model-based offline approaches on continuous control and autonomous driving benchmarks.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Model-based offline reinforcement learning approaches generally rely on bounds of model error.	background 1.00	0.03	0.02	asserted 0.99
s1	Estimating these bounds is usually achieved through uncertainty estimation methods.	background 1.00	0.04	0.02	asserted 0.98
s2	In this work, we combine parametric and nonparametric methods for uncertainty estimation through a novel latent space based metric.	contribution 0.88	0.93	0.08	asserted 1.00
s3	In particular, we build upon recent advances in Riemannian geometry of generative models to construct a pullback metric of an encoder-decoder based forward model.	method 0.93	0.74	0.03	asserted 0.99
s4	Our proposed metric measures both the quality of out-of-distribution samples as well as the discrepancy of examples in the data.	method 0.78	0.91	0.07	asserted 1.00
s5	We leverage our method for uncertainty estimation in a pessimistic model-based framework, showing a significant improvement upon contemporary model-based offline approaches on continuous control and autonomous driving benchmarks.	result 0.98	0.96	0.91	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from simulation or computation 0.51.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Model-based offline reinforcement learning approaches generally rely on bounds of model error. Estimating these bounds is usually achieved through uncertainty estimation methods. In this work, we combine parametric and nonparametric methods for uncertainty estimation through a novel latent space based metric. In particular, we build upon recent advances in Riemannian geometry of generative models to construct a pullback metric of an encoder-decoder based forward model. Our proposed metric measures both the quality of out-of-distribution samples as well as the discrepancy of examples in the data. We leverage our method for uncertainty estimation in a pessimistic model-based framework, showing a significant improvement upon contemporary model-based offline approaches on continuous control and autonomous driving benchmarks.

Human review

How Pith checks papers → · Full audit of this paper →

References

How the literature uses it

3 arXiv papers cite this in 3 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 2 sentences without a confident label

Cites it as background 1 paper

Other than the uncertainty quantification, previous model-based methods also attempt to constrain the learned policy through BC loss [ref] , advantage-weighted prior [ref] , CQL-style penalty [ref] , and Riemannian submanifold [ref] .

Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learning · Related Work · 0.89

What it cites · 17 sentences without a confident label

Uses the method or tool of 3 papers

In this work we utilize a recently proposed model-based offline RL framework (MOPO, [ref] ).

MOPO: Model-based Offline Policy Optimization · Offline Reinforcement Learning · 0.98
In this work we employ recent progress on the geometry of variational latent models [ref] to control and mitigate the proximity-uncertainty tradeoff.

Geometrically Enriched Latent Spaces · Introduction · 0.81
First, the model was fully trained using a calibrated Gaussian decoder [ref] .

Simple and Effective VAE Training with Calibrated Decoders · Continuous Control · 1.00

Uses data or values from 1 paper

We used D4RL [ref] as a benchmark for all of our experiments.

D4RL: Datasets for Deep Data-Driven Reinforcement Learning · Continuous Control · 0.89

Compares with 2 papers

We compared GELATO and its variants to contemporary model-based offline RL approaches; namely, MOPO [ref] and MBPO [ref] , as well as the standard baselines of SAC [ref] and imitation (behavioral cloning).

When to Trust Your Model: Model-Based Policy Optimization · Continuous Control · 1.00
We compared GELATO and its variants to contemporary model-based offline RL approaches; namely, MOPO [ref] and MBPO [ref] , as well as the standard baselines of SAC [ref] and imitation (behavioral cloning).

MOPO: Model-based Offline Policy Optimization · Continuous Control · 1.00

Cites as background 12 papers

Estimating uncertainty can be achieved via confidence sets (i.e., UCB and LCB) [ref] , ensemble methods [ref] , Monte-Carlo dropout [ref] , or by modeling the distribution over functions [ref] .

MOPO: Model-based Offline Policy Optimization · Introduction · 0.84
Methods based on proximity maintain similarity to the data distribution, constraining the policy to remain close to the behavior in the data [ref] .

Conservative Q-Learning for Offline Reinforcement Learning · Introduction · 0.86
Methods based on proximity maintain similarity to the data distribution, constraining the policy to remain close to the behavior in the data [ref] .

Overcoming Model Bias for Robust Offline Deep Reinforcement Learning · Introduction · 0.86

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.