REVIEW 22 cited by
Contextual Markov Decision Processes
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Contextual Markov Decision Processes
read the original abstract
We consider a planning problem where the dynamics and rewards of the environment depend on a hidden static parameter referred to as the context. The objective is to learn a strategy that maximizes the accumulated reward across all contexts. The new model, called Contextual Markov Decision Process (CMDP), can model a customer's behavior when interacting with a website (the learner). The customer's behavior depends on gender, age, location, device, etc. Based on that behavior, the website objective is to determine customer characteristics, and to optimize the interaction between them. Our work focuses on one basic scenario--finite horizon with a small known number of possible contexts. We suggest a family of algorithms with provable guarantees that learn the underlying models and the latent contexts, and optimize the CMDPs. Bounds are obtained for specific naive implementations, and extensions of the framework are discussed, laying the ground for future research.
Forward citations
Cited by 22 Pith papers
-
World Action Planner: Generalizable Decision-Making with Action-Conditioned World Models
Pairing VLM-generated action proposals with rollouts from a pose-image-conditioned video world model yields high success rates in novel simulated manipulation tasks without end-to-end policy retraining.
-
Partner Capability Estimation for Task-Agnostic Adaptation in Ad-Hoc Teamwork
CE-CM infers discrete task-invariant partner capability vectors online via approximate Bayesian simulate-and-compare, and CE-CM-Div improves estimates when humans use diverse suboptimal strategies.
-
DishSeg24k: A Large-Scale Benchmark for Food Segmentation with Stochastic Expert Decoding
DishSeg24k is a 24k-image dish-level food segmentation benchmark, and the FEAST model reports +3.21 mIoU over prior methods, mostly from its mixture-of-experts decoder.
-
Generalization in offline RL: The structure is more important than the amount of pessimism
Symmetric (even highly) pessimistic value functions can generalize optimally in GTI-ZSPT offline RL while mildly pessimistic asymmetric ones cannot; DAC-Output during extraction is the strongest DA method tested.
-
Generalization in offline RL: The structure is more important than the amount of pessimism
In offline RL, the structure of pessimism (set by dataset coverage) matters more for generalization than its amount; a symmetric overly pessimistic value function can outperform a non-symmetric mildly pessimistic one.
-
Formalizing Task-Space Complexity for Zero-Shot Generalization
Introduces signed divergence to bound generalization gaps and defines task-space complexity as the minimum source contexts needed for ε-coverage under local smoothness, with set-cover reduction and empirical validatio...
-
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
FCGraft synthesizes code policies for embodied agents by grafting KV caches from a library of validated functions, claiming 18.31% higher success rate and 2.3x faster synthesis than prompt-level caching.
-
Curriculum reinforcement learning with measurable task representation learning
A VAE-based latent task representation enables automatic curriculum generation in CRL for non-Euclidean navigation tasks, outperforming interpolation and GAN-based methods in experiments.
-
Ada-Diffuser: Latent-Aware Adaptive Diffusion for Decision-Making
Ada-Diffuser is a causal diffusion model that jointly learns observed interaction structure and underlying latent dynamics from minimal observations for adaptive planning and policy learning.
-
Behavior-Constrained Reinforcement Learning with Receding-Horizon Credit Assignment for High-Performance Control
A behavior-constrained RL framework with receding-horizon credit assignment learns high-performance control policies that stay aligned with expert behavior in race car simulation.
-
Stochastic Optimal Control with Side Information and Bayesian Learning
For infinite-horizon control with unknown context-conditional noise, the Bayesian Bellman value function converges uniformly, and the scaled optimal value is asymptotically normal only if an unproved √N-equivalence as...
-
MDP modeling for multi-stage stochastic programs
Extends policy graphs for decision-dependent uncertainty in MDPs and develops SDDP variants for multi-stage stochastic programs with continuous state and action spaces.
-
Functional Cache Grafting: Robust and Rapid Code-Policy Synthesis for Embodied Agents
By reusing validated function-level KV caches (stitching) and regenerating only localized error spans (patching), FCGraft makes CodeLLM policies for embodied agents faster and more robust than prompt-level caching.
-
MATE: Solving Contextual Markov Decision Processes with Memory of Accumulated Transition Embeddings
MATE uses permutation-invariant sum-aggregated memory of transition embeddings to solve CMDPs with online adaptation and computational advantages over Transformers and RNNs.
-
Task-specific Subnetwork Discovery in Reinforcement Learning for Autonomous Underwater Navigation
Contextual multi-task RL for underwater navigation uses just 1.5% of network weights for task differentiation, mostly from context-variable connections to the first hidden layer.
-
Contextual Intelligence The Next Leap for Reinforcement Learning
Reinforcement learning agents can generalize better by treating context as a first-class primitive that distinguishes slow-changing external factors from fast-changing internal ones and incorporates abstract high-leve...
-
Pushing Forward Pareto Frontiers of Proactive Agents with Behavioral Agentic Optimization
BAO, a behavior-enhanced SFT plus regularized RL pipeline, improves proactive agents' task performance while lowering user-involvement rate, beating UserRL baselines on three UserRL gym tasks.
-
Fully Decentralized Cooperative Multi-Agent Reinforcement Learning is A Context Modeling Problem
DAC models fully decentralized cooperative MARL as a context modeling problem, using latent variables for joint policies to fix non-stationarity in value updates and relative overgeneralization in value estimation.
-
Deep Reinforcement Learning for Spacecraft Attitude Control During Atmospheric Re-Entry
Hybrid RL-PID controllers track angle of attack better and show greater robustness than PID alone within a defined operational envelope for re-entry attitude control.
-
Contextual Multi-Task Reinforcement Learning for Autonomous Reef Monitoring
A context-dependent multi-task RL policy is trained and evaluated in HoloOcean simulation to solve multiple reef monitoring tasks with claimed improvements in sample efficiency, zero-shot generalization, and robustnes...
-
Contextual Multi-Task Reinforcement Learning for Autonomous Reef Monitoring
Contextual multi-task DDQN learns one AUV policy for multiple simulated reef-monitoring tasks that matches mixture-of-experts performance and generalizes better on a discrete toy domain.
-
Adaptive Exploration for Latent-State Bandits
A family of LinUCB-based algorithms using lagged observations and adaptive probing reduces dynamic regret in latent-state two-armed bandits.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.