RecBayes: Recurrent Bayesian Ad Hoc Teamwork in Large Partially Observable Domains

Alberto Sardinha; Francisco S. Melo; Jo\~ao G. Ribeiro; Matthijs Spaan; Yaniv Oren

REVIEW 4 major objections 7 minor 8 references

RecBayes: Recurrent Bayesian Ad Hoc Teamwork in Large Partially Observable Domains

T0 review · 4 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that a recurrent classifier trained only on the ad hoc agent's own partial-observation trajectories can identify known teams and tasks in real time, reaching near-optimal assistance without ever seeing environment states…

desk verdict RecBayes is a credible step forward for partial-observability ad hoc teamwork, with a real distribution-shift gap that needs analysis but is not fatal. read the letter →

arxiv 2506.15756 v1 pith:T74N3RRR submitted 2025-06-18 cs.MA cs.AIcs.LG

João G. Ribeiro , Yaniv Oren , Alberto Sardinha , Matthijs Spaan , Francisco S. Melo This is my paper · ORCID

classification cs.MAcs.AIcs.LG

keywords adhocteamworkpartialobservabilityrecurrentBayesianclassifierteamidentificationtaskPOMDPmulti-agentsystemsRecBayes

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RecBayes tackles ad hoc teamwork under partial observability: an agent is dropped into a team it has not coordinated with and must help. The paper's claim is that the agent can identify which known team and task it faces using only its own actions and partial observations, with no access to the environment's full state and no access to teammates' actions at any stage. That identification is learned by a recurrent Bayesian classifier trained on past trajectories, which approximates the exact Bayesian update over candidate team-task models. In two benchmark domains scaled to roughly 970,000 states and $2^{125}$ possible observations, the classifier's beliefs drive a mixed best-response policy that reaches near-optimal normalized performance in most settings. The payoff is a route to ad hoc teamwork that does not need tabular models of the world or privileged information.

What carries the argument

The load-bearing object is the recurrent Bayesian classifier $\hat{p}_{\theta}$, a recurrent network trained with categorical cross-entropy on labelled trajectories to output a distribution over team-tasks. It stands in for the exact Bayesian update of Equation 1, which would require transition and observation probabilities for every candidate POMDP and is infeasible for large spaces. The classifier's belief $\hat{p}_{\theta}(k \mid \phi(z_t), h_t)$ is combined in Equation 3 with the best-response policy $\pi_{\psi_k}$ for each known team-task to form the agent's mixed policy. Relaxing observations to arbitrary features $\phi(z_t)$ is what removes the need for a tabular model.

What would settle it

Retrain or evaluate RecBayes in a moderately sized partially observable domain where the exact posterior of Equation 1 can be computed from the POMDP models; if the classifier's belief consistently lags behind or disagrees with the exact posterior, or if normalized performance drops below the 0.90 near-optimal threshold when training and evaluation policies differ, the approximation claim would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that a recurrent Bayesian classifier trained on trajectories of the form $\tau=(a_0,z_1,a_1,\ldots,a_{L-1},z_L)$, collected while the ad hoc agent explored each known team-task under partial observability, approximates the exact Bayesian posterior over team-tasks that Equation 1 would compute with full POMDP models. At run time the classifier's output distribution over team-tasks weights the best-response policies of all known team-tasks, producing the mixed policy of Equation 3. Because the classifier consumes arbitrary features $\phi(z_t)$ of the observation rather than a tabular entry, the approach scales to domains with roughly $970{,}200$ states and $2^{125}$ observations. The paper presents evidence that this identifies the correct team-task on the fly in both team-identification and task-identification settings, and that the resulting assistance is near-optimal in most evaluated cases.

Load-bearing premise

The method assumes that trajectories collected while the agent is learning with exploratory policies are representative enough for the classifier to keep identifying team-tasks at evaluation time, when the agent is instead executing the mixed policy and thereby changing what it observes.

Editorial extensions

If this is right

An ad hoc agent that has only ever seen its own past interactions with a team can later assist that same team without any privileged state information being available during training or deployment.
The tabular bottleneck of earlier partial-observability approaches, which were limited to domains of a few thousand states, is broken; the same pipeline applies to much larger state and observation spaces as long as trajectories can be collected.
The team-task identification problem is reduced to recurrent sequence classification, so the classifier and the best-response policies can be trained independently and combined only at run time.
Because the method never needs teammate actions, it applies to settings where other agents' decisions are private or unobservable, which is the rule in many real-world deployments rather than the exception.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper's own experiments, an immediate test of the underlying approximation is to compare the classifier's belief trajectory with the exact posterior of Equation 1 in a small domain where that posterior is computable; a systematic divergence would mean the method succeeds despite, not because of, its Bayesian justification.
The paper trains the classifier on trajectories collected under exploratory policies and evaluates it under the mixed policy of Equation 3; retraining on evaluation-policy trajectories would test whether that distribution shift matters for identification accuracy.
Because the classifier accepts arbitrary observation features $\phi(z_t)$, a natural next step is high-dimensional or continuous observations, such as raw images; the algorithm itself would not need to change.
A scaling stress test the paper does not report is holding the domain fixed while increasing the number of known team-task combinations; identification accuracy should degrade with more alternatives, and the rate of that degradation would guide practical use.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

RecBayes is the first ad hoc teamwork method under partial observability that never needs the state or teammates' actions and can handle environments with a million states, and the experiments mostly back that up. The core move is straightforward and credible: instead of doing the exact tabular Bayesian update from Ribeiro et al. (2023a), train a GRU-based sequence classifier on trajectories labeled by team-task, then use its posterior to weight a library of best-response policies. In Table 2, RecBayes is near-oracle in most 7x7 cases and in several 10x10 cases, and clearly above random everywhere. Figure 4 shows the correct team-task rising to near 1 over time. That is a real empirical result for a real restriction.

The paper is also honest about its scope: it only handles known teams and tasks, and says so in the introduction. The notation error in Eq. 3 is cosmetic.

Now the soft spots. The stress-test concern is the right one. Algorithm 1 collects training trajectories by executing each single best-response policy; Algorithm 3 evaluates under the posterior-weighted mixture. The agent's own action is part of the classifier's evidence, so the action distribution shift changes the observation distribution, and the posterior affects the policy which affects the observations. The authors don't report calibration or classification accuracy on evaluation-distribution prefixes, so the online update's validity is not directly established. This matters, but in the tested settings the aggregate results indicate the shift did not destroy performance. It is a gap, not a refutation.

The other issues are milder. 'Arbitrarily large' is overclaiming: two grid sizes, up to ~1M states, do not demonstrate arbitrary scale. Figure 4 has no error bars or statistical aggregation, so the identification claim is weaker than it looks. The repository is only named as 'Anonymised for Review'; no code link in the preprint, which makes the empirical claims harder to check. All of these are fixable.

Who this is for: people working in ad hoc teamwork, POMDP-based team identification, or MARL with pre-trained teammate libraries. A serious referee should get it; the central idea is clear, the experiments are appropriate, and the main weakness is a missing analysis rather than a fatal flaw. I would accept it with major revision, asking for code, error bars, a calibration analysis, and a softer scaling claim.

Referee Report

4 major / 7 minor

Summary. This paper proposes RecBayes, an ad hoc teamwork method for partially observable environments. RecBayes trains a recurrent classifier on trajectories collected during previous interactions with known team-task combinations, where the agent uses only its own partial observations and actions; at deployment, the classifier's output is used as an approximate posterior over known team-tasks to weight a library of best-response policies. The paper claims that this is the first approach that never requires states of the environment or teammates' actions at any stage and that can scale to large domains, and it evaluates on Level-Based Foraging and Predator-Prey with up to 970,200 states and 2^125 observations, reporting near-optimal normalized performance in most settings.

Significance. If the empirical results hold, RecBayes is a practically useful contribution: it removes the privileged-information requirement and the tabular-model bottleneck of prior ad hoc teamwork methods, and the experimental protocol—including normalization against original teammate and random policies, two domains, and three identification sub-problems—is a reasonable first test. The paper also discloses hyperparameters and reports a large number of trials. The key open question is whether the learned recurrent classifier is a reliable substitute for the exact Bayesian update under the deployment policy, and the current evidence does not yet establish that beyond the tested settings.

major comments (4)

[Section 4, Algorithms 1 and 3] The training and deployment distributions over the agent's own actions differ: Algorithm 1 collects trajectories by executing the dedicated policy π_ψk for each team-task, whereas Algorithm 3 executes the mixture π_t = Σ_k p_t(k) π_ψk, which itself depends on the posterior being estimated. Because the action a_t is part of the evidence fed to p̂_θ, any early error changes future actions and therefore future observations, creating a closed-loop covariate shift. The paper does not analyze this shift or report calibration or classification accuracy on evaluation-distribution prefixes; without such evidence, the central claim that p̂_θ approximates Eq. (1) 'well enough' in general is not supported.
[Section 4, Algorithm 2] Algorithm 2 states only that the cross-entropy loss is computed on complete trajectories τ_i, and it does not specify whether the loss is applied at every recurrent step or only after the full sequence. Algorithm 3, however, queries p̂_θ after each prefix (a_0, z_1, ..., a_{t-1}, z_t) and uses those intermediate outputs as the posterior for the mixture policy. If the classifier is trained only on full-sequence logits, the intermediate beliefs are never directly supervised, and the online posterior updates may be miscalibrated. Please clarify the training objective for prefixes and report prefix-level accuracy or calibration.
[Section 6] The evaluation compares RecBayes only against general RL baselines, the original teammate, and random actions; it does not compare against any existing ad hoc teamwork algorithm such as ATPO, PO-GPL, FEAT, or ODITS. Since the paper's stated contribution is to overcome the limitations of these methods (tabular modeling and privileged information), a direct comparison on the 7x7 settings, where ATPO is feasible, is necessary to validate the claimed advantage over the state of the art.
[Section 6.1, Table 2] The near-optimality claim is based on normalized means with no confidence intervals or statistical tests. Table 2 reports a single number per condition, and the text asserts 'no statistically significant differences' without any test. Given the small number of trials and the high standard deviations in Table 1, values such as 0.86 and 0.85 in the 10x10 Task Identification rows may not be reliably below or above the 0.90 threshold; please add error bars or significance tests.

minor comments (7)

[Section 5.1] The observation space is described as 2^125, but this is only an upper bound on the 5x5x5 tuple observation space; the reachable observation distribution is far smaller, and the phrase 'up to 2^125 observations' may overstate the actual diversity. Please clarify.
[Section 4 and Algorithm 3] The initial prior p_0 is never defined; please state how it is initialized (for example, as uniform over the K team-tasks).
[Eq. (1)] The factor π_t(a_t | h_t) appears inside the sum in Eq. (1), but since it is independent of k it cancels in the normalization constant ρ; removing it would make the update easier to read.
[Eqs. (2) and (3)] The sums run from k=0 to K, while the team-tasks are indexed k ∈ {1,...,K}; this off-by-one notation should be corrected.
[Figure 4] The belief curves appear to be averaged over episodes, but the figure does not report error bars, the number of episodes, or per-episode classification accuracy; the statement that 'all team and tasks were in the end correctly classified' is stronger than what the averaged curves show.
[Section 6.1] The sentence 'All three non-optimal cases were related with Task Identification' should be quantified as two cases for RecBayes-MF and one for RecBayes-MB, based on the Table 2 values.
[Abstract and Introduction] The phrase 'arbitrarily large' is too strong for an empirical evaluation up to 970,200 states; suggest replacing it with 'substantially larger' or another more precise qualifier.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the recurrent classifier is openly trained on labeled trajectories and evaluated on fresh episodes.

full rationale

RecBayes does not present a derivation that reduces to its own inputs. The paper's stated hypothesis is that trajectories from past interactions contain enough information to approximate the Bayesian update of Eq. 1 by framing team-task identification as recurrent classification. This is an empirical claim, not a mathematical reduction. The posterior p̂_θ is trained with categorical cross-entropy against ground-truth team-task labels (Algorithm 2), and the final policy of Eq. 3 simply mixes pre-trained best-response policies with the classifier's output. Nothing in Eq. 3 is a fitted parameter renamed as a prediction; the classifier is explicitly a learned estimator, and the evaluation in Section 6 tests it on held-out interaction episodes. All cited prior work, including the authors' own ATPO, is used only as motivation, baseline, or normalization convention, and none of these citations carries the weight of the paper's central claim. The self-citations are contextual and not load-bearing. The train/evaluation distribution shift highlighted by the reader is a genuine generalization concern about covariate shift, but it is not circularity: the target labels do not depend on the fitted parameters, and the evaluation performance is not forced by construction.

Assumptions & free parameters 2 free parameters · 5 assumptions · 1 invented entities

The ledger shows the paper's contribution is an architectural and experimental pipeline, not a derivation. The only genuinely new entity is the learned recurrent classifier, which is a standard sequence model. The free parameters are the usual deep-learning hyperparameters, with no sensitivity analysis. The main domain assumptions are the known-team-task restriction and the sufficiency of the ad hoc agent's own trajectory collection, both disclosed in the text.

free parameters (2)

GRU and MLP classifier architecture dimensions = Unknown from text; partially indicated in Figure 3, full values in unavailable hyperparameters.yaml
The recurrent classifier architecture is manually fine-tuned and only partially specified in the paper. These dimensions affect the classification capacity and are load-bearing for the identification results.
PPO and DreamerV3 hyperparameters = Fine-tuned with Optuna; exact values in unavailable hyperparameters.yaml
The best-response policies are RL policies with tuned hyperparameters. They are standard tuning choices needed to achieve near-optimal policy libraries, but no sensitivity analysis is reported.

assumptions (5)

domain assumption The set of known team-task combinations K is fixed and known in advance, and the ad hoc agent only needs to identify among them.
Section 1 states the paper only considers known teams and tasks and does not address new teammates or tasks. This is an explicit limitation but also the set-up of the problem.
domain assumption Training trajectories collected by the ad hoc agent's own policies are sufficient experience to define the team-task models.
Algorithm 1 collects trajectories by the ad hoc agent's own exploration policies; the paper assumes these are representative of each team-task so the classifier learns a useful mapping from histories to labels.
domain assumption Partial observations are of fixed 5x5x5 tuple shape with five channels and a limited field of view.
Section 5.1 fixes the observation function; the method's generality to other observation shapes is asserted but not demonstrated.
domain assumption The statistical standard for effective identification is that the belief for the correct team-task eventually becomes the most-likely, without formal error bars or significance tests.
Section 6.2 concludes all team-tasks were correctly classified from Figure 4 plots, but no aggregated classification accuracy or confidence intervals are reported.
standard math Standard RL assumptions for PPO and DreamerV3 to compute near-optimal best-response policies.
The policies are trained with standard RL algorithms and the paper validates them by comparing to the Original Teammate in Table 2, where the RL baselines are near-parity.

invented entities (1)

Recurrent Bayesian classifier p_theta
purpose: Learned approximation of the posterior over team-tasks given partial-observation histories, replacing the tabular Bayesian update of Eq. 1.
This is a new component introduced by the paper, but it is an architecture choice, a GRU plus MLP, not a new physical or mathematical entity with external falsifiable predictions. Its evidence is the empirical identification curves in Figure 4, which are internal to the paper's own experiments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of RecBayes: Recurrent Bayesian Ad Hoc Teamwork in Large Partially Observable Domains." pith.science (2026). https://pith.science/paper/T74N3RRR

@misc{pith2026250615756,
  author       = {Pith},
  title        = {Pith review of: RecBayes: Recurrent Bayesian Ad Hoc Teamwork in Large Partially Observable Domains},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T74N3RRR}},
  note         = {Machine review of arXiv:2506.15756}
}

read the original abstract

This paper proposes RecBayes, a novel approach for ad hoc teamwork under partial observability, a setting where agents are deployed on-the-fly to environments where pre-existing teams operate, that never requires, at any stage, access to the states of the environment or the actions of its teammates. We show that by relying on a recurrent Bayesian classifier trained using past experiences, an ad hoc agent is effectively able to identify known teams and tasks being performed from observations alone. Unlike recent approaches such as PO-GPL (Gu et al., 2021) and FEAT (Rahman et al., 2023), that require at some stage fully observable states of the environment, actions of teammates, or both, or approaches such as ATPO (Ribeiro et al., 2023) that require the environments to be small enough to be tabularly modelled (Ribeiro et al., 2023), in their work up to 4.8K states and 1.7K observations, we show RecBayes is both able to handle arbitrarily large spaces while never relying on either states and teammates' actions. Our results in benchmark domains from the multi-agent systems literature, adapted for partial observability and scaled up to 1M states and 2^125 observations, show that RecBayes is effective at identifying known teams and tasks being performed from partial observations alone, and as a result, is able to assist the teams in solving the tasks effectively.

Figures

Figures reproduced from arXiv: 2506.15756 by the authors.

**Figure 2.** Example tuple observations of shape 5 × 5 × 5. Five different channels of size 5 × 5 provide the agent a limited field-of-view which detects both entities and other information in the near environment. Each other agent is given a separate channel since indexes are relevant in these domain, used for instance for stalemate resolution or other decision processes. smallest ones, containing 117649 total states, up to the… view at source ↗

**Figure 3.** Encoder (or input) networks used both by the Bayesian classi [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Identification of the correct team and task by RecBayes’s Bayesian Classifier pˆ. In each timestep, pˆ outputs a probability distribution over all team-tasks, with each entry representing the belief over its respective team-task. The belief for the correct team-task is…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

8 extracted references · 4 canonical work pages

[4]

Helping people on theﬂy: Ad hoc teamwork for human-robot teams

João G Ribeiro, Miguel Faria, Alberto Sardinha, and Francisco S Melo. Helping people on theﬂy: Ad hoc teamwork for human-robot teams. InProgress in Articial Intelligence: 20th EPIA Con- ference on Articial Intelligence, EPIA 2021, Virtual Event, September 7–9, 2021, Proceedings 20, pp. 635–647. Springer,

work page 2021
[5]

Making friends in the dark: Ad hoc teamwork under partial observability

João G Ribeiro, Cassandro Martinho, Alberto Sardinha, and Francisco S Melo. Making friends in the dark: Ad hoc teamwork under partial observability. InECAI 2023, pp. 1954–1961. IOS Press, 2023a. João G Ribeiro, Gonçalo Rodrigues, Alberto Sardinha, and Francisco S Melo. Teamster: Model- based reinforcement learning for ad hoc teamwork.Articial Intelligenc...

work page 2023
[2010]

ISBN 978-0-9826571-1-9

International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-0-9826571-1-9. URL http://dl.acm.org/citation.cfm?id=1838206.1838223. Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. InTwenty-F ourth AAAI Conference on Articial Intelligence,

work page arXiv
[2016]

A survey of ad hoc teamwork: Deﬁnitions, methods, and open problems.arXiv preprint arXiv:2202.10450,

11 Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork: Deﬁnitions, methods, and open problems.arXiv preprint arXiv:2202.10450,

arXiv
[2021]

Lillicrap

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P . Lillicrap. Mastering diverse domains through world models.CoRR, abs/2301.04104,

arXiv
[2022]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv
[2023]

DOI: 10.48550/ARXIV .2301.04104. F. Melo and A. Sardinha. Ad hoc teamwork by learning teammates’ task.Autonomous Agents and Multi-Agent Systems, 30(2):175–219,

arXiv doi:10.48550/arxiv.2301.04104
[2024]

Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

Pedro P Santos, Diogo S Carvalho, Miguel V asco, Alberto Sardinha, Pedro A Santos, Ana Paiva, and Francisco S Melo. Centralized training with hybrid execution in multi-agent reinforcement learning.arXiv preprint arXiv:2210.06274,

work page Pith review arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [4]

Helping people on theﬂy: Ad hoc teamwork for human-robot teams

João G Ribeiro, Miguel Faria, Alberto Sardinha, and Francisco S Melo. Helping people on theﬂy: Ad hoc teamwork for human-robot teams. InProgress in Articial Intelligence: 20th EPIA Con- ference on Articial Intelligence, EPIA 2021, Virtual Event, September 7–9, 2021, Proceedings 20, pp. 635–647. Springer,

work page 2021

[2] [5]

Making friends in the dark: Ad hoc teamwork under partial observability

João G Ribeiro, Cassandro Martinho, Alberto Sardinha, and Francisco S Melo. Making friends in the dark: Ad hoc teamwork under partial observability. InECAI 2023, pp. 1954–1961. IOS Press, 2023a. João G Ribeiro, Gonçalo Rodrigues, Alberto Sardinha, and Francisco S Melo. Teamster: Model- based reinforcement learning for ad hoc teamwork.Articial Intelligenc...

work page 2023

[3] [2010]

ISBN 978-0-9826571-1-9

International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978-0-9826571-1-9. URL http://dl.acm.org/citation.cfm?id=1838206.1838223. Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. InTwenty-F ourth AAAI Conference on Articial Intelligence,

work page arXiv

[4] [2016]

A survey of ad hoc teamwork: Deﬁnitions, methods, and open problems.arXiv preprint arXiv:2202.10450,

11 Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork: Deﬁnitions, methods, and open problems.arXiv preprint arXiv:2202.10450,

arXiv

[5] [2021]

Lillicrap

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy P . Lillicrap. Mastering diverse domains through world models.CoRR, abs/2301.04104,

arXiv

[6] [2022]

Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347,

arXiv

[7] [2023]

DOI: 10.48550/ARXIV .2301.04104. F. Melo and A. Sardinha. Ad hoc teamwork by learning teammates’ task.Autonomous Agents and Multi-Agent Systems, 30(2):175–219,

arXiv doi:10.48550/arxiv.2301.04104

[8] [2024]

Centralized Training with Hybrid Execution in Multi-Agent Reinforcement Learning

Pedro P Santos, Diogo S Carvalho, Miguel V asco, Alberto Sardinha, Pedro A Santos, Ana Paiva, and Francisco S Melo. Centralized training with hybrid execution in multi-agent reinforcement learning.arXiv preprint arXiv:2210.06274,

work page Pith review arXiv