A Multi-Agent Reinforcement Learning Framework for Public Health Decision Analysis

Ankit Shah; Chaitra Gopalappa; Dinesh Sharma

arxiv: 2311.00855 · v3 · submitted 2023-11-01 · 💻 cs.AI · cs.LG· cs.MA

A Multi-Agent Reinforcement Learning Framework for Public Health Decision Analysis

Dinesh Sharma , Ankit Shah , Chaitra Gopalappa This is my paper

Pith reviewed 2026-05-24 05:31 UTC · model grok-4.3

classification 💻 cs.AI cs.LGcs.MA

keywords HIVmulti-agent reinforcement learningpublic healthresource allocationdecision supportepidemic managementjurisdictional interactionsEnding the HIV Epidemic

0 comments

The pith

A multi-agent reinforcement learning framework optimizes HIV resource allocation across jurisdictions by modeling their epidemiological interactions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a multi-agent reinforcement learning framework to support decisions on allocating HIV prevention, treatment, and diagnosis resources across multiple jurisdictions. It aims to demonstrate that this approach, which factors in how actions in one area influence transmission in connected areas, outperforms methods that optimize each jurisdiction in isolation. Tests on California and Florida data show the multi-agent policies cut new infections more effectively when total spending is held fixed. This matters for initiatives like Ending the HIV Epidemic because they require coordinated choices across regions that differ in burden and are linked by movement and care access. The work presents the system as a decision-support tool that can scale to other public-health planning tasks.

Core claim

We propose a multi-agent reinforcement learning framework that enables jurisdiction-specific decision-making while accounting for cross-jurisdictional epidemiological interactions. Our framework functions as an intelligent resource optimization system, helping policymakers strategically allocate interventions based on dynamic, data-driven insights. Experimental results across jurisdictions in California and Florida demonstrate that MARL-driven policies outperform traditional single-agent reinforcement learning approaches by reducing new infections under fixed budget constraints.

What carries the argument

The multi-agent reinforcement learning framework that lets each jurisdiction choose its own interventions while the shared environment captures transmission effects between them.

If this is right

MARL policies reduce new infections more than single-agent reinforcement learning under identical budget limits.
Decision frameworks must incorporate jurisdictional dependencies to improve outcomes for large-scale public health programs.
The approach advances expert systems for government resource planning in epidemic management.
The framework provides a scalable template for similar resource-allocation problems in healthcare policy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same structure could be tested on other diseases whose spread crosses administrative boundaries, such as tuberculosis or influenza.
Policymakers could run the model on updated surveillance data each year to revise allocation plans without rebuilding the entire system.
National strategies that ignore inter-jurisdictional spillovers may systematically underperform once real movement patterns are included.

Load-bearing premise

The underlying epidemiological simulation model accurately represents real cross-jurisdictional interactions and the effects of interventions on transmission dynamics.

What would settle it

Implement the learned MARL policies versus single-agent policies in one or more real jurisdictions and compare the observed change in new HIV infections over the same budget period.

Figures

Figures reproduced from arXiv: 2311.00855 by Ankit Shah, Chaitra Gopalappa, Dinesh Sharma.

**Figure 2.** Figure 2: Comparing total new incidence from SARL and MARL in a) California and b) Florida, and cost difference [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Figure 3: Comparing total new incidence from MARL with a) initial & increased actions and b) initial & increased [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

read the original abstract

Human immunodeficiency virus (HIV) is a major public health concern in the United States (U.S.), with about 1.2 million people living with it and about 35,000 newly infected each year. There are considerable geographical disparities in HIV burden and care access across the U.S. The 'Ending the HIV Epidemic (EHE)' initiative by the U.S. Department of Health and Human Services aims to reduce new infections by 90% by 2030, by improving coverage of diagnoses, treatment, and prevention interventions and prioritizing jurisdictions with high HIV prevalence. We develop intelligent decision-support systems to optimize resource allocation and intervention strategies. Existing decision analytic models either focus on individual cities or aggregate national data, failing to capture jurisdictional interactions critical for optimizing intervention strategies. To address this, we propose a multi-agent reinforcement learning (MARL) framework that enables jurisdiction-specific decision-making while accounting for cross-jurisdictional epidemiological interactions. Our framework functions as an intelligent resource optimization system, helping policymakers strategically allocate interventions based on dynamic, data-driven insights. Experimental results across jurisdictions in California and Florida demonstrate that MARL-driven policies outperform traditional single-agent reinforcement learning approaches by reducing new infections under fixed budget constraints. Our study highlights the importance of incorporating jurisdictional dependencies in decision-making frameworks for large-scale public initiatives. By integrating multi-agent intelligent systems, decision analytics, and reinforcement learning, this study advances expert systems for government resource planning and public health management, offering a scalable framework for broader applications in healthcare policy and epidemic management.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper applies MARL to HIV resource allocation across interacting jurisdictions and claims better results than single-agent RL, but everything rests on an unvalidated simulator.

read the letter

The central point is that this work frames HIV intervention allocation as a multi-agent problem to capture cross-jurisdictional effects in places like California and Florida, and the experiments say MARL reduces new infections more than single-agent RL under fixed budgets. That framing makes sense given the EHE initiative's focus on high-prevalence areas and the fact that mobility and transmission do cross borders. The paper correctly notes that prior models either zoom in on one city or lump everything nationally, so treating jurisdictions as interacting agents is a direct response to that gap. It also positions the output as a decision-support tool rather than pure theory, which fits the applied side of the work. The soft spot is the complete lack of grounding for the simulator itself. The abstract gives no information on how the epidemiological model was built, where the mobility or transmission parameters came from, or whether simulated baselines match observed county-level case counts from 2015-2022. Without sensitivity checks on the strength of inter-jurisdiction coupling or any external validation against CDC data, the reported advantage could simply reflect how the simulator rewards coordination that may not exist in reality. No training details, error bars, or statistical tests are mentioned either, so the performance difference cannot be assessed on its own terms. This paper is aimed at researchers who build RL systems for public health policy or epidemic resource planning. A reader already working on multi-region models might pick up the jurisdictional interaction angle even if the execution needs more work. It deserves a serious referee because the underlying problem is real and the MARL extension is a logical next step, though any review would have to focus heavily on model validation and reproducibility.

Referee Report

2 major / 1 minor

Summary. The paper proposes a multi-agent reinforcement learning (MARL) framework for optimizing HIV intervention resource allocation and strategies across jurisdictions in California and Florida. It claims that MARL-driven policies outperform traditional single-agent RL by reducing new infections under fixed budget constraints, by incorporating cross-jurisdictional epidemiological interactions, and positions the work as a decision-support system for public health initiatives like Ending the HIV Epidemic.

Significance. If the simulation model accurately captures real dynamics, the framework could advance expert systems for government resource planning by showing the value of multi-agent coordination over single-agent approaches in epidemic management.

major comments (2)

Abstract: the claim that MARL outperforms single-agent RL supplies no information on the epidemiological model structure, data sources, training procedure, validation, statistical tests, or error bars, so the reported performance difference cannot be assessed.
Results (experimental comparison across CA/FL jurisdictions): no sensitivity analysis on inter-jurisdiction coupling strength or comparison of simulated baseline trajectories to observed 2015–2022 county-level case counts is described, which is load-bearing for the central outperformance claim.

minor comments (1)

The abstract and introduction could more explicitly define the state/action spaces and reward function used in the MARL formulation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed and constructive review. We address each major comment below and will incorporate revisions to strengthen the manuscript.

read point-by-point responses

Referee: Abstract: the claim that MARL outperforms single-agent RL supplies no information on the epidemiological model structure, data sources, training procedure, validation, statistical tests, or error bars, so the reported performance difference cannot be assessed.

Authors: We agree the abstract is too concise and omits key methodological details. In the revision we will expand the abstract (within length limits) to briefly describe the compartmental epidemiological model, the use of public CDC and state health department data sources for CA/FL jurisdictions, the centralized-training decentralized-execution MARL procedure, and the reporting of results with multiple random seeds, error bars, and statistical tests. revision: yes
Referee: Results (experimental comparison across CA/FL jurisdictions): no sensitivity analysis on inter-jurisdiction coupling strength or comparison of simulated baseline trajectories to observed 2015–2022 county-level case counts is described, which is load-bearing for the central outperformance claim.

Authors: The current results section focuses on policy performance under the calibrated model but does not include the requested analyses. We will add (1) a sensitivity study varying the inter-jurisdiction mobility/coupling parameter across a range of values and (2) a validation subsection that compares simulated baseline incidence trajectories against observed 2015–2022 county-level case counts, reporting appropriate goodness-of-fit metrics. These additions will be placed in a new subsection of the results. revision: yes

Circularity Check

0 steps flagged

No significant circularity; simulation-based comparison is self-contained

full rationale

The paper proposes a MARL framework for HIV intervention allocation and reports that MARL policies reduce new infections relative to single-agent RL under fixed budgets, as measured inside an epidemiological simulator across CA/FL jurisdictions. This comparison is generated by running both policies through the same forward simulation; the performance metric is not obtained by fitting a parameter to the target outcome and then relabeling it as a prediction, nor does any equation define the outperformance by construction. No self-citation is invoked to establish uniqueness of the MARL formulation or to forbid alternatives. The central claim therefore remains an internal experimental result rather than a tautology. The skeptic concern about simulator fidelity to real data is a question of external validity, not circularity in the derivation chain.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review yields no explicit free parameters, axioms, or invented entities; the framework is described at a high level without mathematical or modeling specifics.

pith-pipeline@v0.9.0 · 5809 in / 1059 out tokens · 29648 ms · 2026-05-24T05:31:36.473071+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

12 extracted references · 12 canonical work pages · 2 internal anchors

[1]

HIV .gov. U.S. Statistics.https://www.hiv.gov/hiv-basics/overview/data-and-trend/statistics , 2023a. (accessed: 06.10.2023). HIV .gov. Viral Suppression and an Undetectable Viral load. https://www.hiv.gov/hiv-basics/ staying-in-hiv-care/hiv-treatment/viral-suppression#:~:text=Viral%20suppression%20is% 20defined%20as,having%20\an%20undetectable%20viral%20l...

work page 2023
[2]

(accessed: 06.10.2022). AHEAD. EHE Goal. https://www.ahead.hiv.gov/faqs, 2022a. (accessed: 08.25.2022). Bohdan Nosyk, Xiao Zang, Emanuel Krebs, Benjamin Enns, Jeong E Min, Czarina N Behrends, Carlos Del Rio, Julia C Dombrowski, Daniel J Feaster, Matthew Golden, et al. Ending the hiv epidemic in the usa: an economic modelling study in six cities. The Lance...

work page 2022
[3]

Impact of improved hiv care and treatment on prep effectiveness in the united states, 2016–2020

Nidhi Khurana, Emine Yaylali, Paul G Farnham, Katherine A Hicks, Benjamin T Allaire, Evin Jacobson, and Stephanie L Sansom. Impact of improved hiv care and treatment on prep effectiveness in the united states, 2016–2020. JAIDS Journal of Acquired Immune Deficiency Syndromes, 78(4):399–405,

work page 2016
[4]

Reinforcement learning for optimization of covid-19 mitigation policies

Varun Kompella, Roberto Capobianco, Stacy Jong, Jonathan Browne, Spencer Fox, Lauren Meyers, Peter Wur- man, and Peter Stone. Reinforcement learning for optimization of covid-19 mitigation policies. arXiv preprint arXiv:2010.10560,

work page arXiv 2010
[5]

Playing Atari with Deep Reinforcement Learning

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,

work page internal anchor Pith review Pith/arXiv arXiv
[6]

Dueling network architectures for deep reinforcement learning

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR,

work page 1995
[7]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

work page internal anchor Pith review Pith/arXiv arXiv
[8]

Accessed: 2023-09-17. HRSA. Ryan White HIV/AIDS Program FY 2023 EHE Awards. https://ryanwhite.hrsa.gov/about/ parts-and-initiatives/fy-2023-ending-hiv-epidemic-awards ,

work page 2023
[9]

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

Accessed: 2023-09-17. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR,

work page 2023
[10]

(accessed: 09.17.2023). CDC. HIV Surveillance Report V olume 27, Number

work page 2023
[11]

(accessed: 09.17.2023). AHEAD. PrEP Coverage. https://ahead.hiv.gov/data/prep-coverage, 2022b. (accessed: 09.17.2023). 12 HIV MARL A Appendix A.1 Hyperparameters Table A1: List of hyperparameters used in PPO Hyperparameters Value Discount factor 0.99 Actor learning rate 0.0003 Critic learning rate 0.0003 Initial exploration 0.4 Final exploration 0.05 Deca...

work page 2023
[12]

13 HIV MARL A.3 Reward Figures A.2 and A.4 show the learning curve for MARL and SARL approaches respectively for jurisdictions in California

Jurisdiction interaction HM HF MSM Same jurisdiction 57% 65% 47% Other jurisdiction in same state 28% 23% 31% Other states 14% 12% 22% 2020 2022 2024 2026 2028 2030 Year a) 3000 3250 3500 3750 4000 4250T otal new infectionsWithout mixing With mixing Alameda County Los Angeles CountyOrange County (CA) Riverside County Sacramento County San Bernardino Count...

work page 2020

[1] [1]

HIV .gov. U.S. Statistics.https://www.hiv.gov/hiv-basics/overview/data-and-trend/statistics , 2023a. (accessed: 06.10.2023). HIV .gov. Viral Suppression and an Undetectable Viral load. https://www.hiv.gov/hiv-basics/ staying-in-hiv-care/hiv-treatment/viral-suppression#:~:text=Viral%20suppression%20is% 20defined%20as,having%20\an%20undetectable%20viral%20l...

work page 2023

[2] [2]

(accessed: 06.10.2022). AHEAD. EHE Goal. https://www.ahead.hiv.gov/faqs, 2022a. (accessed: 08.25.2022). Bohdan Nosyk, Xiao Zang, Emanuel Krebs, Benjamin Enns, Jeong E Min, Czarina N Behrends, Carlos Del Rio, Julia C Dombrowski, Daniel J Feaster, Matthew Golden, et al. Ending the hiv epidemic in the usa: an economic modelling study in six cities. The Lance...

work page 2022

[3] [3]

Impact of improved hiv care and treatment on prep effectiveness in the united states, 2016–2020

Nidhi Khurana, Emine Yaylali, Paul G Farnham, Katherine A Hicks, Benjamin T Allaire, Evin Jacobson, and Stephanie L Sansom. Impact of improved hiv care and treatment on prep effectiveness in the united states, 2016–2020. JAIDS Journal of Acquired Immune Deficiency Syndromes, 78(4):399–405,

work page 2016

[4] [4]

Reinforcement learning for optimization of covid-19 mitigation policies

Varun Kompella, Roberto Capobianco, Stacy Jong, Jonathan Browne, Spencer Fox, Lauren Meyers, Peter Wur- man, and Peter Stone. Reinforcement learning for optimization of covid-19 mitigation policies. arXiv preprint arXiv:2010.10560,

work page arXiv 2010

[5] [5]

Playing Atari with Deep Reinforcement Learning

V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,

work page internal anchor Pith review Pith/arXiv arXiv

[6] [6]

Dueling network architectures for deep reinforcement learning

Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995–2003. PMLR,

work page 1995

[7] [7]

Proximal Policy Optimization Algorithms

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

work page internal anchor Pith review Pith/arXiv arXiv

[8] [8]

Accessed: 2023-09-17. HRSA. Ryan White HIV/AIDS Program FY 2023 EHE Awards. https://ryanwhite.hrsa.gov/about/ parts-and-initiatives/fy-2023-ending-hiv-epidemic-awards ,

work page 2023

[9] [9]

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz

Accessed: 2023-09-17. John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889–1897. PMLR,

work page 2023

[10] [10]

(accessed: 09.17.2023). CDC. HIV Surveillance Report V olume 27, Number

work page 2023

[11] [11]

(accessed: 09.17.2023). AHEAD. PrEP Coverage. https://ahead.hiv.gov/data/prep-coverage, 2022b. (accessed: 09.17.2023). 12 HIV MARL A Appendix A.1 Hyperparameters Table A1: List of hyperparameters used in PPO Hyperparameters Value Discount factor 0.99 Actor learning rate 0.0003 Critic learning rate 0.0003 Initial exploration 0.4 Final exploration 0.05 Deca...

work page 2023

[12] [12]

13 HIV MARL A.3 Reward Figures A.2 and A.4 show the learning curve for MARL and SARL approaches respectively for jurisdictions in California

Jurisdiction interaction HM HF MSM Same jurisdiction 57% 65% 47% Other jurisdiction in same state 28% 23% 31% Other states 14% 12% 22% 2020 2022 2024 2026 2028 2030 Year a) 3000 3250 3500 3750 4000 4250T otal new infectionsWithout mixing With mixing Alameda County Los Angeles CountyOrange County (CA) Riverside County Sacramento County San Bernardino Count...

work page 2020