REVIEW 2 major objections 6 minor 47 references
Dealing with uncertainty in agent-based models for short-term predictions
T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper claims that pairing one-time parameter calibration with real-time particle filtering keeps agent-based bus models accurate for short-term predictions, and demonstrates the combination on synthetic GPS data.
desk verdict A clean synthetic demonstration that CEM calibration plus particle-filter DA improves ABM bus-location forecasts, but the missing PF-only control keeps the 'combination' story incomplete. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a Markovian state-space reformulation of the ABM: the state vector $X_t = [O_t, S_t]$ carries all agent information, and the Particle Filter maintains a weighted particle set $\mathcal{P}_t$ of hypotheses; each particle is one full ABM run state. After a predict step, importance weighting based on the observation vector, and systematic resampling (SIR), the method adds Gaussian roughening noise to the parameter sub-vector $S_t$ so that the calibrated parameters can continue to evolve with the data. The Cross-Entropy Method supplies the initial calibrated parameter distribution that the filter then tracks. This two-stage design is what carries the claim.
What would settle it
Apply the identical calibration-plus-filter pipeline to real bus GPS data where the underlying dwell-time and demand processes differ from the model family, and compare RMSE against calibration-only baselines; if the Particle Filter no longer improves predictions or the errors diverge, the central claim fails under structural misspecification. A cheaper version is to rerun the paper's own scenarios with BusSim-truth replaced by a truth model using a different boarding process or a different dwell-time formula.
Extended reading notes
Core claim
The central claim is that an ABM can be made dynamically optimisable for short-term prediction by calibrating its parameters with the Cross-Entropy Method against historical data and then applying a Particle Filter that assimilates real-time observations into the evolving state vector. In the controlled 'identical twin' experiments, where a richer stochastic-dynamic model (BusSim-truth) generates synthetic historical and real-time GPS data, the two simpler companion ABMs (BusSim-deterministic and BusSim-stochastic) achieve much lower RMSE of predicted bus location when the two stages are combined than in either baseline. For example, at one demand level the RMSE falls from about 335 metres with no calibration to about 125 metres after calibration, and to about 49 metres after calibration plus filtering. The paper interprets this as evidence that the drift caused by stochastic and dynamic change can be corrected at run time, and that the Particle Filter is a suitable data-assimilation method for the non-linear, non-Gaussian structure of ABMs.
Load-bearing premise
The weakest load-bearing premise is that the identical-twin test is diagnostic of real performance: the synthetic 'real-time' data are generated by a truth model whose passenger-boarding, dwell-time, and dynamic-change mechanisms are from the same family as the simpler companion models, so the experiment tests correction of known stochasticity, not reaction to a structurally different bus system.
Editorial extensions
If this is right
- Real-time bus location and arrival-time prediction can be supported by an ABM that is continuously updated, rather than only by a static, once-calibrated simulator.
- The method produces a particle ensemble, giving not a single forecast but a distribution that quantifies state uncertainty.
- In the synthetic experiments, the combined calibration-plus-filter approach yields the lowest RMSE across a range of stochasticity and dynamicity settings.
- The framework is positioned for use in passenger information systems and Intelligent Transport Systems, where live forecasts of bus positions are operationally valuable.
- The paper's framing implies that any ABM expressible as a Markovian state-space model can inherit data-assimilation methods without requiring an analytical linearisation.
Reading between the lines
- Because the truth model and the companion models share the same family of boarding, dwell-time, and dynamic-change equations, the head-to-head comparison measures correction of known stochasticity, not reaction to structural misspecification; a real deployment would need to test against a differently structured bus system.
- The roughening step means parameter uncertainty is represented by particle diversity, so the method's benefit likely grows with system volatility, at least until the jitter-noise scale is miscalibrated.
- The same two-stage recipe could transfer to other streaming ABMs, such as crowd movement or epidemic spread, whenever a finite state vector and an observation vector can be defined.
- The concluding section mentions a 'Scenario 4 (only Particle Filtering)' comparison, but no such experiment appears in the numerical section or in the reported sensitivity table, so the marginal contribution of calibration relative to filtering alone is not empirically pinned down by this paper.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a framework for short-term prediction with agent-based models under uncertainty, combining offline parameter calibration (Cross-Entropy Method) with online data assimilation (Particle Filter). The authors build a stochastic, dynamic 'BusSim-truth' model to generate synthetic historical and real-time GPS bus data, then calibrate two simpler companion models (BusSim-deterministic and BusSim-stochastic). They evaluate three scenarios: no calibration, calibration only, and calibration plus particle filtering, and report RMSE sensitivity analyses over passenger demand and dynamic change rate. Table 3 shows the calibration-plus-filter scenario has lower RMSE than the other two scenarios. The paper concludes that the combined framework outperforms both the no-calibration benchmark and a particle-filter-only scenario, and discusses applications to passenger information systems.
Significance. If the central claim holds, the framework is a useful step toward using ABMs with streaming data, addressing a recognized limitation of the field. The paper's strengths include a controlled synthetic evaluation with a more complex truth model, a sensitivity analysis over key dynamic and stochastic parameters, explicit model documentation in appendices, and public release of source code and data. The main gap is that the reported experiments do not include a particle-filter-only control, so the specific claim that the combination of calibration and DA is superior to DA alone is not directly supported. Nevertheless, the study provides a clear demonstration that calibration plus particle filtering improves prediction relative to a calibrated static model, which is valuable for future work in dynamic ABM calibration.
major comments (2)
- [Section 6 and Section 4] The conclusion states that the framework outperforms 'the no calibration scenario (Scenario 1) and only Particle Filtering scenario (Scenario 4)', but Scenario 4 is never defined in Section 4 and no results for it appear in Table 3. The methodology and sensitivity analysis cover only Scenarios 1-3, so the paper contains no experiment in which a particle filter is applied without prior CEM calibration. As a result, the claimed advantage of the combination over a particle filter alone is not established; adding this control and reporting its RMSE is necessary to support the central claim.
- [Section 4.5 and Eq. (8)] The evaluation in Section 4.5 and the RMSE defined in Eq. (8) do not clearly separate state estimation (filtering) from short-term forecasting. If the particle filter at time t is updated with the observation at time t, then the reported RMSE largely measures the accuracy of the filtered state rather than the skill of predictions made ahead of time. The paper should either clarify that the reported errors are one-step-ahead or multiple-step-ahead forecast errors, or re-run the evaluation at explicit lead times, so that the 'short-term predictions' claim in the title and abstract is directly supported.
minor comments (6)
- [Abstract] The keyword line reads 'First keyword· Second keyword· More' and should be replaced with actual keywords.
- [Section 3.3] In the last paragraph of Section 3.3, 'should be the best guest of the system state' should read 'best guess'.
- [Appendix A, Eqs. (16)-(17)] Equation (17) is missing a closing parenthesis, and the text says that xi > 0 'represents an increase in passenger demand and traffic speed' while the equations and Figure 5 show that positive xi reduces traffic speed and increases arrival rate; please reconcile the wording.
- [Table 2] The 'FleetSize' row in Table 2 gives a parameter name but no value; please provide the number of bus agents used in the experiments.
- [Eq. (8)] The RMSE formula in Eq. (8) has no index for the bus or replication; clarify whether the error is averaged over buses and over the ten replications, or reported per bus.
- [Section 4.6, Table 3] Table 3 reports only mean RMSE over 10 replications; please include standard deviations or confidence intervals so the reader can judge the significance of the differences between scenarios.
Circularity Check
No significant circularity: the headline comparison is against separately generated synthetic real-time data, and no reported prediction reduces by construction to fitted parameters or a self-citation.
full rationale
The derivation chain is not circular in the sense defined by the review criteria. The companion models are calibrated with CEM to one synthetic 'historical' dataset (Section 3.2, Eq. 3), while the RMSE values in Table 3 (Section 4.6, Eq. 8) are computed against a separately generated synthetic 'real-time' dataset produced by BusSim-truth. The paper states that the companion models 'will not know the parameters of BusSim-truth', so the calibrated parameters are not definitionally the target of the prediction. The PF step assimilates observations into the state-space model, but no equation sets the predicted trajectory equal to the assimilated observations; the improvement is reported relative to explicit Scenario 1 (no calibration) and Scenario 2 (calibration only) baselines. The only relevant self-citation is [47] (Ward, Evans, Malleson), used as background on dynamic calibration of ABMs; it is not the load-bearing justification for the proposed combination, and no uniqueness theorem or ansatz is imported from it. Two weaknesses exist, but they are not circularity: the identical-twin design means the synthetic truth and the candidate models share the same modelling family, limiting external validity; and Section 6 claims outperformance over a 'Scenario 4' (PF only) that is never defined or reported in Section 4 or Table 3. These are evidence/completeness gaps that weaken the conclusion, not steps where a prediction reduces to its inputs by definition.
Assumptions & free parameters
free parameters (4)
- jittering noise standard deviation (sigma_jitter) =
not reported
- number of particles (NP) =
not reported
- CEM hyperparameters (rho, alpha, initial mu/sigma) =
not reported
- calibration objective weighting (mean vs std) =
equal weights
assumptions (5)
- domain assumption The bus route system can be represented as a Markovian state-space model with a fixed-size state vector.
- domain assumption Boarding passenger counts follow a homogeneous Poisson process.
- domain assumption Dwell time is a linear function of boarding and alighting counts with fixed coefficients.
- ad hoc to paper Dynamic changes in demand and traffic speed are captured by a single deterministic parameter xi with linear equations.
- standard math The particle filter with SIR resampling and Gaussian parameter jittering is a valid approximation for this non-linear, non-Gaussian ABM.
Cite this review
Pith. "Pith review of Dealing with uncertainty in agent-based models for short-term predictions." pith.science (2026). https://pith.science/paper/QDMF3YAS
@misc{pith2026190808288,
author = {Pith},
title = {Pith review of: Dealing with uncertainty in agent-based models for short-term predictions},
year = {2026},
howpublished = {\url{https://pith.science/paper/QDMF3YAS}},
note = {Machine review of arXiv:1908.08288}
}
read the original abstract
Agent-based models (ABM) are gaining traction as one of the most powerful modelling tools within the social sciences. They are particularly suited to simulating complex systems. Despite many methodological advances within ABM, one of the major drawbacks is their inability to incorporate real-time data to make accurate short-term predictions. This paper presents an approach that allows ABMs to be dynamically optimised. Through a combination of parameter calibration and data assimilation (DA), the accuracy of model-based predictions using ABM in real time is increased. We use the exemplar of a bus route system to explore these methods. The bus route ABMs developed in this research are examples of ABMs that can be dynamically optimised by a combination of parameter calibration and DA. The proposed model and framework can also be used in an passenger information system, or in an Intelligent Transport Systems to provide forecasts of bus locations and arrival times.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Matsim-t: Architecture and simulation times
Michael Balmer, Marcel Rieser, Konrad Meister, David Charypar, Nicolas Lefebvre, and Kai Nagel. Matsim-t: Architecture and simulation times. In Multi-agent systems for traffic and transportation engineering, pages 57–78. IGI Global, 2009
work page 2009
-
[2]
Modeling transit trip time using archived bus dispatch system data
Robert L Bertini and Ahmed M El-Geneidy. Modeling transit trip time using archived bus dispatch system data. Journal of transportation engineering, 130(1):56–67, 2004
work page 2004
-
[3]
Yu Bin, Yang Zhongzhen, and Yao Baozhen. Bus arrival time prediction using support vector machines.Journal of Intelligent Transportation Systems, 10(4):151–158, 2006
work page 2006
-
[4]
Agent based modeling: Methods and techniques for simulating human systems
E Bonabeau. Agent based modeling: Methods and techniques for simulating human systems. Proceedings of the National Academy of Sciences, 99(90003):7280–7287, 2002
work page 2002
-
[5]
Improved particle filter for nonlinear problems
James Carpenter, Peter Clifford, and Paul Fearnhead. Improved particle filter for nonlinear problems. IEE Proceedings-Radar, Sonar and Navigation, 146(1):2–7, 1999
work page 1999
-
[6]
Data assimilation in the geosciences: An overview of methods, issues, and perspectives
Alberto Carrassi, Marc Bocquet, Laurent Bertino, and Geir Evensen. Data assimilation in the geosciences: An overview of methods, issues, and perspectives. Wiley Interdisciplinary Reviews: Climate Change, 9(5):e535, September 2018
work page 2018
-
[7]
Mesoscopic modeling of bus public transportation
Oded Cats, Wilco Burghout, Tomer Toledo, and Haris Koutsopoulos. Mesoscopic modeling of bus public transportation. Transportation Research Record: Journal of the Transportation Research Board, (2188):9–18, 2010. 16 A PREPRINT - AUGUST 23, 2019
work page 2010
-
[8]
Dynamic bus arrival time prediction with artificial neural networks
Steven I-Jy Chien, Yuqing Ding, and Chienhung Wei. Dynamic bus arrival time prediction with artificial neural networks. Journal of Transportation Engineering, 128(5):429–438, 2002
work page 2002
Show all 47 references
-
[9]
Steady-states and kinetics of ordering in bus-route models: connection with the nagel-schreckenberg model
Debashish Chowdhury and Rashmi C Desai. Steady-states and kinetics of ordering in bus-route models: connection with the nagel-schreckenberg model. The European Physical Journal B-Condensed Matter and Complex Systems, 15(2):375–384, 2000
2000
-
[10]
Crooks and Sarah Wise
Andrew T. Crooks and Sarah Wise. GIS and agent-based models for humanitarian assistance. Computers, Environment and Urban Systems, 41:100–111, 2013
2013
-
[11]
A headway-based approach to eliminate bus bunching: Systematic analysis and comparisons
Carlos F Daganzo. A headway-based approach to eliminate bus bunching: Systematic analysis and comparisons. Transportation Research Part B: Methodological, 43(10):913–921, 2009
2009
-
[12]
Calibration/data assimilation approach for integrating grace data into the watergap global hydrology model (wghm) using an ensemble kalman filter: First results
Annette Eicker, Maike Schumacher, Jürgen Kusche, Petra Döll, and Hannes Müller Schmied. Calibration/data assimilation approach for integrating grace data into the watergap global hydrology model (wghm) using an ensemble kalman filter: First results. Surveys in Geophysics, 35(6)...
2014
-
[13]
Waiting time perceptions at transit stops and stations: Effects of basic amenities, gender, and security
Yingling Fan, Andrew Guthrie, and David Levinson. Waiting time perceptions at transit stops and stations: Effects of basic amenities, gender, and security. Transportation Research Part A: Policy and Practice, 88:251–264, 2016
2016
-
[14]
Richiardi, and Mike Tsionas
Jakob Grazzini, Matteo G. Richiardi, and Mike Tsionas. Bayesian estimation of agent-based models. Journal of Economic Dynamics and Control, 77:26–47, April 2017
2017
-
[15]
Real-time bus route state forecasting using particle filter and mesoscopic modeling
Etienne Hans, Nicolas Chiabaut, Ludovic Leclercq, and Robert L Bertini. Real-time bus route state forecasting using particle filter and mesoscopic modeling. Transportation Research Part C: Emerging Technologies, 61:121– 140, 2015
2015
-
[16]
Forecasting, structural time series models and the Kalman filter
Andrew C Harvey. Forecasting, structural time series models and the Kalman filter. Cambridge university press, 1990
1990
-
[17]
A. J. Heppenstall, A.J. Evans, and M H Birkin. Genetic algorithm optimisation of an agent-based model for simulating a retail market. Environment and Planning B: Planning and Design, 34:1051–1070, 2007. Cited by 0018
2007
-
[18]
Numerical analysis of a time-headway bus route model
Scott A Hill. Numerical analysis of a time-headway bus route model. Physica A: Statistical Mechanics and its Applications, 328(1):261–273, 2003
2003
-
[19]
Analysis of a continuous car-following model for a bus route: existence, stability and bifurcations of synchronous motions
HJC Huijberts. Analysis of a continuous car-following model for a bus route: existence, stability and bifurcations of synchronous motions. Physica A: Statistical Mechanics and its Applications, 308(1):489–517, 2002
2002
-
[20]
Realistic bus route model considering the capacity of the bus
Rui Jiang, Mao-Bin Hu, Bin Jia, and Qing-Song Wu. Realistic bus route model considering the capacity of the bus. The European Physical Journal B-Condensed Matter and Complex Systems, 34(3):367–372, 2003
2003
-
[21]
Atmospheric Modeling, Data Assimilation and Predictability
Eugenia Kalnay. Atmospheric Modeling, Data Assimilation and Predictability. Cambridge University Press, 2003
2003
-
[22]
Prediction intervals to account for uncertainties in travel time prediction
Abbas Khosravi, Ehsan Mazloumi, Saeid Nahavandi, Doug Creighton, and JWC Van Lint. Prediction intervals to account for uncertainties in travel time prediction. IEEE Transactions on Intelligent Transportation Systems, 12(2):537–547, 2011
2011
-
[23]
Stochastic collective model of public transport passenger arrival process
Le Minh Kieu and Chen Cai. Stochastic collective model of public transport passenger arrival process. IET Intelligent Transport Systems, 12(9), 2018
2018
-
[24]
Liu, and Wing Hung Wong
Augustine Kong, Jun S. Liu, and Wing Hung Wong. Sequential Imputations and Bayesian Missing Data Problems. Journal of the American Statistical Association, 89(425):278–288, March 1994
1994
-
[25]
Bus travel time prediction using a time-space discretization approach
B Anil Kumar, Lelitha Vanajakshi, and Shankar C Subramanian. Bus travel time prediction using a time-space discretization approach. Transportation Research Part C: Emerging Technologies, 79:308–332, 2017
2017
-
[26]
Lewis, S
John M. Lewis, S. Lakshmivarahan, and Sudarshan Dhall.Dynamic Data Assimilation: A Least Squares Approach. Cambridge University Press, Cambridge, 2006
2006
-
[27]
Simulation of nonhomogeneous poisson processes by thinning.Naval research logistics quarterly, 26(3):403–413, 1979
PA W Lewis and Gerald S Shedler. Simulation of nonhomogeneous poisson processes by thinning.Naval research logistics quarterly, 26(3):403–413, 1979
1979
-
[28]
David J. B. Lloyd, Naratip Santitissadeekorn, and Martin B. Short. Exploring data assimilation and forecasting issues for an urban crime model. European Journal of Applied Mathematics, 27(Special Issue 03):451–478, 2016
2016
-
[29]
A realistic cellular automata model of bus route system based on open boundary
Yong-Ji Luo, Bin Jia, Xin-Gang Li, Can Wang, and Zi-You Gao. A realistic cellular automata model of bus route system based on open boundary. Transportation Research Part C: Emerging Technologies, 25:202–213, 2012
2012
-
[30]
Optimising an Agent-Based Model to Explore the Behaviour of Simulated Burglars
Nick Malleson, Linda See, Andrew Evans, and Alison Heppenstall. Optimising an Agent-Based Model to Explore the Behaviour of Simulated Burglars. In Vahid Dabbaghian and Vijay Kumar Mago, editors, Theories and Simulations of Complex Social Systems, number 52 in Intelligent Syste...
2014
-
[31]
Understanding the kalman filter
Richard J Meinhold and Nozer D Singpurwalla. Understanding the kalman filter. The American Statistician, 37(2):123–127, 1983
1983
-
[32]
Kinetic clustering and jamming transitions in a car-following model for bus route
Takashi Nagatani. Kinetic clustering and jamming transitions in a car-following model for bus route. Physica A: Statistical Mechanics and its Applications, 287(1):302–312, 2000
2000
-
[33]
Bunching transition in a time-headway model of a bus route
Takashi Nagatani. Bunching transition in a time-headway model of a bus route. Physical Review E, 63(3):036115, 2001
2001
-
[34]
Calibration of second order traffic models using continuous cross entropy method
D Ngoduy and MJ Maher. Calibration of second order traffic models using continuous cross entropy method. Transportation Research Part C: Emerging Technologies, 24:102–121, 2012
2012
-
[35]
Jamming transition in a homogeneous one-dimensional system: The bus route model
OJ O’loan, MR Evans, and ME Cates. Jamming transition in a homogeneous one-dimensional system: The bus route model. Physical Review E, 58(2):1404, 1998
1998
-
[36]
Montemayor
Juan Jose Pantrigo, Angel Sanchez, Kostas Gianikellis, and Antonio S. Montemayor. Combining Particle Filter and Population-based Metaheuristics for Visual Articulated Motion Tracking. ELCVIA Electronic Letters on Computer Vision and Image Analysis, 5(3):68–83, November 2005
2005
-
[37]
Pennisi, R
M. Pennisi, R. Catanuto, F. Pappalardo, and S. Motta. Optimal vaccination schedules using simulated annealing. Bioinformatics, 24(15):1740–1742, August 2008
2008
-
[38]
The cross-entropy method for combinatorial and continuous optimization
Reuven Rubinstein. The cross-entropy method for combinatorial and continuous optimization. Methodology and computing in applied probability, 1(2):127–190, 1999
1999
-
[39]
Design and Implementation of An Agent-Based Simulation for Emergency Response and Crisis Management
Timothy Schoenharl and Greg Madey. Design and Implementation of An Agent-Based Simulation for Emergency Response and Crisis Management. Journal of Algorithms & Computational Technology, 5(4):601–622, 2011
2011
-
[40]
The Use of Adjoint Equations in Numerical Modelling of the Atmospheric Circulation
Oliver Talagrand. The Use of Adjoint Equations in Numerical Modelling of the Atmospheric Circulation. In An- dreas Griewank and George F. Corliss, editors, Automatic Differentiation of Algorithms: Theory, Implementation, and Application, pages 169–180. SIAM, Philadelphia, PA, 1991
1991
-
[41]
A bus-following model with an on-line bus station
Tieqiao Tang, Yanfeng Shi, Yunpeng Wang, and Guizhen Yu. A bus-following model with an on-line bus station. Nonlinear Dynamics, 70(1):209–215, 2012
2012
-
[42]
Mesoscopic simulation for transit operations
Tomer Toledo, Oded Cats, Wilco Burghout, and Haris N Koutsopoulos. Mesoscopic simulation for transit operations. Transportation Research Part C: Emerging Technologies, 18(6):896–908, 2010
2010
-
[43]
Transit capacity and quality of service manual
TRB. Transit capacity and quality of service manual. Transit Cooperative Highway Research Program (TCRP) Report 165, 2013
2013
-
[44]
Vadakkepat and L
P. Vadakkepat and L. Jing. Improved Particle Filter in Sensor Fusion for Tracking Randomly Moving Object. IEEE Transactions on Instrumentation and Measurement, 55(5):1823–1832, October 2006
2006
-
[45]
Data assimilation and its applications
Bin Wang, Xiaolei Zou, and Jiang Zhu. Data assimilation and its applications. Proceedings of the National Academy of Sciences, 97(21):11143–11144, 2000
2000
-
[46]
Data assimilation in agent based simulation of smart environments using particle filters
Minghao Wang and Xiaolin Hu. Data assimilation in agent based simulation of smart environments using particle filters. Simulation Modelling Practice and Theory, 56:36–54, 2015
2015
-
[47]
Ward, Andrew J
Jonathan A. Ward, Andrew J. Evans, and Nicolas S. Malleson. Dynamic calibration of agent-based models using data assimilation. Royal Society Open Science, 3(4), 2016. 18
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.