ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory

Jinghui Chen; Lu Lin; Yongkang Du; Yuanpu Cao; Yurui Chang

REVIEW 2 major objections 2 minor 7 references

ForecastCompass stores reusable predictive factors and calibration rules in memory so agents improve forecast accuracy across tasks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-28 23:26 UTC pith:OMFP7YQD

load-bearing objection FoCo splits agent memory into factor and reasoning stores with retrospective verbalized updates for forecasting tasks, but the post-outcome revision step leaves the transfer claim open to hindsight bias. the 2 major comments →

arxiv 2605.30858 v1 pith:OMFP7YQD submitted 2026-05-29 cs.LG

ForecastCompass: Guiding Agentic Forecasting with Adaptive Factor Memory

Yurui Chang , Yongkang Du , Yuanpu Cao , Jinghui Chen , Lu Lin This is my paper

classification cs.LG

keywords agentic forecastingmemory augmentationprobabilistic forecastingadaptive memorycalibrationfactor memoryLLM agentsretrospective analysis

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ForecastCompass (FoCo), a memory framework that organizes past forecasting experience into a hierarchical taxonomy of tasks. It maintains factor memory for reusable predictive dimensions and reasoning memory for probability updating, uncertainty handling, and calibration. Using resolved forecasts as signals, a verbalized revision procedure updates the memory iteratively so that future predictions draw on accumulated knowledge rather than starting from scratch each time. Experiments with GPT-5-mini and Gemini-2.5-Flash on Prophet Arena and FutureX show gains in both probabilistic accuracy and calibration.

Core claim

FoCo organizes forecasting experience with a hierarchical forecasting-task taxonomy and maintains two complementary memory components: factor memory capturing reusable predictive dimensions and reasoning memory encoding probability updating, uncertainty handling, and calibration principles. Retrospective analyses serve as learning signals for a verbalized memory-revision procedure that iteratively refines these memories, enabling the agent to accumulate transferable forecasting knowledge over time.

What carries the argument

Adaptive factor-based memory consisting of factor memory for predictive dimensions and reasoning memory for calibration principles, organized by a hierarchical forecasting-task taxonomy and updated through verbalized revision from retrospective analyses.

Load-bearing premise

Retrospective analyses provide unbiased learning signals that let the verbalized memory-revision procedure accumulate transferable forecasting knowledge without systematic errors or overfitting.

What would settle it

A controlled test in which agents equipped with FoCo memory show no accuracy or calibration gains (or show losses) on held-out forecasting tasks after the memory has been revised from a set of resolved forecasts.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Agents retrieve task-relevant forecasting knowledge more effectively through the taxonomy.
Probabilistic accuracy and calibration both improve on the tested benchmarks.
Knowledge accumulated from resolved forecasts transfers to new prediction tasks without retraining the underlying model.
The dual memory structure separates reusable predictive factors from updating and calibration rules.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same revision loop could be applied to other sequential reasoning domains that produce verifiable outcomes over time.
Compressing experience into explicit factors may reduce reliance on long context windows for multi-turn forecasting.
If the taxonomy is too coarse, retrieval quality could degrade on tasks that fall between categories.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

FoCo splits agent memory into factor and reasoning stores with retrospective verbalized updates for forecasting tasks, but the post-outcome revision step leaves the transfer claim open to hindsight bias.

read the letter

The paper's main move is to give forecasting agents a memory system that keeps reusable predictive factors in one store and probability-updating or calibration rules in another, organized by a hierarchical task taxonomy and revised through verbalized retrospectives after outcomes are known.

It does a clean job identifying why generic agent memory (interaction logs or reflections) falls short for probabilistic forecasting and then spelling out two concrete components plus an iterative revision loop. That separation is a reasonable way to make experience more modular.

The soft spots sit mostly in the evaluation and the learning signal. The abstract reports gains on Prophet Arena and FutureX with two models but supplies no baselines, statistical details, or description of how the memory was built and tested across time. More importantly, the revision procedure runs after ground truth is available, so nothing in the described mechanism prevents the memory from absorbing realized frequencies or post-hoc rationales that would not exist at forecast time. If those updates simply encode task-specific patterns from the training distribution, the reported calibration improvements will not transfer to genuinely prospective use.

This is for people already working on LLM agents for sequential prediction or decision support. A reader focused on memory design would pick up the taxonomy and the factor/reasoning split as useful distinctions, but anyone expecting a robust demonstration of transferable forecasting knowledge will need the full experiments to show temporal separation and ablations that isolate the revision step.

I would send it to peer review. The framing targets a real gap in agent memory work and the proposed structure is concrete enough that referees could usefully press on the bias and transfer questions.

Referee Report

2 major / 2 minor

Summary. The paper proposes ForecastCompass (FoCo), an adaptive factor-based memory framework for agentic forecasting. It organizes experience via a hierarchical forecasting-task taxonomy and maintains factor memory (reusable predictive dimensions) and reasoning memory (probability updating, uncertainty, calibration). Memory is iteratively revised via a verbalized procedure driven by retrospective analyses to accumulate transferable knowledge. Experiments on Prophet Arena and FutureX with GPT-5-mini and Gemini-2.5-Flash report gains in probabilistic accuracy and calibration.

Significance. If the retrospective revision mechanism produces reusable, prospective forecasting knowledge without hindsight bias or task-specific overfitting, the approach could meaningfully advance memory-augmented agentic systems for calibrated forecasting.

major comments (2)

[Experiments] The central claim that retrospective analyses yield transferable knowledge rests on the assumption that post-outcome revision does not encode information unavailable at prediction time. No ablation is described that builds memory exclusively from earlier, non-overlapping tasks and then evaluates on held-out future tasks; without this, the reported accuracy and calibration lifts on Prophet Arena and FutureX cannot be attributed to the claimed mechanism rather than hindsight or distribution-specific artifacts.
[Abstract and Experiments] The abstract states that FoCo improves both probabilistic accuracy and calibration, yet supplies no information on baselines, statistical tests, data splits, or controls for confounds. This information is required to evaluate whether the observed gains are load-bearing for the framework's contribution.

minor comments (2)

Clarify how the hierarchical forecasting-task taxonomy is constructed and whether it is fixed or learned.
Specify the exact verbalized memory-revision procedure, including prompts and revision criteria.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which identify key areas where additional evidence is needed to support the claims of transferable knowledge and experimental transparency. We address each major comment below and commit to revisions that directly respond to the concerns raised.

read point-by-point responses

Referee: [Experiments] The central claim that retrospective analyses yield transferable knowledge rests on the assumption that post-outcome revision does not encode information unavailable at prediction time. No ablation is described that builds memory exclusively from earlier, non-overlapping tasks and then evaluates on held-out future tasks; without this, the reported accuracy and calibration lifts on Prophet Arena and FutureX cannot be attributed to the claimed mechanism rather than hindsight or distribution-specific artifacts.

Authors: We agree that the absence of a strict temporal ablation leaves open the possibility of hindsight effects. In the revised manuscript we will add a new ablation in which factor memory and reasoning memory are constructed exclusively from earlier, non-overlapping tasks and then evaluated on subsequent held-out tasks within both Prophet Arena and FutureX. This will isolate the prospective value of the retrospective revision procedure. revision: yes
Referee: [Abstract and Experiments] The abstract states that FoCo improves both probabilistic accuracy and calibration, yet supplies no information on baselines, statistical tests, data splits, or controls for confounds. This information is required to evaluate whether the observed gains are load-bearing for the framework's contribution.

Authors: We acknowledge that the current abstract is too concise to convey these experimental details. We will expand the abstract to include brief references to the baselines (vanilla LLM agents and non-adaptive memory variants), the statistical tests employed, the temporal data splits used, and the main controls for confounds. These elements will also be made more prominent in the Experiments section. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical framework with no self-referential derivations

full rationale

The provided abstract and description present FoCo as an empirical agent-memory framework whose central claims rest on experimental results (Prophet Arena, FutureX) rather than any mathematical derivation chain. No equations, fitted-parameter predictions, self-citations, uniqueness theorems, or ansatzes are quoted or referenced that would reduce outputs to inputs by construction. Retrospective revision is described as a learning signal but is not shown to be tautological with evaluation metrics; the claims remain externally falsifiable via the stated benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review supplies no concrete free parameters, axioms, or invented entities; full text would be required to populate the ledger.

pith-pipeline@v0.9.1-grok · 5729 in / 1007 out tokens · 26178 ms · 2026-06-28T23:26:13.716598+00:00 · methodology

0 comments

read the original abstract

Agentic forecasting is important for decision-making in dynamic environments, but it remains challenging because agents must reason from incomplete, time-limited evidence and produce calibrated probabilities before outcomes are resolved. Memory provides a natural mechanism for transferring experience from resolved forecasts to future prediction tasks. However, existing agent-memory methods are not tailored to forecasting, as they typically store past interactions, reflections, or factual associations without explicitly representing reusable predictive factors or calibration knowledge. We propose ForecastCompass (FoCo), an adaptive factor-based memory framework for agentic forecasting. FoCo organizes forecasting experience with a hierarchical forecasting-task taxonomy, enabling retrieval task-relevant forecasting knowledge. It maintains two complementary memory components: factor memory, which captures reusable predictive dimensions, and reasoning memory, which encodes probability updating, uncertainty handling, and calibration principles. Using retrospective analyses as learning signals, FoCo iteratively revises memory through a verbalized memory-revision procedure, enabling the agent to accumulate transferable forecasting knowledge over time. Experiments on Prophet Arena and FutureX with GPT-5-mini and Gemini-2.5-Flash show that FoCo improves both probabilistic accuracy and calibration.

Figures

Figures reproduced from arXiv: 2605.30858 by Jinghui Chen, Lu Lin, Yongkang Du, Yuanpu Cao, Yurui Chang.

**Figure 1.** Figure 1: Generic memory v.s. FOCO memory on an example forecasting task. However, existing memory mechanisms for LLM agents are largely designed for question-answering or problem-solving settings, where memory helps recover an answer, reuse reasoning traces, or avoid past mistakes [Lewis et al., 2020, Borgeaud et al., 2022, Chhikara et al., 2025]. Forecasting requires memory beyond that used for question answeri… view at source ↗

**Figure 2.** Figure 2: Framework overview. A: the hierarchical forecasting memory organizes subcategory-level factor memory and reasoning memory. B: a new question is classified into a subcategory, relevant memory is retrieved, and the agent produces a calibrated forecast. C: the memory is updated by contrasting original trajectories with retrospective trajectories and iteratively revised. transferability across time and models,… view at source ↗

**Figure 3.** Figure 3: Transferability analysis of FOCO on FutureX. We evaluate both across-time transferability and across-model transferability. Across-model Transferability For across-model transfer, we construct FOCO memory with GPT-5- mini and evaluate the same memory with Gemini-2.5-Flash. Figure 3c shows that FOCO provides consistent gains across different backbone models on Brier scores. Full results including the ECE sc… view at source ↗

**Figure 4.** Figure 4: Memory quality comparison on Prophet Arena using LLM-as-a-Judge . We use GPT-5.2 to assess whether memory generated by Gemini-2.5-Flash on Prophet-Arena captures reusable forecasting knowledge rather than event-specific details. Memories are scored on five dimensions: generalizability, novelty, completeness, accuracy, and actionability, using the prompt in Appendix E.3. As shown in [PITH_FULL_IMAGE:fig… view at source ↗

**Figure 5.** Figure 5: Case Study of how FOCO improves forecasts through factor selection and probability calibration. 5 Related Work Forecasting. Forecasting aims to make probabilistic predictions about future events under incomplete and evolving evidence [Hyndman and Athanasopoulos, 2018, Tetlock and Gardner, 2016]. Classical approaches typically rely on structured historical data, predefined variables, or domain-specific assu… view at source ↗

**Figure 6.** Figure 6: Across-model memory transfer results on FutureX using GPT-5-mini as the backbone. We [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

**Figure 7.** Figure 7: Growth of forecasting taxonomy during memory construction. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗

**Figure 8.** Figure 8: Brier–ECE trade-off for selecting the number of memory-revision epochs. Each point [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

7 extracted references

[1]

FILTERED bundle: inference under a date-based search cutoff
[2]

keep”, “revise

NO-FILTER bundle: inference without date restriction, used as an oracle reference. Filtered event bundle: {filtered_event_bundle} No-filter reference bundle: {nofilter_event_bundle} Ground truth: {ground_truth} Task: Compare the filtered inference with the no-filter reference and ground truth. Identify which parts of the existing memory were useful, missi...
[3]

generalizability – applies broadly across question types, not overfit to specific examples
[4]

novelty – contains non-obvious insights beyond common knowledge
[5]

completeness – covers the key factors needed for forecasting this question type
[6]

accuracy – factually correct and based on sound reasoning
[7]

look for X

actionability – directly guides forecasting decisions, e.g., “look for X” or “weight Y” Memory artifact: –- {memory_text} –- Reply ONLY with valid JSON, with no markdown fences: {{“generalizability”:<int>, “novelty”:<int>, “completeness”:<int>, “accuracy”:<int>, “actionability”:<int>, “reasoning”: “<one sentence>”}} Table 19: Prompt template for evaluatin...

[1] [1]

FILTERED bundle: inference under a date-based search cutoff

[2] [2]

keep”, “revise

NO-FILTER bundle: inference without date restriction, used as an oracle reference. Filtered event bundle: {filtered_event_bundle} No-filter reference bundle: {nofilter_event_bundle} Ground truth: {ground_truth} Task: Compare the filtered inference with the no-filter reference and ground truth. Identify which parts of the existing memory were useful, missi...

[3] [3]

generalizability – applies broadly across question types, not overfit to specific examples

[4] [4]

novelty – contains non-obvious insights beyond common knowledge

[5] [5]

completeness – covers the key factors needed for forecasting this question type

[6] [6]

accuracy – factually correct and based on sound reasoning

[7] [7]

look for X

actionability – directly guides forecasting decisions, e.g., “look for X” or “weight Y” Memory artifact: –- {memory_text} –- Reply ONLY with valid JSON, with no markdown fences: {{“generalizability”:<int>, “novelty”:<int>, “completeness”:<int>, “accuracy”:<int>, “actionability”:<int>, “reasoning”: “<one sentence>”}} Table 19: Prompt template for evaluatin...