Pith. sign in

REVIEW 2 major objections 5 minor 183 references

SocietyBench turns real social event timelines into audited counterfactual worlds and shows that even the strongest LLM reaches only 75.0 out of 100 when forecasting them.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 04:12 UTC pith:6233DZSR

load-bearing objection SocietyBench is a genuinely new forecasting benchmark with real-event anonymization and solid internal controls, but its core anti-recall claim is not proven: the date-shift offset is recoverable by a model that recognizes the event, and the audit cannot be independently verified. the 2 major comments →

arxiv 2608.04009 v1 pith:6233DZSR submitted 2026-08-04 cs.CL

SocietyBench: Forecasting Counterfactual Social-World Evolution

Zhenran Wang , Zhonghan Bian , Jinsong Li , Zhangyang Qi This is my paper
classification cs.CL
keywords LLM forecastingcounterfactual anonymizationsocial event timelinesprobability calibrationtemporal accuracypublic opinionbenchmarkagent evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

SocietyBench is a benchmark that asks language models to forecast how real social events unfold, rather than merely complete a task. To stop models from relying on memorized arcs, every real event is anonymized—named entities are replaced and dates are shifted by a constant—so the model sees a structurally identical but unrecognizable counterfactual social world. Models answer probability questions about future events and give calendar dates, scored on two separate 100-point axes: probability calibration and temporal accuracy. On five events and 125 prediction points in Chinese and English editions, the strongest LLM scores only 75.0, halfway between the trivial 50 anchor and a perfect score; the two axes dissociate, agent frameworks fail to beat their base model, and the benchmark's purpose is to measure forward social reasoning rather than recall.

Core claim

The central claim is that a score on SocietyBench reflects forward reasoning over social-world evolution rather than recall of the specific real arc the timeline came from. The paper builds five counterfactual worlds from real events through three-phase anonymization, then scores models on calibration (weighted mean absolute error of predicted probabilities against binary outcomes) and temporal accuracy (day-error within 30-day buckets). Empirically, the strongest of six frontier LLMs reaches 75.0, the two axes come apart, agent scaffolds do not improve over their base model, and non-LLM heuristics trail every LLM.

What carries the argument

The load-bearing mechanism is the three-phase anonymization pipeline: entity substitution via longest-match replacement, a uniform per-event date offset, and an adversarial audit loop in which LLM auditors try to re-identify the original event until confidence drops to LOW, followed by semantic-consistency repairs. This turns a real timeline into a counterfactual world whose causal and temporal structure is identical but whose surface labels cannot be matched against pre-training memory. The second piece is dual-axis scoring: weighted MAE with time decay and window weighting for calibration, and bucket-banded day error for temporal accuracy.

Load-bearing premise

The whole benchmark rests on the assumption that replacing names and shifting dates removes every clue that could let a model recognize the original real event; if any model can still identify the true event from the anonymized context, high scores may come from memory rather than forecasting.

What would settle it

Run a de-anonymization attack: give an LLM not used in the auditor panel the anonymized timelines and ask it to name the original real event and its true date window; if it succeeds on any event at above-chance frequency, the counterfactual separation is incomplete. A second check would re-score with true dates restored but different entities, and see whether temporal accuracy jumps, indicating calendar shape itself is a leak channel.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If the anonymization holds, any future run of SocietyBench measures forecasting ability independent of pre-training contamination.
  • The two axes must be reported separately; a single blended score would hide models that are calibration-strong but time-weak or the reverse.
  • Multi-event evaluation is necessary; per-event gaps up to 21.4 points show single-event leaderboards are unstable.
  • Scaffolding an LLM with agent frameworks such as plan-and-solve or multi-agent debate does not automatically improve social forecasting; the base model largely determines performance.
  • The fully automated pipeline means new events can be added as they unfold without expert hand-annotation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The anonymization protocol could be adopted as a general decontamination check for any temporal reasoning benchmark, not just social events.
  • The observed +1.0 calibration shift when real names and dates are restored suggests binary outcome prediction is nearly independent of entity identity; a stronger version of this test could be applied to the temporal axis, where arc shape may still leak.
  • The dissociation between axes points toward reporting two-dimensional model profiles rather than a single leaderboard rank.
  • A testable extension: vary the date-offset magnitude and the event's distance from the model's training cutoff to map how temporal recall fades.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. SocietyBench is a benchmark that measures whether LLMs and agent frameworks can forecast the future evolution of real social events rather than recall memorized arcs. The pipeline collects news and social-media posts, distills them into date-indexed timelines, anonymizes named entities and shifts all dates by a per-event constant, and generates, for each cutoff, a bank of calibration questions (probabilistic outcome predictions) and temporal questions (predicted dates). Models are scored on two 100-point axes, probability calibration and temporal accuracy, with a trivial anchor at 50. The authors evaluate six frontier LLMs, three agent frameworks built on a shared base, and two model-free baselines across five events in Chinese and English editions. GPT-5.5 scores highest at 75.0 overall, the two axes dissociate for several models, agent orchestration does not improve over the base model, and non-LLM baselines trail all LLMs. The anonymized timelines, question banks, ground truth, and scoring code are released, though per-question model responses are not.

Significance. If the anonymization procedure fully removes content-addressable clues to the original events, SocietyBench fills a real gap: existing forecasting benchmarks either use live unresolved events, binary market questions, or simulated environments without objective ground truth. The paper's strengths include a fully automated pipeline, real-event ground truth independent of model outputs, pre-specified scoring rules with robustness ablations, a multi-event design that demonstrates per-event instability, and public release of core artifacts. The internal sanity checks—cutoff-position gradients, composition ablations, scoring-rule ablations, and the +1.0 calibration shift under real identities—are valuable and mostly support the benchmark's construct validity. However, the central claim that high scores reflect forward reasoning rather than pre-training recall rests entirely on the adequacy of the anonymization audit. That audit is performed by LLM auditors of the same kind as the evaluated models, and the paper does not provide an independent way to verify that residual leakage is absent. This is a load-bearing issue for the benchmark's core purpose.

major comments (2)
  1. [Section 3.2] The central claim—that SocietyBench scores reflect forward reasoning over social-world evolution rather than recall of a memorized real arc—depends on the three-phase anonymization removing all content-addressable clues. The only evidence for this is Algorithm 1's panel of three adversarial LLM auditors and the +1.0 calibration-shift check. This is not sufficient. The auditor models are the same kind of systems being evaluated; if the top-scoring model is better at de-anonymization than the auditor panel, residual leakage can pass silently. Moreover, the substitution step changes only named entities and dates; non-entity cues such as timeline shape, node count, issue text, opinion dynamics, and exact numeric facts are preserved and can identify an event without any entity names. Because the original timelines and replacement tables are not released, no independent audit is possible. I re
  2. [Section 3.2] The paper argues that 'recalling the true calendar cannot pay by construction' because predictions are scored against shifted ground truth. This argument overlooks the fact that the date shift is a single per-event constant δ that is visible in every context date. A model that recognizes the original event from any residual clue—even without named entities—can compute δ = anonymized_date − remembered_true_date and then add δ to remembered true future dates, thereby answering temporal questions by recall. The described check only re-scores predictions made on the true calendar; it does not model or test this recognition-plus-shift strategy. The +1.0 calibration-shift check is also on the calibration axis only and cannot detect temporal recall that already succeeded. This is a concrete hole in the paper's central defense, specifically for the temporal accuracy axis. The authors should eith
minor comments (5)
  1. [Section 3.3, Eq. (1)] The axis is called 'probability calibration,' but Eq. (1) is a weighted mean absolute error between predicted probabilities and binary outcomes. This is a measure of probabilistic accuracy, not calibration in the standard sense (e.g., reliability diagrams or expected calibration error). Consider renaming the axis to 'probabilistic accuracy' or adding a calibration-curve analysis to justify the name.
  2. [Section 4.3, Table 5] The text says 'the model ordering survives both' proper scoring rules, but Table 5 reports only the GPT-5.5 vs. Doubao gap, not the full six-model ordering. The claim should either be restricted to the two shown models or extended to the full leaderboard.
  3. [Section 4.2, Table 2] The statement that GPT-5.5 'wins outright on four of the five events' is not directly verifiable from the per-event calibration and temporal columns alone. Please provide per-event overall ranking or specify the metric used for 'win.'
  4. [Section 3.2, Algorithm 1] Algorithm 1's termination behavior is underspecified: if the auditor never returns LOW within K=5 rounds but also never returns HIGH, it is unclear whether the event is rejected or accepted. Add an explicit condition for the case when the loop exhausts K without a LOW verdict.
  5. [Section 4.1 / Data Release] The release includes anonymized timelines, question banks, ground truth, and scoring code, but not per-question model responses. This makes independent verification of the reported leaderboard impossible without re-querying expensive API models. Consider releasing the response logs or a verification script that recomputes Table 2 from them.

Circularity Check

0 steps flagged

No significant circularity: the benchmark's ground truth is external real-event data, scoring formulas are pre-specified and stress-tested, and the LLM-audited anonymization is a control procedure rather than an input to any derived equation.

full rationale

Walking the paper's derivation chain: the task is defined on real social events whose timelines are collected from Web news and social media (Section 3.2); the ground-truth set G_P is simply the timeline nodes from the cutoff onward (Section 3.1), so ground truth is external to model behavior, not fitted to it. The anonymization procedure (Algorithm 1) is a preprocessing filter that replaces entities and shifts dates; it does not define or appear inside the scoring equations. Scoring is given by fixed formulas (Eqs. 1-3), and the paper explicitly ablates alternative calibration rules (Brier, log-loss, different time decays, equal weights), showing the model ordering survives, which addresses the concern that the metric was chosen to flatter a result. The 'recalling the true calendar cannot pay' argument is a mathematical property of a uniform per-event date offset, not a fitted prediction; the +1.0 calibration-shift check is an empirical robustness check, not a model input. The only self-referential element is Algorithm 1's use of adversarial LLM auditors to certify that anonymization removed identifiable clues. That is a control procedure, not a load-bearing step that reduces to its inputs by construction; even if the auditor panel were imperfect, the released timelines, question banks, ground truth, and scoring code make the central claim externally checkable. The paper itself notes 'stronger anonymization' as future work, acknowledging the separation is a matter of degree, but this is a measurement-validity risk, not circularity in the derivation. No equation, fitted parameter, or self-citation chain forces the reported scores, so no circular step meets the evidence bar.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The benchmark introduces no new physical or mathematical entities. The hand-chosen design constants (alpha, window offset, delta_max, bucket size) are explicitly stress-tested in the paper, but they are not derived from first principles. The central dependency is the unverified reliability of the LLM-based anonymization auditor and the factuality of the automated timeline.

free parameters (5)
  • alpha in time-weighting = 0.04
    Chosen for the time decay factor w_time=1/(1+alpha*Delta_t); the paper's ablation shows scores shift by at most 1.2 points for alpha in {0,0.08,0.16}, so it is not load-bearing, but it is a hand-chosen design constant.
  • window-weight offset = 4 days
    In w_win=W/(W+4), the additive 4 controls how quickly short windows are down-weighted; no theoretical derivation is given.
  • date-shift range delta_max = 180 days
    Uniform shift in [-180,+180] is chosen to cross seasons and a reporting cycle; this is a design choice affecting temporal anonymization strength.
  • temporal bucket size = 30 days
    The temporal axis scores events by three 30-day buckets after cutoff; the bucket width defines the granularity of the temporal error metric.
  • adversarial audit rounds K = 5
    Maximum augment-and-rerun iterations in Algorithm 1; chosen as a stopping rule for anonymization.
axioms (5)
  • domain assumption Real news and social-media posts are faithful records of the events they describe.
    The pipeline builds ground truth from these sources without external cross-validation, so any systematic bias in the sources propagates into labels.
  • domain assumption The automated extraction and question-generation pipeline yields accurate labels, and human acceptance audits catch material errors.
    No independent fact-checking or inter-annotator agreement statistics are reported.
  • ad hoc to paper The adversarial LLM audit is a reliable detector of residual de-anonymization clues.
    Algorithm 1 releases an event only when three LLM auditors return LOW confidence; this assumes LLMs can recognize all forms of leakage from an anonymized text.
  • domain assumption LLM performance under temperature 0 with a fixed reasoning budget is representative of the model's forecasting ability.
    The evaluation protocol fixes determinism; stochastic sampling might change scores.
  • standard math Background math of scoring formulas is accepted as standard.
    Weighted mean absolute error and bucket-ratio error are standard metrics; no proof of optimality is claimed.

pith-pipeline@v1.3.0-daily-deepseek · 11362 in / 13750 out tokens · 119155 ms · 2026-08-05T04:12:30.882216+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SocietyBench: Forecasting Counterfactual Social-World Evolution." pith.science (2026). https://pith.science/paper/6233DZSR

@misc{pith2026260804009,
  author       = {Pith},
  title        = {Pith review of: SocietyBench: Forecasting Counterfactual Social-World Evolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6233DZSR}},
  note         = {Machine review of arXiv:2608.04009}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Large language models (LLMs), and the agents built on top of them, are now benchmarked heavily on whether they can finish a task -- fix a bug, drive a browser, operate a GUI. A complementary social ability, namely how well a model understands and forecasts the way real social events unfold, has barely been measured. We introduce SocietyBench, an end-to-end benchmark that takes a one-line event topic, collects Web news and social-media posts across five platforms, distills them into a date-indexed timeline that keeps factual events and a public-opinion layer separate, and then turns every cutoff date on that timeline into an audited bank of forecasting questions. Questions are scored on two orthogonal 100-point axes: probability calibration and temporal accuracy. Before any model sees a timeline, a three-phase procedure replaces every named entity and shifts every date by a per-event constant, turning a real arc into a counterfactual social world -- structurally identical to what happened, but stripped of the surface labels a model could match against pre-training memory. On five heterogeneous events and 125 prediction points in Chinese and English editions, the strongest of six frontier LLMs reaches only 75.0 out of 100, against a trivial anchor of 50. The two axes come apart: a model can be calibration-strong but time-weak, or the reverse. Three agent frameworks built on a shared base model fail to improve on that base, and two model-free heuristics trail every LLM. Per-event gaps reach 21.4 points on a single axis, which is our main argument for evaluating on several events rather than one. All anonymized timelines, question banks, ground truth, and scoring code are released.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

183 extracted references · 43 canonical work pages · 1 internal anchor

  1. [1]

    ECCV , year=

    Beyond the Nav-Graph: Vision and Language Navigation in Continuous Environments , author=. ECCV , year=

  2. [2]

    EMNLP , year=

    Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding , author=. EMNLP , year=

  3. [3]

    Habitat:

    Manolis Savva and Abhishek Kadian and Oleksandr Maksymets and Yili Zhao and Erik Wijmans and Bhavana Jain and Julian Straub and Jia Liu and Vladlen Koltun and Jitendra Malik and Devi Parikh and Dhruv Batra , booktitle=. Habitat:

  4. [4]

    Julian Straub and Thomas Whelan and Lingni Ma and Yufan Chen and Erik Wijmans and Simon Green and Jakob J. Engel and Raul Mur-Artal and Carl Ren and Shobhit Verma and Anton Clarkson and Mingfei Yan and Brian Budge and Yajie Yan and Xiaqing Pan and June Yon and Yuyang Zou and Kimberly Leon and Nigel Carter and Jesus Briales and Tyler Gillingham and Elias M...

  5. [5]

    Chang, Angel and Dai, Angela and Funkhouser, Thomas and Halber, Maciej and Niessner, Matthias and Savva, Manolis and Song, Shuran and Zeng, Andy and Zhang, Yinda , booktitle=

  6. [6]

    IROS , year=

    WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation , author=. IROS , year=

  7. [7]

    ICCV , year=

    Waypoint models for instruction-guided navigation in continuous environments , author=. ICCV , year=

  8. [8]

    CVPR , year=

    Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation , author=. CVPR , year=

  9. [9]

    ECCV , year=

    Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environments , author=. ECCV , year=

  10. [10]

    ICCV , year=

    Gridmm: Grid memory map for vision-and-language navigation , author=. ICCV , year=

  11. [11]

    ICCV , year=

    Scaling data generation in vision-and-language navigation , author=. ICCV , year=

  12. [12]

    TPAMI , year=

    ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments , author=. TPAMI , year=

  13. [13]

    CoRL , year=

    InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environment , author=. CoRL , year=

  14. [14]

    CVPR , year=

    Topological planning with transformers for vision-and-language navigation , author=. CVPR , year=

  15. [15]

    EMNLP , year=

    Language-aligned waypoint (law) supervision for vision-and-language navigation in continuous environments , author=. EMNLP , year=

  16. [16]

    CVPR , year=

    Cross-modal Map Learning for Vision and Language Navigation , author=. CVPR , year=

  17. [17]

    NeurIPS , year=

    Weakly-supervised multi-granularity map learning for vision-and-language navigation , author=. NeurIPS , year=

  18. [18]

    CoRL , year=

    Sim-to-Real Transfer via 3D Feature Fields for Vision-and-Language Navigation , author=. CoRL , year=

  19. [19]

    RSS , year=

    Navid: Video-based vlm plans the next step for vision-and-language navigation , author=. RSS , year=

  20. [20]

    NeurIPS , year=

    Action-Aware Zero-Shot Robot Navigation by Exploiting Vision-and-Language Ability of Foundation Models , author=. NeurIPS , year=

  21. [21]

    COLM , year=

    Tulu 3: Pushing Frontiers in Open Language Model Post-Training , author=. COLM , year=

  22. [22]

    arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv:2501.12948 , year=

    Pith/arXiv arXiv

  23. [23]

    5: Scaling reinforcement learning with llms , author=

    Kimi k1. 5: Scaling reinforcement learning with llms , author=. arXiv:2501.12599 , year=

    Pith/arXiv arXiv

  24. [24]

    NeurIPS , year=

    Training language models to follow instructions with human feedback , author=. NeurIPS , year=

  25. [25]

    Liu, Chris Yuhao and Zeng, Liang and Liu, Jiacai and Yan, Rui and He, Jujie and Wang, Chaojie and Yan, Shuicheng and Liu, Yang and Zhou, Yahui , journal=

  26. [26]

    arXiv:1707.06347 , year=

    Proximal Policy Optimization Algorithms , author=. arXiv:1707.06347 , year=

    Pith/arXiv arXiv

  27. [27]

    2024 , url=

    Hello GPT-4o , author=. 2024 , url=

    2024

  28. [28]

    arXiv:2408.03326 , year=

    Llava-onevision: Easy visual task transfer , author=. arXiv:2408.03326 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv:2409.12191 , year=

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution , author=. arXiv:2409.12191 , year=

    Pith/arXiv arXiv

  30. [30]

    TMLR , year=

    Llava-onevision: Easy visual task transfer , author=. TMLR , year=

  31. [31]

    arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv:2412.19437 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv:1909.08593 , year=

    Fine-tuning language models from human preferences , author=. arXiv:1909.08593 , year=

    Pith/arXiv arXiv 1909

  33. [33]

    NeurIPS , year=

    Learning to summarize with human feedback , author=. NeurIPS , year=

  34. [34]

    ICLR , year=

    Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization , author=. ICLR , year=

  35. [35]

    ICML , year=

    Grounding large language models in interactive environments with online reinforcement learning , author=. ICML , year=

  36. [36]

    ACL , year=

    Aligning large multimodal models with factually augmented rlhf , author=. ACL , year=

  37. [37]

    Charlie Victor Snell and Ilya Kostrikov and Yi Su and Sherry Yang and Sergey Levine , booktitle=. Offline

  38. [38]

    ICML , year=

    LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Models , author=. ICML , year=

  39. [39]

    ICML , year=

    ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RL , author=. ICML , year=

  40. [40]

    ICLR , year=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. ICLR , year=

  41. [41]

    arXiv:2412.16720 , year=

    Openai o1 system card , author=. arXiv:2412.16720 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv:2402.03300 , year=

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models , author=. arXiv:2402.03300 , year=

    Pith/arXiv arXiv

  43. [43]

    ICLR , year=

    Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learning , author=. ICLR , year=

  44. [44]

    NeurIPS , year=

    Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents , author=. NeurIPS , year=

  45. [45]

    NeurIPS , year=

    Embodiedgpt: Vision-language pre-training via embodied chain of thought , author=. NeurIPS , year=

  46. [46]

    ICLR , year=

    Large language models as generalizable policies for embodied tasks , author=. ICLR , year=

  47. [47]

    arXiv:2409.12122 , year=

    Qwen2.5-math technical report: Toward mathematical expert model via self-improvement , author=. arXiv:2409.12122 , year=

    Pith/arXiv arXiv

  48. [48]

    arXiv:2402.06332 , year=

    Internlm-math: Open math large language models toward verifiable reasoning , author=. arXiv:2402.06332 , year=

    Pith/arXiv arXiv

  49. [49]

    arXiv:2403.17297 , year=

    Internlm2 technical report , author=. arXiv:2403.17297 , year=

    Pith/arXiv arXiv

  50. [50]

    2024 , booktitle=

    ReFT: Reasoning with Reinforced Fine-Tuning , author=. 2024 , booktitle=

    2024

  51. [51]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv:2409.12186 , year=

    Pith/arXiv arXiv

  52. [52]

    arXiv:2411.16345 , year=

    Preference Optimization for Reasoning with Pseudo Feedback , author=. arXiv:2411.16345 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv:2412.00154 , year=

    o1-coder: an o1 replication for coding , author=. arXiv:2412.00154 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv:2410.05605 , year=

    Codedpo: Aligning code models with self generated and verified source code , author=. arXiv:2410.05605 , year=

    Pith/arXiv arXiv

  55. [55]

    arXiv:2309.14525 , year=

    Aligning large multimodal models with factually augmented rlhf , author=. arXiv:2309.14525 , year=

    Pith/arXiv arXiv

  56. [56]

    arXiv:2311.16839 , year=

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization , author=. arXiv:2311.16839 , year=

    Pith/arXiv arXiv

  57. [57]

    ICLR Workshop , year=

    Aligning modalities in vision large language models via preference fine-tuning , author=. ICLR Workshop , year=

  58. [58]

    ACL Findings , year=

    Aligning large multimodal models with factually augmented rlhf , author=. ACL Findings , year=

  59. [59]

    Yu, Tianyu and Yao, Yuan and Zhang, Haoye and He, Taiwen and Han, Yifeng and Cui, Ganqu and Hu, Jinyi and Liu, Zhiyuan and Zheng, Hai-Tao and Sun, Maosong and others , booktitle=

  60. [60]

    ICLR , year=

    Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models , author=. ICLR , year=

  61. [61]

    Yu, Tianyu and Zhang, Haoye and Yao, Yuan and Dang, Yunkai and Chen, Da and Lu, Xiaoman and Cui, Ganqu and He, Taiwen and Liu, Zhiyuan and Chua, Tat-Seng and others , booktitle=

  62. [62]

    arXiv:1807.06757 , year=

    On evaluation of embodied navigation agents , author=. arXiv:1807.06757 , year=

    Pith/arXiv arXiv

  63. [63]

    Information Fusion , year=

    Embodied navigation with multi-modal information: A survey from tasks to methodology , author=. Information Fusion , year=

  64. [64]

    TMLR , year=

    Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models , author=. TMLR , year=

  65. [65]

    CVPR , year=

    Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments , author=. CVPR , year=

  66. [66]

    NeurIPS , year=

    Object goal navigation using goal-oriented semantic exploration , author=. NeurIPS , year=

  67. [67]

    CVPR , year=

    Embodied question answering , author=. CVPR , year=

  68. [68]

    The International Journal of Robotics Research (IJRR) , year=

    Person-following by autonomous robots: A categorical overview , author=. The International Journal of Robotics Research (IJRR) , year=

  69. [69]

    ICLR , year=

    Habitat 3.0: A co-habitat for humans, avatars and robots , author=. ICLR , year=

  70. [70]

    AAAI , year=

    RSPT: reconstruct surroundings and predict trajectory for generalizable active object tracking , author=. AAAI , year=

  71. [71]

    ECCV , year=

    Empowering embodied visual tracking with visual foundation models and offline rl , author=. ECCV , year=

  72. [72]

    CVPR , year=

    Towards learning a generalist model for embodied navigation , author=. CVPR , year=

  73. [73]

    ICRA , year=

    Visual language maps for robot navigation , author=. ICRA , year=

  74. [74]

    ICRA , year=

    Discuss before moving: Visual language navigation via multi-expert discussions , author=. ICRA , year=

  75. [75]

    ECCV , year=

    Navgpt-2: Unleashing navigational reasoning capability for large vision-language models , author=. ECCV , year=

  76. [76]

    CoRL , year=

    Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action , author=. CoRL , year=

  77. [77]

    NeurIPS , year=

    Towards versatile embodied navigation , author=. NeurIPS , year=

  78. [78]

    IEEE Robotics and Automation Letters , year=

    Towards target-driven visual navigation in indoor scenes via generative imitation learning , author=. IEEE Robotics and Automation Letters , year=

  79. [79]

    ICRA , year=

    Benchmarking reinforcement learning techniques for autonomous navigation , author=. ICRA , year=

  80. [80]

    IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI) , year=

    A survey of embodied ai: From simulators to research tasks , author=. IEEE Transactions on Emerging Topics in Computational Intelligence (TETCI) , year=

Showing first 80 references.