Pith. sign in

REVIEW 4 major objections 5 minor 125 references

Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A live, contamination-free benchmark for AI future forecasting is feasible, and it shows search-and-reasoning agents in the lead while humans keep the edge on hard events.

desk verdict This submission cannot be reviewed as submitted: the title and abstract describe a micro-expression paper while the full text is an entirely different future-prediction benchmark, and that benchmark itself needs an audit of its answer-extraction pipeline before its leaderboard can be trusted. read the letter →

arxiv 2508.11988 v2 pith:6AJWRL66 submitted 2025-08-16 cs.CV

Nicolas Mastropasqua , Ignacio Bugueno-Cordova , Rodrigo Verschae , Daniel Acevedo , Pablo Negri , Maria E. Buemi This is my paper · ORCID
classification cs.CV
keywords futurepredictionLLMagentslivebenchmarkdatacontaminationdeepresearchforecastingevaluationwebsearchevent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FutureX is a live benchmark that evaluates LLM agents by asking them to predict real, time-stamped future events—such as stock prices, rankings, sports outcomes, and data releases—before the answers exist anywhere. The paper's claim is that this forward-looking design is the only methodologically sound way to measure forecasting ability, because any retrospective test leaks the outcome into the model's training data and into search results. The benchmark runs on a semi-automated daily pipeline: roughly 500 events per week are curated from 195 websites, 25 models answer on each event's start date, and after the resolution date an automated crawler retrieves the ground truth and scores every stored prediction. The paper reports that search-and-reasoning models dominate the leaderboard, with Grok-4 first, while a base model wins the easy retrieval tiers, and that even the best agents trail human experts on the hardest open-ended events. The work matters because it proposes a standard for gauging whether autonomous agents can do the analytical forecasting that human professionals do in finance, politics, and economics—and it documents how far they currently fall short.

What carries the argument

The load-bearing mechanism is the fully automated, closed-loop daily pipeline with four stages. Event database construction starts from 2,008 candidate websites collected by the AIME agent, filters them through LLM checks and manual review to 195 sources. Future event daily curation turns these sources into roughly 500 weekly prediction questions via prediction-market crawling and randomized question templates, then filters out harmful, subjective, and binary events. Agent daily prediction runs 25 models on 70-100 events per day with a 30-minute cap per question. Answer daily acquisition crawls each resolved event's source up to four times daily and uses the Seed1.5-Thinking model to extract the answer, achieving a claimed acquisition success rate above 97%. The validity argument is carried by the four-tier difficulty stratification (Basic, Wide Search, Deep Search, Super Agent) with tier-specific metrics—0-1 accuracy, F1 for multi-choice, set-overlap partial credit for rankings, and volatility-adjusted tolerance for numerical forecasts—and by the one-week evaluation delay that makes it impossible to optimize against recent feedback.

What would settle it

Take a random sample of resolved events from one week and hand-check the pipeline's extracted answer against the archived content of the source page on the resolution date, then recompute each model's score using only the corrected answers; if extraction accuracy is materially below the claimed 97%, or if correcting errors changes the leaderboard, the benchmark's validity claim fails. A second decisive check is to run a trivial baseline that predicts the current observed value for every open-ended numerical event and compare it with the agent scores on the volatility-based metric.

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that future prediction can be turned into a rigorous, contamination-proof agent evaluation: because ground truth does not exist at prediction time, the benchmark is "contamination-impossible by design." On the first two weeks of operation (1,272 events, July 20 to August 3), model performance declines monotonically across the four difficulty tiers, search and reasoning become the decisive capabilities at the harder tiers, and the top model (Grok-4) beats both open- and closed-source deep research agents while remaining below human expert scores on three of the four tiers. The paper additionally shows that most deep research agents can be steered toward a false outcome by a fabricated webpage, that agents retrieve resolved "past" outcomes far better than they predict unresolved ones, and that no tested model beats professional sell-side analysts on more than about a third of financial forecasting tasks.

Load-bearing premise

The load-bearing premise is that the automated pipeline—LLM-based event curation and answer extraction, with a claimed success rate above 97%—produces accurate, unbiased ground truth, and that the 195 curated websites give a balanced and representative sample of forecastable events; if extraction is systematically wrong or the sources are skewed, the scores do not measure forecasting ability.

Editorial extensions

If this is right

  • A contamination-free, live evaluation of forecasting agents is feasible at scale and on a daily schedule, not just for static knowledge benchmarks.
  • Search and reasoning, not model size or internal knowledge, are the capabilities that separate agents on future prediction; the largest gains appear exactly where the benchmark becomes open-ended.
  • The four-tier scoring with 10/20/30/40 weights gives a stable leaderboard, and the factor analysis (R-squared = 0.418) supports treating difficulty and domain as first-order drivers of performance.
  • No agent in the study is yet a reliable financial forecaster: even the best models beat professional sell-side analysts on at most 37.5% of revenue and 32.3% of EPS predictions.
  • Deep research agents are vulnerable to adversarial web content; three of four tested closed-source agents were consistently deceived by fabricated pages, so robustness to misinformation must be part of any deployment claim.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the benchmark is live by design, its contamination protection applies at prediction time but erodes as resolved answers accumulate on the public web and enter future training corpora; the daily randomization and fresh event supply are what actually preserve it over time.
  • A persistence baseline that simply repeats the current observed value for numerical events would test whether the volatility-based scoring rewards genuine forecasting or recency tracking; the paper does not report such a baseline.
  • The human-expert comparison used a different question set from the model runs; re-running the same 300 questions through both would give a cleaner estimate of the human-model gap.
  • The fake-website attack protocol, with its LLM-generated pages and iterative feedback loop, could be packaged as a reusable adversarial robustness test for research agents generally.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript, as provided in the body, presents FutureX, a live benchmark for evaluating LLM agents on future prediction. The pipeline collects future-event questions from 195 curated websites, runs 25 models (base LLMs, think-and-search LLMs, open- and closed-source deep research agents) at each event's start date, then automatically scrapes the resolved outcome and scores the stored predictions. The authors report overall scores and analyses across four difficulty tiers, eleven domains, and a factor regression, plus three out-of-benchmark case studies on financial forecasting, fake-website vulnerability, and real-time information retrieval. The stated ambitions are that FutureX is the largest and most diverse live future-prediction benchmark, that it is contamination-free by design, and that the automated answer-acquisition pipeline supports reliable evaluation with a reported success rate above 97%. An additional human-expert comparison is presented as a rough benchmark of human performance. As submitted, there is a major front-matter discrepancy: the abstract and title describe a different paper on event-based facial micro-expression analysis, while the full text describes FutureX.

Significance. If the benchmark's validity holds, FutureX would be a valuable community resource. A live, prospective benchmark for agent future prediction addresses the known pitfalls of backtesting and retrieval contamination, and the paper's scale (25 models, daily updates, 1,272 events over two weeks) is substantially larger than prior work such as FutureBench. The open-ended and high-volatility event types, the case studies on adversarial websites, and the planning/search analyses are useful and go beyond simple leaderboard reporting. However, the significance is contingent on two key assumptions: that the automatically extracted ground-truth answers are correct and unbiased, and that the hand-chosen scoring parameters do not drive the reported rankings. The paper provides no direct evidence for the first assumption, and only weak internal evidence for the second. The front-matter mismatch also prevents the manuscript from being assessed as a coherent submission in its present form.

major comments (4)
  1. [§3.2.4] The reported >97% answer acquisition success rate is not an answer-accuracy rate; it counts events for which a scraped answer was obtained, not events for which the extracted answer is correct. The pipeline uses Seed1.5-Thinking to extract the precise answer, and extraction errors are handled by manual review and prompt tweaking, but no error rate, confusion matrix, or independent audit is reported. Because every downstream result (overall leaderboard, difficulty-tier comparisons, domain analyses, factor regression, human comparison) is computed against these labels, systematic extraction errors correlated with domain or answer format would directly corrupt the reported rankings. The authors should provide a manually validated sample of extracted answers with accuracy broken down by event type and website category, and should demonstrate that label noise is small and uncorrelated with model identity or search behavior.
  2. [Title and Abstract vs. Full Text] The manuscript's front matter identifies it as 'Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis' (arXiv:2508.11988, cs.CV), but the full text is 'FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction' (cs.AI). This is not a typographical issue but an inconsistency in the paper's core identity: the abstract, title, and application domain do not match the content. The authors must correct the title, abstract, and metadata to match the body, or submit the intended manuscript. As submitted, the document cannot be assessed as a coherent paper.
  3. [§3.4.3 and §4.1] The evaluation protocol introduces several hand-chosen parameters with no sensitivity analysis: the tier weights (10%/20%/30%/40%), the 80% partial credit for ranking overlap, the 1-standard-deviation tolerance for numerical predictions, and the 7-day window for computing that standard deviation. The overall leaderboard and the factor-analysis conclusions depend on these choices. The authors should show that the ranking of models and the main qualitative findings (e.g., Grok-4 first, search-augmented models better on harder tiers) are stable under reasonable perturbations of these parameters, such as equal tier weights, partial credit between 70% and 90%, tolerance between 0.5 and 1.5 standard deviations, and windows between 3 and 14 days. Without such robustness checks, the quantitative comparisons may partly reflect metric choices.
  4. [§4.2, Finding 1] The claim that 'our difficulty tiers accurately reflect the complexity of the events' is supported by observing that model performance declines across the four tiers. This is a circular consistency check: the tiers were defined by event type and volatility, and the same model scores are used both to validate the tiers and to report the benchmark's main results. Independent validation is needed, for example human-rated difficulty, calibration against prediction-market prices, or item-response-theory analysis of the event bank. Without external evidence, the tier ordering and the tier-weighted overall score should be treated as a convention rather than a validated difficulty scale.
minor comments (5)
  1. [Global rendering] The submitted text contains multiple garbled or missing tokens, such as 'the������� ��� ���� �������' in Section 1 and '�' placeholders throughout Section 3. These appear to be rendering or conversion artifacts from the PDF-to-text process. The authors should provide a clean, complete version of the manuscript so that all content is legible.
  2. [§2 vs. Table 1] The Related Work text states FutureBench includes a 'very small number of events (≥30)' while Table 1 reports 42 events. The text should be made consistent with the table.
  3. [§4.2, Finding 5] The authors already note that the human comparison uses a different question set from the model evaluation and label the comparison as 'rough indicators.' Given this limitation, the subsequent statement that 'humans significantly outperform LLM agents on Level 1, Level 3, and Level 4' should be rephrased as a preliminary observation, or the evaluation should be repeated on a matched question set.
  4. [§3.2.3] The footnote explaining that GPT-o1, GPT-o3, and GPT-o4-mini were excluded because they frequently refused to make predictions is useful, but the authors should also state whether any of the other 25 models had systematic refusal or missing-prediction patterns that could bias the missingness analysis in §3.4.2.
  5. [§3.2.2] In the event-filtering description, the text says between July 15 and July 22, 294 unsuitable events were dropped as '(both harmful and subjective events).' The phrase 'both' is ambiguous because the preceding sentence lists three categories; the authors should clarify whether the 294 figure covers harmful and subjective only, or also some trivial events.

Circularity Check

1 steps flagged · score 3.0 of 10

Central leaderboard is not circular, but the paper's difficulty-tier validation is a self-referential consistency check using the benchmark's own scores.

  1. other [Section 4.2, Finding 1 (difficulty-tier validation; Table 3 and Figures 8/17)]
    "Our difficulty tiers accurately reflect the complexity of the events. We observe a clear, consistent decline in performance across the 4 defined difficulty tiers, which strongly supports the validity of our task stratification. ... This trend indicates that our difficulty labels effectively correspond to increasing levels of task complexity."

    The only evidence offered that the tiers 'accurately reflect complexity' is the decline in model scores computed on those same tiers. Because the tier labels were constructed by the authors from event type and volatility, and the scores are produced by the same benchmark whose validity is being asserted, this is a self-consistency check rather than an independent validation. The paper provides no external complexity measure, such as human-rated difficulty annotations or a held-out set, against which the tier labels are verified. This circularity is peripheral: the leaderboard and model rankings rest on prospective predictions compared with scraped ground-truth outcomes, so the central benchmark result is not reduced to its own inputs.

full rationale

No significant circularity is present in the core evaluation chain. FutureX scores are obtained by recording agents' predictions before each event's resolution date and then comparing them with automatically scraped ground-truth outcomes; the reported rankings (Grok-4, Gemini-2.5-flash Deep Research, GPT-o4-mini Think&Search) are empirical outcomes of that prospective pipeline, not quantities fitted from the benchmark's inputs. The use of ByteDance's Seed1.5-Thinking for question curation and answer extraction is a self-dependence that could affect label quality, but it does not by construction force any model's score, and the ground truth for each event is an external real-world outcome rather than a model output. The paper's claim that the difficulty tiers 'accurately reflect complexity' is the one circular step: it is validated by the same benchmark scores that the tier labels are supposed to explain, with no independent measure of complexity. This self-validation is not load-bearing for the main leaderboard, which is why the score is only 3 rather than higher. Separately, the reported >97% answer acquisition success rate is a fetch-success rate, not a label-accuracy rate, and the paper provides no confusion matrix or error audit; this is a threat to evaluation validity but is not a circularity, since extraction errors would inject noise rather than reproduce the benchmark's assumptions. Overall, the central contribution is a live empirical benchmark with a non-circular evaluation protocol.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The FutureX text introduces no new physical or conceptual entities such as particles, forces, or conserved quantities. The benchmark itself is a dataset and evaluation protocol, not an invented entity in the sense of this ledger. The main burdens are the hand-set scoring parameters and the unverified assumptions about pipeline accuracy and benchmark validity.

free parameters (4)
  • Difficulty tier weights = 10%, 20%, 30%, 40%
    Used to compute the overall score in Section 4.1; chosen by the authors to emphasize harder tiers, not derived from data or independent principles.
  • Ranking partial credit factor = 80%
    Open-ended ranking metric gives 80% credit for set overlap in Section 3.4.3; a hand-set scoring rule that affects model rankings.
  • Numerical prediction tolerance = 1 standard deviation of past 7 days
    Open-ended numerical metric awards partial credit within one historical standard deviation; a hand-set threshold in Section 3.4.3.
  • Website selection and filtering = 195 websites selected from 2,008
    The choice of sources and the filtering rules in Section 3.2.1 and 3.2.2 determine which events are in the benchmark, shaping difficulty and domain distribution.
assumptions (4)
  • domain assumption Automated answer extraction is accurate enough that a >97% acquisition success rate implies correct ground truth for almost all events.
    Section 3.2.4 claims the success rate but gives no error analysis for extraction mistakes, which would directly bias model scores.
  • domain assumption Future prediction performance on the curated questions is a valid proxy for real-world agent competence, including planning, search, and reasoning.
    The paper's motivation in Section 1 rests on this link; it is not independently established beyond the benchmark itself.
  • domain assumption The 195 selected websites provide a balanced, unbiased sample of verifiable future events across domains.
    Section 3.2.1 describes curation from 2,008 sites, but selection bias is not quantified, and binary, harmful, and subjective events are filtered out in Section 3.2.2.
  • ad hoc to paper The scoring rules (tier weights, partial credit, std-based tolerance) are reasonable and do not distort model rankings.
    Section 3.4.3 introduces metrics with hand-set thresholds, and Section 4.1 sets tier weights 10/20/30/40 without external justification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis." pith.science (2026). https://pith.science/paper/6AJWRL66

@misc{pith2026250811988,
  author       = {Pith},
  title        = {Pith review of: Exploring Spatial-Temporal Dynamics in Event-based Facial Micro-Expression Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6AJWRL66}},
  note         = {Machine review of arXiv:2508.11988}
}
read the original abstract

Micro-expression analysis has applications in domains such as Human-Robot Interaction and Driver Monitoring Systems. Accurately capturing subtle and fast facial movements remains difficult when relying solely on RGB cameras, due to limitations in temporal resolution and sensitivity to motion blur. Event cameras offer an alternative, with microsecond-level precision, high dynamic range, and low latency. However, public datasets featuring event-based recordings of Action Units are still scarce. In this work, we introduce a novel, preliminary multi-resolution and multi-modal micro-expression dataset recorded with synchronized RGB and event cameras under variable lighting conditions. Two baseline tasks are evaluated to explore the spatial-temporal dynamics of micro-expressions: Action Unit classification using Spiking Neural Networks (51.23\% accuracy with events vs. 23.12\% with RGB), and frame reconstruction using Conditional Variational Autoencoders, achieving SSIM = 0.8513 and PSNR = 26.89 dB with high-resolution event input. These promising results show that event-based data can be used for micro-expression recognition and frame reconstruction.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

125 extracted references · 12 canonical work pages

  1. [1]

    Babyagi, 2024

    babyagi. Babyagi, 2024. URLhttps://github.com/yoheinakajima/babyagi

    2024

  2. [2]

    LangChain, October 2022

    Harrison Chase. LangChain, October 2022. URLhttps://github.com/langchain-ai/langchain

    2022

  3. [3]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. InForty-firstInternational Conference on Machine Learning, 2024

    2024

  4. [4]

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    arXiv 2025

  5. [5]

    Investigating data contamination in modern benchmarks for large language models

    Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. Investigating data contamination in modern benchmarks for large language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8698–8711, 2024

    2024

  6. [6]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. URLhttps://arxiv.org/abs/2306.06070

    arXiv 2023

  7. [7]

    Gemini deep research — your personal research assistant, 2025

    Google Gemini Team. Gemini deep research — your personal research assistant, 2025. URLhttps://gemini. google/overview/deep-research/. Accessed: 2025-07-28

    2025

  8. [8]

    Openep: Open-ended future event prediction

    Yong Guan, Hao Peng, Xiaozhi Wang, Lei Hou, and Juanzi Li. Openep: Open-ended future event prediction. arXiv preprint arXiv:2408.06578, 2024

    arXiv 2024

Show all 125 references
  1. [9]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  2. [10]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations, 2021

    2021

  3. [11]

    Deep research agents: A systematic examination and roadmap.arXiv preprint arXiv:2506.18096, 2025

    Yuxuan Huang, Yihang Chen, Haozheng Zhang, Kang Li, Meng Fang, Linyi Yang, Xiaoguang Li, Lifeng Shang, Songcen Xu, Jianye Hao, et al. Deep research agents: A systematic examination and roadmap.arXiv preprint arXiv:2506.18096, 2025

    2025 arXiv

  4. [12]

    Livecodebench: Holistic and contamination free evaluation of large language models for code

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974, 2024

    2024 arXiv

  5. [13]

    Swe-bench: Can language models resolve real-world github issues? InThe TwelfthInternational Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe TwelfthInternational Conference on Learning Representations, 2024

    2024

  6. [14]

    Forecastqa: A question answering challenge for event forecasting with temporal text data

    Woojeong Jin, Rahul Khanna, Suji Kim, Dong-Ho Lee, Fred Morstatter, Aram Galstyan, and Xiang Ren. Forecastqa: A question answering challenge for event forecasting with temporal text data. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics...

    2021

  7. [15]

    Forecastbench: A dynamic benchmark of ai forecasting capabilities

    Ezra Karger, Houtan Bastani, Chen Yueh-Han, Zachary Jacobs, Danny Halawi, Fred Zhang, and Philip Tetlock. Forecastbench: A dynamic benchmark of ai forecasting capabilities. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  8. [16]

    Gonzalez, and Ion Stoica

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline, 2024

    2024

  9. [17]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InThe TwelfthInternational Conference on Learning Representations, 2024

    2024

  10. [18]

    Gaia: a benchmark for general ai assistants

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. InThe TwelfthInternational Conference on Learning Representations, 2023. 33

    2023

  11. [19]

    Navigating tomorrow: Reliably assessing large language models performance on future event prediction.arXiv preprint arXiv:2501.05925, 2025

    Petraq Nako and Adam Jatowt. Navigating tomorrow: Reliably assessing large language models performance on future event prediction.arXiv preprint arXiv:2501.05925, 2025

    2025 arXiv

  12. [20]

    Introducing SWE-Bench verified, 2024

    OpenAI. Introducing SWE-Bench verified, 2024. URL https://openai.com/index/ introducing-swe-bench-verified/

    2024

  13. [21]

    Introducing deep research, 2025

    OpenAI. Introducing deep research, 2025. URL https://openai.com/index/introducing-deep-research. Accessed: 2025-07-28

    2025

  14. [22]

    Pitfalls in evaluating language model forecasters

    Daniel Paleka, Shashwat Goel, Jonas Geiping, and Florian Tramèr. Pitfalls in evaluating language model forecasters. arXiv preprint arXiv:2506.00723, 2025

    2025 arXiv

  15. [23]

    Training software engineering agents and verifiers with swe-gym

    Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym. InForty-second International Conference on Machine Learning, 2025

    2025

  16. [24]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior, 2023. URLhttps://arxiv.org/abs/2304.03442

    2023 arXiv

  17. [25]

    Open- source deepresearch – freeing our search agents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Merve Noyan, Thomas Wolf, and Clémentine Fourrier. Open- source deepresearch – freeing our search agents, 2025. URLhttps://huggingface.co/blog/open-deep-research. Accessed: 2025-07-28

    2025

  18. [26]

    ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    Aymeric Roucher, Albert Villanova del Moral, Thomas Wolf, Leandro von Werra, and Erik Kaunismäki. ‘smola- gents‘: a smol library to build great agentic systems.https://github.com/huggingface/smolagents, 2025

    2025

  19. [27]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025

    2025

  20. [28]

    Aime: Towards fully-autonomous multi-agent framework.arXiv preprint arXiv:2507.11988, 2025

    Yexuan Shi, Mingyu Wang, Yunxiang Cao, Hongjie Lai, Junjian Lan, Xin Han, Yu Wang, Jie Geng, Zhenan Li, Zihao Xia, et al. Aime: Towards fully-autonomous multi-agent framework.arXiv preprint arXiv:2507.11988, 2025

    2025 arXiv

  21. [29]

    Significant Gravitas. AutoGPT. URLhttps://github.com/Significant-Gravitas/AutoGPT

  22. [30]

    Futurebench: Evaluating agents’ future prediction capabilities, 2025

    Together.ai. Futurebench: Evaluating agents’ future prediction capabilities, 2025. URLhttps://www.together. ai/blog/futurebench. Accessed: 2025-07-27

    2025

  23. [31]

    Superglue: A stickier benchmark for general-purpose language understanding systems.Advances in neural information processing systems, 32, 2019

    Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems.Advances in neural information processing systems, 32, 2019

    2019

  24. [33]

    Browsecomp: A simple yet challenging benchmark for browsing agents, 2025

    Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents, 2025. URLhttps://arxiv.org/abs/2504.12516

    2025 arXiv

  25. [34]

    Livebench: A challenging, contamination-limited llm benchmark

    Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, et al. Livebench: A challenging, contamination-limited llm benchmark. In The Thirteenth International Conference on Learning Re...

    2025

  26. [35]

    Bench to the future: A pastcasting benchmark for forecasting agents.arXiv preprint arXiv:2506.21558, 2025

    Jack Wildman, Nikos I Bosse, Daniel Hnyk, Peter Mühlbacher, Finn Hambly, Jon Evans, Dan Schwarz, Lawrence Phillips, et al. Bench to the future: A pastcasting benchmark for forecasting agents.arXiv preprint arXiv:2506.21558, 2025

    2025 arXiv

  27. [36]

    Agentic reasoning: A streamlined framework for enhancing llm reasoning with agentic tools

    Junde Wu, Jiayuan Zhu, Yuyuan Liu, Min Xu, and Yueming Jin. Agentic reasoning: A streamlined framework for enhancing llm reasoning with agentic tools. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 28489...

    2025

  28. [37]

    Reasoning or memorization? unreliable results of reinforcement learning due to data contamination

    Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Yanwei Fu, Qin Liu, et al. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507.10532, 2025. 34

    2025

  29. [38]

    Evaluation-driven development of llm agents: A process model and reference architecture.arXiv preprint arXiv:2411.13768, 2024

    Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing, Dehai Zhao, and Hao Zhang. Evaluation-driven development of llm agents: A process model and reference architecture.arXiv preprint arXiv:2411.13768, 2024

    2024

  30. [39]

    Swe-smith: Scaling data for software engineering agents.arXiv preprint arXiv:2504.21798, 2025

    John Yang, Kilian Leret, Carlos E Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, and Diyi Yang. Swe-smith: Scaling data for software engineering agents.arXiv preprint arXiv:2504.21798, 2025

    2025 arXiv

  31. [40]

    React: Synergizing reasoning and acting in language models, 2023

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models, 2023. URLhttps://arxiv.org/abs/2210.03629

    2023 arXiv

  32. [41]

    InThe Thirteenth International Conference on Learning Representations, 2025

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik R Narasimhan.���-bench: A benchmark for�underline �T� ool-�underline �A� gent-�underline �U� ser interaction in real-world domains. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  33. [42]

    Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025

    Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli- Scheuer. Survey on evaluation of llm-based agents.arXiv preprint arXiv:2503.16416, 2025

    2025 arXiv

  34. [43]

    Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    Daoguang Zan, Zhirong Huang, Wei Liu, Hanwu Chen, Linhao Zhang, Shulin Xin, Lu Chen, Qi Liu, Xiaojian Zhong, Aoyan Li, et al. Multi-swe-bench: A multilingual benchmark for issue resolving.arXiv preprint arXiv:2504.02605, 2025

    2025 arXiv

  35. [44]

    Swe-bench goes live!arXiv preprint arXiv:2505.23419, 2025

    Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, et al. Swe-bench goes live!arXiv preprint arXiv:2505.23419, 2025

    2025 arXiv

  36. [45]

    Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving.arXiv preprint arXiv:2506.12508, 2025

    Wentao Zhang, Ce Cui, Yilei Zhao, Yang Liu, and Bo An. Agentorchestra: A hierarchical multi-agent framework for general-purpose task solving.arXiv preprint arXiv:2506.12508, 2025

    2025 arXiv

  37. [46]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160, 2025

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160, 2025

    2025 arXiv

  38. [47]

    Webarena: A realistic web environment for building autonomous agents

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. In The TwelfthInternational Conference on Learning Representations, 2024

    2024

  39. [48]

    Forecasting future world events with neural networks.Advances in Neural Information Processing Systems, 35:27293–27305, 2022

    Andy Zou, Tristan Xiao, Ryan Jia, Joe Kwon, Mantas Mazeika, Richard Li, Dawn Song, Jacob Steinhardt, Owain Evans, and Dan Hendrycks. Forecasting future world events with neural networks.Advances in Neural Information Processing Systems, 35:27293–27305, 2022. 35 Appendix A Exam...

    2022

  40. [49]

    Detailed accounts of the June 2025 Los Angeles riots consistently omit any mention of fatalities

    2025

  41. [50]

    Subsequent protests in July 2025 across the state were predominantly non-lethal in nature

    2025

  42. [51]

    The tragic deaths of three Los Angeles County Sheriff’s deputies in an explosion were explicitly identified as an isolated, non-riot-related incident, as they occurred during the handling of explosive devices as an occupational hazard

  43. [52]

    garbage in, garbage out,

    This event’s lack of fatalities stands in stark contrast to historical precedents like the 1965 Watts Riots (34 deaths) and the 1992 Los Angeles Riots (50-63 deaths), highlighting a significant difference in outcomes. Based on the exhaustive review of the provided data, the es...

    1965

  44. [53]

    By June 11, 12 deaths had been confirmed

  45. [54]

    On July 10, the Southern California farm raid resulted in 1 death

  46. [55]

    national security threat

    On July 18, the explosion at the Los Angeles County training center caused the deaths of three police officers, but whether this incident is directly related to the riots is yet to be confirmed. It is important to emphasize that the above statistics may have omissions, especia...

    2025

  47. [56]

    Progress of negotiations between the federal and state governments: If both sides can reach a compromise, it may ease tensions and reduce violent clashes

  48. [57]

    Such large-scale protests carry a high risk of violence

    Scale and intensity of protest activities: During the Independence Day period, protests of varying scales were held in many parts of the United States, from Chicago to Los Angeles, from Roanoke, Virginia, to Sioux Falls, South Dakota, and Eugene, Oregon. Such large-scale prote...

  49. [58]

    Changes in enforcement intensity: If law enforcement continues to use high-intensity suppression tactics, it may lead to more casualties. 39

  50. [59]

    Expert analysis points out that there are fundamental differences between these riots and the 1992 Los Angeles riots

    Economic factors: As the world’s fourth-largest economy, California’s economic interests may prompt all parties to seek a peaceful resolution. Expert analysis points out that there are fundamental differences between these riots and the 1992 Los Angeles riots. The 1992 riots s...

    1992

  51. [60]

    This prediction is based on the current relatively stable death toll trend and the likely scale of conflicts

    Most Likely Scenario: An increase of 0–2 deaths in the coming week, bringing the total to 13–15. This prediction is based on the current relatively stable death toll trend and the likely scale of conflicts

  52. [61]

    Optimistic Scenario: If the federal and state governments can reach a compromise and protests remain peaceful, there may be no new deaths in the coming week, keeping the total at around 13

  53. [62]

    However, given the current military deployment and law enforcement capabilities, the possibility of this happening is relatively low

    Pessimistic Scenario: If the conflict escalates further, especially if large-scale violent events similar to 1992 occur, the death toll could increase significantly. However, given the current military deployment and law enforcement capabilities, the possibility of this happen...

    1992

  54. [63]

    Changes in political factors: The relationship between the federal and state governments could change significantly in a short time, affecting the situation’s development

  55. [64]

    3.Impact of sudden events: Unexpected incidents, such as the July 18 explosion, could cause a sudden increase in the death toll

    Possibility of external intervention: Other political forces or international factors might intervene, altering the nature and intensity of the conflict. 3.Impact of sudden events: Unexpected incidents, such as the July 18 explosion, could cause a sudden increase in the death toll

  56. [65]

    Unpredictability of protest activities: The development of large-scale protests is often uncertain and can escalate or subside rapidly. Despite these uncertainties, based on existing data and analysis, we have reason to believe that the total number of fatalities from the Cali...

    2025

  57. [66]

    Facts Survey (a) Facts to look up (b) Facts to derive

  58. [67]

    The outcome be $4400 B

    Plan �������� What price will Ethereum hit July 21-27? A. The outcome be $4400 B. The outcome be $4300 C. The outcome be $4200 D. The outcome be $4100 E. The outcome be $4000 F. The outcome be $3700 G. The outcome be $3600 H. The outcome be $3500 I. The outcome be $3400 J. The...

  59. [68]

    Facts given in the task � The event to be predicted concerns the price of Ethereum (ETH) during the week of July 21-27, 2025, with reference time around 2025-07-28T04:00:00Z

    Facts survey 1.1. Facts given in the task � The event to be predicted concerns the price of Ethereum (ETH) during the week of July 21-27, 2025, with reference time around 2025-07-28T04:00:00Z. � Several options are specified, corresponding to price thresholds ETH might reach o...

    2025

  60. [69]

    Look up the current price of Ethereum (ETH) in USD, as well as its price trend over the past year

  61. [70]

    Gather historical ETH price data from prior years, especially from June/July to the following July, to understand typical annual volatility and trend direction

  62. [71]

    Search for major events, upgrades, crypto-specific and macroeconomic risks, and catalysts expected between now and July 2025 that might affect Ethereum’s price

    2025

  63. [72]

    Look up analyst price predictions, industry reports, and any well-regarded forecasts for ETH in mid-late 2025

    2025

  64. [73]

    Search for information from the derivatives markets (particularly options and futures pricing for ETH with expiry around July 2025) to infer market-implied price expectations and volatility

    2025

  65. [74]

    Synthesize all gathered information to construct a reasoned, data-driven price range for ETH during the week of July 21-27, 2025

    2025

  66. [75]

    Match the forecasted price range to the specified answer options (A–J) and identify all plausible thresholds ETH could hit during the target week

  67. [76]

    42 ��������������

    Format the prediction in the required boxed format, including all plausible letter options. 42 ��������������

  68. [77]

    Facts given in the task �Asset to Predict:Ethereum (ETH)

    Facts survey 1.1. Facts given in the task �Asset to Predict:Ethereum (ETH). �Prediction Timeframe:The week of July 21-27, 2025. �Reference Timestamp:The prediction is contextualized around 2025-07-28T04:00:00Z. �Prediction Options: �A. Ethereum reach $4400 �B. Ethereum reach $...

    2025

  69. [78]

    Use the search agent to find the current price of Ethereum and its all-time high to establish a baseline for the prediction

  70. [79]

    Use the search agent to gather a consensus of Ethereum price predictions for mid-2025 from multiple reputable financial analysts and crypto research firms

    2025

  71. [80]

    Use the search agent to research the Ethereum development roadmap, specifically looking for information on major upgrades or updates planned for late 2024 and the first half of 2025 that could act as catalysts

    2024

  72. [81]

    Use the search agent to find macroeconomic forecasts for mid-2025, focusing on projected interest rates and overall economic health, which will influence investment in risk assets like Ethereum

    2025

  73. [82]

    Use the search agent to investigate the current and expected regulatory environment for Ethereum in key markets, particularly concerning the status and impact of spot Ether ETFs

  74. [83]

    Analyze all the gathered information (price history, expert forecasts, technical catalysts, macroeconomics, and regulation) to synthesize a coherent outlook and derive a probable price range for Ethereum in July 2025

    2025

  75. [84]

    Compare the derived probable price range against the provided options (A-J) and select all options that are plausible based on the analysis

  76. [85]

    ���������

    Provide the final answer by listing the letters of all selected plausible options, separated by commas, in the specified\boxed{}format. ���������

  77. [86]

    Facts given in the task �The event to predict is the price of Ethereum (ETH) on July 21–27, around 2025-07-28T04:00:00Z

    Facts survey 1.1. Facts given in the task �The event to predict is the price of Ethereum (ETH) on July 21–27, around 2025-07-28T04:00:00Z. �There are multiple potential outcomes listed as options A through J. 1.2. Facts to look up � Historical and recent price data for Ethereu...

    2025

  78. [87]

    Use the ‘search-agent‘ team member to gather historical and recent Ethereum price data from reliable sources

  79. [88]

    Use the ‘search-agent‘ team member to research any upcoming events, news, or macroeconomic factors that could affect Ethereum’s price during the specified period

  80. [89]

    Use the ‘search-agent‘ team member to collect technical analysis and expert forecasts regarding Ethereum’s expected performance

  81. [90]

    Analyze the gathered data to determine whether Ethereum is more likely to rise or dip between July 21–27, 2025

    2025

  82. [91]

    Evaluate the likelihood of each option (A–J) based on the compiled data and logical reasoning

  83. [92]

    Identify all plausible options based on the derived probabilities

  84. [93]

    �����������

    Use the ‘final-answer‘ tool to present the final prediction in the required format. �����������

  85. [94]

    �The prediction options range from $4400 (highest) to $3300 (lowest)

    Facts survey ���� ����� ����� �� ��� ���� �The event to predict is the price of Ethereum (ETH) between July 21–27, 2025. �The prediction options range from $4400 (highest) to $3300 (lowest). �The prediction must be formatted as\boxed{A, B, ...}for plausible options. ���� �����...

    2025

  86. [95]

    Research Ethereum’s historical price trends: Analyze ETH’s price movements in similar timeframes (e.g., 15 months post-Bitcoin halving) to identify patterns

  87. [96]

    Grab current market data: Retrieve ETH’s current price, volatility, and institutional interest to assess short-term momentum

  88. [97]

    Identify upcoming Ethereum events: Look for confirmed upgrades or events in 2025 that could drive price action

    2025

  89. [98]

    Analyze macroeconomic factors: Check for broader market trends (e.g., Fed rate cuts, regulatory clarity) that could impact ETH

  90. [99]

    Compile expert predictions: Aggregate mid-2025 price forecasts from reputable analysts and institu- tions

    2025

  91. [100]

    Derive plausible price range: Combine historical trends, current data, and expert opinions to narrow down the most likely options

  92. [101]

    �� �� �� � �����������

    Finalize prediction: Select the most plausible options (e.g., if $4000–$4300 is the consensus range, choose B, C, D, E) and format the answer as\boxed{A, B, ...}. �� �� �� � �����������

  93. [102]

    Facts given in the task �The event to predict is the price of Ethereum around July 21-27, 2025

    Facts survey 1.1. Facts given in the task �The event to predict is the price of Ethereum around July 21-27, 2025. � The potential price points provided are: $4400, $4300, $4200, $4100, $4000, $3700, $3600, $3500, $3400, and $3300. 1.2. Facts to look up � Historical price data ...

    2025

  94. [103]

    Gather historical price data for Ethereum around July from previous years using CoinMarketCap or CoinGecko

  95. [104]

    Analyze the historical price trends to identify patterns or significant events that could provide insight into Ethereum’s price movements during July in past years

  96. [105]

    Search for cryptocurrency market predictions or analysis related to Ethereum for the year 2025 using news websites or financial analysis platforms

    2025

  97. [106]

    Review and compile the significant factors that could affect Ethereum’s price between now and July 2025, including potential regulatory developments or technological advancements

    2025

  98. [107]

    Synthesize the information collected, relating historical trends to potential future outcomes

  99. [108]

    Make a prediction based on the derived data, focusing on the most plausible price point options

  100. [109]

    ����������

    Format the final answer according to the specified format, ensuring to list all plausible price options identified in the box format. ����������

  101. [110]

    Facts given in the task � The task asks for a prediction about Ethereum’s price between July 21-27 around 2025-07-28T04:00:00Z �The possible options for prediction are: �A

    Facts survey 1.1. Facts given in the task � The task asks for a prediction about Ethereum’s price between July 21-27 around 2025-07-28T04:00:00Z �The possible options for prediction are: �A. $4400, B. $4300, C. $4200, D. $4100, E. $4000 �F. $3700, G. $3600, H. $3500, I. $3400,...

    2025

  102. [111]

    Conduct a comprehensive search on the current price of Ethereum, its recent performance, and current market sentiment

  103. [112]

    Investigate historical price patterns of Ethereum to identify any recurring trends or cycles that could help in predicting future prices

  104. [113]

    Search for expert forecasts and market analyses specifically addressing Ethereum’s price expectations for 2025

    2025

  105. [114]

    Research macroeconomic factors, regulatory developments, and technological advancements that could impact Ethereum’s price in 2025

    2025

  106. [115]

    Look for any specific information about events or developments in the Ethereum ecosystem expected around July 2025

    2025

  107. [116]

    Analyze all gathered information to determine the most likely price range for Ethereum during July 21-27, 2025

    2025

  108. [117]

    Map the derived price range to the specific price levels provided in the options (A-J)

  109. [118]

    Calculate probabilities for each option based on the analysis and select the most plausible options

  110. [119]

    C Prompts C.1 Prompt for proposing new question for future prediction Please help me find some websites of data/lists/rankings/statistical tables

    Provide the final answer using the exact box format with the identified options. C Prompts C.1 Prompt for proposing new question for future prediction Please help me find some websites of data/lists/rankings/statistical tables. Requirements:

  111. [120]

    The data/lists/rankings/statistical tables are real and released by authoritative and well-known institu- tions

  112. [121]

    The data/lists/rankings/statistical tables have official websites

  113. [122]

    This point is very important

    The data/lists/rankings/statistical tables areregularly updated, such as daily, weekly, or monthly. This point is very important

  114. [123]

    This point is very important

    Each data/list/ranking/statistical table is always updated on afixed and unchangingwebsite. This point is very important

  115. [124]

    fixed and unchanging

    The data/lists/rankings/statistical tables aredirectly visible, that is, you can directly see the complete content of the data/list/ranking/statistical table when you open the website, without any need for clicking, drop-down selection, page-turning, downloading, etc. This poi...

  116. [125]

    My Reasoning:{reasoning} (The reasoning must clearly explain how you reached the prediction from your observations)

  117. [126]

    Otherwise, repeat the above steps

    If a correct prediction can be made, stop. Otherwise, repeat the above steps. Annotation Bonus � If the screencast and solution process meet the requirements, you will receive thebasic reward, even if the prediction is incorrect. � If the final prediction is correct and the th...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.