REVIEW 3 major objections 3 minor 89 references
FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation
T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read FinReportBench measures institution-grade financial reports with a 35-item rubric, and a distilled skill raises mean report identity by 33.85 points while preserving deliverability.
desk verdict Serious benchmark with a real judge-validation gap: expert calibration is too thin to carry the item-level claims, but the contributions warrant review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the frozen 35-item rubric with a G0-G1-G2 hierarchy: four preflight deliverability checks, four report-identity items, and 27 institutional-completeness items. The rubric is constructed from expert partial orders over a small set of reports, contrastive multimodal evidence mining, and expert audits of decision boundaries, and it is applied by an automated multimodal judge that renders each HTML report and returns an item decision with a short evidence citation. The 244-task benchmark separates the public query, the reconstructed research trajectory, and the hidden source packet, so each model must continue from an organized evidence state rather than generate directly from a query. The improvement machinery is benchmark-guided skill distillation: recurrent item-level failures across models are turned into compact plan, write, and review constraints, selected on disjoint external cases, frozen, and then evaluated once on a locked FinReportBench subset against a paired no-skill condition.
What would settle it
Take a random sample of reports from the 244 tasks outside the 18 calibration reports, have senior financial experts independently score each of the 35 items and rank reports within queries, and compare their item decisions and partial orders with the automated judge; if agreement falls materially below the 98.6 to 100 percent constraint rate reported on the calibration set, or if paired no-skill versus skill reports do not receive correspondingly higher expert readiness scores, the benchmark's criterion validity and the measured skill gains are not supported.
Extended reading notes
Core claim
The paper's central claim is that 'institution-grade' financial reporting is a staged decision: a report must first be deliverable (visible, readable, continuous, and unbroken), then recognizable as institutional research rather than a webpage, dashboard, slide deck, or generic summary, and only then complete in its institutional components such as front-page framing, compliance, source and chart discipline, and generation-trace control. Fluency alone is not enough; in the expert pilot, all 75 model-generated reports received the lowest readiness score. FinReportBench's 35-item rubric, applied by an automated multimodal judge to rendered pages and extracted text, reproduces expert preference ordering at 98.6 to 100 percent across three independent judge families. The same item-level signals can be distilled into a compact set of planning, writing, and self-review constraints, and when that skill is frozen and tested on a locked benchmark subset, it improves mean G1 by 33.85 points and mean G2 by 13.83 points over paired no-skill runs while preserving G0 in every pair.
Load-bearing premise
The load-bearing premise is that the automated judge's 35-item decisions are trustworthy on all 244 tasks, while expert calibration was limited to 18 reports from 3 queries and 71 pairwise constraints; if the judge diverges from senior experts outside that calibration set, the leaderboard order, bottleneck analysis, and the reported skill improvements all inherit the divergence.
Editorial extensions
If this is right
- Because deliverability is nearly saturated across nine model families, further gains in institution-grade report generation will come from report identity and institutional completeness, not from basic formatting or fluency.
- The largest cross-model gaps are generation-trace control (31.1 points), information density (13.4 points), and data discipline (13.2 points), giving developers a concrete repair list rather than a single quality score.
- A compact skill frozen before the locked test transfers across model families, so benchmark diagnostics can be converted into reusable generation and self-review constraints without updating model weights or exposing test answers.
- Three independent judge families preserve near-ceiling expert partial-order agreement, so another group could run the same frozen rubric with a different judge model and expect similar rankings, with score-rank correlations between 0.949 and 0.981.
- Language is the largest source of rank variation among slices, with Spearman correlations of 0.650 for Chinese and 0.917 for English, indicating that language should be monitored as a diagnostic slice rather than treated as a separate benchmark.
Reading between the lines
- Beyond the paper: the G0-to-G1-to-G2 staged-decision structure is a general template for evaluating other accountable document genres, such as legal filings, clinical notes, or policy briefs, where a document must first be readable, then recognizable as the right genre, and finally complete in its institutional obligations.
- Beyond the paper: because the leaderboard and skill gains are measured with the same frozen judge family, a fresh expert item-level review on reports outside the 18-report calibration set would be the direct test of whether the reported rankings and the 33.85-point and 13.83-point gains reflect human-valued quality rather than judge-model preference.
- Beyond the paper: with G0 near saturation, a more informative next benchmark would concentrate on the high-discrimination G1 and G2 items, especially trace control, and add tasks that push systems beyond report framing into compliance and source discipline under harder evidence conditions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces FinReportBench, a benchmark for evaluating institution-grade financial report generation. The authors derive a 35-item hierarchical rubric (G0 deliverability, G1 report identity, G2 institutional completeness) from expert partial orders, contrastive evidence mining, and expert audits; curate 244 bilingual tasks with separated public queries, reconstructed research trajectories, and hidden source packets; evaluate nine model families using an LLM judge with a frozen rubric; and propose benchmark-guided skill distillation that improves mean G1 by 33.85 points and mean G2 by 13.83 points over paired no-skill runs while preserving G0. The paper is carefully presented in several respects: scores are reported as a provisional total, small leaderboard gaps are not overinterpreted, skill-evolution results include confidence intervals and paired no-skill baselines, and code and benchmark artifacts are promised.
Significance. If the central claims hold, FinReportBench is a useful contribution: it moves beyond broad coarse-grained dimensions to a fine-grained, observable, hierarchically organized rubric, and it demonstrates a concrete use of benchmark diagnostics for improving report generation. The expert-grounded rubric construction, the item-level failure analysis, the separation of public query from hidden source, and the paired no-skill skill-evaluation protocol are genuine strengths. The significance is contingent, however, on two load-bearing assumptions: that the automatic judge's item decisions match expert judgments across the full item set, and that the reported skill gains reflect improved institution-grade quality rather than optimization against the judge.
major comments (3)
- [§4, Eq. (1)] The scoring formula as printed is dimensionally inconsistent and cannot reproduce the reported totals. With S1 defined as the weighted G1 percentage, S = (100*g1 + (S1/100)*g2)/(d1+d2) equals (100*g1 + (g1/d1)*g2)/(d1+d2), which is not a convex combination of the G1 and G2 percentages. For a report with perfect G1 and G2, the formula gives (100*d1 + d2)/(d1+d2), which is well below 100 unless d2=0. Substituting the Table 1 values for MiniMax M2.7 (G1 ≈ 43.5, G2 ≈ 46.6) yields approximately 5.8, not the reported 22.6. The intended expression is presumably S = (100*g1 + S1*g2)/(d1+d2). Because all leaderboard scores and skill-evolution deltas depend on this equation, the formula must be corrected and all downstream numbers recomputed or confirmed.
- [§6, Human–Evaluator Alignment] The expert calibration rests on only 18 reports from 3 queries and 71 directed pairwise constraints. These constraints verify only that the judge's overall hierarchical score respects a three-tier ordering; they provide no item-level validation of the 4 G1 items or the 27 G2 items. A judge could satisfy all 71 constraints while systematically mis-scoring individual items, and the reported cross-family agreement of 92–97% on item decisions only demonstrates that LLM judges share biases, not that they match experts. Since the leaderboard, the bottleneck analysis (trace control, density, data discipline), and the skill-improvement numbers are all computed from item decisions, the paper's central measurement claim requires item-level expert agreement on a sample spanning all 35 items, or at least on the items driving the reported conclusions. The paper should also explicitly list judge-item generalization as an open risk.
- [§6, Skill-Evolution Evaluation] The skill-evolution loop uses the same automatic judge (GPT-5.6 Luna with the frozen rubric) both as the optimization signal during candidate selection and as the evaluation metric on the locked FinReportBench subset. The +33.85 G1 and +13.83 G2 gains may therefore reflect adaptation to the judge's item decisions rather than improvement on institution-grade quality, especially if the judge rewards surface cues such as bylines, page numbers, or report-like headers. The paper reports that the locked evaluation uses '100 paired task–model comparisons' but does not specify how that subset was selected, whether it covers all five models and both languages proportionally, or whether any expert reviewed the improved outputs. A human evaluation of a sample of K-star outputs, or at least item-level expert labels on the locked subset, is needed to support the claim that the skill improves institution-grade report generation.
minor comments (3)
- [§4, Itemized Multimodal Evaluation] The sentence 'Section reports the main validation results' is missing its section number; it should read 'Section 6.'
- [Figure 1 caption] The caption states that the selected suite covers 95.65% of the reference space 'at cosine similarity≥0.60' but does not name the embedding model used; please specify it for reproducibility.
- [References] The reference list contains duplicate entries for Zheng et al. 2023 and Zheng et al. 2024; please consolidate.
Circularity Check
Skill-improvement claim is evaluated by the same automatic judge that guided the skill, with only tier-level human validation.
-
fitted input called prediction
[Section 'Benchmark-Grounded Skill Evolution', 'Evolution Pipeline' and Section 'Experiments', 'Skill-Evolution Evaluation'.]
"FinReportBench converts their rendered reports into item-level decisions and observable evidence. An optimizer turns recurring failures into a candidate Kt+1, organized as planning, writing, and review rules. ... On the locked FinReportBench evaluation, K* improves mean G1 by 33.85 points (95% CI: 31.17–36.63) and mean G2 by 13.83 points (12.65–15.02), while preserving G0 for every pair."
The optimizer's training signal and the reported success metric are the same instrument: the FinReportBench judge applies the frozen 35-item rubric to produce item decisions, and those item decisions both drive the evolution of K* and define the G1/G2 gains in the locked evaluation. The only direct validation of this judge's decisions is a three-tier expert partial order over 18 reports from 3 queries (71 cross-tier constraints), which checks that the overall hierarchical score respects the expert tier ordering but does not validate the item-level G1/G2 decisions that the skill was optimized to improve.
full rationale
The benchmark's measurement framework is not purely self-referential: the 35-item rubric is derived from expert partial orders, multimodal evidence, and expert audits, and the final partial-order validation uses disjoint report samples. The skill-evolution pipeline also respects a lock: K* is selected on external cases and only then evaluated on a locked benchmark subset, so the improvement is not fitted to the test tasks themselves. However, the central 'improvement' claim is partially circular because the same automatic judge supplies both the optimization feedback and the success measurement. The expert alignment evidence is limited to tier-level ordering on 18 reports, which does not establish item-level fidelity for the G1/G2 scores that drive the headline +33.85 / +13.83 gains. No load-bearing self-citation chain appears; the cited prior work on contrastive mining and skill discovery is contextual rather than definitional. The validity gap in item-level judge fidelity is a correctness risk, and it is this gap that makes the improvement claim partly self-referential rather than fully independent.
Assumptions & free parameters
free parameters (4)
- cosine coverage threshold =
0.60
- rubric item weights and scoring formula coefficients =
not fully specified in main text
- locked evaluation subset size =
100 paired task-model comparisons
- judge model and reasoning effort =
GPT-5.6 Luna, medium
assumptions (5)
- domain assumption The three senior experts' relative preferences are a valid ground truth for institution-grade report quality.
- domain assumption The 18-report, 3-query partial-order validation generalizes to all 244 benchmark tasks.
- domain assumption Reverse-synthesized research trajectories are realistic proxies for real analyst workflows.
- domain assumption The hidden source packet remains hidden and task construction prevents leakage.
- domain assumption The LLM judge applies the 35-item rubric faithfully and without language/model-family bias.
Cite this review
Pith. "Pith review of FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation." pith.science (2026). https://pith.science/paper/P6LO4ERZ
@misc{pith2026260804374,
author = {Pith},
title = {Pith review of: FinReportBench: Measuring and Improving Institution-Grade Financial Report Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/P6LO4ERZ}},
note = {Machine review of arXiv:2608.04374}
}
read the original abstract
Large language models can produce fluent financial analysis, but fluency alone does not establish whether a report is suitable for institutional delivery. We introduce FinReportBench, an expert-grounded benchmark for measuring and improving institution-grade financial report generation. Expert review reveals recurring gaps in report identity, institutional components, source discipline, and visual delivery. We derive a 35-item rubric through expert partial orders, multimodal evidence, and audits of decision boundaries, covering deliverability, report identity, and institutional completeness. Starting from 10,000 balanced Chinese and English financial-research source records, we curate 244 bilingual tasks across three research objects and two input tiers. Each task separates the public query, reconstructed research trajectory, and hidden source packet. Three independent judge families reproduce the expert partial order at near-ceiling rates, showing that bounded, observable criteria support reliable evaluation. Across nine model families, basic deliverability is nearly saturated, while report identity and institutional completeness remain the primary bottlenecks. The largest cross-model gaps concern generation-trace control, information density, and data discipline rather than basic report framing. We then use benchmark-guided skill distillation to turn recurrent failures into reusable generation and self-review constraints. Across five model families, the evolved skill improves mean G1 by 33.85 points and mean G2 by 13.83 points over paired no-skill runs while preserving G0 for every pair. Code and benchmark artifacts are available at https://github.com/MisterBrookT/finreportbench.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Proceedings of EMNLP , year =
FinQA: A Dataset of Numerical Reasoning over Financial Data , author =. Proceedings of EMNLP , year =
-
[2]
Proceedings of EMNLP , year =
ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering , author =. Proceedings of EMNLP , year =
-
[3]
Proceedings of ACL , year =
TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance , author =. Proceedings of ACL , year =
-
[4]
Proceedings of ACL , year =
MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data , author =. Proceedings of ACL , year =
-
[5]
arXiv preprint arXiv:2311.11944 , year =
FinanceBench: A New Benchmark for Financial Question Answering , author =. arXiv preprint arXiv:2311.11944 , year =
-
[6]
Proceedings of ACL , year =
DocFinQA: A Long-Context Financial Reasoning Dataset , author =. Proceedings of ACL , year =
-
[7]
arXiv preprint arXiv:2505.24714 , year =
FinMME: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation , author =. arXiv preprint arXiv:2505.24714 , year =
-
[8]
arXiv preprint arXiv:2402.12659 , year =
FinBen: A Holistic Financial Benchmark for Large Language Models , author =. arXiv preprint arXiv:2402.12659 , year =
Show all 89 references
-
[9]
Advances in Neural Information Processing Systems , year =
PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Finance , author =. Advances in Neural Information Processing Systems , year =
-
[10]
arXiv preprint arXiv:2303.17564 , year =
BloombergGPT: A Large Language Model for Finance , author =. arXiv preprint arXiv:2303.17564 , year =
-
[11]
and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =
Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik , booktitle =. SWE-bench: Can Language Models Resolve Real-World. 2024 , url =
2024
-
[12]
arXiv preprint arXiv:2404.07972 , year =
OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments , author =. arXiv preprint arXiv:2404.07972 , year =
-
[13]
arXiv preprint arXiv:2112.09332 , year =
WebGPT: Browser-assisted Question-Answering with Human Feedback , author =. arXiv preprint arXiv:2112.09332 , year =
-
[14]
Proceedings of ICLR , year =
GAIA: a benchmark for General AI Assistants , author =. Proceedings of ICLR , year =
-
[15]
arXiv preprint arXiv:2504.12516 , year =
BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents , author =. arXiv preprint arXiv:2504.12516 , year =
-
[16]
Proceedings of ICLR , year =
WebArena: A Realistic Web Environment for Building Autonomous Agents , author =. Proceedings of ICLR , year =
-
[17]
Advances in Neural Information Processing Systems , year =
Mind2Web: Towards a Generalist Agent for the Web , author =. Advances in Neural Information Processing Systems , year =
-
[18]
arXiv preprint arXiv:2506.21506 , year =
Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge , author =. arXiv preprint arXiv:2506.21506 , year =
-
[19]
arXiv preprint arXiv:2404.07738 , year =
ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models , author =. arXiv preprint arXiv:2404.07738 , year =
-
[20]
arXiv preprint arXiv:2503.09516 , year =
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning , author =. arXiv preprint arXiv:2503.09516 , year =
-
[21]
arXiv preprint arXiv:2505.22648 , year =
WebDancer: Towards Autonomous Information Seeking Agency , author =. arXiv preprint arXiv:2505.22648 , year =
-
[22]
2025 , url =
Weng, Luoxuan and Tang, Yinghao and Feng, Yingchaojie and Chang, Zhuo and Chen, Ruiqin and Feng, Haozhe and Hou, Chen and Huang, Danqing and Li, Yang and Rao, Huaming and Wang, Haonan and Wei, Canshi and Yang, Xiaofeng and Zhang, Yuhui and Zheng, Yifeng and Huang, Xiuqi and Zh...
2025
-
[23]
2026 , url =
Tang, Yinghao and Xie, Yupeng and Feng, Yingchaojie and Lan, Tingfeng and Lao, Jiale and Cheng, Yue and Chen, Wei , journal =. 2026 , url =
2026
-
[24]
2026 , url =
Tang, Yinghao and Liu, Xueding and Zhang, Boyuan and Lan, Tingfeng and Xie, Yupeng and Lao, Jiale and Wang, Yiyao and Li, Haoxuan and Gao, Tingting and Pan, Bo and Weng, Luoxuan and Huang, Xiuqi and Zhu, Minfeng and Feng, Yingchaojie and Luo, Yuyu and Chen, Wei , booktitle =. ...
2026
-
[25]
IEEE Transactions on Visualization and Computer Graphics , year =
Exploring Multimodal Prompt for Visualization Authoring with Large Language Models , author =. IEEE Transactions on Visualization and Computer Graphics , year =. doi:10.1109/TVCG.2026.3701510 , url =
2026
-
[26]
2025 , url =
Pan, Bo and Fu, Yixiao and Wang, Ke and Lu, Junyu and Pan, Lunke and Qian, Ziyang and Chen, Yuhan and Wang, Guoliang and Zhou, Yitao and Zheng, Li and Tang, Yinghao and Wen, Zhen and Wu, Yuchen and Lu, Junhua and Zhu, Biao and Zhu, Minfeng and Zhang, Bo and Chen, Wei , journal...
2025
-
[27]
arXiv preprint arXiv:2606.09174 , year =
Demonstrating chart-plot: Closing the Last Mile of Academic Chart Generation , author =. arXiv preprint arXiv:2606.09174 , year =
-
[28]
Xie, Yupeng and Luo, Yuyu and Li, Guoliang and Tang, Nan , journal =
-
[29]
Xie, Yupeng and Zhang, Zhiyang and Wu, Yifan and Lu, Sirong and Zhang, Jiayi and Yu, Zhaoyang and Wang, Jinlin and Hong, Sirui and Liu, Bang and Wu, Chenglin and others , journal =
-
[30]
Xie, Yupeng and Ma, Chen and Wang, Zhenyang and Wang, Liangwei and Zhu, Jiayi and Zeng, Chuxuan and Shen, Zhouan and Li, Boyan and Luo, Yuyu , journal =
-
[31]
Li, Boyan and Peng, Yiran and Xie, Yupeng and Lu, Sirong and Zhu, Yizhang and Mu, Xing and Liu, Xinyu and Luo, Yuyu , booktitle =
-
[32]
2025 , organization =
Chen, Yiyu and Wu, Yifan and Shen, Shuyu and Xie, Yupeng and Shen, Leixian and Xiong, Hui and Luo, Yuyu , booktitle =. 2025 , organization =
2025
-
[33]
You Don't Know Until You Click: Automated
Bian, Yutong and Lin, Xianhao and Xie, Yupeng and Liu, Tianyang and Zhuge, Mingchen and Lu, Siyuan and Tang, Haoming and Wang, Jinlin and Zhang, Jiayi and Chen, Jiaqi and others , journal =. You Don't Know Until You Click: Automated
-
[34]
Proceedings of ACL , year =
PROXYQA: An Alternative Framework for Evaluating Long-Form Text Generation , author =. Proceedings of ACL , year =
-
[35]
arXiv preprint arXiv:2409.02076 , year =
LongGenBench: Benchmarking Long-Form Generation in Long-Context LLMs , author =. arXiv preprint arXiv:2409.02076 , year =
-
[36]
arXiv preprint arXiv:2410.04199 , year =
LongGenBench: Long-Context Generation Benchmark , author =. arXiv preprint arXiv:2410.04199 , year =
-
[37]
arXiv preprint arXiv:2406.05761 , year =
The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models , author =. arXiv preprint arXiv:2406.05761 , year =
-
[38]
arXiv preprint arXiv:2406.04770 , year =
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild , author =. arXiv preprint arXiv:2406.04770 , year =
-
[39]
arXiv preprint arXiv:2411.14199 , year =
OpenScholar: Synthesizing Scientific Literature with Retrieval-augmented Language Models , author =. arXiv preprint arXiv:2411.14199 , year =
-
[40]
Findings of ACL , year =
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning , author =. Findings of ACL , year =
-
[41]
Proceedings of EMNLP , year =
OpenCQA: Open-ended Question Answering with Charts , author =. Proceedings of EMNLP , year =
-
[42]
Proceedings of ACL , year =
Chart-to-Text: A Large-Scale Benchmark for Chart Summarization , author =. Proceedings of ACL , year =
-
[43]
Findings of EMNLP , year =
SciCap: Generating Captions for Scientific Figures , author =. Findings of EMNLP , year =
-
[44]
Proceedings of WACV , year =
PlotQA: Reasoning over Scientific Plots , author =. Proceedings of WACV , year =
-
[45]
arXiv preprint arXiv:2301.11843 , year =
Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checking , author =. arXiv preprint arXiv:2301.11843 , year =
-
[46]
arXiv preprint arXiv:2305.14761 , year =
UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoning , author =. arXiv preprint arXiv:2305.14761 , year =
-
[47]
Proceedings of NAACL , year =
MMC: Advancing Multimodal Chart Understanding with Large-scale Instruction Tuning , author =. Proceedings of NAACL , year =
-
[48]
Findings of ACL , year =
ChartInstruct: Instruction Tuning for Chart Comprehension and Reasoning , author =. Findings of ACL , year =
-
[49]
arXiv preprint arXiv:2407.04172 , year =
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild , author =. arXiv preprint arXiv:2407.04172 , year =
-
[50]
IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
An Empirical Evaluation of the GPT-4 Multimodal Language Model on Visualization Literacy Tasks , author =. IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
-
[51]
Advances in Neural Information Processing Systems , year =
STaR: Bootstrapping Reasoning With Reasoning , author =. Advances in Neural Information Processing Systems , year =
-
[52]
arXiv preprint arXiv:2308.08998 , year =
Reinforced Self-Training (ReST) for Language Modeling , author =. arXiv preprint arXiv:2308.08998 , year =
-
[53]
Advances in Neural Information Processing Systems , year =
Toolformer: Language Models Can Teach Themselves to Use Tools , author =. Advances in Neural Information Processing Systems , year =
-
[54]
Proceedings of ICLR , year =
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving , author =. Proceedings of ICLR , year =
-
[55]
arXiv preprint arXiv:2310.12823 , year =
AgentTuning: Enabling Generalized Agent Abilities for LLMs , author =. arXiv preprint arXiv:2310.12823 , year =
-
[56]
arXiv preprint arXiv:2410.07706 , year =
AgentBank: Towards Generalized LLM Agents via Fine-Tuning on 50000+ Interaction Trajectories , author =. arXiv preprint arXiv:2410.07706 , year =
-
[57]
Advances in Neural Information Processing Systems , year =
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author =. Advances in Neural Information Processing Systems , year =
-
[58]
Advances in Neural Information Processing Systems , year =
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback , author =. Advances in Neural Information Processing Systems , year =
-
[59]
Proceedings of ICLR , year =
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets , author =. Proceedings of ICLR , year =
-
[60]
Proceedings of ICML , year =
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference , author =. Proceedings of ICML , year =
-
[61]
arXiv preprint arXiv:2410.12784 , year =
JudgeBench: A Benchmark for Evaluating LLM-based Judges , author =. arXiv preprint arXiv:2410.12784 , year =
-
[62]
arXiv preprint arXiv:2410.10934 , year =
Agent-as-a-Judge: Evaluate Agents with Agents , author =. arXiv preprint arXiv:2410.10934 , year =
-
[63]
IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
FinFlier: Automating Graphical Overlays for Financial Visualizations with Knowledge-Grounding Large Language Models , author =. IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
-
[64]
IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
Data Formulator: AI-powered Concept-driven Visualization Authoring , author =. IEEE Transactions on Visualization and Computer Graphics (TVCG) , year =
-
[65]
arXiv preprint arXiv:2408.16119 , year =
Data Formulator 2: Iteratively Creating Rich Visualizations with AI , author =. arXiv preprint arXiv:2408.16119 , year =
-
[66]
Proceedings of ACL System Demonstrations , year =
LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Models , author =. Proceedings of ACL System Demonstrations , year =
-
[67]
arXiv preprint arXiv:2302.02094 , year =
Chat2VIS: Generating Data Visualizations via Natural Language using ChatGPT, Codex and GPT-3 Large Language Models , author =. arXiv preprint arXiv:2302.02094 , year =
-
[68]
IEEE Transactions on Visualization and Computer Graphics , year =
NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queries , author =. IEEE Transactions on Visualization and Computer Graphics , year =
-
[69]
arXiv preprint arXiv:2406.03753 , year =
VisLTR: Visualization-in-the-Loop Table Reasoning , author =. arXiv preprint arXiv:2406.03753 , year =
-
[70]
Computer Graphics Forum , year =
AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making , author =. Computer Graphics Forum , year =
-
[71]
IEEE Transactions on Visualization and Computer Graphics , year =
ChartInsighter: An Approach for Mitigating Hallucination in Time-series Chart Summary Generation with A Benchmark Dataset , author =. IEEE Transactions on Visualization and Computer Graphics , year =
-
[72]
2024 , howpublished =
GPT-Vis: AI-native Visualization Library , author =. 2024 , howpublished =
2024
-
[73]
2024 , howpublished =
VMind: LLM-powered Visualization Component Library , author =. 2024 , howpublished =
2024
-
[74]
, journal =
Bradley, Ralph Allan and Terry, Milton E. , journal =. Rank Analysis of Incomplete Block Designs:
-
[75]
arXiv preprint arXiv:2508.11847 , year =
Dropping a Handful of Preferences Can Flip Arena Rankings , author =. arXiv preprint arXiv:2508.11847 , year =
-
[76]
Identity-Link
Chen, Wei and others , journal =. Identity-Link
-
[77]
arXiv preprint arXiv:2505.15240 , year =
Generalised Probabilistic and Uncertainty-Driven Evaluation , author =. arXiv preprint arXiv:2505.15240 , year =
-
[78]
Judgment Becomes Noise: Multidimensional Collapse in
Park, Jinhee and others , journal =. Judgment Becomes Noise: Multidimensional Collapse in
-
[79]
arXiv preprint arXiv:2502.01754 , year =
Coupled Token Generation for Pairwise Comparison , author =. arXiv preprint arXiv:2502.01754 , year =
-
[80]
arXiv preprint arXiv:2512.11150 , year =
Causal Judge Evaluation: 5\ author =. arXiv preprint arXiv:2512.11150 , year =
-
[81]
arXiv preprint arXiv:1606.08842 , year =
Active Ranking using Pairwise Comparisons , author =. arXiv preprint arXiv:1606.08842 , year =
-
[82]
Chatbot Arena: An Open Platform for Evaluating
Zheng, Lianmin and others , journal =. Chatbot Arena: An Open Platform for Evaluating
-
[83]
2026 , month = jul, publisher =
Jin, Jiajie and Zhang, Yuyao and Xu, Yimeng and Qian, Hongjin and Zhu, Yutao and Dou, Zhicheng , booktitle =. 2026 , month = jul, publisher =
2026
-
[84]
Findings of the Association for Computational Linguistics: ACL 2026 , year =
Cogito: A Cognitive Agentic Framework Driven by Dynamic Graph of Thoughts for Financial Report Generation , author =. Findings of the Association for Computational Linguistics: ACL 2026 , year =. doi:10.18653/v1/2026.findings-acl.430 , url =
2026 doi
-
[85]
2026 , url =
Li, Xiangyi and others , journal =. 2026 , url =
2026
-
[86]
arXiv preprint arXiv:2605.23899 , year =
From Raw Experience to Skill Consumption: A Systematic Study of Model-Generated Agent Skills , author =. arXiv preprint arXiv:2605.23899 , year =
-
[87]
2026 , url =
Alzubi, Ahmad and others , journal =. 2026 , url =
2026
-
[88]
2026 , url =
Shen, Shuaike and others , journal =. 2026 , url =
2026
-
[89]
2026 , url =
Zhang, Hanrong and others , journal =. 2026 , url =
2026
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.