Pith. sign in

REVIEW 4 major objections 2 minor 5 cited by

ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking

T0 review · 4 major / 2 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper develops an automated framework that turns real S&P 500 annual-report tables into masked-span prediction tests, and shows that even leading LLMs make 10–20% errors on multi-step financial reasoning.

desk verdict The abstract describes ReasoningTrack, but the full text is a different paper about a financial hallucination benchmark; as a submission, the central claim is unsupported. read the letter →

arxiv 2508.05221 v1 pith:JIR63VE3 submitted 2025-08-07 cs.CV cs.AIcs.LG

Xiao Wang , Liye Jin , Xufeng Lou , Shiao Wang , Lan Chen , Bo Jiang , Zhipeng Zhang This is my paper
classification cs.CVcs.AIcs.LG
keywords intrinsichallucinationfinancialLLMtabularreasoningmaskedspanprediction10-Kannualreportsbenchmarkconstructionnumerical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to establish that intrinsic hallucination in financial LLMs can be measured at scale by turning real annual-report numbers into a masked-span prediction test. The proposed FAITH framework automatically extracts numerical claims from S&P 500 10-K filings, masks the values, and asks models to reconstruct them, with the recovered value checked against the document's own ground truth. The central empirical finding is a reliability hierarchy: frontier proprietary models stay near 91–95% overall accuracy, but all models degrade sharply as reasoning complexity increases, with even top models showing 10–20% error rates on the most complex multi-step calculations and many smaller open-source models scoring near zero. A sympathetic reader would care because this adds a scalable, domain-specific measurement for a failure mode that general-domain benchmarks miss, and it identifies multi-step numerical consistency as the current bottleneck for safe financial automation.

What carries the argument

The central machinery is a masking-plus-filtering pipeline: numerical spans from real 10-K MD&A text are masked, and only spans that three frontier LLMs unanimously classify as answerable are retained, stratified into four financial reasoning types (lookup, comparative, bivariate, multivariate). This converts an otherwise unlabeled corpus into a verifiable hallucination benchmark, with the document itself supplying ground truth.

What would settle it

Collect a random sample of the spans that the three LLM annotators did not unanimously agree were answerable, obtain expert labels for their true answerability, and score models on them. If model accuracy on those spans is materially lower than the reported benchmark numbers, FAITH's filtering step has removed the hardest cases and the benchmark underestimates intrinsic hallucination.

Watch

Extended reading notes

Core claim

FAITH treats an LLM's intrinsic hallucination on financial tables as a context-aware masked span prediction problem. Given a sentence from a company's Management's Discussion and Analysis section, the numerical span is masked and the model is prompted to produce the missing value and unit together with a step-by-step rationale. The labels are not synthetic: the true value exists in the XBRL filing, so correctness is verifiable. The paper contributes a four-way taxonomy of the reasoning needed to recover a span—Direct Lookup, Comparative Calculation, Bivariate Calculation, and Multivariate Calculation—and shows experimentally that this complexity axis is the main driver of hallucination. On t

Load-bearing premise

The benchmark's difficulty estimates assume the unanimous LLM answerability filter does not systematically exclude the hardest or most ambiguous spans.

Editorial extensions

If this is right

  • Any institution with proprietary financial documents can build a hallucination benchmark without manual annotation, as long as the filings contain explicit numerical ground truth.
  • Accuracy reported per reasoning type becomes a complexity profile, letting buyers of LLM systems see in advance where a model becomes unreliable.
  • The consistent 10–20% failure floor on multi-step calculations sets a concrete upper bound for trust in automated financial analysis with current frontier models.
  • The case study's scale-error pattern implies that value-only accuracy metrics hide a distinct unit/magnitude failure mode, suggesting evaluation should score value and unit separately.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The unanimous-consensus filter in the Main Split likely makes the reported error rates a lower bound; the 2-1 disagreement rows of the paper's own agreement table suggest a pool of ambiguous spans that would likely be harder for all models, so real-world hallucination rates on messy documents may be higher than the benchmark shows.
  • If the masking strategy transfers, the same recipe could produce hallucination tests for other numeric-heavy domains—medical lab reports, insurance claims, supply-chain disclosures—where verified source documents exist.
  • The case study suggests that the binding limitation is cross-modal integration of prose and tables; a direct test of this inference would be to prompt models with explicit pointers to the relevant table rows and measure whether the multivariate gap closes.
  • The four-way reasoning taxonomy is a natural complexity ladder, and a testable extension is to check whether the same accuracy hierarchy holds on non-financial tabular corpora.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 2 minor

Summary. The submitted artifact is internally inconsistent in a way that undermines its central claim. The abstract introduces ReasoningTrack, a vision-language tracking framework based on Qwen2.5-VL with SFT and GRPO optimization, a new long-term tracking benchmark TNLLT with 200 video sequences, 20 re-trained baseline trackers, and experiments on multiple vision-language tracking benchmarks. The full body, however, is an entirely different paper: "FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance," with its own title, authors, ACM DOI, abstract, sections, and references. The body contains no description of ReasoningTrack, no Qwen2.5-VL integration, no SFT/GRPO formulation, no TNLLT dataset, no tracking head, and no tracking experiments or figures. The body's FAITH content is a complete independent study of financial hallucination evaluation, but it cannot serve as evidence for the stated vision-language tracking contribution. The mismatch is not a missing ablation or a local omission; it is a structural absence of the paper's claimed contribution.

Significance. If the abstract's tracking contribution were present, it could represent a useful step in long-term vision-language tracking, particularly the use of chain-of-thought reasoning with a large vision-language model. However, as submitted, the paper has no assessable content on that topic. The FAITH body itself has credible methodological elements that should be credited: the human-annotated pilot has high inter-annotator agreement (Fleiss Kappa 0.905), ground-truth answer labels are derived from real SEC filings rather than model-generated content, and the four-way reasoning taxonomy is a sensible organizing principle for financial numerical spans. Those strengths, however, belong to a different manuscript and a different research area. They do not provide support for the abstract's claims about Qwen2.5-VL, GRPO, TNLLT, or tracking performance. The paper as a whole cannot be considered a valid submission on its stated topic.

major comments (4)
  1. [Abstract / Full text] The abstract claims a vision-language tracking framework (ReasoningTrack), a new TNLLT dataset, SFT/GRPO training, 20 re-trained baseline trackers, and tracking experiments. The full text is the FAITH financial-hallucination paper, with a different title, authors, ACM DOI, and complete unrelated content. No section, equation, figure, or table in the body addresses ReasoningTrack, Qwen2.5-VL, tracking architecture, or TNLLT. The GitHub URL in the abstract points to Open_VLTrack, not to a repository associated with the body. This is not a local gap: the paper's central claim has zero support in the submitted artifact. The body cannot be used as evidence for the abstract's claims, so the submission is not reviewable as a paper on reasoning-based vision-language tracking.
  2. [Section 4.1 / Section 7] The Main Split is constructed by retaining a span only if GPT-4.1, Claude-Sonnet-4, and Gemini-2.5-Pro unanimously classify it as answerable, and the same class of frontier LLMs is later evaluated on that split (Table 3). Section 7 acknowledges that this filter "may systematically exclude the most challenging or ambiguous spans—precisely those that elicit disagreement or uncertainty across models." This means reported accuracy numbers are likely optimistic with respect to the true difficulty of financial hallucination. Because the paper's empirical claim is that even state-of-the-art models show 10–20% error rates on multi-step reasoning, this filtering decision is load-bearing. A concrete remedy would be to report performance on the pilot disagreement subsets (Table 1: all 2-1 splits, 210 spans total) and to state CIs or robustness checks under alternative annotation thresholds.
  3. [Table 3] The headline result about complexity-driven degradation, especially the Multivariate Calculation scenario, rests on extremely small samples: n=8 in the Pilot split and n=10 in the Main Split. With n=10, a single example changes the reported accuracy by 10 percentage points, so claims such as Gemini-2.5-Pro at 90.0% and Claude-Sonnet-4 at 80.0% in scenario D are statistically fragile. No confidence intervals or error bars are provided. The paper's central claim of a "10-20% error rate" on multi-step numerical reasoning therefore is not established with the reported evidence. The authors should either substantially enlarge the Multivariate sample or temper the conclusion to a qualitative observation.
  4. [Section 5.1] For the Main Split, scenario labels (A/B/C/D) are assigned by the models themselves via self-classification in the evaluation prompt, not by human annotators. The per-scenario accuracy numbers in Table 3 therefore do not measure performance against an external ground-truth label for reasoning type. If a model misidentifies its own reasoning process, the corresponding accuracy cell is corrupted. This is especially important for the claim in Section 5.2.2 that accuracy systematically decreases with reasoning complexity. The Pilot split uses human labels, but the broader conclusions rely mainly on the Main split. A random Main-split subset should be human-labeled for scenario type to validate the self-classification protocol.
minor comments (2)
  1. [Table 3] The table header is hard to read: the Pilot and Main split blocks are not visually separated, and the n values for scenario columns are listed only in the header row without clear correspondence to the A/B/C/D columns. Add explicit column spans or per-column n labels.
  2. [Section 5.1] The text says "For the main split, we use the strategy stated in section 3.4," but the submitted body does not contain a Section 3.4. Either the numbering is inconsistent or part of the manuscript is missing; this needs correction.

Circularity Check

3 steps flagged · score 4.0 of 10

FAITH body is not centrally circular (ground truth from filings), but Main Split answerability filtering and self-reported scenario labels create partial self-reference; the ReasoningTrack abstract is unsupported by the body.

  1. other [Section 4.1 (LLM-based Answerability Annotation); Table 1/Table 3; Section 7]
    "The LLMs used are GPT-4.1, Claude-sonnet-4, and Gemini-2.5-pro. ... A span is retained in the final dataset only if all three models unanimously classified it as answerable."

    The Main Split is filtered by the unanimous consensus of exactly the same three frontier models that Table 3 evaluates as top performers. Thus the benchmark's difficulty is partially self-selected: spans that these models judge answerable are retained, and their accuracy is then reported as an external finding. The pilot study validates the heuristic against human labels, but the Main Split itself remains LLM-filtered, so the loop is not broken. This is a selection-by-construction concern rather than a forced answer, and the paper's Sec. 7 admission that hard/ambiguous spans may be excluded confirms the bias.

  2. self definitional [Section 5.1 (Experimental Setup) and Table 3 scenario columns]
    "During evaluation, we prompt the models to generate a step-by-step rationale before predicting the final masked value and to self-classify their reasoning process into one of the four financial scenarios (A-D). ... For the main split, we use the strategy stated in section 3.4."

    Main Split scenario labels are generated by the same model that is being scored. The per-scenario accuracy in Table 3 is therefore computed against labels the model itself produced, so a model's scenario breakdown is not independent of its own reasoning classification. Cross-model comparisons of Direct Lookup vs. Multivariate Calculation on the Main Split reflect each model's self-classification tendencies as much as its ability, making the 'reasoning complexity drives hallucination' result partially self-definitional for the Main Split.

1 more flagged steps
  1. other [Abstract vs. Full Text title/body]
    "Abstract: 'this paper proposes a novel reasoning-based vision-language tracking framework, named ReasoningTrack... we propose a large-scale long-term vision-language tracking benchmark dataset, termed TNLLT...' Full text title: 'FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance.'"

    This is not a circular derivation but a complete absence of the claimed derivation chain. The abstract's ReasoningTrack, Qwen2.5-VL SFT/GRPO, TNLLT, and tracking experiments do not appear anywhere in the body, which is a different paper with different authors and an ACM DOI. Per the review rule, this missing support is flagged explicitly; it is a severe integrity defect but does not raise the circularity score because there is no derivation to reduce.

full rationale

The FAITH body's core answer labels are external numbers from SEC 10-K filings, so the overall hallucination rates are not purely constructed by the evaluated models. However, two steps partially close the loop. First, the Main Split retains only spans that GPT-4.1, Claude-Sonnet-4, and Gemini-2.5-Pro unanimously judge answerable; those are the same frontier models later scored in Table 3. The paper's own limitation (Sec. 7) concedes this 'may systematically exclude the most challenging or ambiguous spans,' meaning the 10-20% error rates on complex reasoning are measured on a set filtered by the evaluated models' consensus. Second, Main Split scenario labels are the models' self-classifications, so per-scenario accuracies in Table 3 are defined by the same model's own output; this makes cross-model scenario comparisons and the 'complexity-driven degradation' claim partially tautological. The pilot's human annotation validates the filter's accuracy but does not break the loop for the Main Split. Separately, the submission's abstract promises a ReasoningTrack/TNLLT tracking paper, while the body is an unrelated FAITH paper; this is a missing-support/integrity defect, not a circular derivation, so it is flagged but does not by itself raise the circularity score. Overall circularity is partial: score 4.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

There are no invented physical or mathematical entities. The central new object is the benchmark itself, which rests on design choices: the unanimous-consensus threshold for filtering, the assumption that masked spans with unique ground truth capture hallucination, and the choice of exact-match accuracy as the primary metric.

free parameters (1)
  • LLM unanimous consensus threshold = 3/3 agreement across GPT-4.1, Claude-sonnet-4, Gemini-2.5-pro
    Hand-chosen filtering rule in Sec. 4.1; determines which spans enter the Main Split and thus shapes all reported accuracy numbers.
assumptions (3)
  • domain assumption Masked numerical spans with unique ground truth in the source document are valid units for measuring intrinsic hallucination.
    The whole benchmark defines hallucination as failure to recover the masked value; Sec. 1 and Sec. 3 motivate this but do not prove it is the right operationalization for financial risk.
  • domain assumption Unanimous consensus of three frontier LLMs is a reliable proxy for human answerability judgment.
    Validated only on 1,124 pilot spans (Sec. 3.4); the Main Split relies entirely on this heuristic (Sec. 4.1), including the acknowledged risk it excludes hard cases (Sec. 7).
  • domain assumption Exact value and unit accuracy is the primary measure of hallucination severity.
    Sec. 5.1 defines accuracy as the primary metric; scale errors are counted as wrong, but the paper does not weight error severity by financial impact.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking." pith.science (2026). https://pith.science/paper/JIR63VE3

@misc{pith2026250805221,
  author       = {Pith},
  title        = {Pith review of: ReasoningTrack: Chain-of-Thought Reasoning for Long-term Vision-Language Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIR63VE3}},
  note         = {Machine review of arXiv:2508.05221}
}
read the original abstract

Vision-language tracking has received increasing attention in recent years, as textual information can effectively address the inflexibility and inaccuracy associated with specifying the target object to be tracked. Existing works either directly fuse the fixed language with vision features or simply modify using attention, however, their performance is still limited. Recently, some researchers have explored using text generation to adapt to the variations in the target during tracking, however, these works fail to provide insights into the model's reasoning process and do not fully leverage the advantages of large models, which further limits their overall performance. To address the aforementioned issues, this paper proposes a novel reasoning-based vision-language tracking framework, named ReasoningTrack, based on a pre-trained vision-language model Qwen2.5-VL. Both SFT (Supervised Fine-Tuning) and reinforcement learning GRPO are used for the optimization of reasoning and language generation. We embed the updated language descriptions and feed them into a unified tracking backbone network together with vision features. Then, we adopt a tracking head to predict the specific location of the target object. In addition, we propose a large-scale long-term vision-language tracking benchmark dataset, termed TNLLT, which contains 200 video sequences. 20 baseline visual trackers are re-trained and evaluated on this dataset, which builds a solid foundation for the vision-language visual tracking task. Extensive experiments on multiple vision-language tracking benchmark datasets fully validated the effectiveness of our proposed reasoning-based natural language generation strategy. The source code of this paper will be released on https://github.com/Event-AHU/Open_VLTrack

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

    cs.CV 2026-01 unverdicted novelty 8.0 of 10

    Molmo2 delivers state-of-the-art open-weight video VLMs with new grounding datasets and training methods that outperform prior open models and match or exceed some proprietary ones on pointing and tracking tasks.

  2. STORM: End-to-End Referring Multi-Object Tracking in Videos

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    STORM is an end-to-end MLLM for referring multi-object tracking that uses task-composition learning to leverage sub-task data and introduces the STORM-Bench dataset, achieving SOTA results.

  3. Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Bridge-STG decouples spatio-temporal alignment via semantic bridging and query-guided localization modules to achieve state-of-the-art m_vIoU of 34.3 on VidSTG among MLLM methods.

  4. VPTracker: Global Vision-Language Tracking via Visual Prompt

    cs.CV 2025-12 conditional novelty 7.0 of 10

    VPTracker enables global object tracking in videos by using multimodal large language models with location-aware visual prompts to search entire images while reducing distractions from similar objects.

  5. Dynamic Parsing and Updating Natural Language Specification using VLMs for Robust Vision-Language Tracking

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    A language dependency parsing mechanism combined with Qwen-VL enables adaptive updates to textual descriptions for improved vision-language tracking performance on benchmarks like TNL2K and LaSOT.

Reference graph

Works this paper leans on

34 extracted references · 11 canonical work pages · cited by 5 Pith papers

  1. [1]

    Shayan Ali Akbar, Md Mosharaf Hossain, Tess Wood, Si-Chi Chin, Erica Sali- nas, Victor Alvarez, and Erwin Cornejo. 2024. HalluMeasure: Fine-grained Hallucination Measurement Using Chain-of-Thought Reasoning. InProceed- ings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2024. Association for Computational Linguistics, ...

    doi:10.18653/v1/2024.emnlp-main.837 2024

  2. [2]

    Amos Azaria and Tom Mitchell. 2023. The Internal State of an LLM Knows When It’s Lying. InFindings of the Association for Computational Linguistics: EMNLP

    work page 2023

  3. [3]

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. HalluLens: LLM Hallucination Benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 24128–24156. https://doi.org/10.4855...

    arXiv doi:10.48550/arxiv.2504.17550 2025

  4. [4]

    Yupeng Cao, Zhi Chen, Qingyun Pei, Nathan Lee, K. P. Subbalakshmi, and Papa Momar Ndiaye. 2024. ECC Analyzer: Extracting Trading Signal from Earnings Conference Calls using Large Language Model for Stock Volatility Pre- diction. InProceedings of the 5th ACM International Conference on AI in Finance, ICAIF 2024. ACM, 257–265. https://doi.org/10.1145/367705...

    arXiv 2024

  5. [5]

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. 2021. FinQA: A Dataset of Numerical Reasoning over Financial Data. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Lingui...

    work page 2021

  6. [6]

    Yuzhe Gu, Ziwei Ji, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen

    work page

  7. [7]

    Xiangkun Hu, Dongyu Ru, Lin Qiu, Qipeng Guo, Tianhang Zhang, Yang Xu, Yun Luo, Pengfei Liu, Yue Zhang, and Zheng Zhang. 2024. Knowledge-Centric Hallucination Detection. InProceedings of the 2024 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, 6953–6975. https://doi.org/10.18653/v1/2024.emnlp-main.395

    doi:10.18653/v1/2024.emnlp-main.395 2024

  8. [8]

    Haoqiang Kang and Xiao-Yang Liu. 2023. Deficiency of Large Language Models in Finance: An Empirical Examination of Hallucination.CoRR(2023). https: //doi.org/10.48550/ARXIV.2311.15548

    arXiv doi:10.48550/arxiv.2311.15548 2023

Show all 34 references
  1. [9]

    Michael Krumdick, Rik Koncel-Kedziorski, Viet Dac Lai, Varshini Reddy, Charles Lovering, and Chris Tanner. 2024. BizBench: A Quantitative Reasoning Bench- mark for Business and Finance. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 doi

  2. [10]

    Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluE- val: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. InProceedings of the 2023 Conference on Empirical Methods in Natu- ral Language Processing. Association for Computati...

    2023 doi

  3. [11]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring How Models Mimic Human Falsehoods. InProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Dublin, Ir...

    2022

  4. [12]

    Wen Luo, Tianshu Shen, Wei Li, Guangyue Peng, Richeng Xuan, Houfeng Wang, and Xi Yang. 2024. HalluDial: A Large-Scale Benchmark for Automatic Dialogue- Level Hallucination Evaluation.CoRR(2024). https://doi.org/10.48550/ARXIV. 2406.07070

    2024 doi arXiv

  5. [13]

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine- grained Atomic Evaluation of Factual Precision in Long Form Text Generation. InProceedings of the 2023 Conference on Empiric...

    2023 doi

  6. [14]

    The Moon is Made of Marshmallows

    Yifei Ming, Senthil Purushwalkam, Shrey Pandit, Zixuan Ke, Xuan-Phi Nguyen, Caiming Xiong, and Shafiq Joty. 2024. FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows".CoRR(2024). https://doi.org/10.48550/ARXIV.2410.03727

    2024 doi arXiv

  7. [15]

    Monetary Authority of Singapore. 2024. Artificial Intelligence (AI) Model Risk Management. (dec 2024). Available at: https://www.mas.gov.sg/- /media/mas-media-library/publications/monographs-or-information- paper/imd/2024/information-paper-on-ai-risk-management-final.pdf

    2024

  8. [16]

    Syed Shariyar Murtaza, Yifan Nie, Elias Avan, Utkarsh Soni, Wanyu Liao, Adam Carnegie, Cyril John Mathias, Junlin Jiang, and Eugene Wen. 2025. Implementing Retrieval Augmented Generation Technique on Unstructured and Structured Data Sources in a Call Center of a Large Financia...

    2025

  9. [17]

    Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, KaShun Shum, Randy Zhong, Juntong Song, and Tong Zhang. 2024. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computati...

    2024 doi

  10. [18]

    Jio Oh, Soyeon Kim, Junseok Seo, Jindong Wang, Ruochen Xu, Xing Xie, and Steven Euijong Whang. 2024. ERBench: An Entity-Relationship based Automat- ically Verifiable Hallucination Benchmark for Large Language Models.CoRR (2024). https://doi.org/10.48550/ARXIV.2403.05266

    2024 doi arXiv

  11. [19]

    Sohini Roychowdhury. 2024. Journey of Hallucination-minimized Generative AI Solutions for Financial Decision Makers. InProceedings of the 17th ACM International Conference on Web Search and Data Mining (WSDM ’24). Association for Computing Machinery, 1180–1181. https://doi.org...

    2024

  12. [20]

    Sohini Roychowdhury, Andres Alvarez, Brian Moore, Marko Krema, Maria Paz Gelpi, Punit Agrawal, Federico Martin Rodriguez, Angel Rodriguez, Jose Ramon Cabrejas, Pablo Martinez Serrano, and Arijit Mukherjee. 2023. Hallucination- minimized Data-to-answer Framework for Financial D...

    2023

  13. [21]

    Bhaskarjit Sarmah, Dhagash Mehta, Stefano Pasquali, and Tianjie Zhu. 2024. Towards reducing hallucination in extracting information from financial reports using Large Language Models. InProceedings of the Third International Conference on AI-ML Systems (AIMLSystems ’23). Assoc...

    2024

  14. [22]

    Agam Shah, Liqin Ye, Sebastian Jaskowski, Wei Xu, and Sudheer Chava. 2025. Be- yond the Reported Cutoff: Where Large Language Models Fall Short on Financial Knowledge.CoRR(2025). https://doi.org/10.48550/ARXIV.2504.00042

    2025 doi arXiv

  15. [23]

    Yan Wang, Yang Ren, Lingfei Qian, Xueqing Peng, Keyi Wang, Yi Han, Dongji Feng, Xiao-Yang Liu, Jimin Huang, and Qianqian Xie. 2025. FinTagging: An LLM-ready Benchmark for Extracting and Structuring Financial Information. CoRR(2025). https://doi.org/10.48550/ARXIV.2505.20650

    2025 doi arXiv

  16. [24]

    Pengzuo Wu, Yuhang Yang, Guangcheng Zhu, Chao Ye, Hong Gu, Xu Lu, Ruix- uan Xiao, Bowen Bao, Yijing He, Liangyu Zha, Wentao Ye, Junbo Zhao, and Haobo Wang. 2025. RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis.CoRR(2...

    2025 doi

  17. [25]

    Xianjie Wu, Jian Yang, Linzheng Chai, Ge Zhang, Jiaheng Liu, Xeron Du, Di Liang, Daixin Shu, Xianfu Cheng, Tianzhen Sun, Tongliang Li, Zhoujun Li, and Guanglin Niu. 2025. TableBench: A Comprehensive and Complex Benchmark for Table Question Answering. InAAAI-25, Sponsored by th...

    2025 doi

  18. [26]

    Xiaodong Yu, Hao Cheng, Xiaodong Liu, Dan Roth, and Jianfeng Gao. 2024. ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks. InFindings of the Asso- ciation for Computational Linguistics: NAACL 2024, Mex...

    2024 doi

  19. [27]

    Yilun Zhao, Yunxiang Li, Chenying Li, and Rui Zhang. 2022. MultiHiertt: Nu- merical Reasoning over Multi Hierarchical Tabular and Textual Data. InProceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for ...

    2022 doi

  20. [28]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.CoRR(2023). https://doi.org/10.48550...

    2023 doi arXiv

  21. [29]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, and Zheyan Luo

  22. [30]

    Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. 2021. TAT-QA: A Question An- swering Benchmark on a Hybrid of Tabular and Textual Content in Finance. In

    2021

  23. [33]

    InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)

    LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations). Association for Computational Linguistics, 400–410. https://doi.org/10.18653/v1/202...

    2024 doi

  24. [2023]

    https://doi.org/10

    Association for Computational Linguistics, 967–976. https://doi.org/10. 18653/v1/2023.findings-emnlp.68

    2023

  25. [2024]

    https://doi.org/10.48550/ARXIV.2407.04693

    ANAH-v2: Scaling Analytical Hallucination Annotation of Large Language Models.CoRR(2024). https://doi.org/10.48550/ARXIV.2407.04693

    2024 doi arXiv

  26. [3711]

    https://doi.org/10.18653/v1/2021.emnlp-main.300

    2021 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.