Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Expect the Unexpected: FailSafe Long Context QA for Finance

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces FailSafeQA, a 220-item long-context financial QA benchmark, and claims that no tested LLM simultaneously maintains accurate answers under input perturbations and refrains from hallucinating when context is missing or…

desk verdict A useful benchmark with a promising robustness/refusal split, but the tables contradict themselves and the headline rankings are not reproducible as printed. read the letter →

arxiv 2502.06329 v1 pith:5RHF46A3 submitted 2025-02-10 cs.CL

Kiran Kamble , Melisa Russak , Dmytro Mozolevskyi , Muayad Ali , Mateusz Russak , Waseem AlShikh This is my paper · ORCID
classification cs.CL
keywords FailSafeQAlong-contextquestionansweringfinancialNLPbenchmarkLLMrobustnesshallucinationpreventioncontextgroundingLLM-as-a-judge10-Kfilings
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces FailSafeQA, a benchmark built from 220 long-context financial question-answer tasks derived from 10-K filings, and uses it to score 24 off-the-shelf LLMs under six realistic input failures: misspelled, incomplete, and out-of-domain queries, alongside missing, OCR-degraded, and irrelevant documents. Its central claim is that dependable deployment requires two behaviours that no tested model exhibits together: answering correctly under messy input (Robustness) and refusing to answer when the context cannot support an answer (Context Grounding). The paper reports a sharp trade-off: the most robust model, o3-mini, fabricates information in 41% of unanswerable cases, while the most grounded model, Palmyra-Fin-128k-Instruct, drops robust predictions in 17% of test cases. A new Compliance score formalizes this trade-off, and the highest score any model achieves is 0.81. If the benchmark is right, single-number accuracy evaluations hide the failure mode that matters most for financial use.

What carries the argument

The central object is the FailSafeQA benchmark itself: 220 examples from 10-K filings, each paired with an original query, three perturbed query variants (misspelled, incomplete, out-of-domain), an OCR-corrupted context, and missing- and irrelevant-context conditions. The mechanism that carries the argument is the pair of scores. Robustness is defined as $R = \frac{1}{n}\sum_i \min_j c_{\ge4}(\text{model}(T_j(x_i)), y_i)$, and Context Grounding as $G = \frac{1}{2n}\sum_{j=1}^2\sum_{i=1}^n c_{\ge4}(\text{model}(T_j(x_i), Y))$, where $c_{\ge4}$ maps a 1–6 relevance rating into a binary compliant-versus-fabricated label (rating at least 4 means compliant). The two are combined into a Compliance score $\mathrm{LLMC}_{\beta} = \frac{(1+\beta^2)RG}{\beta^2 G + R}$ with $\beta = 0.5$, which prioritizes refusal over answering. The design makes judging tractable by giving the judge LLM short supporting citations instead of full 25k-token contexts, trading long-context degradation for a simpler verification task.

What would settle it

Select a random subset of about 50 FailSafeQA items covering all six perturbation types, have two human raters independently apply the paper's 1-6 criteria, and compare their labels to the Qwen2.5-72B-Instruct judge labels, measuring agreement with Cohen's kappa. If agreement falls below 0.6, or if the judge gives Qwen-family answers systematically higher relevance than humans do, every model score and the reported robustness-grounding trade-off would need to be recomputed.

Watch

Extended reading notes

Core claim

FailSafeQA claims that dependable deployment of LLMs in finance requires two separable behaviours—answering correctly under messy user input (Robustness) and refusing to answer when the available context cannot support an answer (Context Grounding)—and that current models cannot do both. Across 24 models, every one loses accuracy under perturbation, with the largest drops from OCR-degraded context and out-of-domain phrasing. More striking, the best-answering models fail the refusal test: o3-mini, the most robust model at 0.90, fabricates answers in 41% of unanswerable cases, and reasoning-focused models fabricate in 41 to 70% of those cases. Palmyra-Fin-128k-Instruct has the best Context Grounding at 0.80 but drops robust predictions in 17% of test cases. The paper's compliance metric formalizes this robustness-versus-grounding trade-off and ranks no tested model above 0.81.

Load-bearing premise

The entire ranking assumes the judge model Qwen2.5-72B-Instruct rates answers as accurately as a human rater for all 24 models, an assumption the paper does not test with a human-agreement study.

Editorial extensions

If this is right

  • Robustness and Context Grounding should be reported as separate axes in any financial QA evaluation; averaging them together hides the 41% hallucination rate of the most robust model.
  • Refusal behavior is a trainable capability: the Compliance score gives a concrete objective, and the 0.81 ceiling shows measurable headroom for models that learn when to say the context is insufficient.
  • OCR-degraded and out-of-domain inputs are the hardest perturbations, so systems deployed on scanned contracts should detect document quality early rather than rely on the LLM to recover.
  • Text-generation queries such as 'write a blog post' are more prone to hallucination than question-answering queries, so generation pipelines should retrieve grounded facts first and assemble text later.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the robustness-grounding trade-off holds beyond these 24 models, interventions that improve refusal, such as a separate abstention classifier, may be more practical than trying to make one LLM excel at both behaviours.
  • The judge premise is testable independently: a human-agreement audit on a 50-item sample, including answers from Qwen2.5-72B-Instruct itself, would show whether any model is systematically favored.
  • The same six perturbations could be ported to legal, medical, or technical documentation with minimal change, since the only domain-specific step is generating out-of-domain query paraphrases.
  • One could also use FailSafeQA-style empty-context items to measure whether retrieval-augmented systems that explicitly signal 'no documents retrieved' reduce hallucination rates below the raw model scores reported here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces FailSafeQA, a 220-example long-context financial QA benchmark built from truncated 10-K filings, with six human-interface failure modes: misspelled, incomplete, and out-of-domain queries, plus missing, OCR-degraded, and irrelevant contexts. It evaluates 24 long-context LLMs using Qwen2.5-72B-Instruct as an LLM judge on a 1–6 relevance rubric, and summarizes performance through three metrics: Robustness (Eq. 2), Context Grounding (Eq. 3), and a beta-weighted Compliance score (Eq. 4). The headline findings are that the most robust model (OpenAI o3-mini) fails to refuse or fabricates in a large share of unanswerable cases, that the most compliant model (Palmyra-Fin-128k-Instruct) loses robustness in 17% of cases, and that no tested model is simultaneously robust and well-grounded.

Significance. The benchmark targets a real and under-served problem: dependability of financial QA systems under realistic user and document failures. Strengths include a public dataset, a clearly described perturbation taxonomy, citation-based judging to keep the judge's context short, and an explicit trade-off metric for answer-versus-refuse behavior. If the evaluation were independently validated, this would be a useful resource for the long-context and financial NLP communities. However, the current empirical payload—the 41% fabrication rate, the model ordering, and the robustness-versus-grounding trade-off—rests on a single unvalidated judge that is itself a scored model, and it conflicts with the numbers in Table 2. The benchmark may still be valuable, but this draft does not establish the claimed comparative results.

major comments (3)
  1. [Table 2; §3.2, Eq. (3)] The Context Grounding column in Table 2 is not the value defined by Eq. (3). Under the paired dataset design, Context Grounding should be the equal-weighted average of the Irrelevant Ctx and No Ctx columns. For Gemini 2.0 Flash Exp this gives (0.81+0.66)/2 = 0.735, but the table reports 0.77; for OpenAI o3-mini it gives (0.67+0.51)/2 = 0.59, but the table reports 0.63; for Qwen2.5-72B-Instruct it gives 0.645, but the table reports 0.68. Moreover, the table's QA and TG Context Grounding columns cannot be sub-aggregates of the reported total: for Gemini 2.0 Flash Exp, 0.83×0.46+0.17×0.74 ≈ 0.51, not 0.77. The Compliance column, by contrast, is consistent with Eq. (4) if the equal-weighted average is used (for Palmyra-Fin, (0.95+0.66)/2 = 0.805 together with R=0.83 yields 0.81, matching the table). The paper therefore appears to use two different Context Grounding values in one table. This matters directly for the headline: the Abstract and §7's claim that o3-mini 'fabricated information in 41% of tested cases' corresponds to the equal-weighted average (1−0.59=0.41), whereas the reported Context Grounding of 0.63 implies 37%. The tables, the text, and the metric definitions must be reconciled before any of the comparative conclusions can be assessed.
  2. [§4.2 Judging; Tables 1–2] All scores in Tables 1 and 2 are produced by Qwen2.5-72B-Instruct, which is itself one of the 24 evaluated models (it has a row in both tables). The paper reports no human-rater agreement, no judge calibration against the 1–6 rubric of Appendix C.3, and no check for judge self-preference. The citation-based argument in §4.2 explains why the judge's context is short, but it does not address systematic grading bias across model families or between answer and refusal responses. Because the entire model ranking and all absolute rates are outputs of this single judge, a modest self-preference or rubric miscalibration would change the reported scores and the claimed trade-off. I would like to see a stratified validation sample (covering all 24 models and both answerable and unanswerable scenarios) double-scored by human annotators, with per-model and per-scenario agreement reported.
  3. [§3.2.1, Eq. (4)] The Compliance metric is a free-parameter harmonic score with beta=0.5, and no sensitivity analysis is given. This is load-bearing for the 'most compliant model' claim because Robustness and Context Grounding move in opposite directions for several leading models: OpenAI o3-mini has R=0.90 with Context Grounding around 0.59–0.63, while Palmyra-Fin-128k-Instruct has R=0.83 with Context Grounding around 0.805–0.83. The ranking under Eq. (4) can change as beta moves, so the paper should report Compliance for a range of beta values (e.g., 0.25, 0.5, 1, 2) or explicitly show that the ordering asserted in Section 7 is stable.
minor comments (6)
  1. [§5] The text says 'The best Context Grounding score of 0.80 is achieved by Palmyra-Fin-128k-Instruct', but Table 2 lists Palmyra-Fin's Context Grounding as 0.83; one of these is wrong.
  2. [§3.2, Eq. (3)] Equation (3) is typeset as 'G = 1/nj 2X j=1 nX i=1 ...', which is unreadable; it should be written as a simple average over the two context-failure conditions, e.g., G = (1/2)(G_Missing + G_Irrelevant).
  3. [§2.4] The text states that each data point includes 'an irrelevant query', but the perturbation described in §2.3 is an irrelevant context; please clarify the wording and whether the 'irrelevant query' is a question paired with an irrelevant 10-K excerpt.
  4. [Tables 1–2] No confidence intervals or significance tests are reported. With 220 examples, adjacent scores in the tables often differ by 0.01–0.02, which is well within the standard error of a proportion; bootstrap confidence intervals for at least the headline scores would help the reader judge whether the observed ordering is meaningful.
  5. [§2.3, OCR Errors] The 10% cap on character error probability is described as 'empirically chosen', but no sensitivity analysis or distribution of injected error rates is reported; please state how Robustness changes under a lower or higher cap, or provide the error-rate distribution.
  6. [Figures 5–6] The captions of Figures 5 and 6 are close to verbatim copies, and Figure 5's caption contains sentences about Context Grounding even though the title says 'Robustness vs. Query Type'; the captions should be corrected to describe the actual content of each figure.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation circularity found; the benchmark metrics are explicitly defined and the central empirical claims are computed from measured ratings, not from the conclusions themselves.

full rationale

The paper's derivation chain is self-contained in the sense that its metrics are explicit transformations of raw judge ratings. Robustness (Eq. 2) is the per-item minimum of binary compliance over answerable perturbations, Context Grounding (Eq. 3) is the average compliance over the two unanswerable conditions, and Compliance (Eq. 4) is a weighted harmonic blend of the two. None of these equations takes a headline conclusion as an input. The abstract's 'fabricated information in 41%' for o3-mini is the complement of the average of the two context-failure compliance columns in Table 2, which is exactly the paper's own definition of Context Grounding, so it is arithmetic from the reported scores, not a hidden fit. The use of Qwen2.5-72B-Instruct as both judge and one of the evaluated models is a genuine measurement-bias and self-preference concern, and the table contains internal inconsistencies (e.g., o3-mini's Ctx Grounding is 0.63 while its two columns average 0.59; several Compliance values do not reproduce Eq. 4 with beta=0.5). These are correctness, reproducibility, and validation weaknesses, not examples of a prediction being equivalent to an input by construction. Likewise, the post-hoc introduction of LLMC_beta with beta=0.5 after observing results is a form of HARKing and a free-parameter sensitivity gap, but the 'most compliant' ranking is still a direct computation from measured R and G under a stated formula, not a fitted parameter disguised as a prediction. For these reasons, per the hard rule requiring a specific reduction or fitted-parameter-as-prediction, no circular step can be substantiated from the paper's own equations or citations.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The main uncharged inputs are the validity of the judge, the semantic equivalence of the out-of-domain perturbation, and the beta weighting. These are assumptions and design choices, not parameters fitted to external truth, and they directly affect the reported model rankings.

free parameters (2)
  • Compliance metric beta = 0.5
    Eq. 4 defines LLM Compliance with a free beta; the authors fix beta=0.5 without sensitivity analysis, and the ranking of the 'most compliant model' depends on this choice.
  • OCR character error probability cap = 10%
    Section 2.3 states the upper limit was 'empirically chosen' to balance readability and realism; this affects the difficulty of the OCR scenario but is not fitted to an external target.
assumptions (3)
  • domain assumption Qwen2.5-72B-Instruct judge scores correspond to true answer quality for all 24 models, including stronger models and the judge itself.
    Invoked in Section 4.2; no human validation, no inter-annotator agreement, and the judge is one of the evaluated models.
  • domain assumption Rephrased Out-of-Domain queries are semantically equivalent to the original queries.
    Section 2.2 constructs these rephrasings with Llama 3.1 405B but does not verify semantic equivalence; a meaning change would incorrectly count as a robustness failure.
  • domain assumption The Llama-based generation and filtering pipeline produces accurate ground truths and supporting citations.
    Section 2.1 retains only citation-supported pairs, but no human audit of the final 220 examples is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expect the Unexpected: FailSafe Long Context QA for Finance." pith.science (2026). https://pith.science/paper/5RHF46A3

@misc{pith2026250206329,
  author       = {Pith},
  title        = {Pith review of: Expect the Unexpected: FailSafe Long Context QA for Finance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5RHF46A3}},
  note         = {Machine review of arXiv:2502.06329}
}
read the original abstract

We propose a new long-context financial benchmark, FailSafeQA, designed to test the robustness and context-awareness of LLMs against six variations in human-interface interactions in LLM-based query-answer systems within finance. We concentrate on two case studies: Query Failure and Context Failure. In the Query Failure scenario, we perturb the original query to vary in domain expertise, completeness, and linguistic accuracy. In the Context Failure case, we simulate the uploads of degraded, irrelevant, and empty documents. We employ the LLM-as-a-Judge methodology with Qwen2.5-72B-Instruct and use fine-grained rating criteria to define and calculate Robustness, Context Grounding, and Compliance scores for 24 off-the-shelf models. The results suggest that although some models excel at mitigating input perturbations, they must balance robust answering with the ability to refrain from hallucinating. Notably, Palmyra-Fin-128k-Instruct, recognized as the most compliant model, maintained strong baseline performance but encountered challenges in sustaining robust predictions in 17% of test cases. On the other hand, the most robust model, OpenAI o3-mini, fabricated information in 41% of tested cases. The results demonstrate that even high-performing models have significant room for improvement and highlight the role of FailSafeQA as a tool for developing LLMs optimized for dependability in financial applications. The dataset is available at: https://huggingface.co/datasets/Writer/FailSafeQA

Figures

Figures reproduced from arXiv: 2502.06329 by the authors.

Figure 1
Figure 1. FailSafeQA: Robustness and Context Grounding Evaluation We evaluate the resilience of an LLM￾based QA system in two case studies: Query Failure and Context Failure. In the Query Failure scenario, we perturb the original query into three variants: containing spelling errors (Misspelled Query), query-term form (Incomplete Query), rephrased to exclude in-domain terminology (Out-of-Domain Query). In the Context Failure … view at source ↗
Figure 2
Figure 2. The Dataset Analysis of root verbs and their direct objects from the first sentence of each normal￾ized query shows the top 20 verbs and their top five direct objects1 . This distribution can be used as a proxy measure for the diversity of tasks in the dataset, with 83.0% related to question answering (QA) and 17.0% involving text generation (TG). Prompts used for generating and filtering question-answer pairs can b… view at source ↗
Figure 3
Figure 3. Answer Relevance Classes We evaluate two scenarios in our benchmark: when models should provide an answer (ANSWER QUERY) and when they must decline to answer (REFUSE QUERY) due to lack of relevant context. Our findings reveal that all the tested models are more adept at offering suitable answers than providing a justified refusal in situations where the context lacks sufficient information. Among all models evaluate… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Robustness and Compliance (Left) All models lose with respect to the baseline when input perturbations are applied. The biggest drop is observed for Out-Of-Domain and OCR context perturbations. Among the 24 tested models, OpenAI o3-mini is the most robust. (Right) Reas…
Figure 5
Figure 5. Figure 5: Robustness vs. Query Type. (Left) Across all models, the decrease in robustness is more prominent in text generation (TG) than in question-answering (QA) tasks. (Right) Similar statement also holds true for Context Grounding - when a model is asked to generate text (e.…
Figure 6
Figure 6. Figure 6: Context Grounding vs. Query Type. (Left) Across all models, the decrease in robustness is more prominent in text generation (TG) than in question-answering (QA) tasks. (Right) Similar statement also holds true for Context Grounding - when a model is asked to generate t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Are the Financial Reasoning from LLMs Credible? A Real World Test over Long-Horizon Statements

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A new benchmark shows LLMs' financial calculation accuracy collapses without explicit formulas and degrades further when they must generate multi-metric tables.

Reference graph

Works this paper leans on

41 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Martin Cai, Qin Cai, Vishrav Chaudhary, Dong Chen, Dongdong Chen, Weizhu Chen, Yen-Chun Chen, Yi-Ling Chen, Hao Cheng, Parul Chopra, Xiyang Dai, Matt...

    arXiv 2024

  2. [2]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. https://doi.org/10.1145/3442188.3445922 On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610–623, New York, NY, USA. Association for Computing...

    arXiv 2021

  3. [3]

    O'Reilly Media, Inc

    Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. " O'Reilly Media, Inc."

    2009

  4. [4]

    Mikaël Chelli, Jules Descamps, Vincent Lavoué, Christophe Trojani, Michel Azar, Marcel Deckert, Jean-Luc Raynier, Gilles Clowez, Pascal Boileau, and Caroline Ruetsch-Chelli. 2024. https://doi.org/10.2196/53164 Hallucination rates and reference accuracy of chatgpt and bard for systematic reviews: Comparative analysis . Journal of Medical Internet Research,...

    doi:10.2196/53164 2024

  5. [5]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    arXiv 2025

  6. [6]

    Li Du, Zhuoye Ding, Zhuoye Ding, Xuanjing Huang, and Zhongyu Wei. 2023. Quantifying and attributing the hallucination of large language models via association analysis. arXiv preprint arXiv:2309.05217

    arXiv 2023

  7. [7]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    arXiv 2024

  8. [8]

    Yue Guo, Zian Xu, and Yi Yang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.58 Is C hat GPT a financial expert? evaluating language models on financial natural language processing . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 815--821, Singapore. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.findings-emnlp.58 2023

Show all 41 references
  1. [9]

    Giwon Hong, Eunsol Jang, Jeonghyeon Kim, Jamin Shin, Soyoung Yoon, Sungdong Yoon, Jongwon Shin, and Minjoon Seo. 2024. The hallucinations leaderboard -- an open effort to measure hallucinations in large language models. arXiv preprint arXiv:2404.05904

    2024 arXiv

  2. [10]

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. https://openreview.net/forum?id=kIoBbc76Sy RULER : What s the real context size of your long-context language models? In First Conference on Language Modeling

    2024

  3. [11]

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023. https://arxiv.org/abs/2311.11944 Financebench: A new benchmark for financial question answering . arXiv preprint arXiv:2311.11944

    2023 arXiv

  4. [12]

    Ziwei Ji, Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, and Kai Chen. 2024. https://doi.org/10.18653/v1/2024.acl-long.442 ANAH : Analytical annotation of hallucinations in large language models . In Proceedings of the 62nd Annual Meeting of the Association for Computational ...

    2024 doi

  5. [13]

    Che Jiang, Biqing Qi, Xiangyu Hong, and Dayuan Fu. 2024. On large language models' hallucination with regard to known facts. arXiv preprint arXiv:2403.20009

    2024 arXiv

  6. [14]

    Lee and Katrina A

    John D. Lee and Katrina A. See. 2004. https://doi.org/10.1518/hfes.46.1.50_30392 Trust in automation: Designing for appropriate reliance . Human Factors, 46(1):50--80

    2004 doi

  7. [15]

    Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2024. https://doi.org/10.18653/v1/2024.acl-long.586 The dawn after the dark: An empirical study on factuality hallucination in large language models . In Proceedings of the 62nd Annual Mee...

    2024 doi

  8. [16]

    Yinheng Li, Shaofei Wang, Han Ding, and Hang Chen. 2023. https://doi.org/10.1145/3604237.3626869 Large language models in finance: A survey . In Proceedings of the Fourth ACM International Conference on AI in Finance, ICAIF '23, page 374–382, New York, NY, USA. Association for...

    2023

  9. [17]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Aco...

    2023

  10. [18]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024 a . https://doi.org/10.1162/tacl_a_00638 Lost in the middle: How language models use long contexts . Transactions of the Association for Computational Linguistics,...

    2024 doi

  11. [19]

    Shu Liu, Shangqing Zhao, Chenghao Jia, Xinlin Zhuang, Zhaoguang Long, Jie Zhou, Aimin Zhou, Man Lan, Qingquan Wu, and Chong Yang. 2024 b . https://arxiv.org/abs/2401.02982 Findabench: Benchmarking financial data analysis ability of large language models . arXiv preprint arXiv:...

    2024 arXiv

  12. [20]

    Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. https://doi.org/10.18653/v1/2022.acl-long.556 Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity . In Proceedings of the 60th Annual Meeting of th...

    2022 doi

  13. [21]

    Carsten Maple, Lukasz Szpruch, Gregory Epiphaniou, Kalina Staykova, Simran Singh, William Penwarden, Yisi Wen, Zijian Wang, Jagdish Hariharan, and Pavle Avramovic. 2023. https://arxiv.org/abs/2308.16538 The ai revolution: Opportunities and challenges for the finance sector . a...

    2023 arXiv

  14. [22]

    Filip Ginter Mathieu Timmerman. 2023. https://github.com/TurkuNLP/ocr_errors_simulator Ocr errors simulator

    2023

  15. [23]

    OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennet...

    2024 arXiv

  16. [24]

    OpenAI. 2024. https://openai.com/index/hello-gpt-4o Hello gpt-4o

    2024

  17. [25]

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. 2024. https://openreview.net/forum?id=RIu5lyNXjT Quantifying language models' sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting . In The Twelfth International...

    2024

  18. [26]

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, Soroosh Mariooryad, Yifan Ding, et al. 2024. https://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millio...

    2024 arXiv

  19. [27]

    The Alan Turing Institute . 2023. https://www.turing.ac.uk/news/publications/impact-large-language-models-finance-towards-trustworthy-adoption The impact of large language models in finance: Towards trustworthy adoption . Technical report, The Alan Turing Institute. Accessed o...

    2023

  20. [28]

    United States Bankruptcy Court for the Southern District of Florida . 2020. Signatures and document retention. https://www.flsb.uscourts.gov/local-rule/signatures-and-document-retention. Accessed on October 10, 2024

    2020

  21. [29]

    Simon Valentin, Jinmiao Fu, Gianluca Detommaso, Shaoyuan Xu, Giovanni Zappella, and Bryan Wang. 2024. Cost-effective hallucination detection for llms. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD '24, New York, NY, USA. Associati...

    2024

  22. [30]

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024. https://arxiv.org/abs/2410.01257 Helpsteer2-preference: Complementing ratings with preferences . Preprint, arXiv:2410.01257

    2024 arXiv

  23. [31]

    Qianqian Xie, Weiguang Han, Zhengyu Chen, Ruoyu Xiang, Xiao Zhang, Yueru He, Mengxi Xiao, Dong Li, Yongfu Dai, Duanyu Feng, Yijing Xu, Haoqiang Kang, Ziyan Kuang, Chenhan Yuan, Kailai Yang, Zheheng Luo, Tianlin Zhang, Zhiwei Liu, Guojun Xiong, Zhiyang Deng, Yuechen Jiang, Zhiy...

    2024 arXiv

  24. [32]

    Qianqian Xie, Weiguang Han, Xiao Zhang, Yanzhao Lai, Min Peng, Alejandro Lopez-Lira, and Jimin Huang. 2023. https://openreview.net/forum?id=vTrRq6vCQH PIXIU : A comprehensive benchmark, instruction dataset and large language model for finance . In Thirty-seventh Conference on ...

    2023

  25. [33]

    Jingfeng Xu, Yikang Shen, Yanan Ou, Xiang Liang, Xing Xie, and Meng Jiang. 2024 a . Measuring and reducing llm hallucination without gold-standard answers. arXiv preprint arXiv:2402.10412

    2024 arXiv

  26. [34]

    Ziyue Xu, Peilin Zhou, Xinyu Shi, Jiageng Wu, Yikang Jiang, Bin Ke, and Jie Yang. 2024 b . https://arxiv.org/abs/2406.12009 Fintruthqa: A benchmark dataset for evaluating the quality of financial information disclosure . Preprint, arXiv:2406.12009

    2024 arXiv

  27. [35]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  28. [36]

    Ziqi Yin, Hao Wang, Kaito Horio, Daisuke Kawahara, and Satoshi Sekine. 2024. https://arxiv.org/abs/2402.14531 Should we respect llms? a cross-lingual study on the influence of prompt politeness on llm performance . Preprint, arXiv:2402.14531

    2024 arXiv

  29. [37]

    Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. 2024. https://arxiv.org/abs/2409.02897 Longcite: Enabling llms to generate fine-grained citations in long-context qa . Preprint, arXiv:2409.02897

    2024 arXiv

  30. [38]

    Huaqin Zhao, Zhengliang Liu, Zihao Wu, Yiwei Li, Tianze Yang, Peng Shu, Shaochen Xu, Haixing Dai, Lin Zhao, Gengchen Mai, Ninghao Liu, and Tianming Liu. 2024. https://arxiv.org/abs/2401.11641 Revolutionizing finance with llms: An overview of applications and insights . arXiv p...

    2024

  31. [39]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://openreview.net/forum?id=uccHPGDlao Judging LLM -as-a-judge with MT -bench and chatbot ...

    2023

  32. [40]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  33. [41]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.