Pith. sign in

REVIEW 4 major objections 5 minor 21 references

Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read To compete in multilingual financial QA, route each language to its own model and score options directly, not by text generation.

desk verdict Useful competition write-up with a genuinely interesting Greek routing result, but the evaluation split contamination makes the headline numbers unreliable until the authors show deduplication. read the letter →

arxiv 2607.22841 v1 pith:Y3CB2HPD submitted 2026-07-24 cs.IR cs.AIcs.CL

Justice Ayela , Kabir Sahni This is my paper
classification cs.IRcs.AIcs.CL
keywords multilingualquestionansweringfinancialexamQAretrieval-augmentedgenerationdirectoptionscoringlanguageroutingmodelselectionreciprocalrankfusionchain-of-thought
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that effective multilingual financial question answering requires language-aware design at every pipeline stage, not a single uniform LLM. It reports large, systematic language-asymmetric model differences: a newer 14B model beats its predecessor on Arabic, Chinese, and Hindi while losing 22.9 points on English, and a smaller 8B model wins Greek by 19.5 points. It also shows that chain-of-thought prompting collapses Greek accuracy from 90.7% to 20.9%, while direct option-letter scoring (RADS) ties or beats generation everywhere and eliminates parse failures. The authors conclude that per-language model routing and deliberate scoring strategy selection are correctness requirements, and that the remaining gap to frontier API models on Hindi is base-model capacity, not retrieval quality. This matters because deploying a single 'best' model across languages silently forfeits tens of accuracy points.

What carries the argument

The load-bearing mechanism is the language-routed hybrid retrieval and direct-scoring pipeline. A language detector picks a per-language vector-index shard, or a global cross-lingual index when native exemplars are scarce, and low-resource languages fuse per-language and global rankings via weighted reciprocal rank fusion (a formula summing 1/(k+rank) per retriever with trust weights). The top-5 retrieved exemplars are formatted into a prompt, and the answer is the argmax of the model's next-token log-probability over the question's valid option letters (RADS). Critical details: the underlying model must have its 'thinking mode' disabled so the answer letter remains at the next-token positio

What would settle it

Run the English comparison on a larger held-out set (e.g., 200+ questions) with retrieval built only from training splits, and check for any overlap between evaluation queries and retained exemplars; if the newer model does not regress by ~23 points or if gold-bearing exemplars are retrieved, the routing claim lacks support. Similarly, test Greek on a larger set to see whether the smaller model's ~20-point lead holds.

Watch

Extended reading notes

Core claim

The paper's central claim is that in multilingual financial exam QA, no single LLM is best across languages and the differences are large and reproducible. Under identical retrieval and scoring, a 14-billion-parameter model improves over its predecessor on Arabic (+2.0), Chinese (+4.2), and Hindi (+6.0) but regresses on English by 22.9 points; a smaller 8-billion-parameter model beats both larger models on Greek by 19.5 points. The authors also find that chain-of-thought with self-consistency is sharply language- and task-asymmetric: it degrades Greek classification 90.7%→20.9% and Hindi by −19 points while marginally helping Chinese. They propose a language-routed pipeline: per-language ret

Load-bearing premise

That the reported measurements are clean and representative: evaluation questions' gold answers are absent from the 30,209-exemplar knowledge base, and small per-language splits (35 and 43 questions) are enough to support the headline claims.

Editorial extensions

If this is right

  • Any production multilingual QA system should evaluate per-language model choice; a single 'best' model on aggregate can be disastrously wrong for a specific language.
  • Direct option scoring should be the default for multiple-choice QA because it eliminates parse failures and is deterministic; generation-based evaluation introduces measurement bias especially at small test sizes.
  • Chain-of-thought prompting is not universally beneficial: it helps on quantitative reasoning but can wipe out accuracy on classification-style tasks; strategy selection should be task-type-aware.
  • The near-zero gain from adding 5,982 native Hindi exemplars versus cross-lingual proxies suggests that retrieval infrastructure is not the bottleneck for low-resource languages; model capacity is, so future work should focus on model adaptation.
  • Benchmark reporting should separate open-weight from API-backed systems to make comparisons fair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension: build an automated language+task-type router that selects both retrieval index and reasoning strategy (RADS vs CoT) based on predicted category (classification vs derivation); the Greek collapse predicts this will recover more than 60 points on classification tasks.
  • The paper's English regression claim rests on 35 questions; we would not be surprised if that effect shrinks on a larger sample. A re-run with 200+ held-out English questions is the cleanest check.
  • The inclusion of public test splits in the knowledge base (e.g., the Arabic test split inside the KB) means the reported accuracies could be inflated if any evaluation query has its gold answer among the retrieved exemplars; checking for KB/query overlap is a necessary sanity check before relying on the numbers.
  • If the language-asymmetry result generalizes, model evaluation practices should report per-language results rather than macro-averages, because averaging masks these systematic shifts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents DS@GT's submission to FinMMEval 2026 Task 1, a multilingual financial QA system. The pipeline detects the query language, retrieves top-5 exemplars from a 30,209-entry multilingual knowledge base using BGE-M3 embeddings and FAISS indices, fuses per-language and global indices via weighted Reciprocal Rank Fusion for low-resource languages, and answers multiple-choice questions by reading next-token log-probabilities over candidate option letters (RADS). Model selection is language-routed (Eq. 4). The paper's main empirical claims are: (i) RADS matches or beats few-shot generation and eliminates parse failures; (ii) chain-of-thought with self-consistency sharply degrades Greek and (claimed) Hindi accuracy, while slightly helping Chinese; (iii) no single model is best across languages—Qwen3-14B regresses on English by 22.9 pp relative to Qwen2.5-14B, and Llama-3.1-8B beats Qwen variants on Greek by 19.5 pp; (iv) adding native Hindi exemplars yields only +1 pp, implying base-model capacity rather than retrieval quality is the bottleneck. The authors also report official test-set results showing a mid-pack ranking across four languages.

Significance. If the dev-set measurements are clean, the paper's language-asymmetric findings are interesting and practically relevant. The system is reproducible (code link provided), the RADS method is simple and parse-free, and the paper makes falsifiable predictions about model routing and reasoning strategies. The explicit documentation of the Qwen3 thinking-mode interaction with RADS is a useful practical insight. However, the central claims rest on internal validation splits that appear to be included in the retrieval KB (a potential leakage path), and several key results are based on very small samples or incomplete runs. The official leaderboard results are clean but only validate the end-to-end system's mid-pack ranking; they do not independently support the routing/CoT claims, which were not run on the official test set. The contribution is more of an empirical systems study than a new method, and its value depends on the integrity of the ablation measurements.

major comments (4)
  1. [§3.2, Table 1, Tables 3–5] The central empirical claims in Tables 3–5 are vulnerable to knowledge-base leakage. §3.2 states that 'the original split names on the dataset cards do not restrict usage, and they were reorganized as needed during training as per policy,' and Table 1 places public test/val splits inside the 30,209-entry KB: BhashaBench-Finance English test (13,451) and Hindi test (5,982), CFinBench single_choice/val (8,877), SahmBench Arabic test (183+167), and Greek Plutus-MultiFin val (43). The retriever returns top-5 exemplars from this KB. If a query in the internal validation splits (e.g., the Greek n=43 set used in §4.2 and §4.3) is also present in the KB, the gold answer appears in the prompt as an exemplar, and RADS can copy that letter. This would inflate accuracy and distort relative solver/model comparisons. Please exclude all evaluation splits from the KB, or provide an exact and near-duplic
  2. [§4.3, Table 5; §5] Headline model-routing and CoT findings are based on very small samples with no uncertainty quantification. The English regression is n=35 (80.0% vs 57.1% is 28 vs 20 correct out of 35); the Greek routing comparison is n=43 (Llama 90.7% vs Qwen3 71.2% is 39 vs 31 correct). The paper calls these effects 'structural, not noise' (§5) and 'significantly degrading' (§4.2, Table 4), but no confidence interval, bootstrap, or significance test is reported. For n=35, a 22.9 pp absolute difference is about 8 questions; for n=43, the Greek CoT collapse is 30 questions, which is large but still needs a test, especially given the leakage concern. Please report exact counts and a statistical test (e.g., Fisher's exact test or bootstrap CIs), or soften the strong causal language.
  3. [§4.2, Table 4 footnote, §4.2 text] The Chinese CoT+SC result is not a completed measurement. The footnote to Table 4 says Solver D+SC was discontinued at ≈63% completion (5,580/8,869 records) and the reported value is a rolling estimate at stop time. Nevertheless, §4.2 concludes: 'The only positive signal is on Chinese, where preliminary D+SC tracking reached ∼66% — comparable to Qwen3 Solver B at 66.6% but ∼5× slower per query.' This incomplete run is used as evidence for the language-asymmetry claim. Either complete the run or remove this result from the comparative analysis and conclusions; a rolling estimate on an early prefix is not a valid accuracy on the full set.
  4. [§4.2, §7] The claimed Hindi penalty from CoT+SC is not supported by any data in the paper. The abstract and conclusions state that self-consistency decoding degrades Hindi by −19 pp, and §4.2 says 'CoT+SC is language-asymmetric: Solver D+SC degrades Greek significantly ... and Hindi (−19 pp).' However, Table 4 does not include a Hindi row, and no other table or text reports the Hindi Solver D accuracy or the baseline from which −19 pp is computed. Please add the Hindi Solver D result (with n, base model, and the comparison accuracy) or remove the claim from the abstract, §4.2, and §7.
minor comments (5)
  1. [Abstract, Table 7] The abstract says the benchmark spans English, Spanish, Greek, Chinese, and Hindi, but Table 7 reports official results only for Arabic, Chinese, English, and Hindi. Clarify the status of Spanish and Greek in the official test set.
  2. [§4.1] The text says 'Figure 2 explains the adaptive multi-scope routing results comprehensively,' but the figure is not included in the provided text. Either include the figure or describe the routing results in prose.
  3. [Table 4 vs Table 1] Table 4 lists Chinese N=8,869 while Table 1 shows CFinBench single_choice/val as 8,877. Explain the discrepancy (e.g., exclusion of malformed items or different filtering).
  4. [§3.5] The statement 'with thinking enabled, RADS accuracy on Arabic dropped from 71.7% to near-chance levels' gives no exact near-chance number. Provide the measured value or remove the quantitative-sounding qualifier.
  5. [§3.4] The RRF damping constant k=60 and trust weights w=2.0/1.0 are presented as chosen values; the paper does not describe how they were selected or whether performance is sensitive to them. A sentence on this would help.

Circularity Check

3 steps flagged · score 3.0 of 10

No derivation-to-fit circularity; main concern is measurement integrity, not circularity. Minor circularity in tuned components (RRF weights, routing) being evaluated on the same dev splits used to tune them.

  1. fitted input called prediction [Section 3.4 (Equation 2) and Section 4.1 (Table 3)]
    "we use w=1.0 for the global cross-lingual index and w=2.0 for per-language indices) ... and k=60 is the RRF damping constant ... Dev-leaderboard accuracy under KB v1 vs. KB v2. All language-specific scores hold constant or improve under v2; no regressions are observed."

    The RRF trust weights (w_per_language=2.0, w_global=1.0) and damping constant k=60 are introduced as fixed design choices without a grid search, and the weighted variant is described as 'well-suited' to favoring native-language indices. The system's superiority over a uniform retriever is then reported on the same internal dev splits used to make that configuration choice. This is a mild form of tuning-on-the-test-set: because the weights were selected against these splits, the reported dev accuracies for the fused configuration are partly optimized rather than independently predicted. However, the weights are simple and the paper does not present them as a fitted prediction, so this is a moderate concern rather than a full circular reduction.

  2. other [Section 3.2, Table 1, and Section 3.4 (Language Detection)]
    "The knowledge base aggregates the datasets that have been made publicly available through the FinMMEval Hugging Face collection; the original split names on the dataset cards do not restrict usage, and they were reorganized as needed during training as per policy. ... Table 1: BhashaBench-Finance English Test 13,451; Greek Plutus-MultiFin Val 43; CFinBench single_choice/val 8,877; SahmBench Arabic test 183+167."

    The reported dev accuracies in Tables 3-5 are computed on internal validation splits whose exact items are listed in Table 1 as part of the retrieval KB. Equation 3's direct option scoring can copy the gold letter from a retrieved exemplar if a query or near-duplicate is itself in the KB. This does not make the method circular in the derivation sense, but it can systematically inflate the reported accuracies on the same splits used to justify the paper's central claims (routing necessity, CoT collapse). The paper provides no deduplication or exact-match audit, so the headline numbers are not independently trustworthy. This is a measurement-integrity caveat, not a definitional circularity.

1 more flagged steps
  1. fitted input called prediction [Section 4.3 (Equation 4) and Table 5]
    "The underlying LLM θ is selected per query language ℓ across four model families ... on language-specific dev sets. The final routing is: θ(ℓ)= Qwen3-14B for {ar,zh,hi}, Qwen2.5-14B for en, Llama-3.1-8B for el. ... Per-language model ablation under RADS."

    Equation 4 is literally the argmax of Table 5's per-language dev accuracies: the routing is fitted to those same dev splits. The paper then uses those splits in Section 4.3 to assert that 'model routing is not optional' and that uniform model selection 'would have erased over 20 accuracy points on English and a comparable margin on Greek.' The gain from routing is therefore not a predicted generalization; it is the training-score of the fitted routing operator on the data used to fit it. The official test results (Table 7) are independent, but the paper never runs the routing-vs-uniform comparison on that test set, so the central routing claim is not independently validated.

full rationale

This is an empirical systems paper, not a mathematical derivation: there is no free-parameter or analytical chain that secretly encodes its own answer. The RADS scoring (Eq. 3), RRF fusion (Eq. 2), and the KB construction are all transparently described. The most serious issues are not circularity but data hygiene: Table 1 places public test/val splits inside the retrieval KB (BhashaBench English/Hindi test, CFinBench val, SahmBench Arabic test, Plutus-MultiFin Greek val), and Section 3.2 explicitly says split names 'do not restrict usage'. If evaluation queries or near-duplicates are in the KB, retrieved exemplars can carry the gold answer and inflate Tables 3-5. That would bias the headline claims (Qwen3's -22.9pp English regression, the Greek CoT collapse, and the routing necessity), but it is a measurement-integrity problem, not a definitional reduction. The one genuinely circular element is the limited-tuning-on-dev issue: the RRF weights (w=2.0/1.0, k=60) and especially the language->model routing (Eq. 4) are selected on the very dev splits whose accuracy gains are then reported as evidence that routing is necessary. That is closer to reporting a training score than an independent prediction, and the paper never reruns the routing-vs-uniform comparison on the clean official test set (Table 7). A score of 3 reflects these moderate concerns; the central architecture and scoring method themselves are not circular.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions (cross-lingual embedding quality, trustworthy gold exemplars, the next-token-position premise behind Eq. 3, and held-out disjointness) plus a set of hand-chosen constants (RRF weights, k=60, top-k=5, τ=20, CoT N/T) and a dev-fitted model routing. None of the constants is ablated, and the routing is selected on the same dev sets used for the headline numbers. No new entities are postulated: 'RADS' is a new label for MMLU-style log-prob scoring over retrieved exemplars, not a new scientific entity.

free parameters (7)
  • RRF trust weights = w_language=2.0, w_global=1.0
    Hand-chosen in §3.4 to favor native-language shards over the global index; no tuning or sensitivity analysis reported.
  • RRF damping constant k = 60
    Standard RRF default (ref [1]) used without adaptation; affects fused scores in low-resource settings.
  • retrieval top-k = 5
    Fixed in §3.4 and Appendix A; no ablation of k.
  • per-language shard coverage threshold τ = 20
    Hand-set threshold in §3.4 deciding dedicated vs. fused vs. global search; Greek (171) and Spanish (230) pass it, but no sensitivity analysis is given.
  • language→model routing (Eq. 4) = Qwen3-14B for {ar,zh,hi}; Qwen2.5-14B for en; Llama-3.1-8B for el
    Selected on the dev splits used to report Table 5; the English leg rests on a 35-question comparison (80.0% vs 57.1%).
  • CoT+SC decoding settings (Solver D) = N=5, T=0.7
    Hand-selected in §4.2; the Greek 90.7%→20.9% and claimed Hindi −19 pp results are reported only under these settings.
  • KB composition (v1/v2 dataset selection) = 6,996 → 30,209 exemplars
    Design choice behind Table 3's +0.67 pp average gain; split provenance is contradictory (§3.2/Table 1 label some entries 'test', §4.1 calls the same corpora dev/val).
assumptions (4)
  • domain assumption BGE-M3 provides a shared 1024-d embedding space in which cross-lingual similarity reliably retrieves useful financial exemplars
    Invoked in §2.2/§3.1 as the foundation of zero-shot cross-lingual retrieval; no comparison against alternative embedders or evidence of embedding quality on financial terminology.
  • domain assumption Gold answers in the public exemplar KB are correct and safe to inject as solved examples
    §3.2 builds the KB from public answers; an incorrect top-5 exemplar answer actively misleads the argmax scorer (Eq. 3).
  • domain assumption The answer letter appears as a single next token after the prompt, making Eq. 3's argmax valid
    Configuration-dependent: §3.5 shows Qwen3 thinking-mode tokens break it (Arabic → near-chance), and letter tokens must be single token-IDs in each model's tokenizer; neither is verified across models.
  • domain assumption Evaluated dev/test queries are disjoint from KB exemplars
    All accuracy numbers are meaningful only if this holds; §3.2/Table 1 aggregate public test splits into the KB and §4.1 describes the same data as dev/val, leaving the assumption unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval." pith.science (2026). https://pith.science/paper/Y3CB2HPD

@misc{pith2026260722841,
  author       = {Pith},
  title        = {Pith review of: Language-Routed RAG and Direct Option Scoring for Multilingual Financial QA: DS@GT at FinMMEval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3CB2HPD}},
  note         = {Machine review of arXiv:2607.22841}
}
read the original abstract

We present DS@GT's submission to FinMMEval 2026 Task 1, a multilingual financial exam question answering benchmark spanning English, Spanish, Greek, Chinese, and Hindi. Financial certification exams such as the CFA, EFPA, and CPA demand structured domain reasoning that standard NLP benchmarks do not capture, and this challenge compounds across languages where retrieval and representation infrastructure is underdeveloped. We build a retrieval-augmented pipeline on LangGraph that detects query language and retrieves semantically relevant exemplars from a 30,209-entry multilingual knowledge base using BGE-M3 embeddings and FAISS indexing. The system then scores answers via Retrieval-Augmented Direct Scoring (RADS), reading next-token log-probabilities over candidate option letters rather than generating free-form output. For low-resource languages, we fuse per-language and cross-lingual retrieval indices using weighted Reciprocal Rank Fusion. Model selection is language-routed: Qwen3-14B for Arabic, Chinese, and Hindi; Qwen2.5-14B for English; and Llama-3.1-8B for Greek, a routing derived from empirical ablations that reveal substantial language-asymmetric performance gaps. Notably, chain-of-thought prompting significantly degrades Greek accuracy (90.7% to 20.9%), and enabling Qwen3's default thinking mode collapses Arabic RADS performance to near-chance levels. Our results indicate that effective multilingual financial reasoning requires language-aware retrieval, model routing, and deliberate scoring strategy selection.

Figures

Figures reproduced from arXiv: 2607.22841 by the authors.

Figure 1
Figure 1. The three-stage multilingual hybrid retrieval and multiple-choice reasoning architecture We treat this as a retrieval-augmented reasoning problem. Our system, built on a LangGraph orchestration pipeline, routes each question through language detection and concept extraction, retrieves semantically relevant exemplars from a multilingual FAISS index built with BGE-M3 embeddings [6], fuses per-language and cross-lingua… view at source ↗
Figure 2
Figure 2. Adaptive Multi-Scope Routing Results [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Per-language accuracy of the four solver families under Qwen2.5-14B (except Greek, which uses Llama-3.1-8B). Solver B (RADS) ties or exceeds Solver A on every language except Arabic with Qwen2.5; the Arabic gap is closed by upgrading to Qwen3-14B (see [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 8 linked inside Pith

  1. [1]

    G. V. Cormack, C. L. A. Clarke, S. Buettcher, Reciprocal rank fusion outperforms condorcet and individual rank learning methods, in: Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, Association for Computing Machinery, 2009, pp. 758–759. doi:10.1145/1571941.1572114

    arXiv 2009

  2. [2]

    https://qwenlm.github.io/blog/qwen3/

    Qwen Team, Qwen3 technical report, 2025. https://qwenlm.github.io/blog/qwen3/

    2025

  3. [3]

    Qwen Team, Qwen2.5 technical report, arXiv preprint arXiv:2412.15115 (2024)

    arXiv 2024

  4. [4]

    Grattafiori, et al., The Llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024)

    A. Grattafiori, et al., The Llama 3 herd of models, arXiv preprint arXiv:2407.21783 (2024)

    arXiv 2024

  5. [5]

    Z. Xie, Y. Dai, R. Elbadry, V. Jani, X. Peng, L. Qian, G. Georgiev, D. Dimitrov, F. Zhang, J. Huang, J. Geng, Y. Chen, Y. Yuan, H. Wu, Y. Wang, I. Koychev, V. Stoyanov, M. Song, Y. Chen, X. Liu, P. Nakov, Overview of FinMMEval 2026: Multilingual and multimodal financial evaluation, in: Experimental IR Meets Multilinguality, Multimodality, and Interaction,...

    2026

  6. [6]

    J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, M3-embedding: Multi-linguality, multi- functionality, multi-granularity text embeddings through self-knowledge distillation, arXiv preprint arXiv:2402.03216 (2024)

    arXiv 2024

  7. [7]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, arXiv preprint arXiv:2005.11401 (2020) 1–22

    arXiv 2005

  8. [8]

    Izacard, E

    G. Izacard, E. Grave, Leveraging passage retrieval with generative models for open domain question answering, in: Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 2021, pp. 1545–1556

    2021

Show all 21 references
  1. [9]

    Douze, A

    M. Douze, A. Guzhva, C. Deng, J. Johnson, G. Szilvasy, P.-E. Mazaré, M. Lomeli, L. Hosseini, H. Jégou, The faiss library (2024).arXiv:2401.08281

    2024 arXiv

  2. [10]

    Johnson, M

    J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with GPUs, IEEE Transactions on Big Data 7 (2019) 535–547

    2019

  3. [11]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, J. Steinhardt, Measuring massive multitask language understanding, in: International Conference on Learning Representations (ICLR), 2021

    2021

  4. [12]

    Koncel-Kedziorski, M

    R. Koncel-Kedziorski, M. Krumdick, V. Lai, V. Reddy, C. Lovering, C. Tanner, Bizbench: A quantitative reasoning benchmark for business and finance, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 8309–8332

    2024

  5. [13]

    Q. Xie, W. Han, Z. Chen, R. Xiang, X. Zhang, Y. He, M. Xiao, D. Li, Y. Dai, D. Feng, Y. Xu, H. Kang, Z. Kuang, C. Yuan, K. Yang, Z. Luo, T. Zhang, Z. Liu, G. Xiong, Z. Deng, Y. Jiang, Z. Yao, H. Li, Y. Yu, G. Hu, J. Huang, X. Liu, A. Lopez-Lira, B. Wang, Y. Lai, H. Wang, M. Pe...

    2024

  6. [14]

    Q. Xie, W. Han, X. Zhang, Y. Lai, M. Peng, A. Lopez-Lira, J. Huang, Pixiu: A large language model, instruction data and evaluation benchmark for finance, arXiv preprint arXiv:2306.05443 (2023)

    2023 arXiv

  7. [15]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. ...

    2005 arXiv

  8. [16]

    L. Gao, J. Tow, B. Abbasi, S. Biderman, S. Black, A. DiPofi, C. Foster, L. Golding, J. Hsu, A. Le Noac’h, H. Li, K. McDonell, N. Muennighoff, C. Ociepa, J. Phang, L. Reynolds, H. Schoelkopf, A. Skowron, L. Sutawika, E. Tang, A. Thite, B. Wang, K. Wang, A. Zou, The language mod...

  9. [17]

    Angulo, V

    J. Angulo, V. Yeste, Agentic mcs: A multilingual clinical summarization framework, in: CLEF 2025 Working Notes, Madrid, Spain, 2025. Notebook for the MultiClinSum Lab at CLEF 2025

    2025

  10. [18]

    C. Guo, G. Pleiss, Y. Sun, K. Q. Weinberger, On calibration of modern neural networks, in: Proceedings of the 34th International Conference on Machine Learning, ICML ’17, 2017, pp. 1321–1330

    2017

  11. [19]

    Y. Nie, B. Yan, T. Guo, H. Liu, H. Wang, W. He, B. Zheng, W. Wang, Q. Li, W. Sun, Y. Wang, D. Tao, Cfinbench: A comprehensive chinese financial benchmark for large language models, arXiv preprint arXiv:2407.02301 (2024)

    2024 arXiv

  12. [20]

    URL: http://www

    PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www. pace.gatech.edu. A. Prompt for Retrieval-Augmented Direct Scoring System message You are an expert financial exam solver. You will be shown a set of solved example questions, followed by a n...

    2017

  13. [2024]

    doi:10.5281/zenodo.12608602

    URL: https://zenodo.org/records/12608602. doi:10.5281/zenodo.12608602

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.