Pith. sign in

REVIEW 4 major objections 7 minor 55 references

V-FiLLM: Verified Financial LLM Reasoning Benchmark

T0 review · 4 major / 7 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that financial question-answer benchmarks can be generated from executable computation trees with answers correct by construction, no annotation cost, and controlled difficulty, and that current open models degrade…

desk verdict A solid, useful benchmark generator for financial tabular reasoning, but the 'correct by construction' claim is stronger than the rendered questions can support until the ambiguity rate is measured. read the letter →

arxiv 2608.11047 v1 pith:SAT4HCF4 submitted 2026-08-11 cs.AI cs.CEcs.LG

Alicia Larsen , Victoire Laurent , Aulia Kharis Rakhamsari , Lara Turgut , Nino Antulov-Fantulin This is my paper · ORCID
classification cs.AIcs.CEcs.LG
keywords financialreasoningbenchmarkgenerationcomputationtreestabularquestionansweringchain-of-thoughtLoRAfine-tuningadversarialrobustnesssyntheticdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces V-FiLLM, a framework that generates financial question-answer benchmarks from executable computation trees grounded in synthetic spreadsheets. Because each question is rendered from a tree that is evaluated symbolically, the answer is correct by construction and no model or human annotator is involved in labeling; the authors claim this removes the generator's error rate and makes arbitrarily large benchmark generation possible. The framework exposes four independently controllable difficulty axes: computation depth, expression breadth, financial concept complexity, and context size. Using the benchmark, the authors find that open models' accuracy falls by up to 51 points as reasoning depth grows and by up to 47 points under adversarial numerical perturbations, and that LoRA fine-tuning on verified chain-of-thought traces lifts Qwen3-4B from 81.1% to 85.6% on held-out problems and by 5 points on FinQA. If the construction is sound, V-FiLLM offers a scalable, verifiable evaluation resource for financial reasoning over tables.

What carries the argument

The load-bearing object is the typed, executable computation tree: leaves are financial atoms (a spreadsheet cell with its concept, company, year, unit, and quantity type), internal nodes are operations such as addition, subtraction, multiplication, ratio, growth, minimum, maximum, and average, and a type-aware sampler rejects incompatible compositions. The tree is evaluated symbolically to produce the ground-truth answer and rendered bottom-up into natural-language questions, so the answer is guaranteed correct for the intended computation. Protected derived financial concepts carry hidden formula trees, and a rejection step prevents a raw arithmetic composition from being mistaken for a named concept. This tree structure is also what makes depth, breadth, value scaling, and multi-turn decomposition directly controllable.

What would settle it

Take a sample of generated questions and ask a blind second process, human annotators or an independent renderer, to reconstruct the expression tree from the question text alone; if any nontrivial fraction of reconstructions differs from the original tree, the correctness-by-construction guarantee fails for those items. A sharper version is to search the generator's output space for two distinct bound expression trees that render to identical question strings with different answers.

Watch

Extended reading notes

Core claim

The paper's central claim is that financial reasoning benchmarks can be generated deterministically rather than collected from human annotation: synthetic financial spreadsheets are decomposed into typed atoms, composed into typed binary expression trees, and rendered into natural-language questions, with the ground truth obtained by symbolically evaluating the tree. This makes answers correct by construction, at arbitrary scale, without inheriting a generator's error rate. The same tree structure gives explicit control over reasoning depth, expression breadth through balanced trees, the presence of named derived financial concepts such as gross profit and operating income, and context size. The empirical payoff is that accuracy degrades sharply with depth and with table perturbations, with unit and scale shifts the most destructive on realistic 10-Q filings, and that multi-turn decomposition and verified-trace LoRA fine-tuning both recover a substantial share of the lost performance.

Load-bearing premise

The load-bearing premise is that the bottom-up renderer turns each computation tree into a natural-language question whose wording unambiguously picks out that computation; if a question can be read as a different calculation, the 'correct by construction' answer is no longer the answer to the question being posed.

Editorial extensions

If this is right

  • Because no human labels or model outputs enter answer generation, the benchmark can be scaled to arbitrary depth and breadth distributions at near-zero marginal cost, enabling fine-grained difficulty sweeps.
  • Per-depth accuracy curves provide a direct diagnostic of where compositional reasoning fails, isolating intermediate arithmetic errors from planning errors.
  • The multi-turn version, which converts each internal node into a conversational turn, exposes intermediate failures that single-turn accuracy hides and benefits the weakest models most.
  • The adversarial perturbation suite, with ground truth invariant by construction, cleanly attributes accuracy drops to model fragility rather than label shift.
  • LoRA fine-tuning on verified chain-of-thought traces gives a small but consistent accuracy gain and transfers to FinQA, indicating that verified synthetic traces are useful training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An extension the paper leaves implicit is that the same tree-rendering pipeline transfers to other structured domains, such as medical records or scientific tables, by swapping the financial concept library; the correctness-by-construction guarantee would carry over.
  • The unit and scale fragility result implies models may be matching digit patterns rather than reading unit semantics; the benchmark's value-scaling axis already provides a direct way to test this.
  • Because every tree node carries ground truth, the verified traces could also support process-supervision or verifier-guided decoding; the paper only demonstrates the LoRA use.
  • A back-rendering check, recovering the expression tree from the question text and comparing it with the original tree, would quantify the ambiguity that the paper's limitations section concedes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces V-FiLLM, a framework that generates financial question-answering items by sampling typed expression trees whose leaves are spreadsheet atoms from synthetic financial data, evaluating the trees symbolically to obtain ground-truth answers, and rendering the trees into natural-language questions. The authors claim these items are correct by construction, scale without annotation cost, and avoid inheriting a generator's error rate, while exposing four independently controllable difficulty axes: computation depth, expression breadth, financial concept complexity, and context size. They evaluate several open-weight models on single-turn and multi-turn versions, report that accuracy declines with depth and under adversarial table perturbations---most dramatically under unit/scale shifts---and find that LoRA fine-tuning on verified chain-of-thought traces improves accuracy on held-out generated questions and yields a small gain on FinQA.

Significance. If the construction is fully validated, V-FiLLM would be a useful evaluation resource for compositional financial reasoning: deterministic symbolic grounding, controllable depth, and machine-checked answers are genuine strengths, and the authors provide code, stored expression-tree metadata, verified chain-of-thought traces, and an external FinQA check. The benchmark also makes falsifiable predictions about model degradation with depth and robustness to input perturbations, which is valuable. The contribution is currently tempered by two load-bearing issues: the correctness guarantee attaches to the expression tree rather than the actual question string, and the text repeatedly describes synthetic data as real filings. Once these are addressed, the resource could be a solid addition to the field, but as written the central claims need revision.

major comments (4)
  1. [Abstract, Section 3.1, Section 5.1, Tables 2 and 4] The 'correct by construction' claim holds for the expression tree, not for the rendered natural-language question. The bottom-up renderer can produce attachment and scope ambiguities: the depth-2 example in Figure 2, 'the difference between total assets scaled by income tax and cost of goods sold scaled by income tax', can be read as total assets minus (COGS times income tax) if the second 'scaled by' phrase attaches only to the second noun phrase, whereas the intended tree is (total assets times income tax) minus (COGS times income tax). Because the stored answer is the tree's value, a model solving the alternative parse will be marked incorrect despite following the text. Section 7 concedes that the questions 'may still contain imperfections' and that steps 'may occasionally be translated into English in ways that are ambiguous or lack sufficient context', but no ambiguity rate, no examples, and no human or parser-based verification are provided. The Section 3.3 augmentation that injects 'minor typographical errors' while asserting the computation graph is preserved adds further risk: a substitution such as 'maximum' to 'minimum' or a changed year would alter the denoted computation while the stored answer stays fixed. Consequently, the abstract's claim of generating items 'without inheriting a generator's error rate' is not established; a deterministic renderer can have a systematic, unmeasured ambiguity rate. Please measure the rate at which independent readers or an independent parser recover the intended computation from the question string, or substantially soften the correctness guarantee.
  2. [Section 4.3, Tables 2-6] The paper repeatedly calls the evaluation data 'real'. The abstract says the benchmark is 'grounded in real tables', and Section 5.1 and Tables 2 and 4 report results 'over real 10-Q filings' and on 'real documents'. Section 3.1, however, states that both data sources are synthetic: the first 'mimics a typical 10-Q filing' and the second is a 'regularized financial sheet'; the values are generated 'to preserve internal coherence while remaining synthetic'. This is not merely a terminology issue, because Section 5.1 explains performance differences by saying 'real documents have messy layouts, footnotes, and mixed units', attributing to real filings properties that actually belong to the synthetic mimics. Please revise the abstract, Section 5, and the table captions to use 'synthetic 10-Q-style filings' or similar, or add genuine real-world filings to ground the external-validity claims.
  3. [Tables 2-4, Section 5] The abstract and Section 4.3 claim the framework 'exposes four independently controllable axes of difficulty'---depth, breadth, financial concept complexity, and context size. The experimental section, however, varies only depth systematically (Table 3). There are no reported experiments that manipulate expression breadth, the frequency or type of derived financial concepts, or the number of companies or rows in the prompt. Section 4.3 also describes 'value scaling' as a difficulty axis, but the unit/scale-shift row in Table 4 is an adversarial perturbation applied to question-irrelevant cells, not the controlled value-scaling manipulation described in that subsection. Either add evaluations for each claimed axis, or revise the claims to match the actually supported axes.
  4. The headline numbers are based on 250 mixed-depth questions in Table 2 and 100 questions per cell in Tables 3 and 4, with no confidence intervals, no repeated trials, and no significance tests. In this regime, reported differences of a few percentage points, such as 98.4% versus 98.0%, are within sampling noise, yet the text draws comparative conclusions such as 'Gemma-31B matches or outperforms both GPT-OSS-120B and Qwen3.7-Plus across all conditions' and attributes this to 'specialized training data and robust table parsing mechanisms'. The Section 5.3 hyperparameter sweep is also selected on the same held-out set of 90 problems, as Section 7 acknowledges; Table 5 should therefore be labeled exploratory. Please add confidence intervals or repeated sampling, and soften claims that outrun the statistical evidence.
minor comments (7)
  1. [Section 3.3] The sentence 'See 2 for an example of such trees' should read 'see Figure 2', and the expression '2i for depthi' needs proper mathematical formatting.
  2. [Section 4.1] The final paragraph of Section 3.2 ('Such perturbations enable ... practical financial applications') is repeated verbatim at the end of Section 3.3; remove the duplicate.
  3. [Section 4.2] The sentence beginning 'We convert T into a multi-turn dialogue under the following rules, we convert only trees...' is ungrammatical; rewrite to introduce the conversion rules cleanly.
  4. [Section 4.4] The citation to Mirzadeh et al. (2025) for 'drops of up to 65% from a single such clause' is out of place in a paragraph about tabular perturbations and should be removed or replaced with a directly relevant table-QA robustness citation.
  5. [Abstract] Section 4.4 says 'A rank of (r=16), a scaling factor (α=32), and a dropout rate of 0.05 were used', but Section 5.3 reports the best configuration as r=8, α=8, dropout 0.10; please reconcile these statements so the reported recipe is unambiguous.
  6. [Section 5.2] The abstract contains a typo in 'outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s)'; the stray ', s)' should be removed.
  7. The text uses 'pertubations' instead of 'perturbations', and the table captions inconsistently spell '10-Q Fillings' with different capitalization; please correct these throughout.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: V-FiLLM's answers are computed from the same expression trees that generate the questions, and the empirical claims are checked against an external dataset, with the only self-citations being contextual background.

full rationale

The generation pipeline is self-contained: leaves are spreadsheet atoms, internal nodes define arithmetic operations, and the ground-truth answer is obtained by symbolically evaluating the bound expression tree, not from any model output or fitted parameter. The rendered question is produced from the same tree by a bottom-up semantic renderer, so the answer is correct relative to the tree by construction; the paper's Section 7 explicitly concedes that the English rendering 'may occasionally be translated into English in ways that are ambiguous or lack sufficient context,' which is a validity risk for the question-answer pairs, but not a circular reduction of the claimed result to its inputs. The LoRA experiments select hyperparameters on the held-out set and report accuracy on that same set, but the paper discloses this in Section 7 ('Because the best-performing configuration was selected using this same constrained set'), and the FinQA evaluation (Table 6) provides an external, non-circular check of generalization. The self-citations (Antulov-Fantulin and Kolm 2023, 2026; Jazbec et al. 2021) appear only as introductory context and do not carry any load-bearing premise. No equation or fitted value is shown to be equivalent by construction to a predicted quantity, so there is no circularity to report.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on synthetic data-generation parameters and several domain assumptions about rendering fidelity, depth as difficulty, and small-sample evaluation. It does not introduce new physical or mathematical entities. The free parameters are reasonable for a synthetic benchmark, but they are not externally anchored, so the resulting difficulty and realism are only as good as the authors' choices.

free parameters (4)
  • Synthetic financial data sampling ranges
    Section 3.1: revenue is initialized from company-specific ranges, growth rates and balance-sheet ratios are sampled from plausible ranges. These distributions determine table realism and benchmark difficulty, but they are chosen by the authors with no external calibration.
  • Template depth bounds and derived-concept probability = depth 0 to 8; derived-concept probability unspecified
    Section 3.2: the sampler takes minimum and maximum depth and a probability of sampling named derived concepts. These parameters control the difficulty mix and are set by the authors for the reported experiments, not derived from any benchmark requirement.
  • Number of companies in the prompt = 6
    Section 3.1: the number of companies is adjustable and set to 6 to balance realistic report scale with LLM context length. This directly affects the context-size difficulty axis.
  • LoRA hyperparameters r, alpha, dropout = best: r=8, alpha=8, dropout=0.10
    Section 5.3 and Table 5: hyperparameters were selected on a held-out set of only 90 problems, and the improvement over baseline is 4 correct answers, so the selected configuration is a small-sample fitted quantity.
assumptions (4)
  • ad hoc to paper The natural-language renderer produces unambiguous questions that match the intended computation tree.
    Section 3.2 describes bottom-up rendering; Section 7 admits that questions "may still contain imperfections" and can be "ambiguous or lack sufficient context." The benchmark's correct-by-construction property depends on this rendering assumption.
  • domain assumption Tree depth is a valid proxy for the number of reasoning steps and for reasoning difficulty.
    Sections 3.2 and 4.3 use depth as the main difficulty axis, but no calibration is provided. The same depth can produce very different numerical complexity depending on the operators and cells involved.
  • domain assumption Synthetic spreadsheet values preserve the internal coherence needed for meaningful financial reasoning.
    Section 3.1 generates income-statement quantities as ratios of revenue and balance-sheet components as percentages of parent quantities. There is no validation that these synthetic tables are representative of real financial filings.
  • domain assumption Evaluation sets of 90 to 250 questions per cell are large enough to draw stable accuracy conclusions.
    Tables 3, 4, and 5 use 100 or fewer questions per cell, and Section 7 explicitly notes the constrained compute and small evaluation set. No confidence intervals are provided, so the reported differences may be noise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of V-FiLLM: Verified Financial LLM Reasoning Benchmark." pith.science (2026). https://pith.science/paper/SAT4HCF4

@misc{pith2026260811047,
  author       = {Pith},
  title        = {Pith review of: V-FiLLM: Verified Financial LLM Reasoning Benchmark},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SAT4HCF4}},
  note         = {Machine review of arXiv:2608.11047}
}
read the original abstract

While existing benchmarks have made substantial progress in evaluating LLMs across STEM domains, financial reasoning over structured data remains comparatively less explored. We introduce V-FiLLM, a framework that generates financial reasoning benchmarks from executable computation trees grounded in real tables, yielding items whose answers are correct by construction. Trees are evaluated symbolically to obtain ground truth and rendered into natural-language questions, removing any model from the labeling loop, so items can be generated at arbitrary scale without annotation cost and without inheriting a generator's error rate. V-FiLLM exposes four independently controllable axes of difficulty including computation depth, expression breadth, financial concept complexity, and context size. By evaluating on open-source models, we find that accuracy falls up to 51% as reasoning depth increases, and up to 47% points under adversarial numerical perturbations, highlighting remaining challenges in robust financial reasoning over tables. We further show that lightweight LoRA fine-tuning on verified chain-of-thought traces improves accuracy from 81.1% to 85.6% on held-out problems and outperforms the base model by 5% points on FinQA (Chen et al., 2022a), s), suggesting that targeted, low-cost adaptation is a promising direction for compositional reasoning in financial QA.

Figures

Figures reproduced from arXiv: 2608.11047 by the authors.

Figure 1
Figure 1. Benchmark generation pipeline. Synthetic spreadsheets are decomposed into typed atoms, composed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Examples of grounded computation trees and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

55 extracted references · 17 canonical work pages

  1. [1]

    Royal Society Open Science , author =

    On the impact of publicly available news and information transfer to financial markets , volume =. Royal Society Open Science , author =. 2021 , pages =

    work page 2021

  2. [2]

    Antulov-Fantulin, Nino and Kolm, Petter N , year =. Graph. The Journal of Portfolio Management , publisher =

    work page

  3. [3]

    Journal of Financial Data Science , author =

    Advances of. Journal of Financial Data Science , author =. 2023 , pages =

    work page 2023

  4. [4]

    doi:10.48550/arXiv.2410.05229 , abstract =

    Mirzadeh, Iman and Alizadeh, Keivan and Shahrokhi, Hooman and Tuzel, Oncel and Bengio, Samy and Farajtabar, Mehrdad , month = aug, year =. doi:10.48550/arXiv.2410.05229 , abstract =

    arXiv doi:10.48550/arxiv.2410.05229

  5. [5]

    and Schärli, Nathanael and Zhou, Denny , month = jul, year =

    Shi, Freda and Chen, Xinyun and Misra, Kanishka and Scales, Nathan and Dohan, David and Chi, Ed H. and Schärli, Nathanael and Zhou, Denny , month = jul, year =. Large. Proceedings of the 40th

    work page

  6. [6]

    Proceedings of the 2024

    Zhou, Wei and Mesgar, Mohsen and Adel, Heike and Friedrich, Annemarie , editor =. Proceedings of the 2024. 2024 , pages =. doi:10.18653/v1/2024.naacl-long.137 , abstract =

    doi:10.18653/v1/2024.naacl-long.137 2024

  7. [7]

    Proceedings of the 61st

    Zhao, Yilun and Zhao, Chen and Nan, Linyong and Qi, Zhenting and Zhang, Wenlin and Tang, Xiangru and Mi, Boyu and Radev, Dragomir , editor =. Proceedings of the 61st. 2023 , pages =. doi:10.18653/v1/2023.acl-long.334 , abstract =

    doi:10.18653/v1/2023.acl-long.334 2023

  8. [8]

    arXiv.org , author =

Show all 55 references
  1. [9]

    Least-to-

    Zhou, Denny and Schärli, Nathanael and Hou, Le and Wei, Jason and Scales, Nathan and Wang, Xuezhi and Schuurmans, Dale and Cui, Claire and Bousquet, Olivier and Le, Quoc and Chi, Ed , month = apr, year =. Least-to-. doi:10.48550/arXiv.2205.10625 , abstract =

    doi arXiv

  2. [10]

    arXiv.org , author =

    Quantifying. arXiv.org , author =

  3. [11]

    Sclar, Melanie and Choi, Yejin and Tsvetkov, Yulia and Suhr, Alane , year =

  4. [12]

    Ribeiro, Marco Tulio and Wu, Tongshuang and Guestrin, Carlos and Singh, Sameer , editor =. Beyond. Proceedings of the 58th. 2020 , pages =. doi:10.18653/v1/2020.acl-main.442 , abstract =

    2020 doi

  5. [13]

    State of

    Mizrahi, Moran and Kaplan, Guy and Malkin, Dan and Dror, Rotem and Shahaf, Dafna and Stanovsky, Gabriel , year =. State of. Transactions of the Association for Computational Linguistics , publisher =. doi:10.1162/tacl_a_00681 , abstract =

    doi

  6. [14]

    Lialin, Vladislav and Deshpande, Vijeta and Yao, Xiaowei and Rumshisky, Anna , month = nov, year =. Scaling. doi:10.48550/arXiv.2303.15647 , abstract =

    doi arXiv

  7. [15]

    doi:10.48550/arXiv.2304.01933 , abstract =

    Hu, Zhiqiang and Wang, Lei and Lan, Yihuai and Xu, Wanyu and Lim, Ee-Peng and Bing, Lidong and Xu, Xing and Poria, Soujanya and Lee, Roy Ka-Wei , month = oct, year =. doi:10.48550/arXiv.2304.01933 , abstract =

    doi arXiv

  8. [16]

    Parameter-

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and Laroussilhe, Quentin de and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , month = jun, year =. Parameter-. doi:10.48550/arXiv.1902.00751 , abstract =

    doi arXiv

  9. [17]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , month = oct, year =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , month = oct, year =. doi:10.48550/arXiv.2106.09685 , abstract =

    doi arXiv

  10. [18]

    and Sanyal, Soumya and Welleck, Sean and Ren, Xiang and Ettinger, Allyson and Harchaoui, Zaid and Choi, Yejin , month = oct, year =

    Dziri, Nouha and Lu, Ximing and Sclar, Melanie and Li, Xiang Lorraine and Jiang, Liwei and Lin, Bill Yuchen and West, Peter and Bhagavatula, Chandra and Bras, Ronan Le and Hwang, Jena D. and Sanyal, Soumya and Welleck, Sean and Ren, Xiang and Ettinger, Allyson and Harchaoui, Z...

    doi arXiv

  11. [19]

    Compositional

    Pasupat, Panupong and Liang, Percy , editor =. Compositional. Proceedings of the 53rd. 2015 , pages =. doi:10.3115/v1/P15-1142 , urldate =

    2015 doi

  12. [20]

    Findings of the

    Chen, Wenhu and Zha, Hanwen and Chen, Zhiyu and Xiong, Wenhan and Wang, Hong and Wang, William Yang , editor =. Findings of the. 2020 , pages =. doi:10.18653/v1/2020.findings-emnlp.91 , abstract =

    2020 doi

  13. [21]

    doi:10.48550/arXiv.2309.17167 , abstract =

    Zhu, Kaijie and Chen, Jiaao and Wang, Jindong and Gong, Neil Zhenqiang and Yang, Diyi and Xie, Xing , month = mar, year =. doi:10.48550/arXiv.2309.17167 , abstract =

    doi arXiv

  14. [22]

    doi:10.48550/arXiv.2311.06602 , abstract =

    Koncel-Kedziorski, Rik and Krumdick, Michael and Lai, Viet and Reddy, Varshini and Lovering, Charles and Tanner, Chris , month = mar, year =. doi:10.48550/arXiv.2311.06602 , abstract =

    doi arXiv

  15. [23]

    doi:10.48550/arXiv.2206.01347 , abstract =

    Zhao, Yilun and Li, Yunxiang and Li, Chenying and Zhang, Rui , month = jun, year =. doi:10.48550/arXiv.2206.01347 , abstract =

    doi arXiv

  16. [24]

    , month = nov, year =

    Rein, David and Hou, Betty Li and Stickland, Asa Cooper and Petty, Jackson and Pang, Richard Yuanzhe and Dirani, Julien and Michael, Julian and Bowman, Samuel R. , month = nov, year =. doi:10.48550/arXiv.2311.12022 , abstract =

    doi arXiv

  17. [25]

    Findings of the

    Li, Haonan and Zhang, Yixuan and Koto, Fajri and Yang, Yifei and Zhao, Hai and Gong, Yeyun and Duan, Nan and Baldwin, Timothy , editor =. Findings of the. 2024 , pages =. doi:10.18653/v1/2024.findings-acl.671 , abstract =

    2024 doi

  18. [26]

    doi:10.48550/arXiv.1909.02164 , abstract =

    Chen, Wenhu and Wang, Hongmin and Chen, Jianshu and Zhang, Yunkai and Wang, Hong and Li, Shiyang and Zhou, Xiyou and Wang, William Yang , month = jun, year =. doi:10.48550/arXiv.1909.02164 , abstract =

    1909 doi arXiv

  19. [27]

    doi:10.48550/arXiv.2509.25160 , abstract =

    Yuan, Fan and Yan, Yuchen and Jiang, Yifan and Zhao, Haoran and Feng, Tao and Chen, Jinyan and Lou, Yanwei and Zhang, Wenqi and Shen, Yongliang and Lu, Weiming and Xiao, Jun and Zhuang, Yueting , month = sep, year =. doi:10.48550/arXiv.2509.25160 , abstract =

    doi

  20. [28]

    doi:10.48550/arXiv.1903.00161 , abstract =

    Dua, Dheeru and Wang, Yizhong and Dasigi, Pradeep and Stanovsky, Gabriel and Singh, Sameer and Gardner, Matt , month = apr, year =. doi:10.48550/arXiv.1903.00161 , abstract =

    1903 doi arXiv

  21. [29]

    doi:10.48550/arXiv.2105.07624 , abstract =

    Zhu, Fengbin and Lei, Wenqiang and Huang, Youcheng and Wang, Chao and Zhang, Shuo and Lv, Jiancheng and Feng, Fuli and Chua, Tat-Seng , month = jun, year =. doi:10.48550/arXiv.2105.07624 , abstract =

    doi arXiv

  22. [30]

    Zico and Ravikumar, Pradeep , month = jun, year =

    Zhai, Runtian and Dan, Chen and Kolter, J. Zico and Ravikumar, Pradeep , month = jun, year =. doi:10.48550/arXiv.2106.06142 , abstract =

    doi arXiv

  23. [31]

    and Haghgoo, Behzad and Chen, Annie S

    Liu, Evan Z. and Haghgoo, Behzad and Chen, Annie S. and Raghunathan, Aditi and Koh, Pang Wei and Sagawa, Shiori and Liang, Percy and Finn, Chelsea , month = jul, year =. Just. Proceedings of the 38th

  24. [32]

    and Finn, Chelsea , month = jul, year =

    Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Ermon, Stefano and Manning, Christopher D. and Finn, Chelsea , month = jul, year =. Direct. doi:10.48550/arXiv.2305.18290 , abstract =

    doi arXiv

  25. [33]

    Evaluating

    Liu, Nelson and Zhang, Tianyi and Liang, Percy , editor =. Evaluating. Findings of the. 2023 , pages =. doi:10.18653/v1/2023.findings-emnlp.467 , abstract =

    2023 doi

  26. [34]

    Enabling

    Gao, Tianyu and Yen, Howard and Yu, Jiatong and Chen, Danqi , editor =. Enabling. Proceedings of the 2023. 2023 , pages =. doi:10.18653/v1/2023.emnlp-main.398 , abstract =

    2023 doi

  27. [35]

    Measuring

    Rashkin, Hannah and Nikolaev, Vitaly and Lamm, Matthew and Aroyo, Lora and Collins, Michael and Das, Dipanjan and Petrov, Slav and Tomar, Gaurav Singh and Turc, Iulia and Reitter, David , month = dec, year =. Measuring. Computational Linguistics , publisher =. doi:10.1162/coli...

    doi

  28. [36]

    Advances in Neural Information Processing Systems , author =

    Judging. Advances in Neural Information Processing Systems , author =. 2023 , pages =

    2023

  29. [37]

    doi:10.48550/arXiv.2605.27710 , abstract =

    Sadeghi, Shaghayegh and Khajavi, Khashayar and Adhikari, Rise and Tessier, Alexander , month = may, year =. doi:10.48550/arXiv.2605.27710 , abstract =

    doi arXiv

  30. [38]

    Evaluating

    Zeng, Zhiyuan and Yu, Jiatong and Gao, Tianyu and Meng, Yu and Goyal, Tanya and Chen, Danqi , month = apr, year =. Evaluating. doi:10.48550/arXiv.2310.07641 , abstract =

    doi arXiv

  31. [39]

    Raina, Vyas and Liusie, Adian and Gales, Mark , month = jul, year =. Is. doi:10.48550/arXiv.2402.14016 , abstract =

    doi arXiv

  32. [40]

    Wang, Victor and Zhang, Michael J. Q. and Choi, Eunsol , month = sep, year =. Improving. doi:10.48550/arXiv.2503.03064 , abstract =

    doi

  33. [41]

    Proceedings of the 63rd

    Li, Haitao and Chen, Junjie and Yang, Jingli and Ai, Qingyao and Jia, Wei and Liu, Youfeng and Lin, Kai and Wu, Yueyue and Yuan, Guozhi and Hu, Yiran and Wang, Wuyue and Liu, Yiqun and Huang, Minlie , editor =. Proceedings of the 63rd. 2025 , pages =. doi:10.18653/v1/2025.acl-...

    2025 doi

  34. [42]

    Dimino, Fabrizio and Sarmah, Bhaskarjit and Pasquali, Stefano , month = mar, year =. Risk-. doi:10.48550/arXiv.2603.10807 , abstract =

    doi

  35. [43]

    Sinhahajari, Soumitra and Majumder, Navonil and Poria, Soujanya , month = jun, year =. On the. doi:10.48550/arXiv.2606.12071 , abstract =

    doi arXiv

  36. [44]

    and Goel, Pranav and Stoehr, Niklas and Ash, Elliott and Hoyle, Alexander Miserlis , editor =

    Licht, Hauke and Sarkar, Rupak and Wu, Patrick Y. and Goel, Pranav and Stoehr, Niklas and Ash, Elliott and Hoyle, Alexander Miserlis , editor =. Measuring scalar constructs in social science with. Proceedings of the 2025. 2025 , pages =. doi:10.18653/v1/2025.emnlp-main.1635 , ...

    2025 doi

  37. [45]

    Cited but

    Onweller, Hailey and Lumer, Elias and Huber, Austin and Ramchandani, Pia and Subbiah, Vamse Kumar and Feld, Corey , month = may, year =. Cited but. doi:10.48550/arXiv.2605.06635 , abstract =

    doi arXiv

  38. [46]

    doi:10.48550/arXiv.2606.19714 , abstract =

    Zhang, Zilong and Hung, Yi-Ting and He, Weiyi and Zhang, Junxi and Ding, Lei and Yeh, Chi-Kuang , month = jun, year =. doi:10.48550/arXiv.2606.19714 , abstract =

    doi arXiv

  39. [47]

    Rethinking

    Sahu, Gaurav and Charlin, Laurent and Pal, Christopher , month = may, year =. Rethinking. doi:10.48550/arXiv.2605.29234 , abstract =

    doi arXiv

  40. [48]

    Citation

    Ovcharov, Volodymyr , month = may, year =. Citation. doi:10.48550/arXiv.2606.00898 , abstract =

    doi arXiv

  41. [49]

    and Ré, Christopher and Chilton, Adam and Narayana, Aditya and Chohlas-Wood, Alex and Peters, Austin and Waldon, Brandon and Rockmore, Daniel N

    Guha, Neel and Nyarko, Julian and Ho, Daniel E. and Ré, Christopher and Chilton, Adam and Narayana, Aditya and Chohlas-Wood, Alex and Peters, Austin and Waldon, Brandon and Rockmore, Daniel N. and Zambrano, Diego and Talisman, Dmitry and Hoque, Enam and Surani, Faiz and Fagan,...

    doi arXiv

  42. [50]

    Findings of the

    Cao, Chuxue and Zhu, Han and Ji, Jiaming and Sun, Qichao and Zhu, Zhenghao and Yinyu, Wu and Dai, Josef and Yang, Yaodong and Han, Sirui and Guo, Yike , editor =. Findings of the. 2025 , pages =. doi:10.18653/v1/2025.findings-acl.721 , abstract =

    2025 doi

  43. [51]

    doi:10.48550/arXiv.2310.04793 , abstract =

    Wang, Neng and Yang, Hongyang and Wang, Christina Dan , month = nov, year =. doi:10.48550/arXiv.2310.04793 , abstract =

    doi arXiv

  44. [52]

    doi:10.48550/arXiv.2311.11944 , abstract =

    Islam, Pranab and Kannappan, Anand and Kiela, Douwe and Qian, Rebecca and Scherrer, Nino and Vidgen, Bertie , month = nov, year =. doi:10.48550/arXiv.2311.11944 , abstract =

    doi arXiv

  45. [53]

    doi:10.48550/arXiv.2512.09506 , abstract =

    Ding, Jinru and Ding, Chao and Pang, Wenrao and Xiao, Boyi and Liu, Zhiqiang and Chen, Pengcheng and Chen, Jiayuan and Yuan, Tiantian and Guan, Junming and Jiang, Yidong and Cheng, Dawei and Xu, Jie , month = dec, year =. doi:10.48550/arXiv.2512.09506 , abstract =

    doi arXiv

  46. [54]

    doi:10.48550/arXiv.2210.03849 , abstract =

    Chen, Zhiyu and Li, Shiyang and Smiley, Charese and Ma, Zhiqiang and Shah, Sameena and Wang, William Yang , month = oct, year =. doi:10.48550/arXiv.2210.03849 , abstract =

    doi arXiv

  47. [55]

    doi:10.48550/arXiv.2109.00122 , abstract =

    Chen, Zhiyu and Chen, Wenhu and Smiley, Charese and Shah, Sameena and Borova, Iana and Langdon, Dylan and Moussa, Reema and Beane, Matt and Huang, Ting-Hao and Routledge, Bryan and Wang, William Yang , month = may, year =. doi:10.48550/arXiv.2109.00122 , abstract =

    doi arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.