Pith. sign in

REVIEW 3 major objections 5 minor 45 references

Diagnosing a language model's weaknesses at the level of individual rubric criteria, then fine-tuning on those precise failures, outperforms both prompt-level and random targeting on most professional-domain benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 21:16 UTC pith:42QOG3PV

load-bearing objection CRAFT is a real granularity shift with a well-controlled empirical core; the unnamed judge and the Table 3 inconsistency are the two things to fix before trusting it. the 3 major comments →

arxiv 2607.16122 v1 pith:42QOG3PV submitted 2026-07-17 cs.AI cs.LG

CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data

Vipul Gupta , Zihao Wang , Razvan-Gabriel Dumitru , MohammadHossein Rezaei , Aakash Sabharwal , Yunzhong He This is my paper
classification cs.AI cs.LG
keywords rubric-based evaluationcapability diagnosishierarchical clusteringweakness targetingsupervised fine-tuning dataLLM-as-a-judgeprofessional-domain benchmarksdynamic granularity selection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that an evaluation should tell you what to train on next, not just how a model scored. CRAFT converts a rubric-annotated dataset into a model-specific diagnosis: each grading criterion becomes a capability probe, the criteria are clustered into a hierarchical capability tree, and the model is scored at every node. Weak nodes are then selected dynamically across tree levels and used to generate targeted supervised fine-tuning data. With data generation, fine-tuning, and evaluation held fixed, CRAFT posts the strongest finance-domain average for all four open-source models tested and the strongest legal average for three of four, with the fourth inside decoding-variance bands of the best baseline. The paper's claim is that the unit of diagnosis — individual rubric criteria rather than whole prompts or categories — is what makes the fine-tuning data more effective.

Core claim

The paper's central discovery is that grouping evaluation data by rubric criterion, rather than by prompt, exposes a model's failures at the granularity where each failure is most clearly expressed, and that fine-tuning on those specific weaknesses transfers to held-out benchmarks. CRAFT flattens a rubric dataset into prompt-criterion pairs, asks a language model to describe the capability each criterion tests, and organizes those descriptions into a tree with fixed domain and subcategory levels on top and learned capability levels below. Pass/fail judgments on every criterion, made by an LLM-as-a-judge, are rolled upward so every node carries a pass rate. A top-down search then picks the we

What carries the argument

The capability tree. Each leaf is a single prompt-rubric pair; internal nodes carry natural-language capability descriptions produced by embedding, k-means grouping, and an LLM summarization pass, so every node has both a semantic description and an empirical pass rate aggregated from the criterion-level judge decisions in its subtree. The second load-bearing piece is dynamic weak-node selection: a top-down search with a relaxed pass-rate threshold that collects low-performing nodes with enough support (more than 30 rubric criteria) at whatever depth the weakness is clearest, excluding ancestors and descendants so selected nodes are disjoint. Fixed top levels come from the dataset's domain a

Load-bearing premise

Everything rests on the LLM judge that assigns a pass/fail label to each rubric criterion, since every node score is an inheritance of those labels; if the judge cannot apply the criteria consistently, the weak-node list is wrong and the fine-tuning gains cannot be attributed to capability targeting.

What would settle it

The paper never names its judge or reports judge accuracy, inter-judge agreement, or calibration. A concrete check: take a random sample of criterion-level pass/fail labels, have human annotators apply the same rubrics, and measure agreement; if agreement is poor, or if swapping in a different judge changes the selected weak nodes and erases the SFT gains, the central claim collapses. A second check: run weak-node selection on shuffled criterion labels; if gains persist, the tree structure or the generation recipe, not the diagnosis, is driving the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Capability diagnosis should be carried out at the level of answer requirements: evaluations that carry fine-grained rubrics become a direct interface to fine-tuning data collection.
  • The scored hierarchy adds value beyond extra in-domain data, because CRAFT outperforms the random baseline that draws from the same domain pool without a tree or weakness targeting.
  • Rubric criteria carry diagnostic information that whole-prompt clusters do not, because CRAFT outperforms the prompt-level tree baseline under an otherwise identical pipeline.
  • Weakness targeting cannot be fixed to a single tree depth: the ablation shows the better fixed level varies by model and domain, while dynamic cross-level selection wins every row.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit: the gain over prompt-level clustering should widen as rubric granularity increases, since the diagnostic datasets here carry roughly 17 to 18 criteria per prompt on average, far more signal than datasets with one or two criteria per prompt. This is testable by running the same pipeline on thinner rubrics.
  • Because the paper's own passage on the judge step admits the dependence, the first robustness check is judge reliability: the reported gains would be consistent with the model being trained toward the judge's biases if the pass/fail labels are noisy.
  • A broader design principle suggested by the dynamic-depth result: data-selection methods should screen for the granularity at which the evidence of weakness is most decisive, rather than committing to a fixed skill taxonomy; this could transfer to domains without rubric metadata by learning the tree entirely.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. CRAFT (Clustering Rubrics for Actionable Fine-Tuning) converts a rubric-annotated evaluation dataset into a model-specific diagnosis of weak capabilities. For each prompt–rubric pair, an LLM extracts a capability description; these descriptions are clustered bottom-up into a hierarchical capability tree; the target model is scored at every leaf by an LLM-as-a-judge on each rubric criterion, and pass rates are aggregated to nodes. Weak nodes are then selected dynamically across tree levels by a top-down threshold search, with support constraints and ancestor/descendant exclusion. The selected nodes are used to condition synthetic fine-tuning data generation. The paper evaluates CRAFT against two baselines under a fixed downstream pipeline: EvalTree (prompt-level clustering) and Random (untargeted in-domain sampling). Experiments cover four open-source models, two domains (finance and legal), and 13 held-out benchmarks disjoint from the diagnostic rubric data. The central empirical claim is that criterion-level diagnosis yields the strongest finance-domain average for all four models and the strongest legal average for three of four, with the fourth within decoding-variance bands of the best baseline.

Significance. If the central claim holds, CRAFT provides a practically useful bridge between rubric-based evaluation and targeted post-training data generation, and it sharpens the evaluation-diagnosis unit from prompts/tasks to individual response requirements. The experimental design is the right shape: the diagnostic rubric data are disjoint from the final benchmarks, Random controls for mere in-domain data exposure, EvalTree controls for hierarchy-vs-flat organization, and repeated temperature decoding with reported variance bands makes the aggregate comparisons more robust than single runs. The dynamic-versus-fixed-level ablation is also a sensible test of the method's key design choice. However, the whole diagnosis rests on an unvalidated, unnamed LLM judge, and the ablation table contains internal numerical inconsistencies with the main results. These two issues are load-bearing for the paper's attribution of downstream gains to criterion-level capability targeting, and they must be resolved before the contribution can be accepted.

major comments (3)
  1. [§3.4, 'Dynamically Identifying Low-Performing Nodes'] The manuscript explicitly states: 'all node scores inherit the criterion-level judge decisions; CRAFT therefore depends on judges that can apply rubric criteria consistently rather than only produce a holistic preference.' Yet the judge model is never named, and no judge accuracy, inter-judge agreement, calibration, or per-criterion-type bias analysis is reported. Since every node pass rate and every weak-node selection is a function of these pass/fail labels, a judge that is systematically lenient on one criterion type (e.g., 'explain/justify') or strict on another (e.g., 'format/citation') would produce a capability tree that targets judge artifacts rather than true capability gaps. The observed downstream improvements over Random and EvalTree would then not be attributable to criterion-level capability diagnosis. This is a specific, unmeasured dependency, not a general objection to LL
  2. [Table 3, 'Ablation: Dynamic vs. Fixed-Level Node Identification'] The 'Dynamic' column in Table 3 does not match the CRAFT averages reported in Tables 1 and 2 for the same models and domains. For example, Qwen3-4B Legal Dynamic is 50.4 while Table 1 reports CRAFT average 53.0; Llama-3.1-8B Legal Dynamic is 50.3 vs 51.0; Qwen3-4B Finance Dynamic is 47.1 vs 46.0; Llama-3.1-8B Finance Dynamic is 43.1 vs 42.5. If the ablation uses the same setting as the main experiments, these values should agree exactly; if it uses different runs, seeds, checkpoint selection, or a different evaluation protocol, that must be stated. As reported, the quantitative claim 'Dynamic selection achieves the best average in every ablation row' cannot be checked, and the fixed-level comparisons may be confounded by evaluation differences. Additionally, the caption says 'weak code selection' where it should say 'weak node selection.'
  3. [§4 and §5.2, downstream pipeline controls] The paper says all methods share the same data-generation, fine-tuning, and evaluation setup, which is the right way to isolate the targeting method. However, the concrete instantiations of several adaptive components are not specified: the formulas or values for nbase and ntop, the exact weak-node target count, the pass-rate threshold schedule, and the criteria for the 30-rubric support threshold. These are not necessarily load-bearing for the CRAFT-vs-baseline comparison because the same pipeline is held fixed, but they are needed for reproducibility and for interpreting the cross-model differences in Table 3. Please provide exact parameter values or the adaptive formulas, at least in an appendix.
minor comments (5)
  1. [Title/Abstract] The title contains a typo: 'Fine-T uning' should be 'Fine-Tuning.'
  2. [Table 3 caption] As noted above, 'weak code selection' should be 'weak node selection.'
  3. [References] Some reference author lists contain stray spaces in initials (e.g., 'P .', 'V .'), and several references are to arXiv preprints without version or access dates; please standardize.
  4. [§6.5] The fixed-level ablation is run on only two of the four models. The conclusion that dynamic selection 'remains the most reliable strategy across models and domains' would be stronger if the ablation covered all four models, especially Qwen3-8B and Gemma-3-4B where the main gains are smaller.
  5. [§6.1] The benchmark descriptions are clear, but the claim that 'none of the benchmarks share the long-form format of the diagnosis data' is not quantified. It would help to state the answer format of each benchmark explicitly in one table.

Circularity Check

0 steps flagged

No significant circularity: the central claim is an empirical transfer result on disjoint held-out benchmarks, with the unvalidated judge and PRBench co-authorship noted as validity concerns but not circular reductions.

full rationale

The paper's derivation chain is empirical rather than formal: CRAFT converts rubric criteria into capability descriptions, clusters them into a hierarchy, scores nodes with LLM-as-a-judge pass/fail labels, selects weak nodes, generates SFT data, and measures transfer on 13 held-out benchmarks explicitly disjoint from the diagnostic rubrics. The final evaluation is not fitted to the diagnostic inputs: the Random baseline controls for exposure to in-domain synthetic data, and the EvalTree baseline controls for prompt-level organization, so the CRAFT-vs-baseline comparison does not reduce to the rubric data itself. No equation or parameter is fitted to the final benchmark scores; node pass rates are intermediate measurements, not predictions of downstream performance. The paper openly flags its main dependency in §3.4: 'all node scores inherit the criterion-level judge decisions; CRAFT therefore depends on judges that can apply rubric criteria consistently rather than only produce a holistic preference.' The judge model is unnamed and no judge-accuracy, inter-judge agreement, or calibration data are reported, so the diagnosis could be biased; that is a validity/robustness limitation, not a circular step. The use of PRBench (§5.4, ref [1]), co-authored by one of the present authors, is a data-source choice and is not invoked as proof of the claim. The Table 3 Dynamic values differing from Tables 1-2 despite claimed identical settings is an internal inconsistency that raises reproducibility concerns, but it is not circularity. No load-bearing step reduces to its own input by definition.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 1 invented entities

CRAFT's central claim rests on: (1) judge reliability, since every node score derives from LLM-as-a-judge pass/fail labels on rubric criteria; (2) unvalidated LLM extraction and clustering producing faithful capability abstractions; (3) transfer from generated SFT data to disjoint benchmarks; and (4) several unstated procedural parameters (node target count, thresholds, decode-run count, adaptive sizing functions). No formal derivation is involved — the method is an empirical pipeline — so the ledger is dominated by domain assumptions rather than free parameters.

free parameters (6)
  • Node support threshold (min rubric criteria per node) = >30 criteria (stated)
    Nodes with fewer than 30 prompt-rubric pairs are ineligible for selection (§3.4); chosen by hand, not justified.
  • Weak-node target count = unspecified
    The search 'relaxes [the threshold] incrementally until the target number of weak nodes is collected' (§3.4); the target is never stated, so the amount of data targeting is uncontrolled and unreported.
  • Pass-rate threshold schedule = unspecified
    Starting threshold and relaxation increments are not reported (§3.4); this determines which nodes are selected.
  • Adaptive cluster sizing (nbase, ntop) = not specified as functions of group size
    nbase and ntop 'are set adaptively from the group size' but the mapping is not given (§3.3), so tree depth/shape is not reproducible.
  • Synthetic data budget per node and total = 40 prompts/node; 1,000 examples total
    Chosen budget for comparing methods (§4); held fixed across methods, but the per-node count interacts with the unknown node count.
  • Decoding temperature and number of repeated runs = temperature 0.5; run count unspecified
    Repeated temperature-0.5 decoding is used (§6) but the number of seeds is not stated, so variance estimates cannot be audited.
axioms (6)
  • domain assumption LLM-as-a-judge applies rubric criteria consistently and accurately
    All node pass rates inherit judge decisions (§3.4: 'all node scores inherit the criterion-level judge decisions'). The judge model is unnamed and no judge-accuracy or agreement numbers are reported.
  • domain assumption LLM-extracted capability descriptions faithfully represent the criterion's required capability
    A separate (unnamed) LM produces 'a short, decontextualized description of the capability needed' (§3.2); no human validation of these descriptions is provided.
  • domain assumption LLM hierarchical clustering produces a stable, semantically meaningful capability tree
    Tree construction relies on LLM summarization, consolidation, routing, and refinement with Gemini 2.5 Pro (§3.3); routing-stability checks use an independent model but are described only as a loop, with no stability statistics reported.
  • domain assumption Performance on the 13 held-out benchmarks measures transfer of the diagnosed capabilities
    The suite mixes MCQ, classification, extraction, and numeric formats 'so gains ... must reflect transfer of the underlying capabilities' (§6.1); this is asserted, and no task-level analysis connecting selected nodes to benchmark gains is provided.
  • domain assumption Diagnostic rubric data and held-out benchmarks are contamination-free
    Disjointness is asserted (§5.4, §6.1) but not demonstrated — no data release and no overlap checks.
  • standard math k-means on text-embedding-3-small embeddings captures capability similarity
    Standard embedding-clustering assumption used in §3.3 for base clusters.
invented entities (1)
  • Capability node / capability tree (CRAFT hierarchy) no independent evidence
    purpose: A hierarchical structuring of extracted capability descriptions over prompt-rubric pairs; each node carries a capability label, pass rate, and supporting pairs, and serves as the targeting unit for SFT data generation.
    The tree is an internal analytical construct of the method. There is no falsifiable handle outside the pipeline: node labels are never validated against human-judged capability taxonomies, and the only indirect evidence is downstream benchmark gain, which is not decomposed per node.

pith-pipeline@v1.3.0-alltime-deepseek · 17345 in / 18813 out tokens · 143167 ms · 2026-08-01T21:16:21.515196+00:00 · methodology

0 comments
read the original abstract

Evaluations should do more than measure a models current performance. They should tell us what to fix for the next model iteration and provide a way to generate targeted post training data. Most evaluation pipelines identify weak examples, topics, or categories, but they leave the underlying capability failure implicit: they say where a model fails, not why. We introduce CRAFT, a method that converts any rubric based evaluation dataset into a model specific diagnosis of weak capabilities. CRAFT treats each grading criterion as a capability probe: it extracts a capability description from every prompt rubric pair, clusters these descriptions into a hierarchical capability tree, scores the target model at every node, and selects low performing nodes dynamically across tree levels, at the granularity where each failure is clearest. The selected weak capabilities then direct the generation of targeted supervised finetuning data. Holding the data generation, finetuning, and evaluation setup fixed, we compare CRAFT against prompt level EvalTree clustering and untargeted random generation on four open source models, two professional domains (finance and legal), and 13 held out benchmarks disjoint from the diagnostic data. CRAFT achieves the strongest finance domain average for all four models under repeated temperature decoding; on legal domain, it is strongest for three of four models and remains within the decoding variance bands of the best baseline on the fourth. Diagnosing weaknesses at the level of rubric criteria, rather than prompts or categories, thus yields both a sharper picture of what a model cannot do and measurably better models after finetuning on that diagnosis.

Figures

Figures reproduced from arXiv: 2607.16122 by Aakash Sabharwal, MohammadHossein Rezaei, Razvan-Gabriel Dumitru, Vipul Gupta, Yunzhong He, Zihao Wang.

Figure 1
Figure 1. Figure 1: CRAFT pipeline. CRAFT extracts per-criterion capabilities, builds and scores a hierarchy, selects weak nodes (red), and uses them to generate targeted fine-tuning data. performance for all four models in the finance domain and strongest performance for three of four tested models in the legal domain. 3. We show that the granularity of weakness selection matters: dynamic cross-level selection outperforms fi… view at source ↗
Figure 2
Figure 2. Figure 2: Repeated-decoding average scores from Tables 1 and 2. CRAFT (teal) leads finance for all four models and legal for three of four. This breadth makes the evaluation conservative. Several benchmarks measure capability families, such as exam-style recall and sentiment classification, that are not obviously aligned with the diagnostic data or generated training examples. Reporting the full suite therefore test… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 9 linked inside Pith

  1. [1]

    A. F. Akyürek, A. Gosai, C. B. C. Zhang, V . Gupta, J. Jeong, A. Gunjal, T. Rabbani, M. Mazzone, D. Randolph, M. M. Meymand, et al. Prbench: Large-scale expert rubrics for evaluating high-stakes professional reasoning.arXiv preprint arXiv:2511.11562, 2025

    arXiv 2025

  2. [2]

    R. K. Arora, J. Wei, R. S. Hicks, P . Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al. Healthbench: Evaluating large language models towards improved human health.arXiv preprint arXiv:2505.08775, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Chen, Y.-M

    C.-C. Chen, Y.-M. Tseng, J. Kang, A. Lhuissier, M.-Y. Day, T.-T. Tu, and H.-H. Chen. Multi-lingual ESG issue identification. In C.-C. Chen, H. Takamura, P . Mathur, R. Sawhney, H.-H. Huang, and H.-H. Chen, editors,Proceedings of the Fifth Workshop on Financial Technology and Natural Language Processing and the Second Multimodal AI For Financial Forecastin...

    2023

  4. [4]

    M. Chen, N. Roberts, K. Bhatia, J. Wang, C. Zhang, F. Sala, and C. Ré. Skill-it! a data-driven skills framework for understanding and training language models.Advances in Neural Information Processing Systems, 36:36000–36040, 2023

    2023

  5. [5]

    Z. Chen, S. Li, C. Smiley, Z. Ma, S. Shah, and W. Y. Wang. Convfinqa: Exploring the chain of numerical reasoning in conversational finance question answering.Proceedings of EMNLP 2022, 2022. 13 Scale AI Research

    2022

  6. [6]

    Comanici, E

    G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, Q. Yue, Y. Ni, G. Xie, R. Xie, Y. Lin, Z. Liu, and M. Sun. ULTRAFEEDBACK: Boosting language models with scaled AI feedback. InForty-first International Conference on Machine Learning, 2024. URLhttps://openreview.net/forum?id=BOorDpKHiJ

    2024

  8. [8]

    Fernandes, A

    R. Fernandes, A. Biedenkapp, F. Hutter, and N. Awad. A llama walks into the ’bar’: Efficient supervised fine-tuning for legal reasoning in the multi-state bar exam, 2025. URL https://arxiv. org/abs/2504.04945

    Pith/arXiv arXiv 2025

  9. [9]

    Grattafiori, A

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    N. Guha, J. Nyarko, D. Ho, C. Ré, A. Chilton, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano, et al. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models.Advances in neural information processing systems, 36:44123–44279, 2023

    2023

  11. [11]

    Gunjal, A

    A. Gunjal, A. Wang, E. Lau, V . Nath, Y. He, B. Liu, and S. Hendryx. Rubrics as rewards: Reinforce- ment learning beyond verifiable domains.arXiv preprint arXiv:2507.17746, 2025

    Pith/arXiv arXiv 2025

  12. [13]

    Hendrycks, C

    D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt. Measuring massive multitask language understanding. InInternational Conference on Learning Representations,

  13. [14]

    Holzenberger, A

    N. Holzenberger, A. Blair-Stanek, and B. V . Durme. A dataset for statutory reasoning in tax law entailment and question answering. In N. Aletras, I. Androutsopoulos, L. Barrett, A. Meyers, and D. Preotiuc-Pietro, editors,Proceedings of the Natural Legal Language Processing Workshop 2020 co-located with the 26th ACM SIGKDD International Conference on Know...

    2020

  14. [15]

    URLhttps://openreview.net/forum?id=d7KBjmI3GmQ

  15. [16]

    G. T. A. Kamath, J. Ferret, S. Pathak, N. Vieillard, R. Merhej, S. Perrin, T. Matejovicova, A. Ram’e, M. Rivière, L. Rouillard, T. Mesnard, G. Cideron, J.-B. Grill, S. Ramos, E. Yvinec, M. Casbon, E. Pot, I. Penchev, G. Liu, F. Visin, K. Kenealy, L. Beyer, X. Zhai, A. Tsitsulin, R. I. Busa-Fekete, A. Feng, N. Sachdeva, B. Coleman, Y. Gao, B. Mustafa, I. B...

    Pith/arXiv arXiv 2025

  16. [17]

    Islam, A

    P . Islam, A. Kannappan, D. Kiela, R. Qian, N. Scherrer, and B. Vidgen. Financebench: A new benchmark for financial question answering, 2023

    2023

  17. [18]

    N. Kolt. Predicting consumer contracts.Berkeley Tech. LJ, 37:71, 2022

    2022

  18. [19]

    S. Kim, J. Shin, J. Jang, S. Longpre, H. Lee, S. Yun, R. Shin, S. Kim, J. Thorne, M. Seo, et al. Prometheus: Inducing fine-grained evaluation capability in language models. InInternational Conference on Learning Representations, volume 2024, pages 29927–29962, 2024

    2024

  19. [20]

    S. Li, J. Zhao, M. Wei, H. Ren, Y. Zhou, J. Yang, S. Liu, K. Zhang, and W. Chen. Rubrichub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. arXiv preprint arXiv:2601.08430, 2026

    arXiv 2026

  20. [21]

    Koreeda and C

    Y. Koreeda and C. Manning. ContractNLI: A dataset for document-level natural language inference for contracts. In M.-F. Moens, X. Huang, L. Specia, and S. W.-t. Yih, editors,Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907–1919, Punta Cana, Dominican Republic, Nov

    2021

  21. [22]

    doi: 10.18653/v1/2021.findings-emnlp.164

    Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-emnlp.164. URL https://aclanthology.org/2021.findings-emnlp.164/

    doi:10.18653/v1/2021.findings-emnlp.164 2021

  22. [23]

    Mahmoud, M

    A. Mahmoud, M. Rezaei, Z. Wang, A. Gunjal, B. Liu, and Y. He. Reward hacking in rubric-based reinforcement learning.arXiv preprint arXiv:2605.12474, 2026

    Pith/arXiv arXiv 2026

  23. [24]

    Liang, R

    P . Liang, R. Bommasani, T. Lee, D. Tsipras, D. Soylu, M. Yasunaga, Y. Zhang, D. Narayanan, Y. Wu, A. Kumar, B. Newman, B. Yuan, B. Yan, C. Zhang, C. Cosgrove, C. D. Manning, C. Re, D. Acosta- Navas, D. A. Hudson, E. Zelikman, E. Durmus, F. Ladhak, F. Rong, H. Ren, H. Yao, J. WANG, K. Santhanam, L. Orr, L. Zheng, M. Yuksekgonul, M. Suzgun, N. Kim, N. Guha...

    2023

  24. [25]

    Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu. G-eval: NLG evaluation using gpt-4 with better human alignment. In H. Bouamor, J. Pino, and K. Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore, Dec

    2023

  25. [26]

    D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: A graduate-level google-proof q&a benchmark. InFirst Conference on Language Modeling,

  26. [27]

    Rezaei, R

    M. Rezaei, R. Vacareanu, Z. Wang, C. Wang, B. Liu, Y. He, and A. F. Akyürek. Online rubrics elicitation from pairwise comparisons.arXiv preprint arXiv:2510.07284, 2025

    arXiv 2025

  27. [28]

    M. Maia, S. Handschuh, A. Freitas, B. Davis, R. McDermott, M. Zarrouk, and A. Balahur. Www’18 open challenge: financial opinion mining and question answering. InCompanion proceedings of the the web conference 2018, pages 1941–1942, 2018

    2018

  28. [29]

    text-embedding-3-small

    OpenAI. text-embedding-3-small. https://platform.openai.com/docs/guides/embeddings, 2024. Accessed: 2026-07-07

    2024

  29. [30]

    Srivastava, A

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slone, A. Rahane, A. S. Iyer, A. J. Andreassen, A. Madotto, A...

    2023

  30. [31]

    Tamkin, M

    A. Tamkin, M. McCain, K. Handa, E. Durmus, L. Lovitt, A. Rathi, S. Huang, A. Mountfield, J. Hong, S. Ritchie, et al. Clio: Privacy-preserving insights into real-world ai use.arXiv preprint arXiv:2412.13678, 2024

    Pith/arXiv arXiv 2024

  31. [32]

    Viswanathan, Y

    V . Viswanathan, Y. Sun, X. Kong, M. Cao, G. Neubig, and T. Wu. Checklists are better than reward models for aligning language models. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2026. URLhttps://openreview.net/forum?id=RPRqKhjrr6

    2026

  32. [33]

    M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh. Beyond accuracy: Behavioral testing of nlp models with checklist. InProceedings of the 58th annual meeting of the association for computational linguistics, pages 4902–4912, 2020

    2020

  33. [34]

    A. Shah, S. Paturi, and S. Chava. Trillion dollar words: A new financial dataset, task & market analysis. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6664–6679, 2023

    2023

  34. [35]

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  35. [36]

    S. Ye, D. Kim, S. Kim, H. Hwang, S. Kim, Y. Jo, J. Thorne, J. Kim, and M. Seo. FLASK: Fine-grained language model evaluation based on alignment skill sets. InICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URLhttps://openreview.net/forum?id=3OfPKwAqPf

    2024

  36. [37]

    Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen. Evaluating large language models at evaluating instruction following. InInternational Conference on Learning Representations, volume 2024, pages 40193–40219, 2024

    2024

  37. [38]

    Z. Wu, Y. Hu, W. Shi, N. Dziri, A. Suhr, P . Ammanabrolu, N. A. Smith, M. Ostendorf, and H. Ha- jishirzi. Fine-grained human feedback gives better rewards for language model training.Advances in Neural Information Processing Systems, 36:59008–59033, 2023

    2023

  38. [39]

    M. Xia, S. Malladi, S. Gururangan, S. Arora, and D. Chen. Less: selecting influential data for targeted 17 Scale AI Research instruction tuning. InProceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024

    2024

  39. [40]

    Y. Zhao, S. R. Damle, S. E. Dekker, S. Geng, K. W. Silva, J. J. Hubbard, M. F. Fernandez, F. Zelada- Arenas, A. Alvarez, B. Flores, et al. Pancanbench: A comprehensive benchmark for evaluating large language models in pancreatic oncology.arXiv preprint arXiv:2603.01343, 2026

    arXiv 2026

  40. [41]

    Zheng, N

    L. Zheng, N. Guha, B. R. Anderson, P . Henderson, and D. E. Ho. When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, ICAIL ’21, page 159–168, New York, NY, USA, 2021. Association for Computing ...

    arXiv 2021

  41. [42]

    F. Zhu, W. Lei, Y. Huang, C. Wang, S. Zhang, J. Lv, F. Feng, and T.-S. Chua. TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. InProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Paper...

    doi:10.18653/v1/2021.acl-long.254 2021

  42. [43]

    Z. Zeng, Y. Wang, H. Hajishirzi, and P . W. Koh. Evaltree: Profiling language model weaknesses via hierarchical capability trees. InSecond Conference on Language Modeling, 2025. URL https: //openreview.net/forum?id=aV2hQN9vkp

    2025

  43. [44]

    Zhang, Z

    J. Zhang, Z. Wang, L. Gui, S. M. Sathyendra, J. Jeong, V . Veitch, W. Wang, Y. He, B. Liu, and L. Jin. Chasing the tail: Effective rubric-based reward modeling for large language model post-training. arXiv preprint arXiv:2509.21500, 2025

    arXiv 2025

  44. [2023]

    doi: 10.18653/v1/2023.emnlp-main.153

    Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.153. URL https://aclanthology.org/2023.emnlp-main.153/. 15 Scale AI Research

    doi:10.18653/v1/2023.emnlp-main.153 2023

  45. [2024]

    URLhttps://openreview.net/forum?id=Ti67584b98