Pith. sign in

REVIEW 4 major objections 5 minor 34 references

PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PEDANTIC is the first public dataset of 14,536 US patent claims annotated with examiner-cited reasons for indefiniteness, and on it, LLM agents fail to beat logistic regression on the binary decision despite often naming the correct…

desk verdict A genuinely useful first dataset for patent definiteness, but the negative-class construction needs direct validation before benchmark conclusions are trusted. read the letter →

arxiv 2505.21342 v3 pith:PBY7Z5ZU submitted 2025-05-27 cs.CL

Valentin Knappich , Annemarie Friedrich , Anna Hätty , Simon Razniewski This is my paper
classification cs.CL
keywords patentdefinitenessindefinitenessclaimsofficeactionsLLM-as-Judgeautomaticdatasetconstructionexaminationclaimclarity
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces PEDANTIC, the first publicly available dataset for automatically examining definiteness in US patent claims: 14,536 claims from 3,710 utility applications in the natural-language-processing area, each labelled definite or indefinite, and every indefiniteness linked to an examiner-cited reason with a category (antecedent basis, undefined term, relative term, and others), a free-form explanation, and the exact claim spans in question. The dataset is built fully automatically by pulling first office actions from the US patent office's public records, isolating the sections that mention 112(b), and using an instruction-following language model to extract rejection reasons in a structured format, with claims from applications whose first office action never mentions '112(b)' serving as the definite class. In experiments, logistic regression with linguistic features reaches an AUROC near 60, and LLM agents of 32B and 72B parameters do not clearly beat it on the binary decision, even though the 32B agent recovers about a third of the examiner's own reasons at a high similarity threshold. The paper also finds that LLM claim-level confidence is poorly calibrated, while reason-level confidence correlates moderately with judge-assessed correctness. If correct, PEDANTIC gives patent AI researchers a reusable, human-validated benchmark and suggests that the bottleneck for LLMs is not finding the right reasons but converting them into a well-calibrated verdict.

What carries the argument

The load-bearing object is the PEDANTIC dataset itself, and the mechanism that carries the argument is its construction-and-evaluation pipeline. On the construction side, the pipeline takes first office actions from the US patent office's public file-wrapper records, selects sections whose headings contain '112', and prompts an LLM to extract each rejection reason as a JSON record with a category from the eight-class schema (antecedent basis, undefined term, relative term, exemplary phrasing, functional claiming, contradicting limitations, omission of essential elements or steps, other), a verbatim free-text explanation, and the recited claim spans, which are fuzzy-matched back into the claim text. The definite class is populated by claims from applications whose first office action never contains the literal '112(b)' string, sampled to match the count of indefinite claims. On the evaluation side, an LLM-as-Judge component scores every model-cited reason against every examiner-cited reason on a 1–5 similarity scale, with the probability-weighted mean (following recent work on judgment distributions) converted to a score, from which thresholded and soft precision/recall/F1 are computed; the agents themselves use verbalized probability expressions mapped to numbers via empirically calibrated perception values.

What would settle it

Sample a few hundred PEDANTIC claims labelled definite, pull their complete prosecution histories (not just the first office action) from the patent office's file-wrapper records, and search for any §112(b) rejection phrased without the literal string '112(b)'—for instance, '112, second paragraph' or a definiteness rejection raised in a later action; if a non-negligible share of those claims were rejected as indefinite, the definite class is contaminated and the reported AUROC and F1 comparisons would need to be recomputed on the cleaned sample.

Watch

Extended reading notes

Core claim

PEDANTIC is claimed to be the first public corpus that annotates patent definiteness at the level of the examiner's reasons rather than as a bare label. The corpus contains 14,536 claims from 3,710 G06F40 applications filed after 2014, evenly split between definite and indefinite; the 9,215 indefiniteness reasons fall into eight categories dominated by antecedent-basis errors and undefined terms (together 73% of reasons). The construction pipeline retrieves the first office action for each application, filters for sections headed with '112', prompts a large language model to output rejection reasons in a fixed JSON schema with verbatim reasoning and recited phrases, and fuzzy-matches those phrases into the claim text; a human validation of 50 claims found every binary label correct, all but one free-text reasoning correct, and 19 of 27 category assignments correct. The paper's experimental claim is that current LLM agents—given tools to fetch parent claims and search the description—achieve at best an AUROC of 60.3 when ensembled with logistic regression, and individually do not outperform the linguistic-feature logistic regression baseline on the binary decision, despite the 32B agent matching 35.4% of examiner-cited reasons at threshold 75 in the LLM-as-Judge evaluation. The authors read this as a disconnect between reason identification and calibrated decision-making: LLMs often know why a claim is indefinite but cannot reliably turn that knowledge into the correct verdict.

Load-bearing premise

Every 'definite' label in PEDANTIC rests on the assumption that an application whose first office action never contains the literal string '112(b)' has no indefinite claims, so a rejection phrased as '112, second paragraph' or raised only in a later office action would silently mislabel that claim as definite and inject noise into every reported comparison.

Editorial extensions

If this is right

  • PEDANTIC provides the first public benchmark on which future definiteness-examination models can be compared reproducibly, including the fine-grained reason annotations needed for explainable predictions.
  • The result that LLM agents recover many examiner reasons but still lag logistic regression on the binary label implies that improving claim-level calibration—rather than better reason extraction—is the immediate target for LLM-based examiners.
  • The pipeline is domain-agnostic over USPTO CPC classes, so the same automatic construction can be rerun to produce definiteness datasets for other technical fields without manual annotation.
  • The pairwise LLM-as-Judge protocol generalizes to other rejection grounds (novelty, non-obviousness) and other explanation-evaluation tasks where free-text reasons must be compared.
  • Because the best model is an ensemble of a linear model and an LLM, the paper's numbers suggest that hybrid systems may be the practical route until LLM calibration improves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The definite class is defined by the absence of the literal string '112(b)' in the first office action, so label noise is a real risk: examiners may write '112, second paragraph' or issue the rejection only in a later action. A quick robustness check would be to re-annotate a sample of definite claims from the full prosecution history and measure how much the benchmark numbers shift.
  • The paper's feature analysis shows that higher readability scores correlate with indefiniteness while higher complexity correlates with definiteness—consistent with the idea that terse, under-specified claims are more likely to be rejected. One could test whether this sign pattern holds across CPC classes and whether it became stronger after the 2014 legal standard change.
  • The LLM-as-Judge has been validated only indirectly against human annotations for the binary label; a direct validation would compare judge scores on reason pairs to human (examiner or attorney) pairwise judgments, which would establish whether the reason-level conclusions hold up.
  • Since reason-level confidence was moderately correlated with correctness while claim-level confidence was not, a natural next step is to build a decision rule that aggregates reason-level confidences into the binary verdict; the paper does not propose such a rule, but its data would support testing one.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces PEDANTIC, a dataset of 14,536 US patent claims from 3,710 NLP-related applications, with binary definite/indefinite labels, indefiniteness categories, free-text reasons, and affected claim spans extracted automatically from first office actions using Gemma 3 27B. The authors validate the extraction on 50 manually inspected claims, finding all binary labels correct, reasoning text correct except for one formatting issue, and categories correct in 19/27 cases. They then benchmark logistic regression and Qwen 2.5 32B/72B agents on binary classification, multi-label category prediction, and an LLM-as-Judge comparison of examiner-cited versus model-cited reasons. The headline empirical finding is that LLM agents barely match or slightly trail logistic regression on binary AUROC (best 60.3), despite identifying many examiner-cited reasons for indefiniteness.

Significance. If the label-construction issues are resolved, PEDANTIC would be a valuable first public resource for automatic definiteness examination, a task with no prior public annotated dataset. The paper deserves credit for grounding labels in external examiner-produced office actions rather than in the evaluated models, for using application-level splits to avoid leakage, for checking time-based and claim-characteristic confounds, and for releasing dataset and code. The human validation, although small, covers the core binary and reasoning annotations. The main significance depends on the reliability of the negative labels and on the validation of the LLM-as-Judge, both of which need strengthening before the benchmark conclusions can be taken at face value.

major comments (4)
  1. [Section 3.4 / Appendix A] The negative class is constructed from the absence of the literal string '112(b)' in the first office action, but the paper's own parsing prompt in Appendix A (line 3) instructs the model to extract rejections under '35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph'. Since office actions frequently phrase the rejection as '112, second paragraph' or '112(b) / pre-AIA 112, second paragraph', and Section 3.2 downloads only the first office action, an unknown fraction of the 7,268 'definite' claims may actually be indefinite. This is not uniform label noise: mislabeled negatives are likely to resemble genuine indefiniteness cases, which can depress AUROC for models that detect real issues, distort the feature weights in Figure 3, and weaken the headline comparison in Table 4. I request a direct audit: for a random sample of 'definite' applications, search all office actions for all 112 formulations, manually verify the resulting labels, report the noise rate, and if material, re-estimate the benchmarks.
  2. [Section 3.6] The human validation checked only 24 definite claims, too few to detect a systematic miss in the definite filter. If the filter misses 5% of definite applications, the probability of seeing zero errors in 24 samples is above 29%; for a 2% miss rate it is above 61%. The statement that 'all binary labels are correct' should therefore be restricted to the validation sample or supported by a larger, stratified audit of definite labels. Without such an audit, the binary benchmark results in Table 4 rest on an unquantified negative-label assumption.
  3. [Section 3.6 / Table 4] The validation study reports only 19/27 correct indefiniteness categories (70%), yet the multi-label F1 scores in Table 4 and the category distribution in Table 2 are presented as dataset outcomes, and the abstract claims 'high-quality annotations'. Since fine-grained categories are a central advertised component of PEDANTIC, the paper should either improve category extraction (for example, by enforcing the 'other' fallback already present in the Appendix A prompt) or report per-category reliability and clearly mark the multi-label benchmark as preliminary. As written, the claim of high-quality annotations overstates the reliability of the category layer.
  4. [Section 5.2] The LLM-as-Judge evaluation drives the conclusion that the Qwen 32B model identifies examiner-cited reasons better than the 72B model, but the judge itself is not validated against human judgments on this patent-specific task. A small human-annotated set of reason pairs would establish whether the 75-point threshold and the probability-weighted mean behave as intended. Without such validation, the differences in Table 5 may reflect judge bias rather than model capability, and the claimed disconnect between binary classification and reason identification remains uncertain.
minor comments (5)
  1. [Figure 3] The feature label 'IOU Claim Description' is not defined in the text; please define the intersection-over-union measure (likely over word sets) and explain how it is computed.
  2. [Table 3] The header layout with 'Independent ✓+ ✗ ✓ ✗' is ambiguous; please add explicit column labels or a caption explaining the four columns under 'Definite' and 'Indefinite'.
  3. [Appendix B] In Guideline 8 of the examination prompt, 'the claim begin rejected' should read 'the claim being rejected'.
  4. [References] Reference [11] appears to have a formatting error in the bibliography entry; the closing parenthesis for the workshop proceedings is missing.
  5. [Section 5.1] The threshold balancing procedure is described in words only; please state explicitly that the threshold is chosen on the validation set and applied to the test set, and clarify whether the percentages in Table 4 reflect this procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the dataset labels come from external USPTO office actions, and the model evaluations use held-out test data with independently tuned thresholds.

full rationale

PEDANTIC's labels are grounded in external USPTO examiner documents, not in the outputs of the models being evaluated. The positive class is drawn from examiner-cited 112(b) rejections parsed into structured reasons, and the negative class is drawn from applications whose first office action does not contain the literal string '112(b)'. Although this negative-class construction is a plausible source of label noise and deserves an audit, it is a data-quality concern rather than a circularity: the model inputs (claim and description) do not contain the office-action string that defines the labels, so the prediction task is not equivalent to its own training signal. The evaluation uses application-level splits into train, validation, and test sets, with confidence thresholds tuned on validation and metrics reported on test, so the headline comparisons are not fitted on the evaluation data. The only mild overlap is that Gemma 3 27B is used both to parse the examiner-cited reasons and as the LLM-as-Judge for reason similarity, but the judge compares model-generated reasons against examiner-written text rather than against the parser's own output, and the evaluated models are Qwen agents, not Gemma. The one self-citation (Pap2pat, reference [4]) is used only as a pointer to automatic drafting systems in related work and is not load-bearing for any central claim. No equation or construction step reduces a predicted quantity to an input parameter or to a definitional variant of the target label, so the paper's derivation chain is self-contained.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a set of data-construction heuristics and analysis assumptions. The most consequential are the definite-label heuristic (absence of '112(b)') and the small validation sample. No free parameters are fitted to data in the construction of the dataset itself; the threshold balancing in experiments is a standard evaluation choice.

assumptions (6)
  • domain assumption The first office action contains the core grounds for rejection; later office actions are ignored.
    Section 3.2: only the first office action is used. Indefiniteness rejections raised only in subsequent actions are missed, potentially undercounting indefinite claims.
  • ad hoc to paper An application whose office action lacks the literal string '112(b)' has no indefiniteness rejections.
    Section 3.4: used to sample definite claims; office actions may use phrasing like '112, second paragraph' or '112 paragraph 2', so this heuristic may introduce label noise.
  • domain assumption The eight category labels in Table 1 (plus 'other') cover the indefiniteness reasons present in the data.
    Used in the parsing prompt and multi-label evaluation; the paper notes some greyed-out categories were filtered and 'other' is a catch-all.
  • domain assumption Gemma 3 27B extracts reasons verbatim and accurately from office actions.
    Section 3.3: parsing is fully automatic; human validation on 50 claims found one formatting error in free-text reasons but category errors in 8/27 reasons.
  • ad hoc to paper The 50-claim human validation sample is representative of the full 14k dataset.
    Section 3.6: small sample; systematic errors in the LLM parser may not appear in 50 random claims.
  • ad hoc to paper The LLM-as-Judge similarity scores (Gemma 3 27B) reflect human judgments of reason equivalence.
    Section 5.2: no human validation of the judge; threshold 75 is arbitrary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims." pith.science (2026). https://pith.science/paper/PBY7Z5ZU

@misc{pith2026250521342,
  author       = {Pith},
  title        = {Pith review of: PEDANTIC: A Dataset for the Automatic Examination of Definiteness in Patent Claims},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PBY7Z5ZU}},
  note         = {Machine review of arXiv:2505.21342}
}
read the original abstract

Patent claims define the scope of protection for an invention. If there are ambiguities in a claim, it is rejected by the patent office. In the US, this is referred to as indefiniteness (35 U.S.C {\S} 112(b)) and is among the most frequent reasons for patent application rejection. The development of automatic methods for patent definiteness examination has the potential to make patent drafting and examination more efficient, but no annotated dataset has been published to date. We introduce PEDANTIC (Patent Definiteness Examination Corpus), a novel dataset of 14k US patent claims from patent applications relating to Natural Language Processing (NLP), annotated with reasons for indefiniteness. We construct PEDANTIC using a fully automatic pipeline that retrieves office action documents from the USPTO and uses Large Language Models (LLMs) to extract the reasons for indefiniteness. A human validation study confirms the pipeline's accuracy in generating high-quality annotations. To gain insight beyond binary classification metrics, we implement an LLM-as-Judge evaluation that compares the free-form reasoning of every model-cited reason with every examiner-cited reason. We show that LLM agents based on Qwen 2.5 32B and 72B struggle to outperform logistic regression baselines on definiteness prediction, even though they often correctly identify the underlying reasons. PEDANTIC provides a valuable resource for patent AI researchers, enabling the development of advanced examination models. We will publicly release the dataset and code.

Figures

Figures reproduced from arXiv: 2505.21342 by the authors.

Figure 1
Figure 1. PEDANTIC example and evaluation methodology. Every sample includes a claim, the patent description, and a list of rejection reasons that cause indefiniteness, each with a free-form reasoning, indefiniteness category, and recited claim spans. We evaluate models on (1) the prediction of the binary label definite or indefinite, (2) the identification of the correct indefiniteness categories as multi-label classificatio… view at source ↗
Figure 2
Figure 2. Definite vs. indefinite claims in our dataset over time. Left y-axis: stacked absolute values. Right y-axis fraction of indefinite claims. The fraction remains around 50%, therefore models cannot determine definiteness by inferring the date. independent. The full list is visualized in [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Feature importance scores of the logistic regression classifier with linguistic features. Positive values (red) indicate indefiniteness, negative values (green) indicate definiteness. The classifier performs similarly on train and test, i.e., the feature weights are not overfitted. determine the threshold with which half of the claims from the validation set are predicted as indefinite. 5.2. Pairwise Reasoning Judge… view at source ↗
Figures from the paper (1 more)
Figure 5
Figure 5. Figure 5: Distributions of reason-level confidence scores and the relation of confidence and correctness according to LLM-as-Judge. The reported judge scores are the soft micro-averaged precision scores, and thus represent the average max similarity per model￾cited reason. For b…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 20 canonical work pages

  1. [1]

    URL: https://www.uspto.gov/web/ offices/pac/mpep/index.html

    USPTO, Manual of Patent Examining Procedure (MPEP), 2024. URL: https://www.uspto.gov/web/ offices/pac/mpep/index.html

    work page 2024

  2. [2]

    USPTO, Pendency | Patents Dashboard | USPTO,

    work page

  3. [3]

    doi: 10.34667/TIND

    World Intellectual Property Organization, World Intel- lectual Property Indicators 2024, World Intellectual Property Organization, 2024. doi: 10.34667/TIND. 50133

    work page doi:10.34667/tind 2024

  4. [4]

    Pap2Pat: Benchmarking Outline-Guided Long-Text Patent Generation with Patent-Paper Pairs

    V. Knappich, S. Razniewski, A. Hätty, A. Friedrich, Pap2pat: Towards automated paper-to-patent drafting using chunk-based outline-guided generation, arXiv preprint arXiv:2410.07009 (2024)

    work page Pith review arXiv 2024

  5. [5]

    Q. Lu, A. Myers, S. Beliveau, USPTO Patent Prosecution Research Data: Unlocking Office Ac- tion Traits, 2017. doi: 10.2139/ssrn.3024621. arXiv:3024621

    work page doi:10.2139/ssrn.3024621 2017

  6. [6]

    URL: https://blog.juristat.com/ most-common-rejections

    The Most Common Rejections: 102, 103, and 112(b), 2019. URL: https://blog.juristat.com/ most-common-rejections

    work page 2019

  7. [7]

    Lo, J.-M

    H.-C. Lo, J.-M. Chu, Pre-trained Transformer-based Classification for Automated Patentability Examina- tion, in: 2021 IEEE Asia-Pacific Conference on Com- puter Science and Data Engineering (CSDE), 2021, pp. 1–5. doi:10.1109/CSDE53843.2021.9718474

    work page arXiv 2021

  8. [8]

    Zheng, W.-L

    L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, I. Stoica, Judging llm-as-a-judge with mt-bench and chatbot arena, in: Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, 2023

    work page 2023

Show all 34 references
  1. [9]

    Risch, N

    J. Risch, N. Alder, C. Hewel, R. Krestel, Patent- Match: A Dataset for Matching Patent Claims & Prior Art, 2020. doi:10.48550/arXiv.2012.13919. arXiv:2012.13919

    2020 doi arXiv

  2. [10]

    Vowinckel, V

    K. Vowinckel, V. D. Hähnke, SEARCHFORMER: Se- mantic patent embeddings by siamese transformers for prior art search, World Patent Information 73 (2023) 102192. doi:10.1016/j.wpi.2023.102192

    2023

  3. [11]

    Blume, G

    M. Blume, G. Heidari, C. Hewel, Comparing com- plex concepts with transformers: Matching patent claims against natural language text, in: 5th Workshop on Patent Text Mining and Semantic Technologies 5https://ceur-ws.org/GenAI/Taxonomy.html (PatentSemTech, volume 3775, CEUR-WS....

    2024

  4. [12]

    Parikh, S

    A. Parikh, S. Dori-Hacohen, ClaimCompare: A data pipeline for evaluation of novelty destroying patent pairs, in: 5th Workshop on Patent Text Mining and Semantic Technologies (PatentSemTech, volume 3775 of CEUR Workshop Proceedings, CEUR-WS.org, 2024, pp. 61–66

    2024

  5. [13]

    Stamatis, M

    V. Stamatis, M. Salampasis, K. Diamantaras, A novel re- ranking architecture for patent search, World Patent Information 78 (2024) 102282. doi: 10.1016/j.wpi. 2024.102282

    2024

  6. [14]

    J. Shan, Q. Zhang, C. Shi, M. Gui, S. Wang, U. Naseem, Structural Representation Learning and Disentangle- ment for Evidential Chinese Patent Approval Predic- tion, in: Proceedings of the 33rd ACM International Conference on Information and Knowledge Man- agement, ACM, 2024, p...

    2024

  7. [15]

    Y. Shen, Z. Lin, PatentGrapher: A PLM-GNNs Hybrid Model for Comprehensive Patent Plagiarism Detection Across Full Claim Texts, IEEE Access 12 (2024) 182717– 182725. doi:10.1109/ACCESS.2024.3508762

    2024

  8. [16]

    T. Wei, D. Feng, S. Song, C. Zhang, An extraction and novelty evaluation framework for technology knowledge elements of patents, Scientometrics (2024). doi:10.1007/s11192-024-04990-9

    2024 doi

  9. [17]

    Ikoma, T

    H. Ikoma, T. Mitamura, Can AI Examine Nov- elty of Patents?: Novelty Evaluation Based on the Correspondence between Patent Claim and Prior Art, 2025. doi:10.48550/arXiv.2502.06316. arXiv:2502.06316

    2025 doi arXiv

  10. [18]

    R. Lee, A. Spangher, X. Ma, PatentEdits: Fram- ing Patent Novelty as Textual Entailment, 2024. arXiv:2411.13477

    2024 arXiv

  11. [19]

    S. Hido, S. Suzuki, R. Nishiyama, T. Imamichi, R. Taka- hashi, T. Nasukawa, T. Id^|^eacute;, Y. Kanehira, R. Yohda, T. Ueno, A. Tajima, T. Watanabe, Modeling Patent Quality: A System for Large-scale Patentability Analysis using Text Mining, Journal of Information Processing 20...

    2012 doi

  12. [20]

    N. Kong, U. Dulleck, A. B. Jaffe, S. Sun, S. Vajjala, Lin- guistic metrics for patent disclosure: Evidence from university versus corporate patents, Research Pol- icy 52 (2023) 104670. doi:10.1016/j.respol.2022. 104670

    2023 doi

  13. [21]

    J. H. Ashtor, Modeling patent clarity, Research Pol- icy 51 (2022) 104415. doi:10.1016/j.respol.2021. 104415

    2022 doi

  14. [22]

    Mersha, K

    M. Mersha, K. Lam, J. Wood, A. K. AlShami, J. Kalita, Explainable artificial intelligence: A survey of needs, techniques, applications, and fu- ture direction, Neurocomputing 599 (2024) 128111. URL: https://www.sciencedirect.com/science/article/ pii/S0925231224008828. doi: htt...

    2024

  15. [23]

    Huang, S

    S. Huang, S. Mamidanna, S. Jangam, Y. Zhou, L. H. Gilpin, Can large language models ex- plain themselves? a study of llm-generated self- explanations, 2023. URL: https://arxiv.org/abs/2310. 11207. arXiv:2310.11207

    2023 arXiv

  16. [24]

    Madsen, S

    A. Madsen, S. Chandar, S. Reddy, Are self-explanations from Large Language Models faithful?, in: L.-W. Ku, A. Martins, V. Srikumar (Eds.), Findings of the Associa- tion for Computational Linguistics: ACL 2024, Associ- ation for Computational Linguistics, 2024, pp. 295–337. doi...

    2024 doi

  17. [25]

    Agarwal, S

    C. Agarwal, S. H. Tanneru, H. Lakkaraju, Faithfulness vs. plausibility: On the (un)reliability of explanations from large language models, 2024. URL: https://arxiv. org/abs/2402.04614. arXiv:2402.04614

    2024 arXiv

  18. [26]

    Nautilus, Inc. v. Biosig Instru- ments, Inc, 572 U.S. 898 (2014), https://supreme.justia.com/cases/federal/us/572/898/, 2014

    2014

  19. [27]

    A. K. et al., Gemma 3 technical report, 2025. URL: https: //arxiv.org/abs/2503.19786. arXiv:2503.19786

    2025 arXiv

  20. [28]

    Fagen-Ulmschneider, Per- ception of Probability Words, https://waf.cs.illinois.edu/visualizations/Perception- of-Probability-Words/, 2025

    W. Fagen-Ulmschneider, Per- ception of Probability Words, https://waf.cs.illinois.edu/visualizations/Perception- of-Probability-Words/, 2025

    2025

  21. [29]

    K. Tian, E. Mitchell, A. Zhou, A. Sharma, R. Rafailov, H. Yao, C. Finn, C. Manning, Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feed- back, in: Proceedings of the 2023 Conference on Em- pirical Meth...

    2023

  22. [30]

    Xiong, Z

    M. Xiong, Z. Hu, X. Lu, Y. LI, J. Fu, J. He, B. Hooi, Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs, in: The Twelfth International Conference on Learning Repre- sentations, 2024. URL: https://openreview.net/forum? id=gjeQKFxFpZ

    2024

  23. [31]

    V. Wang, M. J. Q. Zhang, E. Choi, Improving llm-as-a-judge inference with the judgment distri- bution, 2025. URL: https://arxiv.org/abs/2503.03064. arXiv:2503.03064

    2025

  24. [32]

    rejectedClaims

    M. Yasunaga, L. Shamis, C. Zhou, A. Cohen, J. Weston, L. Zettlemoyer, M. Ghazvininejad, Alma: Alignment with minimal annotation, 2024. URL: https://arxiv.org/ abs/2412.04305. arXiv:2412.04305. A. Office Action Parsing Prompt 1 ### TASK 2 3 Your task is to extract data related ...

    2024 arXiv

  25. [2025]

    URL: https://www.uspto.gov/dashboard/patents/ pendency.html

  26. [5442]

    URL: https://aclanthology.org/2023.emnlp-main. 330/. doi:10.18653/v1/2023.emnlp-main.330

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.