When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation

Lekang Jiang; Stephan Goetz; Wenjun Sun

REVIEW 2 major objections 4 minor 16 references

When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation

T0 review · 2 major / 4 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For patent claims, keeping the model’s reasoning silent produces better drafts than writing the reasoning out first.

desk verdict Solid workshop paper: implicit CoT beats explicit CoT on patent claims, with multi-metric evidence and a usable diagnostic framing; the causal story is a bit stronger than the isolation experiments. read the letter →

arxiv 2607.10480 v1 pith:AZX4KG25 submitted 2026-07-11 cs.CL

Lekang Jiang , Wenjun Sun , Stephan Goetz This is my paper

classification cs.CL

keywords patentclaimgenerationchain-of-thoughtpromptingimplicitCoTexplicitverbalizationbottlenecklegaldraftinglargelanguagemodelsinformation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Patent claims are long, hierarchical legal texts that must preserve exact technical details, dependencies, and formal language. The paper asks whether the usual Chain-of-Thought trick—making a language model write out intermediate reasoning—helps or hurts this task. It introduces a three-stage drafting scaffold (feature extraction, logical grouping, language formalization) and compares two ways of using it: implicit CoT, in which the model plans silently and outputs only the claims, versus explicit CoT, in which the same plan is verbalized before the claims. Across several models and two patent corpora, the silent version improves claim quality relative to a plain prompt, while the verbalized version consistently degrades it. The authors attribute the drop to a “verbalization bottleneck”: forcing the plan into natural language abstracts away legally critical details, interrupts the model’s already-learned drafting patterns, and locks early mistakes into the final text. The practical upshot is that, for detail-preserving legal drafting, reasoning can help—but only when it stays internal.

What carries the argument

The verbalization bottleneck: the same three-stage drafting scaffold is held fixed while the visibility of intermediate reasoning is varied; forcing that reasoning into natural language compresses the model’s rich internal state into a lossy textual plan on which all subsequent claim generation is conditioned.

What would settle it

A controlled experiment that keeps the identical three-stage scaffold and models but finds a patent claim corpus or model family where forcing the intermediate reasoning into text measurably improves content coverage, logical structure, and language quality over the silent version on the same expert and automatic metrics.

Watch

Extended reading notes

Core claim

Reasoning-enhanced prompting can improve patent claim generation, yet implicit CoT (the model is guided to think step-by-step but outputs only the final claims) consistently outperforms explicit CoT (the same intermediate reasoning is written out before the claims). Explicit verbalization creates an information bottleneck that harms the final draft through three mechanisms: abstraction of critical technical details, disruption of the model’s internalized generation patterns, and cascading error propagation across reasoning steps.

Load-bearing premise

That the three-stage scaffold of feature extraction, logical grouping, and language formalization is a faithful model of professional claim drafting, so that any performance gap between its silent and verbalized forms can be attributed purely to verbalization rather than to a misspecified scaffold.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing worth knowing is the clean empirical result: on patent claim generation, guiding the model to reason internally and emit only claims (implicit CoT) beats both plain prompting and the same three-step scaffold with verbalized intermediate steps (explicit CoT). That holds across Llama-3.1-8B, Mixtral-8x7B, GPT-4o, extra models in the appendix, and both EPD and HUPD-DCG. Human attorney scores, PatClaimEval, and LLM-as-judge all move in the same direction, with the biggest lift on GPT-4o under implicit CoT.

What is new is not “CoT can hurt” (Zheng, Wang, Ma already said that) but the first systematic implicit-vs-explicit comparison inside claim drafting, plus a three-mechanism diagnosis (detail abstraction, disruption of internalized drafting patterns, error cascade) backed by stepwise annotation on 80 examples and a concrete case study (EP3060168B1). The task-specific scaffold itself is a reasonable zero-shot design, and the ablation is honest: dropping logical grouping or formalization hurts; dropping extraction slightly helps the implicit variant.

Soft spots, in proportion. Human eval is one rater per sample on 120 items—ordinary for a workshop, not fatal, but it caps how hard you can lean on the absolute scores. Code is not released. The stress-test concern is real but overstated: the paper does not fully isolate verbalization from scaffold misspecification, and the ablation already shows the full three-stage plan is not optimal. Still, the shared scaffold is held fixed between implicit and explicit, so the gap is at least “verbalizing this plan hurts,” which is already actionable. The bottleneck account is interpretive, not causal intervention; the authors say so in the limitations.

Citations look clean; self-cites are to their public datasets and eval tools. Math is not the point—this is comparative NLP. For people who care about legal drafting or when to externalize reasoning, this is worth a careful read. I would send it to peer review for the workshop track without hesitation; the central comparison is well-supported and useful even if the mechanism story gets tightened later.

Referee Report

2 major / 4 minor

Summary. The paper studies Chain-of-Thought prompting for patent claim generation from technical descriptions. It introduces a task-specific zero-shot CoT scaffold with three stages (feature extraction, logical grouping, language formalization) and compares a plain prompt, implicit CoT (reason internally, emit only claims), and explicit CoT (verbalize the stages then emit claims). Across Llama-3.1-8B, Mixtral-8x7B, GPT-4o and additional models, on the EPD and HUPD-DCG datasets, automatic metrics (PatClaimEval, LLM-as-judge, BLEU/ROUGE/BERTScore) and human ratings by licensed patent attorneys (content coverage, logical structure, language quality) show that implicit CoT improves claim quality over the plain baseline while explicit CoT consistently underperforms implicit CoT. The authors attribute the gap to a verbalization bottleneck that abstracts critical technical details, disrupts internalized drafting patterns, and allows cascading errors across stages, supported by stepwise diagnostics on 80 examples, an ablation, and a detailed case study.

Significance. If the empirical pattern holds, the work supplies a concrete, counter-intuitive result for both legal NLP and the CoT literature: for a high-stakes, detail-preserving drafting task, forcing intermediate reasoning into text can degrade rather than improve output. The multi-model, multi-dataset design, attorney-rated human evaluation, stepwise intermediate diagnostics, ablation of the scaffold stages, and side-by-side case study (Tables 1–3, 9) constitute a solid empirical package that is rare in this domain. The three proposed mechanisms and the broader suggestion that the verbalization bottleneck may generalize to other formal legal drafting tasks (contracts, regulatory filings) offer a useful diagnostic template for future AI-for-law work.

major comments (2)

[§6 and Appendix C.3 / Table 6] The central causal claim in §6 (and the abstract) attributes the consistent implicit>explicit gap solely to the act of externalizing intermediate reasoning. This attribution presupposes that the shared three-stage scaffold of §3.2 is a faithful model of professional drafting. The paper’s own ablation (Appendix C.3, Table 6) already shows that removing the feature-extraction stage from the implicit prompt slightly raises human scores (average 93.1 vs 92.7). Without a control that holds an independently validated alternative process fixed while varying only externalization, the observed gap may partly reflect forced commitment to a misspecified plan rather than a pure cost of verbalization. Strengthening or qualifying this isolation is load-bearing for the bottleneck interpretation.
[§4.3 and Table 1] Human expert evaluation (§4.3) is the primary evidence for both the quality gains of implicit CoT and the degradation under explicit CoT (Table 1). Each of the 120 samples is scored by only a single patent attorney; no inter-rater reliability, adjudication protocol, or confidence intervals beyond per-sample standard deviations are reported. Given the modest absolute differences on some models and the centrality of these scores, the single-rater design leaves the human results more vulnerable to individual idiosyncrasy than is ideal for the paper’s strongest claims.

minor comments (4)

[Title and tables] Throughout the manuscript (title, section headings, running headers) the word “Verbalization” appears with a spurious space (“V erbalization”). The same artifact occurs for “Average” in table headers. These are almost certainly formatting/OCR artifacts and should be corrected.
[Table 8] The explicit-CoT prompt template in Table 8 contains the typo “Ouput thinking”. Minor prompt-text errors of this kind should be cleaned before camera-ready.
[§4.3 / Table 2] The stepwise diagnostic (Table 2) is performed only on explicit CoT. A short note clarifying why an analogous internal-state probe for implicit CoT is infeasible (or a proxy) would help readers interpret the cascade analysis.
[§4.1] References to the authors’ own prior datasets and metrics (EPD, HUPD-DCG, PatClaimEval) are appropriate, but a one-sentence reminder that these are publicly released benchmarks would further reduce any appearance of circularity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: purely comparative empirical study of prompting variants against external gold claims and independent benchmarks.

full rationale

The paper's central claims (implicit CoT outperforms explicit CoT; verbalization creates an information bottleneck via abstraction, pattern disruption, and error cascade) rest on controlled comparisons of three prompting conditions (plain, implicit CoT, explicit CoT) that share the same three-stage scaffold, evaluated against granted gold claims on the EPD and HUPD-DCG datasets plus human expert ratings and PatClaimEval/LLM-as-judge metrics. There is no derivation, equation, or fitted parameter that is later re-presented as a prediction. Self-citations (Jiang et al. 2025a/c/d) supply the public datasets and evaluation protocol used as external benchmarks; they do not supply a uniqueness theorem, ansatz, or load-bearing premise that forces the implicit>explicit result. The three-stage scaffold is an explicit design choice motivated by drafting practice, not derived from the outcome it is used to measure. Ablations and stepwise diagnostics further test rather than tautologically restate the claim. Consequently the derivation chain contains no self-definitional loop, fitted-input-as-prediction, or self-citation that collapses the result by construction.

Assumptions & free parameters 3 free parameters · 3 assumptions · 2 invented entities

The work is empirical; its load-bearing commitments are the task-specific three-stage scaffold, the operational definitions of implicit vs explicit CoT, and the interpretive claim that intermediate-step diagnostics reveal the three mechanisms. No free physical constants or new particles; free choices are design and evaluation parameters.

free parameters (3)

temperature / top-p / max tokens
Fixed at 0.1 / 0.95 / 4096 for all runs; not swept, so generation stochasticity is controlled but not characterized.
three-stage CoT scaffold
Feature extraction → logical grouping → language formalization is hand-designed by the authors to mirror professional drafting; ablation shows extraction is optional, so the scaffold itself is a free design choice.
human-evaluation sample size (120 / 80)
Random subsets chosen for cost reasons; no power analysis reported.

assumptions (3)

domain assumption Professional patent claim drafting decomposes into feature extraction, logical grouping of dependencies, and formal language expression.
Stated in Section 3.2 as motivation for the three-stage prompt; not independently validated against attorney process studies.
ad hoc to paper When the model is instructed to 'think step-by-step' but output only claims, it actually performs the three stages internally.
Implicit CoT is defined operationally by the prompt; no probing of internal activations is performed.
domain assumption Single licensed patent attorney ratings on content, logic, and language are a reliable gold standard for claim quality.
Used as primary human metric (Section 4.3); inter-rater reliability not measured.

invented entities (2)

verbalization bottleneck
purpose: Explanatory construct for why explicit CoT underperforms: externalized reasoning compresses rich internal state into a lossy text plan that subsequent generation conditions on.
Introduced in Section 6; supported by diagnostics and case study but not given an independent formal definition or measurement outside the paper’s own experiments.
three mechanisms (abstraction of critical detail, disruption of internalized patterns, cascading error propagation)
purpose: Concrete pathways through which the bottleneck is claimed to operate.
Derived post-hoc from intermediate-step annotations and qualitative comparison (Sections 6.1–6.4); each is plausible but not isolated by controlled intervention.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation." pith.science (2026). https://pith.science/paper/AZX4KG25

@misc{pith2026260710480,
  author       = {Pith},
  title        = {Pith review of: When Reasoning Hurts Legal Drafting: The Verbalization Bottleneck in Patent Claim Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZX4KG25}},
  note         = {Machine review of arXiv:2607.10480}
}

read the original abstract

Patent claim drafting is a challenging legal drafting task that requires technical expertise, precise linguistic control, strict adherence to formal conventions, and the preservation of complex logical relationships among claim elements. While Chain-of-Thought (CoT) prompting has been widely used to improve the reasoning capabilities of large language models (LLMs), recent evidence suggests that its benefits may be limited, or even negative, in highly structured or pattern-sensitive tasks. Therefore, this paper investigates whether CoT prompting benefits patent claim generation. We propose a task-specific CoT method for patent claim generation and evaluate its effectiveness through both automatic metrics and human expert assessment. Our results show that reasoning-enhanced prompting can improve claim quality. Moreover, we demonstrate a counter-intuitive but important empirical finding: implicit CoT, where reasoning is kept internal rather than explicitly verbalized, consistently outperforms explicit CoT. Through systematic analysis, we show that explicit CoT can introduce an unnecessary information bottleneck for claim generation. Verbalized reasoning may compromise the quality of final outputs through three specific mechanisms: abstraction of critical details, disruption of internalized generation patterns, and cascading error propagation. Our findings provide new insights into legal tasks and CoT applications.

Figures

Figures reproduced from arXiv: 2607.10480 by the authors.

**Figure 1.** Overview of the CoT prompting methods. Implicit CoT and Explicit CoT share the same three-step process (Feature Extraction → Logical Grouping → Language Formalization); they differ in whether intermediate reasoning is externalized as text. EPD and are presented in Appendix C. 4.2. Models We mainly focus on Llama-3.1-8B (Dubey et al., 2024), Mixtral-8×7B (Jiang et al., 2024), and GPT-4o (OpenAI, 2024) in the paper to… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 1 linked inside Pith

[1]

findings-emnlp.408/

URL https://aclanthology.org/2025. findings-emnlp.408/. Jiang, L., Scherz, P. A., and Goetz, S. Patent-CR: A dataset for patent claim revision. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:...

doi:10.18653/v1/2025.naacl-long 2025
[2]

naacl-long.116/

URL https://aclanthology.org/2025. naacl-long.116/. Jiang, L., Scherz, P. A., and Goetz, S. Towards better eval- uation for generated patent claims. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3775– 3788, 2025c. Jiang, L., Zhang, C., Scherz, P. A., and Goetz, S. Can large languag...

arXiv doi:10.18653/v1/2026 2025
[3]

0–20: Most essential features are missing or poorly described

Content Coverage (0–100): The extent to which the draft claims encapsulated all critical technical aspects of the invention. 0–20: Most essential features are missing or poorly described. 21–40: Some essential features are present but significant gaps remain. 41–60: Majority of essential features are covered but with minor omissions. 61–80: Almost all ess...
[4]

0–20: Claim structure is confusing or non-compliant; dependencies are incorrect or illogical

Logical Structure (0–100): The degree to which the claim set exhibits a coherent, well-organized, and legally sound logical structure, including appropriate dependencies and hierarchical relationships. 0–20: Claim structure is confusing or non-compliant; dependencies are incorrect or illogical. 21–40: Some structure is evident but many dependency relation...
[5]

0–20: Language is vague, ambiguous, or ungrammatical; terminology is inconsistent or incorrect

Language Quality (0–100): The clarity, precision, and formal appropriateness of the language used in the claim set, including terminology, syntax, and adherence to patent drafting conventions. 0–20: Language is vague, ambiguous, or ungrammatical; terminology is inconsistent or incorrect. 21–40: Basic meaning is somewhat understandable but style is inconsi...
[6]

Read the referenced claims carefully and assume the referenced claims have scores of 100 in all Evaluation Criteria
[7]

Read the draft claims and compare it to the referenced claims
[8]

Example: Referenced Claims:<<Claims>> Draft Claims:<<Claims>> Evaluation Form (scores ONLY): - Content Coverage: X, - Logical Structure: X, - Language Quality: X

Assign a score for each metric based on the Evaluation Criteria. Example: Referenced Claims:<<Claims>> Draft Claims:<<Claims>> Evaluation Form (scores ONLY): - Content Coverage: X, - Logical Structure: X, - Language Quality: X. overall performance. In summary, our findings suggest that logical grouping is the most critical stage. Simplifying the CoT promp...

Show all 16 references

[11]

Output the claims ONLY

Write the claims using formal patent language to ensure clarity, consistency, and technical accuracy. Output the claims ONLY . Explicit CoT Prompt: You are a patent expert. Given the patent description, your task is to write high-quality patent claims. You should think step-by-step:
[13]

Organize these features logically based on their functional relationships and dependencies
[14]

Write the claims using formal patent language to ensure clarity, consistency, and technical accuracy. Example: Input: Patent description:<<Description>> Output: <|beginning of thinking|>Ouput thinking<|end of thinking|> <|beginning of claims|>Output claims<|end of claims|> CoT...
[15]

Organize key technical features logically based on their functional relationships and dependencies
[16]

Output the claims ONLY

Write the claims using formal patent language to ensure clarity, consistency, and technical accuracy. Output the claims ONLY . CoT Prompt without Logical Grouping: You are a patent expert. Given the patent description, your task is to write high-quality patent claims. You shou...
[18]

Output the claims ONLY

Write the claims using formal patent language to ensure clarity, consistency, and technical accuracy. Output the claims ONLY . CoT Prompt without Formalization: You are a patent expert. Given the patent description, your task is to write high-quality patent claims. You should ...
[19]

Identify the key technical features that define the invention and their importance ranking
[20]

Output the claims ONLY

Organize key technical features logically based on their functional relationships and dependencies. Output the claims ONLY . 11 The V erbalization Bottleneck of CoT in Patent Claim Generation Table 9.Example model outputs of patent EP3060168B1. Gold claims1. A combination (10)...

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.

[1] [1]

findings-emnlp.408/

URL https://aclanthology.org/2025. findings-emnlp.408/. Jiang, L., Scherz, P. A., and Goetz, S. Patent-CR: A dataset for patent claim revision. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.),Proceedings of the 2025 Conference of the Nations of the Americas Chap- ter of the Association for Computational Linguistics: Human Language Technologies (Volume 1:...

doi:10.18653/v1/2025.naacl-long 2025

[2] [2]

naacl-long.116/

URL https://aclanthology.org/2025. naacl-long.116/. Jiang, L., Scherz, P. A., and Goetz, S. Towards better eval- uation for generated patent claims. InProceedings of the 63rd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3775– 3788, 2025c. Jiang, L., Zhang, C., Scherz, P. A., and Goetz, S. Can large languag...

arXiv doi:10.18653/v1/2026 2025

[3] [3]

0–20: Most essential features are missing or poorly described

Content Coverage (0–100): The extent to which the draft claims encapsulated all critical technical aspects of the invention. 0–20: Most essential features are missing or poorly described. 21–40: Some essential features are present but significant gaps remain. 41–60: Majority of essential features are covered but with minor omissions. 61–80: Almost all ess...

[4] [4]

0–20: Claim structure is confusing or non-compliant; dependencies are incorrect or illogical

Logical Structure (0–100): The degree to which the claim set exhibits a coherent, well-organized, and legally sound logical structure, including appropriate dependencies and hierarchical relationships. 0–20: Claim structure is confusing or non-compliant; dependencies are incorrect or illogical. 21–40: Some structure is evident but many dependency relation...

[5] [5]

0–20: Language is vague, ambiguous, or ungrammatical; terminology is inconsistent or incorrect

Language Quality (0–100): The clarity, precision, and formal appropriateness of the language used in the claim set, including terminology, syntax, and adherence to patent drafting conventions. 0–20: Language is vague, ambiguous, or ungrammatical; terminology is inconsistent or incorrect. 21–40: Basic meaning is somewhat understandable but style is inconsi...

[6] [6]

Read the referenced claims carefully and assume the referenced claims have scores of 100 in all Evaluation Criteria

[7] [7]

Read the draft claims and compare it to the referenced claims

[8] [8]

Example: Referenced Claims:<<Claims>> Draft Claims:<<Claims>> Evaluation Form (scores ONLY): - Content Coverage: X, - Logical Structure: X, - Language Quality: X

Assign a score for each metric based on the Evaluation Criteria. Example: Referenced Claims:<<Claims>> Draft Claims:<<Claims>> Evaluation Form (scores ONLY): - Content Coverage: X, - Logical Structure: X, - Language Quality: X. overall performance. In summary, our findings suggest that logical grouping is the most critical stage. Simplifying the CoT promp...