{"id":"65b0f91c-32de-4166-a153-7d6cafcfaf83","arxiv_id":"2505.12568","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A new European patent dataset improves claim-generation models on European and US test sets, with a difficult subset exposing remaining weaknesses.","lead":"This paper introduces EPD, a new dataset of European patents granted in 2024, and uses it to fine-tune language models for patent claim generation. The authors report that models trained on EPD generate better claims on European patents than models trained on US data, and even outperform GPT-4o.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'granted vs. application' quality claim is confounded: no within-EPD application/granted comparison, and HUPD-DCG's grant status is internally contradictory.","rationale":"We read the central claim as one about data quality: granted EPO claims are better training data than USPTO applications because they are legally finalized. The experiments do not isolate grant status because EPD and HUPD-DCG differ in jurisdiction, time period, and grant status simultaneously. We agree with the reader's weakest-assumption analysis. We add two sharpenings: (i) the manuscript contradicts itself about whether HUPD-DCG contains granted or application claims (Section 4.1 vs Table 1/Section 5.1), so the comparison is not even well-defined; (ii) the dataset already contains application-granted pairs, so a controlled test is available but unused. The absence of significance testing matters, but we treat it as secondary because the primary issue is causal attribution. The dataset itself is a genuine contribution, and the difficult-subset analysis is useful, so the verdict should remain CONDITIONAL: the paper's claims about quality and cross-domain generalization require the controlled experiment described above.","tokens_in":20215,"tokens_out":7077,"duration_ms":68933,"concrete_test":"Within the 8,007 EPD samples filtered as in Section 4.1, use the 32,988 application-and-granted pairs to build an 'application-claims' training set of 6,972 patents (matched to the granted training set by publication number), fine-tune Llama-3.1-8B with the same LoRA settings, and evaluate on the same 1,035-patent EPD test set using all four metrics and the DeepSeek judge. If the application-trained model's scores are within noise of the granted-trained model's scores, the 'high-quality granted claims' advantage is not supported; if granted-trained clearly wins, the confound is less severe. Run each condition over at least three seeds and report mean ± std.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that fine-tuning on EPD's granted EPO claims yields higher quality and better cross-domain generalization because granted claims are legally finalized ('Quality improvement' in the abstract). The evidence compares EPD (EPO, granted, 2024) with HUPD-DCG (USPTO, applications, 2017) — jurisdiction, grant status, and time all vary, so the outcome cannot be attributed to grant status. This attribution is not merely under-supported; the manuscript is internally inconsistent about HUPD-DCG: Section 4.1 says it consists of patents 'granted by the USPTO', while Table 1 and Section 5.1 describe it as 'application-stage claims'. The paper never resolves this, so the 'granted vs. application' distinction driving the quality narrative is ill-defined. Crucially, EPD itself contains 32,988 application-and-granted pairs (Table 2), enabling a clean within-EPO, within-year control, yet Section 4.1 uses only granted versions for claim-generation fine-tuning, and Appendix D's DPO experiments do not compare fine-tuning on applications versus granted claims. Without that control, the superior performance of the EPD model could equally stem from EPO drafting conventions or 2024-era patent language rather than grant status. In addition, no significance tests or multiple seeds are reported, and the 'cross-domain generalization' claim rests on a single DeepSeek-V3 judge that contradicts the BLEU/ROUGE/BERTScore pattern in Table 4. These issues collectively mean the central causal claim is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EPD, a new dataset of 73,980 European Patent Office (EPO) patents granted in 2024, of which 32,988 have both application and granted versions. The authors evaluate patent claim generation by fine-tuning Llama-3.1-8B on EPD, on the existing HUPD-DCG dataset, and on a mixed dataset, comparing against GPT-4o and SaulLM-7B. They report that the EPD-trained model achieves the highest BLEU, ROUGE, and BERTScore on the EPD test set (BLEU 47.22, R-1 67.94, R-L 51.52, BERTScore 90.40) and also receives the highest LLM-as-a-judge scores on most dimensions on both EPD and HUPD-DCG. They further introduce a 'difficult' subset of EPD based on a ROUGE-L threshold and show that all models degrade on it. The paper also reports auxiliary experiments on patent classification, summarization, and DPO-based RLHF. The central claim is that fine-tuning on EPD's granted claims improves generation quality and cross-domain generalization relative to training on application-stage USPTO claims.","tokens_in":20549,"tokens_out":3516,"duration_ms":34736,"significance":"If substantiated, the paper would provide a valuable and reusable resource: a large, recent, multi-purpose European patent dataset with aligned application/granted pairs, detailed documentation, and experiments across several patent NLP tasks. The construction pipeline is described in sufficient detail to be reproducible, and the authors commit to releasing code and publication numbers. The cross-jurisdiction comparison is a genuinely useful direction for the field. However, the headline claim that EPD improves quality because it contains granted rather than application claims is currently not identifiable from the experiments, because the comparison to HUPD-DCG varies jurisdiction, grant status, and time period simultaneously. The lack of significance tests and the reliance on an unvalidated single LLM judge further weaken the quantitative claims. The dataset contribution stands on its own; the causal quality claim needs additional controlled experiments.","major_comments":[{"comment":"The central quality comparison is confounded. The paper attributes the performance advantage of Llama-3.1-8B-FT (EPD) to granted, legally finalized EPO claims, but the comparison dataset HUPD-DCG differs in jurisdiction (EPO vs. USPTO), grant status (granted vs. application), and time period (2024 vs. 2017) simultaneously. Even if the performance difference is real, it cannot be attributed to grant status without a controlled comparison. EPD itself contains 32,988 paired application and granted versions (Table 2), so a within-EPO, same-year comparison of fine-tuning on application claims versus granted claims is feasible and should be reported. Without such an experiment, the abstract and Section 5.1's claim that 'high-quality training data' (i.e., granted claims) drive the gains is unsupported.","section":"Section 4.1 and 5.1"},{"comment":"The manuscript is internally inconsistent about HUPD-DCG's grant status. Section 4.1 states that HUPD-DCG consists of 'patent documents filed in 2017 and granted by the USPTO,' while Table 1 and Section 5.1 describe it as consisting of 'application-stage claims.' These statements are mutually contradictory. Since the entire 'granted vs. application' quality narrative depends on this distinction, the inconsistency is load-bearing. The authors must correct this and specify exactly which version of USPTO documents HUPD-DCG contains; if HUPD-DCG actually contains granted claims, the quality-improvement argument is further weakened.","section":"Section 4.1 vs. Table 1 and Section 5.1"},{"comment":"No statistical significance testing, confidence intervals, or multiple seeds are reported. For example, on the EPD test set, Llama-3.1-8B-FT (EPD) and Llama-3.1-8B-FT (Mixed) differ by only 0.18 BLEU and 0.83 R-1, which is likely within run-to-run variation for LoRA fine-tuning. The word 'significantly' in the abstract is therefore not supported by the evidence presented. I request error bars or significance tests (e.g., bootstrap or paired tests across test samples), or at least multiple fine-tuning seeds for the main comparisons.","section":"Tables 4 and 5, Section 5.1"},{"comment":"The cross-domain generalization claim rests primarily on a single DeepSeek-V3 judge, which is not validated against human judgments in this paper. The judge's conclusion that Llama-3.1-8B-FT (EPD) outperforms Llama-3.1-8B-FT (HUPD-DCG) on the HUPD-DCG test set directly contradicts Table 4, where the HUPD-trained model has higher BLEU (40.54 vs. 34.05), R-1 (61.89 vs. 58.95), R-L (48.70 vs. 45.16), and BERTScore (89.89 vs. 87.99). Given that lexical metrics and the LLM judge point in opposite directions, a human evaluation or calibration of the judge against human preferences is necessary before claiming cross-domain generalization.","section":"Section 5.1, Figures 1-3"}],"minor_comments":[{"comment":"The difficult-subset threshold ROUGE-L > 0.6 is justified by reference to prior work and Table 4, but this is circular in spirit: the threshold is based on model performance, and the same models are then evaluated on the resulting split. Reporting sensitivity to the threshold (e.g., 0.5, 0.7) would strengthen the construction.","section":"Section 4.1"},{"comment":"Table 2 notes that granted patents in EPD 'do not include the abstract section,' yet Table 1 lists EPD as containing an Abstract. This is confusing and should be clarified in both places; the abstract appears to exist only in the application versions.","section":"Section 3.2"},{"comment":"The LLM-as-a-judge prompt in Table 13 lists eight evaluation criteria, including 'Conciseness,' but the text in Section 4.3 lists seven dimensions and does not mention conciseness. The number of criteria should be made consistent, and it should be stated which scores were actually used in the analysis.","section":"Appendix G"},{"comment":"The 'Lower Risk of Data Leakage' argument is plausible but not verified. Since GPT-4o-2024-08-06 is evaluated on 2024 patents, the paper should acknowledge that the training data cutoff of commercial models is not publicly documented, so leakage cannot be ruled out with certainty.","section":"Section 1"},{"comment":"The DPO experiments are interesting negative results, but they are described as using application-to-granted pairs as preference data; this is a different use of the paired data than the fine-tuning comparison I request in the major comments, and it does not answer the question of whether fine-tuning on application claims versus granted claims affects claim generation quality.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is potentially solid, and the authors have assembled a useful resource with clear documentation. However, the paper's central quality-attribution claim is currently underidentified, and the internal inconsistency about HUPD-DCG's grant status is concerning. The authors can realistically fix this with the EPD paired data already in hand, plus human evaluation or judge calibration. I also recommend the editor ensure that the revised version reports multiple seeds or significance tests, as the current single-run results are not convincing for the word 'significantly.'"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: EPD is a genuinely useful new resource. It is the first EPO granted-patent dataset aimed at claim generation, with 2024 data and both application and granted versions for roughly 33K patents. The dataset construction is documented well, the release via OPS publication numbers is clean, and the additional tasks (classification, summarization, retrieval) make it a solid multi-purpose asset. I would use this dataset.\n\nThe paper's causal claim, that granted EPO claims are higher-quality training data, which is why fine-tuning on EPD beats fine-tuning on HUPD-DCG, is not established. The comparison confounds jurisdiction, grant status, and time period. HUPD-DCG is USPTO applications from 2017; EPD is EPO granted from 2024. There is also an internal contradiction: Section 4.1 calls HUPD-DCG 'granted by the USPTO', while Table 1 and Section 5.1 call it 'application-stage claims.' The paper never resolves this. Since EPD itself contains 32,988 application-granted pairs, the authors could run a within-EPO, within-year control, fine-tuning on applications versus granted claims. That would actually isolate grant status. Without it, the 'granted quality' story is speculative.\n\nOther soft spots: no significance tests or multiple seeds, so 'significantly outperform' is unsupported statistically. The LLM-as-a-judge (DeepSeek-V3) is not human-validated here, and on the HUPD-DCG test set it rates FT(EPD) higher while BLEU/ROUGE/BERTScore all favor FT(HUPD-DCG); that discrepancy needs explanation. Comparing fine-tuned Llama against zero-shot GPT-4o is also not a meaningful 'surpass' claim. The ROUGE-L 0.6 threshold for the difficult split is anchored in prior work, which is acceptable, but it is a free parameter.\n\nThe DPO appendix is honest: it reports that DPO hurts performance and that 98% reward accuracy suggests superficial overfitting. That is good scientific practice.\n\nWho this is for: researchers working on patent NLP, especially claim generation, who need a non-US benchmark. The paper deserves peer review; a serious referee should require the within-EPO control and resolution of the HUPD-DCG contradiction before publication. But the resource itself is worth engaging with now.","headline":"EPD is a genuinely useful new dataset for patent NLP, but the paper's central claim that granted EPO claims are higher-quality training data is confounded and internally inconsistent.","tokens_in":678,"tokens_out":1169,"would_cite":true,"duration_ms":31203,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning on EPD, a dataset of European granted patents, yields patent claim generation that beats models trained on US application data and GPT-4o in quality and cross-domain generalization.","keywords":["patent claim generation","European Patent Office","EPD dataset","large language models","fine-tuning","cross-jurisdiction generalization","LLM-as-a-judge evaluation"],"falsifier":"Train models on four matched cells—EPO applications, EPO granted claims, US applications, and US granted claims from the same years and technical fields—under identical fine-tuning settings. If granted-version training does not beat application-version training within each office, or if EPD-trained gains disappear on the difficult subset, the central quality-and-generalization claim would be refuted.","tokens_in":20052,"feed_emoji":"⚖️","tokens_out":9299,"duration_ms":87530,"temperature":0.7,"pith_summary":"This paper tries to establish that patent claim generation improves when training data comes from granted, legally finalized European claims rather than from US patent applications. It introduces EPD, a dataset of about 107,000 documents drawn from roughly 74,000 European patents granted in 2024, and reports that a Llama-3.1-8B model fine-tuned on EPD outperforms the same model fine-tuned on the earlier US application dataset HUPD-DCG, and also outperforms GPT-4o, on claim quality and cross-jurisdiction generalization. The paper adds a difficult subset, defined by the first claim not being nearly extractable from the description, and shows that every tested model degrades sharply on it. If the central claim holds, the field gains a higher-quality, leakage-resistant, cross-jurisdiction benchmark for a costly professional writing task.","feed_headline":"European granted-patent data beats US-only claim training","feed_subtitle":"A new European Patent Office dataset lifts claim quality and cross-jurisdiction generalization past GPT-4o.","key_machinery":"The central object is EPD itself: about 107,000 patent documents from roughly 74,000 European patents granted between January and August 2024, with titles, claims, descriptions, classifications, citations, and application/granted pairs where available. For claim generation, the paper filters to descriptions under 8,000 tokens, splits into 6,972 training and 1,035 test examples, and fine-tunes Llama-3.1-8B with LoRA on the description-to-claims task. The other mechanism is the difficult-subset split: a patent is easy if any description sentence has ROUGE-L above 0.6 against the first claim and difficult otherwise, which isolates reconstruction from near-verbatim extraction.","core_discovery":"The paper's central claim is that fine-tuning on EPD—a collection of English-language European patents granted in 2024, including both application and granted versions—produces patent claims that are better than those from models fine-tuned on HUPD-DCG (US application-stage claims) and better than GPT-4o. On the EPD test set, the EPD-tuned model reaches BLEU 47.22, ROUGE-1 67.94, ROUGE-L 51.52, and BERTScore 90.40; in LLM-as-judge ratings it leads on content, precision, specificity, dependency, terminology, and scope. When evaluated on the US test set, the EPD-trained model receives higher judge ratings than the HUPD-DCG-trained model on all seven dimensions even though surface overlap scores are lower, which the paper interprets as evidence that high-quality granted claims transfer across jurisdictions. The paper also constructs a difficult subset by flagging patents with no description sentence above ROUGE-L 0.6 against the first claim, and reports that all tested models degrade substantially on those samples.","pith_inferences":["A direct test of the paper's quality explanation would compare EPO application versions against EPO granted versions only; the 32,988 paired documents already included in EPD make that experiment runnable without new collection.","The paper's DPO results suggest naive preference learning on application-to-granted pairs can hurt; an inference is that carefully reformulated preference signals might still exploit those pairs.","If EPD-trained quality transfers to US claims through legal-polish patterns rather than format mimicry, zero-shot evaluation on granted claims from a third jurisdiction should also improve; this is testable.","The ROUGE-L 0.6 threshold is a coarse proxy for extractability; decomposing claims into technical features would likely separate extraction from reconstruction more sharply."],"forward_implications":["EPD gives claim-generation research a European benchmark, so future systems can be compared across European and US drafting conventions.","If grant status is the real driver, dataset builders should prefer legally finalized text over raw applications, since canceled or amended claims in applications weaken training.","The difficult subset measures genuine claim construction rather than extraction, and the large performance drop gives a concrete target for future methods.","Because EPD's patents were granted in 2024, evaluations on it are less likely to be contaminated by LLM pretraining corpora, making reported gains more credible."],"supporting_citations":[{"why":"Supplies HUPD-DCG, the US application-stage dataset that is the comparison baseline, and defines the description-based claim generation task.","marker":"Jiang et al. (2025b)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to train all fine-tuned models.","marker":"Hu et al. (2021)"},{"why":"Provides BLEU, the primary surface-level overlap metric in the comparisons.","marker":"Papineni et al. (2002)"},{"why":"Provides ROUGE metrics and the ROUGE-L threshold used to build the difficult/easy split.","marker":"Lin (2004)"},{"why":"Provides BERTScore, the semantic similarity metric reported alongside BLEU and ROUGE.","marker":"Zhang et al. (2019)"},{"why":"Provides the Deepseek-V3 model used as the LLM-as-a-judge in the qualitative evaluation.","marker":"Liu et al. (2024)"}],"fun_headline_variants":["European patent data sharpens AI claim drafting","EPD beats US data for patent claim generation","Granted European patents lift claim quality past GPT-4o","New European patent dataset boosts LLM claims","European-granted patents outperform US in claim training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument that granted European claims are better training data rests on comparing EPD with an older US application dataset, so grant status is entangled with jurisdiction and time; without controls separating these, the quality explanation is not yet established.","fun_headline_variants_meta":{"raw":{"variants":["European patent data sharpens AI claim drafting","EPD beats US data for patent claim generation","Granted European patents lift claim quality past GPT-4o","New European patent dataset boosts LLM claims","European-granted patents outperform US in claim training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000419,"raw_usage":{"total_tokens":2187,"prompt_tokens":1004,"completion_tokens":1183,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1110}},"tokens_in":620,"tokens_out":1183,"duration_ms":8497,"temperature":1.0,"reasoning_tokens":1110,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:36.591333+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train models on four matched cells—EPO applications, EPO granted claims, US applications, and US granted claims from the same years and technical fields—under identical fine-tuning settings. If granted-version training does not beat application-version training within each office, or if EPD-trained gains disappear on the difficult subset, the central quality-and-generalization claim would be refuted.","supporting_citations":[],"review_version":1}