{"id":"61ed5541-ba99-48aa-bfca-94ba8557303d","arxiv_id":"2412.09796","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multi-agent framework with planning, writing, retrieval, and reviewing agents generates 17K-token patents from drafts and reportedly outperforms GPT-4o when powered by a 7B model, per the new D2P benchmark.","lead":"AutoPatent turns an inventor's rough draft into a full patent document, and the authors built a 1,933-pair benchmark from granted U.S. patents to test it. On that benchmark, their multi-agent system based on a small 7B model outscored much larger models like GPT-4o in expert preference.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Synthetic drafts may leak reference content, inflating reported gains: Section 3.2 builds drafts by asking GPT-4o-mini to summarize the very patent being generated, so the task risks rewarding reconstruction fidelity rather than real drafting ability.","rationale":"The reader identified the synthetic draft construction as the weakest assumption; I agree, and I treat it as the single most load-bearing concern because it threatens the external validity of every reported comparison, including the headline claim. Unlike the length confound or missing inter-annotator agreement (which affect interpretation of specific numbers), the synthetic-draft issue determines whether the task being evaluated is the task claimed in the introduction. The paper itself admits that drafts are generated with GPT-4o-mini from the patents and filtered by GPT-4o (Section 3.2), and that the training set (1,500 pairs) fine-tunes the component writers and planner. The RRAG module explicitly copies from the draft and generated short components into the description, so the framework is heavily optimized for reconstructing a known structure. The proposed concrete test is feasible because the model weights, prompts, and pipeline are described in detail; the only missing ingredient is a new draft source, which the authors can construct cheaply. I would keep the verdict CONDITIONAL, because the human evaluation does show consistent wins and the framework is plausible, but the central claim should not be accepted as stated until the benchmark's construct validity is demonstrated.","tokens_in":18185,"tokens_out":1649,"duration_ms":16013,"concrete_test":"Re-run the comparison on held-out cases where the inventor draft is constructed independently of the target patent, e.g., have a different LLM (or human technical writer) produce a draft from only title, abstract, claims, and figures of a withheld patent, without access to the granted full text. If AutoPatent-Qwen2.5-7B's win rate over GPT-4o zero-shot drops substantially (from majority wins to near chance), the claimed superiority is an artifact of the synthetic draft construction. A cheaper secondary check: measure n-gram overlap between synthetic drafts and reference patents; if BLEU(draft, reference) is high, the benchmark is a summarization-expansion task.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that AutoPatent with Qwen2.5-7B beats much larger zero-shot models in both objective and human evaluation. The load-bearing precondition is that D2P measures real Draft2Patent capability, where an inventor draft contains genuine but incomplete invention information. Section 3.2 violates this: drafts are produced by prompting GPT-4o-mini with five questions about the granted patent itself, then filtered by GPT-4o. The generated draft is therefore not an independent inventor disclosure but a query-conditioned summary of the reference patent, with high lexical and semantic overlap. The task reduces to expanding an LLM-generated outline of the target patent back into a document similar to that patent. Because the AutoPatent fine-tuning (short-component writers, planner, and RRAG retrieval) is trained on target patents, and RRAG retrieves draft and short components verbatim, the pipeline is optimized precisely for the setting where the draft is a compressed version of the reference. BLEU/ROUGE gains then reflect reconstruction fidelity to a text the draft was derived from, not superiority in drafting from real inventor notes. Human evaluations may partly reward completeness and formatting, which can also be artifacts of training on the target distribution. This is an external-validity threat rather than an internal inconsistency, but it directly undermines the abstract's claim that AutoPatent generates better patents than larger LLMs from an inventor's draft.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces Draft2Patent, a new task in which a system converts an inventor's technical draft into a complete patent, and presents the D2P benchmark with 1,933 draft-patent pairs built from HUPD. The authors propose AutoPatent, a multi-agent framework with a planner agent that builds a Patent Writing Guideline Tree (PGTree), five short-component writer agents, a description writer, and an examiner agent that iteratively reviews subsections; a retrieval module (RRAG) conditions description writing on earlier components and the draft. On the D2P test set, AutoPatent with Qwen2.5-7B achieves the highest BLEU/ROUGE/IRR scores among the compared systems and, in a 50-patent pairwise human evaluation, is judged better than zero-shot or SFT baselines from GPT-4o, Qwen2.5-72B, and LLAMA3.1-70B.","tokens_in":18581,"tokens_out":7709,"duration_ms":78283,"significance":"The paper makes a coherent engineering contribution: the modular writer/planner/examiner decomposition, PGTree, and RRAG are natural and promising designs for long structured generation, and the ablation studies show large improvements from these components. The IRR metric is a useful and appropriate addition that correctly exposes the repetition failure of SFT. The D2P benchmark is the first of its kind at this scale and could be valuable if used with appropriate caveats. However, the headline claims about patent quality and about superiority over larger LLMs rest on two validity pillars that are currently weak: benchmark construction, where drafts are generated from the reference patents themselves, and evaluation, where n-gram metrics are length-confounded and the human study is small and under-reported. The central contribution is therefore plausible but not yet established.","major_comments":[{"comment":"Section 3.2 constructs each draft by asking GPT-4o-mini to answer five questions about the granted patent being generated, and then uses GPT-4o to filter drafts for informativeness. The D2P task therefore asks models to reconstruct a patent from a query-conditioned summary of that same patent, rather than to draft a patent from an independent inventor's disclosure. Because the AutoPatent pipeline is trained on the target patents and retrieves draft and short-component text verbatim in RRAG, the high BLEU/ROUGE scores in Table 2 may reflect how faithfully the system expands an LLM summary of the reference back into the reference, not how well it drafts from real inventor notes. This is an external-validity threat to the abstract's central claim, and I would like to see (i) quantitative overlap statistics between drafts and reference patents (e.g., Jaccard or BLEU), (ii) an additional evaluation on drafts produced without conditioning on the target patent text, and (iii) a revised abstract and conclusion that limit the claim to the D2P benchmark unless such evidence is provided.","section":"Section 3.2"},{"comment":"Table 2 shows that zero-shot baselines produce 1,247-3,804 tokens on average, while AutoPatent outputs 11,433-15,481 tokens, and SFT outputs are also long. Section 5.1 itself notes that n-gram metrics reward repeated n-grams and short sentences. This makes the objective-metric comparison in Table 2 length-confounded: a long document that reuses draft phrases can mechanically inflate precision-based metrics, and ROUGE recall can be inflated by length. The SFT rows illustrate the same problem: LLAMA3.1-8B+SFT reaches BLEU 39.62 with 17,052 tokens, but its IRR of 49.17 at t=0.2 shows heavy repetition, and the case study confirms poor quality. The paper therefore needs length-controlled or length-normalized comparisons (e.g., truncation to a common token budget, or per-chunk matching), and IRR should be reported for every row in Table 2 before the cross-model superiority claim can be accepted.","section":"Table 2 and Section 5.1"},{"comment":"The human evaluation is based on only 50 generated patents and three experts, with no inter-annotator agreement, no confidence intervals, and no significance tests. The reported win counts for AutoPatent over models such as Qwen2.5-72B or LLAMA3.1-70B are moderate and could be within sampling variation; the paper should report exact counts, per-expert agreement, and a paired significance test. In addition, because AutoPatent outputs are systematically much longer and include full sections and flowcharts, the 'single-bind' protocol may not prevent evaluators from identifying the AutoPatent condition; this should be acknowledged and, ideally, controlled by matching output length or by asking experts to evaluate sections independently.","section":"Section 5.1 and Figure 4"},{"comment":"The examiner agent described in Section 4.1 and Appendix B.4 checks a rubric of accuracy, logic, comprehensiveness, clarity, coherence, consistency, and alignment with the draft; it does not evaluate novelty, non-obviousness, enablement, or prior art, which are the patentability requirements cited in the Introduction. Yet the Introduction and Conclusion describe AutoPatent as generating 'high-quality complete patent documents' and state that 'the quality of patents generated by AutoPatent rivals that of human authorship.' These claims are not supported by the evaluation, and they are in direct tension with the Ethics Statement, which says the generated patents are 'not yet sufficient to be submitted directly to the IP office' and 'still require modification by a patent agent.' The conclusions should be softened, or the examiner rubric and the benchmark should be extended to cover substantive patentability criteria.","section":"Section 4.1, Appendix B.4, Conclusion"}],"minor_comments":[{"comment":"The notation C2_n in Eq. (2) is not defined; please state explicitly that it is the binomial coefficient (n choose 2) and define the smoothing term epsilon more precisely.","section":"Section 5.1, Eq. (2)"},{"comment":"Figure 2 contains untranslated Chinese text in the diagram and caption (e.g., 'Subsection，质量评估，再返回修改，直到通过'); this should be in English.","section":"Figure 2"},{"comment":"'single-bind review' appears to mean 'single-blind'; please correct the wording.","section":"Section 5.1"},{"comment":"The phrase 'observing significant improvement' is used for BERTScore and other metrics without a statistical test; please describe the evaluation protocol or avoid the word 'significant.'","section":"Section 6.1 and Table 4"},{"comment":"The quality requirement for Question 5 restates the technical-solution requirement rather than requiring figure-by-figure descriptions, which is inconsistent with the definition of q5 in Table 5; please correct the prompt or clarify the intended mapping.","section":"Appendix A.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong engineering core, but the headline claim is broader than the evidence supports. The most serious issue is the closed-loop benchmark construction in Section 3.2, which makes the D2P task a reconstruction task from an LLM summary of the reference patent. I would not accept the paper in its current form; a major revision should add an external-validation experiment or substantially reframe the claims. If the released data contains the GPT-4o-mini-generated drafts, the leakage is baked into the benchmark and should be documented prominently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Put time into this one if you care about long-document generation benchmarks. The novelty is real: Draft2Patent is the first full-length patent generation task from inventor-style drafts, and the D2P benchmark (1,933 pairs, 17K target tokens) fills a gap between claims-only, abstract-to-claims, and paper-to-patent tasks. AutoPatent, with its planner (PGTree), reference-retrieval (RRAG), and examiner loop, is a sensible multi-agent design, and the ablation study gives it internal support. The paper also introduces a new repetition metric (IRR) and uses it to explain why SFT baselines look deceptively good under BLEU/ROUGE. That is honest and useful.\n\nThe soft spots are concentrated in the benchmark construction. Section 3.2 generates each draft by asking GPT-4o-mini to answer five questions about the target patent itself, then filters answers with GPT-4o. So the task is reconstruction of a patent from an LLM summary of that same patent, not drafting from an independent inventor disclosure. This is an external-validity threat to the practical claim, but not an internal contradiction—the benchmark still measures controlled long-document expansion. The abstract overstates by saying 'based on initial drafts' when those drafts are derived from the reference. Also, the objective metric comparisons are confounded by output length: zero-shot models produce 1–4K tokens, AutoPatent 11–15K, and n-gram metrics reward longer outputs. The paper acknowledges repetition biases but does not length-normalize. Human evaluation uses three experts with no inter-annotator agreement reported, and the claim that AutoPatent 'rivals human authorship' is not backed by a direct comparison against human-written patents. Data and code are not yet released, which limits reproducibility.\n\nThe citation pattern looks fine: they cite Pap2Pat, Lee and Hsiang, and related agent frameworks. If the authors release code and data, add a length-controlled or semantic metric, and either validate on a small set of genuine inventor drafts or reframe the task as patent reconstruction, the paper would be solid.\n\nWho is this for: NLP researchers working on long-document generation and multi-agent frameworks, plus people in IP automation. It deserves peer review—the task and benchmark are worth refereeing despite the synthetic draft issue. I would engage with it and ask for those revisions before accepting.","headline":"A genuinely new benchmark and a solid multi-agent pipeline for long-document patent generation, but the synthetic drafts are derived from the target patents themselves, which undermines the real-world claim and inflates the headline results.","tokens_in":19009,"tokens_out":2171,"would_cite":true,"duration_ms":24153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AutoPatent shows that a 7B model can beat GPT-4o at writing complete patents when generation is split into planner, writer, and examiner agents.","keywords":["patent generation","multi-agent LLM framework","Draft2Patent","D2P benchmark","PGTree","RRAG","long document generation"],"falsifier":"Run AutoPatent and zero-shot GPT-4o on a held-out set of genuine pre-filing inventor drafts whose corresponding patents later granted, and have patent-law experts blind-judge which output better matches the disclosure; if Qwen2.5-7B+AutoPatent does not win the majority of expert preferences, the central claim that the framework transfers to real inventor drafting collapses.","tokens_in":17966,"feed_emoji":"📄","tokens_out":8423,"duration_ms":81663,"temperature":0.7,"pith_summary":"The paper proposes a new task, Draft2Patent, in which a model must expand an inventor's technical draft into a complete patent, and builds a 1,933-pair benchmark (D2P) from accepted public patent records. It then introduces AutoPatent, a multi-agent framework that decomposes generation into a planning agent, six writer agents, and an examiner agent. The central claim is that this structure, not raw model scale, matters: Qwen2.5-7B running AutoPatent outperforms zero-shot GPT-4o, Qwen2.5-72B, and LLAMA3.1-70B on n-gram metrics, repetition rate, and blind human expert review. If true, long, highly constrained legal-technical documents become tractable for small open models.","feed_headline":"A 7B model pipeline beats GPT-4o at full-patent drafting","feed_subtitle":"Planner, writer, and examiner agents turn short inventor drafts into complete 17K-token patents.","key_machinery":"The load-bearing mechanisms are PGTree (Patent Writing Guideline Tree) and RRAG (Reference-Review-Augmented Generation). PGTree is a two-layer outline — sections containing subsections — that a planning agent generates from the draft so the 14K+ token description can be written subsection by subsection rather than in one pass. RRAG is the write loop: the description writer retrieves relevant chunks from the reference set (draft plus already-written components), drafts a subsection, the examiner agent reviews it against accuracy, logic, comprehensiveness, clarity, coherence, and consistency, and the writer revises until the examiner passes it. The six writer agents split the remaining short components by style, and the examiner agent also serves as draft-quality gatekeeper during benchmark construction.","core_discovery":"On the paper's own terms, the discovery is that a complete patent — title, abstract, background, summary, claims, and a detailed description averaging over 14K tokens of the roughly 17K total — can be generated by composing small models into specialized roles. AutoPatent's planner builds a two-level PGTree outline of the description; its writers fill each subsection by retrieving from the draft plus the already-generated components; its examiner agent rejects and requests rewrites until each subsection passes. In the D2P test set, the configuration built on Qwen2.5-7B scored above GPT-4o, GPT-4o-mini, Qwen2.5-72B, and LLAMA3.1-70B on BLEU, ROUGE-1/2/L, and IRR, and three human experts familiar with patent law preferred the AutoPatent output in blind pairwise comparisons. The authors also report that removing PGTree collapses generation to under 2K tokens and that removing RRAG lowers all objective scores.","pith_inferences":["Because every draft in D2P was produced by asking an LLM to summarize the final patent, the benchmark may reward reconstructing the reference text; a real inventor's draft would likely be noisier, so deployment gains could be smaller.","The same agent decomposition — outline planning, retrieval from a fixed reference set, and examiner-style review — is a general recipe for other long regulated documents, such as clinical study protocols or regulatory submissions.","A decisive test is to run AutoPatent on genuine pre-filing inventor drafts; if the 7B pipeline no longer beats frontier zero-shot models, the paper's headline claim would be confined to its synthetic benchmark."],"forward_implications":["A 7B open-weight model composed with a planning, retrieval, and review pipeline can generate structured long documents rated above frontier zero-shot outputs, suggesting model scale is not the binding constraint.","The D2P benchmark gives the community a reproducible stress test for tasks requiring 10K+ tokens of highly constrained, domain-specific writing.","If the pipeline is applied beyond patents, it offers a template: decompose a long document into an outline, retrieve from a trusted reference set, and loop an examiner-style reviewer until quality checks pass.","Automatic patent drafting could shorten one of the most labor-intensive steps in intellectual-property practice; however, the authors state the outputs still need human legal review before submission."],"supporting_citations":[{"why":"Supplies the accepted granted patents that the D2P benchmark's draft–patent pairs are built from.","marker":"(Suzgun et al., 2023)"},{"why":"GPT-4o-mini generates the five-answer drafts and GPT-4o filters them; GPT-4o also serves as a zero-shot baseline.","marker":"(Achiam et al., 2023)"},{"why":"Provides the Qwen2.5 series, including the 7B base model used in the strongest reported AutoPatent configuration.","marker":"(Yang et al., 2024)"},{"why":"Provides the LLAMA3.1-8B and LLAMA3.1-70B zero-shot and fine-tuned baselines.","marker":"(Dubey et al., 2024)"},{"why":"Provides the Mistral-7B baseline and fine-tuning target model.","marker":"(Jiang et al., 2023)"},{"why":"Supplies the prior planning-based long-writing pipeline that motivates decomposing long outputs into planner-controlled sections.","marker":"(Bai et al., 2024)"},{"why":"Defines BLEU, one of the objective metrics used to compare generated patents.","marker":"(Papineni et al., 2002)"},{"why":"Defines ROUGE-1/2/L, the other objective metrics used in the evaluation.","marker":"(Lin, 2004)"}],"fun_headline_variants":["7B model beats GPT-4o at full patent drafting","AutoPatent: 7B model outwrites GPT-4o on 17K-token patents","Multi-agent pipeline turns drafts into patents, beats GPT-4o","Small model team beats big LLMs at patent writing","Patents via planner, writer, examiner agents: 7B > GPT-4o"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark drafts are not authentic inventor disclosures: they are generated by asking GPT-4o-mini five questions about the already-granted patent and then filtering the answers with GPT-4o, so the task is to reconstruct a known patent from an LLM's summary of that same patent.","fun_headline_variants_meta":{"raw":{"variants":["7B model beats GPT-4o at full patent drafting","AutoPatent: 7B model outwrites GPT-4o on 17K-token patents","Multi-agent pipeline turns drafts into patents, beats GPT-4o","Small model team beats big LLMs at patent writing","Patents via planner, writer, examiner agents: 7B > GPT-4o"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2887,"prompt_tokens":1020,"completion_tokens":1867,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":1767}},"tokens_in":636,"tokens_out":1867,"duration_ms":13547,"temperature":1.0,"reasoning_tokens":1767,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:42:49.957379+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AutoPatent and zero-shot GPT-4o on a held-out set of genuine pre-filing inventor drafts whose corresponding patents later granted, and have patent-law experts blind-judge which output better matches the disclosure; if Qwen2.5-7B+AutoPatent does not win the majority of expert preferences, the central claim that the framework transfers to real inventor drafting collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the accepted granted patents that the D2P benchmark's draft–patent pairs are built from."}],"review_version":1}