Pith. sign in

REVIEW 3 major objections 8 minor 17 references

MK2 at PBIG Competition: A Prompt Generation Solution

T0 review · 3 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A prompt-only pipeline beat all rival systems on the PBIG patent-ideation leaderboard.

desk verdict A honest shared-task report that checks out numerically; the internal judge loop is the main soft spot, not a fatal flaw. read the letter →

arxiv 2507.08335 v1 pith:NQYYTRT5 submitted 2025-07-11 cs.CL

Yuzheng Xu , Tosho Hirasawa , Seiya Kawano , Shota Kato , Tadashi Kozuno This is my paper · ORCID
classification cs.CL
keywords patent-basedideagenerationpromptengineeringLLM-as-a-judgeEloratingproductideationdomainadaptationsharedtask
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that competitive patent-to-product ideation can be achieved purely with prompt engineering, without any extra training data. The authors build MK2, a pipeline in which one LLM drafts and iteratively refines a prompt while another generates product ideas and a third judges them in an Elo loop. MK2 topped the automatic leaderboard in all three domains and won 25 of 36 criterion-by-domain tests, with human judges confirming its edge in NLP and computer science. The materials-chemistry gap shows the limits of the approach and of automatic evaluation in expert-heavy fields. The result matters because it suggests a cheap, fast route to commercially relevant ideation from patents, and it highlights where automatic and human judgments diverge.

What carries the argument

The load-bearing mechanism is the iterative prompt-optimization loop with an LLM-as-a-judge. Gemini 2.5 acts as the prompt engineer: it reads the competition guidelines, analyzes outputs from underperforming prompts, identifies effective components, and merges them into the current best prompt, a process repeated until the internal leaderboard stops improving. GPT-4.1 acts as the generator that produces the final one-idea-per-patent outputs, and Qwen3-8B acts as the judge that runs pairwise Elo comparisons on all six criteria in a single step, with positions swapped in half of the cases to reduce bias. The loop's effectiveness also depends on a practical length-control trick: restating the character limit at the end of the user prompt, which proved more reliable than post-editing truncated outputs.

What would settle it

Take a held-out set of patents and compare ideas generated by prompts selected through the internal Elo loop against ideas from randomly chosen or manually written prompts, with blind human expert ratings. If the Elo-selected prompts do not outperform the random prompts on human ratings, the loop is tuning to judge artifacts; the materials-chemistry result, where MK2 led on automatic scores but never topped any human criterion, is consistent with such a failure and can be probed directly by checking whether MC outputs' cited claim numbers hold up to expert scrutiny.

Watch

Extended reading notes

Core claim

The paper's central claim is that a lightweight, prompt-centric pipeline can outperform more elaborate systems on the Patent-Based Idea Generation task without any additional training data. The winning recipe is an automated prompt-refinement loop: Gemini 2.5 drafts the initial prompt, analyzes outputs from weaker prompts, grafts their effective fragments into the best prompt, and repeats; GPT-4.1 then uses the final prompt to generate one product idea per patent; and an internal Elo leaderboard judged by Qwen3-8B selects the best prompt. Across three domains, two evaluator types, and six criteria, MK2 placed first in the automatic evaluation and won 25 of 36 tests, demonstrating that prompt engineering alone can deliver clear and original product ideas. The one systematic failure, materials chemistry, suggests that automatic metrics can overrate outputs that lack the scientific rigor experts expect.

Load-bearing premise

The internal judge's preferences transfer to the official automatic judge and to human quality judgments, so that prompts selected by Qwen3-8B's Elo scores are actually the most commercially viable ideas.

Editorial extensions

If this is right

  • If prompt-centric ideation is this competitive, teams without fine-tuning infrastructure can enter patent-ideation tasks with modest budgets.
  • The iterative prompt-grafting loop can likely be fully automated with systematic prompt exploration, removing the remaining manual steps.
  • Restating output constraints at the end of the prompt is a simple, transferable technique for controlling generation length in constrained tasks.
  • Automatic evaluation that aligns with human judgment on general technical domains may still overrate outputs in specialized scientific fields, so hybrid screening plus expert review is a safer evaluation design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The internal Elo loop selects prompts by Qwen3-8B's preferences; if that judge is biased, the loop could be tuning to judge artifacts rather than to genuinely better ideas, which would explain why human judges disagreed in materials chemistry.
  • A testable extension would be replacing the cheap judge with a domain-expert judge in the loop and checking whether the materials-chemistry gap narrows, effectively testing whether judge quality causes the domain failure.
  • The paper's success with NLP-to-CS prompt adaptation suggests that cross-domain prompt transfer works when domains share conceptual structure, but fails when the target domain requires specialized scientific grounding; that boundary could be probed with other hard-science domains.
  • The variability of individual human scores reported in the appendix suggests that part of the MC gap may stem from evaluator inconsistency, not just output quality, which would motivate more reliable expert-rating protocols.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 8 minor

Summary. MK2 at PBIG Competition: A Prompt Generation Solution describes a system for the Patent-Based Idea Generation shared task, in which a model must convert each of 150 patents into a product idea with title, description, implementation outline, and differentiation statement. The pipeline is prompt-centric: Gemini 2.5 drafts and iteratively refines a two-page prompt by grafting effective fragments from weaker candidate outputs; GPT-4.1 generates the final ideas; and an internal Elo leaderboard judged by Qwen3-8B (with GPT-4.1-mini gating) selects the submitted outputs. No training data or fine-tuning is used. The paper reports top automatic-evaluation scores in all three domains (NLP, CS, MC) and human-judge preference in the NLP and CS tracks, with a disclosed gap in MC, corresponding to the abstract's '25 of 36 tests.' The final prompt is fully reproduced in Appendix A, and representative outputs with per-criterion human scores appear in Appendix B.

Significance. If the reported leaderboard outcomes hold, the contribution is modest but useful: it shows that a lightweight, iterative prompt-engineering loop, without additional training data, can top the official PBIG automatic evaluation and satisfy expert human judges in two of three domains. The paper's strengths include the complete reproduction of the final prompt (Appendix A), internal consistency between the abstract's 25-of-36 count and the per-domain prose in Section 5.2, explicit acknowledgment of LLM-judge biases (Section 2.3), and honest disclosure of the MC failure (Sections 5.2 and 6), which functions as a falsifiable indication of the method's limits. The descriptive claims are externally checkable against the organizer's published leaderboard; the causal attribution to the prompt strategy is the part that needs the additional evidence requested below. The paper ships no code, but the prompt and the sample outputs make the generation stage reproducible.

major comments (3)
  1. [§5.1, Table 1] Section 5.1 (Table 1) reports only MK2's own Elo scores; there is no ranking table or pairwise win counts for the other PBIG participants, so the central claims of 'first place' and 'won 25 of 36 tests' cannot be verified from the manuscript alone. Please include the full official leaderboard or per-domain, per-evaluator counts of first-place criteria, and state how many human judges contributed to each cell of Table 1.
  2. [§4.5] Section 4.5 justifies the choice of Qwen3-8B with an unquantified claim of 'high correlation with GPT-4.1' and gives no validation of the internal Elo loop's rankings against the official automatic scores or the human scores. Because Table 1 already shows that the LLM-based automatic evaluation diverges from human experts in the MC track, the manuscript should report the measured agreement between the internal judge and the official evaluator (e.g., rank correlation or pairwise agreement on a development sample), the number of candidate prompts compared in the loop, and the point at which the submission was frozen relative to any official submissions.
  3. [§4.5] Section 4.5 is ambiguous about the final selection protocol: it says 'the best-performing models varied across the three domains' and then 'we selected a model that achieved a balance between ranking and the degree of length-limit violation,' while Sections 4.1 and 4.4 state that GPT-4.1 produced all final outputs. Please state whether the submitted outputs came from a single configuration or from per-domain configurations, and clarify whether the iterative 'resubmission to the leaderboard' in Sections 4.2 and 4.5 refers to the internal or the official PBIG leaderboard; this determines whether the official scores are a held-out test of the strategy.
minor comments (8)
  1. [§5.2] There is a stray space before the comma in 'In the MC domain ,'.
  2. [Appendix B] The character sequences '—', 'σ-65', and 'μ' are mojibake artifacts in the sample outputs and should be repaired, especially since the paper itself is about generation quality.
  3. [References] The reference entries for OpenAI (a stray ':' before the author list) and Zheng et al. (an incorrect 'E. Xing' element) appear corrupted, and 'Shun Shramatsu' in the Hoshino et al. entry is likely a misspelling of 'Shun Shiramatsu'.
  4. [§4.3] The claim that truncation performed better than post-editing is an empirical result; please report the number of examples and the measured scores that support it.
  5. [Appendix B] The '-' entries within the human-score arrays are never explained; please state what they denote (e.g., a criterion not rated by that judge).
  6. [§6] The speculation that the MC gap 'may not stem from our lack of knowledge in MC, but rather from differences in LLMs' understanding across domains' is untested; consider softening it or stating a concrete way to test it.
  7. [§4.1] 'Gemini 2&2.5' should read 'Gemini 2.0 and 2.5,' and the base-model comparison would be more convincing if the summary results of that comparison were reported.
  8. [§5.1] The paper notes that 'Human scores fluctuated widely' in Section 6, but no variance or agreement information is given for the human scores in Table 1; reporting these would strengthen the human-evaluation claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's results are an external competition outcome, not a derivation from its own assumptions.

full rationale

The paper reports a competition system. Its central claim is that MK2 placed first in the PBIG automatic evaluation and won 25 of 36 tests against human judges in NLP/CS. This is an empirical result from an external benchmark (the PBIG leaderboard and human evaluation), not a logical consequence of the paper's own definitions or fits. The internal Elo loop (Section 4.5) selects prompts using Qwen3-8B as judge; the final submission set is determined by that internal leaderboard. One might worry that selection on an LLM proxy could overfit to LLM preferences, and the Materials Chemistry results (Section 5.2) indeed show a divergence between automatic and human scores. However, this is a validity/overfitting concern, not circularity: the official automatic evaluation and human judgments are independent of the internal Elo loop, and the paper explicitly acknowledges the MC gap. There is no equation in which a predicted quantity is defined as a fitted parameter, no load-bearing self-citation, and no imported uniqueness theorem. The only self-citation (Kawano et al., 2024) appears in Related Work as general evidence that LLMs can refine patent claims; it is not load-bearing for MK2's method or results. The paper is self-contained against an external benchmark, so the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No mathematical constants or new entities are introduced. The load bearing assumptions are about LLM generative ability and the validity of LLM based judges as proxies for idea quality.

assumptions (3)
  • domain assumption LLMs can convert patent text into commercially viable product ideas.
    The entire pipeline rests on GPT-4.1's generative quality; competition scores are empirical evidence, but no mechanism level validation is given.
  • domain assumption Qwen3-8B and GPT-4.1-mini are reliable internal judges.
    Section 4.5 says Qwen3-8B was chosen for its high correlation with GPT-4.1, but no correlation data or human alignment evidence is provided.
  • domain assumption The official automatic leaderboard is a valid measure of ideation quality.
    The paper's main claim is based on this evaluator; the MC human disagreement shows it may miss scientific rigor, as the authors admit in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MK2 at PBIG Competition: A Prompt Generation Solution." pith.science (2026). https://pith.science/paper/NQYYTRT5

@misc{pith2026250708335,
  author       = {Pith},
  title        = {Pith review of: MK2 at PBIG Competition: A Prompt Generation Solution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQYYTRT5}},
  note         = {Machine review of arXiv:2507.08335}
}
read the original abstract

The Patent-Based Idea Generation task asks systems to turn real patents into product ideas viable within three years. We propose MK2, a prompt-centric pipeline: Gemini 2.5 drafts and iteratively edits a prompt, grafting useful fragments from weaker outputs; GPT-4.1 then uses this prompt to create one idea per patent, and an Elo loop judged by Qwen3-8B selects the best prompt-all without extra training data. Across three domains, two evaluator types, and six criteria, MK2 topped the automatic leaderboard and won 25 of 36 tests. Only the materials-chemistry track lagged, indicating the need for deeper domain grounding; yet, the results show that lightweight prompt engineering has already delivered competitive, commercially relevant ideation from patents.

Figures

Figures reproduced from arXiv: 2507.08335 by the authors.

Figure 1
Figure 1. Condensed view of the final prompt. The full [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Final prompt used in our submission. (per patent), it cuts secure query compute by up to 99.99%. $2B+ secure analytics market. • Implementation: Core patented row reduc￾tion protocol (Claims 1, 3) is embedded in the secure SQL engine, minimizing dummy row overhead. Built on modern MPC libraries, cloud orchestration, and secure APIs. 3-year plan: prototype, pilot with banks, full SaaS launch. Requires secure infra pa… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 13 canonical work pages

  1. [1]

    *The implementation should clearly justify this 3-year timeline.*

    **Technical Validity & Feasibility** – Launchable within 3 years; core functionality MUST depend on the identified patented NLP mechanism. *The implementation should clearly justify this 3-year timeline.*

    work page

  2. [2]

    In The 2nd Workshop on Agent AI For Scenario Planning (AGENTSCEN2025)

    Agent ideate: A framework for product idea generation from patents using agentic ai. In The 2nd Workshop on Agent AI For Scenario Planning (AGENTSCEN2025). Seiya Kawano, Hirofumi Nonaka, and Koichiro Yoshino. 2024. Claimbrush: A novel framework for automated patent claim refinement based on large language models. In 2024 IEEE International Conference on B...

    work page 2024

  3. [3]

    Avoid overly broad markets (e.g., ’all businesses’) or vague pain points (e.g., ’improving efficiency’)

    **Specificity** – Pin-pointed target users/industry, precise pain points (ideally quantified, e.g., ’X hours wasted weekly per user’, ’Y% error rate leading to $Z losses’), and a concrete use case. Avoid overly broad markets (e.g., ’all businesses’) or vague pain points (e.g., ’improving efficiency’). Focus on a defined user segment (e.g., ’compliance off...

    work page

  4. [4]

    Persuasively explain why the target users will adopt this solution over existing alternatives or inaction, emphasizing the quantifiable benefits

    **Need Validity** – Address a compelling, validated pain point. Persuasively explain why the target users will adopt this solution over existing alternatives or inaction, emphasizing the quantifiable benefits

    work page

  5. [5]

    In The Thirteenth International Conference on Learning Representations

    Can LLMs generate novel research ideas? a large-scale human study with 100+ NLP researchers. In The Thirteenth International Conference on Learning Representations. Qwen Team. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388. Yasunori Terao and Yuuki Tachioka. 2025. Col- laborative invention: Refining patent-based prod- uct ideation via llm-guided...

    arXiv 2025

  6. [6]

    This advantage must be critical to the target users/industry

    **Competitive Advantage** – Demonstrate a durable, significant edge **directly and uniquely enabled by the identified core patented NLP mechanism**, making it difficult for competitors to replicate (e.g., due to patent protection or the unique technical barrier). This advantage must be critical to the target users/industry. # Ideation Process:

    work page

  7. [7]

    In The Thirteenth International Conference on Learning Representations

    JudgeLM: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations. A Final Prompts The final prompt used in our submission is repro- duced in Figure 2. B Representative Output Samples and Human Scores NLP Domain • Title: VerticalIQ: Domain-Adaptive Chatbot for Enterprise IT Helpdesks with...

    work page

  8. [9]

    **Innovativeness** – Clearly show how the patent unlocks a new, superior solution. Specify if the innovation lies in solving an existing problem in a *radically more effective/efficient way* OR if it enables a *completely new application/market* previously unfeasible. *This novelty should be a clear thread through your description and differentiation.*

    work page

Show all 17 references
  1. [12]

    **Market Potential** – The product should target a sizable, growing, or high-value niche market. This aspect, potentially with an indicative market size (e.g., ’$X billion market’), *or the scale/frequency of the problem*, should be briefly and credibly touched upon in the pro...

  2. [14]

    *Consider what makes this specific mechanism distinct from general NLP techniques.* This specific element is the *cornerstone*

    **Deep Patent Analysis** – **Critically, identify and articulate the single most unique, pivotal, and non-obvious technical mechanism, algorithm, or architectural innovation** detailed within the patent (often found in the ’claims’ or ’detailed description’ sections). *Conside...

  3. [15]

    white-space

    **Market & Need Identification** – Locate significant unmet needs or "white-space" opportunities where this specific patented breakthrough can deliver outsized, clearly demonstrable (and ideally quantifiable) value. *Consider current market trends and if the patent addresses a...

  4. [16]

    **Product Conceptualization** – Design a focused product where the **identified patented mechanism** is absolutely indispensable to delivering the core user value and solving the pinpointed, quantified problem

  5. [17]

    title":

    **Strategic Pitch Formulation** – Craft a concise, compelling, VC-ready pitch for your product idea, ensuring all evaluation criteria **are evident in the output fields below** and met within the specified character limits, emphasizing quantifiable achievements. *Remember, eva...

    2000

  6. [2007]

    Information processing & management, 43(5):1216– 1247

    Text mining techniques for patent analysis. Information processing & management, 43(5):1216– 1247. Tomas van Rijn and James Kenneth Timmis. 2023. Patent landscape analysis—contributing to the iden- tification of technology trends and informing re- search and innovation funding...

    2023

  7. [2024]

    arXiv preprint arXiv: 2402.07927

    A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv: 2402.07927. Masaya Shimanuki, Naoto Shimizu, Kentaro Kinu- gasa, and Hiroki Sugisawa. 2025. Business idea generation from patent documents: Knowledge in- tegr...

    2025 arXiv

  8. [2025]

    arXiv preprint arXiv: 2501.12948

    Deepseek-r1: Incentivizing reasoning capabil- ity in llms via reinforcement learning. arXiv preprint arXiv: 2501.12948. Arpad E Elo. 1967. The proposed uscf rating system, its development, theory, and applications. Chess Life, 22(8):242–247. Google DeepMind. 2025. Gemini 2.5: ...

    1967 arXiv

  9. [6603]

    Lennart Meincke, Karan Girotra, Gideon Nave, Chris- tian Terwiesch, and Karl T

    IEEE. Lennart Meincke, Karan Girotra, Gideon Nave, Chris- tian Terwiesch, and Karl T. Ulrich. 2024. Using large language models for idea generation in innovation. The Wharton School Research Paper. Forthcoming. OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, A...

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.