Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

This paper introduces Idea2Plan Bench, a benchmark built from papers published after LLM training cutoffs, to measure how well language models turn a research idea into an executable plan, and reports that GPT-5 leads all tested models whil

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 07:37 UTC pith:UW6WKTQ2

load-bearing objection Solid benchmark for research planning with real contamination control; main risk is that the auto-generated rubric/judge pipeline may reward reconstructing the reference paper rather than general planning skill. the 4 major comments →

arxiv 2510.24891 v2 pith:UW6WKTQ2 submitted 2025-10-28 cs.CL cs.LG

Idea2Plan: Exploring AI-Powered Research Planning

Jin Huang , Silviu Cucerzan , Sujay Kumar Jauhar , Ryen W. White This is my paper
classification cs.CL cs.LG
keywords research planningLLM evaluationbenchmarkrubric-based evaluationLLM-as-a-judgedata contaminationautonomous research agentsresearch ideas
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The authors set out to measure a capability that has been largely unexamined: how well large language models can bridge the gap from a raw research idea to a concrete, testable research plan. To do this, they built a benchmark from 200 ICML 2025 papers released after the models' training cutoff, so the models could not have memorized the papers. Each benchmark item pairs a research idea with a rubric of yes/no questions derived from the published plan, and a language-model judge grades the generated plans. On this measure, GPT-5 and GPT-5-mini outperform all other tested models, but even the best plans cover only about 62 percent of rubric elements, well short of an estimated 81.9 percent upper bound set by giving a model the full paper. The core contribution is a reproducible instrument for tracking research-planning ability, a capability that matters for building autonomous research agents.

Core claim

On the paper's own terms, the central claim is that research planning can be isolated and rigorously evaluated, and that current frontier models are already competent but far from saturated at this task. Using a rubric-based scoring protocol, GPT-5 achieves an average Planning Score of 62.0 percent under a simple naïve prompt, and 61.9 percent under a ReAct agentic setup; GPT-5-mini follows at 56.7 and 61.1 percent respectively. The authors find that a ReAct agent with arXiv search and reading tools does not beat simpler prompting, and that fine-tuning on paper-derived idea–plan pairs degrades performance and increases hallucination. They also report that an upper-bound setup, in which o4-mi

What carries the argument

The central object is the auto-generated grading rubric. For each research idea, the authors extract a reference plan from the corresponding paper, then prompt a model to generate a structured rubric of binary yes/no questions that any valid plan for that idea should satisfy. A separate language-model judge, validated against a small set of expert annotations, grades each plan section by section; the section accuracies are macro-averaged into the Planning Score.

Load-bearing premise

The entire evaluation instrument is auto-generated by one AI model: the ideas come from an AI extraction, the rubrics from an AI prompt, and the grading from an AI judge, with only a small sample of human checks; if that model's notion of a valid plan does not generalize to human standards, the reported scores and ranking could reflect evaluator bias rather than planning ability.

What would settle it

Take a random sample of, say, fifty generated plans from GPT-5 and o4-mini, have human experts grade them against the same auto-generated rubrics, and compare the resulting scores and ranking to the LLM-judge scores. If human graders do not reproduce the GPT-5 advantage, or if the absolute scores shift by more than a few points, the benchmark's validity as a measure of planning ability is falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the benchmark is accepted, research planning becomes a measurable, trackable capability, giving the field a standardized yardstick for autonomous research agents.
  • The reported headroom—from roughly 62 percent to an 81.9 percent upper bound—implies that substantially better planners should be achievable without new scientific knowledge.
  • The finding that retrieval tools did not help the ReAct agent suggests that naive retrieval can introduce noise and that knowledge conflicts are a bottleneck for agentic planning.
  • The failure of straightforward fine-tuning on paper-derived idea–plan pairs points away from imitation-only training and toward other training signals, such as rubric feedback or execution results.
  • The consistently weak performance on literature-review sections across all models identifies a specific capability gap to target next.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the ranking holds beyond this benchmark, GPT-5-class models are a reasonable starting point for building research-planning assistants, but the absolute scores caution against treating current agents as reliable autonomous planners.
  • Because the rubrics and the judge are both generated and run by the same model family, the absolute Planning Scores may be inflated or deflated in ways that a human-only benchmark would correct; the relative ranking is the more robust claim.
  • A natural next experiment is to train a judge on rubric judgments from a different model family to test whether the ranking flips, effectively probing whether the scores measure planning ability or evaluator preference.
  • The benchmark could be extended to other fields by sampling papers from other venues with post-cutoff release dates, but rubric generation would likely need domain-specific adaptation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes the Idea2Plan task — turning a research idea into a structured research plan — and introduces Idea2Plan Bench, built from 200 ICML 2025 Spotlight/Oral papers whose arXiv versions postdate the tested LLMs' training cutoffs. Ideas and reference plans are extracted from paper abstracts/full texts using o4-mini, rubrics are generated from the reference plans with o4-mini, and final grading is performed by an o4-mini judge. The authors validate the extraction and rubric quality with 8 expert-rated papers and validate the judge on 10 plans against human annotations (o4-mini F1=0.91). They evaluate GPT-5, GPT-5-mini, o4-mini, GPT-4.1, DeepSeek-V3/R1, and Phi-4 under naïve, 0-shot, 1-shot, and ReAct-agent settings with three trials per configuration, plus length-controlled and curated-literature probes. The headline results are that GPT-5 and GPT-5-mini score highest (naïve average Planning Score 62.0% and 56.7%; ReAct 61.9% and 61.1%), while a full-paper upper bound gives 81.9%. The paper also reports that ReAct does not beat simpler prompting and that SFT on idea–plan pairs degrades performance.

Significance. If the evaluation instrument is valid, this is a useful contribution: it is one of the first dedicated benchmarks for research planning, with deliberate contamination controls (post-cutoff papers, blocklisted search in ReAct), a transparent rubric-based scoring protocol, a small judge-validation set, and length-robustness checks. The result that frontier models leave large headroom (≈20 points below the upper bound) is a falsifiable, potentially actionable finding. However, the benchmark's validity rests on whether auto-generated rubrics reward any sound plan for an idea rather than plans that reconstruct the reference paper's specific methodological choices. The human validation currently does not test this fairness property, and the provided worked example (Jailbreak-Tax) shows rubric questions that appear to require the specific attack taxonomy of the reference paper. Because the ranking is the central claim, this concern is load-bearing rather than cosmetic.

major comments (4)
  1. [§3.2, Tables 8 and 26–29] The rubric generation prompt (Table 8) instructs the model to 'not ask about anything that is unique or special to this particular research plan' and to judge intent rather than terminology. The worked Jailbreak-Tax rubric in Tables 26–29, however, contains questions that violate this instruction. For example, the Methods section independently requires a prompt-based jailbreak attack, a fine-tuning-based jailbreaking approach, a demonstration-based (many-shot) attack, optimization-based adversarial attacks, genetic/evolutionary strategies, translation-based attacks, and iterative prompt rewriting — seven attack families. The research idea only says 'apply representative jailbreak strategies'; a sound plan could reasonably choose three or four of these families. As written, the rubric mechanically penalizes any plan that does not mirror the reference paper's full attack suite. This conver
  2. [§3.4 and §3.6] The human validation is too weak to establish rubric fairness. In Table 2, experts were asked only whether the extracted rubric questions 'cover the essential parts' of a plan for the idea; they were not asked whether a competent alternative plan that makes different methodological choices would be unfairly penalized. The JudgeEval set (Table 3) is only 10 plans, with a single macro-F1 of 0.91 for o4-mini; no confidence intervals or per-paper breakdown are given. With 200 test rubrics generated by the same model that later serves as judge, a systematic bias in rubric wording (e.g., requiring the reference paper's specific methods) would not be detected by either validation stage. The authors should either (a) run a targeted human study in which expert annotators judge whether each rubric question would be satisfied by a valid alternative plan for the same idea, or (b) release rubrics and
  3. [§3.5, Table 13] The judge prompt compounds the rubric-specificity problem. It instructs 'only answer Yes if the plan explicitly satisfies the rubric question as written' and 'No if the answer is vague or only implied.' Combined with questions that enumerate the reference paper's attack families (e.g., 'Does the plan include a genetic algorithm or evolutionary strategy?'), this strictness converts any omitted-but-optional design element into a 'No.' The JudgeEval F1 of 0.91 against human annotations does not address this, because the human annotations were produced for the same rubric questions and thus inherit the same specificity bias. The paper should report a sensitivity analysis in which the judge is re-run on rubrics that are explicitly pruned of reference-specific items, or in which alternative human-written reference plans are graded.
  4. [§4.2, Figure 2 and Table 4] The paper reports means over three runs but provides no measures of uncertainty or significance testing for the main ranking. The claim that 'GPT-5 and GPT-5-mini substantially outperform other models' relies on differences of a few points (e.g., 62.0 vs 45.7 for o4-mini in naïve), but there are 200 papers and three trials; paired bootstrap or a Wilcoxon test per model pair would be inexpensive and would strengthen the claim. As written, the lack of error bars makes it difficult to assess whether, for example, GPT-5-mini's ReAct score (61.1) is truly distinguishable from GPT-5's ReAct score (61.9). This is secondary to the rubric-fairness issue but should be addressed during revision.
minor comments (5)
  1. [Abstract and Introduction] The abstract says 'GPT-5 achieves the strongest performance' while §1 and §4.2 say 'GPT-5 and GPT-5-mini' lead. Please harmonize the wording. Also, there are missing spaces in the typeset text (e.g., 'Idea2Plantask', 'ReActagent', 'AI SCIENTISTproposes'), likely from LaTeX macro expansion.
  2. [§1, Footnote 1] The statement 'Code will be released upon institutional approval' is not a commitment. Since reproducibility of the benchmark depends on public release of the 200 rubrics and ideas, please clarify whether the dataset artifacts (ideas, plans, rubrics, judge outputs) will be released, and under what license.
  3. [§3.4, Table 2] The expert evaluation uses only 8 papers, but the appendix lists 8 selected papers. The domain coverage is reasonable, but the small N means the reported subsection means (e.g., 4.12±0.83 for Introduction plans) have wide confidence intervals; this should be acknowledged in the text.
  4. [§4.5 and Appendix J] The SFT experiment is described as a 'potential training strategy,' but the negative result is interesting. The explanation that fine-tuning on extracted plans causes hallucination is plausible but not directly measured (no hallucination metric). Please either add a quantitative hallucination check or soften the causal claim.
  5. [References] Some in-text citations in the appendix are not in the reference list (e.g., 'Perdomo, J. C. (2024)' appears in Table 22 but is not listed in the bibliography). Also, several references cited in the Introduction (e.g., 'Baek et al., 2025') are listed, but a few arXiv identifiers in Table 23 have inconsistent formatting.

Circularity Check

0 steps flagged

No load-bearing circularity; the benchmark is externally anchored to published papers and a human-validated judge, and self-citations are not central.

full rationale

The derivation chain runs from published ICML papers (external ground truth) to o4-mini-extracted ideas and reference plans, to rubric questions that the generation prompt (Table 8) explicitly instructs to generalize rather than require the reference plan's specific choices, to o4-mini grading validated against expert annotations in JudgeEval (§3.6, Table 3). The reported GPT-5 / GPT-5-mini lead is an empirical measurement over that externally anchored rubric, not a quantity equal by construction to a fitted parameter or to the rubric generator's outputs. No equation or construction step defines the Planning Score in terms of the benchmark inputs in a way that forces the ranking. The self-citations present (e.g., Baek et al. 2025; Zhang et al. 2025; Li et al. 2025b) appear in related-work or SFT motivation and are not load-bearing for the central claim. The paper itself flags the residual risk that LLM-based judges may introduce systematic biases (Limitations, §6), and the human validation sample is small (8 papers for plans/rubrics, 10 plans for judges; §3.4, §3.6). Those are validity and robustness concerns, not demonstrated circular steps; without a quotable reduction of the result to the paper's own inputs, no higher circularity score is warranted.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new physical or ontological entities are postulated. The paper introduces operational constructs — the Idea2Plan task, the five-section plan template, the Planning Score metric, and JudgeEval — all with explicit operational definitions grounded in published papers and human annotations. The load-bearing free choices are the contamination cutoff date, the use of o4-mini as both rubric-generator and judge, and the small human-validation sample sizes.

free parameters (5)
  • GPT-5 training cutoff (October 2024) = 2024-10
    Contamination-control anchor (§3.3): papers with arXiv submissions before this date are excluded. If the true cutoff differs, memorization could inflate scores.
  • o4-mini as rubric generator and judge
    A single model family constructs all answer keys and grades all plans (§3.1, §3.5–3.6). The benchmark's validity depends on this unstated design choice.
  • JudgeEval validation sample = 5 papers, 10 plans
    Judge reliability (F1=0.91) is estimated from a random sample of 5 dev-set papers with 2 plans each (§3.6). Small sample bounds the claim that LLM judges are expert-aligned at scale.
  • Expert validation sample = 8 papers
    Eight experts, one paper each (§3.4). The 200-paper rubric set is not expert-validated; the >4.0/5 Likert scores cover only these 8.
  • ReAct tool budgets = 5 searches, 5 reads
    Hand-chosen limits (§3.7) that constrain agent performance; different budgets could change the ReAct null result.
axioms (4)
  • domain assumption Papers with arXiv submission after October 2024 have not been memorized by the evaluated LLMs.
    §3.3 contamination control. The paper itself concedes partial memorization from overlapping text sources is possible (Limitations §6).
  • domain assumption Rubrics generated from o4-mini-extracted reference plans capture the essential components of any valid plan for the idea.
    §3.2/§3.4. Validated by 8 experts on 8 papers (mean >4.0/5); the remaining 192 test rubrics rest on this assumption without human checks.
  • domain assumption The o4-mini judge grades plans in agreement with human experts across the full test set.
    §3.6 JudgeEval validates on 10 plans (F1=0.91); all 200×models×3 trials are graded by this judge, whose same-family relation to the top models is not examined.
  • domain assumption ICML 2025 Spotlight/Oral papers are representative high-quality AI research ideas.
    §3.3 source selection. Generalizability to other scientific domains or earlier-stage ideas is waived in Limitations §6.

pith-pipeline@v1.3.0-alltime-deepseek · 31645 in / 17270 out tokens · 155948 ms · 2026-08-04T07:37:51.510515+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields. In this work, we investigate how LLMs can handle the transition from conceptual research ideas to well-structured research plans. Effective research planning not only supports scientists in advancing their research but also represents a crucial capability for the development of autonomous research agents. Despite its importance, the field lacks a systematic understanding of LLMs' research planning capability. To rigorously measure this capability, we introduce the Idea2Plan task and Idea2Plan Bench, a set of benchmarks built from ICML 2025 and Nature Mental Health papers released after major LLM training cutoffs. Each benchmark instance includes a research idea and a grading rubric capturing the key components of valid plans. We further propose Idea2Plan JudgeEval, a complementary benchmark to assess the reliability of LLM-based judges against expert annotations. Experimental results show that GPT-5 achieves the strongest performance on the benchmark, though substantial headroom remains for improvement. Our study provides new insights into LLMs' capability for research planning and lays the groundwork for future progress.

Figures

Figures reproduced from arXiv: 2510.24891 by Jin Huang, Ryen W. White, Silviu Cucerzan, Sujay Kumar Jauhar.

Figure 1
Figure 1. Figure 1: Idea2Plan overview: Starting from a research idea extracted from the abstract of an academic paper, we [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Average Planning Score for LLMs under four baselines (Naïve, 0-Shot, 1-Shot, ReAct). Scores are means [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Relationship between plan length and Average Planning Score under constrained and expanded generation [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of curated literature on average Plan [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Pairwise win rate matrices across prompting settings. Each cell [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. InquiTree: Evaluating AI Agents in the Scientific Inquiry Loop with Paper-Derived Research Trees

    cs.DB 2026-06 unverdicted novelty 7.0

    InquiTree shows LLM agents suffer from degrading critical capabilities during extended scientific interactions and perform worse on papers published after their training cutoffs.

  2. MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

    cs.LG 2026-05 conditional novelty 6.0

    MLReplicate benchmark evaluates six autonomous systems on 45 manuscripts from ICML 2025 papers, finding that automated reviews accept flawed outputs with fabricated claims while human review exposes methodological fai...

Reference graph

Works this paper leans on

44 extracted references · 9 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Follow these steps in order: Step 1: For each section of the research plan, define what constitutes a high-quality research plan when developing the research idea

    Generalize the Criteria. Follow these steps in order: Step 1: For each section of the research plan, define what constitutes a high-quality research plan when developing the research idea. Step 2: Verify whether the input plan contains these quality characteristics. Step 3: Generate a list of yes/no questions that can be used to grade other research plans...

  2. [2]

    {{SECTION_BY_SECTION_GUIDANCE}}

    Section-by-Section Guidance. {{SECTION_BY_SECTION_GUIDANCE}}

  3. [3]

    sections

    Structure the Rubric in JSON Format. Your rubric should be organized as a JSON object with the major sections of a research plan as top-level keys. For each section, list the key elements to check for, starting from high-level concepts (e.g., overall goals) down to specific details (e.g., dataset types, model architectures, evaluation metrics). Each quest...

  4. [4]

    InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24- 28, 2025

    Navigating the digital world as humans do: Universal visual grounding for GUI agents. InThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24- 28, 2025. OpenReview.net. Patrik Haslum, Nir Lipovetzky, Daniele Magazzeni, Christian Muise, Ronald Brachman, Francesca Rossi, and Peter Stone. 2019.An introduction to ...

    Pith/arXiv arXiv 2025

  5. [5]

    Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B

    OpenReview.net. Tom Silver, Soham Dan, Kavitha Srinivas, Joshua B. Tenenbaum, Leslie Pack Kaelbling, and Michael Katz. 2024. Generalized planning in PDDL do- mains with pretrained large language models. In Thirty-Eighth AAAI Conference on Artificial Intelli- gence, AAAI 2024, Thirty-Sixth Conference on Inno- vative Applications of Artificial Intelligence,...

    Pith/arXiv arXiv 2024

  6. [6]

    RESEARCH PLAN COMPLETE

    End with “RESEARCH PLAN COMPLETE” after your final answer

  7. [7]

    Observation:

    NEVER generate “Observation:” — always wait for the system response READING PAPERS: When you use theread_paperaction, you will receive a comprehensive summary of the paper content instead of the full text. Begin your research planning process now. Start with a “Thought:” about what information you need to gather first. REMEMBER: You must use the ReAct for...

  8. [13]

    LLM-Generated Research Plan from Full Paper - How well does the AI-extracted plan capture the original paper’s content?

  9. [14]

    In the next section we will provide guidelines for you to evaluate this

    LLM-Generated Rubric Questions - How good are the AI-generated evaluation questions for assessing research plans? Definition of a Research Plan <Placeholder for the definition of research plan, see Section X> PART 1: LLM-GENERATED RESEARCH PLAN FROM FULL PAPER TO EV ALUATE Instructions: The content below was automatically generated by an AI system from th...

  10. [15]

    You will be evaluating **ONE SPECIFIC SECTION** of the rubric at a time against the entire research plan

  11. [16]

    Rubric items may be nested (e.g., subsections→questions)

    For the given section: - Traverse all levels of the section hierarchy. Rubric items may be nested (e.g., subsections→questions). - Evaluate each **leaf-level question** (i.e., the final bullet points that are actual rubric questions). - Answer **Yes** only if the research plan clearly and explicitly addresses the rubric question. - Answer **No** if the ru...

  12. [17]

    e.g., dataset A, dataset B

    When rubric questions include examples in parentheses (for example, "e.g., dataset A, dataset B"), these are provided as reference examples to illustrate the type of content being asked about. Do NOT require the research plan to mention these specific examples to answer "Yes". > For example: > - If the rubric asks "Does the plan use mathematical datasets ...

  13. [18]

    section_name

    For each rubric question in the specified section: - **Question**: [rubric question] - **Answer**: Yes / No - **Explanation**: [brief justification] — ## Inputs // begin of research plan rubric section to evaluate {{RESEARCH_PLAN_RUBRIC_SECTION}} // end of research plan rubric section // begin of full research plan {{RESEARCH_PLAN}} // end of full researc...

  14. [23]

    IMPORTANT NOTES: - Base your plan on the provided research idea and your existing knowledge

    Include resource requirements for conducting the research. IMPORTANT NOTES: - Base your plan on the provided research idea and your existing knowledge. Table 16: One-Shot Baseline Prompt One-Shot Baseline Prompt RESEARCH IDEA TO ANALYZE: {{RESEARCH_IDEA}} Your task is to generate a detailed research plan based on the provided research idea. Below is the t...

  15. [24]

    Analyze the research idea thoroughly from the provided research idea

  16. [25]

    Generate a complete research plan following the EXACT template structure above

  17. [26]

    Fill in all sections with relevant, specific content based on the research idea

  18. [27]

    Draw upon your existing knowledge of the research area to provide context and background

  19. [28]

    Thought:

    Include resource requirements for conducting the research. IMPORTANT NOTES: - **The examples provided in the research plan template are for reference only** - replace them with content specific to the given research idea. - Base your plan on the provided research idea and your existing knowledge. 22 Table 17: ReAct Agent Prompt ReAct Agent Prompt You are ...

  20. [29]

    Action Input:

    You MUST STOP after “Action Input:” — do NOT generate “Observation:” or any results

  21. [30]

    The system will execute your action and provide the real observation

  22. [31]

    Wait for the actual tool results before continuing

  23. [32]

    Final Answer:

    Use the EXACT tool names listed above. Available tools are: {{TOOL_NAMES}} FINAL STEP (After gathering sufficient information): When you have collected enough information through your searches and reads, generate your final answer in this EXACT format: Thought: I now have sufficient information to create the research plan. Final Answer:[Start your complet...

  24. [33]

    Use targeted queries to find relevant information

  25. [34]

    Look for recent advances, existing methods, and gaps in the literature

  26. [35]

    Consider both technical approaches and evaluation methods

  27. [36]

    Each tool call should build upon previous findings

  28. [37]

    Choose the most appropriate tool for each information need

  29. [40]

    search_papers: Search for academic papers using Bing Custom Search - Input: A search query string for academic papers - Output: JSON with paper titles, abstracts, arXiv IDs, and publication details - Usage example: Action Input: machine learning attention mechanisms

  30. [41]

    2301.12345

    read_paper: Read and analyze a specific academic paper by its arXiv ID - Input: An arXiv ID (e.g., "2301.12345") - Output: JSON with arXiv_id and paper summary - Usage example: Action Input: 2301.12345 23 Table 19: Paper Summarization Prompt summary_prompt = f""" PAPER TO SUMMARIZE: Title: {paper_title} ArXiv ID: {arxiv_id} Paper Content: {paper_content} ...

    Pith/arXiv arXiv

  31. [42]

    Main Contributions: Key findings and contributions of this paper

  32. [43]

    Key Related Literature: Important prior works referenced and how this paper builds upon or differs from them

  33. [44]

    Methods and Techniques: Approaches, algorithms, or methodologies used

  34. [45]

    Prediction Policy Problems

    Experimental Design and Results: How experiments were conducted, datasets used, evaluation metrics, and key results obtained SUMMARY: """ Table 20: Section-wise Planning Scores (%) withMeanaggregation. Each value represents the mean accuracy across all papers in Idea2Plan Bench. Bold numbers indicate the highest score within each section across all baseli...

    Pith/arXiv arXiv 2015

  35. [46]

    Carefully analyze the research idea to understand its core contributions, methods, and objectives

  36. [47]

    Read the introduction to identify the key technical foundations and prior work this research builds upon

  37. [48]

    title": The exact title of the paper -

    Select exactly 3 papers that serve as direct foundational works by being: - Core methodological predecessors that this work extends or improves - Papers introducing the key techniques, algorithms, or frameworks being built upon - Essential baseline methods or datasets that this work directly compares against or uses Note: Focus on papers that are specific...

    Pith/arXiv arXiv 2025

  38. [299]

    Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang

    Association for Computational Linguistics. Yidong Wang, Qi Guo, Wenjin Yao, Hongbo Zhang, Xin Zhang, Zhen Wu, Meishan Zhang, Xinyu Dai, Min Zhang, Qingsong Wen, Wei Ye, Shikun Zhang, and Yue Zhang. 2024b. Autosurvey: Large lan- guage models can automatically write surveys. In Advances in Neural Information Processing Systems 38: Annual Conference on Neura...

    Pith/arXiv arXiv 2024

  39. [1998]

    Pddl—the planning domain definition lan- guage.Technical Report, Tech. Rep. Lakshya A. Agrawal, Shangyin Tan, Dilara Soylu, Noah Ziems, Rishi Khare, Krista Opsahl-Ong, Ar- nav Singhvi, Herumb Shandilya, Michael J. Ryan, Meng Jiang, Christopher Potts, Koushik Sen, Alexan- dros G. Dimakis, Ion Stoica, Daniel Klein, Matei Zaharia, and Omar Khattab. 2025. GEP...

    Pith/arXiv arXiv 2025

  40. [2016]

    How to write a research proposal?Indian journal of anaesthesia, 60(9):631–634. David Wadden, Kejian Shi, Jacob Morrison, Aakanksha Naik, Shruti Singh, Nitzan Barzilay, Kyle Lo, Tom Hope, Luca Soldaini, Shannon Zejiang Shen, Doug Downey, Hannaneh Hajishirzi, and Arman Cohan

  41. [2022]

    Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others

    Quantifying memorization across neural lan- guage models.arXiv preprint arXiv:2202.07646. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, and 1 others. 2021. Extracting training data from large language models. In30th USENIX Security Symposium (USE...

    Pith/arXiv arXiv 2021

  42. [2023]

    HumanEval

    to evaluate research plans against the gen- erated rubrics, with the Rubric Evaluation Prompt (Table 13) guiding the model to perform assess- ment with strict interpretation of rubric criteria. We use o4-mini (reasoning=high) throughout the study. Because grading an entire plan in one API call can exceed input limits and cause failures, we instead evaluat...

    2023

  43. [2024]

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim

    Sciriff: A resource to enhance language model instruction-following over scientific literature.CoRR, abs/2406.07835. Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023. Plan- and-solve prompting: Improving zero-shot chain- of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting o...

    arXiv 2023

  44. [2025]

    Deepseek-r1: Incentivizing reasoning capa- bility in llms via reinforcement learning.CoRR, abs/2501.12948. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingx- uan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, 9 Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 80 others. ...

    Pith/arXiv arXiv 2024