REVIEW 4 major objections 3 minor 19 references
Refining a small diversity-selected subset beats refining all LLM ideas.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 14:29 UTC pith:BVUZRZJQ
load-bearing objection A well-scoped empirical study of refinement-budget allocation, but the MMR-k advantage is judge-bound and not yet robust. the 4 major comments →
Budgeted Subset Refinement for Execution-Aware LLM Research Ideation
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms: for a fixed pool of raw LLM-generated ideas and a fixed refinement budget, the allocation policy matters more than raw generation quality. The author introduces Budgeted Subset Refinement, which selects k candidates for expensive refinement and leaves the rest unrefined. In a unified shared-candidate-pool benchmark across 10 seeds and 10 research environments, raw first-n and rerank-only yield zero research-strong nonduplicate ideas; fixed uniform refinement yields 21.8; random-k yields 22.18; MMR-k yields 23.70 with the lowest duplicate rate and best cost per strong nonduplicate. The author argues this shows refinement is necessary but should be spent selectively a
What carries the argument
Maximal marginal relevance (MMR): a selection rule that scores each candidate by balancing rank-based relevance against embedding similarity to already-selected candidates, with lambda=0.65 and k=4. It is the mechanism that turns budgeted refinement from random sampling into diversity-aware allocation. The evaluation also relies on a shared raw candidate pool so all strategies face identical inputs, and on a token-based 'cost per research-strong nonduplicate idea' metric that counts only deduplicated ideas above a rubric threshold.
Load-bearing premise
The load-bearing premise is that the LLM judge's execution-aware rubric is a valid proxy for what a human researcher values; if refinement mainly injects rubric-scored details, the measured gains may not survive expert or execution-grounded review.
What would settle it
Take a balanced sample of raw and MMR-refined ideas from a fixed pool and have independent expert researchers (or an execution pipeline) score them blindly; if refined ideas do not outperform raw ones on expert-rated quality, or if uniformly refined ideas match MMR on strong-nonduplicate yield, the central claim would be refuted.
If this is right
- Research ideation systems should be evaluated as budgeted support-allocation systems, not just idea generators.
- New selection heuristics should be compared against random-k refinement, a strong low-cost baseline, rather than only uniform refinement.
- Diversity-aware subset refinement can counteract duplicate collapse, a known failure mode of LLM overgeneration, at the portfolio level.
- Uniform refinement's higher average idea strength does not translate to the best portfolio, separating per-item quality from allocation efficiency.
Where Pith is reading between the lines
- If the LLM judge is rewarding rubric-specific detail injection rather than genuine idea improvement, the raw-vs-refined gap (and possibly the MMR-vs-fixed ranking) could shrink under expert or execution-grounded evaluation; the paper's own external panel already shows per-item rankings vary by judge.
- The MMR advantage should grow as pool redundancy increases; a testable extension would vary pool size and duplication while holding refinement budget fixed.
- The same budgeted-allocation framing could transfer to other expensive post-processing steps, such as retrieval grounding or human-in-the-loop feedback, where per-candidate cost is high.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper frames LLM research ideation as a pre-execution portfolio-allocation problem. Given a shared pool of raw candidates, the authors compare returning raw or reranked ideas, uniformly refining all candidates, and refining only a selected subset. Subset policies are random-k, MMR-k, and micro-low-to-heavy-k. Across 10 seeds and 10 environments, using Qwen2.5-3B-Instruct as generator and judge, raw and rerank-only outputs yield no research-strong nonduplicate ideas at the 6.0 threshold, while all refinement strategies succeed. The paper's headline claim is that MMR-k gives the best overall proxy tradeoff: highest strong-nonduplicate yield, lowest duplicate rate among successful methods, and best cost per strong nonduplicate. A blinded 72-item external-judge panel confirms that refinement broadly improves per-item quality, but does not reproduce the MMR-specific ranking, with fixed refinement receiving the highest composite score.
Significance. The paper's proposal — treating ideation support as a budget-constrained allocation problem rather than pure generation or reranking — is a useful and timely reframing, and the shared-candidate-pool evaluation is a methodological step forward over naive generation comparisons. The paper is transparent about its proxy nature, provides prompts, rubrics, cost decompositions, threshold sensitivity, per-seed/per-topic breakdowns, and a blinded external panel. If the MMR-specific result were robust, it would offer a practical, low-cost design rule for LLM-based ideation systems. The central claim, however, is currently supported only by one judge family on custom portfolio metrics, while the independent panel ranks fixed refinement first per-item; this gap is load-bearing for the headline conclusion.
major comments (4)
- [§5.5, Table 3] The independent external-judge panel does not support the central MMR-specific claim. Fixed refinement receives the highest composite (7.01 vs. 6.77 for MMR) and the best mean rank (1.75 vs. 2.50). The paper argues that MMR's advantage is portfolio-level, but the external judges scored only a 12-item-per-strategy sample; they did not evaluate full shared-pool outputs, duplicate rates, nonduplicate yield, or cost per strong idea. Thus the headline 'best overall tradeoff' is established only under the original Qwen judge. Please either add an external portfolio-level validation or substantially soften the MMR claim.
- [§4 and Appendix C] The raw-vs-refined gap is partly built into the protocol. The heavy-refinement prompt explicitly instructs the model to 'strengthen the dataset or corpus, baseline, metric' fields, while the judge rubric explicitly penalizes vague methods, weak metrics, and missing baselines or datasets. The raw candidates are generated with the same field requirements but are often underspecified. Consequently, the 'refinement is necessary' result may largely measure rubric compliance rather than genuine research-idea improvement. An ablation in which refined outputs are scored by a rubric that omits these fields — or by a human expert panel — is needed before claiming that transformation, rather than detail injection, drives the result.
- [§5.4, Table 7] The MMR-vs-random advantage is not statistically decisive on the primary metric. The seed-paired difference in strong nonduplicates is +1.52 with 95% CI [-0.22, 2.82], and the difference in cost per strong idea is -367 with CI [-667, 71]; both intervals include zero. Because random-k is the strong low-cost baseline, this weakens the statement that MMR 'improves' the random baseline. These intervals appear only in Appendix G; the main text should report them and temper the claim, or provide additional evidence that the effect is not noise.
- [§5.5, Appendix J] The external robustness check is under-powered for its stated purpose. The panel of four independent judges rated only 72 items, 12 per strategy, with per-item scores and small-portfolio duplicate judgments. It never recomputes nonduplicate yield, duplicate rate, or cost per strong nonduplicate over the full shared-pool outputs. Since the paper explicitly frames MMR's advantage as portfolio-level, a per-item external panel cannot arbitrate the central claim. I request either a complete external scoring of the final portfolios or an explicit statement that the MMR-vs-fixed ranking is a single-judge-benchmark finding awaiting multi-judge confirmation.
minor comments (3)
- [Table 5 caption] Random-k per-seed counts are averages over five random draws, and the table shows non-integer values. The caption should state this clearly to avoid confusion with single-run counts.
- [Eq. (3)] The definition of qt(c) depends on 'rankt(c)', and the text says the candidate order is updated after each selection, but the update rule is not fully specified. Please state exactly how the rank is recomputed among remaining candidates.
- [Figures 1 and 2] The cost-yield frontier would benefit from error bars or confidence bands; currently the reader cannot judge whether the MMR/random/fixed differences in Figure 1 are within noise.
Circularity Check
MMR's best-tradeoff claim is partly rubric-aligned and MMR-conditioned; external panel does not confirm the MMR-specific advantage.
specific steps
-
self definitional
[Appendix C (Heavy refinement prompt; Automatic judge rubric); §4 Judging and thresholds; Table 1/§5.1]
"The final experiments use Qwen/Qwen2.5-3B-Instruct as both generator and judge. ... Given a candidate idea, refine it into a more specific and execution-ready research proposal. Strengthen the research question, mechanism, dataset or corpus, baseline, metric, expected result, implementation plan, feasibility assumptions, and failure modes. ... Score each idea on ... specificity, execution-readiness, dataset specificity, baseline specificity, metric specificity ... The aggregate research-strength score combines these dimensions and penalizes vague methods, weak metrics, excessive human-subject"
The headline 'refinement is necessary' is measured by a judge rubric that explicitly penalizes missing dataset/baseline/metric specificity, while the heavy-refinement prompt explicitly instructs the model to strengthen exactly those fields. The raw-vs-refined gap is therefore partly produced by aligning the intervention with the evaluation instrument; it is not an independent measurement of research quality outside the rubric. The external panel supports the broad refinement effect, but the central 'refinement is necessary' finding reduces in part to the rubric's own definition of strength.
-
self definitional
[§3.3 Eq. 3; §4 Deduplication; §5.1/Table 1]
"scoret(c) = λqt(c) −(1−λ)d(c, S t−1), ct = argmaxc∈Ce\St−1 scoret(c), St =S t−1 ∪ {ct}. ... The similarity term uses normalized sentence embeddings, with a lexical fallback if embeddings are unavailable. ... Deduplication uses the all-MiniLM-L6-v2 sentence embedding model [Reimers and Gurevych, 2019] with a near-duplicate threshold of 0.82."
MMR is defined as greedy selection that maximizes relevance minus a penalty for maximum similarity to the already-selected set. The later headline metric 'duplicate rate' is computed using the same kind of embedding similarity threshold. MMR's 'lowest duplicate rate among successful methods' is therefore close to a direct consequence of its selection objective, not an independent empirical discovery. The yield and cost advantages are not fully explained by this alignment, but this specific success metric is definitionally favored by the method.
full rationale
The paper is not circular through self-citation: its load-bearing citations (Si et al., Qwen Team, Carbonell & Goldstein) are prior external work, and no uniqueness theorem is imported from the present author. The main circularity concerns are protocol-level. First, the benchmark's 'research-strong' definition and the heavy-refinement prompt are mutually aligned: the judge rubric scores dataset/baseline/metric specificity, and the refinement prompt instructs the model to strengthen exactly those fields, so the raw-vs-refined gap is partly a rubric-compliance artifact. Second, MMR's best-duplicate-rate result largely restates its construction, since MMR explicitly maximizes diversity while duplicate rate is measured with embedding similarity. The external panel shows per-item rankings vary by judge and gives fixed refinement the highest composite/mean rank, so the MMR-specific portfolio claim is not independently validated; internal MMR-vs-random confidence intervals for yield and cost/strong also include zero. The cost-yield decomposition, threshold sensitivity, and per-topic variation remain genuine empirical outputs, so the circularity is partial rather than total.
Axiom & Free-Parameter Ledger
free parameters (8)
- k (refinement subset size) =
4
- MMR lambda =
0.65
- near-duplicate similarity threshold =
0.82
- research-strength threshold =
6.0
- micro-triage scoring weights =
10, 2, 1, -3
- temperatures (generation/refinement/judge) =
0.9 / 0.7 / 0.6
- random-k trials per seed =
5
- raw pool size per environment =
16
axioms (7)
- domain assumption LLM judge scores under the execution-aware rubric are a valid proxy for research-idea quality.
- domain assumption Sentence-embedding similarity at threshold 0.82 correctly identifies near-duplicate ideas.
- domain assumption The 10 selected research environments and 16 raw candidates per environment are representative of research ideation workloads.
- domain assumption End-to-end token count is an adequate proxy for refinement cost.
- domain assumption The rank-based relevance and cosine similarity in MMR capture the intended quality-diversity tradeoff.
- domain assumption Refinement-induced score gains are genuine improvement rather than rubric compliance.
- domain assumption The four external LLM judge families are independent enough for a robustness check.
read the original abstract
Large language models (LLMs) can generate research ideas that appear novel to expert reviewers, but recent work also shows that such ideas often lack diversity, are difficult for LLMs to evaluate reliably, and may fail to translate into strong executed projects. This paper evaluates a controlled proxy benchmark for a pre-execution scaffolding problem: given a noisy pool of LLM-generated research ideas, how should a system allocate limited refinement effort to construct a stronger, more diverse, more execution-aware portfolio for human researchers under a fixed rubric? We introduce Budgeted Subset Refinement, a family of strategies that refine only a selected subset of candidates rather than refining all candidates uniformly. In a unified shared-candidate-pool evaluation across 10 random seeds and 10 research-ideation environments, raw generation and reranking alone produce no research-strong nonduplicate ideas under the benchmark rubric, while refinement is necessary for strong proxy-rated portfolios. Uniform refinement produces strong individual ideas but is not the best portfolio-level allocation of compute. Random-k refinement is a strong low-cost baseline, while diversity-aware MMR-k refinement gives the best overall proxy tradeoff: the highest research-strong nonduplicate yield, the lowest duplicate rate among successful methods, and the best cost per research-strong nonduplicate idea. A blinded external-judge robustness check on a balanced 72-item sample supports the broad refinement effect across independent model families, while showing that per-item rankings among refined strategies vary by judge. These results suggest that LLM research ideation systems should be evaluated not only as idea generators, but as budgeted support-allocation systems. The claims are scoped to proxy-rated portfolio quality and do not substitute for expert review or execution-grounded validation.
Figures
Reference graph
Works this paper leans on
-
[1]
Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics , year =
ResearchAgent: Iterative Research Idea Generation over Scientific Literature with Large Language Models , author =. Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics , year =
2025
-
[2]
Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year =
The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries , author =. Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , year =
-
[3]
arXiv preprint arXiv:2411.15594 , year =
A Survey on LLM-as-a-Judge , author =. arXiv preprint arXiv:2411.15594 , year =
-
[4]
Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages =
A Class of Submodular Functions for Document Summarization , author =. Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies , pages =
-
[5]
Findings of the Association for Computational Linguistics: EMNLP 2025 , year =
Chain of Ideas: Revolutionizing Research Via Novel Idea Development with LLM Agents , author =. Findings of the Association for Computational Linguistics: EMNLP 2025 , year =
2025
-
[6]
arXiv preprint arXiv:2408.06292 , year =
The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery , author =. arXiv preprint arXiv:2408.06292 , year =
-
[7]
Advances in Neural Information Processing Systems , year =
Self-Refine: Iterative Refinement with Self-Feedback , author =. Advances in Neural Information Processing Systems , year =
-
[8]
arXiv preprint arXiv:2412.15115 , year =
Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =
-
[9]
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , year =
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks , author =. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing , year =
2019
-
[10]
Advances in Neural Information Processing Systems , year =
Reflexion: Language Agents with Verbal Reinforcement Learning , author =. Advances in Neural Information Processing Systems , year =
-
[11]
Proceedings of the Fifth Workshop on Scholarly Document Processing , pages =
Literature-Grounded Novelty Assessment of Scientific Ideas , author =. Proceedings of the Fifth Workshop on Scholarly Document Processing , pages =
-
[12]
International Conference on Learning Representations , year =
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , author =. International Conference on Learning Representations , year =
-
[13]
International Conference on Learning Representations , year =
Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers , author =. International Conference on Learning Representations , year =
-
[14]
arXiv preprint arXiv:2506.20803 , year =
The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas , author =. arXiv preprint arXiv:2506.20803 , year =
-
[15]
arXiv preprint arXiv:2601.14525 , year =
Towards Execution-Grounded Automated AI Research , author =. arXiv preprint arXiv:2601.14525 , year =
-
[16]
Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year =
SciMON: Scientific Inspiration Machines Optimized for Novelty , author =. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , year =
-
[17]
arXiv preprint arXiv:2504.08066 , year =
The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search , author =. arXiv preprint arXiv:2504.08066 , year =
-
[18]
Findings of the Association for Computational Linguistics: ACL 2024 , year =
Large Language Models for Automated Open-domain Scientific Hypotheses Discovery , author =. Findings of the Association for Computational Linguistics: ACL 2024 , year =
2024
-
[19]
Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year =
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author =. Advances in Neural Information Processing Systems Datasets and Benchmarks Track , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.