pith. sign in

arxiv: 2606.12071 · v1 · pith:FZD4JUB7new · submitted 2026-06-10 · 💻 cs.DL · cs.AI

On the Limits of LLM-as-Judge for Scientific Novelty Assessment

Soumitra Sinhahajari , Navonil Majumder , Soujanya Poria This is my paper

Pith reviewed 2026-06-27 07:39 UTC · model grok-4.3

classification 💻 cs.DL cs.AI
keywords LLM judgesresearch questionsscientific noveltybenchmarkAI evaluationnovelty assessmenthuman expert evaluationarXiv papers
0
0 comments X

The pith

LLM judges rate model-generated research questions as more novel than the original author-anchored questions from real papers.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines whether LLMs can serve as reliable judges of scientific novelty by focusing on research questions rather than full ideas. It constructs RQ-Bench from recent arXiv papers, reconstructing author-anchored reference questions directly from each paper's cited background, identified gaps, and stated contributions. Standalone and comparative LLM evaluations both assign high novelty scores to questions generated by models, an effect that strengthens under direct comparison. Domain experts evaluating the same pairs reach the reverse conclusion and favor the author-anchored questions. The work also notes that many generated questions are narrow or tightly bound to their source material, a limitation LLM judges tend to overlook unless explicitly prompted.

Core claim

LLM judges produce a novelty mirage by consistently rating model-generated research questions as highly novel, with the bias intensifying in comparative settings, while domain experts instead prefer the author-anchored reference questions reconstructed from the cited background, gaps, and contributions of actual papers; generated questions are frequently narrow or source-bound, a shortcoming that LLM judges miss unless the dimension is tested directly.

What carries the argument

RQ-Bench, a benchmark that reconstructs author-anchored research questions from real papers' cited background, gaps, and contributions to serve as reference points for novelty judgments.

If this is right

  • Standalone LLM judging rates model-generated RQs as highly novel.
  • Comparative LLM judging increases the preference for model-generated RQs over author-anchored ones.
  • Domain experts reach the opposite conclusion and favor the author-anchored reference questions.
  • Many model-generated RQs are narrow or source-bound, a dimension LLM judges miss unless explicitly tested.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Automated novelty assessment may reinforce incremental or source-derived questions rather than genuinely new directions.
  • The benchmark approach could be extended to evaluate other dimensions of research ideation such as feasibility or empirical promise.
  • Hybrid evaluation pipelines that combine LLM screening with targeted human review may be required for reliable scientific novelty checks.

Load-bearing premise

The reconstructed author-anchored research questions from cited background, gaps, and contributions serve as appropriate and representative reference points for testing novelty judgments.

What would settle it

A controlled study in which domain experts, using the same instructions as the LLM judges, rate the novelty of model-generated questions higher than or equal to the author-anchored references would falsify the reported discrepancy.

read the original abstract

LLMs are increasingly used to generate and judge scientific ideas. This makes novelty evaluation a central problem. Full idea evaluation is difficult because it often requires judging a method, its feasibility, and its empirical promise. We therefore study a cleaner upstream object: the research question (RQ). RQ generation is a prerequisite for scientific ideation, and RQs can be compared against questions pursued in real papers. We introduce RQ-Bench, a benchmark built from recent arXiv papers. For each paper, we reconstruct author-anchored RQs from its cited background, gaps, and contributions. These RQs are not the only valid questions for the same background. They are author-anchored reference points for testing novelty judgments. We evaluate model-generated RQs with standalone LLM judging, comparative LLM judging, and human expert evaluation. LLM judges consistently rate model-generated RQs as highly novel, producing a novelty mirage; in comparative evaluations, this preference becomes even stronger. Domain experts, however, reach the opposite conclusion and prefer the author-anchored reference questions. We further find that many generated RQs are narrow or source-bound, a dimension that LLM judges often miss unless explicitly tested. Overall, the contradictory novelty evaluations between LLM judges and human experts raise a serious concern about the reliability of using LLMs to assess the scientific novelty of research questions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper introduces RQ-Bench, a benchmark built from recent arXiv papers in which author-anchored research questions (RQs) are reconstructed from each paper's cited background, gaps, and contributions. These serve as reference points against which model-generated RQs are evaluated via standalone LLM judging, comparative LLM judging, and human expert evaluation. The central finding is that LLM judges consistently rate model-generated RQs as highly novel (producing a 'novelty mirage' that strengthens under comparative evaluation), whereas domain experts prefer the author-anchored references; the paper also notes that many generated RQs are narrow or source-bound, a dimension often missed by LLMs unless explicitly prompted.

Significance. If the results hold after methodological clarification, the work is significant because it supplies a concrete, paper-grounded benchmark for testing LLM novelty assessment—an increasingly common use case in scientific ideation. The explicit contrast between LLM and expert judgments, together with the observation that LLMs overlook narrowness/source-boundedness, offers a falsifiable empirical probe into LLM judging reliability. The benchmark construction itself is a constructive contribution, though its value depends on the validity of the reference RQs.

major comments (3)
  1. [RQ-Bench construction] RQ-Bench construction (described in the abstract and methods): the reconstruction of author-anchored RQs from a paper's own cited background, gaps, and contributions risks hindsight bias, as the references necessarily incorporate the realized framing and outcomes of the published work. This is load-bearing for the central claim, because the discrepancy between LLM and expert judgments is interpreted as evidence that experts correctly identify higher novelty in the references; without a quantitative check on reconstruction fidelity or inter-expert agreement, the observed preference may partly reflect alignment with published work rather than objective novelty.
  2. [Evaluation results] Evaluation results (abstract and results section): the abstract reports contradictory outcomes between LLM and human judgments but supplies no sample sizes, statistical tests, controls, or details on how RQs were generated and reconstructed. This prevents verification of the data support for the claim that LLM judges produce a novelty mirage while experts reach the opposite conclusion.
  3. [Analysis of narrow/source-bound RQs] Analysis of narrow/source-bound RQs (results/discussion): the observation that many generated RQs are narrow or source-bound and that LLM judges miss this dimension unless explicitly tested is central to the critique of LLM judging, yet the manuscript provides no quantitative measures, inter-rater statistics, or concrete examples to substantiate the claim.
minor comments (2)
  1. [Abstract] The abstract would be strengthened by including at least one key quantitative result (e.g., mean novelty scores or preference rates) alongside the qualitative description of the mirage.
  2. [Methods] Notation for 'standalone' versus 'comparative' LLM judging should be defined explicitly on first use to aid readability.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [RQ-Bench construction] the reconstruction of author-anchored RQs from a paper's own cited background, gaps, and contributions risks hindsight bias, as the references necessarily incorporate the realized framing and outcomes of the published work. This is load-bearing for the central claim, because the discrepancy between LLM and expert judgments is interpreted as evidence that experts correctly identify higher novelty in the references; without a quantitative check on reconstruction fidelity or inter-expert agreement, the observed preference may partly reflect alignment with published work rather than objective novelty.

    Authors: We acknowledge the risk of hindsight bias in RQ reconstruction. The author-anchored RQs are intentionally derived from each paper's own cited background, gaps, and contributions to serve as concrete, paper-specific reference points rather than claims of maximal novelty. We will expand the methods section with a fuller description of the reconstruction protocol and add an explicit discussion of this limitation and its implications for interpretation. We will also report inter-expert agreement from the human evaluation if not already quantified. The core empirical result—the divergence between LLM and expert judgments on identical RQs—remains informative even if absolute novelty is not claimed. revision: partial

  2. Referee: [Evaluation results] the abstract reports contradictory outcomes between LLM and human judgments but supplies no sample sizes, statistical tests, controls, or details on how RQs were generated and reconstructed. This prevents verification of the data support for the claim that LLM judges produce a novelty mirage while experts reach the opposite conclusion.

    Authors: We agree that the abstract and results section require these details for verifiability. In the revision we will insert sample sizes (number of papers, generated RQs, and evaluated items), statistical tests comparing LLM vs. expert preferences, and expanded methodological descriptions of both RQ generation and reconstruction procedures. revision: yes

  3. Referee: [Analysis of narrow/source-bound RQs] the observation that many generated RQs are narrow or source-bound and that LLM judges miss this dimension unless explicitly tested is central to the critique of LLM judging, yet the manuscript provides no quantitative measures, inter-rater statistics, or concrete examples to substantiate the claim.

    Authors: We will strengthen the results and discussion sections by adding quantitative counts of narrow or source-bound RQs, inter-rater agreement statistics for this classification, and representative examples. This will make the analysis of LLM limitations more concrete and reproducible. revision: yes

Circularity Check

0 steps flagged

No significant circularity; derivation relies on external human expert validation

full rationale

The paper constructs RQ-Bench by reconstructing author-anchored RQs from cited background/gaps/contributions of real papers and then compares LLM vs. human novelty judgments on model-generated RQs. No equations, fitted parameters, or self-citations are used in a load-bearing way that reduces any claim to the inputs by construction. The central result (LLM mirage vs. expert preference) is evaluated against independent domain-expert judgments, satisfying the criterion of self-contained external benchmarking. The reconstruction is presented explicitly as one set of reference points rather than a definitional or predictive tautology.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the domain assumption that human experts provide a reliable ground truth for novelty and that author-anchored RQs reconstructed from papers form a valid baseline; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Human expert judgments constitute a reliable ground truth for the scientific novelty of research questions.
    The paper treats expert preference as the counterpoint to LLM judgments.

pith-pipeline@v0.9.1-grok · 5775 in / 1224 out tokens · 29732 ms · 2026-06-27T07:39:13.944527+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

  1. [2]

    we leave X to future work

    it engages a gap in the background — explicit (the paper says “we leave X to future work”, “X is a limitation”, “X remains open”) OR implicit (the paper studies a phenomenon only in one regime, takes a premise for granted without justifying it, mentions a limitation only in passing, or relies on an assumption that is not validated)

  2. [3]

    creative

    it is not a trivial follow-up that any reader would propose. Critical guidance: Judge novelty ONLY against the provided BACKGROUND. Do not invoke outside knowledge of the field. If a question would be novel in the wider literature but is essentially restated in BACKGROUND, it is NOT novel here. Hierarchical scoring.Scoring is HIERARCHICAL. Decide RELEVANC...

    2026

  3. [4]

    its specific framing / object of inquiry is not present in the background

  4. [5]

    it engages a gap in the background — explicit or implicit

  5. [6]

    scores": [ {

    it is not a trivial follow-up that any reader would propose. This is a LISTWISE comparative evaluation. Consider all candidates together to calibrate what counts as more original, more gap-addressing, and more non-obvious for this BACKGROUND. Then assign each candidate its own rubric scores. Critical guidance: • Judge background-relative novelty: assess w...

    2026

  6. [7]

    This measures whether the research question depends on the specific background paper’s method, dataset, benchmark, component, result, behavior, or setting

    SOURCE-BOUNDEDNESS. This measures whether the research question depends on the specific background paper’s method, dataset, benchmark, component, result, behavior, or setting

  7. [8]

    Which failure modes does GRPO show under long-chain reason- ing?

    DIAGNOSTIC / TEST-LIST FRAMING. This measures whether the research question is framed as an analysis, ablation, correlation, causal test, predictor study, failure-case study, or list of measurable factors. These two scores are independent. A question can be: • source-bound and diagnostic: “Which failure modes does GRPO show under long-chain reason- ing?” ...

    2026

  8. [10]

    ?”. Sinhahajari et al. (2026). On the Limits of LLM-as-Judge for Scientific Novelty Assessment. 31 •Each question must be specific, not a generic question such as “how to improve X

    Propose research questions that, if answered, would directly close one or more of those gaps. Multi-paper rule When the user provides more than one background paper, a gap may apply to one paper, several papers, or all of them. Do not force every gap to be shared by every paper. If gaps from different papers converge on the same research question, group t...

    2026

  9. [11]

    The full LaTeX source of a paper

  10. [12]

    arXiv preprint arXiv:2502.01113 , year=

    ONE research question generated independently from that paper’s references. Your job: decide whether the paper’s own idea and contributions ACTUALLY ADDRESS the research question. “Address” means the paper proposes a method, analysis, or result that answers (in whole or substantial part) what the question asks. Be strict: •If the paper only mentions the t...

    work page arXiv 2026

  11. [13]

    Identify the research gaps in the literature: the limitations, weaknesses, unresolved problems, and unaddressed challenges that the work(s) leave open

  12. [14]

    ?”. Output format The final visible answer must be a single JSON object conforming exactly to the following schema: {

    Propose research questions that, if answered, would directly close one or more of those gaps. Multi-paper rule When the user provides more than one background paper, a gap may apply to one paper, several papers, or all of them. Do not force every gap to be shared by every paper. If gaps from different papers converge on the same research question, group t...

    2026