Pith. sign in

REVIEW 3 major objections 5 minor 4 cited by

This paper claims that rebuttal writing is best treated as an evidence-planning task, and that a multi-agent 'verify-then-write' pipeline beats direct LLM generation on coverage, faithfulness, and coherence.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:16 UTC pith:LEYJWNEI

load-bearing objection Useful multi-agent scaffolding and a real benchmark, but the drafter prompt tells the model to invent results, which guts the faithfulness claim and contaminates the LLM-judged gains as reported. the 3 major comments →

arxiv 2601.14171 v2 pith:LEYJWNEI submitted 2026-01-20 cs.AI

Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance

Qianli Ma , Chang Guo , Zhiheng Tian , Siyu Wang , Jipeng Xiao , Yuanhao Yue , Zhipeng Zhang This is my paper
classification cs.AI
keywords rebuttal generationmulti-agent LLMevidence groundingpeer reviewverify-then-writeatomic concernshybrid contextLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

RebuttalAgent claims that writing an effective response to peer reviewers should be treated as an evidence-planning problem, not a text-generation problem. It breaks each review into atomic concerns, builds a context that mixes a compressed view of the manuscript with verbatim passages and, when needed, citation-ready summaries of external work, and produces an inspectable response plan before drafting. On the paper's RebuttalBench, this plan-first pipeline beats direct-to-text prompting with the same underlying models on coverage, traceability, and coherence. If the claim holds, rebuttal assistance becomes a transparent, human-checkable workflow instead of a black box, and smaller models can produce substantially better rebuttals with structured scaffolding.

Core claim

The paper's central discovery is that the bottleneck in automated rebuttal writing is not fluency but the organization of decisions and evidence. RebuttalAgent enforces a verify-then-write workflow in which reviewer comments are atomized into discrete concerns, a query-conditioned hybrid context combines compressed manuscript summaries with high-fidelity excerpts, an on-demand search module assembles citation-ready external evidence briefs, and a strategist agent produces an explicit plan with to-do items and commitment checks before any final text is drafted. The paper reports that this pipeline outperforms strong direct-to-text baselines under matched base models, with the largest gains in

What carries the argument

The central mechanism is the multi-agent 'verify-then-write' pipeline: an extractor atomizes reviews into atomic concerns with coverage checks; a parser and compressor turn the manuscript into a paragraph-indexed compressed representation with fidelity checks; evidence construction builds a concern-conditioned hybrid context (compressed text plus verbatim passages) and, on demand, external literature briefs; a strategist generates an inspectable response plan separating interpretative defense from necessary intervention; checkers verify coverage, evidence linkage, and cross-point consistency; and a drafter realizes the validated plan, rendering unrun experiments as explicit placeholders.

Load-bearing premise

The headline outperformance claim rests on a rubric-based LLM judge as a proxy for rebuttal effectiveness; the real reviewer follow-up signals collected in the benchmark are used only to select papers, not to score responses, so if the judge rewards the agent's structured format rather than the substance of the argument, the central claim collapses.

What would settle it

Take the benchmark's positive and negative reviewer follow-up labels as the outcome instead of the LLM-judge scores, or have human experts score rebuttals with the structured formatting hidden. If RebuttalAgent no longer beats the direct-to-text baselines in either condition, the outperformance claim is an artifact of the judge or the format.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Every argument in a rebuttal can be traced to a specific passage in the manuscript or a specific external reference, making the draft auditable before submission.
  • Concerns that demand new experiments are turned into action items with placeholders, so the model cannot silently fabricate results.
  • Authors can intervene at an inspectable plan stage and retain final control over strategy and wording rather than editing opaque generated prose.
  • Because the gains are larger for weaker base models, structured scaffolding can partly substitute for raw model capability in high-stakes writing tasks.
  • The ablation result that external evidence briefs are the most critical artifact implies that citation-ready literature synthesis deserves priority in similar systems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same verify-then-write decomposition could apply to other high-stakes response documents, such as grant rebuttals, ethics responses, or regulatory replies, where claims must be anchored and unrun work must be flagged; the paper does not test these.
  • The benchmark already collects reviewer follow-up labels indicating whether concerns were resolved; using those labels as a direct outcome metric, instead of the rubric judge, would test whether the agent's format advantage translates into real reviewer satisfaction.
  • A minimal variant that only adds the external-evidence briefs to a direct-to-text baseline might capture much of the observed gain, since ablations show that module drives the largest drops when removed.
  • The structured plan itself could be reused as a checklist for human authors, even without automated drafting, turning the agent into a planning aid rather than a writer.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes RebuttalAgent, a multi-agent framework that reframes rebuttal generation as an evidence-centric planning task. The pipeline decomposes reviews into atomic concerns, builds concern-conditioned hybrid contexts from compressed and raw manuscript text, optionally retrieves external references, generates an inspectable response plan with checkers, and then drafts a formal rebuttal. The authors introduce RebuttalBench, derived from ICLR OpenReview threads, and evaluate RebuttalAgent and direct-to-text LLM baselines using a Gemini-3-Flash judge with a nine-component rubric. They report consistent gains in relevance, argumentation, and communication quality, with ablations indicating that evidence construction is the most critical module.

Significance. If the results held, the contribution would be meaningful: the system produces inspectable intermediate artifacts, distinguishes interpretive defense from required new experiments, and offers a benchmark with real reviewer follow-up signals. The matched-backbone comparison is a fair design, and the rubric is detailed. However, the central faithfulness claim is directly contradicted by the drafting prompt in Appendix D, which instructs the model to invent experimental values. Moreover, all headline results are single-point LLM-judge scores with no human validation, error bars, or significance tests, and the benchmark's own positive/negative follow-up labels are not used as the outcome metric. These issues make the current evidence insufficient for the paper's strong claims, although the architectural ideas remain worth developing.

major comments (3)
  1. [Appendix D, Prompt: Rebuttal Letter Writer; §3.3; §5.1] The final drafting prompt explicitly instructs the model: 'Since you are an AI and cannot perform actual experiments... you must invent plausible, realistic values/results that support the user's argument,' with an asterisk marker. This is the Drafter Agent in the main pipeline. Section 3.3 claims the system 'explicitly inhibits result generation' and renders missing experiments as '[TBD]' placeholders. Since Section 5.1 runs all main experiments in fully automated mode with no human intervention, the evaluated RebuttalAgent outputs can contain fabricated numerical results. The A2 Evidence Support rubric in Appendix D explicitly rewards 'new quantitative results,' so the judge may assign higher scores precisely for the hallucinated content that the paper claims to prevent. This is an internal inconsistency that directly undermines the strict faithfulness claim and contaminates the headli
  2. [§4.1, §4.2, §5.1, Table 1] RebuttalBench collects reviewer follow-up responses and uses them as a 'decisive classification signal' to label instances positive or negative, but these labels are never used as the outcome metric. All Table 1 results are scores from a single closed-source LLM judge, Gemini-3-Flash, with no human validation, no reported variance, and no significance tests. The rubric is authored by the same team, and the judge model is also one of the agent backbones. While a shared judge can be a reasonable comparison tool, the central claim that RebuttalAgent 'outperforms strong baselines in coverage, faithfulness, and strategic coherence' is currently supported only by an unvalidated proxy. The authors should report human agreement, error bars, significance tests, or validate the judge against the follow-up labels they collected.
  3. [§5.3, Table 2] The ablation results are quantitatively weak and partly contradictory to the narrative. Removing Checkers improves Coverage (+0.03) and Logic Consistency (+0.02); removing Input Structuring improves Statement Clarity (+0.05); most other deltas are within 0.1–0.3. Without error bars or significance tests, the text's claim that 'checkers also contribute measurably to overall quality' is unsupported. The conclusion that 'the gains of RebuttalAgent arise from the combination of complementary modules' overstates what Table 2 shows. Report confidence intervals or significance tests before drawing these conclusions.
minor comments (5)
  1. [§6, §3, §5.3] There are name inconsistencies: the conclusion calls the system 'RESPONSEAGENT,' the contributions section says 'REBUTTALGENT,' and Section 5.3 writes 'ebuttalAgent.' Use one consistent name throughout.
  2. [§4.1] The sentence 'we apply automatic filtering to retain instances with sufficiently explicit follow-up signals and discard ambiguous cases' appears twice nearly verbatim in the 'Filtering and corpus construction' paragraph.
  3. [References] Some references are incomplete or informal, e.g., 'xGr', 'goo', and a URL-style citation for 'Gemini Deep Research.' These need to be formatted consistently.
  4. [Appendix E, Case Study] A planned deliverable in the second case study says 'showing strong positive correlation (r >0.9)' as if it were a concrete result. Since this is a plan, it should be marked as a placeholder or target, otherwise it exemplifies the same invented-number problem at the planning stage.
  5. [§5.1, Abstract] The paper claims to be the 'first multi-agents framework' for rebuttal, but the related work already includes multi-agent simulations of peer review (e.g., AgentReview). The novelty claim should be softened to avoid overclaiming.

Circularity Check

1 steps flagged

RebuttalAgent's A2 'Evidence Support' gains are authored by the system's own drafter prompt: the judge rubric rewards new quantitative results, and the prompt commands the model to invent them.

specific steps
  1. fitted input called prediction [Appendix D ('Prompt: Rebuttal Letter Writer') and Appendix D ('Prompt: Unified Rebuttal Evaluation', A2 Evidence Support); Sec. 3.3; Table 1]
    "Since you are an AI and cannot perform actual experiments, but the rebuttal might require empirical evidence (e.g., ablation studies, baseline comparisons), you must invent plausible, realistic values/results that support the user's argument. ... MANDATORY RULE: Any time you generate a specific number, metric, or experimental result that is not present in the input, you MUST append a distinct asterisk symbol (*) right after it. ... A2 Evidence Support ... 5 Backed by new quantitative results, specific comparative data, or rigorous mathematical derivations presented directly in the rebuttal."

    The drafter prompt instructs the model to fabricate the exact content the evaluation rubric rewards: new quantitative results and comparative data. RebuttalAgent therefore receives high A2 'Evidence Support' scores and the headline Argumentation Quality gains in Table 1 by construction, while the unmodified baselines are not given the 'invent' instruction. The paper claims in Sec. 3.3 that the system 'explicitly inhibits result generation' and renders missing experiments as [TBD], but the final-draft prompt does the opposite. Thus the 'faithfulness' improvement is not an independent measurement; it is a self-fulfilling product of the authors' own rubric and their own system prompt. The Appendix E plan even contains 'strong positive correlation (r >0.9)' as a deliverable, showing the patter

full rationale

The central circularity is the evidence-support metric: the authors design a judge rubric that awards top marks for 'new quantitative results' and simultaneously instruct their Drafter Agent to invent exactly such results, with an asterisk as a placeholder. Since main-paper experiments run the full system in automated mode (Sec. 5.1), the reported A2 gains and the 'faithfulness' claim are not validated against verifiable manuscript content; they are produced by the system's own instruction interacting with the authors' own rubric. This is a partial but real circularity: coverage, structure, and communication improvements may still be meaningful, but the headline claim of outperforming baselines in faithfulness and argumentation quality is confounded by construction. The benchmark's reviewer follow-up labels, described as a 'decisive classification signal' (Sec. 4.1), are used only to select papers rather than as outcomes, further weakening external anchoring, though that is a validity issue rather than a circular one. No load-bearing self-citation chain or ansatz-smuggling was found. Score 6.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central empirical claim rests on benchmark-selection thresholds, a single LLM judge, and domain assumptions about reviewer-follow-up validity and LLM self-verification. No new physical or mathematical entities are introduced.

free parameters (4)
  • Tier confidence thresholds = 0.7 and 0.4
    Appendix A uses LLM confidence >=0.7 for Tier 2 and 0.4<=conf<0.7 for Tier 3 to stratify RebuttalBench instances. These hand-set cutoffs affect which review-rebuttal pairs enter the benchmark and therefore all reported scores.
  • Top-20 / >100-reviewer selection rule = 20 papers, >100 reviewers
    Sec. 4.1 constructs RebuttalBench-CHALLENGE by ranking papers by instance count and selecting the top 20 with over 100 reviewers. This hand-set rule shapes the difficulty and diversity of the test suite.
  • Equal rubric weights = 1/3 each for R, A, C
    Appendix B computes the final score as an unweighted average of Relevance, Argumentation, and Communication. The choice of equal weights is not justified and directly affects the headline averages.
  • LLM judge model choice = Gemini-3-Flash
    Sec. 5.1 fixes Gemini-3-Flash as the unified LLM judge for all systems and ablations. All scores depend on this model choice, and no alternative judge or human validation is reported.
axioms (4)
  • domain assumption Reviewer follow-up reactions and score changes validly indicate whether a rebuttal resolved concerns.
    Sec. 4.1 uses follow-up comments as a 'decisive classification signal' to split RebuttalBench into positive and negative instances. If this mapping is noisy, the benchmark's construction is compromised.
  • domain assumption An LLM judge's 0-5 rubric scores faithfully measure rebuttal quality.
    All headline results rely on Gemini-3-Flash judging outputs (Sec. 4.2, 5.1). No correlation with the benchmark's own follow-up labels or human-evaluation calibration is reported.
  • domain assumption Compressing the manuscript and verifying with an LLM preserves all evidence needed for faithful grounding.
    Sec. 3.1 uses a compressor plus an LLM-based consistency checker to build the hybrid context. This is a self-check by the same type of model, not an independent verification of information preservation.
  • domain assumption External search via the arXiv API returns papers that can be safely cited without introducing factual errors.
    Sec. 3.2 retrieves and summarizes external literature into 'citation-ready briefs'. No audit of the selected papers or briefs is reported, so errors in retrieval or summarization could propagate into the rebuttal.

pith-pipeline@v1.3.0-alltime-deepseek · 23082 in / 12080 out tokens · 111340 ms · 2026-08-03T09:16:53.195430+00:00 · methodology

0 comments
read the original abstract

Writing effective rebuttals is a high-stakes task that demands more than linguistic fluency, as it requires precise alignment between reviewer intent and manuscript details. Current solutions typically treat this as a direct-to-text generation problem, suffering from hallucination, overlooked critiques, and a lack of verifiable grounding. To address these limitations, we introduce $\textbf{RebuttalAgent}$, the first multi-agents framework that reframes rebuttal generation as an evidence-centric planning task. Our system decomposes complex feedback into atomic concerns and dynamically constructs hybrid contexts by synthesizing compressed summaries with high-fidelity text while integrating an autonomous and on-demand external search module to resolve concerns requiring outside literature. By generating an inspectable response plan before drafting, $\textbf{RebuttalAgent}$ ensures that every argument is explicitly anchored in internal or external evidence. We validate our approach on the proposed $\textbf{RebuttalBench}$ and demonstrate that our pipeline outperforms strong baselines in coverage, faithfulness, and strategic coherence, offering a transparent and controllable assistant for the peer review process.

Figures

Figures reproduced from arXiv: 2601.14171 by Chang Guo, Jipeng Xiao, Qianli Ma, Siyu Wang, Yuanhao Yue, Zhiheng Tian, Zhipeng Zhang.

Figure 1
Figure 1. Figure 1: Overview of our work. Given a manuscript and reviews, (a) direct text generation (SFT on peer￾review corpora) often fabricates experiment results and prone to hallucination. (b) Interactive prompt￾ing with chat-LLMs depends on manual concern feed￾ing and many iterations. (c) RebuttalAgent reframes rebuttal writing as a decision-and-evidence organiza￾tion problem, performing concern breakdown, query￾conditi… view at source ↗
Figure 2
Figure 2. Figure 2: Overview of RebuttalAgent. Given a manuscript (PDF) and reviewer comments, the system (1) structures inputs by parsing and compressing the paper with fidelity checks and extracting atomic reviewer concerns with coverage checks; (2) builds concern-conditioned evidence by constructing a query-specific hybrid manuscript context and, when needed, retrieving and summarizing external literature into citation-rea… view at source ↗
Figure 3
Figure 3. Figure 3: RebuttalBench statistics and rubric design. (a) Word-cloud and top-word histogram of reviews in REBUTTALBENCH-CORPUS, highlighting recurring reviewer emphases (e.g., clarity, novelty, reproducibility). (b) Motivated by these signals, REBUTTALBENCH evaluates rebuttals with a rubric that mirrors these concerns, scoring Relevance, Argumentation Quality, and Communication Quality rather than fluency alone. con… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OmniPresent: Generating Coherent Presentation Suites from Scientific Papers

    cs.SE 2026-07 conditional novelty 6.5

    A multi-agent HTML pipeline with shared knowledge and cross-artifact verify-and-repair generates coherent poster/slides/video/page suites from papers and beats specialized baselines on OmniPreBench.

  2. Position: Academic Conferences are Potentially Facing Denominator Gaming Caused by Fully Automated Scientific Agents

    cs.CL 2026-05 unverdicted novelty 6.0

    Malicious actors could use AI agents to submit large numbers of fake papers, inflating the submission count and thereby raising the acceptance odds for a small set of chosen legitimate papers under stable conference a...

  3. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 unverdicted novelty 4.0

    The paper delivers a stage-by-stage roadmap for AI in research, showing reliable assistance in retrieval and tool tasks but fragility in novelty and judgment, advocating human-governed collaboration.

  4. AI for Auto-Research: Roadmap & User Guide

    cs.AI 2026-05 conditional novelty 4.0

    AI can generate research artifacts faster than it can verify them, so across all eight lifecycle stages the credible deployment mode is human-governed collaboration rather than full autonomy.

Reference graph

Works this paper leans on

38 extracted references · 7 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Outcome-based Classification:We first cat- egorize entries intoImproved(review score or acceptance status increased) andUnimproved groups based on the final decision

  2. [2]

    Reliability-based Stratification:To ensure data quality, we subdivide these groups into three tiers based on evidence objectivity and LLM confidence:Tier 1 (Gold Standard) comprises cases with objective score increases (initial ̸= final) or explicit revision statements; Tier 2 (High Confidence)includes instances without score changes but where an LLM iden...

  3. [3]

    Ground Truth Curation:From this stratified data, we curate a balanced test set of 20 repre- sentative papers, prioritizing those with high review volumes to ensure diverse coverage of both positive and negative review samples across tiers

  4. [4]

    Jamshid Sourati and James A Evans

    Reflexion: Language agents with verbal rein- forcement learning.Preprint, arXiv:2303.11366. Jamshid Sourati and James A Evans. 2023. Accelerat- ing science with human-aware artificial intelligence. Nature human behaviour, 7(10):1682–1696. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew ...

    Pith/arXiv arXiv 2023

  5. [5]

    arXiv preprint arXiv:2505.07920

    Re 2: A consistency-ensured dataset for full- stage peer review and multi-turn rebuttal discussions. arXiv preprint arXiv:2505.07920. Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji- Rong Wen. 2024. A survey on the memory mecha- nism of large language model based agents.Preprint, arXiv:2404.13501. Lianmin Zhe...

    arXiv 2024

  6. [6]

    Shuyan Zhou, Frank F

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Gra- ham Neubig. 2024. Webarena: A realistic web envi- ronment for building autonomous age...

    Pith/arXiv arXiv 2024

  7. [7]

    Deepreview: Improving llm-based paper re- view with human-like deep thinking process.arXiv preprint arXiv:2503.08569. 10 A Evaluation Dataset To construct a robust benchmark for evaluating rebuttal effectiveness, we derive our data from the RE 2 dataset (Zhang et al., 2025), focusing on the ICLR 2023 subset (approximately 9,310 en- tries). We process this...

    Pith/arXiv arXiv 2025

  8. [11]

    how” vs. “what

    Baseline Generation Protocol:For each pa- per, the baseline runs multi-round rebuttal generation following the author-reviewer dia- logue. Each round uses a fixed prompt (includ- ing intent, required format, and guardrails), concatenating the paper text, the current re- view, and an optional prior-round abstract. The rebuttal is then summarized into a fac...

    2024

  9. [12]

    Granularity

    Consolidate: Merge issues that represent thesame core objectionand can be addressed with thesame response logic. 4.Format: Output strictly according to the traceability requirements. CRITICAL RULES FOR MERGING & SPLITTING (The "Granularity" Logic) •Do NOT Merge (Split them): – Different Evidence Needed: If R1 asks for "Comparison with Baseline X" and R2 a...

    2023

  10. [13]

    Granularity

    Consolidate: Merge issues that represent thesame core objectionand can be addressed with thesame response logic. 4.Format: Output strictly according to the traceability requirements. 14 CRITICAL RULES FOR MERGING & SPLITTING (The "Granularity" Logic) •Do NOT Merge (Split them): – Different Evidence Needed: If R1 asks for "Comparison with Baseline X" and R...

    2023

  11. [14]

    The reviewer explicitly mentions reference papers

  12. [15]

    Thereview_questioncontains specific method names or dataset names that arenotfrom the current paper

  13. [16]

    compare with X / ablation on Y / baseline Z

    The reviewer requests “compare with X / ablation on Y / baseline Z”

  14. [17]

    need_search

    The content of the paper is insufficient to answer the question. When Search Is NOT Required If thepaper_summaryalready contains evidence that can directly answer the reviewer’s question (e.g., existing experiments, tables, section explanations), or the question concerns only minor formatting issues, then no search is needed. Search Query Generation Rules...

    Pith/arXiv arXiv

  15. [18]

    selected_papers

    How it helps the rebuttal of the current problem (brief description in one paragraph) Anti-Redundancy (with explanation): If multiple papers come from the same source or use the same method, only keep the most relevant one. You must output your result in the following JSON format: { "selected_papers": [1,3,6], "reason": "..." } Theselected_papersarray sho...

  16. [19]

    You should take these as your rebuttal strategy

    [rebuttal_idea and to_do_list]: Prepared by your team for each merged question. You should take these as your rebuttal strategy. Note that your output should be specifically answered in combination with each reviewer’s question. Guidelines & Constraints

  17. [20]

    Do not make any mistakes regarding the reviewers’ questions, or confuse the questions of the first reviewer with those of the second reviewer

    You should precisely identify each reviewer’s questions from[review original text], and then, following the order provided, find the corresponding response ideas in[rebuttal_idea and to_do_list]and generate the responses. Do not make any mistakes regarding the reviewers’ questions, or confuse the questions of the first reviewer with those of the second re...

  18. [21]

    We thank the reviewer for the insightful comment

    Tone:Professional, respectful, objective, and grateful. Even if the reviewer is harsh, your 19 response must be diplomatic (e.g., "We thank the reviewer for the insightful comment..."). Respect every reviewer. Do not generate statements that require a particular reviewer to read the response to another reviewer. 3.Format: • Use standard ICLR rebuttal form...

  19. [22]

    Except for the asterisk, it should not be immediately recognizable as an AI-written rebuttal, but should be as close as possible to a real person

    Although the supplementary experimental data in your final output is speculative (marked with an asterisk), you still need to ensure that your output is very formal, just like a real rebuttal. Except for the asterisk, it should not be immediately recognizable as an AI-written rebuttal, but should be as close as possible to a real person. Your output shoul...

  20. [23]

    Upgrade Criteria

    The responses to each split question can include tables to visually present the experimental result numerical data to improve readability. But don’t use tables to specifically present text! Don’t put q1, response to q1, q2, response to q2 in a large table. Instead, list them separately. Prompt: Unified Rebuttal Evaluation You are an EXPERIENCED and DISCER...

  21. [24]

    Draft numbered Proposition 1 with explicit notation and enumerated assumptions

  22. [25]

    3; reformat Appendix A.1 with numbered lemmas

    Write proof sketch for Sec. 3; reformat Appendix A.1 with numbered lemmas

  23. [26]

    Compute per-layerk-NN error, shallow head error, and final model error plots

  24. [27]

    Compute|f knn −0.5|vs layer curves; tabulate PD-separation fractions

  25. [28]

    Compute Pearson/Spearman correlations for PD vs−H Vcnn

  26. [29]

    Create controlled synthetic validation and robustness tables

  27. [30]

    usable" information in Ds is strictly higher than in Di. Improvement:In the final version, we will move the formal definitions of V -information to the main text and add a

    Produce flow diagram and commit scripts to repository. Feasibility:All planned items use existing model checkpoints, intermediate embeddings, k-NN heads, small auxiliary classifiers, or lightweight synthetic datasets. No retraining of large backbones from scratch or new large-scale dataset collection is required. The planned experiments are deliberately s...

  28. [31]

    Upper Bound

    Rewrite Prop 1 exposition to emphasize "Upper Bound" and list assumptions (linear head, bounded NTK). 25

  29. [32]

    Formalize and write the proof for the Monotonicity Lemma in Appendix D

  30. [33]

    Run diagnostics on 4 settings (Toy MLP, Sup-IN, SimCLR, Graph): Compute per-τ AIE and feature distance

  31. [34]

    Generate scatter plots, calculate Pearson/Spearmanρ, and fit linear regression (α·AIE+β)

  32. [35]

    Measuring non-trivial compositionality

    Estimate constants (C 1, C2) on a subset to plot the theoretical bound linec·AIE. Feasibility:The theoretical work involves derivations based on existing Appendix D material. The empirical diagnostics use existing checkpoints and require only lightweight forward passes to compute feature norms and correlations. No training is required. Summary:We resolve ...

  33. [36]

    Decodability:Implement k-NN to classify constituents of u(rij) using u(ri) library; report accuracy

  34. [37]

    3.Linearity:Calculate MSE forz(r ij)≈w iz(ri) +w jz(rj)in the latent space

    Validation:Curate 10-15 public ideograms, manually decompose them, and compute ρ using our pipeline. 3.Linearity:Calculate MSE forz(r ij)≈w iz(ri) +w jz(rj)in the latent space

  35. [38]

    proof" to using it as a

    Ablation:Compare decodability/linearity between descriptive vs. discriminative generation strategies. Feasibility:High. The decodability and latent space analyses use existing trained models and require only lightweight inference scripts (k-NN, linear regression). The real-world validation requires a small manual dataset collection but utilizes the existi...

  36. [2023]

    Preprint, arXiv:2311.12983

    Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983. OpenAI. 2025. Gpt-5 system card. Technical report. Available at: https://cdn.openai.com/gpt-5-system- card.pdf. 9 Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. Gorilla: Large lan- guage model connected with massive apis.Preprint, arXiv:2305.15334. Sukannya ...

    Pith/arXiv arXiv 2025

  37. [2024]

    Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and 1 others

    Reviewer2: Optimizing review genera- tion through prompt generation.arXiv preprint arXiv:2402.10886. Yichen He, Guanhua Huang, Peiyuan Feng, Yuan Lin, Yuchen Zhang, Hang Li, and 1 others. 2025. Pasa: An llm agent for comprehensive academic paper search.arXiv preprint arXiv:2501.10120. Xiang Hu, Hongyu Fu, Jinge Wang, Yifeng Wang, Zhikun Li, Renjun Xu, Yu ...

    Pith/arXiv arXiv 2025

  38. [2025]

    Yen-Ting Lin and Yun-Nung Chen

    Curie: Toward rigorous and automated scien- tific experimentation with ai agents.arXiv preprint arXiv:2502.16069. Yen-Ting Lin and Yun-Nung Chen. 2023. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language mod- els.arXiv preprint arXiv:2305.13711. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingx- uan ...

    Pith/arXiv arXiv 2023