Pith. sign in

REVIEW 3 major objections 2 minor 49 references

Agent-filtered attempts at unsolved math problems improve Qwen3 models by 9.2 points on average.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The authors release ResearchMath-14k, the largest dataset of research-level math problems, and demonstrate that agent-filtered reasoning trajectories from open models improve fine-tuned Qwen3 models by 9.2 points on average.

T0 review reviewed 2026-06-29 challenge →

load-bearing objection The dataset release is the real contribution here, but the claims about curating unsolved research problems and the value of filtered trajectories rest on unverified assumptions. the 3 major comments →

arxiv 2605.28003 v1 pith:GINBK3EX submitted 2026-05-27 cs.CL

ResearchMath-14K: Scaling Research-Level Mathematics via Agents

Guijin Son , Seungyeop Yi , Minju Gwak , Hyunwoo Ko , Wongi Jang , Youngjae Yu This is my paper
classification cs.CL
keywords research-level mathematicsagentic filteringmathematical reasoningdataset curationlanguage model fine-tuningopen problemsreasoning trajectoriesQwen3 models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents ResearchMath-14k, a collection of 14,056 research-level mathematical problems drawn from academic sources through a multi-agent curation process. It also produces ResearchMath-Reasoning, a set of 220,000 reasoning trajectories generated by open models, which exhibit patterns such as non-attempts and fabricated references that grow more common in newer models. After an agentic filtering step removes low-quality traces, fine-tuning Qwen3 models ranging from 4B to 30B parameters produces an average gain of 9.2 points over the untuned baselines. The result indicates that partial or incorrect attempts at open problems can still deliver effective training signals. The authors release the dataset to support further work on advanced mathematical reasoning.

Core claim

A multi-agent pipeline can extract a large set of unsolved research problems and filter generated reasoning traces to yield useful supervision, delivering an average 9.2-point improvement when used to fine-tune Qwen3 models from 4B to 30B parameters even though the traces themselves are not fully correct.

What carries the argument

The multi-agent pipeline that curates research-level problems from academic sources and filters teacher trajectories to retain useful supervision signals.

Load-bearing premise

The multi-agent system accurately selects genuinely unsolved research problems and the filtering step keeps genuine learning value without adding bias or contamination.

What would settle it

Fine-tuned models show no gain over baselines when evaluated on a fresh set of unsolved problems that were never seen during curation or filtering.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Filtered trajectories from open problems supply usable training data even without correct final answers.
  • Newer open models produce 5.6 times more references and 5.0 times more fabricated references per trace than earlier ones.
  • The approach scales across model sizes from 4B to 30B parameters with consistent average gains.
  • Public release of the 14k-problem set enables further experiments on research-level reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same curation and filtering method could be applied to open problems in other technical fields to generate training data.
  • Models might learn to produce more reliable attempts on frontier problems by repeated exposure to filtered traces.
  • Larger-scale versions of the dataset could test whether gains continue to grow with model size or data volume.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper introduces ResearchMath-14K, a dataset of 14,056 research-level mathematical problems curated from academic sources via a multi-agent pipeline, claimed to be the largest such collection. It generates ResearchMath-Reasoning consisting of 220K teacher trajectories from two open models, documents avoidance behaviors (non-attempts, fabricated references) that increase in newer models, applies agentic filtering, and reports that fine-tuning Qwen3 models (4B–30B) on the filtered data yields an average 9.2-point improvement over base models. The work concludes that filtered open-problem attempts supply useful supervision even without fully correct traces and releases the dataset publicly.

Significance. If the curation and filtering steps are shown to reliably isolate genuine research-level unsolved problems while preserving useful (even if imperfect) reasoning signals, the result would be significant for scaling mathematical reasoning beyond contest problems. The public release of ResearchMath-14K is a concrete strength that enables follow-on work. The observed scaling of avoidance behaviors across model generations is also a useful empirical observation.

major comments (3)
  1. [Abstract] Abstract: the central empirical claim of a 9.2-point average improvement after agentic filtering is presented without any description of the evaluation benchmarks, the number of test problems, statistical significance testing, or comparisons against unfiltered trajectories or standard contest-problem fine-tuning. This information is required to assess whether the gains exceed ordinary data-augmentation effects.
  2. [Abstract] Abstract and § on dataset construction: the claim that the 14,056 problems are verifiably research-level and unsolved rests on the multi-agent pipeline alone; no human-expert validation, novelty checks against existing corpora, or overlap analysis with training data are reported. If these problems overlap with standard training sets or are not genuinely open, the interpretation that the gains demonstrate utility of research-level supervision does not follow.
  3. [Abstract] Abstract: the filtering step is asserted to retain useful supervision signals while removing only avoidance behaviors, yet no ablation (e.g., performance with unfiltered trajectories or with trajectories from non-research problems) or human inspection of retained vs. discarded traces is described. This is load-bearing for the claim that imperfect open-problem attempts are beneficial.
minor comments (2)
  1. [Abstract] The abstract states the dataset size as 14,056 but does not clarify how many problems were initially collected versus retained after filtering; adding this breakdown would improve reproducibility.
  2. [Abstract] The observation that newer models produce 5.6× more references and 5.0× more fake references is interesting but would benefit from a precise definition of “fake reference” and inter-annotator agreement if any human labeling was involved.

Simulated Author's Rebuttal

3 responses · 3 unresolved

We thank the referee for their constructive feedback. We address each major comment below. We agree the abstract requires expansion and will revise it accordingly. For the dataset validation and filtering claims, we clarify our methodology while noting practical limitations on additional experiments.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central empirical claim of a 9.2-point average improvement after agentic filtering is presented without any description of the evaluation benchmarks, the number of test problems, statistical significance testing, or comparisons against unfiltered trajectories or standard contest-problem fine-tuning. This information is required to assess whether the gains exceed ordinary data-augmentation effects.

    Authors: We agree that the abstract should provide more context on the evaluation. In the revised version, we will expand the abstract to name the evaluation benchmarks, indicate the number of test problems, note statistical significance testing, and reference comparisons against unfiltered trajectories and contest-problem fine-tuning. revision: yes

  2. Referee: [Abstract] Abstract and § on dataset construction: the claim that the 14,056 problems are verifiably research-level and unsolved rests on the multi-agent pipeline alone; no human-expert validation, novelty checks against existing corpora, or overlap analysis with training data are reported. If these problems overlap with standard training sets or are not genuinely open, the interpretation that the gains demonstrate utility of research-level supervision does not follow.

    Authors: The multi-agent pipeline sources problems from recent academic publications and applies agent-based verification to select likely unsolved research-level problems. We did not conduct or report human-expert validation, novelty checks, or overlap analysis with training data. We will add a limitations paragraph acknowledging this reliance on automated curation. The public release of the dataset enables independent verification by the community. revision: partial

  3. Referee: [Abstract] Abstract: the filtering step is asserted to retain useful supervision signals while removing only avoidance behaviors, yet no ablation (e.g., performance with unfiltered trajectories or with trajectories from non-research problems) or human inspection of retained vs. discarded traces is described. This is load-bearing for the claim that imperfect open-problem attempts are beneficial.

    Authors: The manuscript reports gains from the filtered trajectories over base models but does not include the requested ablations on unfiltered trajectories, non-research problems, or human inspection of traces. These analyses were outside the scope of the current study. We will not add them in this revision and maintain that the observed improvements provide evidence for the utility of the filtered data even without perfect traces. revision: no

standing simulated objections not resolved
  • Human-expert validation of the 14,056 problems as research-level and unsolved
  • Novelty checks against existing corpora and overlap analysis with training data
  • Ablations comparing filtered vs. unfiltered trajectories and human inspection of retained vs. discarded traces

Circularity Check

0 steps flagged

No significant circularity; empirical result independent of curation definitions

full rationale

The paper's core claim is an observed 9.2-point average gain after fine-tuning on agentically filtered trajectories, measured against base models. This is an external empirical outcome from training and (presumably held-out) evaluation rather than a quantity defined by the multi-agent curation or filtering process itself. No equations, self-citations, or ansatzes are shown that would make the reported improvement equivalent to the input filtering criteria by construction. The dataset originates from external academic sources, and the filtering step is presented as a preprocessing choice whose utility is tested downstream rather than presupposed. This is the normal case of a non-circular empirical paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

With only the abstract available, the central claim rests on the unverified effectiveness of the multi-agent curation pipeline and the agentic filtering step; no explicit free parameters, axioms, or invented entities are detailed.

reviewed 2026-06-29 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ResearchMath-14K: Scaling Research-Level Mathematics via Agents." pith.science (2026). https://pith.science/paper/GINBK3EX

@misc{pith2026260528003,
  author       = {Pith},
  title        = {Pith review of: ResearchMath-14K: Scaling Research-Level Mathematics via Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GINBK3EX}},
  note         = {Machine review of arXiv:2605.28003}
}
Share X Bluesky LinkedIn Reddit HN
abstract

The frontier of mathematics is defined by problems whose solutions are not yet known, yet it remains unclear whether language models can meaningfully engage with such problems without human intervention. A major obstacle is the lack of large-scale research-level math datasets. To this end, we introduce ResearchMath-14k, a set of $14{,}056$ problems curated from academic sources via a multi-agent pipeline, making it the largest collection of research-level mathematical problems to date. We further generate ResearchMath-Reasoning, $220$K teacher trajectories from two open models, where we observe recurring avoidance behaviors such as non-attempts and fabricated references. Interestingly, across eight open-weight models, newer generations produce $5.6\times$ more references and $5.0\times$ more fake references per trace. After agentic filtering of ResearchMath-Reasoning, fine-tuning Qwen3 models from 4B to 30B parameters improves over base models by $9.2$ points on average. This shows that filtered open-problem attempts can provide useful supervision even without fully correct reasoning traces. We make ResearchMath-14k publicly available for future works on research-level mathematical reasoning.

Figures

Figures reproduced from arXiv: 2605.28003 by Guijin Son, Hyunwoo Ko, Minju Gwak, Seungyeop Yi, Wongi Jang, Youngjae Yu.

Figure 1
Figure 1. Figure 1: Domain distribution of the RESEARCHMATH￾14K. Bubble size is proportional to the number of prob￾lems in each mathematical area. Logic and Foundations (455 problems, 3.2%) and Other/Cross-disciplinary (311 problems, 2.2%) are omitted from the visualization for readability. models increasingly appear to internalize parts of this curriculum (Alexeev et al., 2026a,b; Zheng et al., 2026). However, the open-sourc… view at source ↗
Figure 2
Figure 2. Figure 2: Agentic construction pipeline for RESEARCHMATH-14K. Starting from curated open-problem lists and research papers, an extractor agent maps document sections, detects candidate open questions, preserves supporting quotes, and rewrites them as standalone questions. A refiner agent then verifies open status, assigns taxonomy labels, and rewrites each candidate into a self-contained research problem before prod… view at source ↗
Figure 3
Figure 3. Figure 3: Elo Ratings for Difficulty Comparison. Ratings are computed from pairwise LLM difficulty judgments. All sources start at 1500 with k = 32; wins score 1, losses score 0, and draws score 0.5. Higher Elo means the source is judged more difficult more often. comparison is judged by GPT-5-mini along the three axes, producing win/loss/draw labels from which we compute Elo ratings. On all three axes, RESEARCHMATH… view at source ↗
Figure 4
Figure 4. Figure 4: Citation behavior across four matched older→newer model pairs. Left: Row-hit-rate deltas (newer minus older, in percentage points) for the three rule-based counters (abandon, assume, cite) across the five benchmarks. Right: Agent-Judge reference verification on 720 RESEARCHMATH-14K traces, one point per model. Newer models (orange) sit upper-right of their predecessors (blue), with more reference-like ment… view at source ↗
Figure 5
Figure 5. Figure 5: Fine-tuning results by benchmark. Bars show mean score for each model averaged across three runs; whiskers show standard deviation over three runs. to test the alternative explanation that any gain comes from learning the output format rather than from research-level content. We fine-tune Qwen3-4B/8B/30B-A3B-base with LoRA on each training set. See Appendix G for training configura￾tions. We evaluate on AI… view at source ↗
Figure 6
Figure 6. Figure 6: Pairwise Similarity Distribution. Empirical distributions of pairwise embedding similarities across all 98,778,540 problem pairs. Dashed vertical lines indicate the maximum similarity values, both below the 0.90 duplicate threshold. Left: similarities between original statements. Right: similarities between self-contained rewrites. D Near-Duplicates Filtering Details D.1 Pairwise Similarity Distribution [… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

49 extracted references · 5 canonical work pages · 3 internal anchors

  1. [1]

    Riemann-Bench: A Benchmark for Moonshot Mathematics

    Mathematical capabilities of chatgpt.Ad- vances in neural information processing systems, 36:27699–27744. Suhaas Garre, Erik Knutsen, Sushant Mehta, and Edwin Chen. 2026. Riemann-bench: A bench- mark for moonshot mathematics.arXiv preprint arXiv:2604.06802. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falk-...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    Proceedings of the 44th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval

    Let’s verify step by step. InInternational Con- ference on Learning Representations, volume 2024, pages 39578–39601. Junteng Liu, Yunji Li, Chi Zhang, Jingyang Li, Aili Chen, Ke Ji, Weiyu Cheng, Zijia Wu, Chengyu Du, Qidi Xu, and 1 others. 2025a. Webexplorer: Ex- plore and evolve for training long-horizon web agents. arXiv preprint arXiv:2509.06501. Zihan...

    work page arXiv 2024

  3. [3]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al

    Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data. InInter- national Conference on Learning Representations, volume 2025, pages 19243–19275. Shubham Toshniwal, Ivan Moshkov, Sean Narenthi- ran, Daria Gitman, Fei Jia, and Igor Gitman. 2024. Openmathinstruct-1: A 1.8 million math instruction tuning dataset.Advances in Ne...

    work page arXiv 2025

  4. [4]

    Qwen3 Technical Report

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Kaiyu Yang, Aidan Swope, Alex Gu, Rahul Chala- mala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J Prenger, and Animashree Anandkumar. 2023. Le- andojo: Theorem proving with retrieval-augmented language models.Advances in Neural Information Processing Systems, 36:21573–21612. Longhui Yu, Weisen Jiang,...

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    Holomorphic symplectic geometry: a problem list

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information pro- cessing systems, 36:46595–46623. 11 Appendix Contents A Example Source Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B Self-Containment Audit . . . . . . . . . . . . . . . . . ....

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    **Function class:** Problem 1 allows **meromorphic** functions; Problem 2 only allows **holomorphic** functions

  7. [7]

    **Range of (n):** Problem 1 assumes (n \ge 3); Problem 2 asks for **all (n \ge 1)**

  8. [8]

    Problem 2 asks for holomorphic normality, with limits **holomorphic or (\infty)**

    **Normality target:** Problem 1 allows subsequential limits to be **meromorphic or (\infty)** in the spherical metric. Problem 2 asks for holomorphic normality, with limits **holomorphic or (\infty)**. So they are related, but **not the same problem**. Table 9:GPT-5.5Judgment Example 3 17 Benchmarkassumerowsciterowsabandonrows Lemma-decomposition rows RES...

  9. [9]

    Read ‘$ARGUMENTS‘ and extract a single source

  10. [10]

    pdf"‘ - ‘source_locator = paper_path‘ - ‘source_name = <local_pdf_stem>‘ - ‘citation_mode =

    If input is a local PDF path: - Verify the file exists. - Use that absolute path as ‘paper_path‘. - Set: - ‘source_kind = "pdf"‘ - ‘source_locator = paper_path‘ - ‘source_name = <local_pdf_stem>‘ - ‘citation_mode = "page_number"‘

  11. [11]

    - Resolve the URL in this order:

    If input is a URL: - Normalize obvious wrappers while preserving the identity of the same source. - Resolve the URL in this order:

  12. [12]

    If the URL is ‘https://arxiv.org/abs/<id>‘, convert it to ‘https://arxiv.org/pdf/<id>.pdf‘

  13. [13]

    <pdf_url>

    If the URL is already a direct PDF URL, or the resolved response is a PDF, download it internally with Bash: - ‘mkdir -p TEMP_DIR‘ - ‘curl -L --fail "<pdf_url>" -o "TEMP_DIR/<paper_slug>.pdf"‘ - set ‘paper_path‘ to the downloaded file - set: - ‘source_kind = "pdf"‘ - ‘source_locator = paper_path‘ - ‘source_name = <arxiv_id_or_pdf_slug>‘ - ‘citation_mode =...

  14. [14]

    If the URL is a DOI URL or a bibliographic/landing page (for example Crossref, OpenAlex, DBLP, zbMATH, HAL, or a publisher landing page), inspect the page and follow the most specific full-text link available: - preferred targets: ‘PDF‘, ‘Download article‘, ‘Download extract‘, ‘document‘, ‘unpaywalled‘, ‘external edition‘, or an explicit article page whos...

  15. [15]

    If the URL is a workshop/problem-session/forum page whose own body contains the relevant problem statements, use the page itself as the primary source. - examples: AIM pages, workshop problem-session pages, MathOverflow threads, Stack Exchange sites (e.g., Physics SE, Biology SE, Cross Validated), domain-specific Q&A forums - for Q&A forum sources, prefer...

  16. [16]

    html"‘ - ‘source_locator = <canonical URL of the chosen page>‘ 20 - ‘source_name = <arxiv_id_or_doi_slug_or_url_slug>‘ - ‘citation_mode =

    If direct fetching fails because of redirects, robots/SSL issues, transient HTML fetch failures, or citation-only landing pages, search for the same source by exact title, DOI, repository identifier, or trailing URL slug and recover a readable artifact for that same work/source. Do not switch to a different work. - If the resolved primary source is HTML r...

  17. [17]

    Prefer the visible title from the chosen primary source itself: - PDF title page, first page heading, or explicit article/workshop title - HTML ‘<title>‘, ‘<h1>‘, main heading, or obvious visible page title

  18. [18]

    If the chosen primary source lacks a reliable visible title, use trusted resolver metadata for the same exact work/source

  19. [19]

    ai_extracted

    If no reliable title is available, infer a concise descriptive title from the readable source body and set ‘title_origin = "ai_extracted"‘

  20. [20]

    filename

    If the source is unreadable and no better title is available, fall back to the local filename stem or URL slug and set ‘ title_origin = "filename"‘. - Never leave ‘source.title‘ blank. ## Source Resolution Rules - Prefer the direct PDF of the exact source document when available. - Otherwise use full HTML text whose visible body actually contains the prob...

  21. [21]

    If the source has a stable identifier, try identifier-specific recovery first: arXiv PDF, DOI landing page, Crossref, OpenAlex, HAL API, zbMATH metadata, or explicit resolver links

  22. [22]

    If metadata provides a DOI, exact title, year, and authors, use them only to find a readable artifact for that same work

  23. [23]

    Use precise queries such as exact DOI, exact title plus ‘PDF‘, exact title plus first author, or repository identifier

    Run at most three exact-work search attempts after API/resolver recovery. Use precise queries such as exact DOI, exact title plus ‘PDF‘, exact title plus first author, or repository identifier. Avoid broad topical searches

  24. [24]

    Accept a candidate only when the visible title or DOI confirms the same exact work/source

    Inspect at most five candidate pages or files from those searches. Accept a candidate only when the visible title or DOI confirms the same exact work/source

  25. [25]

    source": {

    Stop immediately once remaining candidates are only snippets, abstracts, citation metadata, paywalls, blocked pages, or different works. Failure output rules: - Do not extract questions from metadata, abstracts, search snippets, title text, or related papers. - Do not keep retrying equivalent blocked resolver URLs once the same failure mode is established...

  26. [26]

    this", "above

    Self-containedness gate: - no dangling references like "this", "above", "as discussed", "see Section X" unless expanded - if a reader would need source-specific notation, terminology, definitions, experimental conditions, datasets, evaluation criteria, or domain-specific context to understand the question, include them directly in ‘question_text‘ when sup...

  27. [27]

    Evidence coverage gate: - accepted items must include at least one evidence quote with a page number or stable HTML locator

  28. [28]

    Evidence completeness gate - accepted items must include complete sentences or a contiguous self-contained problem statement as evidence; otherwise, move them to ‘needs_review‘

  29. [29]

    No-new-facts gate: - rewritten content, including any added definitions or settings in ‘question_text‘, must not introduce unsupported claims

  30. [30]

    Source adequacy gate: - do not accept items extracted only from abstracts, citation metadata, link labels, thread titles, or search snippets when the technical problem statement is not visible in the chosen source text

  31. [31]

    page": <number_or_string>,

    Schema conformance gate: - accepted items must match the JSON output schema exactly and contain all required fields Any failed item goes to ‘needs_review‘ with specific ‘reason‘. Add one ‘trace‘ entry per candidate with stage ‘quality_gates‘. ## Required Prompt Clauses ## EVIDENCE_CITATION - Every extracted or rewritten question must be backed by verbatim...

  32. [32]

    Create output directory: - ‘RUN_DIR‘

  33. [33]

    Generate output file name from the normalized source name: - ‘<source_name_sanitized>.json‘ (local PDF stem, arXiv id, DOI slug, or URL slug) - Sanitization rule: replace every ‘/‘ with ‘_‘ and every remaining non-alphanumeric character except ‘-‘ and ‘.‘ with ‘_‘; do not use ‘-‘ as a slash replacement

  34. [34]

    Save the exact JSON output to: - ‘RUN_DIR/<source_name_sanitized>.json‘

  35. [35]

    Also write/update: - ‘LATEST_FILE‘

  36. [36]

    If save fails, still return JSON but add a ‘needs_review‘ item with reason ‘persist_failed‘ and include failure details in ‘ trace‘

    Append a ‘trace‘ entry noting save paths under stage ‘persist‘. If save fails, still return JSON but add a ‘needs_review‘ item with reason ‘persist_failed‘ and include failure details in ‘ trace‘. Table 11: Prompt for the Extractor Agent Prompt for the Refiner Agent I will give you an arXiv paper ID and a draft question. Your job is to read the paper itse...

  37. [37]

    - Prefer the nearest official formulation in the paper

    Search the paper and identify the exact statement, conjecture, or open problem that my draft is trying to capture. - Prefer the nearest official formulation in the paper. - Do not replace it with a broader neighboring problem

  38. [38]

    - Check the introduction first

    Read the exact statement and the nearby context that defines notation, scope, and hidden assumptions. - Check the introduction first. - Then check any proposition/theorem/conjecture/remark/hypothesis that supplies equivalent conditions, notation, or a more precise reformulation

  39. [39]

    - Say what is already correct

    Evaluate how complete my draft question is on a 0 to 10 scale. - Say what is already correct. - List every missing or implicit detail needed to make it fully stand-alone and paper-faithful. - Include domain-specific definitions (e.g. variables, operators, experimental conditions, model assumptions, scope restrictions, quantifiers) that a reader outside th...

  40. [40]

    Rewrite the question as one refined research question that is: - self-contained (no unexplained symbols, jargon, or implicit scope), - tightly scoped, - as close as possible to the paper’s original intent, - not broader than the paper, - not vague

  41. [41]

    - Do not invent new variants

    Give at most one alternate formulation only if the paper itself explicitly proves the equivalence. - Do not invent new variants. - Do not generalize unless the paper itself does so in the exact nearby discussion

  42. [42]

    refined_question

    Guardrails: - Do not broaden a specific narrow question into a more general problem unless the paper explicitly treats that generalization as its main target. - Do not drift into side problems, variants, or adjacent subfields unless my draft explicitly asks for them. 25 - Do not add speculative assumptions or remove paper-critical restrictions (e.g. speci...

  43. [43]

    target_in_paper

    Support every nontrivial claim by citing the exact place in the paper: - section number, - conjecture/proposition/theorem/open problem number, - and page number if available. Output requirements: Return ONLY a valid JSON object. The JSON must contain the following fields: - "target_in_paper": string - "completeness_score": number - "correct_elements": str...

  44. [44]

    Copy the complete exact substring from the block for each external reference

  45. [45]

    Every string must be a contiguous substring that appears verbatim in the block

  46. [46]

    Do not paraphrase, normalize spelling, add positions, or add text that is not in the block

  47. [47]

    If one citation contains another marker, output only the larger complete citation/source phrase

    Do not output overlapping strings. If one citation contains another marker, output only the larger complete citation/source phrase

  48. [48]

    If there is no external reference, output []

  49. [49]

    keep this as a fake reference

    Output JSON only. No Markdown, no explanation. Block: <<<BLOCK {block} BLOCK Table 15: Prompt for factuality reference-span extraction Prompt for Factuality Metric: Agent Verification You are verifying one detected external-reference span from a mathematical reasoning trace. Important: - You MUST use web search before deciding. - Return exactly one lowerc...

This paper was first reviewed by grok-4.3 on June 29, 2026.