{"id":"3c0ea05c-4eb8-4e73-8061-a921dd91d9e7","arxiv_id":"2509.16780","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On a 477-question page-level math textbook benchmark, embedding-based RAG with voyage-3-large reaches 99.4% top-10 retrieval accuracy and outperforms GraphRAG for retrieval and answer quality.","lead":"This paper compares retrieval-augmented generation (RAG) and GraphRAG on a new benchmark of 477 math textbook questions, each tied to a page. It finds standard embedding-based RAG retrieves the correct page more often than GraphRAG, and that GraphRAG floods the model with too much context.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GraphRAG comparison is confounded by retrieval-set size: accuracy is computed over ~47K tokens of uncontrolled context, not a top-k page list, so the claimed RAG advantage may be an artifact.","rationale":"Reader's conditional verdict is reasonable. I diverge on the single most load-bearing weakness: the synthetic-data concern mainly limits external validity and is explicitly acknowledged in §5.7, whereas the GraphRAG comparison confound threatens the internal validity of the paper's headline comparison. A benchmark can use synthetic questions and still support a relative comparison of retrievers, but only if the comparison controls for the amount of context each method is allowed to return. Table 1's GraphRAG row is an inclusion test over ~47K tokens, while the embedding rows are top-k page ranks over hundreds-to-thousands of tokens. Consequently, the claimed performance ordering is not established by the reported numbers. The released code and data make a matched-granularity reanalysis feasible, so the appropriate verdict remains CONDITIONAL (here UNCHANGED from the reader's conditional verdict), with the added condition that the authors rerun the GraphRAG comparison under a token/page-matched protocol. I do not see fraud or sloppiness: the paper is transparent about token counts and limitations, and the open-source artifacts are a real contribution. The concern is specifically about comparability of the central comparison.","tokens_in":10682,"tokens_out":4885,"duration_ms":45498,"concrete_test":"Using the public GraphRAG code, for each of the 477 queries capture the retrieved entities and their document_ids; aggregate duplicate pages and rank unique pages by GraphRAG's relevance/community score. Recompute Table 1 accuracy and Table 2 F1 at top-1/3/5/10 pages, and record average context tokens at each cutoff. Then compare against voyage-3-large at the same cutoffs and also at a matched token budget (e.g., truncating/ranking GraphRAG context to ~3.7K tokens). If GraphRAG still trails voyage-3-large under matched page/token budgets, the current conclusion is supported; if it matches or exceeds, the reported RAG superiority is an artifact of retrieval granularity rather than retrieval quality.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that embedding-based RAG outperforms GraphRAG rests on Table 1's retrieval accuracy comparison, but the two columns are not measured in the same regime. Section 4.1 states that GraphRAG accuracy is evaluated by whether the target page appears anywhere in the entity-based retrieval output; there is no page-level ranking or top-k cutoff. Section 5.3 reports that GraphRAG returns ~46,949 tokens per query on average, while voyage-3-large top-5 returns ~3,743 and top-1 ~899 tokens. Thus GraphRAG's 0.914 accuracy in Table 1 is a high-recall, uncontrolled-context measurement, while the RAG numbers are precision-oriented top-k measurements. The comparison is therefore not apples-to-apples: a method given 10-50x more context can trivially achieve higher inclusion rates, and a method restricted to a few pages is disadvantaged. The F1 gap is also small (GraphRAG ~0.52 vs RAG ~0.54), so the qualitative conclusion could flip under a matched token or page budget. This concern is distinct from the synthetic-data issue: even if the 477 questions were perfect, the GraphRAG-vs-RAG comparison as reported does not establish which retriever is better at page-level retrieval; it establishes only that one configuration returns more context. The authors' Section 5.3 discussion of excessive context is a practical cost observation, but it is not a substitute for a controlled retrieval comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an empirical comparison of standard embedding-based RAG against GraphRAG for page-level retrieval and question answering on an undergraduate mathematics textbook. The authors build a dataset of 477 question–answer pairs by prompting gpt-4o-mini with each textbook page, then filtering manually. They evaluate five embedding models at top-1/3/5/10, a closed-book baseline, and GraphRAG, reporting retrieval accuracy (whether the target page appears in the retrieved set) and answer-generation F1. Their headline finding is that embedding-based RAG outperforms GraphRAG: voyage-3-large reaches 0.994 top-10 accuracy, while GraphRAG's accuracy is measured as 0.914 (with o3-mini) but returns far more context (~47K tokens per query). They also report that LLM-based re-ranking gives mixed results. The abstract additionally claims a BM25 baseline, a Qwen3.5-35B-A3B replication, a bootstrap confidence interval, and a same-chapter error analysis, none of which appear in the body.","tokens_in":11011,"tokens_out":5251,"duration_ms":44707,"significance":"If the comparison were methodologically sound, this would be a useful contribution to educational RAG: it addresses page-level grounding, a practically important requirement for AI tutoring systems, and the authors have open-sourced their data and code. The retrieval-accuracy finding for voyage-3-large, the token-cost observation for GraphRAG, and the re-ranking results are potentially informative for practitioners. However, the central RAG-versus-GraphRAG claim is currently undermined by an apples-to-oranges retrieval comparison, the benchmark's synthetic nature is acknowledged but not resolved, and several abstract-level claims are unsupported in the body. The paper's practical implications are real, but the headline comparison needs substantial reworking before the conclusions can be accepted as stated.","major_comments":[{"comment":"The central claim that embedding-based RAG outperforms GraphRAG at page-level retrieval is not established by the reported measurements. In §4.1, GraphRAG accuracy is computed by checking whether the target page appears anywhere in the entity-based retrieval output, with no top-k or page budget; §5.3 states that GraphRAG returns 46,949 tokens per question on average, versus 3,743 tokens for voyage-3-large top-5 and 899 for top-1. Table 1 therefore compares a high-recall inclusion metric on a ~47K-token context against precision-oriented top-k accuracy on a small page list. The 0.914 GraphRAG value shows only that GraphRAG returns more context, not that it is better or worse at page retrieval. The paper should report GraphRAG under matched page/token budgets, or provide precision/recall at a controlled context size, before drawing the headline conclusion.","section":"§4.1, Table 1, §5.3"},{"comment":"The benchmark's validity for the claimed educational use is not established. Each question–answer pair is generated by gpt-4o-mini from the page content that is also the retrieval target, and the same model family (gpt-4o-mini) is used as the generator in the F1 evaluation. This setup can artificially inflate both retrieval accuracy and F1, because the generated questions likely reuse the page's vocabulary and formulation. The authors acknowledge in §5.7 that the content was LLM-generated and call for human-authored future work, but the paper's conclusions depend on these 477 items being a reasonable proxy for genuine student questions. Please provide at least a small human-authored validation set, or explicitly restrict the conclusions to LLM-generated questions.","section":"§3.2, §5.7"},{"comment":"Several results stated in the abstract are not present in the body: (i) the BM25 baseline; (ii) the bootstrap 95% CI [0.644, 0.728] for voyage-3-large top-1; (iii) the Qwen3.5-35B-A3B replication and the +39% vs +16% relative F1 improvements; (iv) the error analysis that 63.3% of top-1 failures retrieve same-chapter content. None of these appear in Sections 3–5, yet they are used in the abstract to support the paper's claims. Either add the corresponding experiments/analyses or remove these statements from the abstract.","section":"Abstract vs. Sections 3–5"},{"comment":"The generative-quality conclusion rests on F1 differences that are small and not shown to be significant. GraphRAG with gpt-4o-mini scores 0.525, GraphRAG with o3-mini 0.524, while the best RAG row is 0.552 (OpenAI text-embedding-3-large top-3) but several RAG conditions are below GraphRAG (e.g., intfloat/multilingual-e5-large-instruct top-1 at 0.514, voyage-3-large top-1 at 0.523). No confidence intervals or paired tests are reported for Table 2. With 477 items a 0.02 difference may be real, but the paper does not demonstrate it. Minimum action: report bootstrap CIs or paired significance tests before interpreting the F1 gap as evidence that GraphRAG's context reduces generation quality.","section":"§4.2, Table 2, §5.3"}],"minor_comments":[{"comment":"The F1 example contains an arithmetic inconsistency: it writes |R∩G| = |{0, natural, number}| = 2, but the set has three elements, and the subsequent precision/recall calculations correctly use 3. Please fix the notation.","section":"§3.5"},{"comment":"Typo: 'mdoel' should be 'model'.","section":"§2.1"},{"comment":"'Base on the fact computed above' should be 'Based on'.","section":"§3.5"},{"comment":"The table formatting is confusing: the 'Retrieved Top 5' header and the arrow notation (e.g., 0.686→0.593) should be explained explicitly in the caption or text.","section":"Table 3"},{"comment":"The random baseline '0.16%' assumes 628 pages, but the final dataset has 477 pages; if retrieval is over the filtered corpus the random baseline is ~0.21%. Please clarify the corpus size used in retrieval.","section":"§5.2"},{"comment":"For GraphRAG rows, clarify which component uses gpt-4o-mini versus o3-mini. The text says generation is always done by gpt-4o-mini, but the table labels are ambiguous.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central RAG-versus-GraphRAG comparison is the paper's main selling point, and the confounding between retrieval-set size and the measured accuracy is serious. If the authors can re-run GraphRAG under a matched page/token budget, or reframe the claim as a practical cost/context observation rather than a retrieval-quality comparison, the paper may become acceptable. The missing abstract claims (BM25, Qwen, bootstrap CI, error analysis) also need to be either reported or removed; as written, the abstract overstates the body. The synthetic-data limitation is acknowledged but currently prevents strong claims about real student questions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the new resource: 477 page-linked QA pairs from a real math textbook, plus a fairly careful comparison of five embedding models on a page-level retrieval task. The re-ranking experiments are a nice addition, and the authors are honest about the synthetic-data limitation in Section 5.7. If you work on educational retrieval, this dataset and the embedding rankings are worth a look.\n\nThe headline claim—embedding RAG outperforms GraphRAG—does not hold up as stated. The stress-test note is right: Table 1 compares top-k page lists (a few pages, ranked) against GraphRAG accuracy measured as \"does the target page appear anywhere in the retrieved entity dump,\" which averages about 47K tokens per query. That is a high-recall, uncontrolled-context measurement. A method allowed to return ten to fifty times more context will trivially look better on inclusion. And the F1 gap separating them is tiny (0.52 vs. 0.54), so the qualitative conclusion could flip under a matched token or page budget. The authors do flag GraphRAG's excessive context as a practical cost, but that is not the same as a controlled retrieval comparison.\n\nThere are also several abstract claims that never show up in the body: the BM25 baseline, the Qwen3.5-35B-A3B replication, the bootstrap CI, and the 63.3% same-chapter error analysis are all absent. That is a real problem. The F1 calculation example also has a set-size error (the intersection set has three elements but is written as 2), which makes the equations internally inconsistent. No significance tests or error bars accompany the main tables, so the differences between embedding models may be noise.\n\nThe dataset generation process is another soft spot: gpt-4o-mini wrote each question from the target page itself, so retrieval may be artificially easy. The authors acknowledge this, but it still limits how far the absolute accuracies can be trusted.\n\nWho is this for? People building page-level AI tutors or evaluating embedding models on textbook-style content will get value from the dataset and the general pattern. But the paper needs a matched-budget GraphRAG comparison, removal or correction of the unsupported abstract claims, and error bars before the central comparison is established. As it stands, it's a legitimate empirical contribution with one load-bearing comparison that is not yet properly controlled. I'd send it to review, but I'd expect serious revision and would not cite the GraphRAG conclusion as is.","headline":"Useful new dataset and a systematic embedding comparison, but the RAG-beats-GraphRAG claim is not apples-to-apples and the abstract overstates what the body shows.","tokens_in":11518,"tokens_out":1755,"would_cite":false,"duration_ms":17132,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Page-level embedding RAG beats GraphRAG on math-textbook retrieval, with 99.4% top-10 accuracy.","keywords":["retrieval-augmented generation","GraphRAG","page-level retrieval","mathematical textbook QA","AI tutoring","knowledge graphs","question answering","retrieval accuracy"],"falsifier":"Ask a group of students to write questions about this textbook, run the same voyage-3-large page retrieval on those questions, and compare top-1 accuracy with the paper's 68.6%; a large drop, or a case where GraphRAG then beats page-level RAG on answer F1, would undercut the central conclusion.","tokens_in":10569,"feed_emoji":"📘","tokens_out":7255,"duration_ms":60458,"temperature":0.7,"pith_summary":"This paper asks whether an AI retrieval system can point a student to the exact page of a math textbook where a question is answered. On 477 page-tied questions, standard embedding-based RAG beats GraphRAG for page-level retrieval: the best embedding model finds the correct page 68.6% of the time at top-1 and 99.4% at top-10, while GraphRAG's accuracy reaches 91.4% only with far more context. The paper's explanation is structural: GraphRAG expands entity neighborhoods and returns roughly 47,000 tokens per question, whereas page-chunked RAG returns about 3,700 tokens, and that excess context drags down answer-generation F1. The result matters for AI tutors because a system that cites exact pages is easier for students to verify, and the paper shows that simpler page-based indexing beats graph-based complexity for this goal.","feed_headline":"Page-level RAG finds the right textbook page 99.4% of the time","feed_subtitle":"For AI tutors that cite exact pages, simpler page-level embedding beats graph-based retrieval on 477 math questions.","key_machinery":"The load-bearing mechanism is page-level chunking for retrieval: the textbook is split page by page, each page is embedded as a vector, and a question retrieves the top-k pages by cosine similarity. GraphRAG is the contrast: it builds an entity-relation graph and retrieves related entities with document IDs attached, which breaks the book's linear page structure. The mismatch between graph fragmentation and page-level organization is the mechanism the paper blames for excessive context length and lower generation quality. Evaluation uses two tools: retrieval accuracy (is the ground-truth page among the retrieved pages) and token-overlap F1 between the generated answer and the reference answe","core_discovery":"On the paper's own terms, the central discovery is that the natural unit for a textbook-answering task is the page, not the knowledge graph. Dense embeddings chunked by page retrieve the correct page for 99.4% of questions when ten candidates are allowed; GraphRAG, which links entities across pages, retrieves broader neighborhoods that include the right content but also drown the generator in roughly 47,000 tokens per query. The authors adapt GraphRAG to attach document IDs to entities so pages can be referenced, yet it still produces lower F1 answers (around 0.52) than page-based RAG (around 0.53–0.55). They also find that re-ranking retrieved pages with an LLM does not reliably improve acc","pith_inferences":["Because the 477 questions were generated from page text by an LLM and then filtered by the authors, a natural next test is to run the same retrieval stack on questions written by students; if top-1 accuracy drops, the reported advantage may not transfer to classrooms.","Token-overlap F1 is a weak proxy for mathematical correctness; a human panel comparing GraphRAG and RAG answers could reveal whether the extra graph context adds pedagogical value that F1 misses, or whether it actively confuses.","A direct test of the paper's diagnosis would be to impose page-level constraints on GraphRAG's graph expansion (for example, stop entity traversal at page boundaries) and see whether its F1 rises to meet embedding RAG.","The token-count gap implies a cost consequence: GraphRAG consumes over 12 times more input tokens per question, so page-level RAG is likely cheaper at scale as well as more accurate."],"forward_implications":["System builders should index textbook pages as individual retrieval units; page-level RAG gives near-perfect coverage at top-10, so AI tutors can confidently offer a short list of candidate pages.","Retrieval accuracy is not the same as answer quality: GraphRAG can locate relevant content but its oversized context lowers generated-answer F1, so context length must be controlled separately.","LLM-based page re-ranking is not a reliable enhancement; it often hurts top-1 accuracy and can invent nonexistent page references, so it should be deployed with caution.","Weaker open-source models benefit proportionally more from RAG than stronger models do, which makes retrieval augmentation especially valuable for cost-sensitive deployments."],"fun_headline_variants":["GraphRAG floods the model; page RAG wins on math textbooks","Simple page retrieval beats knowledge graphs for math Q&A","GraphRAG retrieves 47K tokens per query; page RAG only 3.7K","For math textbooks, page RAG outscores graph-based retrieval","Page-level RAG finds the right page 99.4% of the time"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The 477 test questions were written by an LLM from the very page they are supposed to retrieve, so the benchmark may be easier than real student questions; if students ask differently, the retrieval advantage could shrink.","fun_headline_variants_meta":{"raw":{"variants":["GraphRAG floods the model; page RAG wins on math textbooks","Simple page retrieval beats knowledge graphs for math Q&A","GraphRAG retrieves 47K tokens per query; page RAG only 3.7K","For math textbooks, page RAG outscores graph-based retrieval","Page-level RAG finds the right page 99.4% of the time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000474,"raw_usage":{"total_tokens":2232,"prompt_tokens":826,"completion_tokens":1406,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":1317}},"tokens_in":570,"tokens_out":1406,"duration_ms":9898,"temperature":1.0,"reasoning_tokens":1317,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T16:03:35.650127+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a group of students to write questions about this textbook, run the same voyage-3-large page retrieval on those questions, and compare top-1 accuracy with the paper's 68.6%; a large drop, or a case where GraphRAG then beats page-level RAG on answer F1, would undercut the central conclusion.","supporting_citations":[],"review_version":1}