{"id":"f4171056-8dc0-4a13-858a-9d395d59ee82","arxiv_id":"2508.12682","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A RAG framework combining multi-stage query refinement and RAPTOR retrieval improves grid code question answering quality by about 26% and retrieval recall by roughly 10x over baselines.","lead":"GridCodex is a retrieval-augmented AI system that reads power grid regulations and answers compliance questions, using extra retrieval steps and hierarchical indexing. It reports large gains over plain chatbots and standard RAG, but the evidence is built on a small private dataset and an AI judge, not yet independently reproducible.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation validity is the load-bearing issue: unreleased benchmark and a self-family LLM judge may inflate the reported 27.5% quality gain and 9.8x recall improvement; no human evaluation is reported.","rationale":"The paper's contribution is an engineering system, and the qualitative claim that query refinement and RAPTOR improve retrieval and answer quality is plausible. My review focuses on the central quantitative claim because it is the only evidence for the contribution. That claim is load-bearing: if the evaluation is biased, the paper's headline numbers have no external support. The dataset is proprietary, the gold chunks are not described, and the judge is the same Qwen3-235B model used to generate answers. This is a concrete self-evaluation risk, not a disagreement with consensus. The reader's weakest_assumption points to the same issue; I agree. I additionally note the internal 10-fold/9.8x discrepancy, which is minor but reinforces that the numbers are not carefully checked. I would not reject the paper: the architecture is coherent, the baselines are reasonable, and the qualitative direction is credible. But the specific percentages should be treated as conditional until an independent human evaluation confirms them. Hence the reader's CONDITIONAL verdict remains appropriate; no change.","tokens_in":8747,"tokens_out":5981,"duration_ms":59419,"concrete_test":"Release the 148 QA pairs, gold relevant chunks, and scoring prompt, and run a blinded human evaluation on a stratified random sample of at least 50 questions (about 12 per region): two independent grid-code experts score GridCodex and vanilla RAG outputs using the same accuracy/completeness/usefulness rubric, without knowing which system produced each answer. Compare the human mean score difference with the LLM-judge difference; if the human-verified improvement is not statistically significant or is below, say, half the claimed 27.5%, the headline numbers are not reproducible. Also report inter-expert agreement and agreement between each expert and the Qwen3 judge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—27.5% answer-quality improvement and Recall@30 rising from 9.5% to 93.2%—rests on two evaluation choices that are not independently supported. First, the automated judge is Qwen3-235B-A22B (Models), the same model used for answer synthesis; LLM-as-a-judge self-preference is a known bias that can inflate scores for the system's own outputs. Second, Recall@30 depends on whether the 'information necessary to answer a query' is in the top-30 retrieved chunks, but the paper never describes how the gold relevant chunks were constructed for the proprietary 148-QA dataset (Datasets). If gold chunks were created with the refined query or the system's terminology injection in mind, the recall gain is partially circular. The paper also reports no human evaluation despite the Introduction claiming expert validation, no inter-annotator agreement, and no variance/error bars. The abstract's 'more than 10-fold increase in recall' is contradicted by the reported 93.2/9.5 = 9.8x. These issues do not disprove the qualitative conclusion, but they make the specific 27.5% and 9.8x numbers unverifiable as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GridCodex, a retrieval-augmented generation framework for power grid code reasoning and compliance. It combines separate terminology and factual knowledge bases, a multi-stage query refinement pipeline (terminology expansion, English translation), and RAPTOR-based hierarchical retrieval, with answer generation by open-source LLMs such as Qwen3 and DeepSeek. The authors evaluate on a proprietary 148-question dataset spanning four regulatory regions, comparing against general LLMs and vanilla RAG on answer quality, faithfulness, and Recall@30. They report a 26.4% (or 27.5%) improvement in answer quality and a roughly 9- to 10-fold increase in Recall@30, plus ablations on model size and reasoning capability. The paper also discusses practical lessons for regulatory QA in the energy sector.","tokens_in":8942,"tokens_out":4626,"duration_ms":48152,"significance":"If the reported results hold, GridCodex would be a practically useful contribution: it addresses an underserved problem (automated grid-code interpretation), the system design is clearly described and model-agnostic, and the ablations on model scale and reasoning behavior are informative. The multi-stage query refinement combined with RAPTOR retrieval is a sensible approach for dense, cross-referenced regulatory documents. However, the quantitative claims currently rest on an evaluation that is not independently verifiable: a proprietary dataset with no construction details, an LLM judge from the same model family as the answer generator, no error bars or statistical tests, and no actual human evaluation despite the introduction claiming expert validation. These limitations do not disprove the qualitative conclusion, but they make the specific numeric gains (27.5% and 9.8x) unsubstantiated as reported.","major_comments":[{"comment":"The paper states that Qwen3-235B-A22B is used for both answer synthesis and automated scoring. Because the judge belongs to the same model family as the generator, the reported answer-quality gains may partly reflect self-preference rather than true quality. Please use an independently developed judge (a different model family or human experts) and report agreement metrics such as Cohen's kappa, or otherwise provide evidence that the judge does not systematically favor GridCodex outputs.","section":"Models"},{"comment":"The Recall@30 metric is defined only as whether the information necessary to answer a query is contained within the top 30 retrieved chunks, but the paper never states how the gold relevant chunks were constructed for the proprietary 148-QA dataset. If those gold chunks were produced using the refined queries or the system's terminology injection, the measured recall gain would be partially circular. Specify the annotation procedure, including whether annotators worked only from raw grid-code documents and whether query-refinement outputs were hidden from them.","section":"Datasets"},{"comment":"Table 2 and Figure 4 report single-point estimates for a 148-pair dataset across four regions, with no confidence intervals, no standard deviations, and no significance tests. Given the small per-region sample sizes, the claimed 27.5% improvement in answer quality and the 9x Recall@30 gain need at least a per-question variance analysis, bootstrap confidence intervals, or a paired significance test to be convincing.","section":"Experiments / Results"},{"comment":"The abstract's '26.4% improvement' and 'more than 10 fold increase' are not consistent with Figure 4, which reports 27.5% and 'nearly 9x', and with Table 2, where 93.2/9.5 = 9.8x. Please reconcile these numbers and explicitly state which baseline the percentages refer to (general LLM, vanilla RAG, or the average of both).","section":"Abstract / Results"},{"comment":"The Introduction asserts that the system is 'validated by experts from Southern Grid and other external authorities' and achieves 'answer quality up to 88%', but the Experiments section contains no human evaluation protocol, no number of experts, and no results from such a study. This claim must either be substantiated with a described human evaluation or removed from the paper.","section":"Introduction"}],"minor_comments":[{"comment":"The phrase 'more than 10 fold increase in recall rate' is inconsistent with the reported 9.8x (93.2/9.5); please correct the wording to 'nearly 10-fold' or adjust the numbers.","section":"Abstract"},{"comment":"Table 1's surrounding sentence 'Table 1 have summarized' should be 'Table 1 summarizes'.","section":"Related Work"},{"comment":"The caption includes '(Subject to aesthetic changes.)', which is an editorial artifact and should be removed before publication.","section":"Figure 4"},{"comment":"Several references are incompletely formatted: the 2016 and 2017 entries lack authors, the RAPTOR entry is truncated, and some entries use only a year as the author field. These need to be normalized.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core system design is plausible and the problem is well motivated, but the evaluation is currently not reproducible: the dataset is proprietary, no code is released, and the judge model is from the same family as the generator. Given the emphasis in this venue on trustworthy empirical claims, the editor may want to require a detailed evaluation appendix, an independent judge or human study, and per-question uncertainty estimates before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom,\n\nQuick read of GridCodex. The genuinely new part is the application: I don't know of another published RAG system aimed specifically at grid code compliance across multiple jurisdictions. The engineering is sensible—terminology expansion, query rewrite, translation, then RAPTOR retrieval—and the ablation on model size and reasoning (Tables 3 and 4) is a nice addition. The paper is clearly written and the system architecture is easy to follow.\n\nThe soft spot is the evaluation, and it's the load-bearing one. The 148 QA pairs are proprietary, so we can't check how the gold chunks for Recall@30 were built. The judge is Qwen3-235B, same model family that generates the answers, which invites self-preference. There are no error bars or statistical tests. The intro claims expert validation, but no human evaluation is reported. Faithfulness is nearly identical for vanilla RAG and GridCodex, so it doesn't discriminate. And the abstract says 'more than 10-fold increase' while the actual numbers give 93.2/9.5 = 9.8x—minor, but sloppy.\n\nNone of this disproves the qualitative conclusion. It's entirely plausible that multi-stage refinement plus RAPTOR improves retrieval on this kind of document. But the specific 27.5% and 9.8x numbers are not something I'd quote, and they shouldn't be taken as transferable.\n\nI'd send it to peer review with the expectation of major revisions: release the dataset or a substantial sample, describe gold chunk construction, use a different judge (or include human evaluation), and report variance. The direction is worth publishing; the current evidence is not.\n\nBest.","headline":"A sensible RAG system for grid code compliance whose headline numbers need better evidence before they're quoted.","tokens_in":9507,"tokens_out":2434,"would_cite":false,"duration_ms":23564,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A retrieval-augmented framework for grid-code reasoning rewrites queries with domain terminology and retrieves from a recursive knowledge tree, reporting gains of roughly 27% in answer quality and 9x in recall over baselines.","keywords":["retrieval-augmented generation","grid code compliance","query refinement","RAPTOR","regulatory question answering","power systems regulation","LLM reasoning","domain knowledge bases"],"falsifier":"Have grid-code experts score a held-out set of questions from a jurisdiction not used in the paper, with the system output, vanilla RAG output, and plain LLM output presented blindly, and compare the human scores to the paper's reported answer-quality and Recall@30 numbers. If the expert scores show a much smaller spread than the reported 27.5% quality gain, or if the recall advantage disappears when a human rewrites the queries with the same terminology, the central claim would be refuted.","tokens_in":8523,"feed_emoji":"⚡","tokens_out":8676,"duration_ms":78551,"temperature":0.7,"pith_summary":"Grid code regulations are dense, jargon-heavy, and inconsistent across regions, and interpreting them today depends on human specialists. The paper claims that a retrieval-augmented generation system—one that feeds relevant document passages to a language model at answer time—can automate much of this reasoning by combining two ideas: a query-refinement stage that expands technical terms and translates them into the language of the documents, and tree-structured retrieval that reaches across nested, cross-referenced clauses. On a private 148-question benchmark spanning four regulatory bodies, the system reports answer-quality scores of 0.843–0.946, roughly a 26–27% gain over plain LLM and vanilla RAG baselines, and Recall@30 rising from 9.5% to 93.2%. If this holds, utilities and equipment vendors could check compliance against multiple jurisdictions far faster, with less reliance on scarce local experts.","feed_headline":"27.5% better answers, 9x recall on grid-code rules","feed_subtitle":"Terminology-aware query rewriting and tree-structured retrieval beat plain LLMs and vanilla RAG on compliance questions.","key_machinery":"The load-bearing mechanism is multi-stage query refinement combined with RAPTOR (recursive abstractive processing for tree-organized retrieval). RAPTOR clusters semantically related chunks, summarizes each cluster with an LLM, and re-embeds the summaries recursively so that the resulting tree preserves both local clauses and global context; retrieval can then touch content a flat chunk split would miss. Query refinement supplies the domain lens the retriever needs: a terminology lookup turns a vague question like overvoltage protection into specific keywords such as protection point and protection time, and translation aligns the query with the documents' language. The framework keeps terminology knowledge and factual clause knowledge in separate vector indexes, which lets each stage retrieve at the right granularity.","core_discovery":"The paper's central claim is that an off-the-shelf LLM becomes a reliable grid-code compliance interpreter when it is given a knowledge base built from two separate sources—terminology definitions and translations, plus factual regulatory clauses—and when the user query is refined in stages before retrieval. That refinement first enriches the query with domain-specific term explanations, then translates it into English, then retrieves from a RAPTOR-organized vector index. The reported result is that this configuration consistently outperforms both direct LLM prompting and vanilla RAG across all test regions: answer quality reaches 0.946 for Hong Kong (China), 0.852 for the Netherlands, 0.843 for the EU, and 0.877 for Bangladesh, with Recall@30 above 0.90 in every region.","pith_inferences":["The same query-refinement-plus-tree-retrieval recipe could transfer to other safety-critical regulation domains, such as aviation, financial compliance, or medical device approval, whenever the authoritative text is long, cross-referenced, and multilingual.","The paper does not isolate how much of the gain comes from query rewriting versus RAPTOR's tree; an ablation that turns off each stage separately would make the contribution of each mechanism testable.","Because the judge LLM comes from the same model family as the generator, the reported quality scores may partly reflect stylistic agreement rather than regulatory correctness; an independent human-scored sample would settle this.","If the terminology knowledge base were maintained as a living asset, the framework could shift from one-off question answering toward proactive compliance monitoring, flagging violations as new grid-code versions are published."],"forward_implications":["If the reported numbers hold, an automated pipeline can answer grid-code compliance questions that currently require specialists, with near-complete retrieval coverage of the governing documents.","Multilingual and multi-jurisdiction compliance becomes practical: the same pipeline worked on documents from Hong Kong (China), the Netherlands, the EU, and Bangladesh after query translation.","Retrieval coverage, not the generator, appears to be the main bottleneck: once Recall@30 exceeds 0.90, answer quality rises into the 0.84–0.95 range across all four regions.","The framework's model-agnostic design means deployment can use smaller local models, with the trade-off quantified in the ablation: larger reasoning-capable models improve answer quality but are slightly less strictly faithful to the retrieved text."],"supporting_citations":[{"why":"Supplies RAPTOR, the recursive clustering-and-summarization retrieval method at the core of the framework's document coverage.","marker":"Sarthi et al. 2024"},{"why":"Establishes retrieval-augmented generation as the base paradigm the framework extends with query refinement.","marker":"Lewis et al. 2021"},{"why":"Describes the Qwen3 family used for query refinement, answer generation, and automated scoring.","marker":"Yang et al. 2025"},{"why":"Provides the DeepSeek-R1 reasoning model used for RAPTOR summarization and for the reasoning-capability ablation.","marker":"Guo et al. 2025"},{"why":"Supplies Linq-Embed-Mistral, the embedding model that builds the vector indexes for both terminology and factual knowledge.","marker":"Choi et al. 2024"}],"fun_headline_variants":["GridCodex: 26% better answers, 10x recall on grid rules","RAG framework boosts grid-code answers 26%, recall 10x","Tree-structured RAG lifts grid reasoning: 26% gain, 10x recall","GridCodex beats vanilla RAG: 26% answer boost, 10x retrieval","AI that interprets grid codes: 26% quality gain, 10x recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation leans on a private 148-question dataset scored by an LLM that belongs to the same model family as the answer generator, with no reported human rating and no variance figures; if that judge is biased toward the system's own writing style, the measured gains could shrink substantially in real compliance workflows.","fun_headline_variants_meta":{"raw":{"variants":["GridCodex: 26% better answers, 10x recall on grid rules","RAG framework boosts grid-code answers 26%, recall 10x","Tree-structured RAG lifts grid reasoning: 26% gain, 10x recall","GridCodex beats vanilla RAG: 26% answer boost, 10x retrieval","AI that interprets grid codes: 26% quality gain, 10x recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000198,"raw_usage":{"total_tokens":1318,"prompt_tokens":844,"completion_tokens":474,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":460,"tokens_out":474,"duration_ms":4691,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:18:32.430805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have grid-code experts score a held-out set of questions from a jurisdiction not used in the paper, with the system output, vanilla RAG output, and plain LLM output presented blindly, and compare the human scores to the paper's reported answer-quality and Recall@30 numbers. If the expert scores show a much smaller spread than the reported 27.5% quality gain, or if the recall advantage disappears when a human rewrites the queries with the same terminology, the central claim would be refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies RAPTOR, the recursive clustering-and-summarization retrieval method at the core of the framework's document coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Qwen3 family used for query refinement, answer generation, and automated scoring."}],"review_version":2}