{"id":"ec368792-da53-47a7-ac14-e3e18a790f0c","arxiv_id":"2608.06828","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A rhetorical-role-aware RAG system for Indian Supreme Court judgments is described, but its effectiveness claim is unsupported by baselines, ablations, or human evaluation.","lead":"A team proposes a question-answering system for Indian Supreme Court judgments that splits documents by rhetorical role, such as facts or ratio of the decision, and retrieves passages before generating answers with a large language model. They report high faithfulness and answer relevance scores, but provide no comparison against standard retrieval systems, so the claimed benefit of their design is not yet demonstrated.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Role filtering can exclude the answer before reranking, and the paper never validates the labeler or the intent-to-role mapping; without a no-filter baseline the effectiveness claim is unsupported.","rationale":"The reader's REJECT is well-founded. My stress-test centers on the most load-bearing condition for the abstract's claim: the role filter must not discard the evidence. The paper gives no evidence for this condition. A standard RAG baseline and ablations would help interpret the metrics, but even with baselines, the lossy filter concern is primary because it can invalidate retrieval in principle. The low Contextual Relevancy scores and Section 6's caveat about automated metrics reinforce rather than resolve this. I agree with the reader's weakest_assumption; I would keep the REJECT verdict. The concrete check above would settle whether the concern actually lands: if all gold sentences are covered by Table 3 mappings and labeler agreement is high, the filter may be acceptable; if not, the framework's central claim fails.","tokens_in":7140,"tokens_out":3501,"duration_ms":35230,"concrete_test":"Have two legal annotators mark, for each of the 750 query-judgment pairs, the minimal sentences needed to answer the query. Then (1) compare the Malik et al. labeler against manual role labels on a stratified sample and report per-role agreement; (2) compute the fraction of query-judgment pairs whose gold answer sentences all lie outside the Table 3 target roles for the query's intent—if this fraction is nonzero, role filtering provably excludes the answer for those pairs; (3) rerun the retrieval pipeline with the filter disabled and compare Contextual Relevancy and Answer Relevancy on the same 25 queries. If unfiltered retrieval matches or exceeds filtered scores, the role filter is not providing the claimed benefit.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To justify 'very effective,' the pipeline must retrieve the passage containing the answer. Section 3.3.2 filters candidates to the roles in Table 3 before cross-encoder reranking, so any error in the Malik et al. labeler (Section 3.1) or any omission in the hand-defined intent-to-role mapping is unrecoverable: the correct passage is never scored. The paper provides no labeler accuracy on these 30 judgments, no human validation of role assignments, and no ablation with filtering disabled. The reported Contextual Relevancy scores (0.376–0.443 in Tables 5–7) show that even among passages that survive filtering, most are judged irrelevant; combined with Section 6's admission that all scores come from automated LLM judges, the numbers do not establish that the framework is effective, and they are consistent with a system that works only when the answer happens to fall into the mapped roles. Section 3.1 also oscillates between 'six rhetorical roles' and 'seven rhetorical categories' while listing six, and Table 2 adds RulingByPresentCourt; the role taxonomy itself is not crisply defined. This is a correctness risk, not a style issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a retrieval-augmented generation system for legal question answering over Indian Supreme Court judgments. The pipeline performs sentence-level rhetorical role labeling, role-aware chunking, hybrid BM25/dense retrieval, intent-based role filtering, cross-encoder reranking, and LLM answer generation. The evaluation uses 30 judgments and 25 queries across civil, corporate, and criminal domains, with automated LLM-based metrics (Faithfulness, Answer Relevancy, Contextual Relevancy) reported in Tables 5–7. The authors conclude that the framework is “very effective” for context-heavy legal QA.","tokens_in":7351,"tokens_out":3927,"duration_ms":37705,"significance":"If validated, the idea of exploiting rhetorical roles in legal judgments for chunking and filtering is potentially useful for domain-specific RAG. The paper makes a concrete pipeline choice and compiles a small evaluation corpus, which is a useful starting point. However, the current evidence does not establish the central effectiveness claim: there is no comparison against a standard RAG baseline, no ablation of the role-aware components, no validation of the external role labeler, and the reported Contextual Relevancy scores are low. The strengths are the clearly described architecture and the acknowledgment of limitations in Section 6, but these do not compensate for the missing experimental support.","major_comments":[{"comment":"The central effectiveness claim is not supported because no baseline or ablation isolates the contribution of the proposed components. The reported Contextual Relevancy scores of 0.376–0.443 indicate that most retrieved nodes are judged irrelevant, and without a comparison to a vanilla RAG pipeline, to retrieval without role filtering, or to chunking without rhetorical roles, the high Faithfulness and Answer Relevancy values cannot be attributed to the proposed architecture. A no-filter baseline is especially important because Section 3.3.2 removes passages before reranking.","section":"Section 4 (Tables 5–7); Section 3.3.2"},{"comment":"The intent-to-role mapping is hand-defined and filters candidates before reranking, so an omission or error is unrecoverable. The paper never validates the Malik et al. role labeler on these 30 judgments, reports no labeler accuracy or human agreement, and does not test whether the mapping excludes passages that contain the answer. Without such validation or a filtering-disabled ablation, the retrieval results are consistent with a system that works only when the answer happens to fall in the mapped roles.","section":"Section 3.3.2 and Table 3"},{"comment":"All three metrics are computed by an LLM acting as an evaluation judge, and Section 6 admits there is no human evaluation. Because the same class of LLM both generates answers and judges relevance, the scores are self-referential; no evidence is given that the LLM judge agrees with human judgments or with gold labels. The low Contextual Relevancy values in Tables 5–7 further suggest that the judge is not simply lenient, and the lack of a human-validated subset makes it impossible to interpret the metric values as evidence of effectiveness.","section":"Section 4.1 and Section 6"},{"comment":"The rhetorical-role taxonomy is inconsistently defined. Section 3.1 says “one of the six rhetorical roles” and then lists “seven rhetorical categories” with only six items (Argument, Statute, Fact, Ratio of the Decision, Precedent, Ruling by Lower Court), while Table 2 reports a seventh role, RulingByPresentCourt. This inconsistency matters because the role labels are the input to chunking and filtering, and the paper does not clarify which roles the labeler actually produces or how the intent-to-role mapping was derived.","section":"Section 3.1 and Table 2"}],"minor_comments":[{"comment":"Typos: “labled” should be “labeled”; “correspondoing” should be “corresponding”; “alloted” should be “allotted”; Table 6: “Qwen3-32B A” appears to have a stray “A”.","section":"Section 3.1 and Section 3.3.2"},{"comment":"The reference list contains the same Malik et al. work as [18] and [21]; one entry should be removed or cross-referenced.","section":"References"},{"comment":"The prompt design and retrieval hyperparameters (top-k, rerank size, small-group merging threshold, sub-splitting window and overlap) are not specified; they should be reported for reproducibility.","section":"Section 3.2 and Section 3.3.3"},{"comment":"The abstract claims the framework “demonstrated strong performance” and “proves” effectiveness, but Section 6 acknowledges limitations; the wording should be modest and match the evidence.","section":"Abstract and Conclusion"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication in a serious journal. The evaluation insufficiency is fundamental: the central effectiveness claim is contradicted by the low Contextual Relevancy scores, and the missing baselines, ablations, labeler validation, and human evaluation would require substantial new experiments rather than a light revision. I would not recommend inviting a major revision unless the authors are willing to rebuild the evaluation around controlled comparisons and human-validated judgments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a reasonable idea, but the evaluation doesn't support the headline claim, and one design choice (role filtering) could systematically remove the correct answer before reranking. I'd desk reject, though the idea is worth a resubmission.\n\nWhat's actually new: combining rhetorical-role chunking with query-intent-to-role filtering for QA over Indian Supreme Court judgments is a real, modest integration. The paper correctly identifies that legal judgments have internal structure and that treating them as homogeneous text is suboptimal. It cites closely related systems (TaxFlow, CEFS) and inherits a role labeler from Malik et al. That's honest positioning.\n\nThe soft spots are serious. There is no baseline, no ablation, no error bars, no human evaluation. The reported Contextual Relevancy numbers (0.376–0.443) mean that most retrieved passages are judged irrelevant even after filtering and reranking. That directly contradicts the abstract's claim of 'proven very effective.' The Section 6 limitation note admits all scores come from automated LLM judges, which is honest, but it doesn't rescue the effectiveness claim.\n\nThe stress-test concern lands: Section 3.3.2 filters candidates by the intent-to-role mapping before reranking. If the labeler makes a mistake or the mapping omits the role containing the answer, the correct passage is gone and no reranker can recover it. The paper gives no labeler accuracy on these judgments and no ablation with filtering disabled. That is load-bearing for the claimed benefit of the approach.\n\nMinor but genuine: the role taxonomy is sloppy—Section 3.1 says 'seven rhetorical categories' but lists six, and Table 2 uses a different set including RulingByPresentCourt. This matters because the intended role filtering depends on the taxonomy. Also, the same paper is cited twice as [18] and [21], which is sloppy but easy to fix.\n\nWho this is for: people working on Indian legal RAG might find the pipeline description useful as a starting point, but the evidence doesn't justify deploying it. I'd recommend desk rejection, and I'd tell the authors to resubmit only after adding a no-filter baseline, ablations of each component, human evaluation on a small sample, and validation of the role labels.","headline":"Sensible pipeline, but no baseline, no ablation, and low contextual relevancy scores mean the effectiveness claim is unsupported; desk reject, though the idea is worth a resubmission.","tokens_in":7916,"tokens_out":2893,"would_cite":false,"duration_ms":29663,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a legal question-answering system over Indian Supreme Court judgments is more faithful when retrieval follows the rhetorical structure of judgments—fact, precedent, ratio, ruling—than when it treats the text as a…","keywords":["retrieval-augmented generation","legal question answering","rhetorical roles","Indian Supreme Court judgments","hybrid retrieval","cross-encoder reranking","query intent classification"],"falsifier":"Run the 25 queries on the same 30 judgments twice: once with the full role-filtering pipeline and once with filtering turned off, retrieving from all roles before reranking and generation. If unfiltered retrieval finds the answer-bearing passage at least as often, or if the generator's faithfulness does not drop when filtering is removed, then the intent-to-role mapping is not the cause of the reported performance; the same test could use an oracle role labeler to separate labeling error from mapping error.","tokens_in":6917,"feed_emoji":"⚖️","tokens_out":8656,"duration_ms":78255,"temperature":0.7,"pith_summary":"The paper claims that a retrieval-augmented question-answering system for Indian Supreme Court judgments works better when it respects the rhetorical structure of a judgment rather than treating the text as a flat collection. Each sentence is labelled with a role such as Fact, Precedent, Ratio of the Decision, or Ruling by Present Court, and chunks are built from consecutive sentences sharing a role. A query is assigned to one of five intents, and that intent decides which roles are allowed into the candidate set; dense and BM25 retrieval run in parallel, and a cross-encoder reranks the survivors before an LLM writes the answer. On 30 judgments across civil, corporate, and criminal law, the authors report faithfulness above 0.95 in every domain and answer relevancy near 0.9, which they read as evidence that role-aware retrieval supplies the context needed for grounded legal answers.","feed_headline":"Legal Q&A hits 99% faithfulness via rhetorical-role retrieval","feed_subtitle":"Chunking judgments by fact, precedent, and ratio keeps answers anchored to the right sections.","key_machinery":"The load-bearing mechanism is the rhetorical-role-aware chunking and filtering cycle. A sentence-level labeling model assigns one of seven roles—Argument, Statute, Fact, Ratio of the Decision, Precedent, Ruling by Lower Court, Ruling by Present Court—to every sentence; consecutive same-role sentences are merged into chunks, micro-chunks are folded into neighbors, and long groups are split into overlapping fifteen-sentence windows. A query is classified into one of five intents, and a hand-defined table maps each intent to the roles most likely to hold the answer (for example, inferential queries target Ratio of the Decision and Argument; precedent queries target Precedent and Statute). Retrieval then runs over a dense passage index and a BM25 sparse index in parallel, results are restricted to the allowed roles, and a cross-encoder scores each remaining query-passage pair before the top passages are handed to the generator.","core_discovery":"On the paper's own terms, the central discovery is that the functional role a sentence plays in a judgment—fact, statute, precedent, ratio, ruling—is a useful retrieval signal for legal question answering. By grouping sentences by role into chunks and then filtering candidate passages by the roles mapped to the predicted query intent, the pipeline removes content that may be lexically or semantically similar but legally peripheral. The authors report that this role-aware pipeline, combined with hybrid dense/sparse retrieval and cross-encoder reranking, produces consistently high faithfulness (0.951–0.991 across the three domains) and answer relevancy (0.860–0.923), while contextual relevancy sits much lower, around 0.38–0.44. They interpret these numbers as showing that the retrieved context, even when it carries irrelevant material, still contains enough legally appropriate passages for the generator to stay close to the source.","pith_inferences":["A direct ablation—running the same queries with role filtering disabled—would show what share of the faithfulness gain comes from the intent-to-role maps; the paper does not report such a comparison.","The contextual relevancy scores are markedly lower than faithfulness, suggesting the generator tolerates noisy context; tightening the retriever or the role mapping might raise both without hurting groundedness.","The hand-defined intent-to-role mapping is a natural place to automate, and learned mappings per domain or per court could transfer the approach to lower courts, other jurisdictions, or non-English judgments.","Because the paper's own limitations section notes that scores are automated and not reviewed by legal experts, a human annotation study of the generated answers would be the clearest next validation."],"forward_implications":["Role-aware chunking produces semantically coherent retrieval units, so questions about reasoning are answered from the ratio and argument sections rather than from peripheral text.","Intent classification followed by role filtering narrows the candidate set, letting the cross-encoder rerank within the parts of a judgment that match the query's information need.","Hybrid dense and BM25 retrieval covers both paraphrase-level matches and exact legal identifiers such as section numbers and case names.","The reported faithfulness scores imply that answers generated over role-filtered context are largely supported by the retrieved passages across all three legal domains tested."],"supporting_citations":[{"why":"Supplies the BM25Okapi sparse lexical index used alongside dense retrieval for exact-match legal terms.","marker":"[1]"},{"why":"Establishes the retrieval-augmented generation paradigm that the framework extends with domain-specific structure.","marker":"[4]"},{"why":"Documents a prior hybrid RAG architecture combining dense and sparse retrieval with cross-encoder reranking that the pipeline resembles.","marker":"[11]"},{"why":"Provides the cross-encoder reranking approach for scoring query-passage pairs in legal answer retrieval.","marker":"[14]"},{"why":"Provides the sentence-level rhetorical-role labeling model that produces the role annotations driving chunking and filtering.","marker":"[21]"}],"fun_headline_variants":["Rhetorical roles boost legal QA faithfulness to 99%","Role-based chunking keeps legal answers on point","Legal RAG: chunk by role, answer with confidence","Fact, precedent, ratio: role-aware retrieval for legal QA","Role-aware retrieval sharpens legal QA answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the model labelling each sentence with its role is accurate on these judgments and that the hand-built mapping from query intent to allowed roles never excludes the role that actually holds the answer; if either fails, the correct passage is removed before reranking no matter how well the retrievers perform.","fun_headline_variants_meta":{"raw":{"variants":["Rhetorical roles boost legal QA faithfulness to 99%","Role-based chunking keeps legal answers on point","Legal RAG: chunk by role, answer with confidence","Fact, precedent, ratio: role-aware retrieval for legal QA","Role-aware retrieval sharpens legal QA answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3541,"prompt_tokens":911,"completion_tokens":2630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":2566}},"tokens_in":527,"tokens_out":2630,"duration_ms":17417,"temperature":1.0,"reasoning_tokens":2566,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:02:09.591374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the 25 queries on the same 30 judgments twice: once with the full role-filtering pipeline and once with filtering turned off, retrieving from all roles before reranking and generation. If unfiltered retrieval finds the answer-bearing passage at least as often, or if the generator's faithfulness does not drop when filtering is removed, then the intent-to-role mapping is not the cause of the reported performance; the same test could use an oracle role labeler to separate labeling error from mapping error.","supporting_citations":[{"cited_title":"Foundations and trends®in information retrieval4(1-2), 1–174 (2009)","cited_arxiv_id":null,"evidence_quote":"Supplies the BM25Okapi sparse lexical index used alongside dense retrieval for exact-match legal terms."},{"cited_title":"Applied Artificial Intelligence40(1), 2626097 (2026)","cited_arxiv_id":null,"evidence_quote":"Documents a prior hybrid RAG architecture combining dense and sparse retrieval with cross-encoder reranking that the pipeline resembles."},{"cited_title":"In: European Conference on Information Retrieval, pp","cited_arxiv_id":null,"evidence_quote":"Provides the cross-encoder reranking approach for scoring query-passage pairs in legal answer retrieval."}],"review_version":1}