{"id":"0aa6d909-3871-4f7c-877c-e9fc37b989d1","arxiv_id":"2412.00934","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"QABISAR uses a query-article bipartite graph augmented with legal hierarchy and knowledge distillation to improve statutory article retrieval on BSARD.","lead":"This paper describes QABISAR, a retrieval model that connects legal questions and statute articles in a graph, then trains a lightweight question encoder to imitate the graph's richer question representations. The authors report consistent improvements over previous statutory article retrieval systems on the French-language BSARD benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The distillation teacher is a label-aware graph node: its relevance scores contain the gold answer via graph edges, so the KD gain may reflect memorized training query-article connections rather than transferable semantics for unseen queries.","rationale":"The paper's strongest claim is that adding query-article bipartite interactions plus graph-to-encoder knowledge distillation improves statutory article retrieval beyond a graph encoder that uses only statute topology. Table 1 supports this numerically, and the ablation descriptions in Section 3.2 are consistent with the claim. However, the support is entirely empirical and lacks code, error bars, significance tests, and rendered ablation plots. My specific concern is sharper than 'the KD might not generalize': the teacher's query representation q_g is not merely a stronger encoder; it is a node with direct edges to the gold articles, so the teacher's relevance distribution is conditioned on the answer key for that training query. Distilling this distribution into a text-only student is only justified if the query text alone can predict the same distribution. Given the small training set and the authors' own evidence of overfitting in feature distillation, the load-bearing assumption is exactly that score distillation avoids this memorization trap. The proposed holdout experiment removes a query's edges from the graph before training and then evaluates on that query, directly simulating the inference condition while keeping the architecture unchanged. If the QABISAR gain disappears in that setting, the central claim should be weakened: the article-side graph enrichment may account for the gains, and the KD contribution to unseen queries would be unverified. This does not invalidate the paper's contribution, but it means the CONDITIONAL verdict is appropriate: the claim is plausible and internally consistent, yet the decisive transfer test has not been reported.","tokens_in":7608,"tokens_out":6360,"duration_ms":69898,"concrete_test":"Using the paper's BSARD training split, randomly partition the training queries into 5 folds. For each fold, construct the bipartite graph using only the other 4 folds' queries and all articles; train the bi-encoder baseline and QABISAR (with and without KD) on those same 4 folds; evaluate on the held-out fold. Repeat over folds and seeds. The central claim survives if QABISAR's R@100 gain over the bi-encoder persists on held-out queries (e.g., at least half of the Table 1 gain); if the gain collapses to zero, the graph teacher's advantage existed only for queries whose gold edges were in the graph, showing KD did not transfer generalizable semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 2.2, the teacher scores used for KD are s(q_g, p_g), where q_g is the embedding of a query node in a graph whose edges are exactly the gold query-article relevance labels. The GAT aggregates over those edges, so q_g is answer-aware: it has directly absorbed information from the articles known to be relevant to that training query. The student q_b is a text-only bi-encoder that must reproduce this distribution without access to the graph edges. This is a structural overfitting risk: with only about 776 training queries, the student can imitate the answer-aware teacher by memorizing which articles are connected to which training query text. The ablation 'removing KD' cannot distinguish this from acquiring transferable semantics, since it only measures training-set behavior of the loss. The paper's own observation that feature distillation overfits because of the limited number of queries makes the same risk concrete for score distillation. At inference the only query-side component is the student encoder, so if KD is memorizing training query-article connections, the reported test-time gain is not evidence that bipartite interactions transfer to unseen queries.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes QABISAR, a two-stage framework for statutory article retrieval on the BSARD dataset. Stage one trains a bi-encoder with contrastive learning; stage two constructs a query-article bipartite graph augmented with the statute hierarchy, applies a graph attention network to obtain enriched query and article representations, and uses KL-divergence knowledge distillation to transfer the graph-based query scoring distribution into the query bi-encoder so that unseen test queries can be encoded without graph supervision. Experiments report consistent gains over baselines, with R@100 of 83.7 versus 82.3 for BE+GE-Stat, and ablations indicate that both the bipartite interactions and the distillation loss contribute to the result.","tokens_in":7872,"tokens_out":6030,"duration_ms":60386,"significance":"Statutory article retrieval is a practically relevant legal IR task, and the idea of exploiting many-to-many query-article interactions through a bipartite graph, while retaining an inductive text-only query encoder at inference, is a reasonable and potentially useful contribution. The paper uses a public expert-annotated dataset and compares against sensible baselines, and the ablations are aligned with the design. However, the empirical support for the central claim is currently incomplete: the reported numbers come from a single run with no variance or significance testing, the gap over the strongest baseline is small, and the distillation mechanism carries a real risk of memorizing training query-article links rather than learning transferable semantics. The manuscript does not provide code or data-processing scripts, which further limits reproducibility. If the transfer concern is resolved with additional analysis, the work would be a solid contribution to legal IR.","major_comments":[{"comment":"The teacher scores s(q_g, p_g) are answer-aware: q_g is the representation of a query node whose graph edges are exactly the gold query-article relevance labels, and the GAT aggregates information over those edges. The student q_b is a text-only bi-encoder that must reproduce this distribution without access to the graph edges. With a small training set (BSARD contains 1,108 questions and the paper never states the train/validation/test split), the student can in principle memorize which article IDs are relevant to each training query text. The paper's own observation in Section 3.2 that feature distillation overfits because of the limited number of queries makes this risk concrete. The 'without KD' ablation in Figure 1 does not settle the issue, because it only shows that removing the loss changes performance; it does not show whether the KD-trained student acquired transferable semantics. Please provide explicit transfer evidence, for example separate training and held-out query curves, or a teacher variant whose query representation is computed without aggregating the query's own gold edges, to rule out memorization of answer links.","section":"Section 2.2, KD loss"},{"comment":"All reported results appear to come from a single run: no variance, confidence intervals, or significance tests are provided. The main gain over BE+GE-Stat is 1.4 points on R@100 and 0.5 points on MAP. Given this magnitude, the claim that QABISAR 'consistently outperforms prior works across all metrics' needs to be supported by multiple seeds or a paired significance test (for example, bootstrap over queries), especially since the per-query metric distributions in legal retrieval are typically highly skewed.","section":"Table 1 and Figure 1"},{"comment":"The captions of the ablation figure and the distillation-strategy figure do not state whether the numbers are computed on the test split or on the training split. If these numbers are evaluated on training queries, they do not support the inference-time generalization claim, because the KD signal is only defined for training queries and the graph query nodes are only available for them. Please state the evaluation split explicitly and report held-out test numbers with variance.","section":"Figures 1 and 2, experimental protocol"},{"comment":"The distillation objective is underspecified in a way that affects its validity. The loss sums over a candidate set P, but the paper does not say whether P is the in-batch contrastive set, the full corpus, or a set of hard negatives; this changes the teacher distribution and the difficulty of the distillation task. It is also unclear whether query node features in the graph are refreshed from the evolving bi-encoder during joint training or fixed from the initial encoder at the start of stage two. If they are fixed, the student is trained to match a stale teacher. Please specify these choices and justify them.","section":"Section 2.2 and Appendix A"}],"minor_comments":[{"comment":"Typo: 'corpus of statues P' should read 'corpus of statutes P'.","section":"Section 2, task definition"},{"comment":"Typo: 'in the the same batch' should read 'in the same batch'.","section":"Section 2.1"},{"comment":"The column headers are misaligned: 'R@ MAP MRP' followed by '100 200 500' should be formatted so that each metric column is unambiguous.","section":"Table 1"},{"comment":"The hyperparameters for the graph encoder (number of GAT layers L, number of attention heads K, edge embedding dimension, and the contrastive temperature tau) are not reported; please provide them for reproducibility.","section":"Appendix A"},{"comment":"The sentence 'These bi-partite interactions, lead to enriched representations' contains a grammatical error and should be rephrased.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the venue and the core idea is interesting, but the current empirical evidence is not yet convincing enough for acceptance. The most important risk is the answer-aware distillation teacher: because the graph query node directly aggregates over gold relevance edges, the reported test-time gains could partly reflect memorized training query-article connections rather than transferable semantics. I would like a revision to include an explicit transfer analysis, multi-run statistics, and precise specification of the KD candidate set and evaluation split. The absence of code or data-processing scripts also weakens reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this paper: it extends Louis et al.'s graph-based statutory retrieval by adding query-article bipartite edges and distilling the graph's query representations into the bi-encoder. The central claim—that these two additions improve recall on BSARD—is plausible and the ablations align with the design. I largely agree with the reader's take: this is a useful, incremental contribution, not a breakthrough.\n\nWhat's genuinely new is the bipartite graph itself. Articles in the graph absorb information from multiple training queries simultaneously, and the graph's article embeddings are used at test time. The knowledge distillation step then tries to transfer the query-side benefit to the text-only encoder. That is a reasonable design, and the ablation captions suggest both graph views matter, with score distillation beating feature distillation. The paper is also honest about limitations: it targets recall, needs a re-ranker, and is limited to French/Belgian law.\n\nThe soft spots are evidentiary, not logical. Table 1 shows one run, with no variance or significance tests. The R@100 gap over BE+GE-Stat is 1.4 points; MAP and MRP gaps are under a point. Without error bars, those gains are not reassuring. The train/validation split is not specified, several hyperparameters (temperature, GAT layers, number of heads) are missing from the appendix, and no code or scripts are released. More importantly, the paper omits a comparison to CUSINES, the same group's curriculum negative-sampling method on the same dataset; that is a directly relevant baseline and should have been included or explained.\n\nOn the stress-test concern: yes, the teacher query representation is answer-aware because the gold edges connect it to relevant articles, so the KD signal includes the labels. But the test queries are unseen and never appear in the graph. If the student merely memorized training query-article pairs, it would not help on the test split; the fact that KD improves test performance suggests some transfer, though it does not rule out partial overfitting. A validation split and a fine-grained analysis of which queries benefit would settle it.\n\nWho this is for: anyone working on legal IR, especially statutory retrieval on BSARD. It deserves peer review; a serious referee should push for error bars, the CUSINES comparison, and reproducibility details before acceptance. I would cite it in my own work on graph-based retrieval.","headline":"A clean, incremental extension of graph-based statutory retrieval with a plausible KD transfer; the evidence is thinner than the claims warrant, but the work deserves a serious referee.","tokens_in":8373,"tokens_out":4362,"would_cite":true,"duration_ms":43934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a query-article bipartite graph and knowledge distillation improves statutory article retrieval beyond previous graph-based methods on every reported metric.","keywords":["statutory article retrieval","legal information retrieval","query-article bipartite graph","graph attention network","knowledge distillation","dense retrieval","BSARD","many-to-many relevance"],"falsifier":"Evaluate QABISAR against the prior graph-based model on a held-out query set whose legal topics do not overlap with the training queries; if the reported Recall@100 gain (83.7 vs 82.3) shrinks or disappears, then the distillation is memorizing training queries rather than transferring generalizable query semantics.","tokens_in":7420,"feed_emoji":"⚖️","tokens_out":9001,"duration_ms":75611,"temperature":0.7,"pith_summary":"QABISAR is an attempt to fix a blind spot in statutory article retrieval: treating each query-article pair as independent misses the fact that one article can answer many kinds of questions and one question can need several articles. The paper builds a graph whose nodes are training queries, all articles, and the statute hierarchy, with edges linking each query to its relevant articles. A graph attention network enriches both query and article representations through these multiple interactions, and knowledge distillation transfers the graph's query representations into a fast bi-encoder so unseen test queries still benefit. On the French-language Belgian statute benchmark, the method raises Recall@100 from 82.3 to 83.7 and Recall@200 from 85.1 to 87.9 over the previous graph-based approach. The broader point, if correct, is that many-to-many interaction modeling plus distillation can capture multi-faceted legal semantics without slowing down inference.","feed_headline":"Query–article graph lifts statute search to 83.7 recall","feed_subtitle":"Bipartite query-article links and score distillation beat the prior graph-only model on every metric.","key_machinery":"The load-bearing object is a two-stage training pipeline. Stage one trains a dense bi-encoder with contrastive loss, using a BERT query encoder and a hierarchical article encoder that pools chunk-level BERT representations. Stage two constructs a typed graph containing training queries, every corpus article, and statute hierarchy nodes, with edges for query-article relevance and for containment of articles in sections, chapters, titles, and books; graph attention layers then update each node from typed neighbors. Finally, a KL-divergence distillation loss trains the bi-encoder's query encoder to reproduce the graph teacher's relevance-score distribution over candidate articles, jointly with the graph contrastive loss. This joint training is what lets the bi-encoder act as a surrogate for the graph at inference.","core_discovery":"The paper's central claim is that enriching a dense retriever with a query-article bipartite graph, augmented by the statute hierarchy and processed by graph attention, yields article and query representations that are better than those learned from isolated pairs. Equally central is the claim that knowledge distillation of relevance-score distributions from the graph teacher into the query bi-encoder is what preserves this enrichment for unseen queries at test time. The ablations support the claim by showing that dropping either the bipartite edges or the distillation loss hurts performance, that the bipartite view matters more than the statute-topology view, and that score distillation alone beats feature distillation or a combination of both when training queries are scarce.","pith_inferences":["An implicit consequence is that the same graph-teacher/encoder-student recipe should transfer to other legal first-stage retrievers with unseen queries at test time, such as prior-case or paragraph retrieval, whenever training queries are numerous enough to form informative bipartite links.","A testable extension is to measure how the recall gain depends on training-query coverage: because articles can only be bridged through queries that appear in training, the benefit of bipartite edges should grow as the number of distinct training questions per article increases.","The superiority of score distillation over feature distillation hints at a general design rule for low-resource retrieval distillation: matching a teacher's ranking distribution is safer than forcing embedding equality when the training set is small, since the latter overfits.","Because QABISAR is framed as a recall-oriented pre-fetcher, a natural next check is whether the improved top candidates also raise end-to-end answer accuracy once a precision-focused re-ranker is attached."],"forward_implications":["Retrieval systems for legal corpora should model relevance as many-to-many relationships instead of scoring each query-article pair independently, because adding query-article edges improves Recall@100 from 82.3 to 83.7.","The graph can be confined to training: score-distillation lets a deployment-time bi-encoder reproduce graph-enriched query semantics without building a graph for unseen questions.","Statute hierarchy and query-article interactions are complementary sources of signal; removing either reduces performance, and losing the bipartite interactions hurts more.","When training queries are scarce, distilling ranking distributions is more effective than distilling exact embeddings for transferring graph representations."],"supporting_citations":[{"why":"Provides the BSARD French statutory article retrieval dataset and the dense bi-encoder baseline that QABISAR extends and compares against.","marker":"(Louis and Spanakis, 2022)"},{"why":"Establishes the statute-hierarchy graph idea and the BE+GE-Stat baseline whose article-topology enrichment QABISAR augments with query-article edges.","marker":"(Louis et al., 2023)"},{"why":"Supplies the dual-encoder dense retrieval architecture and the in-batch plus BM25 negative sampling used in the first training stage.","marker":"(Karpukhin et al., 2020)"},{"why":"Provides the knowledge-distillation approach adapted here to transfer graph query representations into the bi-encoder through score distributions.","marker":"(Hinton et al., 2015)"},{"why":"Defines the graph attention layer that aggregates typed neighbor information over the query-article and statute-hierarchy graph.","marker":"(Veličković et al., 2018)"},{"why":"Supplies the hierarchical article encoder that splits long statutes into chunks and pools chunk representations.","marker":"(Pappagari et al., 2019)"},{"why":"Provides the BERT encoder used for query and article chunk representations in the bi-encoder.","marker":"(Devlin et al., 2018)"}],"fun_headline_variants":["Bipartite graph boosts statute retrieval recall to 83.7","Query-article bipartite graph and KD sharpen legal retrieval","Bipartite interactions with distillation improve statute search","QABISAR: graph-aware queries distill statute article retrieval","Query-article bipartite network plus KD beats isolated baseline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the graph's query representations learned from training questions generalize to unseen questions, so distilling them into the bi-encoder transfers real semantic structure rather than memorizing the training queries.","fun_headline_variants_meta":{"raw":{"variants":["Bipartite graph boosts statute retrieval recall to 83.7","Query-article bipartite graph and KD sharpen legal retrieval","Bipartite interactions with distillation improve statute search","QABISAR: graph-aware queries distill statute article retrieval","Query-article bipartite network plus KD beats isolated baseline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000773,"raw_usage":{"total_tokens":3342,"prompt_tokens":783,"completion_tokens":2559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":2477}},"tokens_in":399,"tokens_out":2559,"duration_ms":18578,"temperature":1.0,"reasoning_tokens":2477,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:51:07.506096+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate QABISAR against the prior graph-based model on a held-out query set whose legal topics do not overlap with the training queries; if the reported Recall@100 gain (83.7 vs 82.3) shrinks or disappears, then the distillation is memorizing training queries rather than transferring generalizable query semantics.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the statute-hierarchy graph idea and the BE+GE-Stat baseline whose article-topology enrichment QABISAR augments with query-article edges."}],"review_version":1}