UQLegalAI@COLIEE2025: Advancing Legal Case Retrieval with Large Language Models and Graph Neural Networks

Ruihong Qiu; Yanran Tang; Zi Huang

REVIEW 3 major objections 5 minor 1 cited by

UQLegalAI@COLIEE2025: Advancing Legal Case Retrieval with Large Language Models and Graph Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A graph-neural retrieval system places second on COLIEE 2025 legal case search with F1 0.2962.

desk verdict Credible second-place COLIEE 2025 result, but the paper's claimed contribution is untested: the CaseLink graph and degree regularization are never isolated from the BM25+year post-processing. read the letter →

arxiv 2505.20743 v1 pith:72FAPXR3 submitted 2025-05-27 cs.IR

Yanran Tang , Ruihong Qiu , Zi Huang This is my paper · ORCID

classification cs.IR

keywords LegalcaseretrievalGraphneuralnetworksGlobalContrastivelearningDegreeregularisationCOLIEE2025TextembeddingsBM25

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a legal case retrieval system, CaseLink, that places second on the COLIEE 2025 Task 1 benchmark with an F1 score of 0.2962. The system builds a Global Case Graph that connects cases to each other, cases to legal charges, and charges to similar charges; encodes case and charge text with a large language model specialised for embeddings (e5-mistral-7b-instruct); and trains a graph neural network with an InfoNCE contrastive objective plus a degree regularisation term. At inference, candidates are first pruned to ten by BM25, then filtered by date so that only earlier precedents are considered, and the top five are returned. A sympathetic reader would take the paper's claim to be that graph connectivity among cases, combined with these post-processing steps, is a competitive recipe for precedent retrieval.

What carries the argument

The Global Case Graph (GCG) is the central object: a single undirected, unweighted adjacency matrix that stacks case-case edges (top-K BM25 neighbours), case-charge edges (charge name appears in case text), and charge-charge edges (cosine similarity above a threshold). Node features are embeddings from e5-mistral-7b-instruct, truncated to 4096 tokens. A graph attention network updates these features, and two losses train it: InfoNCE contrastive learning with positive relevant cases and hard negatives from BM25 top-K, plus degree regularisation, which sums the cosine similarities between each candidate node and all other case nodes, penalising high pseudo-degree. Inference is a cosine similarity score, and post-processing prunes to BM25 top-10 and applies a year filter before taking the top five.

What would settle it

Run the identical pipeline with the GNN and degree regularisation removed, keeping BM25 top-10, year filtering, cosine similarity on the raw LLM embeddings, and top-five selection; if the F1 stays at or above 0.2962 on the COLIEE 2025 Task 1 test set, the graph component is not load-bearing.

Watch

Extended reading notes

Core claim

The central claim is that representing an entire legal case collection as one undirected graph, with case-case edges from BM25 similarity, case-charge edges from charge mentions, and charge-charge edges from charge-name similarity, lets a GNN produce case representations that rank relevant precedents well. The paper argues that the added degree regularisation, which penalises the total cosine similarity (pseudo-degree) of candidate nodes in the GNN-updated feature space, makes the model exploit case reference connectivity rather than only surface text. On the COLIEE 2025 Task 1 test set, the method reaches 0.2962 F1, second among all submitted runs, with the three submitted variants differing by only small hyperparameter changes. The authors attribute the gap to the first-place team partly to their own lower recall caused by selecting more candidates in ranking.

Load-bearing premise

The load-bearing premise is that the graph neural network and degree regularisation contribute meaningfully to the reported F1, beyond what the BM25 top-10 pruning, year filtering, and fixed top-five cutoff would achieve alone, and the paper shows no ablation that isolates their contribution.

Editorial extensions

If this is right

If correct, graph connectivity between cases is a usable signal for legal precedent retrieval, not just text similarity.
Degree regularisation gives a concrete training signal from the unlabelled structure of the case pool, so the model can be trained with only the contrastive positives and negatives plus graph structure.
Combining a lexical pruner (BM25 top-10) with a semantic graph ranker and a date filter yields a stable submission: three hyperparameter variants all scored between 0.2940 and 0.2962 F1.
The method's applicability to long legal texts is demonstrated by truncation to 4096 tokens, showing that full documents are not required for competitive retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reported design does not isolate the contribution of the graph and degree regularisation from the BM25 plus year-filter plus top-five pipeline, so a stripped-down baseline without the GNN could plausibly match the result; the paper itself reports no such ablation.
A natural next experiment is to vary the year-filter and top-K pruner while holding the GNN fixed, since date filtering encodes a strong legal prior that may account for much of the gain.
The same graph recipe might transfer to other citation-style retrieval tasks, such as patent or regulatory document retrieval, where explicit reference links exist alongside text.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

This is a five-page workshop paper describing a second-place run in COLIEE 2025 Task 1. The rank and F1 of 0.2962 are externally measured, so they are credible, and the open-source code link means the core pipeline can be reproduced. But the paper's own method text undercuts the novelty claims: Section 4.2 openly says the CaseLink module, including degree regularization, is adopted from the authors' prior SIGIR paper, while the abstract calls the regularization objective "new." That inconsistency is minor on its own, but it signals a more substantive gap: the reported configuration never isolates the graph and the degree regularization from the BM25+year post-processing. Inference in Sections 4.3–4.4 first prunes candidates to the BM25 top 10, filters to years earlier than the query, then picks the top five by CaseLink similarity. So the GNN only reorders a short BM25-selected list. Without a BM25+year+top-5 baseline or an ablation with lambda=0 / no GNN, the conclusion that case connectivity and degree regularization improve retrieval is not established by the evidence in the paper.

What is actually useful here is modest but real: the specific combination of e5-mistral embeddings, the BM25 prefilter, and the year filter works well on the COLIEE dataset, and the three submitted runs show stable precision/recall around +/-0.003. For someone building a legal case retrieval system and looking for a cheap, effective recipe, this is a decent starting point. The related work is standard and the dataset statistics are clearly presented.

The missing pieces are a straightforward ablation, a BM25-only baseline, and the exact hyperparameters of the submitted run — the paper lists search ranges but not the chosen values. These are fixable and would turn this from a credible system description into a useful one. A serious referee should engage with it, but the authors need to add those baselines before anything stronger is claimed.

Referee Report

3 major / 5 minor

Summary. This COLIEE 2025 system paper describes CaseLink, the second-ranked submission to Task 1 (legal case retrieval). The method builds a Global Case Graph with case-case, case-charge, and charge-charge edges; initializes node features with e5-mistral-7b-instruct embeddings; trains a GAT with an InfoNCE loss plus a degree-regularization term; and at inference computes cosine similarity between query and candidate representations. Post-processing first prunes to the BM25 top 10 candidates, applies a latest-date year filter, and returns the top five. The team reports an official F1 of 0.2962, within 0.04 of the first-place JNLP team.

Significance. If the attribution of the result to the graph components were supported, the paper would provide a useful competitive system description for long legal documents, with the credible external evaluation of the COLIEE organizers and an open-sourced codebase based on the prior CaseLink repository. The official F1 and ranking are externally measured and not in doubt. However, the current manuscript does not establish the claimed benefit of graph connectivity and degree regularization over the BM25-plus-year-filter pipeline, so the scientific contribution is currently limited to a rank report.

major comments (3)

[§4.4 and §6] The end-to-end score confounds the contributions of the learned model with the post-processing. Section 4.4.1 says candidate lists are first reduced to ten cases by BM25, then year filtering in Section 4.4.2 removes later cases, and only then are the top five chosen by CaseLink similarity. No baseline of BM25 + year filter + top-5 alone is reported, and no ablation with lambda=0 or without the GNN is reported, even though Section 5.1 lists lambda=0 as part of the search. Therefore the conclusion in Section 6 that the CaseLink graph and degree regularization 'demonstrate effectiveness' is not established by the reported F1; an ablation or baseline is required to support the central attribution claim.
[§4.2.3, Eq. (12)] The objective called 'degree regularization' is defined as the sum of all pairwise cosine similarities between the updated candidate-node representations in the fully connected pseudo-adjacency matrix. This is not a degree term in the usual graph sense, and the paper does not state whether the term is minimized or maximized, nor why this sum should be interpreted as a regularization of node degree. Since this term is presented as a central component of CaseLink, the description is too underspecified to reproduce or to reason about its effect.
[§5.2, Table 2] The three UQLegalAI runs differ by at most about 0.003 in F1, but these are three different hyperparameter configurations rather than repeated runs of the same configuration. The sentence comparing stability with JNLP is also inaccurate: JNLP's two listed F1 values of 0.3353 and 0.3267 have a half-range of about 0.0043, not ±0.01. The stability claim should be reworded or supported with repeated-run statistics, though this does not affect the official ranking.

minor comments (5)

[§3.3] Typo: 'evalutation metic' should be 'evaluation metric'.
[§4.1.2] Eq. (4) builds Case-Case edges from BM25 similarity, not from actual citation/reference relations; the phrase 'intrinsic case connectivity' should be clarified so readers know these edges are lexical-similarity neighbors.
[§4.1.3] The average case length in Table 1 is about 30,000 tokens, but e5-mistral-7b-instruct is truncated to 4096 tokens; the potential information loss from this truncation is not discussed.
[§4.4.2] The year filter uses the latest date appearing in a case as its representative trial date; it is not stated how dates are extracted or how ties (no date) are handled.
[Abstract and §4.2] The abstract says a 'new contrastive objective, incorporating a regularization on the degree of case nodes, is proposed,' but Section 4.2 states that the CaseLink module and degree regularization are adopted from the authors' previous work [18]; the paper should state explicitly which components are new to this submission.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported F1 is an externally measured COLIEE score, and the missing ablation of BM25/year post-processing is an attribution gap, not a circular derivation.

full rationale

The derivation chain runs from the CaseLink pipeline (GCG construction, e5-mistral embeddings, GNN, InfoNCE + degree regularization, BM25 top-10 + year filter + top-5) to the official COLIEE 2025 Task 1 F1 of 0.2962, which is measured by the competition organizers on the held-out test set and reported in Table 2. None of Equations (4)-(14) defines the reported result in terms of an input parameter fitted to the same test labels: the final ranking is computed by CaseLink cosine similarity on GNN-updated representations, and the official score is independent of the authors' own code, citations, or fitted hyperparameters. The absence of an ablation with λ=0 or without BM25 post-processing means the paper does not establish how much of the score comes from graph connectivity or degree regularization, but that is an empirical attribution gap, not circularity. The abstract's 'new' degree-regularization objective is internally inconsistent with Section 4.2, which says the CaseLink module is adopted from the authors' prior work [18]; this is a self-citation/novelty-consistency issue and is not load-bearing for the externally verified ranking. No load-bearing step reduces to its own inputs, so the circularity score is 0.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper relies on several unvalidated modeling choices: graph edge definitions, year filtering, token truncation, and the choice of best run. None of these are accompanied by ablation or sensitivity analysis, and the final hyperparameter values are not reported.

free parameters (6)

K (TopK BM25 neighbors) = selected from {3, 5, 10}
Controls Case-Case edge construction in Eq. 4; value not reported for the best run.
delta (charge-charge similarity threshold) = selected from {0.85, 0.9, 0.95}
Controls Charge-Charge edges in Eq. 5; final value not reported.
lambda (degree regularization coefficient) = selected from {0, 5e-4, 1e-3, 5e-3}
Balances InfoNCE and degree regularization in Eq. 13; includes 0, so the regularization may be absent.
number of hard negatives = selected from {1, 5, 10}
Sampled negatives for InfoNCE in Eq. 10; final value not reported.
number of final retrieved cases per query = fixed at 5
Chosen from training average of 4.1 relevant cases; directly controls precision and recall tradeoff.
LLM truncation length = 4096 tokens
Cases longer than the e5-mistral limit are truncated; no evidence the first 4096 tokens are representative.

assumptions (5)

domain assumption BM25 top-K similarity indicates useful case-to-case connectivity that helps representation learning
Used to build Case-Case edges in Eq. 4; no ablation tests whether these edges improve retrieval.
domain assumption A case-charge edge exists when a charge text appears in the case
Eq. 6 assumes surface mention of a charge name is a meaningful relation.
domain assumption The latest date appearing in a case is its representative trial date
Section 4.4.2 uses this to filter precedents; not validated, and could exclude valid citations.
standard math Contrastive InfoNCE with one positive and sampled negatives is a valid training signal
Adopted from [19]; standard in representation learning, but assumptions about negative sampling are not examined.
domain assumption Truncating long legal cases to 4096 tokens preserves the information needed for retrieval
Section 5.1; average case length is about 30k tokens, so most content is discarded.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UQLegalAI@COLIEE2025: Advancing Legal Case Retrieval with Large Language Models and Graph Neural Networks." pith.science (2026). https://pith.science/paper/72FAPXR3

@misc{pith2026250520743,
  author       = {Pith},
  title        = {Pith review of: UQLegalAI@COLIEE2025: Advancing Legal Case Retrieval with Large Language Models and Graph Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/72FAPXR3}},
  note         = {Machine review of arXiv:2505.20743}
}

read the original abstract

Legal case retrieval plays a pivotal role in the legal domain by facilitating the efficient identification of relevant cases, supporting legal professionals and researchers to propose legal arguments and make informed decision-making. To improve retrieval accuracy, the Competition on Legal Information Extraction and Entailment (COLIEE) is held annually, offering updated benchmark datasets for evaluation. This paper presents a detailed description of CaseLink, the method employed by UQLegalAI, the second highest team in Task 1 of COLIEE 2025. The CaseLink model utilises inductive graph learning and Global Case Graphs to capture the intrinsic case connectivity to improve the accuracy of legal case retrieval. Specifically, a large language model specialized in text embedding is employed to transform legal texts into embeddings, which serve as the feature representations of the nodes in the constructed case graph. A new contrastive objective, incorporating a regularization on the degree of case nodes, is proposed to leverage the information within the case reference relationship for model optimization. The main codebase used in our method is based on an open-sourced repo of CaseLink: https://github.com/yanran-tang/CaseLink.

Figures

Figures reproduced from arXiv: 2505.20743 by the authors.

**Figure 2.** The overall framework of CaseLink [18]. x𝑐𝑖 ∈ R 𝑑 , x𝑐 𝑗 ∈ R 𝑑 . The number of Charge-Charge edges is regulated by a similarity score threshold 𝛿. • Case-Charge Edge. A Case-Charge edge is established when a charge name appears in the case, which shows the high correlation between the charge and case. The adjacency matrix of Case-Charge edges A𝑏 ∈ R 𝑚×𝑛 is designed as: A𝑏𝑖 𝑗 = 1 for 𝑡𝑐𝑖 appears in 𝑡𝑑𝑗 , 0 for Oth… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

NOWJ@COLIEE 2026: Adaptive Pipelines for Legal Retrieval and Reasoning
cs.CL 2026-07 conditional novelty 4.0 of 10

The NOWJ team's multi-stage pipelines took first place in COLIEE 2026 legal case retrieval and statute-law entailment, while its dynamic difficulty-routing idea beat baselines on old data but lost on the official blind test.

Reference graph

Works this paper leans on

22 extracted references · 17 canonical work pages · cited by 1 Pith paper

[1]

Hamilton, Zhitao Ying, and Jure Leskovec

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Represen- tation Learning on Large Graphs. InNeurIPS

work page 2017
[2]

Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval.Journal of Documentation28, 1 (1972), 11–21

1972
[3]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. InICLR

2015
[4]

Kipf and Max Welling

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. InICLR

2017
[5]

Haitao Li, You Chen, Zhekai Ge, Qingyao Ai, Yiqun Liu, Quan Zhou, and Shuai Huo. 2024. Towards an In-Depth Comprehension of Case Relevance for Better Legal Retrieval. InJSAI, Vol. 14741. 212–227

work page 2024
[6]

Haitao Li, Weihang Su, Changyue Wang, Yueyue Wu, Qingyao Ai, and Yiqun Liu. 2023. THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval.CoRRabs/2305.06812 (2023)

work page Pith review arXiv 2023
[7]

Yixiao Ma, Yueyue Wu, Qingyao Ai, Yiqun Liu, Yunqiu Shao, Min Zhang, and Shaoping Ma. 2023. Incorporating Structural Information into Legal Case Re- trieval.ACM Trans. Inf. Syst.(2023)

work page 2023
[8]

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. MTEB: Massive Text Embedding Benchmark.CoRRabs/2210.07316 (2022)

arXiv 2022

Show all 22 references

[9]

Ponte and W

Jay M. Ponte and W. Bruce Croft. 2017. A Language Modeling Approach to Information Retrieval. InSIGIR

2017
[10]

Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu Kano, Masaharu Yosh- ioka, and Ken Satoh. 2022. Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021.Rev. Socionetwork Strateg.16, 1 (2022), 111–133

2022
[11]

Juliano Rabelo, Mi-Young Kim, and Randy Goebel. 2022. Semantic-Based Classi- fication of Relevant Case Law. InJURISIN

2022
[12]

Robertson and Steve Walker

Stephen E. Robertson and Steve Walker. 1994. Some Simple Effective Approxi- mations to the 2-Poisson Model for Probabilistic Weighted Retrieval. InSIGIR

1994
[13]

Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang, and Shaoping Ma. 2020. BERT-PLI: Modeling Paragraph-Level Interactions for Legal Case Retrieval. InIJCAI

2020
[14]

Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov

Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting.J. Mach. Learn. Res.(2014)

2014
[15]

Yanran Tang, Ruihong Qiu, and Xue Li. 2023. Prompt-based Effective Input Reformulation for Legal Case Retrieval.CoRRabs/2309.02962 (2023)

2023 arXiv
[16]

Yanran Tang, Ruihong Qiu, Yilun Liu, Xue Li, and Zi Huang. 2024. CaseGNN++: Graph Contrastive Learning for Legal Case Retrieval with Graph Augmentation. CoRRabs/2405.11791 (2024)

2024
[17]

Yanran Tang, Ruihong Qiu, Yilun Liu, Xue Li, and Zi Huang. 2024. CaseGNN: Graph Neural Networks for Legal Case Retrieval with Text-Attributed Graphs. InECIR

2024
[18]

Yanran Tang, Ruihong Qiu, Hongzhi Yin, Xue Li, and Zi Huang. 2024. CaseLink: Inductive Graph Learning for Legal Case Retrieval. InSIGIR. UQLegalAI@COLIEE2025: Advancing Legal Case Retrieval with Large Language Models and Graph Neural Networks COLIEE 2025, June 20, 2025, Chicago, USA

2024
[19]

Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding.CoRRabs/1807.03748 (2018)

2018 arXiv
[20]

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018. Graph Attention Networks. InICLR

2018
[21]

Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Improving Text Embeddings with Large Language Models. In ACL

2024
[22]

Kun Zhang, Chong Chen, Yuanzhuo Wang, Qi Tian, and Long Bai. 2023. CFGL- LCR: A Counterfactual Graph Learning Framework for Legal Case Retrieval. In SIGKDD

2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Hamilton, Zhitao Ying, and Jure Leskovec

William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive Represen- tation Learning on Large Graphs. InNeurIPS

work page 2017

[2] [2]

Karen Spärck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval.Journal of Documentation28, 1 (1972), 11–21

1972

[3] [3]

Kingma and Jimmy Ba

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Opti- mization. InICLR

2015

[4] [4]

Kipf and Max Welling

Thomas N. Kipf and Max Welling. 2017. Semi-Supervised Classification with Graph Convolutional Networks. InICLR

2017

[5] [5]

Haitao Li, You Chen, Zhekai Ge, Qingyao Ai, Yiqun Liu, Quan Zhou, and Shuai Huo. 2024. Towards an In-Depth Comprehension of Case Relevance for Better Legal Retrieval. InJSAI, Vol. 14741. 212–227

work page 2024

[6] [6]

Haitao Li, Weihang Su, Changyue Wang, Yueyue Wu, Qingyao Ai, and Yiqun Liu. 2023. THUIR@COLIEE 2023: Incorporating Structural Knowledge into Pre-trained Language Models for Legal Case Retrieval.CoRRabs/2305.06812 (2023)

work page Pith review arXiv 2023

[7] [7]

Yixiao Ma, Yueyue Wu, Qingyao Ai, Yiqun Liu, Yunqiu Shao, Min Zhang, and Shaoping Ma. 2023. Incorporating Structural Information into Legal Case Re- trieval.ACM Trans. Inf. Syst.(2023)

work page 2023

[8] [8]

Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2022. MTEB: Massive Text Embedding Benchmark.CoRRabs/2210.07316 (2022)

arXiv 2022