REVIEW 3 major objections 3 minor 1 cited by
Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use
T0 review · 3 major / 3 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Structured RAG retrieves more authoritative fair-use precedents than text-only retrieval.
desk verdict A thoughtful, honest prototype for legal RAG, but its only quantitative result is close to a tautology: the retrieval score and the evaluation metric are the same PageRank signal. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a domain-specific knowledge graph whose nodes are Cases, Courts, Opinions, and six paragraph types—Facts, Purpose, Nature, Amount, Market, and Conclusion—connected by CITED, DECIDED_IN, HAS_OPINION, and APPEALS_TO edges. Retrieval scores are $s_i = w_{\mathrm{text}}\mathrm{TextSim}_i + w_{\mathrm{cit}}\mathrm{Citation}_i + w_{\mathrm{court}}\mathrm{Court}_i$, with the citation and court terms coming from PageRank over the citation network and over the appellate hierarchy respectively. This lets retrieval align a complaint with the statutory factor it raises and then rerank by doctrinal authority, instead of treating every opinion as a flat text chunk.
What would settle it
Have legal experts annotate the factor paragraphs in a sample of the corpus and compare them with the extracted labels. Low agreement (for example, below an expert-acceptance threshold on factor assignment) would show the factor-level graph encodes the LLM's guess rather than doctrine, so the higher PageRank of the retrieved set could no longer be attributed to legal structure.
Extended reading notes
Core claim
The paper's central claim is that injecting legal structure into the retrieval stage—storing fair-use opinions as factor-level paragraphs, linking them through citations and court hierarchy, and scoring them with a convex combination of semantic similarity and two PageRank terms—yields retrieved sets that are more doctrinally authoritative than semantic-only retrieval. The measured evidence is comparative: on 20 unresolved copyright complaints, the structured configuration produced mean PageRank 0.213 (SD 0.315) versus 0.026 (SD 0.114) for standard RAG, while text similarity fell from 0.753 to 0.521. The authors offer this as preliminary evidence that local doctrinal structure can reduce naive-retrieval and inapplicable-authority failures in legal RAG.
Load-bearing premise
The conclusion collapses if the LLM's factor-level extraction is inaccurate: paragraphs assigned to the wrong statutory factor make factor-level retrieval and the graph-based reranking measure the LLM's labels rather than legal doctrine.
Editorial extensions
If this is right
- Fair-use answers can be grounded in cases courts actually treat as controlling, which should reduce the hallucinated or merely textually similar citations observed in existing legal RAG tools.
- Factor-level paragraph nodes let a user interrogate each statutory factor—purpose, nature, amount, market—separately, matching the structure of 17 U.S.C. § 107.
- Exposing the three weights as hyperparameters lets legal experts steer retrieval, favoring court hierarchy in appellate-heavy disputes or text similarity in novel cases.
- Pulling in the cited opinions of retrieved cases simulates how a practitioner follows precedent, adding context for the downstream fair-use analysis.
Reading between the lines
- Editorial: if the PageRank gap is confirmed on a larger, expert-annotated set, the next test is whether retrieval quality alone drives downstream fair-use conclusions, which could be isolated by varying only the retriever while holding the summarizer fixed.
- Editorial: PageRank rewards being cited, not being right for this dispute, so a high-authority retrieved set may still miss the controlling precedent for a quirky fact pattern; a testable extension is to score retrieved sets by expert-rated usefulness rather than by PageRank.
- Editorial: because citation counts penalize recently decided authorities, a time-aware authority score—such as citation velocity or a recency-adjusted PageRank—could be swapped into the same architecture without changing its shape.
- Editorial: the graph schema is doctrine-shaped rather than corpus-shaped, so the same recipe—factor nodes plus citation and hierarchy edges—could be ported to other multi-factor legal tests, with flat RAG as a cheap baseline.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Structured RAG, a retrieval-augmented generation system specialized for U.S. copyright fair use analysis. In addition to standard semantic similarity, it incorporates citation-based PageRank scores and court-hierarchy scores into the retrieval ranking, and it decomposes judicial opinions into factor-level chunks (facts, purpose, nature, amount, market, conclusion) using LLM-based extraction. The system uses Chain-of-Thought and interleaved retrieval. The paper reports a preliminary evaluation on 20 unresolved PACER complaints, comparing Standard RAG (text similarity only) with Structured RAG (weighted combination at w_text = w_cit = w_court = 0.333), and finds that Structured RAG retrieves opinions with higher mean PageRank (0.213 vs 0.026) while Standard RAG achieves higher text similarity (0.753 vs 0.521). The authors interpret this as support for the hypothesis that adding legal structure improves doctrinal relevance of retrieval.
Significance. If validated, the idea of encoding statutory-factor structure and citation authority into RAG for a narrow legal doctrine would be a useful contribution, especially for access-to-justice applications in DMCA counter-notification contexts. The paper is commendable for making a functioning prototype and source code publicly available, and for clearly framing the work as preliminary. However, the current empirical evidence does not support the central claim as stated: the primary outcome metric (PageRank) is also a component of the retrieval scoring function, so the observed improvement is substantially a self-referential artifact. The factor-level extraction is also unvalidated. The significance of the contribution would depend on new evaluation using external doctrinal relevance labels, expert judgment, or downstream inference quality, or on reframing the paper as a system/prototype description rather than a demonstrated retrieval improvement.
major comments (3)
- [Section 4, Table 2, and Eq. (1)] The central claim that Structured RAG improves doctrinal relevance rests on the mean PageRank gap in Table 2 (0.213 vs 0.026). However, Eq. (1) defines the retrieval score as a convex combination in which the PageRank-based citation score is a component, and the Structured RAG configuration sets w_cit = 0.333. PageRank is therefore both the signal being optimized and the outcome being measured. Under min-max scaling and top-k selection, giving positive weight to a signal is expected to raise the mean of that signal in the selected set, regardless of whether the signal captures doctrinal quality. Table 2 thus does not distinguish doctrinally better retrieval from a mechanical re-ranking on the optimized signal. The paper's own caveat that PageRank is an imperfect proxy for doctrinal authority (Sections 3.3 and 5.2) makes the need for an external criterion explicit; expert doctrinal relevance judgments or blind assessment of downstream fair-use analysis from both retrieval methods would be required.
- [Section 3.2] The factor-level decomposition is load-bearing, but the extraction of verbatim paragraphs for facts, the four statutory factors, and the conclusion is performed by an LLM without validation against a gold standard. If paragraphs are misassigned across factors, the factor-level vector search and the graph representation inherit the error, so the apparent doctrinal granularity may be spurious. The authors should provide at least a small manual audit of the extracted paragraphs, ideally with inter-annotator agreement, before using factor-level annotations as evidence for the method's value.
- [Section 4] The quantitative comparison reports no significance testing, confidence intervals, or per-complaint breakdown, and the test set is only 20 PACER complaints. Even setting aside the circularity of the PageRank metric, the statement that these findings 'support our hypothesis' is stronger than the evidence warrants. If the PageRank metric is retained, the authors should report paired distributions and a paired statistical test; if an external label set is added, it should be evaluated with standard IR metrics.
minor comments (3)
- [Section 4] The text says 'Figure 2' when referring to the quantitative comparison, but the comparison is reported in Table 2 and Figure 5; the cross-reference should be corrected.
- [References] Reference [28] (PACER) lacks a publication year and access date; the entry currently contains a placeholder '????' and should be completed.
- [Section 3.4] The sentence beginning 'This prototype was developed mainly for internal testing and refinement of the retrieval process, but is accessible to external users' is a run-on; consider splitting it for readability.
Circularity Check
Structured RAG's reported doctrinal-authority gain is largely assured by construction: Table 2 scores retrieval with the same PageRank signal that Eq. (1) optimizes.
-
self definitional
[Section 3.3, Eq. (1); Section 4, Table 2]
"The retrieved documents are ranked based on a convex combination, where si = w_text TextSim_i + w_cit Citation_i + w_court Court_i (1) such that w_text, w_cit, w_court in [0,1] and w_text + w_cit + w_court = 1. ... Structured RAG yields higher doctrinal relevance with significantly increased PageRank scores (Mean = 0.213, SD = 0.315)."
Eq. (1) defines Structured RAG's retrieval ranking as a weighted sum in which Citation_i and Court_i are PageRank-based authority scores, and the reported configuration sets w_cit = w_court = 0.333. Table 2 then evaluates the two methods by comparing the mean PageRank of the retrieved sets. Thus the evaluation metric is one of the very signals the Structured RAG ranking function is designed to maximize, while Standard RAG gives that signal zero weight. Selecting the top-k documents with positive weight on PageRank will, by construction, produce a retrieved set with higher mean PageRank than selecting purely on text similarity, regardless of doctrinal relevance or downstream fair-use quality.
full rationale
The paper's central empirical claim is that Structured RAG retrieves doctrinally authoritative fair-use precedents better than Standard RAG, and this claim rests on Table 2's mean PageRank gap (0.213 vs 0.026). However, Eq. (1) makes citation-based PageRank and court-hierarchy scores explicit components of the Structured RAG ranking function, with the prototype using uniform weights w_cit = w_court = 0.333. Table 2 evaluates the retrieval methods using the same PageRank signal that Structured RAG is built to optimize, while Standard RAG sets that weight to zero; hence the gap is largely a mathematical consequence of the intervention, not an independent measure of doctrinal quality. The paper honestly labels the results preliminary and notes that PageRank is an imperfect estimate of doctrinal authority, and it defers user studies and ablation studies to future work; these limitations are correctly stated but do not remove the evaluative circularity. No self-citation chain is load-bearing here, and the unvalidated LLM-based factor extraction is a correctness risk rather than a circularity. Because the only quantitative evidence for the central claim reduces to a self-referential comparison, the circularity score is high, though not maximal: an external evaluation could in principle validate the approach, so the derivation is not entirely equivalent to its input by definition.
Assumptions & free parameters
free parameters (3)
- Retrieval weights (w_text, w_cit, w_court) =
w_text=1 for Standard RAG; 1/3 each for Structured RAG
- top-k retrieved documents =
not stated
- n cited opinions to expand =
not stated
assumptions (4)
- domain assumption PageRank over the case citation network and court hierarchy are valid proxies for doctrinal authority in fair use retrieval.
- domain assumption LLM extraction assigns verbatim paragraphs to the correct fair use factors and case sections without validation.
- domain assumption The 20 unresolved PACER complaints are a representative sample for measuring retrieval performance.
- domain assumption Vector cosine similarity from Gecko embeddings captures legally relevant text similarity.
Cite this review
Pith. "Pith review of Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use." pith.science (2026). https://pith.science/paper/S7QBEZN2
@misc{pith2026250502164,
author = {Pith},
title = {Pith review of: Incorporating Legal Structure in Retrieval-Augmented Generation: A Case Study on Copyright Fair Use},
year = {2026},
howpublished = {\url{https://pith.science/paper/S7QBEZN2}},
note = {Machine review of arXiv:2505.02164}
}
read the original abstract
This paper presents a domain-specific implementation of Retrieval-Augmented Generation (RAG) tailored to the Fair Use Doctrine in U.S. copyright law. Motivated by the increasing prevalence of DMCA takedowns and the lack of accessible legal support for content creators, we propose a structured approach that combines semantic search with legal knowledge graphs and court citation networks to improve retrieval quality and reasoning reliability. Our prototype models legal precedents at the statutory factor level (e.g., purpose, nature, amount, market effect) and incorporates citation-weighted graph representations to prioritize doctrinally authoritative sources. We use Chain-of-Thought reasoning and interleaved retrieval steps to better emulate legal reasoning. Preliminary testing suggests this method improves doctrinal relevance in the retrieval process, laying groundwork for future evaluation and deployment of LLM-based legal assistance tools.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Retrieval-Augmented Large Language Models as Components of Cognitive Computing architecture for Regulatory Knowledge Management
RAG-augmented local Bielik and PLLuM models produce more terminologically dense Polish legal answers than bare LLMs, yet the study is small, readability-focused, and still reports fundamental legal mistakes.
Reference graph
Works this paper leans on
-
[1]
U.S. Congress, 17 u.s.c. § 512 - limitations on lia- bility relating to material online, https://www.law. cornell.edu/uscode/text/17/512, 2024. Retrieved from https://www.law.cornell.edu/uscode/text/17/512
work page 2024
-
[2]
U.S. Congress, 17 u.s.c. § 107 - limitations on ex- clusive rights: Fair use, https://www.law.cornell. edu/uscode/text/17/107, 2024. Retrieved from https://www.law.cornell.edu/uscode/text/17/107
work page 2024
-
[3]
J. D. Matteson, Unfair misuse: How section 512 of the dmca allows abuse of the copyright fair use doctrine and how to fix it, Santa Clara high-technology law journal 35 (2018) 1
work page 2018
-
[4]
S. M. Blythe, Freedom of speech and the dmca: Abuse of the notification and takedown process, European intellectual property review 41 (2019) 70–88
work page 2019
-
[5]
T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Ka- plan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sas- try, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, D. Amod...
work page 2020
-
[6]
J. Lai, W. Gan, J. Wu, Z. Qi, P. S. Yu, Large language models in law: A survey, AI Open 5 (2024) 181–196. URL: https://www.sciencedirect.com/science/article/ pii/S2666651024000172. doi: https://doi.org/10. 1016/j.aiopen.2024.09.002
work page 2024
-
[7]
H. Westermann, S. Meeùs, M. Godet, A. Troussel, J. Tan, J. Savelka, K. Benyekhlef, Bridging the gap: Mapping layperson narratives to legal issues with lan- guage models, in: Proceedings of the Sixth Workshop on Automated Semantic Analysis of Information in Legal Text (ASAIL 2023), CEUR Workshop Proceed- ings, Braga, Portugal, 2023. URL: https://ceur-ws.or...
work page 2023
-
[8]
J. Savelka, K. D. Ashley, M. A. Gray, H. Westermann, H. Xu, Can gpt-4 support analysis of textual data in tasks requiring highly specialized domain expertise?, in: Proceedings of the Sixth Workshop on Automated Semantic Analysis of Information in Legal Text (ASAIL 2023), CEUR Workshop Proceedings, Braga, Portugal,
work page 2023
Show all 35 references
-
[9]
N. Guha, J. Nyarko, D. E. Ho, C. Re, A. Chilton, A. Narayana, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano, D. Talisman, E. Hoque, F. Surani, F. Fagan, G. Sarfaty, G. M. Dickinson, H. Po- rat, J. Hegland, J. Wu, J. Nudell, J. Niklaus, J. J. Nay, J. H. Choi, ...
2023
-
[10]
M. Dahl, V. Magesh, M. Suzgun, D. E. Ho, Large le- gal fictions: Profiling legal hallucinations in large lan- guage models, Journal of Legal Analysis 16 (2024) 64–93. URL: https://doi.org/10.1093/jla/laae003. doi:10. 1093/jla/laae003
2024 doi
-
[11]
Magesh, F
V. Magesh, F. Surani, M. Dahl, M. Suzgun, C. D. Manning, D. E. Ho, Hallucination-free? assess- ing the reliability of leading ai legal research tools, 2024. URL: https://arxiv.org/abs/2405.20362. arXiv:2405.20362
2024 arXiv
-
[12]
R. A. Jacobs, M. I. Jordan, S. J. Nowlan, G. E. Hinton, Adaptive mixtures of local experts, Neural Computa- tion 3 (1991) 79–87. doi: 10.1162/neco.1991.3.1. 79
1991 doi
-
[13]
B. J. Gutiérrez, Y. Shu, W. Qi, S. Zhou, Y. Su, From rag to memory: Non-parametric continual learning for large language models, 2025. URL: https://arxiv.org/ abs/2502.14802.arXiv:2502.14802
2025 arXiv
-
[14]
S. Pan, L. Luo, Y. Wang, C. Chen, J. Wang, X. Wu, Uni- fying large language models and knowledge graphs: A roadmap, IEEE Transactions on Knowledge and Data Engineering 36 (2024) 3580–3599. doi:10.1109/TKDE. 2024.3352100
2024
-
[15]
S. Chen, H. Zhang, T. Chen, B. Zhou, W. Yu, D. Yu, B. Peng, H. Wang, D. Roth, D. Yu, Sub-sentence en- coder: Contrastive learning of propositional semantic representations, in: K. Duh, H. Gomez, S. Bethard (Eds.), Proceedings of the 2024 Conference of the North American Chapte...
2024
-
[16]
T. Chen, H. Wang, S. Chen, W. Yu, K. Ma, X. Zhao, H. Zhang, D. Yu, Dense X retrieval: What re- trieval granularity should we use?, in: Y. Al-Onaizan, M. Bansal, Y.-N. Chen (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, Associa...
2024 doi
-
[17]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. H. Chi, Q. V. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, in: Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS ’22, Curran Ass...
2022
-
[18]
Trivedi, N
H. Trivedi, N. Balasubramanian, T. Khot, A. Sabhar- wal, Interleaving retrieval with chain-of-thought rea- soning for knowledge-intensive multi-step questions, in: A. Rogers, J. Boyd-Graber, N. Okazaki (Eds.), Pro- ceedings of the 61st Annual Meeting of the Associa- tion for C...
2023
-
[19]
Lewis, E
P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rock- täschel, S. Riedel, D. Kiela, Retrieval-augmented gen- eration for knowledge-intensive nlp tasks, in: Proceed- ings of the 34th International Conference on Neural Infor...
2020
-
[20]
Sartor, P
G. Sartor, P. Santin, L. D. Caro, Chasing the invisi- ble in the grammar of repetitions: A network analy- sis approach to fiscal state aids, in: Proceedings of the Sixth Workshop on Automated Semantic Analy- sis of Information in Legal Text (ASAIL 2023), CEUR Workshop Proceedi...
2023
-
[21]
Sanmartin, Kg-rag: Bridging the gap between knowledge and creativity, 2024
D. Sanmartin, Kg-rag: Bridging the gap between knowledge and creativity, 2024. URL: https://arxiv.org/ abs/2405.12035.arXiv:2405.12035
2024 arXiv
-
[22]
Sepasdar, S
Z. Sepasdar, S. Gautam, C. Midoglu, M. A. Riegler, P. Halvorsen, Enhancing structured-data retrieval with graphrag: Soccer data case study, 2024. URL: https: //arxiv.org/abs/2409.17580.arXiv:2409.17580
2024 arXiv
-
[23]
URL: http://ceur-ws.org/Vol-3441/, use permitted under Creative Commons License Attribution 4.0 In- ternational (CC BY 4.0)
-
[24]
Team, Gemini: A family of highly capable multi- modal models, 2024
G. Team, Gemini: A family of highly capable multi- modal models, 2024. URL: https://arxiv.org/abs/2312. 11805.arXiv:2312.11805
2024 arXiv
-
[25]
J. Lee, Z. Dai, X. Ren, B. Chen, D. Cer, J. R. Cole, K. Hui, M. Boratko, R. Kapadia, W. Ding, Y. Luan, S. M. K. Duddu, G. H. Abrego, W. Shi, N. Gupta, A. Kusupati, P. Jain, S. R. Jonnalagadda, M.-W. Chang, I. Naim, Gecko: Versatile text embeddings distilled from large language...
2024 arXiv
-
[26]
L. Page, S. Brin, R. Motwani, T. Winograd, The PageR- ank citation ranking: Bringing order to the web., Tech- nical Report, Stanford infolab, 1999
1999
-
[27]
Cushman, M
J. Cushman, M. Dahl, M. Lissner, eyecite: A tool for parsing legal citations, Journal of Open Source Soft- ware 6 (2021) 3617. URL: https://doi.org/10.21105/joss. 03617
2021 doi
-
[28]
Courts, Public ac- cess to court electronic records (pacer), https://pacer
Administrative Office of the U.S. Courts, Public ac- cess to court electronic records (pacer), https://pacer. uscourts.gov, ???? Original source of federal court records
-
[29]
T. F. L. Project, Recap archive, https://www. courtlistener.com/recap/, 2020. Accessed January 23, 2020
2020
-
[30]
Ferrario, M
A. Ferrario, M. Loi, How explainability contributes to trust in ai, in: Proceedings of the 2022 ACM conference on fairness, accountability, and transparency, 2022, pp. 1457–1466
2022
-
[31]
Barabási, R
A.-L. Barabási, R. Albert, Emergence of scaling in random networks, Science 286 (1999) 509–512. doi:10. 1126/science.286.5439.509
1999
-
[32]
Version 5.26.2, Available at: https://neo4j.com/
Neo4j, Inc., Neo4j Graph Database, 2025. Version 5.26.2, Available at: https://neo4j.com/
2025
-
[35]
Beebe, An empirical study of u.s
B. Beebe, An empirical study of u.s. copyright fair use opinions, 1978–2005, University of Pennsylvania Law Review 156 (2008) 549–634. URL: https://scholarship. law.upenn.edu/penn_law_review/vol156/iss3/2/
2008
-
[1609]
URL: https://aclanthology.org/2024.naacl-long. 89/. doi:10.18653/v1/2024.naacl-long.89
2024 doi
-
[2023]
URL: http://ceur-ws.org/Vol-3441/, available un- der CC BY 4.0 license
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.