REVIEW 4 major objections 4 minor 25 references
Bringing legal knowledge to the public by constructing a legal question bank using large-scale pre-trained language model
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper claims that a hybrid GPT-3 prompting strategy can produce legal questions whose accuracy and coverage approach human-written ones, at much lower cost and with greater diversity.
desk verdict The hybrid partitioning strategy is a real engineering contribution with an unusually careful human evaluation, but the representativeness of the 100-page sample is not supported and the cost-effectiveness claim is asserted, not measured. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is "Hybrid partitioning" for prompt construction. Each CLIC-page is split into sections; each paragraph inside a section is labelled; and for every paragraph the model receives the full section text plus the instruction "10 most frequently-asked questions for Paragraph X". This combines section-level context with paragraph-level attention, and the paper shows it outperforms both section-only and paragraph-only prompts on quantity, precision, and coverage. Each generated question is bound to an answer scope s(q) = [page id : paragraph ids], which is what turns raw questions into navigational pointers; a DistilBERT embedding-based deduplication using cosine similarity and single-link clustering at a threshold of 0.95 removes near-duplicates before evaluation.
What would settle it
Select 100 pages from legal topics outside the five sampled, run the same Hybrid GPT-3 pipeline, and have legally trained annotators label each question as answerable by its source page; if precision or paragraph coverage falls materially below 68% or 93%, the claimed generalizability fails.
Extended reading notes
Core claim
On a sample of 100 CLIC-pages spanning landlord-and-tenant, defamation, insurance, personal-data privacy, and intellectual-property law, the paper shows that machine-generated questions can largely substitute for human-composed questions in building a Legal Question Bank. The Hybrid partitioning strategy, which feeds GPT-3 a whole section as context but asks for the ten most frequently asked questions about one labelled paragraph at a time, produces 4,979 deduplicated questions with 68% precision and 93% paragraph coverage; human workers produced 2,686 questions with 98% coverage and higher precision, but almost 90% of human questions were single-paragraph specific ones. The paper also identifies 69 machine questions that were relevant but not answerable by the source pages, which it calls "augmenting questions" and treats as a free by-product that reveals missing content and guides page revision. The overall discovery is that machine and human question creation complement each other: the machine supplies scale, diversity, and gap-finding, while humans supply precision.
Load-bearing premise
The evaluation assumes that 100 pages sampled under five legal topics represent all 1,557 pages of the CLIC platform, so the reported 68% precision and 93% coverage would hold for the other legal topics as well.
Editorial extensions
If this is right
- A legal-information site can scale its question bank to thousands of pages without paying lawyers for every question; the remaining human workload shifts to verifying machine questions, which is faster than writing them.
- Because every question in the bank carries an answer scope, a recommender can route a layperson's free-text description to the specific paragraphs that answer the matched question, directly addressing navigability.
- The "augmenting questions", which are relevant but not yet answered by their source pages, provide page editors with a concrete list of topics where existing legal pages need enrichment.
- Because Hybrid generates both single-paragraph and multi-paragraph questions, the resulting bank serves users who can articulate specific concerns as well as users who only know a general situation.
- A 100,000-question target becomes practical because compute-based generation avoids the large training-data expense of neural question generation, leaving manual effort concentrated on verification rather than composition.
Reading between the lines
- The same hybrid-prompt recipe likely transfers to other explainer corpora, such as medical, tax, or housing information, where plain-language pages exist but users cannot formulate precise queries; that extension is not tested in the paper.
- At 68% precision, a production question bank would need either a human verification pass or a confidence-gated filter, and the paper's cost argument depends on verification being materially cheaper than composition.
- The 32% non-answerable questions contain a signal: the paper counts 69 augmenting questions on 37 of 100 pages, so a screening step that separates "relevant but unanswered" from "irrelevant" could turn the generator into a content-audit tool.
- The reported numbers rest on five sampled topics; testing on structurally different legal areas, such as criminal procedure or immigration, would be the first check of whether the precision and coverage figures generalize.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper addresses the legal knowledge gap by proposing a three-step approach: CLIC pages that explain legal concepts in layperson's terms, a Legal Question Bank (LQB) of model questions with answer scopes, and a CLIC Recommender (CRec) that maps a user's verbal description to relevant LQB questions and CLIC pages. The main technical contribution is the automatic construction of an LQB with GPT-3, comparing three prompting strategies: section-based, paragraph-based, and hybrid partitioning. On a 100-page sample from 5 of 32 CLIC topics, the authors report that hybrid partitioning yields the best balance of quantity, precision (68%), coverage (93%), and diversity, and they argue that machine-generated questions compare favorably with human-composed questions on scalability, cost-effectiveness, and diversity while being somewhat less precise. A prototype of CRec is illustrated with one example. The central claim is that a hybrid GPT-3 prompting strategy can produce a legal question bank of comparable quality to human-composed questions, with better scalability, cost-effectiveness, and diversity.
Significance. If the evaluation is accepted, the paper offers a practical and inexpensive way to populate legal FAQ banks, and the hybrid partitioning idea, section-level context with paragraph-level attention, is simple and potentially transferable to other question-generation tasks. A notable strength is the human verification effort: 11,430 machine-generated questions and 2,686 human-composed questions were manually inspected, taking about 400 person-hours. However, the evidence is drawn from a convenience sample of 100 pages from 5 of 32 topics, and the absence of statistical tests, per-topic breakdowns, and direct cost measurements makes the headline conclusions contingent on assumptions that are not fully demonstrated. The paper is a useful engineering contribution, but the current evaluation does not yet establish the generalizability of the reported precision, coverage, and cost-effectiveness numbers.
major comments (4)
- [Section 4 (sample selection)] The claim that the 100-page sample is "representative" is not supported. The sample covers 5 of 32 topics, selected without a described random or stratified procedure, and the justification that about fourteen thousand questions were evaluated addresses sample size, not representativeness. The sample's question density under Hybrid is 49.8 questions/page (4,979/100) versus 38.4 questions/page for the full 1,557-page run (59,798/1,557), suggesting the sample over-represents content-rich pages or topics. Because the paper's conclusion about "comparable quality" rests on absolute precision (68%) and coverage (93%), this sampling issue directly affects the central claim. Please report per-topic results, confidence intervals, or a sampling justification.
- [Section 4.1, Table 2 and following paragraph] HCQ precision is never reported. The comparison "4,979 x 68% ~ 3,400 correct questions" versus 2,686 HCQs implicitly assigns 100% precision to all human-composed questions. Since the manual task (task 2) explicitly checked MGQ correctness but no equivalent check is reported for HCQs, the quality comparison is incomplete. Either report HCQ correctness verification or state explicitly that HCQs are assumed correct by construction.
- [Section 4.1 (cost-effectiveness)] The abstract and Section 4.1 claim that MGQs are "more cost-effective" and require "much lower (human) cost," but no direct cost measurement is provided. The only cost datum is the total of about 400 person-hours for all manual tasks. No per-question time is given for composing an HCQ versus verifying an MGQ, and no cost model or scaling analysis supports the claim. This is load-bearing because cost-effectiveness is one of the three stated advantages of MGQs over HCQs.
- [Sections 3.3 and 4 (coverage)] The coverage metric is defined over answer scopes that are initialized to all paragraphs in the prompt that generated each question. Unless the manual scope verification is exhaustive and documented, the 93% Hybrid coverage may substantially reflect prompt construction rather than post-hoc content coverage. Please clarify how many scopes were modified during verification, report inter-annotator agreement, and state whether coverage was computed from the initial or the verified scopes.
minor comments (4)
- [Table 4] Table 4 reports 2,685 HCQs while the text and Table 1 report 2,686; please reconcile the discrepancy.
- [Section 4, first paragraph of the evaluation description] The sentence "The evaluation results we present in the rest of this section is therefore representative" contains a subject-verb agreement error; it should be "are therefore representative."
- [Section 3.2] The GPT-3 hyperparameters (temperature, top_p, frequency penalty, presence penalty) are listed without any sensitivity analysis or rationale; even a brief justification would improve reproducibility and help readers understand the robustness of the reported results.
- [Section 3.2 and reproducibility] The manuscript does not specify the exact GPT-3 model variant (e.g., text-davinci-003) or the date of access; please state these details for reproducibility, since different GPT-3 versions can produce substantially different outputs.
Circularity Check
No circularity found: the evaluation is empirical against human judgments, and no derived quantity is equivalent to an input by construction.
full rationale
The paper's derivation chain is self-contained. The central comparisons—Hybrid vs section/ and paragraph-based partitioning, and MGQs vs HCQs—are evaluated by human verifiers with formal legal training, who label each MGQ as 'correct' or not and independently compose HCQs. Precision is the fraction of human-corrected labels, coverage is computed from human-verified answer scopes, and diversity is counted from those scopes. No quantity is fitted to the data that it then 'predicts': the GPT-3 hyperparameters and prompt templates are fixed before evaluation, and the three partitioning strategies are compared on the same human-judged data. The paper contains no load-bearing self-citation: the cited works are external (GPT-3, question-generation literature), and the CLIC platform is described as the domain corpus rather than as a prior uniqueness result. The 'augmenting questions' discussion reinterprets some incorrect MGQs, but it does not alter the reported precision or the head-to-head comparisons; it is an additional qualitative finding, not a redefinition of the measured metric. The representativeness concern raised by the 100-page sample is a sampling/external-validity limitation, not a circularity: it affects generalization, not whether the reported measurements reduce to their inputs. Therefore no circular step can be exhibited, and the score is 0.
Assumptions & free parameters
free parameters (3)
- GPT-3 hyperparameters =
temperature=1.0, top_p=0.9, frequency penalty=0.3, presence penalty=0.1
- Number of FAQs per prompt =
10
- Deduplication threshold =
0.95 cosine similarity
assumptions (3)
- domain assumption CLIC-pages are an appropriate source of plain-language legal knowledge.
- domain assumption Cosine similarity in DistilBERT embedding space reflects semantic similarity of legal questions.
- domain assumption Human legally-trained workers can reliably judge whether a generated question is answered by a CLIC-page.
Cite this review
Pith. "Pith review of Bringing legal knowledge to the public by constructing a legal question bank using large-scale pre-trained language model." pith.science (2026). https://pith.science/paper/YRAQSKKE
@misc{pith2026250504132,
author = {Pith},
title = {Pith review of: Bringing legal knowledge to the public by constructing a legal question bank using large-scale pre-trained language model},
year = {2026},
howpublished = {\url{https://pith.science/paper/YRAQSKKE}},
note = {Machine review of arXiv:2505.04132}
}
read the original abstract
Access to legal information is fundamental to access to justice. Yet accessibility refers not only to making legal documents available to the public, but also rendering legal information comprehensible to them. A vexing problem in bringing legal information to the public is how to turn formal legal documents such as legislation and judgments, which are often highly technical, to easily navigable and comprehensible knowledge to those without legal education. In this study, we formulate a three-step approach for bringing legal knowledge to laypersons, tackling the issues of navigability and comprehensibility. First, we translate selected sections of the law into snippets (called CLIC-pages), each being a small piece of article that focuses on explaining certain technical legal concept in layperson's terms. Second, we construct a Legal Question Bank (LQB), which is a collection of legal questions whose answers can be found in the CLIC-pages. Third, we design an interactive CLIC Recommender (CRec). Given a user's verbal description of a legal situation that requires a legal solution, CRec interprets the user's input and shortlists questions from the question bank that are most likely relevant to the given legal situation and recommends their corresponding CLIC pages where relevant legal knowledge can be found. In this paper we focus on the technical aspects of creating an LQB. We show how large-scale pre-trained language models, such as GPT-3, can be used to generate legal questions. We compare machine-generated questions (MGQs) against human-composed questions (HCQs) and find that MGQs are more scalable, cost-effective, and more diversified, while HCQs are more precise. We also show a prototype of CRec and illustrate through an example how our 3-step approach effectively brings relevant legal knowledge to the public.
Reference graph
Works this paper leans on
-
[1]
Becher, S.I. and U. Benoliel. 2021. Law in books and law in action: The readability of privacy policies and the GDPR , K. Mathis & A. Tor (Eds.), Consumer Law and Economics , 179--204. Springer International Publishing. doi:10.1007/978-3-030-49028-7_9
-
[2]
Brown, T., B. Mann, N. Ryder, M. Subbiah, J.D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei ...
work page 2020
- [3]
-
[4]
Dai, Z., A.T. Chaganty, V.Y. Zhao, A. Amini, Q.M. Rashid, M. Green, and K. Guu 2022. Dialog inpainting: Turning documents into dialogs. In International Conference on Machine Learning, ICML 2022 , pp.\ 4558--4586
work page 2022
-
[5]
Das, R., A. Ray, S. Mondal, and D. Das 2016. A rule based question generation framework to deal with simple and complex sentences. In 2016 International Conference on Advances in Computing, Communications and Informatics (ICACCI) , pp.\ 542--548. doi:10.1109/ICACCI.2016.7732102
arXiv 2016
-
[6]
Du, X., J. Shao, and C. Cardie 2017. Learning to ask: Neural question generation for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pp.\ 1342--1352. doi:10.18653/v1/P17-1123
- [7]
-
[8]
Heilman, M. and N.A. Smith 2010, June. Good question! S tatistical ranking for question generation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics , Los Angeles, California, pp.\ 609--617. Association for Computational Linguistics. doi:10.5555/1857999.1858085
arXiv 2010
Show all 25 references
-
[9]
McCandlish, T
Kaplan, J., S. McCandlish, T. Henighan, T.B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361\/ . doi:arXiv:2001.08361
2020 arXiv
-
[10]
Kim, Y., H. Lee, J. Shin, and K. Jung 2019. Improving neural question generation using answer separation. In Proceedings of the AAAI conference on artificial intelligence , Volume 33, pp.\ 6602--6609. doi:10.1609/aaai.v33i01.33016602
2019 doi
-
[11]
Popowich, J
Lindberg, D., F. Popowich, J. Nesbit, and P. Winne 2013. Generating natural language questions to support learning on-line. In Proceedings of the 14th European Workshop on Natural Language Generation , pp.\ 105--114
2013
-
[12]
Liu, B., H. Wei, D. Niu, H. Chen, and Y. He 2020. Asking questions the human way: Scalable question-answer generation from text corpus. In Proceedings of The Web Conference 2020 , pp.\ 2032--2043. doi:10.1145/3366423.3380270
2020
-
[13]
Min, B., H. Ross, E. Sulem, A.P.B. Veyseh, T.H. Nguyen, O. Sainz, E. Agirre, I. Heinz, and D. Roth. 2021. Recent advances in natural language processing via large pre-trained language models: A survey. arXiv preprint arXiv:2111.01243\/ . doi:arXiv:2111.01243
2021 arXiv
-
[14]
Mommers, L. 2011. Access to law in Europe , Innovating Government , 383--398. Springer. doi:10.1007/978-90-6704-731-9_21
2011 doi
-
[15]
Voermans, W
Mommers, L., W. Voermans, W. Koelewijn, and H. Kielman. 2009. Understanding the law: improving legal knowledge dissemination by translating the contents of formal sources of law. Artificial Intelligence and Law\/ 17\/ (1): 51--78. doi:10.1007/s10506-008-9073-5
2009 doi
-
[16]
New Zealand Law Reform Commission . 2008. New Zealand Parliamentary Counsel's Office (2008) Presentation of New Zealand statute law ( NZLC R104 ). https://www.lawcom.govt.nz/sites/default/files/projectAvailableFormats/NZLC\
2008
-
[17]
Radford, A., J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog\/ 1\/ (8): 9
2019
-
[18]
Shazeer, A
Raffel, C., N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P.J. Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research\/ 21\/ (140): 1--67
2020
-
[19]
Ruohonen, J. 2021. Assessing the readability of policy documents on the digital single market of the European Union . In 2021 Eighth International Conference on eDemocracy & eGovernment (ICEDEG) , pp.\ 205--209. IEEE. doi:10.1109/ICEDEG52154.2021.9530996
2021
-
[20]
Schick, T. and H. Sch \"u tze 2021. Generating datasets with pretrained language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pp.\ 6943--6951. doi:10.18653/v1/2021.emnlp-main.555
2021 doi
-
[21]
Song, L., Z. Wang, W. Hamza, Y. Zhang, and D. Gildea 2018, June. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, ...
2018 doi
-
[22]
Filighera, T
Steuer, T., A. Filighera, T. Tregel, and A. Miede. 2022. Educational automatic question generation improves reading comprehension in non-native speakers: A learner-centric case study. Frontiers in Artificial Intelligence\/ 5. doi:10.3389/frai.2022.900304
2022
-
[23]
Wang, S., Z. Wei, Z. Fan, Y. Liu, and X. Huang 2019. A multi-agent communication framework for question-worthy phrase extraction and question generation. In Proceedings of the AAAI Conference on Artificial Intelligence , Volume 33, pp.\ 7168--7175. doi:10.1609/aaai.v33i01.33017168
2019 doi
-
[24]
Valdez, D
Wang, Z., J. Valdez, D. Basu Mallick, and R.G. Baraniuk 2022. Towards human-like educational question generation with large language models. In International Conference on Artificial Intelligence in Education , pp.\ 153--166. Springer. doi:10.1007/978-3-031-11644-5_13
2022 doi
-
[25]
write newline
" write newline "" before.all 'output.state := FUNCTION output.doi doi empty skip "doi:" doi * "" * output if FUNCTION format.archive archivePrefix empty "" archivePrefix ":" * if FUNCTION format.primaryClass primaryClass empty "" " [" primaryClass * "] " * if FUNCTION format....
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.