REVIEW 3 major objections 5 minor 1 cited by
Brief training raises GenAI adoption and exam grades for law students; access alone does not.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-15 14:52 UTC pith:TJLQACCL
load-bearing objection Clean three-arm RCT isolating brief training under optional LLM access; main ITT results are real, but the adoption-vs-effectiveness mechanism claim is under-identified by the paper’s own bounds. the 3 major comments →
Training for Technology: Adoption and Productive Use of Generative AI in Legal Analysis
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On a timed issue-spotting exam, optional LLM access without training did not improve performance and was associated with shorter answers and more case misstatements. Adding a short training intervention raised self-reported use (41% versus 26%) and raised mean grade point by 0.27 relative to untrained LLM access, while reducing rule misstatements. Principal-stratification bounds under strict mean dominance place the adoption channel above the effectiveness channel, though intervals are wide.
What carries the argument
Principal stratification that partitions participants into always-users, never-users, and induced users, then partially identifies the adoption effect for induced users and the effectiveness effect for always-users under monotone treatment response, exclusion for never-users, support bounds, and mean dominance.
Load-bearing premise
The claim that training works mainly by inducing new users rather than by improving existing users rests on assumptions (no defiers, exclusion for never-users, and mean dominance) whose resulting bounds have wide overlapping confidence intervals.
What would settle it
A larger pre-registered experiment that measures pre-treatment ability independently, records actual tool logs rather than self-report, and finds either no adoption increase among high-ability students or no grade-point gain once adoption is held fixed would overturn the central claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports a three-arm randomized experiment (N=164 law students) on a contract-law issue-spotting exam: no GenAI, optional DeepSeek access, or optional access plus a brief training video and quiz. Untrained access does not improve (and on some margins worsens) performance relative to no access; training raises self-reported adoption (41% vs 26%, p=0.044), mean grade point by 0.27 relative to untrained access (p=0.027), and reduces rule misstatements (p=0.014). Principal-stratification bounds under monotonicity, never-user exclusion, and mean dominance are used to argue that the training effect operates primarily through adoption rather than effectiveness among always-users, though the paper notes that bootstrap CIs are wide. The authors further interpret ability-quartile patterns as evidence that, without training, higher-ability students opt out while lower-ability students adopt unproductively.
Significance. If the main ITT contrasts hold, the paper supplies clean experimental evidence that a low-cost, task-specific training intervention can reverse the null or negative effect of optional GenAI access on a judgment-intensive professional task. That finding is directly relevant to legal education, firm training policy, and the broader debate on whether GenAI levels or amplifies skill differences. Strengths include pre-registration of main hypotheses, random assignment, blinded grading, transparent flagging of non-pre-registered tests, and an explicit (if partial) identification strategy for the adoption-versus-effectiveness channels. The contribution is empirical and policy-relevant rather than theoretical; the stylized model in §2 is illustrative scaffolding.
major comments (3)
- The abstract and §5.5/§6 claim that training "operates primarily through adoption rather than effectiveness" and that "the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance." Table 3 shows that under γ=0 the adoption lower-bound 95% bootstrap CI is [-0.364, 1.740] and the effectiveness upper-bound CI is [-0.049, 0.938]; the intervals overlap substantially. The ordering is therefore a point-estimate comparison under an untestable dominance assumption that equalizes the bounds already at γ≈0.64. The main ITT results do not depend on this decomposition; the mechanism language should be rewritten to match the paper's own identification limits (e.g., "point estimates under strict mean dominance favor adoption, but CIs do not reject effectiveness as the larger channel").
- Adoption is measured solely by post-exam self-report (§5.1). With optional access and no server logs or prompt records, differential reporting (e.g., trained participants more willing to admit use) cannot be ruled out and would inflate both the first-stage and the principal-stratum proportions. A robustness check that treats non-response or that bounds the adoption contrast under plausible misreporting rates would strengthen the central claim; alternatively the limitation should be stated more prominently when interpreting the 15 pp difference.
- Table 4 and the accompanying discussion in §6 treat post-exam grade-point quartiles as a proxy for pre-treatment ability and conclude that training induces adoption among high-ability students (0% → 42% in Q4). Quartile cell sizes are small (n=6 in Group 2 Q4), the analysis is not pre-registered, and the ability measure is the outcome itself, so selection and reverse causality are possible. The claim that "higher-ability practitioners opt out while lower-ability users adopt but unproductively" is load-bearing for the paper's challenge to the leveling narrative; it should be either supported by a pre-treatment ability measure or clearly labeled as exploratory and post-hoc.
minor comments (5)
- Several two-tailed tests on length and case citations (§5.3–5.4) are flagged as non-pre-registered; keep that discipline consistently in the abstract and introduction so that the reader does not over-weight them.
- Figure 4 density overlay and the grade-point bar chart (Figure 5) would be clearer with exact sample sizes and a note that grade point is discrete (1–4.3).
- The theoretical model in §2 uses ΔY_T and L(c,e_T) without stating units or how they map to the grade-point scale; a short sentence linking the model objects to the empirical outcomes would help.
- Typo / cross-reference: "Error! Reference source not found." appears in §6 before Table 4.
- Appendix preference/helpfulness results are null; a one-sentence summary in the main text would prevent the reader from wondering whether attitudes moved.
Circularity Check
No circularity: RCT contrasts and principal-stratification bounds are self-contained empirical results, not forced by definition or self-citation.
full rationale
The paper's load-bearing claims are direct experimental contrasts (adoption rates 41% vs 26%, grade-point difference 0.27, rule-misstatement reduction) from a three-arm RCT of 164 law students, plus partial-identification bounds under stated assumptions (monotonicity, never-user exclusion, mean dominance). The stylized productivity model in §2 is illustrative only and is not used to generate or constrain the empirical estimates. Principal stratification follows Frangakis–Rubin with support and dominance restrictions; the resulting bounds (Table 3) are computed from observed conditional means and stratum proportions, not from parameters fitted to the target effects themselves. No equation equates a claimed prediction to a fitted input by construction; no uniqueness theorem or ansatz is imported via self-citation; references are external (Acemoglu, Autor, Choi/Schwarcz, etc.). The mechanism ordering (adoption lower bound > effectiveness upper bound at γ=0) is a point-estimate comparison under an untestable assumption whose CIs overlap—an identification limitation, not circularity. The derivation chain is therefore self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
free parameters (1)
- mean-dominance tolerance γ =
0 (strict) and grid up to 1.665
axioms (6)
- domain assumption Random assignment of participants to the three experimental arms (SUTVA and no interference).
- domain assumption Monotone treatment response: training never decreases DeepSeek use (no defiers).
- domain assumption Exclusion restriction for never-users: training does not affect outcomes of participants who never use DeepSeek.
- domain assumption Global support restriction: grade points lie in [1.0, 4.3] under all potential outcomes.
- ad hoc to paper Mean dominance (up to γ): trained induced users perform at least as well as trained always users minus γ.
- domain assumption Self-reported DeepSeek use is a valid measure of actual adoption during the exam.
invented entities (1)
-
always / induced / never users (principal strata)
no independent evidence
read the original abstract
Can targeted user training unlock the productive potential of generative artificial intelligence in professional settings? We study this question using a randomized experiment in which 164 law students completed an issue-spotting examination under one of three conditions: no GenAI access, optional access to a large language model (LLM), or LLM access with a brief training intervention. Untrained LLM access proved counterproductive: relative to participants without any LLM access, untrained users wrote significantly shorter answers, committed more case misstatements, and scored marginally lower, though most differences fall short of conventional significance. Training reversed this pattern. Trained participants adopted the LLM at higher rates (41% vs. 26%; p = 0.044), scored 0.27 grade points higher than untrained users--roughly one fine grade--(p = 0.027), and stated applicable rules more accurately (p = 0.014). Principal stratification analysis suggests training operates primarily through adoption rather than effectiveness--the adoption lower bound (1.06) exceeds the effectiveness upper bound (0.42) at strict mean dominance--though confidence intervals are wide. More broadly, these findings challenge the view that GenAI primarily benefits lower-skilled workers: without training, higher-ability practitioners opt out while lower-ability users adopt but unproductively. Realizing GenAI's productivity gains requires investment in both access and instruction.
Figures
Forward citations
Cited by 1 Pith paper
-
Eskwai for Students: Generative AI Assistant for Legal Education in Ghana
Eskwai for Students is a RAG-based AI system for Ghanaian legal questions, deployed over 30 months to 3.1K students who submitted 32K queries, with evaluation of helpfulness and identification of ethical concerns in u...
Reference graph
Works this paper leans on
-
[1]
The Simple Macroeconomics of AI
“The Simple Macroeconomics of AI.” Economic Policy 40 (121): 13–58. https://doi.org/10.1093/epolic/eiae042. Acemoglu, Daron, and David Autor
-
[2]
https://doi.org/10.1016/S0169-7218(11)02410-5
Elsevier. https://doi.org/10.1016/S0169-7218(11)02410-5. Acemoglu, Daron, and Pascual Restrepo
-
[3]
Automation and New Tasks: How Technology Displaces and Reinstates Labor
“Automation and New Tasks: How Technology Displaces and Reinstates Labor.” Journal of Economic Perspectives 33 (2): 3–30. https://doi.org/10.1257/jep.33.2.3. Acemoglu, Daron, and Pascual Restrepo
-
[4]
Robots and Jobs: Evidence from US Labor Markets
“Robots and Jobs: Evidence from US Labor Markets.” The Journal of Political Economy 128 (6): 1–57. https://doi.org/10.1086/705716. Agrawal, Ajay K., John McHale, and Alexander Oettl
-
[5]
Enhancing Worker Productivity Without Automating Tasks: A Different Approach to AI and the Task- Based Model. Working Paper No. 34781. National Bureau of Economic Research. https://doi.org/10.3386/w34781. American Bar Association
-
[6]
Angrist, Joshua D., Guido W
https://www.americanbar.org/groups/law_practice/resources/law- technology-today/2025/the-legal-industry-report-2025/. Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin
2025
-
[7]
Identification of Causal Effects Using Instrumental Variables
“Identification of Causal Effects Using Instrumental Variables.” Journal of the American Statistical Association 91 (434): 444–55. https://doi.org/10.1080/01621459.1996.10476902. Autor, David H
-
[8]
Applying AI to Rebuild Middle Class Jobs. Working Paper No. 32140. National Bureau of Economic Research. https://doi.org/10.3386/w32140. Autor, David H., Frank Levy, and Richard J. Murnane
-
[9]
The Skill Content of Recent Technological Change: An Empirical Exploration
“The Skill Content of Recent Technological Change: An Empirical Exploration.” The Quarterly Journal of Economics 118 (4): 1279–333. https://doi.org/10.1162/003355303322552801. Bauer, Emmanuel, Dominik Stammbach, Nianlong Gu, and Elliott Ash
-
[10]
General Purpose Technologies ‘Engines of Growth’?
“General Purpose Technologies ‘Engines of Growth’?” Journal of Econometrics 65 (1): 83–108. https://doi.org/10.1016/0304- 4076(94)01598-T. Brynjolfsson, Erik, Danielle Li, and Lindsey R. Raymond
-
[11]
Generative AI at Work. Working Paper No. 31161. National Bureau of Economic Research. https://doi.org/10.3386/w31161. Charlotin, Damien. n.d. “AI Hallucination Cases.” Accessed January 28,
-
[12]
Lawyering in the Age of Artificial Intelligence
“Lawyering in the Age of Artificial Intelligence.” Minnesota Law Review 109 (1): 147–218. https://doi.org/10.24926/265535.4225. Choi, Jonathan H., and Daniel B. Schwarcz
-
[13]
Understanding the Duty of Competence for Attorneys Using Generative Ai
“Understanding the Duty of Competence for Attorneys Using Generative Ai.” North Carolina Journal of Law & Technology 27: 1–26. https://doi.org/10.2139/ssrn.5053423. Cui, Kevin Zheyuan, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz
-
[14]
The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers
“The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers.” Preprint, Massachusetts Institute of Technology, Department of Economics, February. https://economics.mit.edu/sites/default/files/2024- 01/draft_copilot_experiments.pdf. Dahl, Matthew, Varun Magesh, Mirac Suzgun, and Daniel E. Ho
2024
-
[15]
Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models
“Large Legal Fictions: Profiling Legal Hallucinations in Large Language Models.” Journal of Legal Analysis 16 (1): 64–93. https://doi.org/10.1093/jla/laae003. Dell’Acqua, Fabrizio, Edward McFowland, Ethan R. Mollick, et al
-
[16]
“Navigating the Jagged Technological Frontier: Field Experimental Evidence of the Effects of AI on Knowledge Worker Productivity and Quality.” SSRN Electronic Journal, ahead of print. https://doi.org/10.2139/ssrn.4573321. Denniston, Alex, and Peter Duffy
-
[17]
GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models
“GPTs Are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models.” arXiv:2303.10130. Preprint, arXiv, August
-
[18]
Engel, Christoph, and Johannes Kruse
https://doi.org/10.48550/arXiv.2303.10130. Engel, Christoph, and Johannes Kruse
-
[19]
Principal Stratification in Causal Inference
“Principal Stratification in Causal Inference.” Biometrics 58 (1): 21–29. https://doi.org/10.1111/j.0006- 341X.2002.00021.x. Freitas, Julian
-
[20]
Why People Resist Embracing AI
“Why People Resist Embracing AI.” Harvard Business Review, January. https://hbr.org/2025/01/why-people-resist-embracing-ai. Humlum, Anders, and Emilie Vestergaard
2025
-
[21]
Confidence Intervals for Partially Identified Parameters
“Confidence Intervals for Partially Identified Parameters.” Econometrica 72 (6): 1845–57. https://doi.org/10.1111/j.1468-0262.2004.00555.x. Lam, Kwok-Yan, Victor C. W. Cheng, and Zee Kin Yeong
-
[22]
“Building a Better Lawyer: Experimental Evidence That Artificial Intelligence Can Increase Legal Work Efficiency.” Journal of Empirical Legal Studies 21 (4): 979–1022. https://doi.org/10.1111/jels.12396. Noy, Shakked, and Whitney Zhang
-
[23]
Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence
“Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence.” Science 381 (6654): 187–92. https://doi.org/10.1126/science.adh2586. Peng, Sida, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer
-
[24]
The Impact of AI on Developer Productivity: Evidence from GitHub Copilot
“The Impact of AI on Developer Productivity: Evidence from GitHub Copilot.” arXiv:2302.06590. Preprint, arXiv, February
-
[25]
https://doi.org/10.48550/arXiv.2302.06590. Rubin, Donald B
-
[26]
Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies
“Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.” Journal of Educational Psychology 66 (5): 688–701. https://doi.org/10.1037/h0037350. Rubin, Donald B
-
[27]
“Inference and Missing Data.” Biometrika 63 (3): 581–92. https://doi.org/10.1093/biomet/63.3.581. Rubin, Donald B
-
[28]
Estimating Causal Effects from Large Data Sets Using Propensity Scores
“Estimating Causal Effects from Large Data Sets Using Propensity Scores.” Annals of Internal Medicine 197 (8_Part_2): 757–63. https://doi.org/10.7326/0003-4819-127-8_Part_2-199710151-00064. Savelka, Jaromir
work page doi:10.7326/0003-4819-127-8_part_2-199710151-00064
-
[29]
“Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts.” Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, June 19, 447–51. https://doi.org/10.1145/3594536.3595161. Schwarcz, Daniel, and Jonathan H. Choi
-
[30]
AI Tools for Lawyers: A Practical Guide
“AI Tools for Lawyers: A Practical Guide.” Minnesota Law Review Headnotes 108 (1): 1–39. https://minnesotalawreview.org/wp-content/uploads/2023/10/FL1-Choi- Schwarcz.pdf. Schwarcz, Daniel, Sam Manning, Patrick James Barry, David R. Cleveland, J. J. Prescott, and Beverly Rich
2023
-
[31]
“AI-Powered Lawyering: AI Reasoning Models, Retrieval Augmented Generation, and the Future of Legal Practice.” Preprint, SSRN. https://doi.org/10.2139/ssrn.5162111. Splawa-Neyman, Jerzy, D. M. Dabrowska, and T. P. Speed
-
[32]
On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9
“On the Application of Probability Theory to Agricultural Experiments. Essay on Principles. Section 9.” Statistical Science 5 (4). https://doi.org/10.1214/ss/1177012031. Thomson Reuters
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.