Pith. sign in

REVIEW 4 major objections 8 minor 33 references

Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Arguments

T0 review · 4 major / 8 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that current LLMs are mostly faithful when generating factor-based legal arguments but often omit available factors and, when instructed to abstain because no shared factors exist, most generate spurious arguments anyway.

desk verdict A useful, honest pipeline paper for legal-argument evaluation; the headline numbers rest on an unvalidated LLM extractor, so treat rankings as indicative until the extractor is measured. read the letter →

arxiv 2506.00694 v2 pith:B2BHYCRV submitted 2025-05-31 cs.CL cs.AIcs.LG

Li Zhang , Morgan Gray , Jaromir Savelka , Kevin D. Ashley This is my paper
classification cs.CLcs.AIcs.LG
keywords LLMevaluationlegalargumentgenerationhallucinationfactor-basedreasoningabstentionautomatedmetricsinstructionfollowingcase-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces an automated pipeline for evaluating how faithfully large language models generate structured 3-ply case-based legal arguments. It defines hallucination as citing a factor not present in the input case materials, factor utilization as how many of the available ground-truth factors the model mentions, and abstention as the model's willingness to stop when the cases share no common factors. Applying the pipeline to eight LLMs on three tests, the paper argues that models are largely faithful on viable argument-generation tasks but systematically incomplete, and that most models fail the abstention test despite explicit instructions to stop. The interest lies in showing that factual accuracy and instruction following can be measured separately and automatically, and that the second is the weaker behavior.

What carries the argument

The pipeline is built on factor-encoded case representation, where each case is a fixed set of 26 trade-secret factors with plaintiff or defendant valence, taken from the HYPO/CATO tradition of case-based reasoning. An external evaluator LLM (GPT-4.1) extracts the factor sets the test model asserted for each case in the triple, and three metrics compare those sets to ground truth: Hallucination Accuracy (1 minus normalized hallucinated factors), Factor Utilization Recall (normalized correct factor mentions), and Abstention Ratio (successful abstentions over abstention-required triples). These metrics carry the argument because they turn qualitative fidelity judgments into reproducible scores that rank the eight models.

What would settle it

Have legal experts manually annotate the factor mentions in the same 90 generated argument triples and compare their annotation to the GPT-4.1 extraction; if the human-extracted sets diverge systematically from the automatic sets on certain models or phrasings, the reported AccH, RecU, and model rankings would change, showing the scores are extraction artifacts rather than true behavior differences.

Watch

Extended reading notes

Core claim

The paper's central empirical claim is that current LLMs, when given factor-encoded case triples and asked to generate a plaintiff-defendant-rebuttal argument, achieve high Hallucination Accuracy (over 90% on Tests 1 and 2) while Factor Utilization Recall varies widely, from roughly 40% to 85%, with most models far below completeness. Its sharper claim concerns Test 3: when the current case shares no factors with either precedent, most models fail to follow the instruction to abstain, with several scoring 0% Abstention Ratio, and instead generate spurious arguments even though those arguments are often still built from factors genuinely present in the input. The paper presents this as evidence that reliable legal deployment is blocked less by hallucination than by incomplete reasoning and weak adherence to negative constraints.

Load-bearing premise

The validity of every score depends on the external GPT-4.1 extractor reading each generated argument and producing the exact set of factor mentions, without silently missing factor references or inserting factors that were never actually named.

Editorial extensions

If this is right

  • If the pipeline works as claimed, faithfulness, completeness, and abstention in factor-based legal argument generation can be measured at scale without manual annotation, enabling large model comparisons.
  • The observed gap between high hallucination accuracy and low factor recall implies that LLMs' legal arguments are often factually safe but shallow, missing supporting and distinguishing factors a competent advocate would use.
  • The abstention results imply that strong generative performance does not transfer to instruction following under negative constraints, so legal deployment needs explicit abstention training or evaluation rather than assuming it follows from general capability.
  • The three metrics give future work a concrete, automated target: improving factor utilization and abstention can be verified directly by these scores, not just by human reading.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported hallucination accuracy may be partly an artifact of the extractor: if the GPT-4.1-based evaluator misses factors the test model mentioned in prose without the factor-number label, the metric would overstate faithfulness, so a human-annotation comparison would likely shift the numbers.
  • The abstention failure is plausibly a general negative-instruction-following weakness rather than a legal-reasoning failure, which means Test 3 could be repurposed as a cheap probe for that behavior in any domain with structured inputs and a clear stop condition.
  • Because factor misattribution (citing a real factor in the wrong case) is counted as a hallucination by the current metric, a case-attribution-aware variant of AccH would separate invented factors from misplaced real ones, giving a finer diagnostic.
  • Extending the pipeline from synthetic factor-encoded triples to real legal documents would require a robust factor-extraction step from raw text; if that step becomes reliable, the same three metrics could evaluate arguments on authentic case law.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. This paper presents an automated pipeline for evaluating LLM-generated 3-ply, factor-based legal arguments in the U.S. trade secret domain. A synthetically generated set of 90 case triples (30 per mode) is given to eight LLMs under three conditions: standard argument generation (Test 1), generation with swapped precedent roles (Test 2), and a non-arguable abstention condition (Test 3). An external LLM (GPT-4.1) extracts factor mentions attributed to each case from the generated arguments, and three metrics are computed: hallucination accuracy (AccH), factor utilization recall (RecU), and abstention ratio (RatioAbstain). The main findings are that most models exceed 90% AccH on Tests 1 and 2, RecU ranges from roughly 42% to 85%, and most models rarely abstain in Test 3 despite explicit instructions, with only GPT-4o abstaining in 86.67% of triples.

Significance. The paper's metrics are simple, transparent, and well matched to the factor-based CBR tradition: AccH, RecU, and RatioAbstain separately quantify three distinct failure modes, and the controlled synthetic dataset makes the ground truth unambiguous. The abstention finding is practically important: nearly all of the evaluated models, including reasonably capable ones, generate spurious arguments when explicitly instructed to stop, which bears directly on deployment risk in legal settings. The qualitative error analysis with concrete output excerpts independently corroborates the abstention failure. The paper ships no code, data, or human-validated extractor evaluation, however, so the quantitative rankings are currently the raw output of an unvalidated measurement instrument; the significance of the paper as a measurement methodology therefore rests on extractor validation that is currently missing.

major comments (4)
  1. [§3.4.1, §3.5, §8] The central methodological claim that the GPT-4.1-based extraction pipeline measures factor mentions reliably enough to rank the eight models is not supported by the evidence presented. Section 3.5 reports only informal 'spot checks' of extractor accuracy, Section 8 concedes that 'the accuracy of our automated metrics inherently depends on the performance of the external LLM used for factor extraction,' and no precision/recall figures against human annotation, no extraction error taxonomy, and no released outputs are provided. This is load-bearing because the headline results (AccH and RecU in Tables 2 and 3) are computed from F_Ext, the extraction of model assertions, and the eight models produce widely differing output styles: a model that paraphrases factors ('the plaintiff had agreed not to disclose') rather than emitting factor codes ('F4 Agreed-not-to-disclose') may have its mentions missed, which would systematically depress RecU; and a model that writes 'F5 is not present in the current case' may be recorded as mentioning F5 for the current case, which would simultaneously inflate NH and depress NU. Without a human-annotated validation sample stratified by model, the reported rankings cannot be distinguished from extraction artifacts. The authors should validate the extractor on a sample of outputs and report per-model extraction precision and recall, or release the data and outputs so that readers can do so.
  2. [§5.3, Table 4] No uncertainty quantification is reported anywhere, and for the abstention test the sample is only 30 triples per model. RatioAbstain in Table 4 is a binomial proportion with n=30, for which the standard error reaches about 9 percentage points; accordingly Llama-3-70B (3.33%), DeepSeek (23.33%), and Qwen (56.67%) are not all pairwise distinguishable at conventional significance levels, and the statement that several models 'failed to abstain in any of the test instances' should be accompanied by a confidence interval. The same issue affects the factor-level metrics in Tables 2 and 3, since they are computed over roughly 36 ground-truth factor slots per triple and the paper gives no measure of dispersion across the 30 triples. I request confidence intervals or an exact test for the proportions, and per-triple distributions for AccH and RecU.
  3. [§3.4.4, §6, Appendix A] The abstention criterion appears to be exact-phrase matching. The prompt quoted in Appendix A instructs models to say 'No common factor between the input current case and the TSC1/TSC2,' whereas Section 4.2 states the phrase is e.g. 'Cannot generate argument due to lack of common factors' and Section 6 describes the required phrase as 'No common factor between the current case and the TSC1/TSC2'; these three formulations are inconsistent, and the exact-match requirement means that a model that recognizes the impossibility but phrases the abstention slightly differently is scored as a failure. This operationalization conflates recognition of task impossibility with verbatim instruction following, and Section 6's 'Incorrect Abstention Phrase' category shows that such near-misses occur. Please state precisely which phrase was required, and report the near-miss abstention rate separately from the exact-match RatioAbstain.
  4. [§5.3, Table 2] The claim in Section 5.3 that GPT-4o's 'spurious arguments were largely free of hallucinated factors' is not supported by the aggregate Test 3 AccH of 99.16%, because that figure includes the 86.67% of triples in which the model abstained; an abstention produces an empty extraction set and therefore trivially contributes NH=0 to the numerator. The faithfulness of the spurious arguments should be assessed by conditioning on the non-abstained outputs only, as the qualitative example in Box 3 does.
minor comments (8)
  1. [§1 (Introduction)] The phrase 'of utomost importance' contains a typo; it should be 'of utmost importance.'
  2. [Table 2] The Llama-4-Maverick row renders as '96.9698.1594.35' with no separators between the three test columns; this formatting error must be fixed.
  3. [§4.1] The word 'controlls' should be 'controls.'
  4. [§4.1 vs §4.3] Test 2 is described in Section 4.3 as 'Also using the Arguable triples' with swapped order, while Section 4.1 says the separate 'Reordered' triples are used for Test 2; please clarify which dataset was actually used and whether the outcome labels were swapped.
  5. [Table 1] The layout of Table 1 makes it difficult to determine which outcome label belongs to which case in each mode; please reformat so each case's outcome is unambiguous.
  6. [§4.4] Please state whether any generated outputs were truncated by the 500-token limit (or the 5,000-token limit for reasoning models), since truncation would mechanically depress RecU.
  7. [§3.4.3] RecU counts every correct per-case mention of a ground-truth factor as utilization regardless of argumentative role; since the three plies only require shared factors for analogy and differing factors for distinction, please justify why all ground-truth factors are the appropriate recall target.
  8. [§3.5] The claim that the pipeline enables 'reproducible' evaluation would be strengthened by releasing the 90 triples, the generated outputs, and the extraction outputs; please add a data and artifact availability statement.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the metrics compare GPT-4.1-extracted factor sets directly against the provided ground-truth factor sets; all central claims are empirical measurements, not predictions derived from assumptions that include the results.

full rationale

The paper's derivation chain is straightforward: synthetic case triples are generated with known ground-truth factor sets (F_GT); models under test generate 3-ply arguments; GPT-4.1 extracts factor sets from those arguments (F_Ext); and AccH, RecU, and RatioAbstain are computed by plain set arithmetic (Eqs. 1-3). Hallucination is defined as F_Ext containing a factor not in F_GT for the same case, and utilization is F_Ext ∩ F_GT. Nothing is fitted, and no quantity is defined in terms of the quantity it is claimed to predict. The abstention ratio counts whether the model emitted the required abstention phrase, which is again a direct behavioral measurement. The only self-citation is to the authors' pilot work [3] as motivation, but the automated metrics are independent of that work and do not rely on its conclusions. The paper explicitly concedes in Section 8 that the metrics' accuracy 'inherently depends on the performance of the external LLM used for factor extraction'; this is an acknowledged validity limitation about measurement error, not a circular step. No uniqueness theorem, ansatz smuggled via citation, or renaming of a known result is present. Therefore there is no circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central empirical findings rest on two load-bearing assumptions: (1) GPT-4.1 extracts factor mentions from generated arguments accurately enough for the metrics to be trustworthy, and (2) the synthetically generated factor-encoded cases constitute a valid test bed for the behaviors of interest. The exact-phrase abstention criterion is a third, acknowledged operationalization choice that affects RatioAbstain. No free parameters are fit to data; the metrics are deterministic set comparisons.

assumptions (4)
  • domain assumption The 26-factor HYPO representation is a valid ground-truth encoding for judging factual faithfulness.
    Section 3.2: cases are represented as factor lists; the evaluation uses these lists as truth. This is standard in CBR, but it means findings only apply to factor-encoded inputs, not raw legal text.
  • ad hoc to paper GPT-4.1 accurately extracts every factor mention from generated arguments.
    Section 3.4.1 and 3.5: the evaluator LLM is the measurement instrument; only informal spot checks are reported, and Section 8 concedes metric accuracy depends on it.
  • domain assumption The synthetic case triples are correctly constructed, including the absence of shared factors in non-arguable sets.
    Section 4.1: the custom generation tool is not described in enough detail to verify; if construction is flawed, Test 3 would not test what it claims.
  • domain assumption The exact abstention phrase is the only acceptable abstention; semantically equivalent refusals count as failures.
    Section 3.4.4 and 6: 'Incorrect Abstention Phrase' is listed as an error, so models that convey the same meaning without the exact string are scored as non-abstaining. Section 8 says this operationalization could be refined.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Arguments." pith.science (2026). https://pith.science/paper/B2BHYCRV

@misc{pith2026250600694,
  author       = {Pith},
  title        = {Pith review of: Measuring Faithfulness and Abstention: An Automated Pipeline for Evaluating LLM-Generated 3-ply Case-Based Legal Arguments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2BHYCRV}},
  note         = {Machine review of arXiv:2506.00694}
}
read the original abstract

Large Language Models (LLMs) demonstrate potential in complex legal tasks like argument generation, yet their reliability remains a concern. Building upon pilot work assessing LLM generation of 3-ply legal arguments using human evaluation, this paper introduces an automated pipeline to evaluate LLM performance on this task, specifically focusing on faithfulness (absence of hallucination), factor utilization, and appropriate abstention. We define hallucination as the generation of factors not present in the input case materials and abstention as the model's ability to refrain from generating arguments when instructed and no factual basis exists. Our automated method employs an external LLM to extract factors from generated arguments and compares them against the ground-truth factors provided in the input case triples (current case and two precedent cases). We evaluated eight distinct LLMs on three tests of increasing difficulty: 1) generating a standard 3-ply argument, 2) generating an argument with swapped precedent roles, and 3) recognizing the impossibility of argument generation due to lack of shared factors and abstaining. Our findings indicate that while current LLMs achieve high accuracy (over 90%) in avoiding hallucination on viable argument generation tests (Tests 1 & 2), they often fail to utilize the full set of relevant factors present in the cases. Critically, on the abstention test (Test 3), most models failed to follow instructions to stop, instead generating spurious arguments despite the lack of common factors. This automated pipeline provides a scalable method for assessing these crucial LLM behaviors, highlighting the need for improvements in factor utilization and robust abstention capabilities before reliable deployment in legal settings. Link: https://lizhang-aiandlaw.github.io/An-Automated-Pipeline-for-Evaluating-LLM-Generated-3-ply-Case-Based-Legal-Arguments/

Figures

Figures reproduced from arXiv: 2506.00694 by the authors.

Figure 2
Figure 2. Overall Automated Evaluation Pipeline Flowchart The core argument generation step takes the factor￾represented case triple as input and invokes the chosen model with specific parameters (Section 4.4) to produce the 3-ply textual argument. The LLM output, along with metadata, is logged for subsequent analysis. 3.4. Automated Metrics Definition To evaluate the generated 3-ply arguments quantitatively, we developed aut… view at source ↗
Figure 1
Figure 1. Three-ply Legal Argument Generation Scheme 3.2. Factor-Based Case Representation Cases are represented using a standardized set of 26 le￾gal factors pertinent to U.S. trade secret misappropriation law, derived from foundational work in legal AI [13, 30]. These factors encapsulate key factual aspects, such as cir￾cumstances surrounding disclosure, security measures im￾plemented, characteristics of the information, an… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 18 canonical work pages

  1. [1]

    Ariai, G

    F. Ariai, G. Demartini, Natural language processing for the legal domain: A survey of tasks, datasets, mod- els, and challenges, arXiv preprint arXiv:2410.21306 (2024)

    arXiv 2024

  2. [2]

    D. M. Katz, D. Hartung, L. Gerlach, A. Jana, M. J. Bom- marito II, Natural language processing in the legal domain, arXiv preprint arXiv:2302.12039 (2023)

    arXiv 2023

  3. [3]

    M. A. Gray, L. Zhang, K. D. Ashley, Generating case- based legal arguments with llms, in: Proceedings of the 4th ACM Computers and Law Symposium, 2025

    work page 2025

  4. [4]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosse- lut, E. Brunskill, et al., On the opportunities and risks of foundation models, arXiv preprint arXiv:2108.07258 (2021)

    arXiv 2021

  5. [5]

    Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, P. Fung, Survey of hallucina- tion in natural language generation, ACM Computing Surveys 55 (2023) 1–38

    work page 2023

  6. [6]

    D. U. S. de la Osa, N. Remolina, Artificial intelli- gence at the bench: Legal and ethical challenges of in- forming—or misinforming—judicial decision-making through generative ai, Data & Policy 6 (2024) e59

    work page 2024

  7. [7]

    J. J. Avery, P. S. Abril, A. del Riego, Chatgpt, esq.: Recasting unauthorized practice of law in the era of generative ai, Yale JL & Tech. 26 (2023) 64

    work page 2023

  8. [8]

    N. Guha, J. Nyarko, D. Ho, C. Ré, A. Chilton, A. Chohlas-Wood, A. Peters, B. Waldon, D. Rockmore, D. Zambrano, et al., Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models, Advances in Neural Information Processing Systems 36 (2024)

    work page 2024

Show all 33 references
  1. [9]

    H. Li, Y. Chen, Q. Ai, Y. Wu, R. Zhang, Y. Liu, Lexe- val: A comprehensive chinese legal benchmark for evaluating large language models, arXiv preprint arXiv:2409.20288 (2024)

    2024 arXiv

  2. [10]

    Z. Fei, X. Shen, D. Zhu, F. Zhou, Z. Han, S. Zhang, K. Chen, Z. Shen, J. Ge, Lawbench: Benchmarking legal knowledge of large language models, arXiv preprint arXiv:2309.16289 (2023)

    2023 arXiv

  3. [11]

    Touvron, T

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., Llama: Open and efficient foundation language models, arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  4. [12]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023)

    2023 arXiv

  5. [13]

    K. D. Ashley, Modeling Legal Argument: Reasoning with Cases and Hypotheticals, The MIT Press, Cam- bridge, MA, 1990

    1990

  6. [14]

    Aleven, K

    V. Aleven, K. D. Ashley, Teaching case-based argu- mentation through a model and examples: Empirical evaluation of an intelligent learning environment, in: Artificial intelligence in education, volume 39, Cite- seer, 1997, pp. 87–94

    1997

  7. [15]

    E. L. Rissland, D. B. Skalak, Cabaret:rule interpretation in a hybrid architecture, International Journal of Man- machine Studies 34 (1991) 839–887

    1991

  8. [16]

    Brüninghaus, K

    S. Brüninghaus, K. D. Ashley, Predicting outcomes of case based legal arguments, in: Proceedings of the 9th International conference on Artificial Intelligence and Law, ACM, 2003, pp. 233–242

    2003

  9. [17]

    Grabmair, Predicting trade secret case outcomes us- ing argument schemes and learned quantitative value effect tradeoffs, in: Proceedings of the 16th ICAIL, 2017, pp

    M. Grabmair, Predicting trade secret case outcomes us- ing argument schemes and learned quantitative value effect tradeoffs, in: Proceedings of the 16th ICAIL, 2017, pp. 89–98

    2017

  10. [18]

    J. F. Horty, Modifying precedential constraint, Journal of Artificial Intelligence Law 30 (2021) 1–24

    2021

  11. [19]

    M. A. Gray, J. Savelka, W. M. Oliver, K. D. Ashley, Em- pirical legal analysis simplified: reducing complexity through automatic identification and evaluation of legally relevant factors, Philosophical Transactions of the Royal Society A 382 (2024) 20230155

    2024

  12. [20]

    Savelka, K

    J. Savelka, K. D. Ashley, The unreasonable effective- ness of large language models in zero-shot semantic annotation of legal texts, Frontiers in Artificial Intelli- gence 6 (2023) 1279794

    2023

  13. [21]

    Y. A. Yadkori, I. Kuzborskij, A. György, C. Szepesvári, To believe or not to believe your llm, arXiv preprint arXiv:2406.02543 (2024)

    2024 arXiv

  14. [22]

    Zhang, Y

    Y. Zhang, Y. Li, L. Cui, D. Cai, L. Liu, T. Fu, X. Huang, E. Zhao, Y. Zhang, Y. Chen, et al., Siren’s song in the ai ocean: a survey on hallucination in large language models, arXiv preprint arXiv:2309.01219 (2023)

    2023 arXiv

  15. [23]

    Dhuliawala, M

    S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, X. Li, A. Celikyilmaz, J. Weston, Chain-of-verification re- duces hallucination in large language models, arXiv preprint arXiv:2309.11495 (2023)

    2023 arXiv

  16. [24]

    T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le, et al., Freshllms: Refreshing large language models with search engine augmentation, arXiv preprint arXiv:2310.03214 (2023)

    2023 arXiv

  17. [25]

    Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp

    C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81

    2004

  18. [26]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine transla- tion, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  19. [27]

    Zhang, V

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  20. [28]

    B. Wen, J. Yao, S. Feng, C. Xu, Y. Tsvetkov, B. Howe, L. L. Wang, Know your limits: A survey of ab- stention in large language models, arXiv preprint arXiv:2407.18418 (2024)

    2024 arXiv

  21. [29]

    S. Feng, W. Shi, Y. Wang, W. Ding, V. Balachandran, Y. Tsvetkov, Don’t hallucinate, abstain: Identifying llm knowledge gaps via multi-llm collaboration, arXiv preprint arXiv:2402.00367 (2024)

    2024 arXiv

  22. [30]

    K. D. Ashley, Artificial intelligence and legal analytics: new tools for law practice in the digital age, Cambridge University Press, 2017

    2017

  23. [31]

    M. Gray, J. Savelka, W. Oliver, K. Ashley, Automatic identification and empirical analysis of legally rele- vant factors, in: Proceedings of the Nineteenth Inter- national Conference on Artificial Intelligence and Law, 2023, pp. 101–110. A. Prompt Structure for 3-Ply Argument ...

    2023

  24. [32]

    No common factor between the input current case and the TSC1/TSC2

    IMPORTANT:If there is no common factor be- tween the current case and the TSC1/TSC2, you need to say "No common factor between the input current case and the TSC1/TSC2" and stop generating any argument. 2.Construct a 3-Ply Argument: a) Plaintiff’s Argument:Present an argument ...

  25. [33]

    Use Provided Factors: Base your arguments on the factors outlined, ensuring logical consistency. Example Input and Output Example Current Case •F1 Disclosure-in-negotiations (D) •F4 Agreed-not-to-disclose (P) •F6 Security-measures (P) •F10 Secrets-disclosed-outsiders (D) •F12 ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.