Pith. sign in

REVIEW 5 major objections 5 minor 83 references

The paper claims that a systematically prompted 7-billion-parameter generalist model, aided by chunking and two transparent candidate-selection heuristics, outperforms a fine-tuned legal model on CUAD contract review by about 9% in human ev

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

On CUAD legal contracts, a prompt-engineered QWEN-2 pipeline with chunking and two answer-selection heuristics reportedly outperforms the fine-tuned DeBERTa-large baseline by about 9%, reaching claimed state-of-the-art performance.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A well-written prompt-engineering pipeline for legal QA with a new assembly of known pieces, but the headline 9% gain is undercut by an under-specified location prior that may overlap the evaluation set. the 5 major comments →

arxiv 2509.02241 v1 pith:XULEWEAC submitted 2025-09-02 cs.AI

LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents

Strahinja Klem , Noura Al Moubayed This is my paper
classification cs.AI
keywords prompt engineeringlong document question answeringlegal NLPCUADchunkingcandidate selection heuristicslarge language modelscontract review
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a general-purpose large language model, prompted systematically and paired with two simple heuristics, can answer clause-level questions about long legal contracts as well as or better than a task-specific fine-tuned model. The authors work on the CUAD contract-review benchmark, where documents are far longer than QWEN-2's context window, so they split documents into chunks, add overlapping copies to reconnect cut-off context, and ask the model to answer per chunk. A location prior and an inverse-cardinality weighting scheme then choose among the many candidate answers. Against DeBERTa-large, human evaluation shows about 9% more correct answers per question, and the broader claim is that structured prompt engineering is a cheap, transparent, updateable alternative to expensive fine-tuning in law.

Core claim

The central claim is that the combination of content-agnostic chunking with overlap augmentation, an engineered prompt template, and the Distribution-Based Localisation (DBL) and Inverse Cardinality Weighting (ICW) heuristics lets QWEN-2 (7B) match or exceed the extractive fine-tuned DeBERTa-large on the CUAD legal question-answering dataset without any fine-tuning. The paper's headline comparison is human evaluation over DeBERTa's 102-document test set, where the generative pipeline shows an average increase in correctness of about 9% per question and 250 more correct answers total; when true negatives are excluded, the per-question gain is about 6%. A 2x2 factorial comparison supports prom

What carries the argument

The mechanism is a three-stage pipeline. First, chunking and augmentation turn a long contract into uniform-size pieces, with an overlap step that creates a new chunk from every pair of sequential chunks so split context is reconnected. Second, a finalised prompt template asks the model to quote the answer word-for-word or respond only with 'Does not exist'. Third, two heuristics select among candidates: DBL builds a normalized 100-bin prior over answer locations from labeled documents and up-weights chunks in likely regions, while ICW embeds candidate answers with GritLM, clusters them with DBSCAN, and weights answers inversely to cluster size on the assumption that correct answers are the

Load-bearing premise

The method assumes legal documents keep roughly the same proportional layout as they grow, so a location prior learned from labeled documents transfers to new ones; it also matters that the same labeled set may be used to build that prior and to run the final evaluation, which would make the gain an in-sample fit rather than a prediction.

What would settle it

Run the same pipeline on a set of contracts that were never used to build the DBL location distribution and never used to pick prompts, then compare human-evaluated correctness with DeBERTa-large on the same 41 questions. If the per-question gain disappears, the 9% came from the prior seeing the evaluation documents; if the gain persists, the prompting and chunking transfer on their own.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the method holds, legal question answering can be improved without retraining a model: updating the prompt or the heuristics replaces model fine-tuning.
  • The full pipeline runs on a local, consumer-grade setup, since the 7B QWEN-2 variant is used via a local API, making the approach reproducible without specialised hardware.
  • Because the candidate-selection heuristics are explicit rules, a reviewer can inspect why an answer was chosen, which is a step toward accountability in legal AI.
  • Automatic metrics such as ROUGE, METEOR, and cosine similarity rank the generative outputs differently from human judgement on short legal answers, indicating that better evaluation metrics are needed for this setting.
  • The factorial comparison indicates that overlap augmentation helps only when the prompt is sufficiently explicit, so the two design choices interact rather than contributing independently.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • For unseen contracts in a different jurisdiction or drafting style, the DBL location prior may not transfer; a clean test would hold out entire document types and refit the prior on a separate set.
  • The paper does not state which split feeds DBL. If the location distribution is built from the same 102 documents used for the human evaluation, the 9% gain could partly reflect an in-sample fit rather than a true prediction; a replication on a disjoint document set would settle this.
  • Inverse-cardinality weighting is a general candidate-selection principle for generative QA over long documents: rare embedding clusters are trusted over common ones, and it could transfer to non-legal domains where a model over-generates variants of a single true answer.
  • Because prompt selection used only short documents and layperson-friendly questions, the reported performance on long, clause-typed questions likely comes more from the heuristics than from the prompt; isolating each contribution would clarify what the prompt itself buys.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a structured prompting methodology for long legal document question answering on CUAD, using QWEN-2-7B as a generalist model. Long documents are split into fixed-size chunks with a reduplication/augmentation step, each chunk is prompted with an engineered prompt, and candidate answers are selected via two heuristics: Distribution-Based Localisation (DBL), which builds a location prior from labelled documents, and Inverse Cardinality Weighting (ICW), which clusters candidate answers by embedding similarity. The authors report that their approach outperforms the DeBERTa-large baseline from the original CUAD paper by about 9% average correctness per question in human evaluation, claiming state-of-the-art performance on CUAD. They also compare prompt variants in a 2x2 factorial design using automatic metrics. The paper argues that structured prompt engineering can be a transparent, scalable alternative to fine-tuning for legal information retrieval.

Significance. If the central claim were fully supported, the paper would make a useful contribution: it demonstrates a zero-fine-tuning pipeline for long legal documents using a locally runnable 7B model, with transparent heuristics that partially address LLM black-box concerns. The chunking-with-augmentation idea and the two candidate-selection heuristics are simple and testable, and the explicit call to action on the inadequacy of current automatic metrics for generative QA is timely. The use of a reproducible open model and a public dataset is a strength. However, the significance is currently limited by the evaluation weaknesses described below: the main comparison rests on an undocumented human evaluation, the DBL prior may be fitted with evaluation labels, and the SOTA claim is asserted against a single baseline.

major comments (5)
  1. [§3.4, Figure 5, Figure 6] The DBL location prior is built from 'the set of documents D from the test set' with 'associated answers to questions A' (Section 3.4). Section 3.3 defines that test set as 50 documents of at most 1000 words restricted to five question categories (document name, parties, agreement date, effective date, expiration date). The final headline results, Figures 5-6, are then reported on 'DeBERTa's 102 document test set'. The paper never states whether the 50 documents used to fit L_d are a subset of the 102 evaluation documents. If they overlap, the DBL weights are derived from the very answer locations being scored, making the candidate-selection step partially supervised by ground truth. This would invalidate the reported 9% improvement as a fair held-out comparison. This is load-bearing and must be clarified, and the experiments re-run with a strict train/test split if overlap exists.
  2. [§3.4, Figures 5-6] Independently of the split question, DBL is fit only on the five layperson-evaluable question categories, yet Figures 5 and 6 report results across all 41 CUAD questions. For the remaining 36 question types, the location prior L_d is undefined. The paper does not explain how candidate selection operates for those questions, nor whether their scores contribute to the headline 'about 9% per question' and '250 correct answers' figures. Without this specification, the main result is not reproducible and the DBL component's scope is unclear.
  3. [§4, Figures 5-6] The primary comparison against DeBERTa-large relies entirely on human evaluation, but no protocol is provided. The paper does not report the number or expertise of annotators, the instructions given, whether annotations were independent, inter-annotator agreement, adjudication for disagreements, or any confidence intervals. The statements 'average increase in correctness of about 9% per question' and 'absolute jump of 250 correct answers total' are thus unverifiable. Because the automatic metrics are explicitly deemed inadequate, the human evaluation is the sole support for the central claim and must be reported with full methodological detail and per-question numbers.
  4. [Table 4] The automatic metric thresholds (ROUGE 0.60, METEOR 0.68, cosine 0.79) are 'extrapolated' from the ParaQA paraphrase dataset and applied to CUAD legal QA without any validation on legal text. Table 4, which reports the factorial comparison of prompt variants and augmentation, depends entirely on these thresholds. The transferability of paraphrase-oriented thresholds to short legal answers is not self-evident and should be justified or validated against a labeled CUAD subset. As presented, Table 4's absolute and percentage scores have unknown reliability.
  5. [Abstract, §4] The claim of reaching 'state-of-the-art performance' is unsupported as stated. The only comparative baseline is DeBERTa-large from the original CUAD paper. No comparison is made to other published CUAD results, to more recent legal-domain models, or to other long-document QA approaches. The authors should either qualify this claim to 'better than the original CUAD baseline' or provide a systematic comparison with existing SOTA methods on the same test set.
minor comments (5)
  1. [Figures 5-6] The figures lack axis labels and error bars, and the definition of 'correctness' used to compute per-question percentages is not stated in the caption or text. Adding a table of raw correct/total counts per question would substantially aid verification.
  2. [§3.2] The choice of chunk size 1000 is reported as 'empirically found', but no sensitivity analysis is given. Since chunk size is a free parameter that interacts with augmentation and downstream candidate selection, reporting the tested values and resulting performance would strengthen the methodology.
  3. [§3.3] The split terminology is confusing: the paper calls the prompt-development set the 'test set' and the held-out set the 'verification set'. This is opposite to common usage and contributes to the ambiguity about whether DBL is fit on evaluation data. Consider renaming to 'development' and 'evaluation' sets.
  4. [Abstract and §1] Minor typographical issues: 'tacking' should be 'tackling' in the abstract; 'uninterruptible' in §2.3 should likely be 'uninterpretable' or 'continuous'. These do not affect content but should be corrected.
  5. [Table 4] The factorial design is only reported as absolute and percentage summary scores; no interaction term or statistical test is given. Given the paper's emphasis on the interaction between prompting complexity and augmentation, a more detailed breakdown (e.g., per-question or per-document) would be informative.

Circularity Check

0 steps flagged

No significant circularity: the DBL location prior is a supervised component defined on a separately described 50-document test set, not shown to be the 102-document evaluation set.

full rationale

The derivation chain is self-contained. QWEN-2 produces candidate answers from prompted, chunked documents; ICW weights candidates by cluster size; DBL constructs a location prior from labeled documents. The final SOTA comparison is against DeBERTa on CUAD's 102-document test set. The only potentially circular point is DBL's use of 'the test set' to build Lf. However, Section 3.3 defines that test set as 50 documents of at most 1000 words restricted to five layperson-evaluable question types, whereas Section 4 evaluates on DeBERTa's 102-document test set. The paper never identifies these two sets as identical, and the text is consistent with a disjoint split in which DBL is fitted on prompt-development documents and evaluated on a held-out official test set. No equation or definition forces the DBL fitting labels to be the evaluation labels. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of known results. The undisclosed relationship between the DBL fitting set and the evaluation set is a reproducibility/leakage risk worth flagging, but it is not an exhibited circular reduction under the standard that a prediction must be equivalent to its input by construction. Therefore the appropriate circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central claim depends on several domain assumptions and tuned numeric choices: a fixed chunk size, a location-prior length, automatic thresholds, and unstated clustering parameters. None of these are derived from theory; they are tuned on data, and the paper does not demonstrate that the choices transfer across splits.

free parameters (4)
  • chunk size = 1000
    Chosen empirically; affects reduplication overhead and model accuracy (Section 3.2).
  • DBL partition length L = 100
    Chosen empirically to balance granularity against overhead (Section 3.4).
  • automatic metric thresholds = ROUGE 0.60, METEOR 0.68, cosine 0.79
    Extrapolated from ParaQA paraphrase data, then applied to legal QA for the Table 4 factorial analysis (Section 3.5).
  • DBSCAN hyperparameters (eps, min_samples) = not reported
    Clustering groups and noise points depend on these values; they are not specified in the paper (Section 3.4).
axioms (5)
  • domain assumption Legal documents have proportional structural regularity: answer locations scale linearly with document length.
    Stated in Section 3.4 as the basis for DBL; if false, the location prior mis-weights chunks.
  • domain assumption Correct answers from the model appear less often than incorrect ones, so inverse cardinality weighting favors correct answers.
    Stated in Section 3.4 as the empirical basis for ICW; no evidence is provided for this ratio.
  • domain assumption Similarity thresholds from ParaQA transfer to legal contract QA correctness judgments.
    Used in Section 3.5 to set automatic decision boundaries; no legal-domain validation is provided.
  • ad hoc to paper Human evaluation is a valid ground truth for the main comparison, despite no protocol details.
    Section 4 relies on human evaluation for the 9% claim but gives no annotator count, instructions, blinding, or agreement metrics.
  • domain assumption The 50 short documents and 5 question types in the prompt-selection test set are representative enough to optimize prompts for the full CUAD task.
    Section 3.3 restricts prompt selection to layperson-evaluable questions and documents under 1000 words; this may not reflect hard clauses in longer contracts.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents." pith.science (2026). https://pith.science/paper/XULEWEAC

@misc{pith2026250902241,
  author       = {Pith},
  title        = {Pith review of: LLMs for LLMs: A Structured Prompting Methodology for Long Legal Documents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XULEWEAC}},
  note         = {Machine review of arXiv:2509.02241}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The rise of Large Language Models (LLMs) has had a profoundly transformative effect on a number of fields and domains. However, their uptake in Law has proven more challenging due to the important issues of reliability and transparency. In this study, we present a structured prompting methodology as a viable alternative to the often expensive fine-tuning, with the capability of tacking long legal documents from the CUAD dataset on the task of information retrieval. Each document is first split into chunks via a system of chunking and augmentation, addressing the long document problem. Then, alongside an engineered prompt, the input is fed into QWEN-2 to produce a set of answers for each question. Finally, we tackle the resulting candidate selection problem with the introduction of the Distribution-based Localisation and Inverse Cardinality Weighting heuristics. This approach leverages a general purpose model to promote long term scalability, prompt engineering to increase reliability and the two heuristic strategies to reduce the impact of the black box effect. Whilst our model performs up to 9\% better than the previously presented method, reaching state-of-the-art performance, it also highlights the limiting factor of current automatic evaluation metrics for question answering, serving as a call to action for future research. However, the chief aim of this work is to underscore the potential of structured prompt engineering as a useful, yet under-explored, tool in ensuring accountability and responsibility of AI in the legal domain, and beyond.

Figures

Figures reproduced from arXiv: 2509.02241 by Noura Al Moubayed, Strahinja Klem.

Figure 1
Figure 1. Figure 1: A unified view of our approach with the solutions to long document, information retrieval and candidate selection problems, coloured in red, yellow and blue respectively. 3.1 CUAD and QWEN-2 The Contract Understanding Atticus Dataset [29] is an American legal dataset, curated by the non-profit legal organisation "The Atticus Project", with the primary task of information retrieval in the form of contract r… view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of document lengths in CUAD. 3.2 Chunking and Augmentation The long document problem may be solved by targeting the length of either the context window or the input text. As discussed in Section 2, approaches such as Rope and Yarn address the former of the two, usually with a significant trade-off in accuracy and especially outside of the original training context, which is not appropriate for… view at source ↗
Figure 3
Figure 3. Figure 3: The augmentation (reduplication) step remedies the context splitting problem caused by having cut off points in chunking. This is done by creating a new chunk for every pair of existing sequential chunks such that the new ones contain the latter part of the first chunk and the former part of the second chunk. For example, A/B contains the second half of A, and the first half of B, thereby reconnecting the … view at source ↗
Figure 4
Figure 4. Figure 4: Methodology for creating the base template. Paraphrases are created by taking different combinations from the categories, and are trialled on the test set to evaluate performance. In our experimentation, we found "Identify the part of the question that corresponds to [Q]" to perform the best. 3.4 Candidate Selection Heuristics After the process of inference is complete, each document will have an answer as… view at source ↗
Figure 5
Figure 5. Figure 5: We compare the effectiveness of the two approaches across the 41 questions in the dataset. We find that QWEN-2, with few exceptions, matches or surpasses DeBERTa￾large on almost all questions. This represents an average increase in correctness of about 9% per question, and absolute jump of 250 correct answers total [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: With the same criteria as [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

83 extracted references · 40 canonical work pages · 8 internal anchors

  1. [1]

    arXiv preprint arXiv:2401.14423 (2024)

    Amatriain, X.: Prompt design and engineering: Introduction and advanced methods. arXiv preprint arXiv:2401.14423 (2024)

    Pith/arXiv arXiv 2024

  2. [2]

    Nature Reviews Physics5(5), 277–280 (2023)

    Birhane, A., Kasirzadeh, A., Leslie, D., Wachter, S.: Science in the age of large language models. Nature Reviews Physics5(5), 277–280 (2023)

    2023

  3. [3]

    Advances in neural information processing systems33, 1877–1901 (2020) 16 Klem et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Nee- lakantan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems33, 1877–1901 (2020) 16 Klem et al

    1901

  4. [4]

    think like a lawyer

    Burton, K.: " think like a lawyer" using a legal reasoning grid and criterion- referenced assessment rubric on irac (issue, rule, application, conclusion). Journal of Learning Design10(2), 57–68 (2017)

    2017

  5. [5]

    In: International Conference on Business Process Modeling, Development and Support

    Busch, K., Rochlitzer, A., Sola, D., Leopold, H.: Just tell me: Prompt engineering in business process management. In: International Conference on Business Process Modeling, Development and Support. pp. 3–11. Springer (2023)

    work page 2023

  6. [6]

    Amicus Curiae35, 28 (2001)

    Butt, P.: Legalese versus plain language. Amicus Curiae35, 28 (2001)

    work page 2001

  7. [7]

    arXiv preprint arXiv:2407.06204 (2024)

    Cai, W., Jiang, J., Wang, F., Tang, J., Kim, S., Huang, J.: A survey on mixture of experts. arXiv preprint arXiv:2407.06204 (2024)

    Pith/arXiv arXiv 2024

  8. [8]

    Metaverse Basic and Applied Research2, 33–33 (2023)

    Cano, C.A.G., Castillo, V.S., Gallego, T.A.C.: Unveiling the thematic landscape of generative pre-trained transformer (gpt) through bibliometric analysis. Metaverse Basic and Applied Research2, 33–33 (2023)

    work page 2023

  9. [9]

    IEEE Transactions on Knowledge and Data Engineering (2024)

    Cao, H., Tan, C., Gao, Z., Xu, Y., Chen, G., Heng, P.A., Li, S.Z.: A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering (2024)

    work page 2024

  10. [10]

    arXiv preprint arXiv:2404.01322 (2024)

    Carolan, K., Fennelly, L., Smeaton, A.F.: A review of multi-modal large language and vision models. arXiv preprint arXiv:2404.01322 (2024)

    Pith/arXiv arXiv 2024

  11. [11]

    arXiv preprint arXiv:2010.02559 (2020)

    Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Aletras, N., Androutsopoulos, I.: Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559 (2020)

    Pith/arXiv arXiv 2010

  12. [12]

    arXiv preprint arXiv:2110.00976 (2021)

    Chalkidis, I., Jana, A., Hartung, D., Bommarito, M., Androutsopoulos, I., Katz, D.M., Aletras, N.: Lexglue: A benchmark dataset for legal language understanding in english. arXiv preprint arXiv:2110.00976 (2021)

    Pith/arXiv arXiv 2021

  13. [13]

    arXiv preprint arXiv:2310.00785 (2023)

    Chang, Y., Lo, K., Goyal, T., Iyyer, M.: Booookscore: A systematic exploration of book-length summarization in the era of llms. arXiv preprint arXiv:2310.00785 (2023)

    Pith/arXiv arXiv 2023

  14. [14]

    Sublanguage: Studies of language in restricted semantic domains pp

    Charrow, V.R., Crandall, J.A., Charrow, R.P.: Characteristics and functions of legal language. Sublanguage: Studies of language in restricted semantic domains pp. 175–190 (1982)

    work page 1982

  15. [15]

    arXiv preprint arXiv:2405.07468 (2024)

    Chen, X., Xiang, J., Lu, S., Liu, Y., He, M., Shi, D.: Evaluating large language models in medical applications: a survey. arXiv preprint arXiv:2405.07468 (2024)

    Pith/arXiv arXiv 2024

  16. [16]

    In: International Conference on Applications of Natural Language to Information Systems

    Clavié, B., Ciceu, A., Naylor, F., Soulié, G., Brightwell, T.: Large language models in the workplace: A case study on prompt engineering for job type classification. In: International Conference on Applications of Natural Language to Information Systems. pp. 3–17. Springer (2023)

    work page 2023

  17. [17]

    LegaLMFiT: Efficient Short Legal Text Classification with LSTM Language Model Pre-Training

    Clavié, B., Gheewala, A., Briton, P., Alphonsus, M., Laabiyad, R., Piccoli, F.: Legalmfit: Efficient short legal text classification with lstm language model pre- training. arXiv preprint arXiv:2109.00993 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  18. [18]

    The Cambridge Law Journal5(3), 366–370 (1935)

    Cohn, E.J.: Precedents in continental law. The Cambridge Law Journal5(3), 366–370 (1935). https://doi.org/10.1017/S0008197300124705

    work page doi:10.1017/s0008197300124705 1935

  19. [19]

    In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    Colombo, P., Pires, T., Boudiaf, M., de Melo, R.F.C.P., Hautreux, G., Malaboeuf, E., Charpentier, J., Culver, D., Desa, M.: Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain. In: The Thirty-eighth Annual Conference on Neural Information Processing Systems (2024)

    work page 2024

  20. [20]

    arXiv preprint arXiv:2403.03883 (2024)

    Colombo, P., Pires, T.P., Boudiaf, M., Culver, D., Melo, R., Corro, C., Martins, A.F., Esposito, F., Raposo, V.L., Morgado, S., et al.: Saullm-7b: A pioneering large language model for law. arXiv preprint arXiv:2403.03883 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    arXiv preprint arXiv:2306.16092 (2023) LLMs for LLMs 17

    Cui, J., Li, Z., Yan, Y., Chen, B., Yuan, L.: Chatlaw: Open-source legal large language model with integrated external knowledge bases. arXiv preprint arXiv:2306.16092 (2023) LLMs for LLMs 17

    Pith/arXiv arXiv 2023

  22. [22]

    In: Companion Proceedings of the 29th International Conference on Intelligent User Interfaces

    Desmond, M., Ashktorab, Z., Pan, Q., Dugan, C., Johnson, J.M.: Evalullm: Llm assisted evaluation of generative outputs. In: Companion Proceedings of the 29th International Conference on Intelligent User Interfaces. pp. 30–32 (2024)

    work page 2024

  23. [23]

    arXiv preprint arXiv:2302.12246 (2023)

    Diao, S., Wang, P., Lin, Y., Zhang, T.: Active prompting with chain-of-thought for large language models. arXiv preprint arXiv:2302.12246 (2023)

    Pith/arXiv arXiv 2023

  24. [24]

    Authorea Preprints (2023)

    Ekin, S.: Prompt engineering for chatgpt: a quick guide to techniques, tips, and best practices. Authorea Preprints (2023)

    work page 2023

  25. [25]

    In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD ’96)

    Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for dis- covering clusters in large spatial databases with noise. In: Proceedings of the 2nd International Conference on Knowledge Discovery and Data Mining (KDD ’96). pp. 226–231 (1996)

    work page 1996

  26. [26]

    arXiv preprint arXiv:2312.10997 (2023)

    Gao, Y., Xiong, Y., Gao, X., Jia, K., Pan, J., Bi, Y., Dai, Y., Sun, J., Wang, H.: Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    Pith/arXiv arXiv 2023

  27. [27]

    Annals of biomedical engineering51(12), 2629–2633 (2023)

    Giray, L.: Prompt engineering with chatgpt: a guide for academic writers. Annals of biomedical engineering51(12), 2629–2633 (2023)

    work page 2023

  28. [28]

    arXiv preprint arXiv:2209.06120 (2022)

    Guha, N., Ho, D.E., Nyarko, J., Ré, C.: Legalbench: Prototyping a collaborative benchmark for legal reasoning. arXiv preprint arXiv:2209.06120 (2022)

    Pith/arXiv arXiv 2022

  29. [29]

    arXiv preprint arXiv:2103.06268 (2021)

    Hendrycks, D., Burns, C., Chen, A., Ball, S.: Cuad: An expert-annotated nlp dataset for legal contract review. arXiv preprint arXiv:2103.06268 (2021)

    Pith/arXiv arXiv 2021

  30. [30]

    International Medical Education 2(3), 198–205 (2023)

    Heston, T.F., Khun, C.: Prompt engineering in medical education. International Medical Education 2(3), 198–205 (2023)

    work page 2023

  31. [31]

    In: European Conference on Information Retrieval

    Hou, Y., Zhang, J., Lin, Z., Lu, H., Xie, R., McAuley, J., Zhao, W.X.: Large language models are zero-shot rankers for recommender systems. In: European Conference on Information Retrieval. pp. 364–381. Springer (2024)

    work page 2024

  32. [32]

    arXiv preprint arXiv:2305.15062 (2023)

    Huang, Q., Tao, M., An, Z., Zhang, C., Jiang, C., Chen, Z., Wu, Z., Feng, Y.: Lawyer llama technical report. arXiv preprint arXiv:2305.15062 (2023)

    Pith/arXiv arXiv 2023

  33. [33]

    In: European semantic web conference

    Kacupaj, E., Banerjee, B., Singh, K., Lehmann, J.: Paraqa: a question answer- ing dataset with paraphrase responses for single-turn conversation. In: European semantic web conference. pp. 598–613. Springer (2021)

    work page 2021

  34. [34]

    arXiv preprint arXiv:2307.16883 (2023)

    Kamalloo, E., Jafari, A., Zhang, X., Thakur, N., Lin, J.: Hagrid: A human-llm collaborative dataset for generative information-seeking with attribution. arXiv preprint arXiv:2307.16883 (2023)

    Pith/arXiv arXiv 2023

  35. [35]

    In: JSAI International Symposium on Artificial Intelligence

    Kim, M.Y., Rabelo, J., Goebel, R., Yoshioka, M., Kano, Y., Satoh, K.: Coliee 2022 summary: Methods for legal document retrieval and entailment. In: JSAI International Symposium on Artificial Intelligence. pp. 51–67. Springer (2022)

    work page 2022

  36. [36]

    arXiv preprint arXiv:2312.03718 (2023)

    Lai, J., Gan, W., Wu, J., Qi, Z., Yu, P.S.: Large language models in law: A survey. arXiv preprint arXiv:2312.03718 (2023)

    Pith/arXiv arXiv 2023

  37. [37]

    Law, C.: The common law and civil law traditions (2016)

    work page 2016

  38. [38]

    A Benchmark for Lease Contract Review

    Leivaditi, S., Rossi, J., Kanoulas, E.: A benchmark for lease contract review. arXiv preprint arXiv:2010.10386 (2020)

    work page internal anchor Pith review Pith/arXiv arXiv 2010

  39. [39]

    arXiv preprint arXiv:2101.00190 (2021)

    Li, X.L., Liang, P.: Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 (2021)

    Pith/arXiv arXiv 2021

  40. [40]

    Advances in Neural Information Processing Systems36 (2024)

    Liu, J., Xia, C.S., Wang, Y., Zhang, L.: Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation. Advances in Neural Information Processing Systems36 (2024)

    work page 2024

  41. [41]

    Transactions of the Association for Computational Linguistics12, 157–173 (2024)

    Liu, N.F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., Liang, P.: Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics12, 157–173 (2024)

    2024

  42. [42]

    The Journal of Academic Librarianship49(4), 102720 (2023) 18 Klem et al

    Lo, L.S.: The clear path: A framework for enhancing information literacy through prompt engineering. The Journal of Academic Librarianship49(4), 102720 (2023) 18 Klem et al

    work page 2023

  43. [43]

    arXiv preprint arXiv:2202.07209 (2022)

    Locke, D., Zuccon, G.: Case law retrieval: problems, methods, challenges and evaluations in the last 20 years. arXiv preprint arXiv:2202.07209 (2022)

    Pith/arXiv arXiv 2022

  44. [44]

    In: Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval

    Ma, Y., Shao, Y., Wu, Y., Liu, Y., Zhang, R., Zhang, M., Ma, S.: Lecard: a legal case retrieval dataset for chinese law system. In: Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. pp. 2342–2348 (2021)

    work page 2021

  45. [45]

    ILDC for CJPE: Indian Legal Documents Corpus for Court Judgment Prediction and Explanation

    Malik, V., Sanjay, R., Nigam, S.K., Ghosh, K., Guha, S.K., Bhattacharya, A., Modi, A.: Ildc for cjpe: Indian legal documents corpus for court judgment prediction and explanation. arXiv preprint arXiv:2105.13562 (2021)

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  46. [46]

    In: International Conference on Artificial Intelligence in Education

    Moore, S., Tong, R., Singh, A., Liu, Z., Hu, X., Lu, Y., Liang, J., Cao, C., Khosravi, H., Denny, P., et al.: Empowering education with llms-the next-gen interface and content generation. In: International Conference on Artificial Intelligence in Education. pp. 32–37. Springer (2023)

    work page 2023

  47. [47]

    arXiv preprint arXiv:2402.09906 (2024)

    Muennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T., Singh, A., Kiela, D.: Generative representational instruction tuning. arXiv preprint arXiv:2402.09906 (2024)

    Pith/arXiv arXiv 2024

  48. [48]

    Southeast Europe Journal of Soft Computing12(1), 13–41 (2023)

    Muftić, F., Kadunić, M., Mušinbegović, A., Abd Almisreb, A.: Exploring medical breakthroughs: a systematic review of chatgpt applications in healthcare. Southeast Europe Journal of Soft Computing12(1), 13–41 (2023)

    work page 2023

  49. [49]

    A Brief Report on LawGPT 1.0: A Virtual Legal Assistant Based on GPT-3

    Nguyen, H.T.: A brief report on lawgpt 1.0: A virtual legal assistant based on gpt-3. arXiv preprint arXiv:2302.05729 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  50. [50]

    arXiv preprint arXiv:2306.02069 (2023)

    Niklaus, J., Matoshi, V., Stürmer, M., Chalkidis, I., Ho, D.E.: Multilegalpile: A 689gb multilingual legal corpus. arXiv preprint arXiv:2306.02069 (2023)

    Pith/arXiv arXiv 2023

  51. [51]

    Openlegaldata: Openlegaldata/awesome-legal-data: Collection of datasets for legal text processing, https://github.com/openlegaldata/awesome-legal-data

    work page

  52. [52]

    Behaviour & Information Technology pp

    Oppenlaender, J.: A taxonomy of prompt modifiers for text-to-image generation. Behaviour & Information Technology pp. 1–14 (2023)

    work page 2023

  53. [53]

    Pasquale, F.: A rule of persons, not machines: the limits of legal automation. Geo. Wash. L. Rev.87, 1 (2019)

    work page 2019

  54. [54]

    arXiv preprint arXiv:2309.00071 (2023)

    Peng, B., Quesnelle, J., Fan, H., Shippole, E.: Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071 (2023)

    Pith/arXiv arXiv 2023

  55. [55]

    In: Legal Knowledge and Information Systems, pp

    Ribary, M., Krause, P., Orban, M., Vaccari, E., Wood, T.: Prompt engineering and provision of context in domain specific use of gpt. In: Legal Knowledge and Information Systems, pp. 305–310. IOS Press (2023)

    work page 2023

  56. [56]

    arXiv preprint arXiv:2402.07927 (2024)

    Sahoo, P., Singh, A.K., Saha, S., Jain, V., Mondal, S., Chadha, A.: A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024)

    Pith/arXiv arXiv 2024

  57. [57]

    arXiv preprint arXiv:2305.11430 (2023)

    Santu, S.K.K., Feng, D.: Teler: A general taxonomy of llm prompts for benchmarking complex tasks. arXiv preprint arXiv:2305.11430 (2023)

    Pith/arXiv arXiv 2023

  58. [58]

    In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval

    van Schaik, T.A., Pugh, B.: A field guide to automatic evaluation of llm-generated summaries. In: Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2832–2836 (2024)

    work page 2024

  59. [59]

    School, C.L.: jurisdiction — law.cornell.edu.https://www.law.cornell.edu/wex/ jurisdiction (2022), [Accessed 01-05-2024]

    work page 2022

  60. [60]

    Legal Case Document Summarization: Extractive and Abstractive Methods and their Evaluation

    Shukla, A., Bhattacharya, P., Poddar, S., Mukherjee, R., Ghosh, K., Goyal, P., Ghosh, S.: Legal case document summarization: Extractive and abstractive methods and their evaluation. arXiv preprint arXiv:2210.07544 (2022)

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  61. [61]

    one country, two systems

    So, A.Y.: “one country, two systems” and hong kong-china national integration: A crisis-transformation perspective. Journal of Contemporary Asia41(1), 99–116 (2011) LLMs for LLMs 19

    work page 2011

  62. [62]

    Neurocomputing568, 127063 (2024)

    Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., Liu, Y.: Roformer: Enhanced transformer with rotary position embedding. Neurocomputing568, 127063 (2024)

    2024

  63. [63]

    arXiv preprint arXiv:2407.16252 (2024)

    Sun, J., Dai, C., Luo, Z., Chang, Y., Li, Y.: Lawluo: A chinese law firm co-run by llm agents. arXiv preprint arXiv:2407.16252 (2024)

    Pith/arXiv arXiv 2024

  64. [64]

    Nature medicine29(8), 1930–1940 (2023)

    Thirunavukarasu, A.J., Ting, D.S.J., Elangovan, K., Gutierrez, L., Tan, T.F., Ting, D.S.W.: Large language models in medicine. Nature medicine29(8), 1930–1940 (2023)

    1930

  65. [65]

    arXiv preprint arXiv:2212.02199 (2022)

    Trautmann, D., Petrova, A., Schilder, F.: Legal prompt engineering for multilingual legal judgement prediction. arXiv preprint arXiv:2212.02199 (2022)

    Pith/arXiv arXiv 2022

  66. [66]

    Meta-Radiology p

    Wang, J., Liu, Z., Zhao, L., Wu, Z., Ma, C., Yu, S., Dai, H., Yang, Q., Liu, Y., Zhang, S., et al.: Review of large vision models and visual prompt engineering. Meta-Radiology p. 100047 (2023)

    work page 2023

  67. [67]

    Prompt Engineering for Healthcare: Methodologies and Applications

    Wang, J., Shi, E., Yu, S., Wu, Z., Ma, C., Dai, H., Yang, Q., Kang, Y., Wu, J., Hu, H., et al.: Prompt engineering for healthcare: Methodologies and applications. arXiv preprint arXiv:2304.14670 (2023)

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  68. [68]

    arXiv preprint arXiv:2203.11171 (2022)

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D.: Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171 (2022)

    Pith/arXiv arXiv 2022

  69. [69]

    Advances in neural information processing systems35, 24824–24837 (2022)

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al.: Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems35, 24824–24837 (2022)

    2022

  70. [70]

    arXiv preprint arXiv:2302.11382 (2023)

    White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., Schmidt, D.C.: A prompt pattern catalog to enhance prompt engineering with chatgpt. arXiv preprint arXiv:2302.11382 (2023)

    Pith/arXiv arXiv 2023

  71. [71]

    arXiv preprint arXiv:2303.17564 (2023)

    Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., Mann, G.: Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564 (2023)

    Pith/arXiv arXiv 2023

  72. [72]

    CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain

    Xiao, C., Zhong, H., Guo, Z., Tu, C., Liu, Z., Sun, M., Zhang, T., Han, X., Hu, Z., Wang, H., et al.: Cail2019-scm: A dataset of similar case matching in legal domain. arXiv preprint arXiv:1911.08962 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1911

  73. [73]

    British Journal of Educational Technology55(1), 90–112 (2024)

    Yan, L., Sha, L., Zhao, L., Li, Y., Martinez-Maldonado, R., Chen, G., Li, X., Jin, Y., Gašević, D.: Practical and ethical challenges of large language models in education: A systematic scoping review. British Journal of Educational Technology55(1), 90–112 (2024)

    work page 2024

  74. [74]

    arXiv preprint arXiv:2306.13549 (2023)

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)

    Pith/arXiv arXiv 2023

  75. [75]

    arXiv preprint arXiv:2402.14531 (2024)

    Yin, Z., Wang, H., Horio, K., Kawahara, D., Sekine, S.: Should we respect llms? a cross-lingual study on the influence of prompt politeness on llm performance. arXiv preprint arXiv:2402.14531 (2024)

    Pith/arXiv arXiv 2024

  76. [76]

    arXiv preprint arXiv:2212.01326 (2022)

    Yu, F., Quartey, L., Schilder, F.: Legal prompting: Teaching a language model to think like a lawyer. arXiv preprint arXiv:2212.01326 (2022)

    Pith/arXiv arXiv 2022

  77. [77]

    arXiv preprint arXiv:2401.13601 (2024)

    Zhang, D., Yu, Y., Li, C., Dong, J., Su, D., Chu, C., Yu, D.: Mm-llms: Recent advances in multimodal large language models. arXiv preprint arXiv:2401.13601 (2024)

    Pith/arXiv arXiv 2024

  78. [78]

    CitaLaw: Enhancing LLM with Citations in Legal Domain

    Zhang, K., Yu, W., Dai, S., Xu, J.: Citalaw: Enhancing llm with citations in legal domain. arXiv preprint arXiv:2412.14556 (2024)

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  79. [79]

    Transactions of the Association for Computational Linguistics 11, 1114–1131 (2023) 20 Klem et al

    Zhang, X., Thakur, N., Ogundepo, O., Kamalloo, E., Alfonso-Hermelo, D., Li, X., Liu, Q., Rezagholizadeh, M., Lin, J.: Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics 11, 1114–1131 (2023) 20 Klem et al

    work page 2023

  80. [80]

    arXiv preprint arXiv:2210.03493 (2022)

    Zhang, Z., Zhang, A., Li, M., Smola, A.: Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 (2022)

    Pith/arXiv arXiv 2022

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.