Pith. sign in

REVIEW 4 major objections 7 minor 105 references

A retrieval-augmented generation pipeline for Indian criminal-law questions improves smaller AI models, degrades larger ones, and—under the study's rating protocol—produces answers rated above human reference answers on some queries.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 14:13 UTC pith:DLCDYOK3

load-bearing objection Useful new benchmark for Indian legal QA, but the headline claim that AI answers beat human reference answers rests on a comparison that was never actually run. the 4 major comments →

arxiv 2607.18825 v1 pith:DLCDYOK3 submitted 2026-07-21 cs.CL cs.AIcs.IR

AILQA: Evaluating AI-Driven Legal Question Answering Systems for the Indian Legal System

Shubham Kumar Nigam , Shubham Kumar Mishra , Noel Shallum , Kripabandhu Ghosh , Arnab Bhattacharya This is my paper
classification cs.CL cs.AIcs.IR
keywords Legal Question AnsweringRetrieval-Augmented GenerationIndian criminal lawLarge Language ModelsExpert legal evaluationAll India Bar ExaminationHallucinationSemantic similarity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a retrieval-augmented generation (RAG) pipeline—retrieving the three most relevant chunks from Indian criminal-law documents and feeding them to a large language model—can produce legally useful answers to everyday legal questions in India. Its central empirical claims are that RAG improves answer quality for smaller or older generative models, that it can degrade larger models by introducing noisy or irrelevant context, and that under the study's rating protocol some AI-generated answers received higher expert ratings than the human reference answers from a free legal-advice website. The authors treat the system as a decision-support prototype, not as a substitute for legal professionals, and they explicitly scope the findings to their 150-question dataset and rating criteria. A sympathetic reader would care because the claims, if right, point to a low-cost way to make legal information access more consistent and to a benchmark for future legal QA systems.

Core claim

The paper's central discovery is an asymmetry in how retrieval-augmented generation affects language models in the legal domain: adding retrieved context substantially improved smaller or older models (one model's All India Bar Examination accuracy rose from 45.72% to 51.69%) while larger models that already encode substantial legal knowledge tended to lose accuracy or expert ratings when given retrieved context, because noisy top-3 chunks can distract them or trigger hallucinated citations. A second, more provocative finding is that, under the study's evaluation protocol, some AI answers were rated higher than the reference human answers on a five-point scale that rewards legally accurate s

What carries the argument

The load-bearing mechanism is a controlled dense-retrieval RAG baseline: about 7,200 Indian criminal-law documents—statutes, judicial decisions, and legal articles—are split into 2,000-character chunks with a 250-character overlap; three embedding models each build a vector store; a query is embedded with the same model and the top three chunks by cosine similarity are prepended to a prompt that instructs the generative model to answer for India, be detailed, and cite sections or cases only if confident. The same models are also run without retrieval. The evaluation stack—lexical overlap, semantic similarity, expert Likert ratings, an LLM-based relevance auto-check, and the AIBE multiple-cho

Load-bearing premise

The entire comparison rests on an unblinded expert-rating protocol in which only the AI-generated answers were scored on a five-point scale that rewards correct legal detail; if the human reference answers were scored on the same scale by raters blind to source, the reported superiority of some AI answers could vanish.

What would settle it

Re-run the expert evaluation on the same 150 questions with three changes: (1) score the reference answers from the advice website on the identical five-point scale, (2) blind raters to whether an answer is AI-generated and whether RAG was used, and (3) report inter-rater agreement (e.g., Krippendorff's alpha). If the reference answers then receive equal or higher mean ratings, or if rater agreement is near zero, the paper's central comparative claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RAG becomes a practical, fine-tuning-free lever: older and smaller models gain several accuracy points on legal QA and the bar exam when given retrieved context.
  • Model size is not a monotonic advantage in legal QA: for large models, retrieval quality—chunking, reranking, and filtering—can matter more than raw capacity.
  • A simple dense-retrieval pipeline can clear the 40% All India Bar Examination threshold, with the best configuration at about 71% accuracy, suggesting use in exam preparation and legal research support.
  • Hallucinated legal citations are a documented failure mode without context; retrieved context reduces but does not eliminate them, and irrelevant retrieved context can create new hallucinations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the 'AI beats human answers' claim would be to score the original reference answers on the same five-point scale; the paper never does this, so the comparison is asymmetric.
  • If the size-dependent RAG effect is real, a testable prediction follows: adding a reranker or legal-metadata filtering to the retrieval stage should shrink the performance drop of the larger models and widen the gain of the smaller ones.
  • The bar-exam result suggests an immediate low-stakes use case—automated legal-education tutoring and exam practice—where imperfect answers are still useful, and where hallucinated citations can be audited by a human.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper introduces AILQA, a RAG-based question-answering system for Indian criminal law. It assembles a corpus of judgments, statutes, and articles; builds vector stores with three embedding models; and evaluates six generative LLMs with and without retrieval on 150 VidhiKarya legal queries and on the AIBE multiple-choice benchmark (1,158 questions). Evaluation uses lexical (ROUGE/BLEU), semantic (MPNET), GPT-4-as-judge, and human expert Likert ratings. The main reported findings are that RAG improves smaller/older models (e.g., Llama2-70B AIBE accuracy from 45.72% to 51.69%) while degrading larger ones, and that under the study's protocol some AI-generated answers received higher ratings than the available reference answers. The paper also provides a dataset and code release and an extensive limitations section.

Significance. If substantiated, the paper would provide a useful benchmark and reproducible baseline for Indian legal QA, with an externally anchored AIBE evaluation and a detailed comparison of several off-the-shelf LLMs. Strengths include the use of a public AIBE dataset, the public release of code and data, and a candid limitations section. However, the central comparative claim against human reference answers is not directly measured, and one of the RAG-effect descriptions contradicts the tabulated results. The AIBE results and the resource are valuable, but the headline conclusions as currently stated are not supported by the evidence presented.

major comments (4)
  1. [Abstract; Section 8; Section 5.3] The abstract claims that 'some AI-generated responses received higher ratings than the available reference answers,' and Section 8 states that 'AI-generated answers often surpass the quality of responses provided by human legal experts.' This is an inference, not an empirical result: in Section 5.3, evaluators rated only generated responses on the 5-point Likert scale; the VidhiKarya reference answers were provided as context but never rated on that scale. A rating of 5 is an absolute quality criterion, not a pairwise comparison. Moreover, the scale rewards statutory citations and detail, which the prompts in Table 3 explicitly request, while the reference answers are short and often lack citations (e.g., Figure 1, Table 10). The evaluation was also unblinded (Section 9). To support the headline, the authors must either run a blinded paired evaluation in which reference answers are rated
  2. [Section 6.2 vs Table 5] Section 6.2 states that for Mixtral-8x7B 'the histogram indicates an overall improvement in performance with RAG across all question types.' However, Table 5 reports a mean expert rating of 4.59 without RAG and 4.02 with Mxbai-based RAG for Mixtral-8x7B on Test Set 2 — a decline. The auto-evaluation (Table 6) shows only a small improvement (71 vs 73 questions with score 1). The qualitative narrative in Section 6.2 is thus inconsistent with the quantitative results. This contradiction must be resolved: either Table 5/Figure 3 is mislabeled or the descriptive text is incorrect.
  3. [Abstract; Section 6.1.4] The abstract's broad claim that the findings 'underscore the effectiveness of the Retrieval-Augmented Generation (RAG) paradigm in improving answer quality, particularly in complex legal domains' is not supported by the expert ratings. In Table 5, RAG degrades Llama3-70B (4.43 to 3.37) and Mixtral-8x7B (4.59 to 4.02); in Table 6, Llama3-70B with RAG scores 75 vs 79 without. Only GPT-3.5 Turbo improves slightly (3.43 to 3.55), and AIBE results help smaller models such as Llama2-70B (45.72% to 51.69%). The paper's own Section 6.1.4 concludes 'smaller models benefit from RAG, while larger models may hallucinate with added context.' The abstract should reflect this conditional, model-dependent outcome rather than a general RAG benefit.
  4. [Section 4.2 vs Section 9] There is an internal inconsistency in the reported chunk size: Section 4.2 states the CharacterTextSplitter uses 2000-character chunks with a 250-character overlap, while Section 9 (Limitations) attributes Flan-UL2's poor performance to 'restricting our analysis to only 1000 characters with a 250-character overlap.' These descriptions are contradictory and directly affect reproducibility. Please clarify which chunk size was used, and whether it varied across models or datasets.
minor comments (7)
  1. [Table 9] Model name 'Mixtral 7x8B' should be 'Mixtral-8x7B' for consistency with the rest of the paper.
  2. [Figure 3 caption] The caption uses inconsistent model names ('LLAMA-70b', 'MIXTRAL-8x7b'); please make them uniform.
  3. [Section 6.1.3] The text refers to 'GPT-3 variants,' but the experiments use GPT-3.5 Turbo; please use consistent model naming.
  4. [Section 4.3.3] Minor grammar: 'stored with it's metadata' should be 'stored with its metadata.'
  5. [Section 1] The claim to 'pioneer' RAG for the Indian legal system is strong; consider softening given existing legal RAG work (e.g., CBR-RAG) and the paper's own related-work section.
  6. [Tables 7-8] The p-values are not corrected for multiple comparisons; this should be noted or corrected if multiple pairwise tests are interpreted jointly.
  7. [Section 6.2] The histograms in Figure 3 are difficult to compare across models; overlaying or annotating mean scores would improve clarity.

Circularity Check

0 steps flagged

No load-bearing circularity: the claims are empirical evaluations against external benchmarks and disclosed protocols; the reference-score concern is a validity/design issue, not a definitional reduction.

full rationale

The paper contains no derivation chain that reduces a predicted quantity to a fitted input. Its central empirical claims are measurements: RAG improves Llama2-70B AIBE accuracy from 45.72% to 51.69% on an externally published 1,158-question benchmark (Tiwari et al., 2024); MPNET semantic-similarity scores and GPT-4 auto-evaluation scores are computed against ground-truth answers; and expert Likert ratings are collected under a disclosed protocol. No parameter is fitted to a subset of data and then re-reported as a prediction of a closely related quantity. The AIBE result is an external, independently published benchmark and is therefore self-contained evidence for the RAG claim. The related-work self-citations (Malik et al., Nigam et al., Vats et al.) are contextual and are not used to justify any uniqueness theorem, ansatz, or load-bearing premise. The only plausible concern is the abstract's statement that some AI responses 'received higher ratings than the available reference answers': reference answers were not themselves scored on the same Likert scale, so this is an interpretive comparison rather than a direct measurement. However, the paper itself discloses the unblinded evaluation and the scale's wording in Section 5.3, qualifies the claim as specific to 'the study's evaluation protocol' and 'the evaluated dataset and rating criteria,' and Section 9 acknowledges that lack of blinding 'may have introduced expectation bias into some assessments.' Whether the reference answers are commensurable is a validity/correctness issue, not a circularity in which an output is equivalent to an input by construction. Accordingly, no circular step meeting the quoted-evidence standard is present.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper's contribution is an evaluation, so its load-bearing inputs are not physical constants but measurement choices: which public corpus serves as ground truth, who rates the answers, and which design constants (top-k, chunk size, prompts, rating scale) are set by hand. Four hand-set design parameters and five domain assumptions carry the central claims. The paper discloses the rater-personnel weakness in Section 9, but it does not quantify the effect of any of these choices on the model rankings, and the Section 4.2 vs 9 chunk-size contradiction means even the described pipeline is uncertain.

free parameters (4)
  • Top-k retrieved chunks = k = 3
    The number of chunks fed to the generator as context was fixed at top-3 cosine-similarity chunks (Section 4.4) with no ablation or justification; this choice directly influences which model wins by determining context quality.
  • Chunk size and overlap = 2000 characters / 250 overlap (Section 4.2); stated as 1000 characters / 250 overlap in Section 9
    Fixed-size character chunking can separate a statutory rule from its exceptions, and the two sections state different values, so the reported results may not correspond to the described configuration.
  • Prompt templates = Six model-specific prompts (Table 3)
    Prompts were iteratively refined by manually evaluating responses to a random set of 10 questions (Section 4.5), tuning generator behavior against a sample that may have been drawn from the test distribution.
  • Likert rating criteria = 1-5 scale rewarding 'appropriate statutory provisions or precedents' and useful additional detail
    The author-designed scale (Section 5.3) structurally rewards detailed answers — the property that distinguishes AI outputs from the short reference answers — and was applied by unblinded raters.
axioms (5)
  • domain assumption VidhiKarya free-legal-advice answers are valid ground truth for legal QA quality
    Test Sets 1 and 2 use expert responses from a public legal-advice website as reference answers (Section 3.3); the inclusion criteria for questions and the verification standard for the 'expert' answers are not described.
  • domain assumption AIBE dataset answers (Tiwari et al., 2024) are correct, and recall-based MCQ accuracy is a meaningful legal-QA capability measure
    Section 3.3.1 states AIBE questions 'primarily test recall abilities'; the paper relies on the external dataset's manual verification without independent checking and uses its accuracy as a headline number.
  • domain assumption Consensus ratings of three unblinded law students measure legal answer quality
    Sections 5.3 and 9: raters are senior undergraduate law students, not practicing lawyers; evaluation was not blinded (model identity and RAG status disclosed); no inter-rater reliability coefficient was computed.
  • domain assumption MPNET cosine similarity is a valid measure of legal answer quality
    Section 5.2 uses MPNET similarity to the reference answer as a quality metric and as the basis for all pairwise p-value tests; a semantically similar paraphrase of a wrong answer would score well, and the paper does not validate MPNET against the expert ratings.
  • standard math Unspecified pairwise significance test with a 0.05 threshold and no multiple-comparison correction
    Section 5.4 does not name the test; about 28 pairwise comparisons use p-values without correction, so several 'significant' differences are expected by chance.

pith-pipeline@v1.3.0-alltime-deepseek · 20463 in / 26634 out tokens · 218259 ms · 2026-08-01T14:13:36.464558+00:00 · methodology

0 comments
read the original abstract

This comprehensive study introduces an advanced Artificial Intelligence for Indian Legal Question Answering (AILQA) system tailored to the Indian legal context. AILQA leverages a variety of embedding and generative models, including recent Large Language Models (LLMs), to address the unique challenges posed by the intricate and diverse nature of Indian legal texts and to enhance the accuracy and reliability of responses to legal questions. We conducted rigorous evaluations using both lexical and semantic metrics, enriched by expert legal feedback, to ensure relevance and accuracy. Our findings underscore the effectiveness of the Retrieval-Augmented Generation (RAG) paradigm in improving answer quality, particularly in complex legal domains. Additionally, we assessed performance on standardized tests such as the All India Bar Examination (AIBE), thereby providing a robust benchmark for practical applications. Under the study's evaluation protocol, some AI-generated responses received higher ratings than the available reference answers, particularly when they contained accurate and relevant supporting details. This finding is specific to the evaluated dataset and rating criteria and should not be interpreted as evidence that the models generally outperform qualified legal professionals. We also discuss the challenges encountered, such as the need for precise context and the risks of model hallucination, and propose directions for future research to further refine AI capabilities in the legal field. This study aims to pave the way for enhanced legal decision-support systems, making them more accessible and effective for legal professionals and the public alike.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

105 extracted references · 2 canonical work pages

  1. [1]

    Computer Science Review , volume=

    A survey on legal question--answering systems , author=. Computer Science Review , volume=. 2023 , publisher=

    2023

  2. [2]

    Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence , articleno =

    Huang, Weiyi and Jiang, Jiahao and Qu, Qiang and Yang, Min , title =. Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence , articleno =. 2021 , isbn =

    2021

  3. [3]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  4. [4]

    arXiv preprint arXiv:1808.07036 , year=

    QuAC: Question answering in context , author=. arXiv preprint arXiv:1808.07036 , year=

    Pith/arXiv arXiv

  5. [5]

    International Journal of Research and Reviews in Information Sciences (IJRRIS) , volume=

    The question answering systems: A survey , author=. International Journal of Research and Reviews in Information Sciences (IJRRIS) , volume=

  6. [6]

    arXiv preprint arXiv:1502.05698 , year=

    Towards ai-complete question answering: A set of prerequisite toy tasks , author=. arXiv preprint arXiv:1502.05698 , year=

    Pith/arXiv arXiv

  7. [7]

    arXiv preprint arXiv:1810.04805 , year=

    Bert: Pre-training of deep bidirectional transformers for language understanding , author=. arXiv preprint arXiv:1810.04805 , year=

    Pith/arXiv arXiv

  8. [8]

    Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval , pages=

    BERT with history answer embedding for conversational question answering , author=. Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval , pages=

  9. [9]

    arXiv preprint arXiv:1908.08167 , year=

    Multi-passage bert: A globally normalized bert model for open-domain question answering , author=. arXiv preprint arXiv:1908.08167 , year=

    Pith/arXiv arXiv 1908

  10. [10]

    arXiv preprint arXiv:2005.00766 , year=

    BERT-kNN: Adding a kNN search component to pretrained language models for better QA , author=. arXiv preprint arXiv:2005.00766 , year=

    Pith/arXiv arXiv 2005

  11. [11]

    COLIEE 2022 Summary: Methods for Legal Document Retrieval and Entailment , author=. New Frontiers in Artificial Intelligence: JSAI-isAI 2022 Workshop, JURISIN 2022, and JSAI 2022 International Session, Kyoto, Japan, June 12--17, 2022, Revised Selected Papers , pages=. 2023 , organization=

    2022

  12. [12]

    The Review of Socionetwork Strategies , volume=

    Overview and discussion of the competition on legal information extraction/entailment (COLIEE) 2021 , author=. The Review of Socionetwork Strategies , volume=. 2022 , publisher=

    2021

  13. [13]

    New Frontiers in Artificial Intelligence: JSAI-isAI 2020 Workshops, JURISIN, LENLS 2020 Workshops, Virtual Event, November 15--17, 2020, Revised Selected Papers 12 , pages=

    COLIEE 2020: methods for legal document retrieval and entailment , author=. New Frontiers in Artificial Intelligence: JSAI-isAI 2020 Workshops, JURISIN, LENLS 2020 Workshops, Virtual Event, November 15--17, 2020, Revised Selected Papers 12 , pages=. 2021 , organization=

    2020

  14. [14]

    A summary of the coliee 2019 competition , author=. New Frontiers in Artificial Intelligence: JSAI-isAI International Workshops, JURISIN, AI-Biz, LENLS, Kansei-AI, Yokohama, Japan, November 10--12, 2019, Revised Selected Papers 10 , pages=. 2020 , organization=

    2019

  15. [15]

    Twelfth international workshop on Juris-informatics (JURISIN 2018) , year=

    Overview of japanese statute law retrieval and entailment task at coliee-2018 , author=. Twelfth international workshop on Juris-informatics (JURISIN 2018) , year=

    2018

  16. [16]

    LatinX in Natural Language Processing Research Workshop , year=

    Study of Question Answering on Legal Software Document using BERT based models , author=. LatinX in Natural Language Processing Research Workshop , year=

  17. [17]

    arXiv preprint arXiv:1911.00841 , year=

    Question answering for privacy policies: Combining computational and legal perspectives , author=. arXiv preprint arXiv:1911.00841 , year=

    Pith/arXiv arXiv 1911

  18. [18]

    Ninth International Workshop on Juris-informatics (JURISIN 2015) , year=

    COLIEE-2015: evaluation of legal question answering , author=. Ninth International Workshop on Juris-informatics (JURISIN 2015) , year=

    2015

  19. [19]

    Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10--14, 2022, Proceedings, Part II , pages=

    Expert finding in legal community question answering , author=. Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10--14, 2022, Proceedings, Part II , pages=. 2022 , organization=

    2022

  20. [20]

    2023 , eprint=

    UL2: Unifying Language Learning Paradigms , author=. 2023 , eprint=

    2023

  21. [21]

    2020 , eprint=

    Longformer: The Long-Document Transformer , author=. 2020 , eprint=

    2020

  22. [22]

    2021 , eprint=

    Evaluating Pretrained Transformer Models for Entity Linking in Task-Oriented Dialog , author=. 2021 , eprint=

    2021

  23. [23]

    2020 , eprint=

    BERTScore: Evaluating Text Generation with BERT , author=. 2020 , eprint=

    2020

  24. [24]

    KPQA : A Metric for Generative Question Answering Using Keyphrase Weights

    Lee, Hwanhee and Yoon, Seunghyun and Dernoncourt, Franck and Kim, Doo Soon and Bui, Trung and Shin, Joongbo and Jung, Kyomin. KPQA : A Metric for Generative Question Answering Using Keyphrase Weights. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021. doi:10...

    work page doi:10.18653/v1/2021.naacl-main.170 2021

  25. [25]

    2020 , publisher=

    New Frontiers in Artificial Intelligence: JSAI-isAI International Workshops, JURISIN, AI-Biz, LENLS, Kansei-AI, Yokohama, Japan, November 10--12, 2019, Revised Selected Papers , author=. 2020 , publisher=

    2019

  26. [26]

    Generative AI+ Law Workshop , year=

    Chain Of Reference prompting helps LLM to think like a lawyer , author=. Generative AI+ Law Workshop , year=

  27. [27]

    arXiv preprint arXiv:2308.11462 , year=

    Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models , author=. arXiv preprint arXiv:2308.11462 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2005.05257 , year=

    A dataset for statutory reasoning in tax law entailment and question answering , author=. arXiv preprint arXiv:2005.05257 , year=

    Pith/arXiv arXiv 2005

  29. [29]

    arXiv preprint arXiv:2306.05827 , year=

    Towards the Exploitation of LLM-based Chatbot for Providing Legal Support to Palestinian Cooperatives , author=. arXiv preprint arXiv:2306.05827 , year=

    Pith/arXiv arXiv

  30. [30]

    Advances in Neural Information Processing Systems , volume=

    Retrieval-augmented generation for knowledge-intensive nlp tasks , author=. Advances in Neural Information Processing Systems , volume=

  31. [31]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  32. [32]

    Advances in neural information processing systems , volume=

    An investigation of practical approximate nearest neighbor algorithms , author=. Advances in neural information processing systems , volume=

  33. [33]

    B leu: a Method for Automatic Evaluation of Machine Translation

    Papineni, Kishore and Roukos, Salim and Ward, Todd and Zhu, Wei-Jing. B leu: a Method for Automatic Evaluation of Machine Translation. Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. 2002. doi:10.3115/1073083.1073135

    arXiv 2002

  34. [34]

    ROUGE : A Package for Automatic Evaluation of Summaries

    Lin, Chin-Yew. ROUGE : A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out. 2004

    2004

  35. [35]

    Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =

    Song, Kaitao and Tan, Xu and Qin, Tao and Lu, Jianfeng and Liu, Tie-Yan , title =. Proceedings of the 34th International Conference on Neural Information Processing Systems , articleno =. 2020 , isbn =

    2020

  36. [36]

    Advances in neural information processing systems , volume=

    Xlnet: Generalized autoregressive pretraining for language understanding , author=. Advances in neural information processing systems , volume=

  37. [37]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

    1972

  38. [38]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    LLMs--the Good, the Bad or the Indispensable?: A Use Case on Legal Statute Prediction and Legal Judgment Prediction on Indian Court Cases , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  39. [39]

    Proceedings of the Natural Legal Language Processing Workshop 2021 , pages=

    jurBERT: A Romanian BERT model for legal judgement prediction , author=. Proceedings of the Natural Legal Language Processing Workshop 2021 , pages=

    2021

  40. [40]

    Advances in Neural Information Processing Systems , volume=

    A multi-task benchmark for korean legal language understanding and judgement prediction , author=. Advances in Neural Information Processing Systems , volume=

  41. [41]

    Information Processing & Management , volume=

    Criminal action graph: a semantic representation model of judgement documents for legal charge prediction , author=. Information Processing & Management , volume=. 2023 , publisher=

    2023

  42. [42]

    Chinese Computational Linguistics: 19th China National Conference, CCL 2020, Hainan, China, October 30--November 1, 2020, Proceedings 19 , pages=

    Multi-task legal judgement prediction combining a subtask of the seriousness of charges , author=. Chinese Computational Linguistics: 19th China National Conference, CCL 2020, Hainan, China, October 30--November 1, 2020, Proceedings 19 , pages=. 2020 , organization=

    2020

  43. [43]

    ILDC for CJPE : I ndian Legal Documents Corpus for Court Judgment Prediction and Explanation

    Malik, Vijit and Sanjay, Rishabh and Nigam, Shubham Kumar and Ghosh, Kripabandhu and Guha, Shouvik Kumar and Bhattacharya, Arnab and Modi, Ashutosh. ILDC for CJPE : I ndian Legal Documents Corpus for Court Judgment Prediction and Explanation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International...

    doi:10.18653/v1/2021.acl-long.313 2021

  44. [44]

    arXiv preprint arXiv:1807.02478 , year=

    Cail2018: A large-scale legal dataset for judgment prediction , author=. arXiv preprint arXiv:1807.02478 , year=

    Pith/arXiv arXiv

  45. [45]

    HLDC : H indi Legal Documents Corpus

    Kapoor, Arnav and Dhawan, Mudit and Goel, Anmol and T H, Arjun and Bhatnagar, Akshala and Agrawal, Vibhu and Agrawal, Amul and Bhattacharya, Arnab and Kumaraguru, Ponnurangam and Modi, Ashutosh. HLDC : H indi Legal Documents Corpus. Findings of the Association for Computational Linguistics: ACL 2022. 2022. doi:10.18653/v1/2022.findings-acl.278

    work page doi:10.18653/v1/2022.findings-acl.278 2022

  46. [46]

    Artificial Intelligence and Law , volume=

    Using machine learning to predict decisions of the European Court of Human Rights , author=. Artificial Intelligence and Law , volume=. 2020 , publisher=

    2020

  47. [47]

    Bhattacharya, Paheli and Poddar, Soham and Rudra, Koustav and Ghosh, Kripabandhu and Ghosh, Saptarshi , title =

  48. [48]

    Legal Knowledge and Information Systems: JURIX , pages=

    Identification of rhetorical roles of sentences in indian legal judgments , author=. Legal Knowledge and Information Systems: JURIX , pages=

  49. [49]

    arXiv preprint arXiv:2112.01836 , year=

    Semantic segmentation of legal documents via rhetorical roles , author=. arXiv preprint arXiv:2112.01836 , year=

    Pith/arXiv arXiv

  50. [50]

    arXiv preprint arXiv:2307.05260 , year=

    U-CREAT: Unsupervised Case Retrieval using Events extrAcTion , author=. arXiv preprint arXiv:2307.05260 , year=

    Pith/arXiv arXiv

  51. [51]

    JSAI International Symposium on Artificial Intelligence , pages=

    nigam@ COLIEE-22: Legal Case Retrieval and Entailment using Cascading of Lexical and Semantic-based models , author=. JSAI International Symposium on Artificial Intelligence , pages=. 2022 , organization=

    2022

  52. [52]

    arXiv preprint arXiv:2201.13125 , year=

    Corpus for automatic structuring of legal documents , author=. arXiv preprint arXiv:2201.13125 , year=

    Pith/arXiv arXiv

  53. [53]

    Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023) , pages=

    Nonet at SemEval-2023 Task 6: Methodologies for Legal Evaluation , author=. Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023) , pages=

    2023

  54. [54]

    Proceedings of 19th International Conference on Artificial Intelligence and Law - ICAIL 2023 , year =

    Paul, Shounak and Mandal, Arpan and Goyal, Pawan and Ghosh, Saptarshi , title =. Proceedings of 19th International Conference on Artificial Intelligence and Law - ICAIL 2023 , year =

    2023

  55. [55]

    Advances in Neural Information Processing Systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in Neural Information Processing Systems , volume=

  56. [56]

    Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    LEGAL-BERT: The Muppets straight out of Law School , author=. Findings of the Association for Computational Linguistics: EMNLP 2020 , pages=

    2020

  57. [57]

    Proceedings of the eighteenth international conference on artificial intelligence and law , pages=

    When does pretraining help? assessing self-supervised learning for law and the casehold dataset of 53,000+ legal holdings , author=. Proceedings of the eighteenth international conference on artificial intelligence and law , pages=

  58. [58]

    Gordon (ed.), Legal Knowledge and Information Systems

    Atefeh Farzindar and Guy Lapalme,'LetSum, an automatic Legal Text Summarizing system in T. Gordon (ed.), Legal Knowledge and Information Systems. Jurix 2004: The Seventeenth Annual Conference. Amsterdam: IOS Press, 2004, pp. 11-18 , author=. Legal knowledge and information systems: JURIX 2004, the seventeenth annual conference , volume=. 2004 , organization=

    2004

  59. [59]

    Proceedings of the 3rd International Conference on Information Science and Systems , pages=

    Legal judgement prediction for uk courts , author=. Proceedings of the 3rd International Conference on Information Science and Systems , pages=

  60. [60]

    2020 International Conference on Smart Electronics and Communication (ICOSEC) , pages=

    A novel approach on argument based legal prediction model using machine learning , author=. 2020 International Conference on Smart Electronics and Communication (ICOSEC) , pages=. 2020 , organization=

    2020

  61. [61]

    2004 , organization=

    LetSum, an Automatic Text Summarization system in Law field , author=. 2004 , organization=

    2004

  62. [62]

    Proceedings of the AAAI conference on artificial intelligence , volume=

    Summarunner: A recurrent neural network based sequence model for extractive summarization of documents , author=. Proceedings of the AAAI conference on artificial intelligence , volume=

  63. [63]

    Minds and Machines , volume=

    GPT-3: Its nature, scope, limits, and consequences , author=. Minds and Machines , volume=. 2020 , publisher=

    2020

  64. [64]

    OpenAI blog , volume=

    Language models are unsupervised multitask learners , author=. OpenAI blog , volume=

  65. [65]

    2021 6th International Conference on Signal Processing, Computing and Control (ISPCC) , pages=

    Analysis of Legal Case Document Automated Summarizer , author=. 2021 6th International Conference on Signal Processing, Computing and Control (ISPCC) , pages=. 2021 , organization=

    2021

  66. [66]

    Proceedings of COLING 2016, the 26th international conference on Computational Linguistics: System Demonstrations , pages=

    Casesummarizer: a system for automated summarization of legal texts , author=. Proceedings of COLING 2016, the 26th international conference on Computational Linguistics: System Demonstrations , pages=

    2016

  67. [67]

    arXiv preprint arXiv:1903.10318 , year=

    Fine-tune BERT for extractive summarization , author=. arXiv preprint arXiv:1903.10318 , year=

    Pith/arXiv arXiv 1903

  68. [68]

    arXiv preprint arXiv:1907.11692 , year=

    Roberta: A robustly optimized bert pretraining approach , author=. arXiv preprint arXiv:1907.11692 , year=

    Pith/arXiv arXiv 1907

  69. [69]

    Proceedings of NAACL-HLT , pages=

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , author=. Proceedings of NAACL-HLT , pages=

  70. [70]

    arXiv preprint arXiv:2311.13350 , year=

    Fact-based Court Judgment Prediction , author=. arXiv preprint arXiv:2311.13350 , year=

    Pith/arXiv arXiv

  71. [71]

    arXiv preprint arXiv:2209.06049 , year=

    Pre-training transformers on indian legal text , author=. arXiv preprint arXiv:2209.06049 , year=

    Pith/arXiv arXiv

  72. [72]

    Publications Manual , year = "1983", publisher =

    1983

  73. [73]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

    arXiv 1981

  74. [74]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  75. [75]

    Dan Gusfield , title =. 1997

    1997

  76. [76]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

    2015

  77. [77]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  78. [78]

    2024 , eprint=

    Aalap: AI Assistant for Legal & Paralegal Functions in India , author=. 2024 , eprint=

    2024

  79. [79]

    arXiv preprint arXiv:2212.09741 , year=

    One embedder, any task: Instruction-finetuned text embeddings , author=. arXiv preprint arXiv:2212.09741 , year=

    Pith/arXiv arXiv

  80. [80]

    Advances in neural information processing systems , volume=

    Mpnet: Masked and permuted pre-training for language understanding , author=. Advances in neural information processing systems , volume=

Showing first 80 references.