Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A retrieval-augmented generation system that grounds answers in state and federal statutes outperforms leading commercial chatbots on four legal-question-answering metrics, the paper reports.

desk verdict Useful statute corpus and clear system description, but the headline comparison against commercial LLMs is not a fair test. read the letter →

arxiv 2505.18426 v1 pith:6TA2IEHT submitted 2025-05-23 cs.CL cs.AI

Khandakar Ashrafi Akbar , Md Nahiyan Uddin , Latifur Khan , Trayce Hockstad , Mizanur Rahman , Mashrur Chowdhury , Bhavani Thuraisingham This is my paper · ORCID
classification cs.CLcs.AI
keywords retrieval-augmentedgenerationlargelanguagemodelshallucinationmitigationtransportationcybersecuritylegislativeanalysisstatestatuteslegalquestionansweringfactualconsistencymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a retrieval-augmented generation (RAG) system, which pulls relevant state and federal statutes from a curated vector database before answering, can give policymakers more faithful answers about transportation cybersecurity law than general-purpose commercial chatbots. The authors build a legal corpus of 1,795 legislative documents and a 59-question dataset, then compare their RAG-powered GPT with leading commercial LLMs on AlignScore, ParaScore, BERTScore, and ROUGE. On every metric, the RAG system reports the highest F1 scores, which the authors interpret as evidence that retrieval grounds the model in current law and reduces the hallucination that plagues unaugmented models. If the claim holds, legislative analysts could use such a tool to surface gaps in state and federal statutes without retraining LLMs on every new law.

What carries the argument

The machinery is a two-stage retrieval-augmented generation pipeline built with the LlamaIndex framework. Legislative texts are split into chunks, embedded, and stored in a vector database; at query time a retriever finds the top-k relevant chunks and the generator (gpt-3.5-turbo) writes an answer using those chunks as context. Two retrieval strategies are compared: Whole Data Index (WDI), which searches all states at once, and State-Wise Index (SWI), which first identifies states named in the query and then searches only their indexes. The evaluation metrics — AlignScore for factual consistency, ParaScore for paraphrase quality, BERTScore for token-level semantic similarity, and ROUGE for n-gram overlap — are what carry the headline claim of outperformance.

What would settle it

Re-run the same 59-question evaluation but prepend the identical retrieved statute chunks to the prompts given to ChatGPT, Claude, and Gemini; if their AlignScore, ParaScore, BERTScore, and ROUGE F1 scores rise to at least the RAG system's reported values, the paper's outperformance claim is explained by information access rather than by the RAG architecture itself.

Watch

Extended reading notes

Core claim

The central claim is that grounding an LLM in retrieved statutory text is enough to make it outperform standalone commercial LLMs on domain-specific legal Q&A. The paper reports that its RAG-powered GPT achieves an AlignScore F1 of 0.73201, ParaScore F1 of 0.70433, BERTScore F1 of 0.87033, and ROUGE-L F1 of 0.37340, all higher than ChatGPT-3.5, ChatGPT-4o, Gemini, and Claude on the same 59-question subset. The authors take this as evidence that RAG mitigates hallucination in a rapidly evolving legal domain, because the generator is constrained by up-to-date context retrieved from a vector database rather than relying on its static training data.

Load-bearing premise

The comparison assumes it is fair to give the RAG system the full statute corpus while the commercial chatbots must answer from memory alone; if the baselines received the same retrieved passages, the gap in Table 6 would likely shrink.

Editorial extensions

If this is right

  • A RAG-based legal assistant can answer questions about newly enacted statutes without retraining the underlying LLM, since updates enter through the vector database.
  • The reported score gaps imply that commercial chatbots' legal answers are less faithful than retrieval-grounded ones for extractive, statute-specific questions.
  • The State-Wise Index method gives more accurate multi-state comparisons than the Whole Data Index, with comparable response time, making it the better default for jurisdictional queries.
  • If the approach scales, policymakers could use it to identify legislative gaps across all 50 states, as the paper demonstrates with questions about liability, encryption, and critical infrastructure definitions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison may overstate the RAG system's advantage: the reference answers were drafted from the same corpus the RAG retriever searches, while the commercial baselines were given no documents, so part of the gap is information access rather than reasoning quality.
  • A fairer test would feed the identical retrieved statute chunks to the commercial LLMs as context; the metrics would likely compress, and the residual difference would isolate RAG-specific benefits.
  • The 59-question set is small and extractive-heavy, so the claim of 'outperforming' should be read as proof-of-concept for one curated benchmark, not a general statement about legal reasoning ability.
  • One could extend the framework by measuring whether retrieved-context answers actually reduce downstream human error, for example by asking legal experts to flag inaccuracies in the RAG outputs versus baseline outputs.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents a Retrieval-Augmented Generation (RAG) based legal question-answering system for U.S. transportation cybersecurity statutes. The authors compile 1,795 federal, state, and international legislative documents, index them into a vector database, and propose two retrieval variants, Whole Data Index and State-Wise Index. They create a dataset of 59 Q&A pairs with attorney-authored reference answers and compare the RAG-powered GPT system (gpt-3.5-turbo with LlamaIndex retrieval) against ChatGPT-3.5/4o, Claude, and Gemini on AlignScore, ParaScore, BERTScore, and ROUGE, reporting that the RAG system outperforms all baselines. The paper also reports latency comparisons between the two retrieval methods and claims that RAG reduces hallucinations and produces more factually accurate legal insights.

Significance. If the comparison were controlled, the result would be useful evidence that retrieval grounding improves legal Q&A faithfulness. The corpus compilation by a licensed attorney and the source-citation feature are practical contributions, and the two-index design addresses multi-state queries in a useful way. However, as reported, the headline comparison is not evidence of superior reasoning: the RAG system is given access to the exact statute corpus from which the reference answers were drafted, while commercial baselines are not, and none of the four metrics verifies output against the source statutes. The paper is therefore best read as a system demonstration rather than a controlled benchmark, and the abstract's claim of outperformance is not currently supported.

major comments (4)
  1. [§8.2, Table 6; §6.4] The central comparative claim is undermined by an asymmetric evaluation protocol. Section 6.4 states that the reference answers used as ground truths were curated from the same U.S. statutes that populate the RAG vector database (Section 5), while Section 8.2 and Table 5 show that the commercial baselines received only the question. For extractive questions such as the Code of Ala. § 13A-8-111 definition in Table 4, the RAG system retrieves and reproduces the statute text, producing near-verbatim overlap with the reference answer, whereas the baselines either decline to answer or rely on general knowledge. The score gap in Table 6 therefore conflates access to the answer source with outperformance on legal reasoning. The authors should rerun the comparison with baselines given the same retrieved statute chunks, or with the same generator with and without retrieval, and use a held-out reference set not derived from the retrieval corpus.
  2. [§8.2, Evaluation Metrics] None of the four metrics measures hallucination against the source law. AlignScore is computed between the generated response and the reference answer, ParaScore between the generated response and the reference, and BERTScore and ROUGE quantify lexical or embedding overlap with the reference. A system that copies the statute can score highly, while a cautious baseline that refuses to answer, as ChatGPT-4o does in Table 5, is penalized. The abstract's claim that RAG produces more factually accurate responses and reduces hallucinations is therefore not established. The authors should add a source-grounded faithfulness evaluation, for example by checking generated assertions against the retrieved statute documents, and report a human-verified error analysis.
  3. [§6.4 and §8] The evaluation set appears to be the same set used to develop the system. Section 6.4 says the 59 questions were used to train the LLM, but no fine-tuning or training is described anywhere in the paper; Section 8 then says the comparison is run on a subset of the Q&A dataset without specifying a held-out split or whether the reference answers were written before system outputs were generated. If the same questions and references were used during prompt tuning or retrieval parameter selection, such as top-k, the aggregate scores in Table 6 are optimistically biased. Please define a clear train/evaluation split, report per-question results, and separate extractive questions from synthesis questions.
  4. [§8.2, Table 6; Table 1] The comparison is confounded by model choice. The RAG generator is gpt-3.5-turbo (Table 1), while the baselines include GPT-4o, Claude, and Gemini, which are substantially more capable models. The reported advantage may reflect retrieval access rather than the RAG framework per se, and the comparison does not isolate the effect of retrieval augmentation. Please include a no-RAG gpt-3.5-turbo arm, and report the exact versions and prompting protocols used for each baseline.
minor comments (6)
  1. [Table 5] The second column header of Table 5 reads 'Response from TraCR RAG,' but the table is captioned as presenting baseline LLMs and the content in that column is from ChatGPT-4o, Claude, and Gemini; this appears to be a copy-paste error that should be corrected.
  2. [§6.4] The phrase '59 initial questions that were used to train the LLM' is misleading because no LLM training is described; the questions appear to be used for evaluation and system development, so the wording should be revised.
  3. [References] Reference 24 lists 'Claudia' instead of 'Claude,' and reference 32's URL contains a typo ('vector-dabase'); reference 23 also includes an odd 'ChatGPT-' URL fragment that should be cleaned.
  4. [Figure 9] The sample output in Figure 9 displays local file paths such as 'C:\Users\User\Box\UTD\PhDResearch\RAG-app\...' as sources; these machine-specific paths should be anonymized or replaced with document names and citations.
  5. [§10] The limitation statement in Section 10, which says the study lacks a dedicated module for comparing responses across different legislative bodies, partially contradicts the paper's claimed contribution of identifying legislative gaps; the conclusions should reconcile this limitation with the stated contribution.
  6. [§7.2, Table 7] The text claims that State-Wise Index shows 'no significant difference' in response time, but no statistical test is reported, and the claimed accuracy advantage of SWI is not quantified with retrieval metrics such as precision@k or recall@k.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported RAG outperformance is largely self-referential because the reference answers used as ground truth were curated from the same statute corpus that populates the RAG retriever, while commercial baselines were not given that corpus.

  1. self definitional [Section 6.4 (Question and Answer Dataset), used by Section 8.2 / Table 6]
    "We have also curated the reference answers to these demonstrative questions that eventually serve as ground truths."

    The RAG vector database is populated with the same statutes from which the reference answers were drafted: Section 6.1 says these statutes 'serve as essential contextual cues for generating responses,' and Section 6.4 says the reference answers 'serve as ground truths.' The metrics in Section 8.2 (AlignScore, ParaScore, BERTScore, ROUGE) all score candidate responses against those reference answers. Table 4 shows the RAG answer for Code of Ala. §13A-8-111 is nearly verbatim statute text, and the reference answer is the same statutory text. Thus, for extractive questions, high scores reduce to retrieval success from the corpus that also generated the ground truth.

full rationale

The central comparative claim rests on an evaluation set that is not independent of the RAG system's knowledge source. Section 6.4 states that the reference answers were curated by the authors to serve as ground truths, and Section 6.1 establishes that the same state and federal statutes are the foundational content of the RAG pipeline. Since all four evaluation metrics compare generated responses to those reference answers, and since Table 4 shows the RAG output reproducing the retrieved statute almost verbatim for an extractive question, the reported outperformance in Table 6 is substantially built into the benchmark design. The commercial baselines, described only as standalone commercial LLMs, were evidently not given access to the corpus, making the comparison asymmetric. This does not make the entire paper circular: non-extractive questions require synthesis, and a retrieval system could still fail even with the corpus. However, the headline claim that RAG 'outperforms leading commercial LLMs' and produces 'more factually accurate' responses is not independently validated by this self-referential protocol. The self-citations in the paper (e.g., references [2] and [8]) are background citations and are not load-bearing for the evaluation, so they do not add to the circularity score.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The evaluation rests on unproven assumptions: the authors' reference answers are treated as legal gold standard without independent validation; text-similarity metrics are used as a proxy for hallucination reduction with no direct hallucination measurement; the single-attorney curated corpus is assumed complete and current; and retrieval recall is assumed adequate without measurement. The top-k retrieval setting is an unstated hyperparameter.

free parameters (1)
  • Top-k retrieved chunks = unspecified
    The retrieval depth k is a manually chosen hyperparameter that affects response quality; the paper says k is variable (e.g., 5, 10, 15) but never reports the value used for the Table 6 evaluation.
assumptions (4)
  • domain assumption Reference answers are correct, complete ground truth for the 59 questions
    The evaluation treats the authors' curated reference answers as the standard of truth; the paper does not validate them independently, for example through external legal review or inter-annotator agreement.
  • domain assumption Text-similarity metrics (AlignScore, ParaScore, BERTScore, ROUGE) measure factual accuracy and hallucination reduction
    These metrics measure lexical and semantic overlap with a reference, not legal correctness. A response can be similar yet legally wrong, or dissimilar yet correct, and the paper does not calibrate the metrics against any legal gold standard.
  • domain assumption The statute corpus assembled by one licensed attorney is complete and representative of current U.S. transportation cybersecurity law
    Section 6 describes a single-attorney LexisNexis search over five search terms with post hoc exclusions (election security, cyberbullying, taxation, etc.); completeness across all 50 states and federal sources is not verified.
  • domain assumption Retrieved top-k chunks contain all information needed to answer the query
    RAG quality depends on retrieval recall; the paper never measures retrieval recall or analyzes failure cases where the needed statute was not retrieved.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps." pith.science (2026). https://pith.science/paper/6TA2IEHT

@misc{pith2026250518426,
  author       = {Pith},
  title        = {Pith review of: Retrieval Augmented Generation-based Large Language Models for Bridging Transportation Cybersecurity Legal Knowledge Gaps},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TA2IEHT}},
  note         = {Machine review of arXiv:2505.18426}
}
read the original abstract

As connected and automated transportation systems evolve, there is a growing need for federal and state authorities to revise existing laws and develop new statutes to address emerging cybersecurity and data privacy challenges. This study introduces a Retrieval-Augmented Generation (RAG) based Large Language Model (LLM) framework designed to support policymakers by extracting relevant legal content and generating accurate, inquiry-specific responses. The framework focuses on reducing hallucinations in LLMs by using a curated set of domain-specific questions to guide response generation. By incorporating retrieval mechanisms, the system enhances the factual grounding and specificity of its outputs. Our analysis shows that the proposed RAG-based LLM outperforms leading commercial LLMs across four evaluation metrics: AlignScore, ParaScore, BERTScore, and ROUGE, demonstrating its effectiveness in producing reliable and context-aware legal insights. This approach offers a scalable, AI-driven method for legislative analysis, supporting efforts to update legal frameworks in line with advancements in transportation technologies.

Figures

Figures reproduced from arXiv: 2505.18426 by the authors.

Figure 1
Figure 1. Hallucination in Large Language Models 2. CONTRIBUTION OF THE RESEARCH The primary contribution of this research is tackling the hallucination issue prevalent in commercial state-of-the-art LLMs when generating responses in specialized and rapidly evolving domains, such as transportation cybersecurity. LLM hallucination in legal contexts is critical due to: 1) the sensitivity of legal matters demanding heightened ac… view at source ↗
Figure 2
Figure 2. Architectural Overview of RAG 5. SYSTEM OVERVIEW In this section, we present the RAG pipeline. This pipeline employs two distinct methods for relative context retrieval. The first method generates the retrieval index on the entire dataset. When a question is posed to the LLM, the retriever searches through all indices to find the most relevant ones, subsequently generating the answer. Although this process is more t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 44 canonical work pages

  1. [1]

    INTRODUCTION The rise of connected and automated transportation systems underscores the pressing need to address legal uncertainties at the federal, state, and international levels. The intricate nature of these systems—particularly their reliance on networks of Internet of Things (IoT) devices—raises complex questions around data privacy, infrastructure ...

    work page

  2. [2]

    CONTRIBUTION OF THE RESEARCH The primary contribution of this research is tackling the hallucination issue prevalent in commercial state-of-the-art LLMs when generating responses in specialized and rapidly evolving domains, such as transportation cybersecurity. LLM hallucination in legal contexts is critical due to: 1) the sensitivity of legal matters dem...

    work page

  3. [3]

    We explore how generative models have evolved, the nature and taxonomy of hallucinations, and the approaches taken to detect and mitigate them

    RELATED WORK To contextualize our study, this literature review critically examines the development of LLMs with a focus on hallucination—a significant limitation that undermines their reliability. We explore how generative models have evolved, the nature and taxonomy of hallucinations, and the approaches taken to detect and mitigate them. Our goal is to ...

    work page

  4. [4]

    To mitigate these complexities, context -augmented LLMs offer a practical solution

    RETRIEVAL AUGMENTED GENERATION (RAG) Sustaining LLMs presents significant challenges, primarily due to the enormous data requirements for training and fine-tuning and the computational resources needed. To mitigate these complexities, context -augmented LLMs offer a practical solution. Data frameworks such as LlamaIndex [19] facilitate the development of ...

    work page

  5. [5]

    Alabama" and the state of

    SYSTEM OVERVIEW In this section, we present the RAG pipeline . This pipeline employs two distinct methods for relative context retrieval. The first method generates the retrieval index on the entire dataset. When a question is posed to the LLM, the retriever searches through all indices to find the most relevant ones, subsequently generating the answer. A...

    work page

  6. [6]

    statutes and legislation

    DATASET PREPARATION 2 3 To develop a comprehensive dataset of currently enacted U.S. state -level legislation relevant to 4 cybersecurity governance, a systematic review of all 50 state statutory codes was conducted. The 5 purpose of this exercise was to identify, categorize, and extract legislative provisions that address 6 core areas of cybersecurity la...

    work page 2012

  7. [7]

    Figure 9 showcases a sample output 4 produced by our RAG-powered GPT architecture

    RAG EVALUATION 1 2 We made great strides in resolving the problem of insufficiently trained LLMs lacking up-to-date 3 information and the issue of hallucination by utilizing RAG. Figure 9 showcases a sample output 4 produced by our RAG-powered GPT architecture. This architecture leverages RAG to transform 5 datasets collected at federal, state, and intern...

    work page 2025

  8. [9]

    This limitation is 29 significant, as inaccurate outputs can misidentify legislative gaps and complicate the work of legal 30 and policy stakeholders

    CONCLUSIONS 26 27 These findings suggest that even state -of-the-art LLMs face challenges when responding to 28 specialized queries, particularly those involving recently enacted legislation. This limitation is 29 significant, as inaccurate outputs can misidentify legislative gaps and complicate the work of legal 30 and policy stakeholders. Nonetheless, t...

    work page

Show all 53 references
  1. [12]

    Definitions.pdf (14) Identification document

    Current Cybersecurity Law\Alabama\Digital Crime Act\_ 13A-8-111. Definitions.pdf (14) Identification document. — Any document containing data that is issued to an individual and which that individual, and only that individual, uses alone or in conjunction with any other inform...

  2. [13]

    California California: I am sorry, I could not find any information to answer the question you asked. None. 2 18 Table 5: Case Study on the Effectiveness of Baseline LLMs 1 Question Response from TraCR RAG Reference Answers What is the definition of identification document acc...

  3. [15]

    § 13A-8-111:

    Alabama Alabama: According to Code of Ala. § 13A-8-111:

  4. [16]

    Identification document. — Any document containing data that is issued to an individual and which that individual, and only that individual, uses alone or in conjunction with any other information for the primary purpose of establishing his or her identity or accessing his or ...

  5. [17]

    Definitions.pdf

    Current Cybersecurity Law\Alabama\Digital Crime Act\_ 13A-8-111. Definitions.pdf

  6. [18]

    Current Cybersecurity Law\Alabama\Data Breach Notification Act\Files(12).pdf

  7. [19]

    identification document

    Current Cybersecurity Law\Alabama\Digital Crime Act\_ 13A-8-111. Definitions.pdf ChatGPT-4o: Sorry, I can't guarantee accurate information about voter- related laws and procedures in the U.S., including Code of Ala. § 13A-8-111. I recommend checking the official Alabama state ...

  8. [20]

    The comparison focuses on the 4 factual accuracy of the responses produced by each model

    COMPARATIVE EVALUATION OF RAG-POWERED GPT ARCHITECTURE 1 2 We compare the performance of our RAG -powered LLM with other state -of-the-art commercial 3 LLMs in generating responses for a subset of our Q&A dataset. The comparison focuses on the 4 factual accuracy of the respons...

    2022

  9. [21]

    https://openai.com/index/gpt-4o-and-more-tools-to-chatgpt-free/ (Date Accessed: May 5, 2024)

    2024

  10. [22]

    These methods can scrutinize both questions and outputs for adherence to 25 standards

    FUTURE ADVANCEMENTS 22 23 Comprehensive methods can be developed to ensure factual accuracy in the responses generated 24 for our dataset. These methods can scrutinize both questions and outputs for adherence to 25 standards. From our observations, w e have seen that the same ...

  11. [23]

    Department of Transportation National University 29 Transportation Center) headquartered at Clemson University, Clemson, South Carolina, USA

    ACKNOWLEDGMENTS 26 27 This work is based upon the work supported by the National Center for Transportation 28 Cybersecurity and Resiliency (TraCR) (a U.S. Department of Transportation National University 29 Transportation Center) headquartered at Clemson University, Clemson, S...

  12. [24]

    All authors reviewed the results 42 and approved the final version of the manuscript

    AUTHOR CONTRIBUTIONS 38 39 The authors confirm their contributions to the paper as follows: Khandakar Ashrafi Akbar and Md 40 Nahiyan Uddin contributed equally, with responsibilities including the design and assessment of 41 the framework and the development of the RAG -powere...

  13. [25]

    CONFLICT OF INTEREST 1 2 The authors declared no potential conflicts of interest with respect to the research, authorship, 3 and/or publication of this article.4

  14. [26]

    Giannaros, A., Karras, A., Theodorakopoulos, L., Karras, C., Kranias, P., Schizas, N., Kalogeratos, G., & Tsolis, D. (2023). Autonomous Vehicles: Sophisticated Attacks, Safety Issues, Challenges, Open Topics, Blockchain, and Future Directions. Journal of Cybersecurity and Priv...

    2023 doi

  15. [27]

    Hockstad, Trayce & Rahman, Mizanur & Jones, Steven & Chowdhury, Mashrur. (2024). A regulatory gap analysis in transportation cybersecurity and data privacy. Transportation Journal. 64. 10.1002/tjo3.12036

    2024 doi

  16. [28]

    Moss, eds

    Balleisen, Edward J., and David A. Moss, eds. Government and Markets: Toward a New Theory of Regulation. Cambridge: Cambridge University Press, 2010

    2010

  17. [29]

    Transforming Legal Aid with AI: Training LLMs to Ask Better Questions for Legal Intake

    Nick Goodson, Rongfei Lu, “Transforming Legal Aid with AI: Training LLMs to Ask Better Questions for Legal Intake”, March 15, 2024, https://law.stanford.edu/2024/03/15/transforming-legal-aid-with-ai-training-llms-to-ask- better-questions-for-legal-intake/ (Date Accessed: Decem...

    2024

  18. [30]

    How LLM’s Are A Game Changer In Legal Research

    A3Logics, "How LLM’s Are A Game Changer In Legal Research", Aug 5, 2024, https://www.a3logics.com/blog/llm-for-legal-research/ (Date Accessed: December 1, 2024)

    2024

  19. [31]

    GPT-4 Technical Report: https://cdn.openai.com/papers/gpt-4.pdf (Date Accessed: May 5, 2024)

    2024

  20. [32]

    Devlin, J., Chang, M.W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech...

    2019

  21. [33]

    Hu, Y ., Hosseini, M., Skorupa Parolin, E., Osorio, J., Khan, L., Brandt, P., & D′Orazio, V . (2022). ConfliBERT: A Pre-trained Language Model for Political Conflict and Violence. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Comput...

    2022

  22. [34]

    Sun, H., Xu, G., Deng, J., Cheng, J., Zheng, C., Zhou, H., Peng, N., Zhu, X., & Huang, M. (2021). On the Safety of Conversational Models: Taxonomy, Dataset, and Benchmark. ArXiv. /abs/2110.08466

    2021 arXiv

  23. [35]

    T., Bi, W., Shi, F., & Shi, S

    Zhang, Y ., Li, Y ., Cui, L., Cai, D., Liu, L., Fu, T., Huang, X., Zhao, E., Zhang, Y ., Chen, Y ., Wang, L., Luu, A. T., Bi, W., Shi, F., & Shi, S. (2023). Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models. ArXiv. /abs/2309.01219

    2023 arXiv

  24. [36]

    Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., & Liu, T. (2023). A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. ArXiv. /abs/2311.05232

    2023 arXiv

  25. [37]

    Dziri, N., Milton, S., Yu, M., Zaiane, O., & Reddy, S. (2022). On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? ArXiv. /abs/2204.07931

    2022 arXiv

  26. [38]

    Huo, S., Arabzadeh, N., & Clarke, C. L. (2023). Retrieving Supporting Evidence for Generative Question Answering. ArXiv. https://doi.org/10.1145/3624918.3625336 26

    2023

  27. [39]

    Liu, T., Zhang, Y ., Brockett, C., Mao, Y ., Sui, Z., Chen, W., & Dolan, B. (2021). A Token- level Reference-free Hallucination Detection Benchmark for Free-form Text Generation. ArXiv. /abs/2104.08704

    2021 arXiv

  28. [40]

    Gao, T., Yen, H., Yu, J., & Chen, D. (2023). Enabling Large Language Models to Generate Text with Citations. ArXiv. /abs/2305.14627

    2023 arXiv

  29. [41]

    Hallucinating Law: Legal Mistakes with Large Language Models Are Pervasive

    “Hallucinating Law: Legal Mistakes with Large Language Models Are Pervasive.” Stanford HAI, https://hai.stanford.edu/news/hallucinating-law-legal-mistakes-large- language-models-are-pervasive (Date Accessed: Aug 1, 2024)

    2024

  30. [42]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, & Douwe Kiela. (2021). Retrieval-Augmented Generation for Knowledge- Intensive NLP Tasks

    2021

  31. [43]

    G., Jain, N., Shen, S., Zaharia, M., Stoica, I., & Gonzalez, J

    Zhang, T., Patil, S. G., Jain, N., Shen, S., Zaharia, M., Stoica, I., & Gonzalez, J. E. (2024). RAFT: Adapting Language Model to Domain Specific RAG. ArXiv. /abs/2403.10131

    2024 arXiv

  32. [44]

    LlamaIndex

    “LlamaIndex.” LlamaIndex Documentation, https://docs.llamaindex.ai/en/stable/ (Date Accessed: August 1, 2024)

    2024

  33. [45]

    Lewis, P., Perez, E., Piktus, A., Petroni, F., Karpukhin, V ., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., & Kiela, D. (2021). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. arXiv preprint arXiv:2005.11401. Retrieved from https:/...

    2021 arXiv

  34. [47]

    https://www.anthropic.com/news/claude-3-family (Date Accessed: May 5, 2024)

    2024

  35. [48]

    (2024, July 26)

    Wikipedia contributors. (2024, July 26). ChatGPT. In Wikipedia, The Free Encyclopedia. Retrieved 21:02, July 26, 2024, from https://en.wikipedia.org/w/index.php?title=ChatGPT-&oldid=1236692130

    2024

  36. [49]

    (2024, May 7)

    Wikipedia contributors. (2024, May 7). Claudia. In Wikipedia, The Free Encyclopedia. Retrieved 21:07, July 26, 2024, from https://en.wikipedia.org/w/index.php?title=Claudia- &oldid=1222632185

    2024

  37. [50]

    (2024, July 23)

    Wikipedia contributors. (2024, July 23). Gemini (language model). In Wikipedia, The Free Encyclopedia. Retrieved 21:11, July 26, 2024, from https://en.wikipedia.org/w/index.php?- title=Gemini_(language_model)&oldid=1236223829

    2024

  38. [51]

    Zha, Yuheng & Yang, Yichi & Li, Ruichen & Hu, Zhiting. (2023). AlignScore: Evaluating Factual Consistency with A Unified Alignment Function. 11328-11348. 10.18653/v1/2023.acl-long.634

    2023 doi

  39. [52]

    Shen, L., Liu, L., Jiang, H., & Shi, S. (2022). On the Evaluation Metrics for Paraphrase Generation. In Y . Goldberg, Z. Kozareva, & Y . Zhang (Eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 3178–3190). Association for Comput...

    2022 doi

  40. [53]

    Q., & Artzi, Y

    Zhang, T., Kishore, V ., Wu, F., Weinberger, K. Q., & Artzi, Y . (2020). BERTScore: Evaluating Text Generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. https://openreview.net/forum?id=SkeHuCVFDr 27

    2020

  41. [54]

    Lin, C.-Y . (2004). ROUGE: A Package for Automatic Evaluation of Summaries. Text Summarization Branches Out, 74–81. https://aclanthology.org/W04-1013

    2004

  42. [55]

    OpenAI Platform

    “OpenAI Platform.” Prompt Engineering, https://platform.openai.com/docs/guides/prompt-engineering (Date Accessed: Aug 1, 2024)

    2024

  43. [56]

    https://www.meilisearch.com/blog/how-meilisearch-updates-a-millions-vector- embeddings-database-in-under-a-minute?utm_source=chatgpt.com (Date Accessed: May 5, 2024)

    2024

  44. [57]

    https://www.weka.io/learn/guide/ai-ml/vector-dabase/ (Date Accessed: May 5, 2024)

    2024

  45. [2023]

    What 37 are the maximum penalties for failing to follow the data breach notification statutes in Ohio and 38 Oklahoma?

    These models, including the latest Claude 3.5 Sonnet, use Constitutional AI, a training 25 method designed to make them helpful, honest, and harmless. Claude's iterations have introduced 26 features like image analysis, expanded token capacity, and advanced reasoning capabilit...

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.