Pith. sign in

REVIEW 4 major objections 4 minor 78 references

LLMs to Support a Domain Specific Knowledge Assistant

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A fully LLM-based pipeline, with industry classification and a fine-tuned generator, beats a retrieval-augmented baseline on IFRS sustainability questions.

desk verdict Useful domain dataset and honest engineering, but the headline accuracy numbers rest on an in-sample evaluation and unvalidated LLM self-scores. read the letter →

arxiv 2502.04095 v1 pith:2RZILX4I submitted 2025-02-06 cs.CL cs.AI

Maria-Flavia Lovin This is my paper · ORCID
classification cs.CLcs.AI
keywords syntheticquestion-answerdatasetIFRSsustainabilityreportingretrievalaugmentedgenerationLLMfine-tuningindustryclassificationdomain-specificQALoRAevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tackles a domain where no public question-answer dataset exists: IFRS sustainability reporting standards. It builds a synthetic dataset of 1,063 QA pairs using chain-of-thought and few-shot prompting, scores them with LLM judges on faithfulness, relevance, and specificity, and then uses the dataset to fine-tune a compact open model and to train an industry classifier. Two final pipelines are compared: one that retrieves chunks by vector search, and one that lets an LLM select the relevant context from whole industry reports. The fully LLM-based pipeline reaches 93.45% accuracy on single-industry multiple-choice questions and 80.30% on cross-industry questions, beating the RAG pipeline by about eight percentage points in each setting and the baseline by larger margins. If these numbers hold, the work shows that a high-accuracy domain assistant can be built from synthetic data in a regulated reporting domain without human-labelled QA pairs.

What carries the argument

The load-bearing mechanism is the pairing of two components: a prompt-based multi-label industry classifier that identifies which IFRS industry document a query concerns, and a low-rank-adapted (LoRA) fine-tuned Llama 3.1 8B model that answers from the selected context. In the fully LLM-based pipeline, context selection is itself delegated to an LLM that reads the entire industry markdown and returns query-specific chunks, replacing fixed chunking and embedding similarity. For dataset construction, the machinery is a chain-of-thought plus few-shot prompting pipeline that first selects reference sentences, then writes a question and a verbatim answer, then filters by LLM-judged faithfulness, relevance, domain specificity, single-best-answer status, and duplicate similarity.

What would settle it

Have a human expert confirm the correct answer for all 1,063 multiple-choice questions, then re-run both pipelines on questions that were set aside before fine-tuning; the central claim fails if the fully LLM-based pipeline's corrected accuracy drops below the custom RAG pipeline's numbers.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully LLM-based architecture is the better design for IFRS sustainability-reporting QA: classify the industry with an LLM prompt, let an LLM pick relevant context sections directly from each industry's markdown report, and generate the answer with a LoRA-fine-tuned Llama 3.1 8B model. The same classifier plus a fine-tuned generator inside a vector-search RAG pipeline is the runner-up. On the 1,063-pair synthetic benchmark, the LLM pipeline scores 93.45% single-industry and 80.30% cross-industry MCQ accuracy, versus 85.32% and 72.15% for the custom RAG pipeline and 80.65% and 52.94% for the baseline. The paper also claims that its synthetic QA generation and evaluation pipeline, with chain-of-thought reference-text selection, few-shot question styles, custom faithfulness/relevance/specificity scoring, and single-best-answer filtering, produces a reusable dataset that averages 8.16 out of 10 on LLM-based quality metrics.

Load-bearing premise

The reported accuracies stand or fall on the assumption that the LLM-generated multiple-choice labels are correct and that the questions used to teach the model were not also used to grade it, since the paper does not describe either check.

Editorial extensions

If this is right

  • A company preparing an IFRS S1 or S2 sustainability report can receive accurate guidance on metrics, units, and calculations from a chatbot built without any human-labelled QA data.
  • Cross-industry questions are the limiting case: both pipelines lose accuracy there, so future work should focus on multi-document retrieval and reasoning rather than on single-industry factual lookup.
  • Domain-specific fine-tuning improves a compact generator's accuracy even without retrieval, but the best results still depend on supplying the right context, so fine-tuning and context selection are complementary.
  • The 1,063-pair synthetic dataset supplies a reusable evaluation suite for sustainability-reporting QA, filling a gap where no public dataset existed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same two-step recipe, LLM industry classification followed by per-industry context selection, is likely to transfer to other multi-standard regulatory domains, since nothing in it depends on IFRS-specific embeddings or retrieval tuning.
  • The reported 8.16/10 quality score is better read as an upper-bound estimate, because the labels and quality judgments were produced by LLMs without full human verification; a human audit would probably lower the score and could change the reported accuracies.
  • A testable extension would be to run the same generation pipeline on another disclosure framework and check whether the fully LLM-based pipeline again beats the RAG pipeline by roughly the same margin.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper develops a synthetic QA dataset for IFRS sustainability reporting, generated and evaluated through a custom LLM pipeline, and then uses that dataset to build and evaluate two question-answering architectures: a RAG pipeline and a fully LLM-based pipeline, both incorporating an industry classifier and a LoRA fine-tuned Llama 3.1 8B generator. The dataset comprises 1,063 QA pairs with an average LLM-assessed quality score of 8.16/10. The proposed pipelines report MCQ accuracies of 85.32% (RAG) and 93.45% (LLM-based) on single-industry questions and 72.15% and 80.30% on cross-industry questions, respectively, outperforming a baseline RAG configuration.

Significance. If the reported accuracies reflect genuine generalization, the paper would provide a useful domain-specific benchmark and a practical recipe for building knowledge assistants in a document-heavy regulatory domain. The work is systematic in its exploration of chunking, retrieval, query transforms, and generator model choices, and it documents the prompt engineering and evaluation pipeline in sufficient detail for replication. The main scientific value is the dataset itself and the comparison of pipeline architectures; however, the current evaluation design does not yet support the headline generalization claims because the fine-tuned model and classifiers are evaluated on the same QA data used for training and tuning, and the dataset quality scores are produced by the same type of LLM that generated the data, without human validation. These are fixable with a proper held-out split and a human calibration study, but they are load-bearing for the paper's central claims.

major comments (4)
  1. [§4.2.2, §4.3.4, Table 4.7] The paper states in §4.2.2 that Llama 3.1 8B is 'fine-tuned using the final MCQ QA pairs', and then §4.3.4 reports MCQ accuracy on 'the QA dataset' in Table 4.7. No train/test split is described anywhere in the evaluation protocol. This means the headline accuracies (93.45% local, 80.30% cross-industry) are measured on data that may have been seen during LoRA fine-tuning, so the results may reflect memorization rather than generalization. A held-out split (or a clear statement that the fine-tuned model is evaluated only on unseen questions, with the split described) is necessary to support the claim that the proposed pipelines outperform the baseline on new queries.
  2. [§4.2.3, Table 4.6] The industry classifiers (DistilBERT, Random Forest, XGBoost, MLP) are trained on the QA dataset, and Table 4.6 reports their macro F1 scores without describing a train/test split or cross-validation procedure. Because the classifier is a component of both proposed pipelines, its reported performance and the downstream pipeline accuracies are both affected by potential overfitting to the same question set used for evaluation. The paper should specify how the classification training data was partitioned from the evaluation data, and report results on an independent test set.
  3. [§3.3.4, §5.1] The dataset quality claim (average 8.16/10 on faithfulness, relevance, and specificity) rests entirely on LLM-based self-assessment: the same generation model family (Claude 3.5 Sonnet) is used to generate questions and to score them, and the paper itself acknowledges in §5.1 that 'Question quality was checked manually... though this was not done on all 1,063 questions.' Without a human validation study on a representative sample that calibrates the LLM scores against expert judgments, the 'high-quality dataset' claim is not empirically established. The authors should either provide such a calibration study or soften the claim to 'LLM-assessed quality'.
  4. [§3.2.3, §3.2.4] The evaluation pipeline uses thresholds (reference faithfulness/relevance below 0.7 excluded; local questions must score at least 9, cross-industry at least 7) that are chosen based on the same experimental data used to report the final average scores. Since the filtering and the reported quality metrics are derived from the same LLM evaluations on the same questions, the reported average is at least partially a consequence of the filtering rule rather than an independent measurement of dataset quality. An external validation set with human labels, or at least a sensitivity analysis showing that the average 8.16 is robust to threshold choice, would address this circularity concern.
minor comments (4)
  1. [§4.3.2 vs. §4.3.1 and Table 4.1] The model name is inconsistent: §4.3.2 and Table 4.5 refer to 'Llama 3.1 40B', while Table 4.1 and the surrounding text refer to 'Llama 3 70B'. The authors should verify which model was actually used and standardize the naming.
  2. [§4.2.4 and Figure 4.1] The RAG pipeline description states 'The top 5 most relevant chunks are then retrieved from the filtered database using KNN', but for cross-industry queries the classifier may output up to five industries, and it is unclear whether top-5 means five chunks total or five per industry. Clarify the retrieval procedure for multi-industry queries.
  3. [§3.2.4] The similarity filter threshold is described as '0.99% similarity' in the text; this is likely intended as a cosine similarity threshold of 0.99 (or 0.99 on a 0-1 scale). Please clarify the unit and the exact threshold used.
  4. [Table 4.4 and §4.4.1] The multi-query transform is said to 'significantly underperform in local MCQs (48.39%)', but the table shows no error bars or repeated runs, so it is not possible to determine whether this is a real effect or run-to-run variance. Reporting variability (e.g., across multiple seeds or test subsets) would strengthen the comparison.

Circularity Check

1 steps flagged · score 5.0 of 10

Fine-tuned on the same final MCQ set used to report accuracy, with no held-out split: the headline 93.45%/80.30% numbers are substantially in-sample, and the 8.16 dataset-quality score rests on LLM self-assessment.

  1. fitted input called prediction [Section 4.2.2 (Fine-tuning) and Section 4.3.4 / Section 5.1 (evaluation)]
    "The Llama 3.1 8B model was fine-tuned using the final MCQ QA pairs to enhance its performance on multiple-choice questions without RAG ... Specifically, the dataset was used to fine-tune a model for question-answering and evaluate all the methods tested on various levels of complexity."

    The fine-tuned Llama 3.1 8B is a component of both final pipelines (Section 4.2.4). The headline accuracies in Table 4.7 (93.45% local, 80.30% cross-industry MCQ) are computed on the same final MCQ QA pairs used for LoRA fine-tuning, and the paper never describes a train/test split. The model has therefore seen the exact questions and their correct options during training, so the reported improvement over the baseline is partly an in-sample fit rather than a prediction on unseen queries. Table 4.5 (49%/52% without RAG) indicates memorization is not complete, so the result is not wholly forced, but the generalization claim is unverified. This reduces the central accuracy claim to a fitted-input evaluation.

full rationale

The paper's QA-generation pipeline itself is largely self-contained: it starts from IFRS PDFs, generates questions with Claude 3.5 Sonnet under CoT/few-shot prompting, and filters with custom LLM judges; none of those steps is equivalent to the final accuracy numbers. The main circularity is in the evaluation protocol. Section 4.2.2 fine-tunes Llama 3.1 8B on the final MCQ QA pairs, and Section 4.3.4 reports MCQ accuracy on that same dataset without any held-out split; since the fine-tuned model is used in both the custom RAG and fully LLM pipelines, the headline improvements are at least partly in-sample. Section 5.1 explicitly confirms the dual use ('used to fine-tune a model ... and evaluate all the methods tested'). The dataset-quality claim (8.16/10) is also weakened by the paper's own admission that QA evaluation is LLM-based and that human evaluation of all 1,063 questions was not performed; however, that is a validity limitation rather than a derivation that reduces to its inputs. No load-bearing self-citations or imported uniqueness theorems appear. Because the central accuracy claim is evaluated on training data, but the architectures and generation pipeline have independent content, a score of 5 is appropriate.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper relies on hand-chosen thresholds and a temperature setting, plus strong assumptions that LLM judgments reflect human quality and that the synthetic labels are correct. No new physical or conceptual entities are introduced; the claimed inventions are pipeline components and a dataset, all of which are unreleased.

free parameters (6)
  • Reference faithfulness/relevance exclusion threshold = 0.7
    QA pairs whose reference text scores below 0.7 on faithfulness or relevance are discarded before averaging; chosen from manual inspection (Sec. 3.2.3).
  • Quality improvement thresholds = 9 (local), 7 (cross-industry)
    Conditional thresholds for applying the quality improvement function (Sec. 3.3.3).
  • Similarity filter threshold = 0.99
    Questions with similarity above 0.99 are removed as duplicates (Sec. 3.2.4).
  • Generation temperature = 0.5
    Chosen for all final question generation after showing little temperature sensitivity (Sec. 3.4.2).
  • Top K retrieval = 5
    Number of retrieved chunks used in RAG experiments and the custom RAG pipeline (Sec. 4.2.1 and 4.2.4).
  • Few-shot example count = 10-12
    Number of question structures provided in the CoT + few-shot prompt (Sec. 3.2.2).
assumptions (4)
  • domain assumption The parsed IFRS markdown documents accurately represent the official IFRS sustainability standards, including correct handling of crossed-out and underlined text.
    The whole dataset and assistant are built on these 72 markdown files; parsing errors would propagate into every QA pair (Sec. 3.1).
  • ad hoc to paper LLM-based faithfulness, relevance, and domain specificity scores are valid proxies for human judgment of QA quality.
    The dataset quality claim (8.16/10) is computed by LLM evaluators; no human evaluation of the full dataset is performed (Sec. 3.2.3, 5.1).
  • domain assumption The synthetic QA pairs provide correct labels for evaluating the assistant pipelines.
    The pipeline accuracies are measured against these LLM-generated answers, so any incorrect labels directly distort the reported accuracies (Sec. 3.3.4, 4.3.4).
  • domain assumption The industry classifier, whether LLM-prompted or trained, produces correct industry labels for query filtering.
    Both proposed pipelines rely on the industry classifier to restrict retrieval; misclassification would remove relevant chunks (Sec. 4.2.4).

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLMs to Support a Domain Specific Knowledge Assistant." pith.science (2026). https://pith.science/paper/2RZILX4I

@misc{pith2026250204095,
  author       = {Pith},
  title        = {Pith review of: LLMs to Support a Domain Specific Knowledge Assistant},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2RZILX4I}},
  note         = {Machine review of arXiv:2502.04095}
}
read the original abstract

This work presents a custom approach to developing a domain specific knowledge assistant for sustainability reporting using the International Financial Reporting Standards (IFRS). In this domain, there is no publicly available question-answer dataset, which has impeded the development of a high-quality chatbot to support companies with IFRS reporting. The two key contributions of this project therefore are: (1) A high-quality synthetic question-answer (QA) dataset based on IFRS sustainability standards, created using a novel generation and evaluation pipeline leveraging Large Language Models (LLMs). This comprises 1,063 diverse QA pairs that address a wide spectrum of potential user queries in sustainability reporting. Various LLM-based techniques are employed to create the dataset, including chain-of-thought reasoning and few-shot prompting. A custom evaluation framework is developed to assess question and answer quality across multiple dimensions, including faithfulness, relevance, and domain specificity. The dataset averages a score range of 8.16 out of 10 on these metrics. (2) Two architectures for question-answering in the sustainability reporting domain - a RAG pipeline and a fully LLM-based pipeline. The architectures are developed by experimenting, fine-tuning, and training on the QA dataset. The final pipelines feature an LLM fine-tuned on domain specific data and an industry classification component to improve the handling of complex queries. The RAG architecture achieves an accuracy of 85.32% on single-industry and 72.15% on cross-industry multiple-choice questions, outperforming the baseline approach by 4.67 and 19.21 percentage points, respectively. The LLM-based pipeline achieves an accuracy of 93.45% on single-industry and 80.30% on cross-industry multiple-choice questions, an improvement of 12.80 and 27.36 percentage points over the baseline, respectively.

Figures

Figures reproduced from arXiv: 2502.04095 by the authors.

Figure 2.1
Figure 2.1. Basic RAG pipeline. 7 [PITH_FULL_IMAGE:figures/full_fig_p012_2_1.png] view at source ↗
Figure 3.1
Figure 3.1. Multi-modal PDF to markdown parsing pipeline. 19 [PITH_FULL_IMAGE:figures/full_fig_p024_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Faithfulness and relevance QA evaluation pipeline Listing 3.5: Prompt for question faithfulness/relevance evaluation 1 system = " You are an expert question evaluator . Given a question , you assess its faithfulness and relevance relative to the reference text . 2 3 user = f """ 4 Critically evaluate the following question based on the provided context for the relevant { ’ industry ’ if len ( industries ) == 1 else … view at source ↗
Figures from the paper (3 more)
Figure 3.3
Figure 3.3. Figure 3.3: Sample t-SNE visualisation of question embeddings, coloured by industry group 3.3.2 Experimental Results The three question generation methods outlined in Section 3.2.2 are used to gener￾ate a sample of questions for quality evaluation experiments. For each method, t…
Figure 4.1
Figure 4.1. Figure 4.1: Custom RAG pipeline. 52 [PITH_FULL_IMAGE:figures/full_fig_p057_4_1.png]
Figure 4.2
Figure 4.2. Figure 4.2: Custom LLM pipeline. 54 [PITH_FULL_IMAGE:figures/full_fig_p059_4_2.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

78 extracted references · 30 canonical work pages

  1. [1]

    https://www.ey.com/en\_gl/insights/strategy/why-sustainability-has-become-a-corporate-imperative

    work page

  2. [2]

    https://www.ifrs.org/projects/completed-projects/2023/general-sustainability-related-disclosures/, a

    work page 2023

  3. [3]

    Human-computer interaction in customer service: the experience with ai chatbots—a systematic literature review

    Luminița Nicolescu and Monica Teodora Tudorache. Human-computer interaction in customer service: the experience with ai chatbots—a systematic literature review. Electronics, 11 0 (10): 0 1579, 2022

    work page 2022

  4. [4]

    From rag to qa-rag: Integrating generative ai for pharmaceutical regulatory compliance process, 2024

    Jaewoong Kim and Moohong Min. From rag to qa-rag: Integrating generative ai for pharmaceutical regulatory compliance process, 2024. URL https://arxiv.org/abs/2402.01717

    arXiv 2024

  5. [5]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023

    2023

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding, 2019

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019. URL https://arxiv.org/abs/1810.04805

    arXiv 2019

  7. [7]

    Improving language understanding by generative pre-training, 2018

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training, 2018

    2018

  8. [8]

    Siren's song in the ai ocean: a survey on hallucination in large language models

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023 a

    arXiv 2023

Show all 78 references
  1. [9]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay S...

    2024 arXiv

  2. [10]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020

  3. [11]

    Prompt programming for large language models: Beyond the few-shot paradigm

    Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pages 1--7, 2021

    2021

  4. [12]

    Large language models are human-level prompt engineers

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022

    2022 arXiv

  5. [13]

    Autoprompt: Eliciting knowledge from language models with automatically generated prompts

    Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010.15980, 2020

    2010 arXiv

  6. [14]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 0 24824--24837, 2022

    2022

  7. [15]

    Large language models are zero-shot reasoners, 2023

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2023. URL https://arxiv.org/abs/2205.11916

    2023 arXiv

  8. [16]

    Gemba-mqm: Detecting translation quality error spans with gpt-4

    Tom Kocmi and Christian Federmann. Gemba-mqm: Detecting translation quality error spans with gpt-4. arXiv preprint arXiv:2310.13988, 2023

    2023 arXiv

  9. [17]

    Large language models are diverse role-players for summarization evaluation

    Ning Wu, Ming Gong, Linjun Shou, Shining Liang, and Daxin Jiang. Large language models are diverse role-players for summarization evaluation. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 695--707. Springer, 2023 a

    2023

  10. [18]

    Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt, 2023

    Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt, 2023

    2023

  11. [19]

    Human-like summarization evaluation with chatgpt

    Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554, 2023

    2023 arXiv

  12. [20]

    Are large language model-based evaluators the solution to scaling up multilingual evaluation? arXiv preprint arXiv:2309.07462, 2023

    Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language model-based evaluators the solution to scaling up multilingual evaluation? arXiv preprint arXiv:2309.07462, 2023

    2023 arXiv

  13. [21]

    Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models

    Yen-Ting Lin and Yun-Nung Chen. Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. arXiv preprint arXiv:2305.13711, 2023

    2023 arXiv

  14. [22]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021. URL https://arxiv.org/abs/2106.09685

    2021 arXiv

  15. [23]

    Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021

    2021

  16. [24]

    Retrieval-augmented generation for large language models: A survey, 2024

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024

    2024

  17. [25]

    https://platform.openai.com/docs/guides/embeddings

  18. [26]

    https://docs.voyageai.com/docs/embeddings

  19. [27]

    Semantic cosine similarity

    Faisal Rahutomo, Teruaki Kitasuka, Masayoshi Aritsugi, et al. Semantic cosine similarity. In The 7th international student conference on advanced science and technology ICAST, volume 4, page 1. University of Seoul South Korea, 2012

    2012

  20. [28]

    Dense passage retrieval for open-domain question answering, 2020

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering, 2020

    2020

  21. [29]

    Accessed 21 July 2024

    Why your rag is not working? URL https://medium.com/@saurabhgssingh/why-your-rag-is-not-working-96053b4d5305. Accessed 21 July 2024

    2024

  22. [30]

    Least-to-most prompting enables complex reasoning in large language models, 2023

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-most prompting enables complex reasoning in large language models, 2023. URL https://arxiv.org/abs/2205.10625

    2023 arXiv

  23. [31]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311–318, USA, 2002. Association for Computational ...

    2002

  24. [32]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81, 2004

    2004

  25. [33]

    Bertscore: Evaluating text generation with bert

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  26. [34]

    Bleurt: Learning robust metrics for text generation

    Thibault Sellam, Dipanjan Das, and Ankur P Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020

    2004 arXiv

  27. [35]

    Interpretable unified language checking

    Tianhua Zhang, Hongyin Luo, Yung-Sung Chuang, Wei Fang, Luc Gaitskell, Thomas Hartvigsen, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Interpretable unified language checking. arXiv preprint arXiv:2304.03728, 2023 b

    2023 arXiv

  28. [36]

    Factscore: Fine-grained atomic evaluation of factual precision in long form text generation

    Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023

    2023 arXiv

  29. [37]

    Gptscore: Evaluate as you desire

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166, 2023

    2023 arXiv

  30. [38]

    Is chatgpt a good nlg evaluator? a preliminary study

    Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048, 2023 a

    2023 arXiv

  31. [39]

    Ragas: Automated evaluation of retrieval augmented generation

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation. arXiv preprint arXiv:2309.15217, 2023

    2023 arXiv

  32. [40]

    Large language models are not fair evaluators

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators. arXiv preprint arXiv:2305.17926, 2023 b

    2023 arXiv

  33. [41]

    Halueval: A large-scale hallucination evaluation benchmark for large language models

    Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747, 2023

    2023 arXiv

  34. [42]

    Computer-aided generation of multiple-choice tests

    Ruslan Mitkov and Le An Ha. Computer-aided generation of multiple-choice tests. In Proceedings of the HLT-NAACL 03 Workshop on Building Educational Applications Using Natural Language Processing - Volume 2, HLT-NAACL-EDUC '03, page 17–22, USA, 2003. Association for Computation...

    2003

  35. [43]

    The first question generation shared task evaluation challenge, 2010

    Vasile Rus, Brendan Wyse, Paul Piwek, Mihai Lintean, Svetlana Stoyanchev, and Cristian Moldovan. The first question generation shared task evaluation challenge, 2010

    2010

  36. [44]

    Learning to ask: Neural question generation for reading comprehension

    Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. arXiv preprint arXiv:1705.00106, 2017

    2017 arXiv

  37. [45]

    Answer-focused and position-aware neural question generation

    Xingwu Sun, Jing Liu, Yajuan Lyu, Wei He, Yanjun Ma, and Shi Wang. Answer-focused and position-aware neural question generation. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 3930--3939, 2018

    2018

  38. [46]

    Leveraging context information for natural question generation

    Linfeng Song, Zhiguo Wang, Wael Hamza, Yue Zhang, and Daniel Gildea. Leveraging context information for natural question generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...

    2018

  39. [47]

    End-to-end synthetic data generation for domain adaptation of question answering systems

    Siamak Shakeri, Cicero dos Santos, Henghui Zhu, Patrick Ng, Feng Nan, Zhiguo Wang, Ramesh Nallapati, and Bing Xiang. End-to-end synthetic data generation for domain adaptation of question answering systems. In Proceedings of the 2020 Conference on Empirical Methods in Natural ...

    2020

  40. [48]

    Synthetic qa corpora generation with roundtrip consistency

    Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. Synthetic qa corpora generation with roundtrip consistency. arXiv preprint arXiv:1906.05416, 2019

    1906 arXiv

  41. [49]

    Training question answering models from synthetic data

    Raul Puri, Ryan Spring, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. Training question answering models from synthetic data. arXiv preprint arXiv:2002.09599, 2020

    2002 arXiv

  42. [50]

    Asking questions the human way: Scalable question-answer generation from text corpus

    Bang Liu, Haojie Wei, Di Niu, Haolan Chen, and Yancheng He. Asking questions the human way: Scalable question-answer generation from text corpus. In Proceedings of The Web Conference 2020, pages 2032--2043, 2020

    2020

  43. [51]

    Improving question answering model robustness with synthetic adversarial data generation

    Max Bartolo, Tristan Thrush, Robin Jia, Sebastian Riedel, Pontus Stenetorp, and Douwe Kiela. Improving question answering model robustness with synthetic adversarial data generation. arXiv preprint arXiv:2104.08678, 2021

    2021 arXiv

  44. [52]

    Generative ai for synthetic data generation: Methods, challenges and the future, 2024

    Xu Guo and Yiqiang Chen. Generative ai for synthetic data generation: Methods, challenges and the future, 2024. URL https://arxiv.org/abs/2403.04190

    2024 arXiv

  45. [53]

    Multilingual generation and answering of questions from texts and knowledge graphs

    Kelvin Han and Claire Gardent. Multilingual generation and answering of questions from texts and knowledge graphs. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), pages 13740--13756. Association for Computational Linguistics, 2023

    2023

  46. [54]

    https://www.confident-ai.com/blog/the-definitive-guide-to-synthetic-data-generation-using-llms, a

  47. [55]

    https://mlflow.org/docs/latest/llms/rag/notebooks/question-generation-retrieval-evaluation.html

  48. [56]

    Realm: Retrieval-augmented language model pre-training, 2020

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training, 2020

    2020

  49. [57]

    Rae, Erich Elsen, and Laurent Sifre

    Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore,...

    2022 arXiv

  50. [58]

    Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering, 2022

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering, 2022. URL https://arxiv.org/abs/2210.02627

    2022 arXiv

  51. [59]

    Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E

    Tianjun Zhang, Shishir G. Patil, Naman Jain, Sheng Shen, Matei Zaharia, Ion Stoica, and Joseph E. Gonzalez. Raft: Adapting language model to domain specific rag, 2024. URL https://arxiv.org/abs/2403.10131

    2024 arXiv

  52. [60]

    Finqa: A dataset of numerical reasoning over financial data, 2022

    Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, and William Yang Wang. Finqa: A dataset of numerical reasoning over financial data, 2022

    2022

  53. [61]

    Financebench: A new benchmark for financial question answering

    Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. Financebench: A new benchmark for financial question answering. arXiv preprint arXiv:2311.11944, 2023

    2023 arXiv

  54. [62]

    Bloomberggpt: A large language model for finance, 2023 b

    Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance, 2023 b . URL https://arxiv.org/abs/2303.17564

    2023 arXiv

  55. [63]

    Benchmarking retrieval-augmented generation for medicine, 2024

    Guangzhi Xiong, Qiao Jin, Zhiyong Lu, and Aidong Zhang. Benchmarking retrieval-augmented generation for medicine, 2024. URL https://arxiv.org/abs/2402.13178

    2024 arXiv

  56. [64]

    https://www.ifrs.org/use-around-the-world/use-of-ifrs-standards-by-jurisdiction/\#analysis-of-use-of-ifrs-accounting-standards-around-the-world, b

  57. [65]

    https://www.ifrs.org/issued-standards/ifrs-sustainability-standards-navigator/ifrs-s1-general-requirements/, c

  58. [66]

    https://www.ifrs.org/issued-standards/ifrs-sustainability-standards-navigator/ifrs-s2-climate-related-disclosures/, d

  59. [67]

    https://www.ifrs.org/projects/completed-projects/2023/climate-related-disclosures/appendix-b-industry-based-disclosure-requirements/, e

    2023

  60. [68]

    https://sasb.ifrs.org/company-use/sasb-reporters/

  61. [69]

    https://docs.llamaindex.ai/en/stable/module\_guides/evaluating/

  62. [70]

    https://github.com/confident-ai/deepeval, b

  63. [71]

    Can large multimodal models uncover deep semantics behind images? arXiv preprint arXiv:2402.11281, 2024

    Yixin Yang, Zheng Li, Qingxiu Dong, Heming Xia, and Zhifang Sui. Can large multimodal models uncover deep semantics behind images? arXiv preprint arXiv:2402.11281, 2024

    2024 arXiv

  64. [72]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. URL https://arxiv.org/abs/1701.06538

    2017 arXiv

  65. [73]

    Lost in the middle: How language models use long contexts

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12: 0 157--173, 2024

    2024

  66. [74]

    https://lmsys.org/blog/2023-06-29-longchat/

    2023

  67. [75]

    Financial report chunking for effective retrieval augmented generation

    Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde, and Leah Li. Financial report chunking for effective retrieval augmented generation. arXiv preprint arXiv:2402.05131, 2024

    2024 arXiv

  68. [76]

    Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers

    Kunal Sawarkar, Abhilasha Mangal, and Shivam Raj Solanki. Blended rag: Improving rag (retriever-augmented generation) accuracy with semantic search and hybrid query-based retrievers. arXiv preprint arXiv:2404.07220, 2024

    2024 arXiv

  69. [77]

    Precise zero-shot dense retrieval without relevance labels

    Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. Precise zero-shot dense retrieval without relevance labels. arXiv preprint arXiv:2212.10496, 2022

    2022 arXiv

  70. [78]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2020. URL https://arxiv.org/abs/1910.01108

    2020 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.