Pith. sign in

REVIEW 5 major objections 7 minor 38 references

RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems

T0 review · 5 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A modular RAG testbed shows hybrid vector-keyword retrieval plus structured self-evaluation prompting improves performance, with Qwen 2.5 reaching a 72.7% pass rate on the authors' nine-metric evaluation.

desk verdict Useful local RAG evaluation harness with a believable hybrid-search result, but the headline 72.7% pass rate is not independently anchored until the metric thresholds and LLM judge are validated. read the letter →

arxiv 2412.12322 v1 pith:PXHXEMG2 submitted 2024-12-16 cs.LG cs.AIcs.CLcs.IR

Ioannis Papadimitriou , Ilias Gialampoukidis , Stefanos Vrochidis , Ioannis (Yiannis) Kompatsiaris This is my paper · ORCID
classification cs.LGcs.AIcs.CLcs.IR
keywords Retrieval-AugmentedGenerationhybridretrievalvectorsearchkeywordpromptengineeringreasoning-and-actingagentsRAGevaluationmetricscompletenessgain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

RAG Playground is an open-source testbed that asks a practical question: which retrieval strategy and prompt design actually improve retrieval-augmented generation? The paper claims that hybrid search, combining dense vector similarity with keyword matching and a reranking step, outperforms naive vector search and vector-plus-reranking alone, and that a reasoning-and-acting agent prompted to self-evaluate its steps adds further gains. On a curated set of 319 question-answer pairs scored by a weighted nine-metric framework, the best configuration (hybrid retrieval, custom self-evaluation prompt, Qwen 2.5) reaches a 72.7% pass rate, versus 63.0% for the same model with naive vector search. The authors also introduce a "completeness gain" metric that credits systems for finding information beyond the human ground truth. The broader point the paper is trying to establish is that retrieval strategy and prompt engineering can move RAG performance more than the choice of language model does.

What carries the argument

The load-bearing object is the RAG Playground evaluation pipeline itself. It implements three retrieval strategies — naive vector search, vector search with cross-encoder reranking, and hybrid vector-keyword search with union combination and reranking — and two prompt variants of a reasoning-and-acting agent: the base version and a structured self-evaluation version that forces explicit step-by-step reasoning, confidence scoring, and search-result-quality analysis. The evaluation layer combines programmatic metrics (key-terms precision, token recall), LLM-based metrics (truthfulness, completeness, source relevance, context faithfulness), and hybrid metrics (semantic F1, answer relevance, and the novel completeness gain), with weights summing so that programmatic metrics carry 25%, LLM-based 45%, and hybrid 30% of the weighted total. A response passes if at least six of eight primary metrics clear their thresholds; completeness gain is tracked separately. This metric machinery is what turns the qualitative claim "hybrid and self-evaluation are better" into the quantitative 72.7% pass rate.

What would settle it

Take a random sample of responses from the best configuration (hybrid plus self-evaluation with Qwen 2.5) and the naive-vector baseline, and have independent human raters score them for factual accuracy; the central claim would be falsified if humans do not rate the best configuration at least as accurate as the baseline, or if the framework's metric scores correlate weakly (e.g., rank correlation below 0.3) with human ratings on the same responses.

Watch

Extended reading notes

Core claim

The central claim is that RAG quality is governed less by model choice alone and more by how retrieval is combined and how the agent is prompted. Concretely, the paper reports that a hybrid retriever that unions dense-vector and BM25-style keyword results and then reranks to the top four chunks, paired with a reasoning-and-acting agent whose prompt forces step-by-step reasoning, confidence scores, and analysis of search-result quality, yields the highest performance across all configurations tested. With Qwen 2.5, this combination achieves a 72.7% pass rate on the authors' weighted nine-metric evaluation (passing at least six of eight primary metrics plus a separate numerical-accuracy check), compared with 63.0% for naive vector search and 66.5% for vector search with reranking. The same prompt engineering lifts Llama 3.1 from 62.9% to 66.1% in overall pass rate, and improves numerical accuracy by 5 to 10 percentage points across configurations. The paper further claims that its locally runnable evaluation framework, including the new completeness gain metric, makes these comparisons reproducible at roughly four dollars of electricity per full evaluation suite.

Load-bearing premise

The entire pass-rate comparison rests on the assumption that the framework's nine weighted metrics and their thresholds actually measure answer quality; the authors set the weights and thresholds through their own empirical testing, with no human rating or external benchmark used for calibration, so if those metrics do not track true quality the 72.7% figure is not meaningful.

Editorial extensions

If this is right

  • Hybrid vector-keyword retrieval with reranking should be the default retrieval configuration in RAG systems, since it outperformed both single-strategy baselines on nearly every metric in the paper's experiments.
  • Structured self-evaluation prompting is a near-zero-cost optimization: it raised pass rates by 3 to 5 percentage points and numerical accuracy by up to 10 points compared with base prompting.
  • RAG evaluation can be run locally and cheaply (about $4 of electricity per full suite), which makes systematic ablations and reproducibility practical on consumer hardware.
  • The completeness gain metric can identify responses that add correct information beyond the curated ground truth, offering a way to credit systems for going beyond the reference answer.
  • For question-answering workloads of the kind in the 319-pair dataset, the larger Qwen 2.5 model outperformed Llama 3.1 in every configuration, with the gap largest in numerical accuracy (83.9% versus 72.7%).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the framework's metrics track human judgment, the relative ordering of configurations should generalize to other document collections, but the absolute pass-rate numbers are framework-relative and should not be read as an absolute accuracy measure.
  • The structured self-evaluation prompt bundles several changes (step-by-step reasoning, confidence scoring, and result-quality analysis); the paper does not isolate which component drives the gain, so an ablation would be a natural test of the mechanism.
  • Because all configurations share the same fixed chunk size and overlap, the reported gains from hybrid search might change under adaptive chunking; the paper itself flags dynamic chunking as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper introduces RAG Playground, an open-source framework for evaluating retrieval strategies and prompt engineering in RAG systems. It implements three retrieval approaches (naive vector search, reranking, and hybrid vector-keyword search) and two ReAct prompting variants (base and structured self-evaluation), and evaluates them with Llama 3.1 8B and Qwen 2.5 14B on 319 curated QA pairs. The evaluation uses a weighted suite of programmatic, LLM-based, and hybrid metrics, with a pass criterion requiring 6 of 8 primary metrics. The main empirical claim is that hybrid search combined with structured self-evaluation prompting achieves up to a 72.7% pass rate with Qwen 2.5, and that this represents a 15.4% relative improvement over naive vector search. The framework is reported to run locally on consumer hardware at roughly $4 per full evaluation suite.

Significance. If the evaluation framework is valid, the paper provides a useful, low-cost, open-source tool for comparing RAG configurations, and the empirical comparisons between retrieval strategies and prompting variants would be practically valuable. The modular design, explicit metric definitions, and reproducible local setup are genuine strengths. However, the central quantitative claim rests entirely on an evaluation framework whose thresholds and weights were tuned on the same dataset and never validated against human judgment or an external benchmark. The absence of statistical significance testing and the unspecified LLM judge model further weaken the empirical conclusions. These are fixable in revision, but as presented the 72.7% pass rate and the relative comparisons are not yet supported independently.

major comments (5)
  1. [Section 4.1, Section 4.2.2] The headline pass rates are defined entirely by the authors' own evaluation thresholds and weights, which Section 4.1 states were 'established through empirical testing' on the same dataset. There is no independent anchor: no human ratings, no held-out calibration split, and no comparison against an established RAG benchmark. Consequently the absolute 72.7% pass rate (Section 6.1.1) and the 15.4% relative improvement (Section 7.1) are threshold-dependent rather than externally validated. I request a validation study of the metric suite against human judgments and a sensitivity analysis showing how pass rates move under reasonable threshold variations.
  2. [Section 4.1.2] The truthfulness, completeness, source relevance, and context faithfulness metrics are computed via 'structured LLM prompting', but the judge model is never identified. If the judge is the same model being evaluated (Qwen 2.5 or Llama 3.1), then the differences between configurations might reflect self-preference or prompt-sensitivity of the judge rather than answer quality. Please specify the judge model, its prompt, and its temperature, and run at least one configuration with a different, independent judge model to test robustness.
  3. [Section 6.1, Section 7.1] No statistical significance testing is reported for any of the pass-rate differences. With only 319 QA pairs, the gap between 63.0% and 72.7% (or between 62.9% and 66.1% for prompt engineering) could plausibly arise from evaluation noise. Report paired bootstrap confidence intervals or a suitable significance test (e.g., McNemar or permutation test over QA pairs) for each comparison, with correction for multiple comparisons across the configurations.
  4. [Section 4.2.1] The completeness gain metric uses a threshold of 0.501, which is just above the neutral 0.5 point. This effectively passes any response with a positive gain over ground-truth coverage, and its weight is only 0.05 while it is excluded from the 6-of-8 primary-pass requirement. The paper should justify this threshold choice independently and clarify what marginal contribution the metric makes to the aggregate score; otherwise the metric appears to be tuned to maximize pass rates rather than to measure quality.
  5. [Section 7.3] The limitations subsection acknowledges dataset size and fixed chunk sizes, but it does not acknowledge the strongest limitation: the evaluation framework itself has not been validated against human judgment or an external benchmark. The validity of every quantitative finding in Sections 6 and 7 depends on that validation, so the limitations discussion should state this directly and the experiments should include such a check.
minor comments (7)
  1. [Section 1.2] In the fourth bullet of Section 1.2 there is an extra space before the period in 'challenges . [8], [9]'.
  2. [Section 3.2.2] The paper shows only a partial prompt template; please include the full system prompts for all variants in an appendix so the prompt engineering results are reproducible.
  3. [Section 3.2.3] The 'Context Engineering for Retrieval' component is described but not explicitly distinguished from the structured self-evaluation prompting in the experiments; clarify whether it is a separate configuration or a subpart of the custom ReAct agent.
  4. [Section 4.1.2] The source relevance metric is described as employing an 80/20 weighting scheme, but the text does not define what the 80/20 split applies to; please clarify the formula.
  5. [Section 6.2.2] Correlation coefficients are reported without confidence intervals or significance tests; please add them or temper the claims.
  6. [References] Reference [25] contains a broken line break in the URL; please fix the citation.
  7. [Section 7.1] The sentence 'This was expected as the Qwen model is 75% bigger...' attributes the performance gap to model size, but no ablation or controlled comparison supports this causal claim; rephrase as a hypothesis.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity; only a minor self-definitional completeness-gain threshold.

  1. self definitional [Section 4.2.1 (Completeness Gain Metric)]
    "A key innovation in our framework is the completeness gain metric (weight: 0.05, threshold: 0.501), which evaluates response quality relative to the full available context rather than just the ground truth."

    The pass threshold for completeness gain is set at 0.501, while the metric's own definition in Section 4.3 states that 0.5 indicates equal coverage to ground truth and scores above 0.5 represent improved coverage. Thus passing this metric is true by construction for any response that covers even one additional point beyond the ground truth; the threshold does not establish an independent standard of improvement but merely restates the metric's internal definition. This is a genuine definitional shortcut, but it is minor because the metric has weight 0.05 and is explicitly excluded from the 6-of-8 primary-metric pass criterion, so it does not drive the headline 72.7% pass rate.

full rationale

The paper does not derive a first-principles result; it reports an ablation study under its own evaluation framework. The central comparison (hybrid > reranking > vector; custom ReAct > base ReAct) is a direct empirical measurement, and no fitted parameter is renamed as a prediction. The thresholds in Section 4.1 were 'established through empirical testing' without a human-calibration or held-out split, which limits the external interpretability of the absolute 72.7% pass rate, but this is a validity limitation rather than a circular reduction: the same thresholds are applied to all configurations, so the relative ordering is not forced by construction. The only definitional shortcut is the completeness-gain threshold (0.501), set just above the metric's own parity definition (0.5), making a pass equivalent to any positive gain; this metric is minor and excluded from the primary pass criterion. The paper itself acknowledges dataset-size and parameter-exploration limits in Section 7.3 and notes in Section 2.4 that human/automated hybrid validation is valuable for validating the evaluation framework, but that validation is not reported. No self-citation chains or imported uniqueness arguments appear, so overall circularity is minor.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The framework's evaluation results depend on hand-tuned weights, thresholds, and retrieval hyperparameters, and on unvalidated assumptions about the reliability of LLM judges and the representativeness of the self-curated dataset. The completeness gain threshold of 0.501 is the most explicit example of a parameter chosen to produce pass outcomes.

free parameters (5)
  • Evaluation metric weights = 0.15, 0.15, 0.2, 0.1, 0.05, 0.1, 0.1, 0.1, 0.05; category weights 25/45/30
    Weights for the nine metrics and three categories are hand-selected to reflect 'relative importance and reliability' (Section 4.1). No sensitivity analysis is provided.
  • Metric pass thresholds = 0.7 for most, 0.6 for semantic F1, 0.501 for completeness gain
    Thresholds were 'established through empirical testing' (Section 4.1) on the same dataset, so the pass/fail variable is fitted to make results look sensible.
  • Chunk size and overlap = 256 tokens, 50 overlap
    Chosen from preliminary experimentation (Section 5.3.1); no ablation shows sensitivity.
  • Retrieval top-k = k=4 for naive, k=20 then rerank to 4 for hybrid
    Set by hand; not systematically varied.
  • Completeness gain pass threshold = 0.501
    A threshold just above the neutral 0.5, so any positive relative coverage counts as a pass; this is effectively a permissive hand-set parameter.
assumptions (4)
  • domain assumption LLM-based metrics provide valid measures of truthfulness, completeness, source relevance, and context faithfulness
    Section 4.1.2 uses LLM judges but never validates them against human ratings or established benchmarks.
  • domain assumption The 319 QA pair dataset is representative of real-world RAG queries
    Section 5.1 describes a curation process but provides no distribution analysis or comparison to public benchmarks.
  • ad hoc to paper A response passing 6 of 8 primary metrics is 'successful'
    The aggregation rule in Section 4.2.2 is arbitrary and not derived from any task objective.
  • ad hoc to paper Semantic F1 and answer relevance hybrid metrics are meaningful
    These combine embeddings and LLM judgment without any calibration; they are treated as ground truth.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems." pith.science (2026). https://pith.science/paper/PXHXEMG2

@misc{pith2026241212322,
  author       = {Pith},
  title        = {Pith review of: RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PXHXEMG2}},
  note         = {Machine review of arXiv:2412.12322}
}
read the original abstract

We present RAG Playground, an open-source framework for systematic evaluation of Retrieval-Augmented Generation (RAG) systems. The framework implements and compares three retrieval approaches: naive vector search, reranking, and hybrid vector-keyword search, combined with ReAct agents using different prompting strategies. We introduce a comprehensive evaluation framework with novel metrics and provide empirical results comparing different language models (Llama 3.1 and Qwen 2.5) across various retrieval configurations. Our experiments demonstrate significant performance improvements through hybrid search methods and structured self-evaluation prompting, achieving up to 72.7% pass rate on our multi-metric evaluation framework. The results also highlight the importance of prompt engineering in RAG systems, with our custom-prompted agents showing consistent improvements in retrieval accuracy and response quality.

Figures

Figures reproduced from arXiv: 2412.12322 by the authors.

Figure 1
Figure 1. Performance analysis across different configurations and metrics. Top: Overall performance metrics including mean scores and pass rates. [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 12 canonical work pages

  1. [1]

    Language models are open knowledge graphs,

    C. Wang, X. Liu, and D. Song, “Language models are open knowledge graphs,” arXiv preprint arXiv:2010.11967, 2020

    arXiv 2010

  2. [2]

    Retrieval-augmented generation for large language models: A survey,

    Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997, 2023

    arXiv 2023

  3. [3]

    Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection,

    A. Asai, Z. Wu, Y. Wang, A. Sil, and H. Hajishirzi, “Self-rag: Learn- ing to retrieve, generate, and critique through self-reflection,” arXiv preprint arXiv:2310.11511, 2023

    arXiv 2023

  4. [4]

    Rarr: Researching and revising what language models say, using language models,

    L. Gao, Z. Dai, P . Pasupat, A. Chen, A. T. Chaganty, Y. Fan, V . Y. Zhao, N. Lao, H. Lee, D.-C. Juan et al. , “Rarr: Researching and revising what language models say, using language models,” arXiv preprint arXiv:2210.08726, 2022

    arXiv

  5. [5]

    Lever- aging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach,

    S. Kuzi, M. Zhang, C. Li, M. Bendersky, and M. Najork, “Lever- aging semantic and lexical matching to improve the recall of document retrieval systems: A hybrid approach,” arXiv preprint arXiv:2010.01195, 2020

    arXiv 2010

  6. [6]

    Neural information retrieval: A literature review,

    Y. Zhang, M. M. Rahman, A. Braylan, B. Dang, H.-L. Chang, H. Kim, Q. McNamara, A. Angert, E. Banner, V . Khetan et al. , “Neural information retrieval: A literature review,” arXiv preprint arXiv:1611.06792, 2016

    arXiv 2016

  7. [7]

    Colbert: Efficient and effective pas- sage search via contextualized late interaction over bert,

    O. Khattab and M. Zaharia, “Colbert: Efficient and effective pas- sage search via contextualized late interaction over bert,” in Pro- ceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, 2020, pp. 39–48

    work page 2020

  8. [8]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P . Liang, “Lost in the middle: How language models use long contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    work page 2024

Show all 38 references
  1. [9]

    Challenges and applications of large language mod- els,

    J. Kaddour, J. Harris, M. Mozes, H. Bradley, R. Raileanu, and R. McHardy, “Challenges and applications of large language mod- els,” arXiv preprint arXiv:2307.10169, 2023

    2023 arXiv

  2. [10]

    Ragas: Automated evaluation of retrieval augmented generation,

    S. Es, J. James, L. Espinosa-Anke, and S. Schockaert, “Ragas: Automated evaluation of retrieval augmented generation,” arXiv preprint arXiv:2309.15217, 2023

    2023 arXiv

  3. [11]

    Benchmarking large language models in retrieval-augmented generation,

    J. Chen, H. Lin, X. Han, and L. Sun, “Benchmarking large language models in retrieval-augmented generation,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 16, 2024, pp. 17 754–17 762

    2024

  4. [12]

    Switchprompt: Learning domain-specific gated soft prompts for classification in low-resource domains. corr, abs/2302.06868,

    K. Goswami, L. Lange, J. Araki, and H. Adel, “Switchprompt: Learning domain-specific gated soft prompts for classification in low-resource domains. corr, abs/2302.06868,” 2023

    2023 arXiv

  5. [13]

    Retrieval aug- mented language model pre-training,

    K. Guu, K. Lee, Z. Tung, P . Pasupat, and M. Chang, “Retrieval aug- mented language model pre-training,” inInternational conference on machine learning. PMLR, 2020, pp. 3929–3938

    2020

  6. [14]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P . Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Ad- vances in Neural Information Processing Systems , vol. 33, pp. 9459– 9474, 2020

    2020

  7. [15]

    Sparse, dense, and attentional representations for text retrieval,

    Y. Luan, J. Eisenstein, K. Toutanova, and M. Collins, “Sparse, dense, and attentional representations for text retrieval,” Trans- actions of the Association for Computational Linguistics , vol. 9, pp. 329–345, 2021

    2021

  8. [16]

    Retrieval, re-ranking and multi-task learning for knowledge-base question answering,

    Z. Wang, P . Ng, R. Nallapati, and B. Xiang, “Retrieval, re-ranking and multi-task learning for knowledge-base question answering,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume , 2021, pp. 347–357

    2021

  9. [17]

    Introducing a new hyper- parameter for rag: Context window utilization,

    K. Juvekar and A. Purwar, “Introducing a new hyper- parameter for rag: Context window utilization,” arXiv preprint arXiv:2407.19794, 2024

    2024 arXiv

  10. [18]

    The chronicles of rag: The retriever, the chunk and the generator,

    P . Finardi, L. Avila, R. Castaldoni, P . Gengo, C. Larcher, M. Piau, P . Costa, and V . Carid´a, “The chronicles of rag: The retriever, the chunk and the generator,” arXiv preprint arXiv:2401.07883, 2024

    2024 arXiv

  11. [19]

    LlamaIndex: https://docs.llamaindex.ai/en/stable/

  12. [20]

    OpenAI Evals: https://github.com/openai/evals

  13. [21]

    Rageval: Scenario specific rag evaluation dataset generation framework,

    K. Zhu, Y. Luo, D. Xu, R. Wang, S. Yu, S. Wang, Y. Yan, Z. Liu, X. Han, Z. Liu et al. , “Rageval: Scenario specific rag evaluation dataset generation framework,” arXiv preprint arXiv:2408.01262 , 2024

    2024 arXiv

  14. [22]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  15. [23]

    Prompt engineering in large language models,

    G. Marvin, N. Hellen, D. Jjingo, and J. Nakatumba-Nabende, “Prompt engineering in large language models,” in International 10 conference on data intelligence and cognitive informatics . Springer, 2023, pp. 387–402

    2023

  16. [24]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

  17. [25]

    Tree of thoughts: Deliberate problem solving with large language models, 2023,

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y. Cao, and K. Narasimhan, “Tree of thoughts: Deliberate problem solving with large language models, 2023,” URL https://arxiv. org/pdf/2305.10601. pdf, 2023

    2023 arXiv

  18. [26]

    Bertscore: Evaluating text generation with bert,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” arXiv preprint arXiv:1904.09675, 2019

    1904 arXiv

  19. [27]

    Faith- fulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods,

    W. Li, W. Wu, M. Chen, J. Liu, X. Xiao, and H. Wu, “Faith- fulness in natural language generation: A systematic survey of analysis, evaluation and optimization methods,” arXiv preprint arXiv:2203.05227, 2022

    2022 arXiv

  20. [28]

    Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models,

    Y.-T. Lin and Y.-N. Chen, “Llm-eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models,” arXiv preprint arXiv:2305.13711, 2023

    2023 arXiv

  21. [29]

    Mix-of- granularity: Optimize the chunking granularity for retrieval- augmented generation,

    Z. Zhong, H. Liu, X. Cui, X. Zhang, and Z. Qin, “Mix-of- granularity: Optimize the chunking granularity for retrieval- augmented generation,” arXiv preprint arXiv:2406.00456, 2024

    2024 arXiv

  22. [30]

    Dense text retrieval based on pretrained language models: A survey,

    W. X. Zhao, J. Liu, R. Ren, and J.-R. Wen, “Dense text retrieval based on pretrained language models: A survey,” ACM Transac- tions on Information Systems, vol. 42, no. 4, pp. 1–60, 2024

    2024

  23. [31]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  24. [32]

    Simcse: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “Simcse: Simple contrastive learning of sentence embeddings,” arXiv preprint arXiv:2104.08821, 2021

    2021 arXiv

  25. [33]

    Transformer memory as a dif- ferentiable search index,

    Y. Tay, V . Tran, M. Dehghani, J. Ni, D. Bahri, H. Mehta, Z. Qin, K. Hui, Z. Zhao, J. Gupta et al. , “Transformer memory as a dif- ferentiable search index,” Advances in Neural Information Processing Systems, vol. 35, pp. 21 831–21 843, 2022

    2022

  26. [34]

    https://huggingface.co/BAAI/bge-base-en-v1.5

  27. [35]

    https://huggingface.co/cross-encoder/ms-marco-MiniLM-L-2-v2

  28. [36]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Let- man, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  29. [37]

    Qwen2.5: A party of foundation models,

    Q. Team, “Qwen2.5: A party of foundation models,” September

  30. [2024]

    Available: https://qwenlm.github.io/blog/qwen2

    [Online]. Available: https://qwenlm.github.io/blog/qwen2. 5/

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.