Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework

Eoin Thomas; Hongliu Cao; Ilias Driouich

REVIEW 3 major objections 6 minor 33 references

Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A three-agent pipeline claims to generate synthetic QA pairs for RAG evaluation with higher semantic diversity than evolutionary or direct-prompting baselines, while masking sensitive entities across personal, workplace, and medical domains

desk verdict The pipeline is sensible and the diversity results are directionally encouraging, but the paper never tests whether the final QA pairs leak masked entities, so the central privacy claim is currently unsupported. read the letter →

arxiv 2508.18929 v1 pith:YGBKI3PA submitted 2025-08-26 cs.CL cs.AI

Ilias Driouich , Hongliu Cao , Eoin Thomas This is my paper

classification cs.CLcs.AI

keywords syntheticQAgenerationRAGevaluationmulti-agentframeworksemanticdiversityPIIpseudonymizationprivacy-preservingdataEUAIActLLM-as-a-judge

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Retrieval-augmented generation (RAG) is only as trustworthy as the datasets used to evaluate it, yet most benchmarks ignore topical coverage and privacy. This paper claims that a three-agent pipeline—clustering-based sample selection, sensitive-entity pseudonymization, and QA generation from masked text—produces synthetic question–answer sets that are more semantically diverse than existing evolutionary or direct-prompting generators, while masking personal, workplace, and medical entities with high per-label accuracy. The authors test diversity on the EU AI Act corpus using an LLM judge and a cosine-similarity metric, and test masking on three domain-specific datasets. If correct, it offers a practical recipe for building evaluation sets that reflect real-world breadth and reduce exposure of sensitive data, supporting compliance-minded AI auditing. The load-bearing caveat is that the paper checks masking on the input text, not on the final QA pairs.

What carries the argument

The carrying mechanism is the three-stage pipeline itself. First, a Diversity agent applies k-means clustering to semantic embeddings and picks representative samples per cluster, aiming for wide topic coverage. Second, a Privacy agent detects sensitive entities according to predefined categories, pseudonymizes them with context-aware replacements, and logs a structured privacy report. Third, a QA Curation agent generates question–answer pairs from the pseudonymized samples, producing evaluation-ready ground truth. The pipeline is modular: each stage can be swapped or extended independently.

What would settle it

Run a prompt-extraction or membership-inference attack on the generated QA pairs themselves, not on the input text, to see whether original masked entities (names, salaries, diagnoses, and so on) can be recovered. If original entities appear in or are inferable from the QA pairs, the paper's stronger privacy claim fails; if they cannot be recovered, the claim holds.

Watch

Extended reading notes

Core claim

The paper claims that decomposing synthetic QA dataset creation into three specialized agents—topic-diverse representative sampling via clustering, entity-level pseudonymization, and QA generation from the masked texts—yields evaluation sets that are semantically more diverse than evolutionary and direct-prompting alternatives, while masking sensitive entities at high label-level accuracy across personal, workplace, and medical domains. The reported diversity scores rise with test set size (from 7.8 at 10 samples to 9.0 at 100 samples on an LLM judge scale), and cosine-similarity-to-diversity moves consistently toward zero, indicating greater semantic spread. Privacy masking accuracy per ent

Load-bearing premise

The privacy claim rests on the untested assumption that generating questions from pseudonymized text cannot reintroduce or infer the masked entities in the final QA pairs.

Editorial extensions

If this is right

Larger synthetic QA sets generated this way keep gaining topic diversity, so scaling evaluation sets is a practical route to broader coverage.
The pipeline can be applied to domain-specific corpora to produce evaluation sets without exposing the underlying sensitive fields.
The modular agent design means a change in clustering or masking strategy can be dropped in without redesigning the whole generation process.
The twofold evaluation recipe—LLM judge plus cosine-similarity diversity, plus per-entity masking accuracy—offers a repeatable protocol for comparing synthetic QA generators.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The strongest privacy claim—that the QA pairs themselves are private—is not actually tested; only input-text masking is measured, and the conclusion defers attack-resistance testing to future work.
LLM-as-a-judge diversity scores may partly reflect the judge model's own preferences; correlating these scores with retrieval or answer quality on downstream RAG tasks would test whether the diversity is functionally useful.
Because representative sampling is cluster-based, rare or outlier topics may be under-sampled even when average diversity improves; a coverage check on low-density clusters would clarify this.
Formal guarantees such as differential privacy could be layered onto pseudonymization if regulations require more than entity masking.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The multi-agent pipeline is a reasonable extension of RAGAS-style synthetic QA generation: cluster for topical coverage, pseudonymize, then generate QA pairs. Using the EU AI Act as a testbed and comparing against evolutionary and direct-prompting baselines is a sensible setup, and the label-wise masking accuracies across PWI/PHI/PII give useful detail. The diversity gains over baselines are consistent across dataset sizes, which is a good sign. I credit the authors for a modular design and for surfacing a structured privacy report.

The soft spots are real, and one is load-bearing. The paper claims to produce 'private synthetic QA datasets,' but the experiments only measure masking accuracy on the input texts (Sec. 4.3.2). The actual deliverable is the QA pairs generated by the curator from the pseudonymized text. No test checks whether those pairs contain or permit recovery of originally masked entities. The authors themselves list attack-resistance testing as future work in Sec. 5, which confirms the gap. Given that label-wise accuracies run between 0.75 and 0.90, a nontrivial fraction of entities are never masked; those can be copied verbatim into questions or answers. Even correctly pseudonymized entities could be reintroduced via contextual inference or parametric knowledge. Measuring masking accuracy on input text cannot substitute for leakage testing on the generated set. This is an addressable flaw, but it means the central privacy claim is not established.

The diversity evaluation has its own weaknesses: a single LLM judge, no error bars or multiple runs, and no human validation. The cosine metric description is internally inconsistent as written. The authors say lower values indicate greater semantic spread, but then interpret scores moving closer to zero (e.g., from -0.36 to -0.15) as reflecting that questions are 'increasingly dissimilar.' Closer to zero means higher cosine similarity, so the opposite. Either the metric is defined differently or the interpretation is backwards; the table can't be read reliably without a fix. This is a minor-to-moderate reporting issue, but it undermines confidence in the quantitative diversity claim. No prompts or code are released, which is not fatal for a workshop paper but would help.

There is no fitted-parameter circularity: the hyperparameters are choices, not tuned to the results. The self-citation to the prior multi-agent judge work is contextual and not a problem.

Who is this for? People building synthetic evaluation sets for RAG in privacy-sensitive domains. The paper deserves a serious referee, not a desk reject, because the framework is plausible and the privacy gap is testable. I would send it to peer review with the clear expectation of major revision: add leakage tests on the final QA pairs, fix the metric description, and include variance or multiple judge runs. With those changes, the claims could be supported.

Referee Report

3 major / 6 minor

Summary. The paper proposes a three-stage multi-agent pipeline for generating synthetic QA datasets for RAG evaluation. Stage 1 uses embeddings and k-means clustering to select diverse representative samples; Stage 2 detects and pseudonymizes private entities; Stage 3 generates QA pairs from the pseudonymized samples. The authors evaluate diversity on the EU AI Act by comparing against RAGAS-style evolutionary generation and direct prompting, using a GPT-4.1 judge and a cosine-based similarity metric. Privacy is evaluated by label-wise entity masking accuracy on three AI4Privacy datasets (PWI, PHI, PII). The central claims are that the generated sets outperform baselines in diversity and that the framework achieves robust privacy masking.

Significance. If the claims are established, the paper addresses a real gap: many RAG evaluation sets are static and not privacy-aware, and the proposed modular pipeline is a reasonable, easy-to-replicate design. Using public privacy datasets and comparing to established generation paradigms are good practices. The diversity results are directionally consistent across set sizes, and the privacy agent obtains respectable entity-level accuracy on three domains. However, the evidence currently has two serious gaps: the privacy claim is only measured at the masking stage, not on the final QA pairs that the framework actually outputs, and the quantitative diversity metric is reported with an internally inconsistent interpretation. Because both gaps concern the paper's central claims, the manuscript needs substantial revision before the conclusions are supported.

major comments (3)

[§4.3.2 and §5] The privacy claim is load-bearing but is only evaluated at the masking stage. Figure 1 reports label-wise accuracy of entity detection/masking on the input PWI/PHI/PII texts, but the final deliverable is the QA pairs produced by Stage 3. A non-trivial fraction of entities are missed (accuracies 0.75–0.90), so those entities remain in the text consumed by the QA curator and may appear verbatim in questions/answers. Moreover, even correctly pseudonymized entities could be re-inferred from context or from the LLM's parametric knowledge. Section 5 explicitly defers attack-resistance testing to future work. To support 'private synthetic QA datasets', the authors need to test the generated QA pairs for leakage (e.g., entity extraction from the QA set, membership inference, or manual audit) and compare against a baseline. Without this, the abstract's 'robust privacy masking' applies only to an
[§4.2.4, Table 1] The cosine metric is described inconsistently, making Table 1's quantitative diversity evidence unclear. §4.2.3 defines CosineSimilaritytoDiversity as a value that 'inverts the average pairwise cosine similarity' and states that 'lower values indicate greater semantic spread.' But §4.2.4 says the score 'becomes less negative (closer to zero), reflecting that questions are increasingly dissimilar.' Under the natural inversion (score = −average pairwise cosine), moving from −0.36 to −0.15 means the average pairwise cosine increased from 0.36 to 0.15? Actually the definition is ambiguous: if lower values mean greater spread, then −0.36 should indicate more spread than −0.15, so the reported trend would be a decrease in diversity, not an increase. If the intended metric is something else (e.g., 1 − average cosine), then the sign and the 'lower/higher' wording need correction. As written, the
[§4.2.3–4.2.4] The diversity evaluation relies on a single GPT-4.1 judge with no error bars, no repeated runs, and no inter-judge agreement. Since the questions themselves are generated by GPT-4o/GPT-4.1, there is a real risk that the judge is biased toward a particular style or that the differences in Table 1 are within prompt variability. The authors cite their own prior multi-agent judge paper [5], but that does not validate this specific use. I recommend reporting variance across several judge runs or multiple judges, and/or providing a human-annotated sample of the pairs. This would make the 'outperform baselines in diversity' claim more robust.

minor comments (6)

[§2.3] Minor wording issue: 'contributing a privacy and and diversity-aware framework' contains a duplicated 'and'.
[Algorithm 1] In the initialization line, 'Rep ortpriv' appears to be a typo for 'Report_priv'.
[Table 2] The header 'Total entities number' is awkward; consider 'Number of entities'.
[Reference [30]] Reference [30] lists the venue as '????'. This needs to be corrected.
[§4.2.3] The metric is called 'CosineSimilaritytoDiversity' but appears as 'Cosine Sim. to Diversity' in Table 1. Please use consistent naming and provide the exact formula.
[§4.3.1] The description of the privacy datasets says they contain 'private entities and their corresponding masked versions.' It would help to clarify whether the provided masked versions are used as ground truth for evaluation and whether the concatenation changes the entity annotations.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the diversity and privacy claims are evaluated against external baselines and external datasets; the privacy gap is an evidentiary limitation, not a circular derivation.

full rationale

I walked the paper's derivation chain. The proposed pipeline (Algorithm 1) takes input documents D, clusters them in Stage 1, pseudonymizes selected samples in Stage 2, and synthesizes QA pairs in Stage 3. The diversity claim is tested against two external baselines (RagasGen, DirPmpt) using (i) a GPT-4.1 judge and (ii) CosineSimilarityToDiversity [32], an external quantitative metric computed on embeddings. Neither metric is defined in terms of the method's own outputs in a way that forces the reported result; clustering selects diverse source documents, but the diversity rating is applied to the generated questions and compared with baselines that did not use clustering. The privacy evaluation in Section 4.3.2 measures label-wise masking accuracy on AI4Privacy's external PWI/PHI/PII datasets; this is an external benchmark, not a self-defined target. No fitted parameter is later renamed as a prediction. The only self-citation is reference [5] in the introduction, which supports the general importance of golden evaluation sets and is not load-bearing for either experimental claim. The paper itself flags a real limitation in Section 5: 'Future work will also include rigorous evaluation of the framework's resilience to privacy attacks.' Since no experiment tests whether the final QA pairs leak or allow recovery of originally masked entities, the privacy claim for the deliverable QA sets is under-evidenced. That is a correctness/evidential gap, not a circularity: the paper does not define 'privacy' as input-masking accuracy for the purpose of deriving the QA privacy claim; it simply omits the leakage test. No equation or construction equates output to input, so there is no circular step.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No invented physical or formal entities. The pipeline's burden lies in unstated assumptions about embedding quality, pseudonymization sufficiency, and judge validity.

free parameters (4)

number of clusters k = not reported
Selected via intra-cluster distance scores (Section 4.1); affects topical sampling breadth.
representative samples per cluster = not reported
The size of each S_i in Algorithm 1 is unspecified and controls both coverage and QA count.
QA pairs per private sample (n) = not reported
Algorithm 1 generates n QA pairs per input sample; n is not given.
document chunk size = 256 tokens
Fixed chunking before embedding and clustering; affects cluster quality (Section 4.1).

assumptions (4)

domain assumption text-embedding-3-small embeddings support semantically meaningful k-means clusters
Used in Stage 1 (Section 4.1); if embeddings are poor, representative sampling fails.
domain assumption pseudonymized source text yields privacy-preserving QA pairs
The final QA privacy is assumed, not measured; only source masking is evaluated (Section 4.3).
domain assumption GPT-4.1 LLM judge ratings are valid diversity measures
Diversity scores rely on a single judge with no human calibration (Section 4.2.3).
domain assumption concatenating AI4Privacy sentences preserves realistic PII context
Privacy benchmarks are constructed by concatenating samples into paragraphs (Section 4.3.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework." pith.science (2026). https://pith.science/paper/YGBKI3PA

@misc{pith2026250818929,
  author       = {Pith},
  title        = {Pith review of: Diverse And Private Synthetic Datasets Generation for RAG evaluation: A multi-agent framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGBKI3PA}},
  note         = {Machine review of arXiv:2508.18929}
}

read the original abstract

Retrieval-augmented generation (RAG) systems improve large language model outputs by incorporating external knowledge, enabling more informed and context-aware responses. However, the effectiveness and trustworthiness of these systems critically depends on how they are evaluated, particularly on whether the evaluation process captures real-world constraints like protecting sensitive information. While current evaluation efforts for RAG systems have primarily focused on the development of performance metrics, far less attention has been given to the design and quality of the underlying evaluation datasets, despite their pivotal role in enabling meaningful, reliable assessments. In this work, we introduce a novel multi-agent framework for generating synthetic QA datasets for RAG evaluation that prioritize semantic diversity and privacy preservation. Our approach involves: (1) a Diversity agent leveraging clustering techniques to maximize topical coverage and semantic variability, (2) a Privacy Agent that detects and mask sensitive information across multiple domains and (3) a QA curation agent that synthesizes private and diverse QA pairs suitable as ground truth for RAG evaluation. Extensive experiments demonstrate that our evaluation sets outperform baseline methods in diversity and achieve robust privacy masking on domain-specific datasets. This work offers a practical and ethically aligned pathway toward safer, more comprehensive RAG system evaluation, laying the foundation for future enhancements aligned with evolving AI regulations and compliance standards.

Figures

Figures reproduced from arXiv: 2508.18929 by the authors.

**Figure 1.** Privacy agent accuracy per entity type across the PHI, PWI, and PII datasets. 5. Conclusion and future work In this work, we introduced a modular multi-agent framework for the generation of synthetic QA datasets tailored to the rigorous evaluation of RAG systems. Our approach decomposes the dataset construction process into distinct, specialized agents, each focused on enriching semantic diversity, enforcing privacy… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 21 canonical work pages

[5]

H. Cao, I. Driouich, R. Singh, E. Thomas, Multi-agent llm judge: automatic personalized llm judge design for evaluating natural language generation applications, arXiv preprint arXiv:2504.02867 (2025)

work page Pith review arXiv 2025
[1]

Siriwardhana, R

S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, S. Nanayakkara, Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering, Transactions of the Association for Computational Linguistics 11 (2023) 1–17

work page 2023
[2]

Cao, Recent advances in text embedding: A comprehensive review of top-performing methods on the mteb benchmark, arXiv preprint arXiv:2406.01607 (2024)

H. Cao, Recent advances in text embedding: A comprehensive review of top-performing methods on the mteb benchmark, arXiv preprint arXiv:2406.01607 (2024)

arXiv 2024
[3]

M. R. Parvez, W. Ahmad, S. Chakraborty, B. Ray, K.-W. Chang, Retrieval augmented code generation and summarization, in: Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 2719–2734

work page 2021
[4]

A. Chen, G. Stanovsky, S. Singh, M. Gardner, Evaluating question answering evaluation, in: A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, D. Chen (Eds.), Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 119–124. URL: https://aclanthology.org/D19-5817/. doi:10.1...

work page doi:10.18653/v1/d19-5817 2019
[6]

Joshi, E

M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1601–1611

work page 2017
[7]

J. Chen, H. Lin, X. Han, L. Sun, Benchmarking large language models in retrieval-augmented generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 17754–17762

work page 2024
[8]

Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, E. Chen, Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models, arXiv preprint arXiv:2401.17043 (2024)

arXiv 2024

Show all 33 references

[9]

H. Cao, Enhancing negation awareness in universal text embeddings: A data-efficient and computational-efficient approach, Proceedings of the 28th European Conference on Artificial Intelligence (ECAI-2025) (2025)

2025
[10]

Bruckhaus, Rag does not work for enterprises, 2024

T. Bruckhaus, Rag does not work for enterprises, 2024. URL: https://arxiv.org/abs/2406.04369. arXiv:2406.04369

2024 arXiv
[11]

S. Es, J. James, L. Espinosa Anke, S. Schockaert, RAGAs: Automated evaluation of retrieval augmented generation, in: N. Aletras, O. De Clercq (Eds.), Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrati...

2024
[12]

Commission, Eu artificial intelligence act (ai act), https://artificialintelligenceact.eu, 2024

E. Commission, Eu artificial intelligence act (ai act), https://artificialintelligenceact.eu, 2024. Ac- cessed: 2025-07-15

2024
[13]

S. Zeng, J. Zhang, P. He, Y. Xing, Y. Liu, H. Xu, J. Ren, S. Wang, D. Yin, Y. Chang, et al., The good and the bad: Exploring privacy issues in retrieval-augmented generation (rag), ACL Findings (2024)

2024
[14]

Y. Ding, W. Fan, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, Q. Li, A survey on rag meets llms: Towards retrieval-augmented large language models, arXiv preprint arXiv:2405.06211 (2024)

2024 arXiv
[15]

Yunxiang, L

L. Yunxiang, L. Zihan, Z. Kai, D. Ruilong, Z. You, Chatdoctor: A medical chat model fine-tuned on llama model using medical domain knowledge, arXiv preprint arXiv:2303.14070 (2023)

2023 arXiv
[16]

Lewis, E

P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in Neural Information Processing Systems 33 (2020) 9459–9474

2020
[17]

Y. Gao, Y. Xiong, X. Gao, K. Jia, J. Pan, Y. Bi, Y. Dai, J. Sun, H. Wang, Retrieval-augmented generation for large language models: A survey, arXiv preprint arXiv:2312.10997 (2023)

2023 arXiv
[18]

Shuster, S

K. Shuster, S. Poff, M. Chen, D. Kiela, J. Weston, Retrieval augmentation reduces hallucination in conversation, arXiv preprint arXiv:2104.07567 (2021)

2021 arXiv
[19]

Z. Shao, Y. Gong, Y. Shen, M. Huang, N. Duan, W. Chen, Enhancing retrieval-augmented large language xmodels with iterative retrieval-generation synergy, arXiv preprint arXiv:2305.15294 (2023)

2023 arXiv
[20]

Cheng, D

X. Cheng, D. Luo, X. Chen, L. Liu, D. Zhao, R. Yan, Lift yourself up: Retrieval-augmented text generation with self memory, arXiv preprint arXiv:2305.02437 (2023)

2023 arXiv
[21]

D. P. Panagoulias, M. Virvou, G. A. Tsihrintzis, Augmenting large language models with rules for enhanced domain-specific interactions: The case of medical diagnosis, Electronics 13 (2024) 320

2024
[22]

Huang, S

Y. Huang, S. Gupta, Z. Zhong, K. Li, D. Chen, Privacy implications of retrieval-based language models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 2023

2023
[23]

Z. Qi, H. Zhang, E. Xing, S. Kakade, H. Lakkaraju, Follow my instruction and spill the beans: Scalable data extraction from retrieval-augmented generation systems, arXiv preprint arXiv:2402.17840 (2024)

2024 arXiv
[24]

J. Ye, J. Gao, Q. Li, H. Xu, J. Feng, Z. Wu, T. Yu, L. Kong, Zerogen: Efficient zero-shot learning via dataset generation, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 11653–11669

2022
[25]

Y. Meng, J. Huang, Y. Zhang, J. Han, Generating training data with language models: Towards zero-shot language understanding, Advances in Neural Information Processing Systems 35 (2022) 462–477

2022
[26]

J. Gao, R. Pi, L. Yong, H. Xu, J. Ye, Z. Wu, W. Zhang, X. Liang, Z. Li, L. Kong, Self-guided noise- free data generation for efficient zero-shot learning, in: International Conference on Learning Representations (ICLR 2023), 2023

2023
[27]

D. Chen, C. Lee, Y. Lu, D. Rosati, Z. Yu, Mixture of soft prompts for controllable data generation, in: Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 14815–14833

2023
[28]

Y. Yu, Y. Zhuang, J. Zhang, Y. Meng, A. J. Ratner, R. Krishna, J. Shen, C. Zhang, Large language model as attributed training data generator: A tale of diversity and bias, Advances in Neural Information Processing Systems 36 (2024)

2024
[29]

X. Tang, R. Shin, H. A. Inan, A. Manoel, F. Mireshghallah, Z. Lin, S. Gopi, J. Kulkarni, R. Sim, Privacy-preserving in-context learning with differentially private few-shot generation, arXiv preprint arXiv:2309.11765 (2023)

2023 arXiv
[30]

C. Xie, Z. Lin, A. Backurs, S. Gopi, D. Yu, H. A. Inan, H. Nori, H. Jiang, H. Zhang, Y. T. Lee, et al., Differentially private synthetic data via foundation model apis 2: Text, in: ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, ????

2024
[31]

S. Zeng, J. Zhang, P. He, J. Ren, T. Zheng, H. Lu, H. Xu, H. Liu, Y. Xing, J. Tang, Mitigating the privacy issues in retrieval-augmented generation (rag) via pure synthetic data, 2025. URL: https://arxiv.org/abs/2406.14773. arXiv:2406.14773

2025 arXiv
[32]

H. Gao, Y. Zhang, Vrsd: Rethinking similarity and diversity for retrieval in large language models,
[2024]

arXiv:2407.04573

URL: https://arxiv.org/abs/2407.04573. arXiv:2407.04573

arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [5]

H. Cao, I. Driouich, R. Singh, E. Thomas, Multi-agent llm judge: automatic personalized llm judge design for evaluating natural language generation applications, arXiv preprint arXiv:2504.02867 (2025)

work page Pith review arXiv 2025

[2] [1]

Siriwardhana, R

S. Siriwardhana, R. Weerasekera, E. Wen, T. Kaluarachchi, R. Rana, S. Nanayakkara, Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering, Transactions of the Association for Computational Linguistics 11 (2023) 1–17

work page 2023

[3] [2]

Cao, Recent advances in text embedding: A comprehensive review of top-performing methods on the mteb benchmark, arXiv preprint arXiv:2406.01607 (2024)

H. Cao, Recent advances in text embedding: A comprehensive review of top-performing methods on the mteb benchmark, arXiv preprint arXiv:2406.01607 (2024)

arXiv 2024

[4] [3]

M. R. Parvez, W. Ahmad, S. Chakraborty, B. Ray, K.-W. Chang, Retrieval augmented code generation and summarization, in: Findings of the Association for Computational Linguistics: EMNLP 2021, 2021, pp. 2719–2734

work page 2021

[5] [4]

A. Chen, G. Stanovsky, S. Singh, M. Gardner, Evaluating question answering evaluation, in: A. Fisch, A. Talmor, R. Jia, M. Seo, E. Choi, D. Chen (Eds.), Proceedings of the 2nd Workshop on Machine Reading for Question Answering, Association for Computational Linguistics, Hong Kong, China, 2019, pp. 119–124. URL: https://aclanthology.org/D19-5817/. doi:10.1...

work page doi:10.18653/v1/d19-5817 2019

[6] [6]

Joshi, E

M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2017, pp. 1601–1611

work page 2017

[7] [7]

J. Chen, H. Lin, X. Han, L. Sun, Benchmarking large language models in retrieval-augmented generation, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 2024, pp. 17754–17762

work page 2024

[8] [8]

Y. Lyu, Z. Li, S. Niu, F. Xiong, B. Tang, W. Wang, H. Wu, H. Liu, T. Xu, E. Chen, Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models, arXiv preprint arXiv:2401.17043 (2024)

arXiv 2024