{"id":"97b281ab-fc57-4d86-bccb-0f8d8a437da2","arxiv_id":"2508.18929","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A multi-agent LLM framework generates synthetic QA datasets for RAG evaluation by combining clustering-based sampling, PII pseudonymization, and QA curation, with reported diversity gains and 0.75-0.90 masking accuracy.","lead":"This paper describes a three-stage, LLM-driven pipeline that builds question-answer datasets for testing RAG systems, using clustering to increase topic variety, pseudonymization to mask private information, and a curation step that writes the QA pairs. The generated sets are reported to be more diverse than two baselines, and the masking step achieves 75-94% accuracy across tested entity types.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy of final QA pairs is not established: masking accuracy is measured only on input text (Sec. 4.3.2), and Sec. 5 defers attack-resistance testing; QA generation could reintroduce or infer masked entities.","rationale":"The strongest_claim includes both diversity and privacy. The diversity claim has its own weaknesses (single-run LLM judge, contradictory Cosine-Similarity-to-Diversity description in Sec. 4.2.3 vs 4.2.4), but the privacy gap is more load-bearing because the title and motivation promise 'private synthetic datasets,' and the authors themselves defer attack resistance to future work. Even if diversity results are accepted, the framework cannot be called privacy-preserving without measuring the final artifact. This aligns with the reader's weakest_assumption, so I agree. A targeted leakage test on QA pairs would settle the issue; if it comes back clean at a reasonable sample size, the conditional verdict can be upgraded; if not, the privacy claim needs to be downgraded to 'input-masking accuracy only.'","tokens_in":7654,"tokens_out":5154,"duration_ms":46566,"concrete_test":"Use a held-out subset (e.g., 200 documents from each of PWI, PHI, PII), run the full pipeline, and collect the final QA pairs. For every sensitive entity in the original documents (including any the privacy agent failed to mask), perform exact-string and fuzzy (token-overlap or embedding-similarity) matching against the QA pairs. Then run an adversarial recovery pass: prompt GPT-4 with each QA pair and ask it to infer the original entity values from context, treating any unmasked spans in the source as auxiliary clues. Report entity-level leakage precision/recall for both checks. If any originally masked entity is found in or recoverable from the QA pairs, the 'private evaluation set' claim is falsified; if recovery is zero across a reasonably sized sample, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the framework generates 'private synthetic QA datasets' is not supported by the reported experiments. Section 4.3.2 evaluates the privacy agent's entity detection/masking accuracy on the input PWI/PHI/PII texts, but the actual deliverable is the QA pairs produced by the QA curation agent (Stage 3, Algorithm 1). No experiment checks whether these QA pairs contain or permit recovery of originally masked entities. This is not a hypothetical gap: the reported label-wise accuracies are between 0.75 and 0.90, so a substantial fraction of sensitive entities are not masked at all and remain in the text that the QA curator consumes; those entities can be copied verbatim into questions or answers. Moreover, even for correctly pseudonymized entities, the QA generation step may reintroduce the original value through contextual inference or the LLM's parametric knowledge. The conclusion explicitly lists 'rigorous evaluation of the framework's resilience to privacy attacks' as future work (Sec. 5), confirming that the privacy claim for final QA sets is untested. Measuring masking accuracy on input text cannot substitute for leakage testing on the generated evaluation set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a three-stage multi-agent pipeline for generating synthetic QA datasets for RAG evaluation. Stage 1 uses embeddings and k-means clustering to select diverse representative samples; Stage 2 detects and pseudonymizes private entities; Stage 3 generates QA pairs from the pseudonymized samples. The authors evaluate diversity on the EU AI Act by comparing against RAGAS-style evolutionary generation and direct prompting, using a GPT-4.1 judge and a cosine-based similarity metric. Privacy is evaluated by label-wise entity masking accuracy on three AI4Privacy datasets (PWI, PHI, PII). The central claims are that the generated sets outperform baselines in diversity and that the framework achieves robust privacy masking.","tokens_in":7951,"tokens_out":4354,"duration_ms":40634,"significance":"If the claims are established, the paper addresses a real gap: many RAG evaluation sets are static and not privacy-aware, and the proposed modular pipeline is a reasonable, easy-to-replicate design. Using public privacy datasets and comparing to established generation paradigms are good practices. The diversity results are directionally consistent across set sizes, and the privacy agent obtains respectable entity-level accuracy on three domains. However, the evidence currently has two serious gaps: the privacy claim is only measured at the masking stage, not on the final QA pairs that the framework actually outputs, and the quantitative diversity metric is reported with an internally inconsistent interpretation. Because both gaps concern the paper's central claims, the manuscript needs substantial revision before the conclusions are supported.","major_comments":[{"comment":"The privacy claim is load-bearing but is only evaluated at the masking stage. Figure 1 reports label-wise accuracy of entity detection/masking on the input PWI/PHI/PII texts, but the final deliverable is the QA pairs produced by Stage 3. A non-trivial fraction of entities are missed (accuracies 0.75–0.90), so those entities remain in the text consumed by the QA curator and may appear verbatim in questions/answers. Moreover, even correctly pseudonymized entities could be re-inferred from context or from the LLM's parametric knowledge. Section 5 explicitly defers attack-resistance testing to future work. To support 'private synthetic QA datasets', the authors need to test the generated QA pairs for leakage (e.g., entity extraction from the QA set, membership inference, or manual audit) and compare against a baseline. Without this, the abstract's 'robust privacy masking' applies only to an","section":"§4.3.2 and §5"},{"comment":"The cosine metric is described inconsistently, making Table 1's quantitative diversity evidence unclear. §4.2.3 defines CosineSimilaritytoDiversity as a value that 'inverts the average pairwise cosine similarity' and states that 'lower values indicate greater semantic spread.' But §4.2.4 says the score 'becomes less negative (closer to zero), reflecting that questions are increasingly dissimilar.' Under the natural inversion (score = −average pairwise cosine), moving from −0.36 to −0.15 means the average pairwise cosine increased from 0.36 to 0.15? Actually the definition is ambiguous: if lower values mean greater spread, then −0.36 should indicate more spread than −0.15, so the reported trend would be a decrease in diversity, not an increase. If the intended metric is something else (e.g., 1 − average cosine), then the sign and the 'lower/higher' wording need correction. As written, the","section":"§4.2.4, Table 1"},{"comment":"The diversity evaluation relies on a single GPT-4.1 judge with no error bars, no repeated runs, and no inter-judge agreement. Since the questions themselves are generated by GPT-4o/GPT-4.1, there is a real risk that the judge is biased toward a particular style or that the differences in Table 1 are within prompt variability. The authors cite their own prior multi-agent judge paper [5], but that does not validate this specific use. I recommend reporting variance across several judge runs or multiple judges, and/or providing a human-annotated sample of the pairs. This would make the 'outperform baselines in diversity' claim more robust.","section":"§4.2.3–4.2.4"}],"minor_comments":[{"comment":"Minor wording issue: 'contributing a privacy and and diversity-aware framework' contains a duplicated 'and'.","section":"§2.3"},{"comment":"In the initialization line, 'Rep ortpriv' appears to be a typo for 'Report_priv'.","section":"Algorithm 1"},{"comment":"The header 'Total entities number' is awkward; consider 'Number of entities'.","section":"Table 2"},{"comment":"Reference [30] lists the venue as '????'. This needs to be corrected.","section":"Reference [30]"},{"comment":"The metric is called 'CosineSimilaritytoDiversity' but appears as 'Cosine Sim. to Diversity' in Table 1. Please use consistent naming and provide the exact formula.","section":"§4.2.3"},{"comment":"The description of the privacy datasets says they contain 'private entities and their corresponding masked versions.' It would help to clarify whether the provided masked versions are used as ground truth for evaluation and whether the concatenation changes the entity annotations.","section":"§4.3.1"}],"recommendation":"major_revision","confidential_remarks":"The privacy gap is especially salient given the paper's stated motivation (privacy-preserving evaluation for RAG) and the workshop's trustworthiness theme. The authors should be asked to add an end-to-end leakage experiment on the final QA pairs; this may be more important than additional diversity metrics. Also note the paper does not provide code or data, so reproducibility is currently limited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The multi-agent pipeline is a reasonable extension of RAGAS-style synthetic QA generation: cluster for topical coverage, pseudonymize, then generate QA pairs. Using the EU AI Act as a testbed and comparing against evolutionary and direct-prompting baselines is a sensible setup, and the label-wise masking accuracies across PWI/PHI/PII give useful detail. The diversity gains over baselines are consistent across dataset sizes, which is a good sign. I credit the authors for a modular design and for surfacing a structured privacy report.\n\nThe soft spots are real, and one is load-bearing. The paper claims to produce 'private synthetic QA datasets,' but the experiments only measure masking accuracy on the input texts (Sec. 4.3.2). The actual deliverable is the QA pairs generated by the curator from the pseudonymized text. No test checks whether those pairs contain or permit recovery of originally masked entities. The authors themselves list attack-resistance testing as future work in Sec. 5, which confirms the gap. Given that label-wise accuracies run between 0.75 and 0.90, a nontrivial fraction of entities are never masked; those can be copied verbatim into questions or answers. Even correctly pseudonymized entities could be reintroduced via contextual inference or parametric knowledge. Measuring masking accuracy on input text cannot substitute for leakage testing on the generated set. This is an addressable flaw, but it means the central privacy claim is not established.\n\nThe diversity evaluation has its own weaknesses: a single LLM judge, no error bars or multiple runs, and no human validation. The cosine metric description is internally inconsistent as written. The authors say lower values indicate greater semantic spread, but then interpret scores moving closer to zero (e.g., from -0.36 to -0.15) as reflecting that questions are 'increasingly dissimilar.' Closer to zero means higher cosine similarity, so the opposite. Either the metric is defined differently or the interpretation is backwards; the table can't be read reliably without a fix. This is a minor-to-moderate reporting issue, but it undermines confidence in the quantitative diversity claim. No prompts or code are released, which is not fatal for a workshop paper but would help.\n\nThere is no fitted-parameter circularity: the hyperparameters are choices, not tuned to the results. The self-citation to the prior multi-agent judge work is contextual and not a problem.\n\nWho is this for? People building synthetic evaluation sets for RAG in privacy-sensitive domains. The paper deserves a serious referee, not a desk reject, because the framework is plausible and the privacy gap is testable. I would send it to peer review with the clear expectation of major revision: add leakage tests on the final QA pairs, fix the metric description, and include variance or multiple judge runs. With those changes, the claims could be supported.","headline":"The pipeline is sensible and the diversity results are directionally encouraging, but the paper never tests whether the final QA pairs leak masked entities, so the central privacy claim is currently unsupported.","tokens_in":8404,"tokens_out":2102,"would_cite":false,"duration_ms":20951,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-agent pipeline claims to generate synthetic QA pairs for RAG evaluation with higher semantic diversity than evolutionary or direct-prompting baselines, while masking sensitive entities across personal, workplace, and medical domains","keywords":["synthetic QA generation","RAG evaluation","multi-agent framework","semantic diversity","PII pseudonymization","privacy-preserving data generation","EU AI Act","LLM-as-a-judge"],"falsifier":"Run a prompt-extraction or membership-inference attack on the generated QA pairs themselves, not on the input text, to see whether original masked entities (names, salaries, diagnoses, and so on) can be recovered. If original entities appear in or are inferable from the QA pairs, the paper's stronger privacy claim fails; if they cannot be recovered, the claim holds.","tokens_in":7575,"feed_emoji":"🛡️","tokens_out":6446,"duration_ms":58389,"temperature":0.7,"pith_summary":"Retrieval-augmented generation (RAG) is only as trustworthy as the datasets used to evaluate it, yet most benchmarks ignore topical coverage and privacy. This paper claims that a three-agent pipeline—clustering-based sample selection, sensitive-entity pseudonymization, and QA generation from masked text—produces synthetic question–answer sets that are more semantically diverse than existing evolutionary or direct-prompting generators, while masking personal, workplace, and medical entities with high per-label accuracy. The authors test diversity on the EU AI Act corpus using an LLM judge and a cosine-similarity metric, and test masking on three domain-specific datasets. If correct, it offers a practical recipe for building evaluation sets that reflect real-world breadth and reduce exposure of sensitive data, supporting compliance-minded AI auditing. The load-bearing caveat is that the paper checks masking on the input text, not on the final QA pairs.","feed_headline":"Three-agent pipeline beats baselines on RAG test-set diversity","feed_subtitle":"It also masks personal, workplace, and medical entities with per-label accuracy up to 0.94.","key_machinery":"The carrying mechanism is the three-stage pipeline itself. First, a Diversity agent applies k-means clustering to semantic embeddings and picks representative samples per cluster, aiming for wide topic coverage. Second, a Privacy agent detects sensitive entities according to predefined categories, pseudonymizes them with context-aware replacements, and logs a structured privacy report. Third, a QA Curation agent generates question–answer pairs from the pseudonymized samples, producing evaluation-ready ground truth. The pipeline is modular: each stage can be swapped or extended independently.","core_discovery":"The paper claims that decomposing synthetic QA dataset creation into three specialized agents—topic-diverse representative sampling via clustering, entity-level pseudonymization, and QA generation from the masked texts—yields evaluation sets that are semantically more diverse than evolutionary and direct-prompting alternatives, while masking sensitive entities at high label-level accuracy across personal, workplace, and medical domains. The reported diversity scores rise with test set size (from 7.8 at 10 samples to 9.0 at 100 samples on an LLM judge scale), and cosine-similarity-to-diversity moves consistently toward zero, indicating greater semantic spread. Privacy masking accuracy per ent","pith_inferences":["The strongest privacy claim—that the QA pairs themselves are private—is not actually tested; only input-text masking is measured, and the conclusion defers attack-resistance testing to future work.","LLM-as-a-judge diversity scores may partly reflect the judge model's own preferences; correlating these scores with retrieval or answer quality on downstream RAG tasks would test whether the diversity is functionally useful.","Because representative sampling is cluster-based, rare or outlier topics may be under-sampled even when average diversity improves; a coverage check on low-density clusters would clarify this.","Formal guarantees such as differential privacy could be layered onto pseudonymization if regulations require more than entity masking."],"forward_implications":["Larger synthetic QA sets generated this way keep gaining topic diversity, so scaling evaluation sets is a practical route to broader coverage.","The pipeline can be applied to domain-specific corpora to produce evaluation sets without exposing the underlying sensitive fields.","The modular agent design means a change in clustering or masking strategy can be dropped in without redesigning the whole generation process.","The twofold evaluation recipe—LLM judge plus cosine-similarity diversity, plus per-entity masking accuracy—offers a repeatable protocol for comparing synthetic QA generators."],"supporting_citations":[{"why":"Supplies the evolutionary-generation baseline and the synthetic evaluation paradigm the paper compares against.","marker":"[11]"},{"why":"Establishes prompt-based extraction privacy risks in retrieval-augmented generation, motivating the privacy agent.","marker":"[13]"},{"why":"Demonstrates scalable data extraction attacks on retrieval-augmented systems, the threat the pipeline aims to counter.","marker":"[23]"},{"why":"Proposes a two-stage agent-based synthetic-data pipeline for privacy in RAG, the approach this framework extends.","marker":"[31]"},{"why":"Provides the cosine-similarity-to-diversity metric used to quantify semantic spread of the generated questions.","marker":"[32]"}],"fun_headline_variants":["Multi-agent framework crafts diverse and private RAG test sets","Three agents make RAG eval sets diverse and privacy-safe","Clustering, masking, and QA agents boost RAG dataset quality","Synthetic QA sets get semantic spread and entity masking","RAG evaluation data: diverse and private via multi-agent pipeline"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The privacy claim rests on the untested assumption that generating questions from pseudonymized text cannot reintroduce or infer the masked entities in the final QA pairs.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent framework crafts diverse and private RAG test sets","Three agents make RAG eval sets diverse and privacy-safe","Clustering, masking, and QA agents boost RAG dataset quality","Synthetic QA sets get semantic spread and entity masking","RAG evaluation data: diverse and private via multi-agent pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1241,"prompt_tokens":746,"completion_tokens":495,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":490,"tokens_out":495,"duration_ms":4940,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:04:29.740916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a prompt-extraction or membership-inference attack on the generated QA pairs themselves, not on the input text, to see whether original masked entities (names, salaries, diagnoses, and so on) can be recovered. If original entities appear in or are inferable from the QA pairs, the paper's stronger privacy claim fails; if they cannot be recovered, the claim holds.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evolutionary-generation baseline and the synthetic evaluation paradigm the paper compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes prompt-based extraction privacy risks in retrieval-augmented generation, motivating the privacy agent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the cosine-similarity-to-diversity metric used to quantify semantic spread of the generated questions."}],"review_version":1}