{"id":"1e46b148-2469-43b5-9109-8e7790e332e8","arxiv_id":"2608.12675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEAG uses a locally fine-tuned 3-4B model to replace sensitive entities with consistent aliases in RAG prompts, letting external LLMs answer while keeping original values hidden; reported User accuracy is over 80% but full-entity hiding accuracy is only 75-78%.","lead":"This paper introduces SEAG, a framework that replaces sensitive names and numbers in user queries and documents with consistent fake aliases before sending them to an external AI like GPT-5, then swaps the aliases back after the answer is generated. It reports that small locally run models can hide enough sensitive information to let organizations use powerful cloud LLMs without directly exposing private data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own Table 2 contradicts the central privacy claim: entity replacement accuracy is only 74.91–77.83%, so roughly a quarter of sensitive entities remain in the prompts sent to the external generator.","rationale":"The reader's weakest assumption consisted of two related points: that the external generator cannot recover original entities from the aliased prompt, and that the SEAG model must identify and replace all sensitive entities, with Table 2 showing this condition is not met. My analysis converges on the second point, which is more directly falsifying: the paper's own reported entity replacement accuracy means that the external generator still receives original sensitive entities in roughly 20–25% of entity occurrences. The paper's metrics are designed to measure question-answering success and hiding of the question-critical entity, not the confidentiality of all sensitive content. Thus the stated conclusion in the abstract and Section 7 that the framework prevents the external generator from accessing sensitive information is not supported by the evidence. The framework remains plausible as a proof-of-concept, and the appropriate verdict remains CONDITIONAL with the requested revisions: report entity-level leakage across all sensitive entities, provide baselines, add error bars, and test against an adversarial reconstruction attack. My concern does not change the reader's verdict; it reinforces it with a more concrete internal inconsistency.","tokens_in":13305,"tokens_out":3357,"duration_ms":34207,"concrete_test":"Using the released dataset and the SEAG models' outputs, recompute the proportion of evaluation samples (out of 600) in which at least one entity from the manual ground-truth entity list is left unreplaced in the prompt that is actually sent to the external generator. If this sample-level leakage fraction is non-zero, then the generator receives original sensitive information in a non-negligible share of cases, directly contradicting the claim that it receives only the altered version. A complementary, second check is to present the replaced prompts to a strong external LLM with instructions to infer the original sensitive entities and measure the reconstruction success rate, but the first check alone is sufficient to falsify the current claim if leakage is observed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SEAG prevents the external generator from accessing sensitive information. For that to hold, every sensitive entity in the query and retrieved documents must be replaced before the prompt is sent to the generator. Section 4.2 states this requirement explicitly: the generator 'receives only the altered version of the information' and 'does not have direct access to the original sensitive information.' However, Section 5.4 and Table 2 show that the SEAG models replace only 74.91%–77.83% of sensitive entities across all six domains. This means that in roughly one out of four entity occurrences, the original sensitive value remains visible in the prompt that is forwarded to the external LLM. The two headline metrics do not detect this leakage. The Privacy metric in Section 5.2 only checks whether the specific entity needed to answer the user's question is hidden, and the User metric only checks whether the final answer shown to the user is correct. Neither metric measures whether other sensitive entities in the document were left unreplaced. Consequently, the paper's evidence does not support the unqualified statement that the external generator cannot access sensitive information. The paper itself acknowledges the required condition in Section 4.2 but does not address the gap in the evaluation. This is an internal inconsistency between the stated threat model and the measured behavior, not merely a disagreement with an external privacy standard. Additionally, for entities that are replaced, the aliases are chosen to be semantically consistent (e.g., Saudi Arabia with Qatar and Riyadh with Doha), which could allow a capable external LLM to infer the original values from world knowledge or contextual co-occurrence; this reconstruction risk is never tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SEAG (Sensitive Entity Alias Generator), a privacy-preserving framework for RAG. A locally deployed fine-tuned model (Qwen-3, LLaMA-3.2, or Phi-4) identifies sensitive entities in the user query and retrieved documents, generates aliases for them, and constructs an entity replacement table. The query and documents are anonymized with this table before being sent to an external LLM generator (GPT-5 or Claude-4 sonnet); the generator's answer is then reverse-mapped to restore the original values for the user. The authors construct two datasets (one for fine-tuning, one for evaluation), fine-tune three 3-4B models with QLoRA, and report ORG, Privacy, and User metrics. Results show ORG scores near 99%, Privacy scores up to 89.67%, and User scores above 80% for all combinations. The paper also reports an entity-level evaluation in Table 2 showing total entity replacement accuracy of 77.83% (Qwen-3), 76.73% (LLaMA-3.2), and 74.91% (Phi-4).","tokens_in":13457,"tokens_out":4535,"duration_ms":44614,"significance":"If the central claim holds—that SEAG prevents external generators from accessing sensitive information while preserving answer quality—the framework would be practically valuable: organizations could use powerful third-party LLMs over private documents without exposing raw sensitive values, at the cost of deploying only a small local model. The paper provides a concrete pipeline, two constructed datasets, and a reproducibility-oriented release (datasets and code on GitHub). It also applies a consistent entity-alignment methodology and evaluates across multiple base models and two commercial generators. However, the significance is currently limited by a gap between the stated threat model and the measured behavior: the paper's own entity-level evaluation shows that roughly one in four sensitive entities is not replaced, which directly undermines the unqualified privacy claim. The significance will be substantially stronger if the authors address this gap and provide a direct measure of what the external generator actually receives.","major_comments":[{"comment":"The threat model in Section 4.2 states a necessary condition: 'The fine-tuned model must correctly identify and replace all sensitive entities consistently. A missing entity may expose private information.' Yet Table 2 shows that total entity replacement accuracy is only 74.91%–77.83%, meaning about one in four sensitive entity occurrences in the documents remains in its original form and is forwarded to the external generator. The Privacy metric in Section 5.2 only checks the specific entity required to answer the user's question, so it does not detect leakage of other sensitive entities. Consequently, the evidence does not support the abstract's claim that SEAG 'prevents the external generator from accessing sensitive information.' The authors should either modify the claim to reflect probabilistic protection, or add a fail-closed mechanism (e.g., abstaining from sending prompts containing unreplaced entities) so that the stated condition is actually met.","section":"§4.2 vs. §5.4 (Table 2)"},{"comment":"The Privacy metric is defined operationally as the generator failing to produce the ground-truth answer (e.g., answering 'Mike' instead of 'John'). This conflates 'hiding' with 'answer failure': the generator might fail for reasons unrelated to replacement (e.g., ambiguity in the aliased prompt), or it might correctly infer the original entity from context even when the entity is replaced, which would be a privacy failure not captured by this metric. The paper does not directly inspect the anonymized prompts to verify that the original sensitive strings are absent. A direct leakage measurement—for example, checking the forwarded prompt for occurrences of original entities—is needed to substantiate the privacy claim.","section":"§5.2 (Privacy metric)"},{"comment":"The threat model treats the external generator as a passive recipient of the altered prompt and assumes that the generator 'does not have direct access to the original sensitive information.' However, confidentiality also requires that the generator cannot reconstruct the original values from the aliases, context, or world knowledge. The paper provides no test of whether a capable LLM can reverse the replacement (e.g., inferring 'New Zealand' from 'Australia' or '4.2%' from '5%' in the running example). If such inference is possible, even perfect replacement would not protect confidentiality. The limitations section should discuss this, and ideally the authors should run an empirical reconstruction-attack evaluation.","section":"§4.2 and §6 (threat model scope)"}],"minor_comments":[{"comment":"The text says 'There has been a strong emphasis on extracting all name entities, not just a few'—replace 'name entities' with 'named entities.'","section":"§3.1"},{"comment":"The phrase 'theORGmetric' lacks a space; it should read 'the ORG metric.'","section":"§5.2"},{"comment":"The replacement table lists entity types such as 'Figure 1.1' and 'Figure 2.2' as original entities, which appear to be figure references rather than sensitive entities; clarify whether these are part of the evaluated entity set or an artifact of the example.","section":"Appendix A (Table A.3)"},{"comment":"The limitations section discusses document length and entity density but does not acknowledge the incomplete-replacement issue revealed by Table 2. Adding a discussion of the 25% leakage rate and its implications for the privacy guarantee would make the limitations more complete.","section":"§6 (Limitations)"}],"recommendation":"major_revision","confidential_remarks":"The paper's central privacy claim is currently contradicted by its own entity-level evaluation (Table 2). This is not a matter of style but of substance: the threat model requires complete replacement, and the data show it is not achieved. The authors should be encouraged to either reframe the contribution as risk reduction rather than absolute prevention, or implement a fail-closed architecture that never forwards unreplaced sensitive entities. Also, the absence of any direct test of alias invertibility weakens the privacy argument. I would not support acceptance without these points being addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this one for the idea, not for the headline claim. SEAG is a sensible system: a locally deployed 3-4B model that builds a consistent alias table for sensitive entities before the query and retrieved docs go to an external LLM, then maps the aliases back in the answer. The threat model—the generator, not the user, is the untrusted party—is a real gap in RAG privacy work, and the authors are upfront that prior work focuses on unauthorized users. The two new datasets and three fine-tuned models (Qwen-3, LLaMA-3.2, Phi-4) are concrete contributions, and the writing is clear, with a limitation section that names document length and entity density.\n\nThe soft spot is load-bearing. The paper's own Table 2 shows total entity-replacement accuracy of 74.91–77.83%. That means roughly one in four sensitive entity occurrences stays in the prompt sent to the external generator. Section 4.2 states the condition for the framework to be 'fully effective': the model must identify and replace all sensitive entities consistently. It doesn't. The Privacy metric only checks the entity needed to answer the user's question, so it can score 89% while the generator still sees many other private values. The stress-test note is correct: this is an internal inconsistency between the stated threat model and the measured behavior, not just a disagreement with an external privacy standard.\n\nTwo more issues, minor in comparison. Alias choices are semantically coherent (Saudi Arabia→Qatar, Riyadh→Doha), so a capable external LLM could plausibly infer originals from world knowledge; the paper never tests reconstruction. And the evaluation rests entirely on self-constructed datasets; there are no baselines, no error bars, no external benchmarks.\n\nNone of this kills the proof-of-concept. The architecture is sensible, the restoration logic works when replacement succeeds, and the User scores around 80–84% show the approach can preserve utility. But the 'privacy-preserving' label is not supported by the evidence as presented. This deserves a serious referee, not a desk reject. The fix is concrete: report leakage over all entities, add baselines (raw RAG, random masking, synthetic data), give variance across runs, and include a reconstruction attack by the generator. Then it could be a useful contribution to the privacy-preserving RAG subfield.\n\nFor your reading group: maybe. For citing: not yet.\n\nRecommendation: conditional accept with major revision.","headline":"Plausible privacy-preserving RAG wrapper that overclaims: its own entity-replacement accuracy leaves a quarter of sensitive entities exposed to the external generator.","tokens_in":14203,"tokens_out":2027,"would_cite":false,"duration_ms":19028,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The SEAG framework hides sensitive entities from external LLMs in RAG pipelines while preserving answer quality.","keywords":["Retrieval-Augmented Generation (RAG)","Privacy-Preserving AI","Large Language Models (LLMs)","Sensitive Entity Replacement","Parameter-Efficient Fine-Tuning (PEFT)","Alias Generation","Entity Replacement Table","External Generator Privacy"],"falsifier":"Send the SEAG-replaced prompts from the evaluation dataset to a capable LLM with an instruction to infer the original entities, and measure how often it recovers the true values; if recovery succeeds on a nontrivial fraction, the framework's confidentiality claim fails.","tokens_in":12980,"feed_emoji":"🛡️","tokens_out":4169,"duration_ms":36667,"temperature":0.7,"pith_summary":"This paper claims that a small locally deployed language model can act as a privacy gatekeeper in retrieval-augmented generation (RAG) by replacing all sensitive entities in a query and its retrieved documents with consistent aliases before the prompt is sent to an external generator. The SEAG framework's fine-tuned model builds an entity replacement table, swaps every sensitive value for a semantically consistent alias, and reverses the swap in the generator's answer before returning it to the user. The paper reports that all tested SEAG models achieved over 80% accuracy on its User metric, meaning the user gets the correct answer while the external generator only sees aliases, with the best Privacy score at 89.67%. If correct, these results would let organizations use powerful third-party LLMs over private documents without exposing raw sensitive values.","feed_headline":"A small local model can hide secrets from GPT-5 and Claude-4","feed_subtitle":"SEAG swaps sensitive entities for aliases before RAG prompts leave your organization, and restores answers afterward.","key_machinery":"The entity replacement table is the central mechanism. The locally deployed SEAG model examines the user query and the retrieved documents, identifies the sensitive entities, and emits a table mapping each original entity to a consistent alias, for example New Zealand to Australia and 4.2% to 5%. The same table is applied to every occurrence of each entity across the query and all documents, so the external generator sees coherent, semantically plausible text rather than inconsistent placeholders. After the generator answers, the table is applied in reverse to any aliases appearing in the response, restoring the original values before the answer reaches the user. This two-way table is what lets the framework preserve both privacy and utility.","core_discovery":"The central discovery is that a fine-tuned 3–4 billion parameter model can reliably locate sensitive entities in a RAG prompt, generate meaningful and consistent aliases, and restore the original values in the final answer, so the external generator never sees the real sensitive information. The paper's experiments use three locally fine-tuned models (Qwen-3, LLaMA-3.2, and Phi-4) and two external generators (GPT-5 and Claude-4 sonnet). On the Privacy metric, which checks whether the entities needed to answer the question were hidden, all models scored between 84.83% and 89.67%. On the User metric, which checks whether the final answer presented to the user is correct after alias restoration, all models scored above 80%. The paper argues this shows the framework succeeds at shielding sensitive content from the generator while keeping answer quality high.","pith_inferences":["If the external generator can infer original entities from context or auxiliary knowledge, the alias replacement alone does not guarantee confidentiality; the paper does not test this adversarial de-anonymization.","The framework could be extended to structured knowledge bases or knowledge graphs, where alias consistency across relations becomes even more critical.","A natural testable extension is to evaluate SEAG against a generator explicitly prompted to recover the original entities, measuring how often aliases are reversible.","The entity replacement table could be combined with differential privacy or synthetic data techniques to strengthen protection beyond what a single alias mapping provides."],"forward_implications":["Organizations can keep using third-party LLM generators in RAG systems without sending raw sensitive entities to the generator.","A small local model of roughly 3–4 billion parameters is sufficient for the anonymization step, avoiding the cost of hosting a large generator locally.","Answer quality remains high: all tested configurations exceeded 80% User accuracy, so users still receive correct answers after alias restoration.","Privacy scores up to 89.67% indicate that most question-critical entities are hidden from the external generator, though full-document entity hiding is lower, around 75–78%."],"supporting_citations":[{"why":"Supplies the QLoRA parameter-efficient fine-tuning method used to train the SEAG models.","marker":"[16]"},{"why":"Used to construct the entity replacement tables that form the fine-tuning dataset.","marker":"[33]"},{"why":"One of the base models fine-tuned as a SEAG model.","marker":"[13]"},{"why":"One of the base models fine-tuned as a SEAG model.","marker":"[14]"},{"why":"One of the base models fine-tuned as a SEAG model.","marker":"[15]"},{"why":"One of the two external generators used to evaluate the framework.","marker":"[7]"},{"why":"The second external generator used to evaluate the framework.","marker":"[8]"}],"fun_headline_variants":["Local model hides sensitive data from GPT-5 and Claude-4","SEAG keeps RAG secrets from external LLMs","Local alias generator hides RAG data from GPT-5 and Claude-4","Hide sensitive entities before RAG with a local model","Small local model shields RAG prompts from prying LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The privacy guarantee assumes the external generator cannot recover the original sensitive entities from the aliased prompt, but the paper never tests whether a capable LLM can infer the original values from context, alias patterns, or world knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Local model hides sensitive data from GPT-5 and Claude-4","SEAG keeps RAG secrets from external LLMs","Local alias generator hides RAG data from GPT-5 and Claude-4","Hide sensitive entities before RAG with a local model","Small local model shields RAG prompts from prying LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000608,"raw_usage":{"total_tokens":2873,"prompt_tokens":1028,"completion_tokens":1845,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":1759}},"tokens_in":644,"tokens_out":1845,"duration_ms":13370,"temperature":1.0,"reasoning_tokens":1759,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:48:43.816387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Send the SEAG-replaced prompts from the evaluation dataset to a capable LLM with an instruction to infer the original entities, and measure how often it recovers the true values; if recovery succeeds on a nontrivial fraction, the framework's confidentiality claim fails.","supporting_citations":[{"cited_title":"Dettmers, A","cited_arxiv_id":null,"evidence_quote":"Supplies the QLoRA parameter-efficient fine-tuning method used to train the SEAG models."},{"cited_title":"URLhttps://github.com/meta-llama/lla ma-models/blob/main/models/llama3_2/MO DEL_CARD.md","cited_arxiv_id":null,"evidence_quote":"One of the base models fine-tuned as a SEAG model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the two external generators used to evaluate the framework."},{"cited_title":"URLhttps://www.anthropic.com/news/c laude-4","cited_arxiv_id":null,"evidence_quote":"The second external generator used to evaluate the framework."}],"review_version":1}