{"id":"bea492ea-a62a-45f5-907c-5692e4e2035b","arxiv_id":"2505.00951","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A hybrid system that filters sensitive purchases out of LLM-based recommendation prompts and generates those recommendations locally nearly matches full-data recommendation quality while keeping most sensitive data off the server.","lead":"This paper builds a privacy shield for AI shopping recommenders: a local classifier hides sensitive purchases (like health items) from the cloud, while a small on-device AI generates those recommendations locally. The hybrid system nearly matches the recommendation quality of sharing everything, but still leaks about 22% of sensitive items through misclassification.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Privacy guarantee lacks external anchor: sensitivity labels and scores are generated by ChatGPT-4o, the same model family as the cloud recommender, so the reported 22.28% leakage and the privacy-utility trade-off are ungrounded.","rationale":"The reader's weakest_assumption identifies the same concern, and I agree: the privacy-leakage metric is anchored to ChatGPT-4o-generated labels and scores. This is the most load-bearing issue because the paper's central claim is explicitly about preserving privacy while maintaining utility. If the ground truth for sensitivity is not externally valid, then both the privacy guarantee (22.28% leakage) and the utility-recovery measurements (deobfuscator operates on the classifier's sensitive set) are called into question. The paper transparently documents the use of ChatGPT-4o for labeling and scoring, but it never validates those labels against human judgments. Other weaknesses—the undefined 'Only Local' baseline, the 0.3 vs 0.5 threshold contradiction, missing error bars—are real and should be fixed, but they are secondary to the missing external anchor because they affect reproducibility rather than the validity of the privacy metric itself. The proposed concrete test—human annotation of a random sample followed by recalculation of PL_b and PL_s—would settle whether the circularity is harmful. Since the reader's CONDITIONAL verdict already requires independent labels, my assessment does not change the verdict; it sharpens the reason it is needed.","tokens_in":21899,"tokens_out":8213,"duration_ms":79716,"concrete_test":"Select a random sample of 500 products from the evaluation set (or from the BERT test set). Have three independent human annotators label each product as sensitive/nonsensitive and assign sensitivity scores using the operational definition in Section 2.1. Compute inter-annotator agreement (e.g., Fleiss' kappa) and agreement with ChatGPT-4o labels. Recompute the BERT classifier's false-negative rate and Table 4's PL_b and PL_s against the human labels. If agreement is poor (kappa < 0.6) or PL_b/PL_s shift by more than 5 percentage points, the privacy claim is not robust. This directly tests whether the self-referential labeler supports the central privacy-utility trade-off.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that BERT Obf + Deobf preserves privacy while nearly matching baseline utility depends on Table 4's privacy-leakage numbers (PL_b=22.28%, PL_s=10.99%). These are computed against ground-truth sensitivity labels (Section 5.3) and sensitivity scores (Section 3.2.2) that are both produced by ChatGPT-4o via few-shot prompts (Appendices A.3/A.4). The cloud recommender is also ChatGPT-4o. As a result, the BERT obfuscator is trained to mimic ChatGPT-4o's sensitivity judgments, and 'leakage' is defined as disagreement with those same judgments. If ChatGPT-4o's notion of sensitivity diverges from what users or regulators treat as private, the actual fraction of sensitive products exposed to the server could be substantially different from the reported 22%. The deobfuscator also only sees products the ChatGPT-4o-trained classifier labels sensitive, so the utility-recovery numbers are conditional on the same unvalidated labeler. The paper's Limitations section concedes that sensitivity is subjective and user-specific, but provides no human or external validation of the labels. The privacy guarantee is therefore not grounded in an independent standard.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a hybrid privacy-preserving framework for LLM-based product recommendation. A fine-tuned BERT classifier (trained with focal loss on ChatGPT-4o-labeled products) serves as an obfuscator that removes sensitive products from a user's purchase history; the remaining nonsensitive products are sent to a cloud ChatGPT-4o recommender, while a local Llama 3.2 1B model generates recommendations for the withheld sensitive products; the two lists are merged. Using the Amazon Reviews 2023 dataset, the authors evaluate HR@10 (categorical and semantic), category-distribution L1/L2 distances relative to a full-history baseline, and two privacy-leakage metrics. The central empirical claim is that BERT Obf + Deobf (R'') nearly matches baseline utility (Categorical HR@10 0.6061 vs 0.6263; Semantic HR@10 0.3117 vs 0.3045) while incurring 22.28% binary privacy leakage, and that the approach runs on consumer hardware.","tokens_in":22067,"tokens_out":6047,"duration_ms":54771,"significance":"If the central numbers are accepted, the paper demonstrates a practical design point for LLM recommendation with on-device filtering: rather than encrypting or perturbing all prompts, the system withholds only sensitive products and uses a small local model to recover the lost sensitive recommendation channel. The hardware measurements (Table 7, Section 7.2) and the explicit prompt appendices are useful engineering contributions, as is the frank Limitations section. However, the privacy guarantee is currently not externally anchored: the labels and sensitivity scores used to measure leakage are generated by ChatGPT-4o, the same model family used as the cloud recommender. Because the central privacy claim depends on this unvalidated ground truth, and because the utility numbers are single-run stochastic outputs with small differences, the paper's headline conclusion is not yet established to the standard of a journal publication.","major_comments":[{"comment":"The privacy metrics PL_b and PL_s are computed against ground-truth sensitivity labels (Section 5.3) and sensitivity scores (Section 3.2.2) that are both generated by ChatGPT-4o via few-shot prompts (Appendices A.3 and A.4); since the cloud recommender is also ChatGPT-4o (Section 4.2.1), the 22.2833% binary leakage in Table 4 measures the BERT classifier's disagreement with ChatGPT-4o's sensitivity judgments rather than leakage relative to an independent standard. Please validate the labels and scores against human annotation or an established medical/privacy taxonomy on at least a held-out sample, and report inter-annotator agreement; without this, the privacy-utility trade-off claim is ungrounded.","section":"Section 3.2.2, Section 5.3, Table 4"},{"comment":"The inference rule in Section 4.1.3 states that products with sensitivity scores > 0.5 are classified as sensitive and scores <= 0.5 as nonsensitive, but Section 5.3 states that the decision threshold for the sensitive class was lowered to 0.3 to increase recall; the paper does not say which threshold produced the Table 4 results. This ambiguity affects both the reported 22.28% leakage and the HR@10 numbers, so reproducibility requires stating the operational threshold and reporting sensitivity of the main results to threshold choice.","section":"Section 4.1.3 vs Section 5.3"},{"comment":"Table 4 includes an 'Only Local' configuration with Categorical HR@10 0.4667 and Semantic HR@10 0.2398, but Section 5.2 defines only Baseline, Only Obfuscator, and Obfuscator + Deobfuscator; the 'Only Local' input, prompt, and merge procedure are never specified. Please define this baseline precisely or remove it, since the text uses it to argue that server-side nonsensitive data contributes to utility.","section":"Section 5.2 and Table 4"},{"comment":"The utility results in Table 4 appear to be single runs of stochastic LLMs with no repetition, seeds, or error bars; the headline utility differences are small (Categorical HR@10 0.6061 vs 0.5960 for BERT Obf Only, and Semantic HR@10 0.3117 vs 0.3045 for the no-privacy Baseline), so without confidence intervals or significance tests the claim that BERT Obf + Deobf 'closely approaches' the baseline is not statistically grounded. Please report multiple runs (at least five) with mean and variance, or bootstrap confidence intervals, for both HR@10 and the distance metrics.","section":"Section 6.1, Table 4"}],"minor_comments":[{"comment":"The phrase 'to a large extend' should be 'to a large extent'.","section":"Abstract"},{"comment":"The row labeled 'Categorical Obf Only (R)' uses R instead of R-prime; the notation in Section 5.2 reserves R-prime for obfuscation-only setups.","section":"Table 4"},{"comment":"The recovery percentage is computed as 41.52% in footnote 1 but reported as 41.42% in Table 6 for the same numbers; please reconcile the arithmetic.","section":"Section 6.4"},{"comment":"The text before Section 6.2 contains a garbled passage ('Analysis of Average Category Distribution Distances ies...') that appears to be an editing artifact; please fix the sentence.","section":"Section 6"},{"comment":"The word 'noticable' appears in Section 3.2.1 and Section 6.3; it should be 'noticeable'.","section":"Section 3.2.1"},{"comment":"Inference times are reported as bare numbers (0.1808, 6.3816, 2.7428, 3.8196) without units; please state that these are seconds and clarify whether the deobfuscation time refers to the 8-bit quantized model described in that section.","section":"Section 7.2"},{"comment":"Reference [3] contains a typo ('ccessed' instead of 'Accessed').","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core concern is the circularity of the privacy evaluation: the same model family provides the ground-truth sensitivity labels, the sensitivity scores, and the cloud recommendations. If the authors cannot supply an external validation (for example, a human-annotated sensitivity sample), I would recommend that the privacy claims be substantially downgraded and the paper be reframed as a utility-preserving obfuscation study. I do not see the issues as fatal, but they are load-bearing and require additional experiments rather than copyediting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful system paper: the BERT-obfuscator plus local Llama 3.2 deobfuscator combination for e-commerce LLM recommendations is new as far as I can tell, and the utility numbers are close to the full-data baseline. Second, the privacy guarantee is not yet anchored to anything external: the sensitivity labels that train the BERT classifier and the sensitivity scores used to compute leakage are both generated by ChatGPT-4o, the same model family that serves as the cloud recommender. If ChatGPT-4o's notion of 'sensitive' diverges from what users or regulators consider private, the reported 22.28% leakage is not meaningful.\n\nThe paper does several things well. The system design is clear and modular; the threat model is sane (semi-honest server, trusted on-device processing); the evaluation uses real Amazon Reviews data; and the hardware measurements show the local components fit on consumer-grade devices, including an 8-bit Llama result on a laptop GPU. The strongest result is that BERT Obf + Deobf (R'') gets Categorical HR@10 of 0.6061 and Semantic HR@10 of 0.3117, essentially matching the baseline's 0.6263 and 0.3045. That is a meaningful demonstration that product-level filtering plus local generation can recover most utility.\n\nThe soft spots are real but mostly addressable. The circularity is the load-bearing one: both the training labels (Appendix A.3) and the leakage measurement scores (Appendix A.4) come from ChatGPT-4o. The paper's own Limitations section concedes that sensitivity is subjective and user-specific, yet no human or external validation is provided. A reviewer should ask for a human-annotated set of product labels and a re-computation of leakage against that set. Also, the utility and leakage numbers are single runs with no error bars; LLM outputs are stochastic, and a rerun could shift HR@10 by a few points. There is a threshold inconsistency: inference is described as scores >0.5 being sensitive in Section 4.1.3, but training says the threshold was lowered to 0.3 in Section 5.3. The 'Only Local' baseline appears in Table 4 but is never defined in Section 5.2. And the 22.28% leakage tempers the abstract's 'to a large extend' claim—that phrase should be softened or the leakage reduced.\n\nNone of these are fatal. They are the kind of thing a serious referee round can fix. The category-distribution improvement is partly enforced by prompt instructions and the proportional allocation formula, so I wouldn't overweight that result, but the HR@10 comparisons are still informative.\n\nBottom line: this is a solid engineering contribution that deserves peer review. I'd send it out, but with a clear request for independent sensitivity labels, error bars, and a resolved threshold definition.","headline":"A clean system paper with a real engineering contribution, but the privacy claim is unanchored until the sensitivity labels get external validation.","tokens_in":22694,"tokens_out":3087,"would_cite":true,"duration_ms":29530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tries to establish that a hybrid split-processing scheme lets cloud-based LLM recommenders produce near-baseline recommendations while sensitive purchases never leave the user's device.","keywords":["LLM-based recommendation","privacy preservation","obfuscation","deobfuscation","BERT classifier","local inference","hit rate","category distribution"],"falsifier":"Re-annotate a sample of the products with human raters from diverse backgrounds, then rerun the BERT classifier and the PL_b and PL_s computations against those human labels; if the measured leakage rises well above 22.28% or the classifier's F1 score drops materially, the reported privacy guarantee does not transfer to real users.","tokens_in":21613,"feed_emoji":"🔒","tokens_out":4435,"duration_ms":43592,"temperature":0.7,"pith_summary":"The paper proposes a hybrid privacy-preserving framework for LLM-based product recommendation. A fine-tuned BERT classifier separates a user's purchase history into sensitive and nonsensitive products, only the nonsensitive part is sent to a large server-side LLM, and a small local Llama model generates the sensitive recommendations on-device. Experiments on real e-commerce data report that this merged list recovers most of the utility lost by filtering: Categorical HR@10 drops only from 0.6263 (full-data baseline) to 0.6061, while Semantic HR@10 rises slightly from 0.3045 to 0.3117. The remaining privacy leakage of 22.28% is attributed to BERT misclassifications. If the results hold, users of cloud recommenders could obtain near-baseline personalization while keeping health-related purchase history off the server.","feed_headline":"Privacy-preserving LLM recommendations nearly match full-data quality","feed_subtitle":"Health-related purchases stay on-device while a hybrid pipeline keeps category hit rate within 4% of the baseline.","key_machinery":"The load-bearing mechanism is the obfuscator-deobfuscator pair. The obfuscator is a BERT-based binary classifier, fine-tuned with focal loss, class weighting, and a lowered decision threshold of 0.3, that labels each product in a purchase history as sensitive or nonsensitive. Nonsensitive products go to a server-side ChatGPT-4o recommender whose prompt requests category proportions matching the nonsensitive history, while sensitive products go to a locally running Llama 3.2 1B deobfuscator. The two recommendation lists are merged in proportion to the sizes of the two subsets. The evaluation apparatus includes ChromaDB vector retrieval to map generated text to real products, a sentence-transformer embedding model for semantic similarity, and binary and semantic privacy-leakage metrics.","core_discovery":"The central claim is that obfuscating sensitive products before a cloud call costs little recommendation utility when a local deobfuscator restores the sensitive side of the recommendation list. For the best configuration, BERT Obfuscation plus local Deobfuscation (R''), the paper reports Categorical HR@10 of 0.6061 and Semantic HR@10 of 0.3117, closely approaching the no-privacy baseline of 0.6263 and 0.3045, while binary privacy leakage stays at 22.2833% because the BERT classifier fails to flag some sensitive items. The paper also reports that deobfuscation reduces category-distribution distances to the baseline and that the whole pipeline can run on a consumer-grade laptop with roughly 3.8 seconds of added latency.","pith_inferences":["The privacy guarantee is only as strong as ChatGPT-4o's notion of sensitivity, because the same model family generates both the classifier's training labels and the leakage scores; an independent human-annotated benchmark could shift the reported 22.28% leakage substantially.","The paper's sensitivity definition is health-centric; extending the classifier to income, relationship status, or other private attributes is described as straightforward but untested, and cumulative purchase-history inference remains a leakage channel the paper itself acknowledges.","A testable extension: replacing the local Llama 3.2 1B deobfuscator with a stronger local ranker should push Semantic HR@10 further above the full-data baseline, since the paper already reports that the semantic metric slightly exceeds the baseline in its best configuration."],"forward_implications":["Cloud LLM recommender services could offer a privacy mode that requires no server-side changes and delivers near-baseline recommendation quality.","The utility gap between the privacy-preserving system and the full-data baseline is tied to classifier accuracy, so better sensitivity classification would simultaneously reduce leakage and further close that gap.","The local deobfuscator restores sensitive-category diversity that obfuscation-only schemes lose, as measured by the reported L1 and L2 category-distribution distances.","The measured hardware overhead suggests the approach is deployable on consumer devices: the paper reports about 3.8 seconds of added end-to-end latency on a laptop with an 8-bit quantized local model."],"supporting_citations":[{"why":"Supplies the pretrained BERT architecture that is fine-tuned into the product-sensitivity obfuscator.","marker":"[17]"},{"why":"Is the server-side ChatGPT-4o recommender and also the model used to generate sensitivity labels and sensitivity scores.","marker":"[43]"},{"why":"Provides the Llama 3.2 model card for the lightweight local deobfuscator used to generate sensitive recommendations on-device.","marker":"[2]"},{"why":"Supplies the Amazon Reviews 2023 dataset from which purchase histories and product metadata are drawn.","marker":"[32]"},{"why":"Provides the focal loss used to train the BERT classifier with emphasis on the harder-to-classify sensitive class.","marker":"[38]"},{"why":"Provides the ChromaDB vector-based retriever that maps generated recommendations to actual product entries for evaluation.","marker":"[16]"},{"why":"Supplies the all-MiniLM-L6-v2 embedding model used to compute Semantic HR@10 via cosine similarity.","marker":"[22]"}],"fun_headline_variants":["Privacy-preserving LLM recs almost as good as full-data","Local deobfuscation restores sensitive picks in private LLM recommender","Hybrid LLM recommender: only nonsensitive data goes to cloud","Consumer-grade hardware enables private LLM recommendations","Obfuscation plus local recovery keeps LLM rec utility high"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The privacy account rests on ChatGPT-4o-generated sensitivity labels being a trustworthy ground truth; if users' actual privacy judgments diverge from that model's, both the classifier's error rate and the measured leakage could be very different.","fun_headline_variants_meta":{"raw":{"variants":["Privacy-preserving LLM recs almost as good as full-data","Local deobfuscation restores sensitive picks in private LLM recommender","Hybrid LLM recommender: only nonsensitive data goes to cloud","Consumer-grade hardware enables private LLM recommendations","Obfuscation plus local recovery keeps LLM rec utility high"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1617,"prompt_tokens":916,"completion_tokens":701,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":611}},"tokens_in":532,"tokens_out":701,"duration_ms":6874,"temperature":1.0,"reasoning_tokens":611,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:30:49.946738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a sample of the products with human raters from diverse backgrounds, then rerun the BERT classifier and the PL_b and PL_s computations against those human labels; if the measured leakage rises well above 22.28% or the classifier's F1 score drops materially, the reported privacy guarantee does not transfer to real users.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the all-MiniLM-L6-v2 embedding model used to compute Semantic HR@10 via cosine similarity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the server-side ChatGPT-4o recommender and also the model used to generate sensitivity labels and sensitivity scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Llama 3.2 model card for the lightweight local deobfuscator used to generate sensitive recommendations on-device."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ChromaDB vector-based retriever that maps generated recommendations to actual product entries for evaluation."}],"review_version":1}