{"id":"f860a4ed-8746-406e-9338-8eb26f472644","arxiv_id":"2501.19287","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DPS-MOZO samples each generated token from the product of per-example distributions mixed with the zero-shot distribution to make in-context learning differentially private without additive noise.","lead":"This paper proposes a decoding method that mixes one-shot example outputs with a zero-shot output before sampling, aiming to give language models differential privacy without adding noise. If it worked as claimed, providers could serve in-context learning generation tasks under privacy budgets like epsilon=2 with near-nonprivate quality.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Renyi additivity is misapplied in Eq. (6): common factors in a normalized pointwise product over one vocabulary do not cancel, so Theorem B.3's per-token RDP bound is unproven and the central DP claim collapses.","rationale":"The reader's weakest assumption identifies the same load-bearing error: Theorem B.2's additivity applies to product measures over independent components, whereas Algorithm 1 Line 16 samples from a normalized pointwise product of distributions over a single vocabulary. The proof of Theorem B.3 uses Eq. (6) to cancel the product over j ≠ i, which is invalid because the normalization constant of each candidate output distribution depends on all factors, and the common factors remain inside the Renyi integral rather than vanishing. This is not a minor typographical issue; it is the only step connecting the per-demonstration divergence constraint to the claimed per-token RDP guarantee. Without it, the central privacy claim has no theoretical support, and every downstream privacy number (ε = 1, 2, 4 in Table 1) rests on an unproven bound. The paper's empirical utility results and membership-inference experiments are interesting but cannot substitute for a valid privacy proof. I see no reason to change the reader's REJECT verdict, so the recommendation remains unchanged. A corrected proof might establish a different, possibly larger bound, but that would require new analysis and likely different parameter accounting.","tokens_in":19773,"tokens_out":11530,"duration_ms":110160,"concrete_test":"For a two-token vocabulary with zero-shot q = (1/2, 1/2), take n = 2 demonstrations with f_1 = (0.9, 0.1), f'_1 = (0.1, 0.9), and f_2 = (0.9, 0.1), each satisfying the symmetric Renyi constraint D↔_α(f·, q) ≤ βα for a sufficiently large β. Compute the normalized product distributions P_D ∝ f_1 · f_2 and P_{D'} ∝ f'_1 · f_2, then evaluate D_α(P_D ∥ P_{D'}) numerically and compare it with D_α(f_1 ∥ f'_1). If the two values differ—which they must because the normalization ratio Z_{D'}/Z_D and the surviving common factor inside the integral are non-trivial—Eq. (6)'s cancellation step is invalid. This directly settles whether the proof of Theorem B.3 holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 Line 16 samples yt from the normalized pointwise product P_D(y) = (1/Z_D) ∏_{i=1}^{nshots} f_i(y), where f_i = p_{θ,λ_i}(·|x_i,y_i,x,y_<t). Theorem B.3 treats the output distributions for adjacent datasets D, D' as if their Renyi divergence decoupled into a term for the differing demonstration plus a zero term for the common factors (Eq. (6)). This would require additivity of Renyi divergence for product measures over independent spaces (Theorem B.2). That theorem does not apply here: all f_i are densities on the same token vocabulary, and the normalized pointwise product is not a product measure. Writing P_D(y) = (1/Z_D) ∏_i f_i(y) and P_{D'}(y) = (1/Z_{D'}) f'_i(y) ∏_{j≠i} f_j(y), the ratio is P_D/P_{D'} = (Z_{D'}/Z_D) · f_i/f'_i. The constant Z_{D'}/Z_D depends on all common factors, and inside the Renyi integral the common factors appear to the first power (α + (1−α) = 1), so they do not cancel. Thus Eq. (6) is false for nshots ≥ 2. Since subsampling amplification, composition over ntest·Tmax, and all reported (ε, δ) numbers are derived from Theorem B.3, the privacy guarantee is unsupported. The empirical results are encouraging but do not repair this mathematical gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DPS-MOZO, a decoding framework for differentially private in-context learning in text generation. For each next token, the method subsamples a few demonstrations from a private dataset, constructs per-demonstration one-shot token distributions mixed with a zero-shot distribution under a Renyi divergence constraint (with mixing weight lambda chosen by a bisection search), and samples the next token from the normalized pointwise product of the mixed distributions. Privacy is claimed via a per-token Renyi differential privacy bound, privacy amplification by subsampling, and composition over ntest*Tmax token generations. The paper reports strong empirical utility, e.g., only a 0.3% ROUGE-L F1 drop at epsilon=2 on SAMSum with Gemma 2 2B, and includes ablations and membership inference attacks.","tokens_in":20077,"tokens_out":7640,"duration_ms":66903,"significance":"If the privacy analysis were sound, DPS-MOZO would be a valuable contribution: it proposes noise-free DP for open-ended generation via sampling, avoids the heavy ensemble cost of prior work such as Wu et al. (2023), and reports favorable utility-privacy tradeoffs across multiple models and datasets. The empirical evaluation is reasonably thorough, with ablations over model size, number of shots, number of queries, and top-k, as well as membership inference experiments. However, the central theoretical claim -- the per-token Renyi divergence bound in Theorem B.3 -- is not established, and the error is load-bearing: the subsampling amplification, composition, and all reported (epsilon, delta) figures depend on it. The empirical results, while encouraging, do not repair the mathematical gap.","major_comments":[{"comment":"The proof applies Theorem B.2 (additivity of Renyi divergence for product measures) to the sampling distribution of Algorithm 1 Line 16, but that distribution is a normalized pointwise product of per-demonstration distributions over the same token vocabulary, not a product measure over independent components. For adjacent datasets D and D' differing in demonstration i, the ratio of the sampling densities is P_D(y)/P_{D'}(y) = (Z_{D'}/Z_D) * f_i(y)/f'_i(y), where Z_D and Z_{D'} are the normalization constants and depend on all common factors. The common factors therefore do not cancel inside the Renyi integral, and Eq. (6) is false for nshots >= 2. Since subsampling amplification (Theorem A.4), composition over ntest*Tmax tokens, and the reported (epsilon, delta) values all rely on Theorem B.3, the central privacy guarantee is unsupported.","section":"Appendix B, Theorem B.3 and Eq. (6)"},{"comment":"Even if the additivity step were replaced by a valid bound, the proof would need a new argument showing that the divergence between the normalized pointwise products is controlled by the per-factor divergence bounds; the current proof simply cancels common factors in the product, which is invalid. No alternative inequality (e.g., a tensorization or strong-data-processing argument for the specific product-of-experts distribution) is supplied. The paper therefore does not establish a per-token RDP bound for DPS-MOZO, and the claimed privacy guarantee should not be relied upon.","section":"Appendix B, Eq. (7) and surrounding argument"}],"minor_comments":[{"comment":"The pseudocode sets logits to -infinity for all yt in K, where K is the top-k public indices; this removes the most likely tokens rather than keeping them. If the intent is top-k truncation, the lines should read 'forall yt notin K' (and similarly in the text describing the truncation). As written, the pseudocode conflicts with the description and with the top-k ablation results.","section":"Algorithm 1, Lines 7 and 10"},{"comment":"The proof refers to 'Lemma B.2' when applying the additivity theorem, but the additivity result is stated as Theorem B.2; the citation should be corrected.","section":"Appendix B, proof of Theorem B.3"},{"comment":"The bullet point says '0.3% ROGUE-L' but the metric is ROUGE-L; the typo should be fixed.","section":"Contributions, Section 1"},{"comment":"The caption says the figure plots lambda(t) evaluated on 'SAMSum, E2E, and Spider', but the subplot labels and the rest of the text refer to WikiLarge; the dataset name should be corrected.","section":"Figure 3 and Section 5.3"},{"comment":"The notation 'epsilon_1 alpha' and 'epsilon_2 alpha' in the statement of the triangle-like inequality is confusing; please clarify whether these are bounds on D_alpha or scaled bounds, and state the exact quasi-triangle inequality being used.","section":"Theorem B.1"}],"recommendation":"reject","confidential_remarks":"The paper builds heavily on the authors' own prior work (Flemings et al., 2024) for the mixing formula and the lambda optimization, without independent verification of that prior result; while that reliance alone would not be disqualifying, it compounds the concern that the current proof is not self-contained. The empirical results, including the membership inference attacks, are encouraging, but they cannot substitute for a correct formal privacy guarantee. Given that the central theorem is invalid as written, I recommend rejection; a future revision with a genuinely correct privacy proof would need to be re-reviewed from scratch."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before spending time on it. First, the problem it targets is real — DP for in-context learning on generation tasks is under-served, and Wu et al.'s 100x-ensemble approach is expensive. Second, the paper's central claim, the DP guarantee, is not established. The privacy proof in Appendix B misapplies Rényi additivity, and everything downstream of that step (subsampling amplification, composition over ntest·Tmax tokens, the reported ε=2 numbers) inherits the gap.\n\nThe core idea is clean: mix each one-shot output distribution with the zero-shot output so each mixed distribution lies in a divergence ball around the public distribution, then sample from the product of the mixed distributions. The connection to Flemings et al. (2024) is explicit, and the offline/online variants are a legitimate new setting. The experiments are a genuine strength: three datasets, two model families, ablations over model size, shots, queries, and top-k, plus membership-inference attacks that land near chance. The writing is clear and the limitations (logit access, token-only threat model) are stated plainly.\n\nThe soft spot is load-bearing. In Theorem B.3, the output distribution of Algorithm 1 is the normalized pointwise product P_D(y) = (1/Z_D) ∏_i f_i(y) over a single vocabulary, but Eq. (6) applies the additivity theorem for product measures over independent spaces, splitting the divergence into a \"common factors\" term and the changed demonstration. Those factors do not cancel: they appear inside the normalization constants Z_D and Z_D', and the ratio Z_D'/Z_D depends on every common factor. For adjacent D and D', the likelihood ratio is (Z_D'/Z_D)·(f_i/f'_i), and that constant does not drop out of the Rényi integral. The stress-test note holds up on reading; I could not find a way to rescue Eq. (6) as stated. Since every reported (ε, δ) number derives from Theorem B.3, the privacy guarantee is unsupported. The MIA results are reassuring but do not repair the math.\n\nMinor points: the mixing formula and λ selection come from the same group's prior work, which is acceptable but worth an independent check; λ is capped at 1.5 by heuristic; the bisection solver for β and λ can yield loose bounds.\n\nThis is for anyone working on private prediction or DP ICL. It deserves a serious referee — the problem matters, the experiments are real, and the flaw is specific enough to be fixable (a direct analysis of the normalized product, or a different sampling scheme). Send it to review, but the authors should expect a major revision aimed at the privacy proof.","headline":"The paper's DP guarantee rests on a misapplied Rényi additivity step in Eq. (6), so the central privacy claim is unsupported; the empirical study is solid but does not repair the math.","tokens_in":20661,"tokens_out":10350,"would_cite":false,"duration_ms":87985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that sampling from a product of one-shot and zero-shot next-token distributions, each kept in a small Renyi-divergence ball around the zero-shot output, gives differential privacy for in-context text generation without…","keywords":["differential privacy","in-context learning","large language models","text generation","Renyi divergence","subsampling amplification","decoding framework","privacy-utility tradeoff"],"falsifier":"Build a three-token vocabulary and two adjacent demonstration sets differing in one example, run Algorithm 1 on a fixed query, compute the exact normalized product distributions for both datasets, and evaluate their Renyi divergence for small integer $\\alpha$; if any token yields $D_\\alpha > 4\\beta\\alpha$ for the $\\beta$ chosen by Eq. (8), the paper's per-token privacy bound is violated.","tokens_in":19555,"feed_emoji":"🔒","tokens_out":15794,"duration_ms":125084,"temperature":0.7,"pith_summary":"In-context learning can leak private demonstrations through generated text, and prior differentially private generation schemes pay for privacy with injected noise or hundreds of extra inferences. This paper proposes DPS-MOZO, a decoding rule that for each next token builds several one-shot output distributions, each conditioned on one private demonstration, mixes each with the public zero-shot output, and then samples from their product. The mixing is controlled so that every per-demonstration distribution stays within a Renyi-divergence ball around the zero-shot output, which the paper proves yields an $(\\alpha, 4\\beta\\alpha)$-RDP guarantee per token and, with subsampling and composition across $n_{\\text{test}} T_{\\max}$ tokens, an $(\\varepsilon,\\delta)$-DP guarantee for the whole response. Because privacy comes from the bounded mixing and from sampling, no calibrated noise is added, which the paper argues preserves generation quality. On SAMSum with Gemma 2 2B the method reports $\\epsilon=2$ with a 0.3% ROUGE-L F1 drop relative to non-private few-shot ICL.","feed_headline":"Mix one-shot and zero-shot outputs for DP text with 0.3% ROUGE-L loss","feed_subtitle":"New decoding scheme hits epsilon=2 DP for in-context learning with only 0.3% quality loss.","key_machinery":"The engine of the method is a mix-then-sample one-token decoder. For each sampled demonstration, the algorithm interpolates the one-shot logits with the zero-shot logits through a weight $\\lambda_i$, and chooses the largest $\\lambda_i$ that keeps the symmetric Renyi divergence between the mixed and zero-shot distributions at most $\\beta\\alpha$. The next token is then drawn from the normalized product of these mixed distributions over the single shared vocabulary; the product is the ensemble-decoding step, and the normalization is the part of the algorithm that matters for the proof. A public top-k truncation of the vocabulary, taken from the zero-shot logits, reduces the output space without revealing private information.","core_discovery":"The paper's central claim is that private text generation from in-context examples can be obtained without additive noise by constraining how much each demonstration may influence the next-token distribution. For each sampled demonstration $(x_i,y_i)$, Algorithm 1 forms the mixed distribution $p_{\\theta,\\lambda_i}(\\cdot|x_i,y_i,x,y_{<t}) = \\operatorname{softmax}(\\lambda_i \\operatorname{logit}_\\theta(\\cdot|x_i,y_i,x,y_{<t}) + (1-\\lambda_i)\\operatorname{logit}_\\theta(\\cdot|x,y_{<t}))$ and chooses the largest $\\lambda_i$ such that the symmetric Renyi divergence to the zero-shot distribution is at most $\\beta\\alpha$. The next token is then sampled from the normalized product of these mixed distributions over the shared vocabulary, with the token set first truncated to public top-k indices from the zero-shot logits. After subsampling $n_{\\text{shots}}$ demonstrations without replacement, Theorem B.3 bounds the per-token privacy loss by $4\\beta\\alpha$; subsampling amplification and composition over all tokens convert this to an $(\\varepsilon,\\delta)$-DP guarantee, reported at $\\epsilon=2$ with minimal utility loss.","pith_inferences":["A testable extension the paper does not run: on tasks where a demonstration sharply changes the next-token distribution, the allowed $\\lambda_i$ will shrink, so the utility gap versus zero-shot should widen; the reported $\\lambda(t)$ trajectories across datasets are consistent with this but do not confirm it.","The same divergence-ball recipe could be combined with other logit-level decoding transforms, such as nucleus sampling or repetition penalties, as long as the final distribution remains inside the ball; the paper only experiments with top-k truncation of the public token set.","The proof's additivity step in Eq. (6) is the one place a reader should re-derive from the normalized product directly; an exact toy-model check of that step would determine whether the reported $\\epsilon$ values are the true guarantee or an optimistic one."],"forward_implications":["An operator can set the privacy budget by choosing the number of demonstrations, queries, and the divergence radius $\\beta$ rather than by adding noise, so the method scales to generation settings where the output space is too large for standard private aggregation.","The offline variant DPS-MOZO+Off produces differentially private few-shot demonstrations once and then permits unlimited non-private decoding from them, because differential privacy is preserved under post-processing.","At $\\epsilon=2$ on SAMSum with Gemma 2 2B, the method stays within 0.3% ROUGE-L F1 of non-private few-shot ICL, and it matches or exceeds the prior embedding-aggregation baseline at equal privacy across SAMSum, E2E, and WikiLarge.","The measured mixing weights are smallest at the first token and grow after the third or fourth token, meaning the privacy budget is consumed mostly when the model is choosing how to start the response."],"supporting_citations":[{"why":"Provides the embedding-subspace aggregation baseline for private ICL generation that DPS-MOZO is compared against and outperforms.","marker":"Wu et al. (2023)"},{"why":"Supplies the subsampled Renyi differential privacy amplification bound used as Theorem A.4.","marker":"Wang et al. (2019)"},{"why":"Supplies the RDP definition, composition rule, and post-processing theorem used in the privacy accounting.","marker":"Mironov (2017)"},{"why":"Supplies the triangle-like inequality for Renyi divergence used to derive the factor 4 in Theorem B.3.","marker":"Steinke (2022)"},{"why":"Supplies the additivity property of Renyi divergence invoked in Eq. (6) to cancel common product factors.","marker":"Van Erven & Harremos (2014)"},{"why":"Provides the prior result that sampling from divergence-bounded distributions gives privacy, and the lambda-mixing optimization template.","marker":"Flemings et al. (2024)"},{"why":"Provides the logit-mixing formula and the choice of lambda values that DPS-MOZO adapts.","marker":"Xu (2023)"},{"why":"Establishes the offline/online two-phase private ICL setup and the generation of differentially private few-shot demonstrations.","marker":"Tang et al. (2023)"},{"why":"Supplies the membership-inference attack protocol and the AG News/Trec classification setup used for the empirical privacy evaluation.","marker":"Duan et al. (2024)"}],"fun_headline_variants":["Mix one-shot and zero-shot outputs for DP text with 0.3% loss","Noiseless DP: sampling mixed outputs gives strong privacy","Epsilon=2 DP via output mixing with minimal utility loss","In-context learning made private by mixing demonstrations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof's load-bearing step is that the common per-demonstration factors inside the product distribution cancel when comparing two adjacent datasets, but the algorithm renormalizes the product over one shared vocabulary, so those common factors affect the normalization constant and cannot be canceled.","fun_headline_variants_meta":{"raw":{"variants":["Mix one-shot and zero-shot outputs for DP text with 0.3% loss","Noiseless DP: sampling mixed outputs gives strong privacy","Epsilon=2 DP via output mixing with minimal utility loss","In-context learning made private by mixing demonstrations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000898,"raw_usage":{"total_tokens":3923,"prompt_tokens":1058,"completion_tokens":2865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2793}},"tokens_in":674,"tokens_out":2865,"duration_ms":17993,"temperature":1.0,"reasoning_tokens":2793,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:42:07.749850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a three-token vocabulary and two adjacent demonstration sets differing in one example, run Algorithm 1 on a fixed query, compute the exact normalized product distributions for both datasets, and evaluate their Renyi divergence for small integer $\\alpha$; if any token yields $D_\\alpha > 4\\beta\\alpha$ for the $\\beta$ chosen by Eq. (8), the paper's per-token privacy bound is violated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the subsampled Renyi differential privacy amplification bound used as Theorem A.4."},{"cited_title":"Flocks of stochastic parrots: Differentially private prompt learning for large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the membership-inference attack protocol and the AG News/Trec classification setup used for the empirical privacy evaluation."}],"review_version":1}