{"id":"864ec6f3-39b0-43e8-a232-c90f0e1525e6","arxiv_id":"2507.15997","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Twelve DP experts converged on a core set of parameters, including epsilon, delta, and the unit of privacy, that a standardized differential privacy label should disclose to technical audiences.","lead":"Researchers interviewed 12 differential privacy experts to ask which technical details a public 'privacy label' should disclose, and built a prototype label from the answers. The study offers a first expert-backed checklist for standardizing how real-world privacy guarantees are reported.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Expert consensus is strong only for privacy parameters and the unit of privacy; most label categories show moderate or weak consensus, so the claim that expert consensus supports the comprehensive DP label overstates the evidence.","rationale":"The reader's verdict is CONDITIONAL and cites sample representativeness as the weakest assumption. I find a more direct internal problem: within the interviewed sample, consensus is strong for only two of eight substantive parameter categories. The central claim uses the strength of expert consensus as its warrant; if that consensus is weak for most of the label's content, the claim overstates what the data show. The delta example illustrates the issue: the paper counts three experts emphasizing delta, yet the headline example includes delta as a consensus parameter. This is not a matter of external generalizability; it is a mismatch between the reported evidence and the claim. I therefore partially agree with the reader: the reader's formal weakest assumption (representativeness) is valid, but the more load-bearing concern is the internal weakness of the consensus itself. The proposed transcript re-coding would settle this by providing per-parameter endorsement counts. If the counts confirm weak consensus, the paper should be accepted only with revisions that distinguish consensus-backed core parameters from author-proposed extensions; the verdict therefore remains CONDITIONAL, and I recommend no change to the reader's verdict.","tokens_in":25604,"tokens_out":10461,"duration_ms":108549,"concrete_test":"Re-code the 12 interview transcripts from Section 3.3 to count, for each parameter displayed in Figure 1, how many experts independently endorsed it as necessary for a DP label (excluding prompted agreement and merely mentioning the term). Compare these counts with the consensus tiers in Table 1 and with the label's inclusion of borderline parameters such as deployment model, empirical privacy metrics, and privacy semantics. If any included parameter is independently endorsed by fewer than half of the experts, the Section 4 claim of 'significant consensus' should be narrowed to the core set, and the label should be described as part author-proposed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, stated in Section 4, is that 'experts agreed that important parameters like epsilon, delta, and the unit of privacy are vital for transparency in DP deployments' and that this consensus supports a standardized DP label. The paper's own consensus data do not support this for most of the label. Table 1 assigns the strongest consensus tier only to 'Privacy parameters' and 'Unit of privacy' (8-12 experts). 'Utility information,' 'Mechanism used,' and 'Algorithm hyperparameters' fall in the moderate tier (4-7 experts), and 'Deployment model,' 'Empirical privacy metrics,' and 'Privacy interpretation-semantics' fall in the weakest tier (1-3 experts). Section 5.1 states that when expert opinions diverged, the authors 'used our best judgment,' so the comprehensive label is not fully consensus-backed. The specific example of delta is especially problematic: Section 4.1.1.2 reports only three experts emphasizing delta's role, and Table 3 reports only three 'yes' responses for delta for technical audiences, while one expert (P05) explicitly argued that if delta is set small by consensus, there is no need to convey it. Thus the 'significant consensus' framing conflates a strong core (epsilon and unit of privacy) with the authors' design choices for many other parameters. This is an internal-validity problem: even if the 12 experts were representative, the evidence would not establish consensus for most of the label.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a qualitative interview study with 12 DP experts (May–August 2024) conducted to identify which parameters should be included in a standardized 'DP privacy label' and how those parameters should be presented. The authors use semi-structured interviews, hybrid thematic analysis with two independent coders, and report counts of expert mentions for nine parameter categories. Based on these data, they propose a two-layer interactive HTML label targeted at technical users, and they discuss typical ranges, audience relevance, and presentation formats. The paper's central claim is that experts reached significant consensus about what to communicate, specifically epsilon, delta, and the unit of privacy, and that this consensus motivates the proposed label.","tokens_in":25848,"tokens_out":7070,"duration_ms":67960,"significance":"The study addresses a real gap: DP deployments are proliferating but lack standardized transparency documentation. If the findings are interpreted with appropriate scope, the paper makes a useful contribution by (i) providing a systematically coded list of expert-nominated parameters, (ii) documenting expert opinions on ranges and audiences, and (iii) offering a concrete, publicly available label prototype. Strengths include the published codebook (Appendix E), the two-coder process, the explicit discussion of limitations, and the caution about not over-claiming for end-user communication. The main concern is that the 'consensus' framing overstates what the data show; with more careful hedging, the work would be a solid basis for future standardization efforts.","major_comments":[{"comment":"The abstract and the Section 4 introduction claim 'significant consensus about what to communicate' and that experts agreed that 'epsilon, delta, and the unit of privacy are vital' for a DP label, but Table 1 and Section 4.1 show that this strong consensus tier (8–12 experts) covers only the combined 'Privacy parameters' category and 'Unit of privacy.' 'Utility information,' 'Mechanism used,' and 'Algorithm hyperparameters' fall in the moderate tier (4–7), and 'Deployment model,' 'Empirical privacy metrics,' and 'Privacy interpretation-semantics' fall in the weakest tier (1–3). Section 5.1 then states that 'When expert opinions diverged, we used our best judgment,' which means the label's comprehensive contents are not fully consensus-backed. Because the paper's contribution is explicitly framed as an 'Expert-Informed Privacy Label' and as a basis for standardization, the manuscript should either report consensus levels per parameter throughout, or sharply separate the consensus-backed core (epsilon, unit of privacy, and arguably the other privacy-loss measures) from the authors' design choices for the remaining categories. As written, the 'significant consensus' framing overstates the evidence.","section":"Section 4, Table 1"},{"comment":"Delta is the clearest instance of the gap between the consensus claim and the underlying counts. Only three experts are reported as emphasizing delta's role (Section 4.1.1.2), and Table 3 lists only three 'yes' responses for delta for a technical audience, with one expert (P05) explicitly arguing that if delta is set small by consensus there is no need to convey it. Yet delta is placed at the top of the proposed label in Figure 1 and treated as a core parameter throughout. This is not necessarily wrong, but the paper should explicitly acknowledge that delta's inclusion is a design decision informed by theoretical importance rather than by a demonstrated expert consensus, and it should explain the authors' judgment for including it despite the sparse support. Without that, a reader could infer that the label's most prominent parameters are all equally expert-endorsed.","section":"Section 4.1.1.2 and Table 3"},{"comment":"The two-layer structure is one of the paper's main design outputs, but the supporting evidence is partly an artifact of the interview protocol. The authors proposed the two-layer structure to experts (Appendix B, Focus 3) and disclose that experts did not independently suggest it; they then report that 'Experts unanimously supported our proposed layered structure.' This wording overstates the agreement: two experts (P01, P11) proposed adding a third layer, so the support was not unanimous in the sense of unqualified endorsement. Moreover, the question format may invite socially desirable agreement with the interviewer's proposal. The manuscript should present this result as 'all experts who commented on the proposed two-layer design responded positively, with two suggesting an additional intermediate layer,' and add a sentence noting the possibility of acquiescence bias in this portion of the data.","section":"Section 4.5 and Appendix B"}],"minor_comments":[{"comment":"The counts in Table 3 are incomplete: for example, Epsilon–General Audience reports 8 yes and 2 no, totaling 10 of 12 participants, and Delta–Technical Audience reports only 3 yes, leaving 9 participants unaccounted for. Please state whether the remaining experts declined to answer or were not asked, or report 'n' and missing values explicitly.","section":"Table 3"},{"comment":"The text reports 'Eleven experts praised our idea of a DP label with standardized contents' and 'Ten experts favored the two-layer structure,' but no counts are given for the remaining experts; please specify the number of experts who addressed each question and how non-responses were handled.","section":"Section 4.5"},{"comment":"The phrase 'Experts unanimously supported our proposed layered structure' is inconsistent with the immediately following statement that two experts (P01, P11) proposed a third layer; consider rephrasing to 'supported the layered concept.'","section":"Section 4.5"},{"comment":"In the 'Two-Layer Design' paragraph, 'plan-language descriptions' should be 'plain-language descriptions.'","section":"Section 5.1"},{"comment":"The eligibility survey asks 'Do you currently reside in the United States?' but Section 3.1 reports one participant from Europe; please clarify whether the residency criterion was waived or whether the question was only a screening preference.","section":"Appendix A"},{"comment":"The paragraph on counting notes that mentions are not an attempt to quantify findings, but Table 1 and Table 3 use count-based consensus tiers; clarify the counting unit (e.g., whether an expert mentioning any sub-parameter counts once for the category) to make those tiers interpretable.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid qualitative contribution, but the 'consensus' claim in the abstract and results needs to be tempered. The reader's assessment is accurate: the sample size and recruitment limit generalizability, and the authors already acknowledge this. The manuscript is suitable for a usable privacy/security venue, but not as-is; the revision should focus on aligning the claims with the reported counts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: it is a careful, honest qualitative study that gives the DP transparency community a first pass at a structured label, but the central claim of expert consensus only holds for a small core of parameters. The stress-test note is right. Table 1 puts only privacy parameters and unit of privacy in the strong tier (8–12 experts); utility, mechanism, and hyperparameters are moderate (4–7); deployment model, empirical metrics, and semantics are weak (1–3). The abstract and introduction nonetheless say experts agreed on epsilon, delta, and unit of privacy as vital. Epsilon and unit of privacy do have strong support; delta gets only three mentions in Section 4.1.1.2 and three 'yes' responses in Table 3. So the overstatement is real, and it matters because the prototype label includes all eight categories, with the authors explicitly filling gaps with 'best judgment' where experts diverged. That is a defensible design choice; calling it expert consensus is not.\n\nWhat the paper does well deserves credit. It is the first systematic interview study focused on what to disclose rather than how to explain a single parameter. The unit of privacy as a first-class disclosure item is a genuinely useful finding, and the list of overlooked parameters (sub-budgets, data universe, group privacy) is a concrete contribution. The methods are transparent: two coders, a published codebook, pilot interviews, and an honest limitations section. The HTML prototype is shipping, reproducible, and clearly labeled as a starting point, not a finished standard. These are real assets.\n\nSoft spots beyond the consensus framing: the two-layer label structure was proposed by the authors and then endorsed, so that part of the 'expert feedback' is partly a framing artifact. The sample is 12 self-selected experts, eleven US-based, mostly academic or industry researchers; the authors acknowledge this, but it still limits any claim to stakeholder-wide validity. Tables 3 and 4 have count inconsistencies that need reconciliation—epsilon general audience shows 8 yes, 2 no, leaving two experts unaccounted for. None of this sinks the paper; it all points to revisions.\n\nWho should read this: people working on DP registries, usable privacy, or transparency standards. It gives them a concrete parameter list and a prototype to react to, but it should be read as an expert-elicitation study, not as a consensus standard.\n\nI would send it to peer review. The topic is timely, the methods are solid for a qualitative study, and the mismatch between claims and data is fixable with a more careful framing and cleaned-up counts. A serious referee would push for that, not reject.","headline":"Useful expert-elicitation study with a solid prototype, but the 'significant consensus' frame outruns the data for most label categories.","tokens_in":26392,"tokens_out":1404,"would_cite":true,"duration_ms":17074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Twelve DP experts agree: a privacy label must say more than epsilon.","keywords":["differential privacy","privacy label","transparency","epsilon","delta","unit of privacy","expert interviews","standardization"],"falsifier":"Run the same interview protocol with a larger, preregistered sample that includes regulators, policymakers, legal scholars, and downstream data users in addition to DP researchers and engineers; if the resulting parameter list does not reproduce the nine categories or does not place $\\epsilon$, $\\delta$, and the unit of privacy at the top, the claimed expert consensus does not generalize beyond the original sample.","tokens_in":25392,"feed_emoji":"🏷️","tokens_out":6713,"duration_ms":70771,"temperature":0.7,"pith_summary":"Differential privacy guarantees become misleading when deployments disclose only $\\epsilon$, the paper argues. Through interviews with 12 experts, the authors try to establish what a complete disclosure should contain, and they find consensus around a shared list: the privacy parameters $\\epsilon$ and $\\delta$, the unit of privacy, the mechanism used, utility information, algorithm hyperparameters, deployment model, empirical privacy metrics, and privacy semantics. On that basis they build a prototype two-layer privacy label aimed at technical readers. If the expert consensus generalizes, this label offers a concrete starting point for standards and registries that document real-world differential privacy deployments.","feed_headline":"12 DP experts agree: disclose more than epsilon","feed_subtitle":"Experts say a standard label for differential privacy should report epsilon, delta, and the unit of privacy.","key_machinery":"The load-bearing artifact is the proposed DP label: a table modeled on nutrition labels that lists each expert-endorsed parameter with its value, ordered with privacy parameters at the top. Its design follows a two-layer structure, with a compact primary layer for summaries and a secondary layer, reached through an interactive interface, holding technical detail and plain-language explanations. The label is generated from a qualitative machinery: semi-structured interviews with 12 DP experts, transcribed and coded with a hybrid deductive-inductive thematic analysis, with mention counts indicating consensus strength.","core_discovery":"The paper's central claim is that expert consensus already exists on the content of a differential privacy disclosure: reporting $\\epsilon$ alone is insufficient and potentially misleading, while a standardized label that includes $\\epsilon$, $\\delta$, the unit of privacy (what entity or event is protected), and contextual parameters would let technical stakeholders accurately compare and assess deployments. The interview study yields nine parameter categories, with privacy parameters and the unit of privacy receiving the strongest expert support. The authors further report expert guidance on typical ranges (e.g., $\\epsilon \\le 4$ as an ideal, $\\delta$ near $10^{-5}$ to $10^{-6}$), on which parameters suit general versus technical audiences, and on presentation formats. Synthesizing this guidance, they design a two-layer, nutrition-style differential privacy label for technical users, leaving communication to the general public as an open problem.","pith_inferences":["If the label were machine-readable and embedded in data-release metadata, disclosure of the unit of privacy could become an automatic compliance check for deployments claiming differential privacy.","The contested status of utility information suggests that a single label may not serve all stakeholders; regulators, analysts, and data subjects may each need a tailored view.","A direct empirical test would be to give the prototype to practitioners comparing two DP releases and see whether it prevents the kind of unit-of-privacy misreadings documented in the paper.","The expert-suggested ranges (e.g., $\\epsilon$ up to 20 for high-dimensional data) could be benchmarked against actual published deployments to see whether industry practice matches expert norms."],"forward_implications":["A standard differential privacy disclosure should report $\\epsilon$ and $\\delta$ together with the unit of privacy; $\\epsilon$-only reporting is the transparency failure the paper targets.","Privacy registries and data-release documentation can adopt the two-layer label format, giving technical readers a full parameter list and non-experts a summary layer.","Draft normal ranges (e.g., $\\epsilon$ ideally at or below 4, $\\delta$ around $10^{-5}$ to $10^{-6}$) provide initial benchmarks, though the paper stresses context-dependence.","How to communicate differential privacy guarantees to the general public remains unsolved; the paper positions that as future work.","Future evaluation should test the label with technical users first, then iterate with end users through participatory design."],"supporting_citations":[{"why":"Supplies the formal definition of differential privacy that anchors the paper's account of $\\epsilon$ as the core privacy parameter.","marker":"[17]"},{"why":"Gives the algorithmic foundations of DP and the standard treatment of privacy parameters and guarantees.","marker":"[18]"},{"why":"Prior work the paper extends: explains only epsilon to non-experts, leaving the question of what to disclose unaddressed.","marker":"[41]"},{"why":"Proposes an Epsilon Registry for documenting DP deployments, which the label design builds on and redirects toward external communication.","marker":"[16]"},{"why":"Argues that incomplete or unclear DP descriptions can erode trust, motivating the paper's call to disclose the full parameter set.","marker":"[14]"},{"why":"Introduces the nutrition-label approach to privacy communication that the DP label adopts in tabular form.","marker":"[32]"},{"why":"Prototype layered privacy/security label for IoT devices whose two-layer structure the interviewers proposed and experts endorsed.","marker":"[19]"},{"why":"Demonstrates in a real deployment how omitting the unit of privacy led experts to misjudge a DP guarantee, motivating the label's emphasis on that parameter.","marker":"[29]"},{"why":"A privacy deployments registry that records parameters but does not always emphasize the unit of privacy, a gap the DP label addresses.","marker":"[48]"}],"fun_headline_variants":["Differential privacy gets a nutrition-style label, per experts","Epsilon alone misleading: experts draft DP standard label","Expert-backed DP label: what to disclose beyond epsilon","New DP label from 12 experts: full guarantee breakdown","Standardizing DP: experts agree on label parameters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 12 interviewed experts, mostly US-based academics and industry practitioners recruited through the authors' professional networks and a DP mailing list, represent the wider DP community well enough to anchor a standardization effort; if policymakers, regulators, or other stakeholders name a different set of essential parameters, the consensus list and label design lose their grounding.","fun_headline_variants_meta":{"raw":{"variants":["Differential privacy gets a nutrition-style label, per experts","Epsilon alone misleading: experts draft DP standard label","Expert-backed DP label: what to disclose beyond epsilon","New DP label from 12 experts: full guarantee breakdown","Standardizing DP: experts agree on label parameters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000786,"raw_usage":{"total_tokens":3408,"prompt_tokens":826,"completion_tokens":2582,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":442,"completion_tokens_details":{"reasoning_tokens":2505}},"tokens_in":442,"tokens_out":2582,"duration_ms":21124,"temperature":1.0,"reasoning_tokens":2505,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:19:56.807327+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same interview protocol with a larger, preregistered sample that includes regulators, policymakers, legal scholars, and downstream data users in addition to DP researchers and engineers; if the resulting parameter list does not reproduce the nine categories or does not place $\\epsilon$, $\\delta$, and the unit of privacy at the top, the claimed expert consensus does not generalize beyond the original sample.","supporting_citations":[{"cited_title":"Calibrating noise to sensitivity in private data analysis","cited_arxiv_id":null,"evidence_quote":"Supplies the formal definition of differential privacy that anchors the paper's account of $\\epsilon$ as the core privacy parameter."},{"cited_title":"What are the chances? explaining the epsilon parameter 13 Onyinye Dibia, Mengyi Lu, Prianka Bhattacharjee, Joseph P","cited_arxiv_id":null,"evidence_quote":"Prior work the paper extends: explains only epsilon to non-experts, leaving the question of what to disclose unaddressed."},{"cited_title":"Differential privacy in practice: Expose your epsilons! Journal of Privacy and Confidentiality , 9(2), 2019","cited_arxiv_id":null,"evidence_quote":"Proposes an Epsilon Registry for documenting DP deployments, which the label design builds on and redirects toward external communication."},{"cited_title":"Centering policy and practice: Research gaps around usable differential privacy","cited_arxiv_id":null,"evidence_quote":"Argues that incomplete or unclear DP descriptions can erode trust, motivating the paper's call to disclose the full parameter set."},{"cited_title":"nutrition label","cited_arxiv_id":null,"evidence_quote":"Introduces the nutrition-label approach to privacy communication that the DP label adopts in tabular form."},{"cited_title":"Ask the experts: What should be on an iot privacy and security label? In 2020 IEEE Symposium on Security and Privacy (SP) , pages 447–464","cited_arxiv_id":null,"evidence_quote":"Prototype layered privacy/security label for IoT devices whose two-layer structure the interviewers proposed and experts endorsed."},{"cited_title":"On the difficulty of achieving differential privacy in practice: user-level guarantees in aggregate location data","cited_arxiv_id":null,"evidence_quote":"Demonstrates in a real deployment how omitting the unit of privacy led experts to misjudge a DP guarantee, motivating the label's emphasis on that parameter."},{"cited_title":"The privacy deployments registry","cited_arxiv_id":null,"evidence_quote":"A privacy deployments registry that records parameters but does not always emphasize the unit of privacy, a gap the DP label addresses."}],"review_version":1}