{"id":"75cf25e8-c84f-40d4-a1ac-05490071cecb","arxiv_id":"2507.05872","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LDP3 is a multi-threaded open-source toolkit implementing six LDP protocols, seven post-processing methods, and utility metrics, with experiments showing that choosing the right combination improves accuracy and parallel execution speeds up runs.","lead":"This paper introduces LDP3, an open-source toolkit that bundles local differential privacy protocols, post-processing methods, and utility metrics, and runs experiments in parallel threads. It could help privacy researchers and engineers pick the best accuracy setting for their data without building every component from scratch.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Norm post-processing column equals 'w/o PP' in every table row, contradicting Eq. 17 for five of the six protocols; the reported utility comparisons do not yet establish that LDP3's PP implementations are faithful.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the Norm column equals the 'w/o PP' column in all three tables, which is inconsistent with Eq. 17 for protocols whose raw estimates do not sum to 1. This is not a minor typo. The paper's central experimental claim is that LDP3 provides faithful implementations of multiple PP methods and that choosing a good protocol/PP combination substantially improves utility over a bad or random choice. If Norm is implemented as a no-op, then at least one of the seven PP methods is not faithfully implemented, and the tables cannot be used to rank PP methods or to support the average improvement claim. The concern is concrete and testable because the repository is public and the required additive shift for OUE is large enough to make the observed equality impossible. I do not see a more serious issue: the protocol formulas are standard, the multi-threading design is reasonable, and the rest of the PP methods mostly show plausible differences from the raw estimates. The appropriate disposition remains CONDITIONAL, matching the reader's verdict; the condition is that the Norm discrepancy must be resolved and the tables or code corrected. Thus no change to the reader's verdict is needed.","tokens_in":11862,"tokens_out":4678,"duration_ms":56986,"concrete_test":"Clone the public LDPcube repository and run the OLH protocol on Kosarak with ε = 1, with and without Norm, using the paper's reported settings. Independently implement Eq. 17 by adding (1 − sum(hat_f))/|D| to the raw estimates and compute the ℓ1 distance to the true frequencies. If the toolkit's Norm output equals its raw output, the implementation is a no-op; if it differs from the independent computation, the implementation is incorrect. Either way, rerun the full Table I with the corrected Norm implementation: if any Norm entries change by more than 0.01×10⁻³, the published tables do not reflect the toolkit's actual behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that LDP3 is a reliable benchmarking toolkit and that selecting a good protocol/PP combination improves utility depends on each post-processing implementation matching its published algorithm. The Norm column violates this condition. Equation 17 says Norm adds a constant σ to every estimated frequency so that the sum becomes 1. For GRR, raw estimates already sum to 1, so Norm is a legitimate no-op. But for OLH, OUE, BLH, RAPPOR, and SS, the raw estimates do not sum to 1. For OUE with domain size d and ε = 1, the sum of the raw estimates is 2(e^ε + 1 − d)/(e^ε − 1); with d = 128 this is roughly −144.6, so Norm would add roughly 1.14 to every frequency and change the ℓ1 distance dramatically. Yet Tables I–III report Norm exactly equal to 'w/o PP' for every protocol and dataset. This is not a rounding artifact: such a shift cannot leave all reported ℓ1 distances unchanged at the displayed precision. Either the Norm implementation is a no-op in the code, or the table columns were copied incorrectly. Both possibilities undermine the experimental evidence that the PP methods are faithfully implemented and that the reported rankings of PP methods are reliable. Since Norm is one of the seven methods and the 'Avg. w/ PP' column depends on all methods, the headline benchmarking conclusion is not yet supported. The toolkit's open-source repository may still be correct, but the paper must clarify or correct this discrepancy before the utility claims can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LDP3, an open-source, modular, multi-threaded Python toolkit for local differential privacy (LDP) protocols and post-processing (PP) methods. It implements six LDP protocols (GRR, RAPPOR, OUE, BLH, OLH, SS), seven PP methods (Base-Pos, Norm, Norm-Cut, Norm-Sub, Norm-Mul, Power, PowerNS), utility metrics, and a command-line interface. The experimental evaluation reports ℓ1-distance-based utility for protocol-PP combinations on three real-world datasets (Kosarak, BMS-POS, Porto) at ε=1, and execution-time speedups from multithreading. The central claims are that post-processing consistently improves utility, that LDP3 enables users to identify good protocol-PP combinations, and that multithreading substantially reduces execution times.","tokens_in":12041,"tokens_out":6742,"duration_ms":69510,"significance":"If the implemented formulas match the published algorithms, LDP3 is a potentially valuable benchmarking resource for the LDP community: it is open-source, modular, extensible, and combines protocols, PP methods, and utility metrics under one interface. The protocol formulas in Section III-A are standard and appear correct, and the multi-threaded execution design is a practical engineering contribution. The authors also make the code publicly available, which aids reproducibility. However, the experimental evidence contains an internal inconsistency in the reported Norm column and an overstated claim that post-processing consistently helps; these issues currently limit the strength of the empirical conclusions.","major_comments":[{"comment":"The Norm column in Tables I-III is identical to the w/o PP column for every protocol and dataset (e.g., OLH/Kosarak both 1.56, OUE/BMS-POS both 2.17). This is inconsistent with Eq. (17), which defines Norm as adding a constant σ to every estimated frequency so that the frequencies sum to 1. For GRR the raw estimates already sum to 1, so Norm being a no-op is mathematically expected, but for OLH, OUE, BLH, RAPPOR, and SS the raw estimates do not sum to 1. For example, for OUE at ε=1 and |D|=128, the sum of the raw estimates from Eq. (8) is approximately 2(e^ε+1−d)/(e^ε−1) ≈ −144.6, so Norm would need to add roughly 1.14 to every frequency, which would substantially change the ℓ1 distance. The reported identical columns therefore indicate either that the Norm implementation does not implement Eq. (17) or that the table values were copied incorrectly. This undermines the claim in Section IV-B that the PP implementations are faithfully benchmarked, and because the Avg. w/ PP column includes Norm, the averages are also affected. The authors should correct the tables or the implementation, or explain why Norm is a no-op for all protocols.","section":"Section IV-B, Tables I-III"},{"comment":"The text states that 'Post-processing methods consistently help in reducing errors.' This is contradicted by the reported results for Power and PowerNS. For example, in Table I, Power yields higher ℓ1 distance than w/o PP for OLH (1.59 > 1.56), OUE (1.50 > 1.47), RAPPOR (1.71 > 1.65), and BLH (1.84 > 1.83). In Tables II and III, Power is worse than w/o PP for most protocols as well (e.g., Table II OLH 2.52 > 2.14, BLH 2.91 > 2.45, OUE 2.53 > 2.17; Table III OLH 1.27 > 1.17, BLH 1.45 > 1.34). The claim of consistency is therefore too strong; it should be restricted to the average over all methods or to specific methods that actually show improvement. This overstatement affects the interpretation of the utility benefits of post-processing as a general phenomenon.","section":"Section IV-B, Tables I-III"},{"comment":"The description of the Power method is not sufficiently detailed to be reproducible. Eq. (21) states a generic minimization over a distribution P, but the paper does not specify the family of distributions (despite the name suggesting a power-law family), the parameter estimation method, the initialization, or the numerical optimization procedure. The PowerNS method inherits this ambiguity. Since the experimental results for Power and PowerNS are part of the benchmark, the current description prevents readers from verifying the implementation or reproducing the results. The authors should either provide the full algorithm or cite a precise source and describe how the toolkit implements it.","section":"Section III-B, Eq. (21)"},{"comment":"All utility comparisons are averages over 10 repetitions with no standard deviations, confidence intervals, or significance tests. Many reported differences are small (e.g., Table I: Norm-Sub 1.27 vs. Norm-Mul 1.39 for OLH; Table III: Norm-Mul 0.94 vs. Base-Pos 0.93 for SS), and without error bars the claim that one PP method is 'best' for a given protocol and dataset is not statistically supported. Adding variance estimates or paired significance tests would materially strengthen the benchmarking conclusions.","section":"Section IV-A, IV-B"}],"minor_comments":[{"comment":"The Norm-Cut method sets a threshold θ with the condition that the post-processed frequencies sum to 1, but the choice of θ is ambiguous when multiple thresholds produce the same sum. The paper should specify the exact selection rule (e.g., the largest θ such that all remaining frequencies are positive and sum to 1).","section":"Section III-B, Eq. (18)"},{"comment":"The Power method is named 'Power' but the distribution in Eq. (21) is generic; consider clarifying the name or the intended distribution family.","section":"Section III-B"},{"comment":"The text says execution times decrease as threads increase from 1 to 8, but the figures show that for some protocols the decrease saturates or even reverses at 8 threads (e.g., OUE in Figure 4); this behavior should be discussed.","section":"Section IV-C, Figures 3-5"},{"comment":"The abstract and introduction refer to 'LDP$^3$' with superscript formatting while the body uses 'LDP3' throughout; unify the notation for clarity.","section":"Section I"},{"comment":"The table captions note that all values are ×10^{-3}, but the running text sometimes refers to values without recalling the scaling (e.g., '1.57' vs. '> 2.0'); consistent use of the scaling would improve readability.","section":"Section IV-B, Tables I-III"}],"recommendation":"major_revision","confidential_remarks":"The Norm column discrepancy is the most pressing issue and is directly checkable against the public GitHub repository. If the code implements Norm correctly and the tables are simply wrong, a corrected set of tables and a rerun of the average column may be sufficient. The paper is within the scope of a systems/benchmarking venue, but the experimental claims need to be made statistically defensible and the Power method should be specified precisely. No concerns about citation patterns or novelty disclosure beyond the need to fix the experimental presentation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate software contribution, not a theory paper. LDP3 knits together six standard LDP protocols and seven post-processing methods in a modular, multi-threaded, open-source package, and the authors are honest about how it sits relative to PURE-LDP, Multi-Freq-LDPy, and LDP-Plens. The protocol formulas in Section III-A are the standard ones and look correct. The execution-time results showing real speedups from multi-threading are plausible and useful. If I were benchmarking an LDP protocol, I'd use this toolkit as a starting point.\n\nThe soft spot is the one the stress-test note flags. In Tables I-III, the Norm column is identical to the w/o PP column for every protocol and every dataset. That cannot be right for OLH, OUE, BLH, RAPPOR, or SS, because their raw estimates do not sum to 1, and Norm (Eq. 17) adds a constant to force the sum to 1. For OUE with the parameters used here, the shift would be large enough to change every reported l1 distance at the displayed precision. Either the code's Norm is a no-op, or the table columns were copied from the w/o PP column. Both possibilities mean the paper has not shown that the PP implementations are faithful, and the relative rankings of PP methods are in question. This is the load-bearing part of the utility claim, so it needs a straight answer.\n\nTwo smaller issues: there are no error bars or variance measures despite 10 repetitions, which matters for ranking methods whose l1 values are close; and the Power method is described only as fitting a distribution and minimizing expected squared error, without saying which distributions were tried or how fitting was done. That makes the Power results hard to reproduce.\n\nMy read on the circularity angle: there is none. The paper implements published formulas and measures them on real data. The conclusion that post-processing helps is empirical and not derived from fitted toolkit parameters.\n\nSo: the toolkit itself is worth having, the comparison to prior toolkits is accurate, and the reviewer time would be well spent. But the paper as submitted needs a major revision to resolve the Norm discrepancy, add variance information, and specify the Power fitting procedure before the headline benchmarking conclusion is supportable. I hope the authors fix it, because a clean version of this would be a useful reference for the LDP community.","headline":"A genuinely useful LDP benchmarking toolkit, but the reported utility tables contain an unexplained Norm-column artifact that must be fixed before the experimental claims can be trusted.","tokens_in":12639,"tokens_out":1223,"would_cite":true,"duration_ms":15019,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LDP3 is an open-source toolkit claiming that selecting a protocol and post-processing method through its benchmarking pipeline substantially improves frequency-estimation utility over a bad or random choice.","keywords":["local differential privacy","LDP3","post-processing methods","frequency estimation","privacy toolkit","multi-threading","utility benchmarking","open-source software"],"falsifier":"Take a synthetic dataset whose raw LDP frequency estimates are known not to sum to 1, run LDP3 with the Norm method, and compare the post-processed output with a direct computation of Eq. 17; if the Norm output equals the raw estimates, the implementation is not adding the constant $\\sigma$ required to make the sum equal 1, and the post-processing comparisons in Tables I-III would need to be redone.","tokens_in":11578,"feed_emoji":"🔒","tokens_out":9492,"duration_ms":91765,"temperature":0.7,"pith_summary":"This paper presents LDP3, an open-source toolkit that bundles six local differential privacy (LDP) protocols, seven post-processing methods, and several utility metrics in a modular, multi-threaded design. In LDP, each user randomizes their own data on-device before sharing it, and the server estimates population frequencies; LDP3 is built to find which protocol and post-processing correction gives the most accurate estimates. The paper's central claim is that using LDP3 to choose a combination substantially improves utility compared with a bad or random choice, and that its multi-threaded execution cuts running times by splitting the dataset across threads and averaging the per-thread estimates. A sympathetic reader would take the contribution to be a practical benchmarking resource: a standard place to compare new protocols against existing ones and to pick a configuration for a real deployment.","feed_headline":"LDP3 finds the protocol and method that best cut estimation error","feed_subtitle":"Six LDP protocols, seven correction methods, and parallel execution in one open-source benchmarking package.","key_machinery":"The central object is the modular pipeline made of five modules: Protocol, Post-Processing, Utility Measurement, Multi-Threading, and Execution. The Protocol Module standardizes each LDP method as two functions, user-side perturbation and server-side estimation, covering GRR, RAPPOR, OUE, BLH, OLH, and SS. The Post-Processing Module applies correction formulas such as Base-Pos, Norm, Norm-Cut, Norm-Sub, Norm-Mul, Power, and PowerNS to the raw estimates $\\hat{f}(v)$ to produce $\\tilde{f}(v)$. The Multi-Threading Module splits the user population into $t$ equal chunks, runs a protocol on each chunk in parallel with its own random seed, and averages the per-chunk estimates before post-processing. This pipeline is what lets LDP3 benchmark many combinations and repeat experiments enough times to tame the randomness of LDP.","core_discovery":"The paper's central claim is that LDP3 lets a user experimentally identify a high-utility combination of LDP protocol and post-processing method, and that this matters because no protocol and no post-processing method dominates across datasets. Across the Kosarak, BMS-POS, and Porto datasets at $\\varepsilon = 1$, the reported $\\ell^1$ errors show post-processing consistently reducing error relative to no post-processing, while the best method varies by protocol and dataset; for instance, Norm-Mul is best under GRR while PowerNS is best under SS. The paper further reports that increasing from one to eight threads roughly halves or thirds execution times for many protocols, with non-linear scaling because dataset splitting and result combination remain serial. The intended consequence is that practitioners can benchmark their own data through LDP3's command-line interface and deploy the best configuration, and researchers can add new protocols and post-processing methods through its standardized interfaces.","pith_inferences":["A direct extension the paper leaves untested is how the best combination changes with the privacy budget; the reported utility tables fix $\\varepsilon = 1$, so the selection guidance is not shown at smaller or larger budgets.","The same modular split between protocol, post-processing, and utility metric could apply to downstream tasks such as heavy hitter identification or multidimensional frequency estimation, because the toolkit's interfaces are written independently of the metric.","A reader relying on the paper's utility conclusions should first verify that each post-processing implementation matches its published formula, since a silent mismatch in one method would change the rankings of all the others."],"forward_implications":["A practitioner with a surrogate dataset, a privacy budget, and a utility metric can use LDP3 to find the protocol and post-processing method with the lowest error for deployment.","A researcher adding a new protocol can benchmark it against GRR, RAPPOR, OUE, BLH, OLH, and SS, and can search which post-processing method works best with it.","Because the best post-processing method changes across protocols and datasets, a fixed or random choice can be substantially worse than the best combination, which is the paper's main argument for experimenting with LDP3.","Multi-threading reduces wall-clock time for large experiments, with the largest gains for protocols such as OLH and BLH whose per-user cost is high.","The reported experiments find no universally best post-processing method, so the paper's utility claim is about selection rather than a single winning method."],"supporting_citations":[{"why":"Supplies the frequency-estimation protocols GRR, BLH, OLH, RAPPOR, and OUE along with their default parameter choices, which the Protocol Module implements.","marker":"[6]"},{"why":"Supplies the consistency-based post-processing methods such as Norm, Norm-Cut, Norm-Sub, Norm-Mul, Power, and PowerNS that the Post-Processing Module implements and benchmarks.","marker":"[10]"},{"why":"Introduces the SS protocol and the PURE-LDP package, giving LDP3 a protocol to include and a closest existing toolkit to compare against.","marker":"[8]"},{"why":"Provides LDPLENS, an existing protocol-analysis framework without post-processing, used to position LDP3's contribution.","marker":"[7]"},{"why":"Provides Multi-Freq-LDPy, an existing LDP toolkit focused on multidimensional and longitudinal estimation, cited to show LDP3's different scope.","marker":"[22]"},{"why":"Provides earlier calibration and prior-knowledge methods for frequency estimation, cited as a source of post-processing approaches.","marker":"[9]"}],"fun_headline_variants":["LDP3 pinpoints the protocol-PP pair that minimizes error","LDP3 speeds up benchmarking with multi-threaded runs","LDP3 selects the best protocol+PP combo for your dataset","LDP3: parallelized LDP benchmarking for optimal protocol choice","LDP3 cuts runtime and finds the best protocol-PP pair"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusion that post-processing consistently improves utility rests on the assumption that every post-processing implementation in LDP3 matches the published algorithm, and the paper's own tables make this assumption doubtful: for every protocol and dataset, the Norm column is identical to the no-post-processing column, which is inconsistent with Eq. 17 unless the raw estimates already sum to 1.","fun_headline_variants_meta":{"raw":{"variants":["LDP3 pinpoints the protocol-PP pair that minimizes error","LDP3 speeds up benchmarking with multi-threaded runs","LDP3 selects the best protocol+PP combo for your dataset","LDP3: parallelized LDP benchmarking for optimal protocol choice","LDP3 cuts runtime and finds the best protocol-PP pair"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00128,"raw_usage":{"total_tokens":5232,"prompt_tokens":945,"completion_tokens":4287,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":561,"completion_tokens_details":{"reasoning_tokens":4197}},"tokens_in":561,"tokens_out":4287,"duration_ms":30658,"temperature":1.0,"reasoning_tokens":4197,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:16:20.976796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a synthetic dataset whose raw LDP frequency estimates are known not to sum to 1, run LDP3 with the Norm method, and compare the post-processed output with a direct computation of Eq. 17; if the Norm output equals the raw estimates, the implementation is not adding the constant $\\sigma$ required to make the sum equal 1, and the post-processing comparisons in Tables I-III would need to be redone.","supporting_citations":[{"cited_title":"Locally differentially private pro- tocols for frequency estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the frequency-estimation protocols GRR, BLH, OLH, RAPPOR, and OUE along with their default parameter choices, which the Protocol Module implements."},{"cited_title":"Locally differentially private frequency estimation with consistency,","cited_arxiv_id":null,"evidence_quote":"Supplies the consistency-based post-processing methods such as Norm, Norm-Cut, Norm-Sub, Norm-Mul, Power, and PowerNS that the Post-Processing Module implements and benchmarks."},{"cited_title":"Frequency estimation under local differential privacy,","cited_arxiv_id":null,"evidence_quote":"Introduces the SS protocol and the PURE-LDP package, giving LDP3 a protocol to include and a closest existing toolkit to compare against."},{"cited_title":"An adversarial approach to protocol analysis and selection in local differential privacy,","cited_arxiv_id":null,"evidence_quote":"Provides LDPLENS, an existing protocol-analysis framework without post-processing, used to position LDP3's contribution."},{"cited_title":"Multi-freq-LDPy: Multiple frequency estimation under local differential privacy in python,","cited_arxiv_id":null,"evidence_quote":"Provides Multi-Freq-LDPy, an existing LDP toolkit focused on multidimensional and longitudinal estimation, cited to show LDP3's different scope."},{"cited_title":"Calibrate: Frequency estimation and heavy hitter identification with local differential privacy via incorporating prior knowledge,","cited_arxiv_id":null,"evidence_quote":"Provides earlier calibration and prior-knowledge methods for frequency estimation, cited as a source of post-processing approaches."}],"review_version":1}