{"id":"b03a8722-420f-4db3-a02f-b046670f9ddc","arxiv_id":"2502.05219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OpenMined's privacy-enhancing infrastructure enabled two pilot AI audits, and the authors argue the remaining barriers to routine external scrutiny are legal rather than technical.","lead":"This paper describes privacy-preserving technology from OpenMined that lets outside researchers study AI systems without seeing private data, and reports two real-world pilot audits. It argues that this makes independent AI scrutiny feasible and that policymakers should focus on legal support for auditors.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim presumes secure-enclave attestation guarantees both integrity and confidentiality; the paper provides no evidence that the enclave enforces output policies preventing a malicious model from exfiltrating the secret evaluation dataset.","rationale":"The reader's weakest_assumption identified both the trustworthiness of enclave attestation and the owner-approval workflow as key concerns. I focus narrowly on the enclave attestation claim because it is the technical foundation for the central privacy guarantee in the paper's most advanced case study. The paper's language, 'certificates proving that it only ran the evaluation code that both Anthropic and UK AISI approved,' overstates what attestation certificates can demonstrate: they prove which code image was loaded and that it runs in a genuine enclave, but they do not prove that the code is free of output channels or side channels that can leak the secret dataset. The paper does not discuss output filtering, adversarial models, or any independent validation of the security properties. This is a concrete and falsifiable weakness. The owner-approval issue is real but the paper explicitly acknowledges it and proposes legal remedies; it does not hinge on an unverified technical claim. My proposed test would directly probe whether the confidentiality guarantee is trustworthy. I agree partially with the reader: the enclave concern is shared, but the specific mechanism I emphasize (output-channel exfiltration) is more granular than the reader's general 'hardware, side-channel, or governance failures.' The verdict should remain CONDITIONAL because the paper's descriptive value and policy synthesis are still useful, but the central claim requires independent validation before being accepted as a basis for policy conclusions.","tokens_in":5280,"tokens_out":4050,"duration_ms":39842,"concrete_test":"Obtain or reconstruct the exact enclave configuration and evaluation code used in the UK AISI–Anthropic trial, then run an adversarial test: replace the benign model with a 'leaky' model that steganographically encodes its input into output logits, and provide the secret dataset as inputs. Check whether any encoded data appears in the enclave's externally visible outputs or in the attestation evidence. If a leak is observable, the confidentiality claim for Case Study 2 fails in a realistic adversarial scenario, and the paper's 'succeeded' conclusion would need to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim is that OpenMined's infrastructure enables privacy-preserving audits and has succeeded in real-world governance scenarios. In Case Study 2 (§4), the key guarantee is mutual secrecy between the model owner (Anthropic) and the evaluator (UK AISI). The only technical support offered is that a secure enclave 'produced certificates proving that it only ran the evaluation code that both Anthropic and UK AISI approved.' This conflates load-time integrity with runtime confidentiality. Attestation certificates establish which code was loaded into a genuine enclave; they do not by themselves prove that the approved evaluation code prevents a hostile model from exfiltrating the secret dataset through its outputs. A malicious model could receive the secret Q&A as inputs and embed them into its generated outputs, or leak via timing and other side channels. Unless the enclave's evaluation code strictly filters all output channels, and that filter is itself verified or proven, the certificate provides no confidentiality guarantee. The paper cites no independent security evaluation, no formal verification, and no specification of the attestation quote's semantics. This is load-bearing because if a malicious model can leak the dataset to the model owner, the claimed mutual secrecy collapses and the conclusion that 'trustworthy privacy-preserving technical solutions ... have succeeded in real-world governance scenarios' is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that privacy-enhancing technologies (PETs), as operationalized by OpenMined in the open-source PySyft library, now make it feasible for external researchers to audit AI systems without learning sensitive user data, proprietary model weights, or secret evaluation datasets. It describes two deployments: a Christchurch Call–Dailymotion collaboration in which external researchers analyzed recommendation-algorithm behavior without seeing impression data, and a UK AI Safety Institute–Anthropic trial in which a secure enclave hosted both a model (GPT-2) and an evaluation dataset (CAMEL Biology) under claimed mutual secrecy. The paper concludes that these technical solutions have succeeded in real-world governance scenarios and that policymakers should now focus on legal empowerment of researchers, such as safe harbor and mandated access.","tokens_in":5541,"tokens_out":2924,"duration_ms":28446,"significance":"If the central claim is substantiated, the paper would describe a meaningful advance in AI governance: routine external scrutiny of consequential AI systems could become technically feasible without forcing companies to expose user data, weights, or audit datasets. The paper is valuable as a descriptive account of deployed PET infrastructure and as a policy-oriented synthesis, and it names concrete organizations and public assets. However, the evidence base is thin and largely self-reported by the developer community: the Dailymotion result is stated without effect sizes or methodology, and the secure-enclave trial used public assets as an exercise rather than sensitive ones. The contribution is therefore primarily programmatic and descriptive rather than a demonstrated or independently verified security result.","major_comments":[{"comment":"The claimed 'mutual secrecy' between Anthropic and UK AISI rests on secure-enclave certificates that 'proved that it only ran the evaluation code that both Anthropic and UK AISI approved.' This conflates load-time integrity with runtime confidentiality: attestation can certify which code was loaded into a genuine enclave, but it does not by itself show that the approved evaluation code prevents a malicious model from exfiltrating the secret dataset through its generated outputs, timing, or other side channels. The paper provides no evidence of output filtering, side-channel mitigation, or formal verification of the evaluation code, so the confidentiality guarantee for the UK AISI biology dataset is unsupported as stated.","section":"Sec. 4"},{"comment":"The Dailymotion case study reports that 'the researchers found that Dailymotion's recommendation algorithms do not disproportionately amplify highly suggestive content,' but the paper gives no effect sizes, statistical analysis, or methodological details behind that finding. Without any quantitative or qualitative analysis of the data, the reader cannot assess whether the finding is robust, which is load-bearing for the general claim that privacy-preserving external audits 'have succeeded in real-world governance scenarios.'","section":"Sec. 3"},{"comment":"The central case-study evidence consists of OpenMined's own blog posts and the Christchurch Call write-up authored by the same developer community. There is no independent security evaluation, no third-party verification of the privacy guarantees, and no adversarial testing of the claimed protections. Case Study 2 itself acknowledges that both the model and the dataset were public assets (GPT-2 and CAMEL Biology), so the exercise is a trial run rather than a demonstration that the setup can keep genuinely sensitive model weights or evaluation data private in a high-stakes setting; the conclusion that 'trustworthy privacy-preserving technical solutions ... have succeeded in real-world governance scenarios' therefore overstates what the paper actually shows.","section":"Secs. 3-4 and refs [1, 19, 20]"},{"comment":"The paper asserts that the described technologies 'enable users to verify their trust in a system's relevant privacy and security properties' and that PETs have reached 'a new level of maturity,' but it provides no explicit threat model, adversarial model, or formal specification of which security properties each setup guarantees. Without these, the paper cannot support the normative conclusion that AI model owners 'can no longer use privacy, security, and IP as conclusive excuses' for denying access, because the extent to which these setups actually protect against malicious models or malicious model owners remains unspecified.","section":"Sec. 2"}],"minor_comments":[{"comment":"The author list in reference [19] contains the name 'Kellye Trask' twice; please remove the duplicate.","section":"Ref. [19]"},{"comment":"The statement that 'running the evaluation took under two hours' is given without any hardware specification, enclave type, or dataset size, making the runtime claim difficult to interpret or reproduce.","section":"Sec. 4"},{"comment":"The future-work discussion mentions verification mechanisms to prevent model owners from deleting prediction logs or switching out models, but it does not cite any existing implementation, prototype, or research paper for such mechanisms.","section":"Sec. 5"},{"comment":"The paper would benefit from a dedicated limitations paragraph that explicitly addresses adversarial scenarios in which the model itself tries to leak data, or the model owner uses its approval rights to steer or block research; the current text touches on this in Section 6 but does not integrate it into the technical assessment.","section":"Sec. 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is more of a descriptive policy report than a technical security contribution. Its main evidence is self-reported by the developer community, and the strongest technical claim—secure-enclave confidentiality—is not backed by independent analysis. I would recommend the editor consider whether the journal's scope can accommodate a paper whose core evidence is a set of advocacy-oriented blog posts, or whether the authors should be asked to re-frame the claims as a report on ongoing experiments with clearly stated limitations. The revisions suggested in the major comments are feasible within the manuscript's scope and would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should read this as a policy synthesis, not a technical paper. It compiles two real deployment pilots—OpenMined with Dailymotion and with UK AISI/Anthropic—into a readable argument that PETs now make external audits technically feasible. The descriptive core is accurate as far as I can tell, and the authors are honest that the workflow still gives model owners approval rights. The legal recommendations (safe harbor, mandated compliance) are sensible and appropriately grounded.\n\nThe soft spots are real but not fatal to the policy argument. First, all the evidence comes from OpenMined's own blog posts and the Christchurch Call write-up by the same developer community. That is a circularity burden the paper never addresses. Second, the strong conclusion that 'trustworthy privacy-preserving technical solutions have succeeded' is undercut by the details: the UK AISI case used public GPT-2 and public CAMEL data, so it was a trial run with no actual secrets at stake. The stress-test note about enclave attestation is on point—certificates prove what code was loaded, not that a hostile model cannot exfiltrate secret data through its outputs. The paper conflates load-time integrity with runtime confidentiality. That matters if you generalize from these pilots to high-stakes evaluations.\n\nI also think the Dailymotion finding, that the algorithm does not amplify suggestive content, is stated without any effect sizes or statistical analysis. For a governance audience that may be okay, but it is thinner evidence than the prose implies.\n\nWho benefits? Policymakers and governance researchers who want a compact overview of how PETs could fit into audit regimes. It will not teach a security researcher anything new, and the future-directions section is admittedly speculative. The authors are clear thinkers—they identify the owner-approval problem themselves and propose legal backstops. I would not call this a groundbreaking contribution, but it deserves a serious referee because it addresses an important question and could be strengthened with relatively modest revisions: independent security assessment, more methodological detail, and a softened conclusion.","headline":"A clear policy synthesis of two PET-based audit pilots, but the stronger privacy and independence claims outrun the self-reported evidence.","tokens_in":6008,"tokens_out":1198,"would_cite":true,"duration_ms":13222,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Privacy-enhancing technologies can make external scrutiny of AI systems feasible without exposing user data, model weights, or secret evaluation sets.","keywords":["AI governance","external scrutiny","privacy-enhancing technologies","secure enclaves","mutual secrecy","AI auditing","model evaluation","differential privacy"],"falsifier":"A direct test would be to run an adversarial red-team exercise against the enclave setup using deliberately malicious evaluation code or an attempt to forge attestation certificates; a second test would compare the findings of an owner-approved audit with those of an unrestricted audit of the same public model and look for systematic blind spots. Either a successfully forged certificate or an owner-approved audit that misses problems the unrestricted audit finds would refute the paper's central claim.","tokens_in":5099,"feed_emoji":"🔐","tokens_out":7734,"duration_ms":63797,"temperature":0.7,"pith_summary":"External scrutiny of AI systems has been blocked by companies' legitimate worries about user privacy, system security, and intellectual property. This paper argues those worries no longer need to be decisive, because privacy-enhancing technologies have matured into end-to-end infrastructure that lets outside researchers run approved analyses without seeing raw user data, model weights, or secret evaluation sets. It supports the argument with two real-world governance pilots: one in which external researchers studied a video platform's recommendation algorithm through a privacy-preserving access tool, and one in which a government safety agency evaluated a frontier model inside a secure enclave while both the model weights and the agency's test dataset remained mutually hidden. If the paper is right, routine external audits of consequential AI systems become technically feasible, and the main remaining task for policymakers is legal: safe harbor for auditors and legal backing for access to systems whose owners decline to cooperate.","feed_headline":"Privacy tools put external AI audits within reach","feed_subtitle":"Two real-world pilots: outside researchers audit AI while data, weights, and tests stay hidden.","key_machinery":"The load-bearing mechanism is a secure enclave, a protected execution environment that can hold an AI model and a sensitive dataset at the same time and emit signed certificates attesting that only computation code approved by both the model owner and the evaluator ran. An open-source Python library orchestrates a workflow in which the researcher proposes questions (and, in the advanced setup, brings code and a private dataset), the model owner approves the computation, and the researcher receives only the outputs. This arrangement realizes 'mutual secrecy': the model owner's weights stay private and the evaluator's dataset, such as a national-security-relevant question set, stays private, while each party retains the power to approve or deny every computation.","core_discovery":"The paper's central claim is that a nonprofit's open-source infrastructure, built around a Python library, combines several privacy-enhancing technologies—secure enclaves, secure multi-party computation, zero-knowledge proofs, federated learning, and differential privacy—into setups that implement a single core workflow: a researcher proposes questions, the model owner approves them, and the researcher receives answers without learning anything else. The simpler setup lets researchers analyze a proprietary dataset held on a secure server through owner-approved code, as demonstrated when external researchers tested whether a video platform's recommender amplified suggestive content and found that it did not disproportionately do so. The more advanced setup uses a secure enclave to host both the model and a secret evaluation dataset simultaneously, producing certificates that only the evaluation code approved by both parties ran; in the second pilot a government evaluator used exactly this arrangement to run a frontier-model safety evaluation while the model owner never saw the test data and the evaluator never saw the weights. The paper's conclusion is that these privacy-preserving technical solutions have succeeded in real-world governance scenarios, so privacy, security, and intellectual-property concerns are no longer conclusive excuses for refusing external access.","pith_inferences":["A likely next test is scaling the enclave setup from public demonstration assets to truly proprietary frontier models; the paper acknowledges significant engineering work remains, and the trust model of enclave vendors has not yet been stress-tested at that scale.","The owner-approval workflow makes these audits 'supervised scrutiny' rather than fully independent oversight; one could test whether approved research questions systematically avoid adversarial or reputationally sensitive topics by comparing approved audits with unrestricted audits on a public model.","If privacy-preserving access becomes standard, data-sharing regulation may shift from compelling disclosure of raw data to compelling computation-with-approval protocols, which would change what protections regulators need to specify.","The same infrastructure could generalize beyond AI to digital-platform research and cross-border statistics, areas the paper notes are already being piloted, suggesting a broader governance template for private data analysis."],"forward_implications":["External researchers could routinely study deployed systems' real behavior, such as whether a recommender system has a partisan lean or a chatbot produces toxic replies, without seeing raw user logs.","Government agencies could evaluate frontier models against secret test sets that they never reveal, while the model owner's weights remain confidential.","Auditors could operate as private businesses because their audit prompts remain proprietary, and model owners would be unable to 'teach to the test' by training on those prompts.","Future setups that keep researcher code private, add verification, and prevent deletion or swapping of models would further strengthen the audit regime.","The decisive policy moves become legal safe harbor for auditors and legislation that backs researcher access, such as the proposed platform transparency law the paper cites."],"supporting_citations":[{"why":"Documents the first real-world use of the integrated privacy-preserving access tool in the video-platform pilot.","marker":"[1]"},{"why":"Establishes the multi-stakeholder initiative that framed and funded the recommendation-algorithm case study.","marker":"[2]"},{"why":"Provides the open-source Python library at the core of the infrastructure.","marker":"[14]"},{"why":"Defines the basic audit workflow and the range of technical setups the paper summarizes.","marker":"[19]"},{"why":"Reports the secure-enclave trial demonstrating mutual secrecy in frontier-model evaluation.","marker":"[20]"},{"why":"Supplies the safe-harbor policy recommendation the conclusion endorses.","marker":"[7]"},{"why":"Supplies the example of proposed legislation requiring platforms to share data with vetted researchers.","marker":"[12]"}],"fun_headline_variants":["Privacy tech unlocks external AI audits without data exposure","OpenMined's PETs enable secure AI scrutiny for real-world governance","External audits of AI now feasible with privacy-enhancing tools","PETs bridge gap: outside researchers can audit AI safely"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the secure enclave's attestation certificates genuinely guarantee confidentiality and integrity in a real high-stakes audit, and that an audit in which the model owner approves every research question still provides meaningful external scrutiny; if either of these gives way, the paper's privacy and independence claims are overstated.","fun_headline_variants_meta":{"raw":{"variants":["Privacy tech unlocks external AI audits without data exposure","OpenMined's PETs enable secure AI scrutiny for real-world governance","External audits of AI now feasible with privacy-enhancing tools","PETs bridge gap: outside researchers can audit AI safely"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000653,"raw_usage":{"total_tokens":3002,"prompt_tokens":965,"completion_tokens":2037,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1969}},"tokens_in":581,"tokens_out":2037,"duration_ms":14777,"temperature":1.0,"reasoning_tokens":1969,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T05:17:14.087069+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to run an adversarial red-team exercise against the enclave setup using deliberately malicious evaluation code or an attempt to forge attestation certificates; a second test would compare the findings of an owner-approved audit with those of an unrestricted audit of the same public model and look for systematic blind spots. Either a successfully forged certificate or an owner-approved audit that misses problems the unrestricted audit finds would refute the paper's central claim.","supporting_citations":[{"cited_title":"AI trans- parency in practice, October 2024","cited_arxiv_id":null,"evidence_quote":"Documents the first real-world use of the integrated privacy-preserving access tool in the video-platform pilot."},{"cited_title":"Christchurch Call initiative on algorithmic outcomes, September 2022","cited_arxiv_id":null,"evidence_quote":"Establishes the multi-stakeholder initiative that framed and funded the recommendation-algorithm case study."},{"cited_title":"PySyft’s documentation, 2021","cited_arxiv_id":null,"evidence_quote":"Provides the open-source Python library at the core of the infrastructure."},{"cited_title":"How to audit an AI model owned by someone else (Part 1), June 2023","cited_arxiv_id":null,"evidence_quote":"Defines the basic audit workflow and the range of technical setups the paper summarizes."},{"cited_title":"Secure enclaves for AI evaluation, November 2024","cited_arxiv_id":null,"evidence_quote":"Reports the secure-enclave trial demonstrating mutual secrecy in frontier-model evaluation."},{"cited_title":"Senator Christopher Coons of Delaware","cited_arxiv_id":null,"evidence_quote":"Supplies the example of proposed legislation requiring platforms to share data with vetted researchers."}],"review_version":1}