{"id":"94c5f645-dfa9-4a0f-ae99-a10a5da5fae5","arxiv_id":"2608.00739","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"EduPluginBench provides a staged assurance pipeline that catches self-constructed plugin violations conventional checks miss, while fresh model generations and real Moodle fixes reveal sharp transfer limits.","lead":"A new benchmark and staged admission pipeline for AI-generated educational plugins checks whether code respects permissions, privacy, provenance, lifecycle, and failure bounds, not just whether it runs. It finds that the full staged pipeline catches far more benchmark-encoded violations than standard checks, yet zero of 600 fresh model outputs passed even the first gate, and a generic scanner missed all 16 real Moodle vulnerabilities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The primary recall claim is a contract-consistency result whose external validity rests on the unvalidated representativeness of the 1,440 author-derived mutants; the paper's own negative transfer studies reinforce that this is the weakest link.","rationale":"The reader identified the weakest assumption as the validity and representativeness of the 1,440 controlled mutants, specifically the shared specification vocabulary between construction oracles and detectors. I agree that this is the most load-bearing concern. The paper is internally careful: it acknowledges the circularity, separates evidence tiers, reports negative transfer results, and avoids overclaiming. The primary quantitative claim, however, is a controlled contract-consistency result. Its value as a benchmark depends on whether the mutants stand in for real release-blocking defects, and this is asserted rather than independently demonstrated. The external Moodle study (S7) tests only a generic PHP detector and returns zero detections, which reinforces the transfer limitation but does not validate the Python plugin mutants. The confirmatory generation study (S6) produced no P0 passes, so it cannot provide downstream effectiveness evidence. Thus the only positive effectiveness signal is the controlled mutants. An independent reimplementation of the mutant-generation process, or an external expert review of the existing mutants, would directly test whether the 100% recall is an artifact of construct sharing. Since the reader's verdict of CONDITIONAL already reflects this concern, no verdict change is needed; the concrete test would either confirm the concern or retire it.","tokens_in":21886,"tokens_out":4053,"duration_ms":36701,"concrete_test":"Commission an independent team, blinded to the P0-P4 detector implementations, to create a fresh set of first-order mutants for a random subset of the 30 specifications using only the natural-language task statements, public API schemas, and visible examples (without access to the construction oracles or mutation registry). Run the frozen P0-P4 pipeline on these independently authored mutants. If recall drops materially below 100%, or if a substantial fraction of the original mutants are judged implausible by independent security reviewers, the primary claim is an artifact of shared vocabulary rather than a property of real defects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (74.7-point recall increase) is only as strong as the assumption that the 1,440 activation-checked mutants represent release-blocking defects. The construction oracles (H_f) and the P3/P4 detectors operationalize the same specification vocabulary, and both are authored by the same team. A mutant is valid if it flips its family oracle, so a correct detector for that family will necessarily reject it. The label-separation and decision-invariance tests (Section 9.1) show the detectors do not read mutation identifiers or labels, but they do not break the conceptual circularity: the oracle and detector are two implementations of the same contract. The paper explicitly acknowledges this and restricts the claim to 'consistency with the encoded contract', and it provides valuable negative transfer evidence (S6: 0/600 unmodified packages pass P0; S7: generic PHP detector finds 0/16 Moodle vulnerabilities). However, the benchmark's positive effectiveness evidence rests entirely on these controlled mutants. If the mutants are not representative of naturally occurring plugin defects, the headline quantitative contribution reduces to an internal tautology. The held-out operator and archetype analyses strengthen internal validity but do not address external representativeness. Independent real-defect validation is the single missing piece.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EduPluginBench, an executable benchmark and a staged P0–P4 admission pipeline for AI-generated educational plugins. The benchmark defines eight defect families, 30 specifications, 120 clean reference packages, and 1,440 activation-checked first-order mutants. The controlled evaluation reports that P0–P4 increases mutant recall from 25.3% under P0–P2 to 100%, a 74.7 percentage-point difference, with 0/120 observed clean-package rejections. The paper also reports three negative transfer results: 0/600 unmodified model-generated candidates pass P0; a frozen generic PHP detector finds 0/16 Moodle vulnerable revisions; and an earlier bounded-repair diagnostic is presented as a separate, clearly post hoc stratum. The authors explicitly frame the controlled result as consistency with the encoded contract, not independent real-defect effectiveness, and they provide a checksummed artifact with row-level data, frozen protocols, and reproduction instructions.","tokens_in":22112,"tokens_out":12523,"duration_ms":116559,"significance":"The paper's main strengths are its unusually explicit evidence hierarchy, protocol locking, specification-clustered inference, label-separation checks, and the clear separation of confirmatory, post hoc, and scenario analyses. The negative transfer studies are a real contribution: they sharply delimit what the benchmark can and cannot claim. The central caveat, stated in Section 9.1 and in the conclusion, is that the construction oracles and the P3/P4 detectors operationalize the same specification vocabulary, so the 74.7-point recall gain is a contract-consistency result rather than evidence of independent real-defect discovery. Because the paper is transparent about this and scopes its claims accordingly, the benchmark remains useful as a controlled, reproducible instrument for measuring contract consistency and clean-package harm, even though it does not by itself establish real-defect detection effectiveness.","major_comments":[],"minor_comments":[{"comment":"The sentence \"Under the frozen parser, 50.0% (300/600; 95% CI 46.01–53.99%)\" is missing the noun after the percentage; it should say \"50.0% parsed\" or \"yielded a parseable package.\"","section":"§7.5"},{"comment":"Because the paper itself states that oracles and detectors are not construct-independent, the abstract should consider renaming \"release-blocking-defect recall\" to \"controlled contract-consistency recall\" at first mention, or adding a one-sentence qualifier immediately after the 74.7 percentage-point number, so that a casual reader does not take the figure as real-defect recall.","section":"§9.1"},{"comment":"Given that all 4,800 operator–parent pairs were applicable, activated, and isolated, the paper should state explicitly whether the activation oracle was designed to be total, so that the 100% activation rate is not read as empirical evidence of mutant representativeness.","section":"§7.1"},{"comment":"The claim that P6 satisfies the preregistered 10% clean-package rejection bound on held-out specifications rests on a 0/20 point estimate whose Wilson upper bound is 16.1%; the paper should phrase this as observed satisfaction and avoid implying the bound is statistically confirmed.","section":"§5.6"},{"comment":"The model-specific S6 result would be clearer if the abstract or results section explicitly stated that all 300 parseable outputs came from a single model and all 300 interface failures came from the other model, since this split is central to interpreting the 50.0% parse yield.","section":"§7.5"},{"comment":"The sentence \"The extent to which each recommendation is supported is summarized by the controlled and bounded-repair evidence supports executable contracts...\" is grammatically tangled and should be rewritten.","section":"§8.4"}],"recommendation":"minor_revision","confidential_remarks":"The paper is unusually honest and reproducible, which is a major point in its favor. The main risk is that the headline 74.7-percentage-point recall figure will be cited by future work without the surrounding caveats; the editor may wish to encourage the authors to add a prominently displayed 'claims and non-claims' box, perhaps near the abstract or in the introduction, to make the contract-consistency interpretation unmissable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear —,\n\nThe headline 74.7-point recall gain is a contract-consistency result, not evidence that the pipeline finds real defects—and the paper says so explicitly. The reason to read it is the methodological discipline and the rare honesty of the two negative transfer studies.\n\nThe benchmark itself is genuinely new. EduPluginBench combines an eight-family threat model, executable domain contracts, staged P0–P4 admission, an activation-checked mutation registry, and an explicit evidence hierarchy. The construction is unusually careful: 4,800 operator–parent pairs audited, 1,440 balanced mutants, specification-clustered bootstrap, predeclared splits, and lineage digests. The paper cleanly separates confirmatory, post hoc, and scenario analyses, and reports 0/600 unmodified-generation P0 passes and 0/16 Moodle vulnerability detections as clear negative results rather than burying them.\n\nThe soft spot is the one the paper itself flags. A mutant is valid only if its family oracle fires, and the P3/P4 detectors implement executable versions of those same family definitions. The 100% recall therefore is a consistency property of the encoding. External validity depends on the representativeness of the 1,440 author-derived mutants, which is asserted through the operator design rather than demonstrated against real defects. The negative transfer results reinforce that boundary. Also, the checksummed compendium is not linked in the preprint, so independent verification is currently impractical. The P6 calibration analysis shows no advantage over simple fail-closed rejection and never exercises quarantine; that part is more modest than the rest of the paper claims.\n\nMinor issue: in the primary held-out analysis, held-out specifications and held-out operators are confounded (the reserved operators appear only on those five specs). The later factorial challenge separates them, but the headline \"held-out\" number should be read as a joint holdout.\n\nWho this is for: people building governed plugin release pipelines, and researchers studying benchmark validity in AI-code assurance. The evidence hierarchy and negative transfer design are worth emulating. It deserves a serious referee. My recommendation: send it to peer review. Require the artifact link, and ideally an independent real-defect validation, before treating the headline recall as effectiveness evidence.","headline":"A carefully constructed benchmark whose headline recall is a contract-consistency result—valuable as a reproducibility model and for its honest negative-transfer evidence, but not yet a demonstration of real-defect detection.","tokens_in":22655,"tokens_out":3483,"would_cite":true,"duration_ms":29864,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Executable contracts lift plugin-defect recall from 25% to 100%.","keywords":["AI-generated code","software assurance","mutation testing","policy as code","software supply chain","educational plugins","executable contracts"],"falsifier":"Run the full P0-P4 pipeline on an independently authored corpus of real educational-plugin vulnerabilities with black-box labels and a fresh specification schema; if recall is materially below 100% or clean-package rejections exceed the 3.1% Wilson bound, the contract-consistency claim is falsified. A cheaper check: strip mutation identifiers and lineage labels from package objects and verify that every P3/P4 decision is unchanged; any decision flip would break the label-separation claim.","tokens_in":21659,"feed_emoji":"🧩","tokens_out":6730,"duration_ms":55862,"temperature":0.7,"pith_summary":"EduPluginBench asks whether an AI-generated plugin should be admitted to a governed ecosystem, not just whether it runs. The paper builds a staged P0-P4 assurance pipeline that layers static security analysis, supply-chain policy, executable domain contracts, and sandboxed behavioral checks over ordinary build and functional tests, then measures it on 1,440 single-defect mutant packages, each constructed and activation-checked from one of 30 educational-plugin specifications. On that controlled benchmark, the full pipeline raises release-blocking-defect recall from 25.3% to 100% (specification-clustered 95% CI 73.4-75.8 percentage points) while rejecting none of 120 clean references (95% Wilson upper bound 3.1%). The paper is careful to claim only consistency with the encoded contract, not independent real-defect discovery: a separately frozen transfer study of 600 unmodified model-generated packages found none that passed even the first stage, and a generic PHP ruleset detected none of 16 independently labelled vulnerable/fixed pairs from a public learning-platform security-advisory corpus. A sympathetic reader would care because plugin platforms need release decisions that span code, dependencies, runtime behavior, and domain policy, and this paper supplies a concrete executable method with its limits stated.","feed_headline":"Executable contracts lift plugin-defect recall from 25% to 100%","feed_subtitle":"Adding domain-contract stages to build, security, and dependency checks finds every seeded defect and rejects no clean package.","key_machinery":"The carrying mechanism is a staged fail-closed pipeline: P0 (package schema, build, visible functional tests, deterministic replay), P1 (pinned Bandit and Semgrep static rules), P2 (lockfile, CycloneDX SBOM, and a frozen advisory snapshot), P3 (executable domain contracts over capability, telemetry, provenance, lifecycle, resource, and accessibility records), and P4 (network-isolated execution with metamorphic replay and behavioral-record checks). Its validity instrument is the mutation registry: 40 first-order operators, five per family, each applied only after a label-separated construction oracle confirms activation and isolation, giving 1,440 valid mutants whose decisions are the primary recall denominator. The two rejection-bearing P4 relations are that an irrelevant event field must not change the result and that the returned evidence version and retrieval time must track the stale-evidence input; remaining probes stay diagnostic.","core_discovery":"The paper's central claim is that release-blocking defects in plugin packages split into eight families—security, dependency, privacy, authority, provenance, lifecycle, reliability, and accessibility—and that after build, static-security, and dependency stages catch the first two, executable domain contracts (P3) and behavioral-record and metamorphic runtime checks (P4) catch the remaining six. On the constructed benchmark, P0-P2 alone recalled 25.3% of 1,440 activation-validated first-order mutants, while P0-P4 recalled 100%, an absolute 74.7-percentage-point difference; P3 uniquely detected 680 mutants and P4 uniquely detected 395. No clean reference was rejected. The paper states that this establishes consistency with the encoded contract, not independent production-defect effectiveness, and the transfer strata confirm that boundary: 300 of 600 unmodified generations parsed but 0 of 600 passed P0, and the frozen generic PHP detector found 0 of 16 vulnerable revisions in the external security pairs.","pith_inferences":["If the zero-P0 transfer results generalize, the practical bottleneck for admitting generated plugins is interface and functional compliance rather than detector sensitivity; even perfect P3-P4 detectors cannot be exercised on packages that never form valid releases.","The shared vocabulary between construction oracles and detectors means the 100% recall is a consistency result within the benchmark's own encoding; an independently authored corpus with black-box labels is the natural next experiment and would likely show lower recall.","The porting recipe generalizes beyond education to other governed plugin ecosystems, such as enterprise workflow or scientific analysis, but the specific effect sizes should not be carried over without rebuilding the specification schema, clean corpus, and domain oracles.","The external zero-detection result suggests that governance needs application-aware contracts rather than generic language rulesets; training and evaluating a domain-specific detector on the same public vulnerable/fixed pairs would be a testable follow-up."],"forward_implications":["Plugin platforms can express admission obligations as versioned executable contracts instead of prose review criteria, covering least privilege, telemetry consent, provenance, lifecycle, and bounded failure.","Conventional build, static-security, and dependency checks alone leave six of eight defect families undetected in the controlled stratum; the contract and behavioral stages are what recover them.","If the benchmark is right, current model-generated plugins fail at interface and visible-functionality gates (0 of 600 unmodified generations passed P0), so downstream assurance estimates remain undefined without repair or better generation.","Generic security rulesets do not transfer to application-specific vulnerabilities: the frozen PHP ruleset found none of 16 independently labelled vulnerable/fixed pairs.","Clean-package harm was not observed in the controlled stratum, but the 95% Wilson upper bound of 3.1% means a nonzero rejection rate is not excluded."],"supporting_citations":[{"why":"Established execution-based functional evaluation of code models, the baseline EduPluginBench contrasts against.","marker":"[8]"},{"why":"Repository-level code-generation benchmark that motivates evaluating whole components rather than isolated functions.","marker":"[13]"},{"why":"Secure-code benchmark showing generated code can contain vulnerabilities, motivating added assurance stages.","marker":"[6]"},{"why":"Argues for outcome-driven oracles that assess functionality and security together, supporting contract-based evaluation.","marker":"[27]"},{"why":"Classic mutation-operator study that grounds the controlled-fault methodology.","marker":"[21]"},{"why":"Mutation-practice assessment motivating activation-checked mutant validity in the registry.","marker":"[16]"},{"why":"The frozen generic PHP ruleset used in the static stage and in the external transfer test.","marker":"[30]"},{"why":"Public vulnerable/fixed announcement pairs used as independently labelled external evidence.","marker":"[19]"}],"fun_headline_variants":["Executable contracts catch all 8 defect families, but real models fail","0 clean rejections, 74.7-point recall jump, but real-world gaps remain","EduPluginBench: synthetic recall 100%, real-world transfer 0%","Contract stages lift recall to 100% on synthetic bugs; real models pass none"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 1,440 activation-checked first-order mutants are valid release-blocking defects, where activation is decided by author-written construction oracles that share the specification vocabulary and public manifest or behavioral fields with the P3/P4 detectors; if those mutants are not representative of real defects, the 100% recall measures consistency with the benchmark's own encoding rather than real-world detection.","fun_headline_variants_meta":{"raw":{"variants":["Executable contracts catch all 8 defect families, but real models fail","0 clean rejections, 74.7-point recall jump, but real-world gaps remain","EduPluginBench: synthetic recall 100%, real-world transfer 0%","Contract stages lift recall to 100% on synthetic bugs; real models pass none"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001056,"raw_usage":{"total_tokens":4471,"prompt_tokens":1025,"completion_tokens":3446,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":641,"completion_tokens_details":{"reasoning_tokens":3359}},"tokens_in":641,"tokens_out":3446,"duration_ms":21813,"temperature":1.0,"reasoning_tokens":3359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:16:59.819629+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full P0-P4 pipeline on an independently authored corpus of real educational-plugin vulnerabilities with black-box labels and a fresh specification schema; if recall is materially below 100% or clean-package rejections exceed the 3.1% Wilson bound, the contract-consistency claim is falsified. A cheaper check: strip mutation identifiers and lineage labels from package objects and verify that every P3/P4 decision is unchanged; any decision flip would break the label-separation claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The frozen generic PHP ruleset used in the static stage and in the external transfer test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Public vulnerable/fixed announcement pairs used as independently labelled external evidence."}],"review_version":2}