{"id":"7e7984a7-5712-43dc-8498-df839c3a25dd","arxiv_id":"2608.10216","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Embedding-cosine thresholds used as meaning gates instead measure lexical overlap, so in the target cases of reversal-versus-paraphrase the gates fire backwards; a matched-pair audit reveals the regime.","lead":"An audit of embedding-cosine similarity gates in agent systems shows the score tracks how much wording changed, not whether meaning was preserved, so a drift guard approved changing 'withhold the study drug' to 'administer the study drug' at cosine 0.96. The paper ships a matched-pair audit instrument and shows that a natural-looking evaluation corpus can produce an inverted verdict.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline deployed-system result rests on one guard's mutation corpus; the paper's weakest link is the unshown denominator of the 0/56 claim.","rationale":"The reader's weakest assumption correctly identified the narrow empirical base (10 pairs per cell, one register, one model/operator, one deployed system). I agree that is a real external-validity limit and that the authors scope it honestly in §10. But the most load-bearing concern for the central claim is not the size of the corpus; it is the missing denominator and selection documentation behind the 0/56 deployed-system result, which is the paper's flagship empirical finding and the one that makes the headline concrete. The paper is careful about corpus construction (§4) and pre-registration of repairs (§7), yet §6 does not describe how the 56 mutations were generated or selected, and §10's list of what the adversarial review did not cover omits the deployed-system result. Without the selection procedure, the reader cannot distinguish \"the guard fires backwards on meaning-breaking mutations\" from \"the guard fires backwards on minimal-edit mutations, which are the only ones we tried.\" This is a correctness risk that is distinct from the external-validity caveat, because it bears on whether the stated result is an accurate estimate even for the audited system, not only on whether it generalizes. A test that settles it is concrete and cheap: inspect the artifact's mutation-generation log. The verdict should be CONDITIONAL rather than REJECT because the concern is about documentation of a denominator and selection rule, not about an identified contradiction; if the artifact already contains the full log, the verdict returns to ACCEPT with the external-validity caveat intact.","tokens_in":9382,"tokens_out":2219,"duration_ms":20436,"concrete_test":"Request the full mutation-generation log from the released artifact (or, if the artifact lacks it, treat that absence as the finding). For each of the 56 mutations, record: (a) the source anchor, (b) the mutation class, (c) the exact edit, (d) token-Jaccard with the anchor, (e) the cosine under the production configuration, and (f) whether the mutation was included in the final set before or after the cosine was computed. Then recompute the headline 0/56 with the inclusion criterion \"all attempted mutations that an independent annotator agrees are meaning-breaking\" and report the distribution of token-Jaccard for included versus excluded items.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central deployed-system claim (\"caught 0 of 56 meaning-breaking mutations\") is the load-bearing result: it converts the small-corpus laboratory finding into a statement about a real production drift guard, and it is also the result the abstract leads with. The paper reports the 56 mutations and 15 faithful controls, and reports per-class AUROC values, but it never reports the full set of mutations that were generated or attempted, nor the selection/authoring procedure that determined which mutations entered the 56. Section 6 says the mutations span an in-sample corpus (26 mutations, 5 faithful controls) and a held-out corpus (30, 10) with the held-out corpus authored under documented isolation, but it gives no inventory of mutation classes attempted versus included, no criteria for inclusion/exclusion, and no count of attempted mutations that were discarded (e.g., because the author judged them not meaning-breaking, or because they changed too much surface form). Without that denominator, the 0/56 finding could reflect a mutation set that was hard by construction (e.g., all mutations were minimal single-token edits that never moved cosine below 0.83), or it could be an overstatement if the 56 are a subset selected after the fact. The paper's own §8 shows how easily the confounder captured the investigators' own claims, and §10 explicitly lists what was not attacked by the adversarial review: the no-threshold results, test-suite results, mechanism-variance results, and projection results—but the deployed-system result itself is not listed as an attack target, so the single most consequential empirical claim carries no adversarial clean bill. The concern is not that the authors are dishonest; it is that the most load-bearing empirical result is the least documented selection procedure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper audits embedding-cosine similarity gates used in agent systems, arguing that these gates measure lexical overlap rather than semantic agreement, and that in the exact cases they are deployed to catch (e.g., instruction reversals), the two quantities are anti-correlated. It presents a 2×2 factorial corpus balancing decision (same/opposite) and lexical overlap (close/distant) per anchor, shows a lexical-axis positive control (AUROC 0.975–1.000 across nine encoder configurations), and measures decision-axis AUROC 0.440–0.815 with the deployed configuration at chance. The paper reports a production drift guard approving 0 of 56 meaning-breaking mutations, including a 'withhold'→'administer' reversal at cosine 0.9608, and shows that obvious repairs (encoder swap, overlap-conditioned gate, NLI drop-in) fail on separately authored held-out corpora. It also demonstrates that a naively constructed evaluation corpus can invert the verdict (decision AUROC 0.000 in 13 of 18 configuration-task cells), reports a no-op hyperbolic projection finding, and releases the corpus, harness, and frozen results.","tokens_in":9580,"tokens_out":9135,"duration_ms":82208,"significance":"If the results hold, the paper makes a strong, actionable case that cosine-threshold gates should not be treated as semantic agreement measures without a per-configuration matched-pair audit. The design is a genuine advance: the balanced factorial corpus and the positive control give the measurements internal validity, and the authors are transparent about the confounder capturing their own claims. The machine-checked hyperbolic no-op proof and the released pinned harness are reproducible contributions. The main claims are appropriately limited in §10, and the paper is careful to state that generalization beyond one project's register and one deployed system is indicated, not established.","major_comments":[{"comment":"The headline deployed-system claim that the audited drift guard 'caught 0 of 56 meaning-breaking mutations' is reported without the denominator of the mutation-authoring process. The manuscript states that the 56 mutations span an in-sample corpus (26) and a held-out corpus (30), with 15 faithful controls, but it does not report how many candidate mutations were generated or attempted, the criteria for classifying a mutation as meaning-breaking versus a discarded edit, or the selection procedure that determined which mutations entered the final sets. Without this inventory, the 0/56 figure could reflect a corpus that was hard by construction (e.g., all mutations being minimal single-token edits) or a post-hoc selection of unfavorable items. Because the abstract and conclusion lead with this number, it is load-bearing; please add a full mutation log (including attempted-but-discarded items), the authoring instructions, and the classification rules to the artifact and summarize the inclusion/exclusion criteria in the paper.","section":"§6, first paragraph"},{"comment":"The 'held-out' corpora used to evaluate the repairs are the same two drift-guard corpora described in §6, and §10(iv) states that all corpora were generated by one model under one operator. The 'separately authored' condition is therefore an isolated prompting context within the same authoring setup, not an author change by an independent human. The conclusion that 'evaluation across an authoring change is what catches this' is stronger than the evidence supports. Please either add an evaluation on corpora produced by an independent operator, or revise the claim to 'a change in authoring protocol' and temper the generalization statements in the abstract and §12 accordingly.","section":"§7, table and final paragraph"}],"minor_comments":[{"comment":"The prevalence observations are counts of opened-and-read sites and are correctly not presented as rates; adding the list of the 11 opened sites to the artifact or an appendix would make the survey auditable.","section":"§2"},{"comment":"The NLI drop-in row reports a 'frozen prior result' but does not state whether its predictions were committed at the same time as the other two repairs; please clarify its provenance and its relation to the pre-registration.","section":"§7, table"},{"comment":"The statement that scoring rules for the adversarial review were 'committed before any result was seen' needs material support; please point to a timestamped file in the artifact.","section":"§8"},{"comment":"The permutation p-values for the simulated naive design are conditional on the balanced corpus construction; a one-sentence reminder that this is a simulation, not a prevalence estimate, would help readers avoid over-generalizing.","section":"§3"},{"comment":"The provenance of the five shipped operating points is given in a parenthetical; since the prevalence survey ships, a pointer to the exact code locations would make these thresholds auditable.","section":"§5"},{"comment":"There is a typo in the abstract ('adifferentquestion' should be 'a different question'); please also proofread for similar spacing artifacts.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good fit for the journal's focus on measurement validity for deployed LLM systems. The central thesis is well supported by the positive control and balanced design; the main gaps are the missing denominator for the 0/56 claim and the same-operator nature of the 'author change'. Both are fixable with additional disclosure or a genuine independent authoring session, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this is the closest thing I've seen to a proper measurement-validity audit of embedding-cosine gates in agent systems. The central claim—that these gates track lexical overlap rather than meaning, and that in the exact pairs they're designed to catch the two are anti-correlated—is supported by a careful factorial design, a positive control, and an unusually honest limitations section. It's not a takedown or a toy: the released harness and frozen results make it reproducible, and the authors' account of how the confounder captured their own headline claims is credible evidence of the phenomenon.\n\nWhat's actually new: the 2x2 factorial matched-pair instrument (decision x lexical overlap, per-anchor balanced), the demonstration that a naively authored corpus can invert the verdict (AUROC 0.000 in 13 of 18 cells), the pre-registered failure of an encoder swap and an overlap-conditioned gate on separately authored held-out data, and the audited deployed drift guard. Prior work (Blagec, Kassner, Hossain, Anschütz) documented embedding insensitivity to negation, but none audited production gates or quantified the structural confounder. The positive control (lexical-axis AUROC 0.975–1.000 across nine configurations) shows the instrument works, so the decision-axis chance results are meaningful. The §9 hyperbolic-projection no-op is a nice bit of algebra, machine-checked in the artifact.\n\nSoft spots: the stress-test note is right that the 0/56 deployed-system claim is the least documented empirical result. The paper reports 56 mutations and 15 faithful controls but not the denominator of attempted mutations or the selection/authoring criteria. That means the headline number could under- or over-state the real miss rate. This is a genuine gap, and it's the one place I'd push for revision. The other limits—one register, one model, one operator, single-annotator labels—are acknowledged in §10 and don't undermine the structural argument. The factorial experiments carry the thesis; the deployed-system story is illustrative, not load-bearing on its own.\n\nWho this is for: anyone building or evaluating semantic-similarity gates in LLM agents, and anyone working on evaluation validity. It deserves a serious referee. I'd recommend acceptance with a request to add the mutation-generation inventory and ideally a second annotator, but even as-is it's a solid, usable contribution.","headline":"A genuinely useful validity audit of cosine-threshold gates, with a reproducible instrument and an honest limitations section; the headline 0/56 deployed-system claim has an underdocumented denominator but the structural finding stands.","tokens_in":10225,"tokens_out":2250,"would_cite":true,"duration_ms":22605,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Similarity gates measure wording, not meaning, so a production drift guard approved an instruction and its exact reversal.","keywords":["embedding cosine similarity","lexical overlap confound","negation","paraphrase","semantic gate audit","matched-pair evaluation","AUROC","agent systems"],"falsifier":"Take a production drift guard of this class and run it on a new 2x2 matched-pair corpus built from long-form narrative text or multilingual dialogue; if a gate configuration catches most minimal-edit reversals (e.g., decision AUROC above 0.9 at matched overlap, or nonzero detection of 56 engineered mutations), the claim that these gates fire backwards would be overturned for that regime.","tokens_in":9118,"feed_emoji":"⚠️","tokens_out":6406,"duration_ms":54867,"temperature":0.7,"pith_summary":"This paper tries to establish that a widely deployed class of agent-system quality gates—cosine thresholds over text embeddings—answers the wrong question. The score tracks how much the wording changed, while the gate exists to ask whether the meaning changed. Because reversing an instruction is often a one-word edit while faithful rewording replaces most of the words, the score and the target run opposite in the very pairs the gates exist to catch. The paper backs this with a production drift guard that fired on none of 56 meaning-breaking mutations, a factorial audit in which balanced accuracy at five shipped operating points never exceeded 0.700 (median 0.525), and a demonstration that a naturally authored evaluation corpus can invert the verdict to decision AUROC 0.000.","feed_headline":"Cosine gate approves a reversal: 56 of 56 mutations missed","feed_subtitle":"A drift guard approved 'withhold the study drug' as 'administer the study drug' at cosine 0.96.","key_machinery":"The instrument is a 2×2 factorial corpus per task, crossing decision (same/opposite) with lexical overlap (close/distant), with all four cells paired to the same anchor and overlap balanced encoder-blind (10 anchors, 10 pairs per cell, 80 pairs per task). Stratified AUROC within overlap strata reads the decision axis while the lexical axis serves as a positive control; this matched-pair design is what separates what cosine measures (wording) from what the gates deploy it to measure (meaning).","core_discovery":"The central claim is that embedding-cosine similarity, as deployed in agent gates, measures lexical overlap rather than decision agreement. In the pairs these gates exist to catch—negation, modal weakening, quantity change—wording and meaning are anti-correlated: a reversal is a minimal additive edit, while an agreement is a full substitutive rewrite. A threshold on this score therefore pairs a sentence with its own negation and approves it; the audited production guard caught 0 of 56 meaning-breaking mutations and approved \"withhold the study drug\" as \"administer the study drug\" at cosine 0.9608. The same confounder corrupts evaluation: a naively built corpus produced decision AUROC exactly 0.000 in 13 of 18 configuration-task cells. The paper further claims that with lexical overlap matched, the decision axis is heterogeneous across encoders but separable for the strongest configurations (AUROC 0.79–0.90), so a valid instrument is buildable if validity is measured per configuration.","pith_inferences":["If cosine gates are lexical-overlap meters, then other minimal-edit semantic changes—scope inversions, modality shifts, quantity and unit changes—are equally likely to pass; the audit's mutation classes suggest these may be even more dangerous than negation.","The additive-vs-substitutive asymmetry is likely register-dependent; running the same 2x2 audit on dialogue, long-form prose, or multilingual text would test whether the anti-correlation generalizes or is a property of short imperative instructions.","The demonstrated inversion of naive evaluation implies that published \"embeddings cannot represent negation\" results may partly be artifacts of corpus design rather than representation failures; re-measuring those benchmarks with matched overlap is a concrete next test.","A practical extension: gate systems could expose the token-Jaccard/overlap stratum to the operator and require a per-stratum threshold audit before a cosine cutoff is trusted as a safety decision."],"forward_implications":["At any of the five shipped operating points, a cosine-threshold gate cannot reliably tell an instruction from its reversal: balanced accuracy across 90 configuration-threshold-task cells topped out at 0.700.","Polarity cases—pairs that share wording and flip the decision, and pairs that share the decision and share no wording—must be added to component test suites; all four suites audited had none.","Any evaluation of such gates that uses naturally authored corpora can return an inverted verdict, so the gate class requires a de-confounded matched-pair audit before deployment.","Calibration-shaped repairs (encoder swaps, overlap-conditioned gating, NLI drop-ins) are not sufficient; the tested attempts landed at chance on separately authored held-out data.","Some encoder configurations can separate reversal from paraphrase at matched overlap (AUROC 0.79–0.90), so the finding is about validity measurement, not a blanket condemnation of embeddings."],"supporting_citations":[{"why":"Documents negation and antonym subsets scoring above genuinely similar pairs across models, supplying the natural-subset evidence for the confounder.","marker":"Blagec et al. (2019)"},{"why":"Shows NLI benchmarks underrepresent negation so thoroughly that models can ignore it, explaining why evaluation layers are blind to this failure.","marker":"Hossain et al. (2020)"},{"why":"Names the Proxy Presumption and the discriminant-validity test that this audit operationalizes for embedding-based gates.","marker":"Li et al. (2026)"},{"why":"Supplies the linguistic basis: negation is a marked additive operation on an affirmative base, making the wording-meaning anti-correlation structural rather than an authoring habit.","marker":"Horn (1989)"},{"why":"NegEx exists because surface similarity does not carry polarity, used as standing evidence that negation needs dedicated machinery rather than similarity thresholds.","marker":"Chapman et al. (2001)"},{"why":"PAWS establishes the resource-side half of the problem: models fail on high-overlap non-paraphrases until trained on them, motivating the need for all four factorial cells.","marker":"Zhang et al. (2019)"},{"why":"Supplies the discriminant-validity notion the 2x2 instrument is built on: the measure must track the construct and fail to track the confounder.","marker":"Campbell and Fiske (1959)"},{"why":"Provides the measurement-validity framing: system harms arise from the mismatch between a theoretical construct and its operationalization.","marker":"Jacobs and Wallach (2021)"}],"fun_headline_variants":["Cosine gate approves drug reversal, catches 0 of 56","Reversal scores 0.96: gate approves administer for withhold","Similarity gate measures wording, not meaning: 0/56 caught","Naive evaluation flips AUROC to 0.000 in 13/18 cells"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline numbers rest on small corpora (10 pairs per cell) in a single register of short imperative software and clinical instructions, generated by one model under one operator, and on one deployed system; if the wording-meaning anti-correlation is weaker in other registers, the inverted verdict may not generalize to the full gate class.","fun_headline_variants_meta":{"raw":{"variants":["Cosine gate approves drug reversal, catches 0 of 56","Reversal scores 0.96: gate approves administer for withhold","Similarity gate measures wording, not meaning: 0/56 caught","Naive evaluation flips AUROC to 0.000 in 13/18 cells"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000878,"raw_usage":{"total_tokens":3903,"prompt_tokens":1157,"completion_tokens":2746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":773,"completion_tokens_details":{"reasoning_tokens":2666}},"tokens_in":773,"tokens_out":2746,"duration_ms":20424,"temperature":1.0,"reasoning_tokens":2666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:10:34.824570+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a production drift guard of this class and run it on a new 2x2 matched-pair corpus built from long-form narrative text or multilingual dialogue; if a gate configuration catches most minimal-edit reversals (e.g., decision AUROC above 0.9 at matched overlap, or nonzero detection of 56 engineered mutations), the claim that these gates fire backwards would be overturned for that regime.","supporting_citations":[],"review_version":1}