{"id":"1c43db7b-dc7d-40d9-8119-31b105522298","arxiv_id":"2509.07022","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A modular safety middleware combining deterministic filters and a single-call LLM self-verdict reports perfect blocking of dieting-related unsafe content on a small synthetic evaluation.","lead":"After the NEDA chatbot Tessa gave harmful diet advice, this paper proposes a lightweight safety layer that blocks dangerous dieting and calorie guidance with a keyword filter plus a single LLM call that judges its own answer. The authors report perfect blocking on a 100-prompt synthetic test set at near-baseline cost, and map the failure to OWASP and NIST controls.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Method F's 100% recall may reflect self-verdict correlation, not independent safety: the paper's own limitations concede the risk, and the evaluation does not show the mandated final deterministic scan was applied.","rationale":"The reader's verdict is CONDITIONAL, which is appropriate. My analysis agrees with the reader's weakest assumption: the core risk is that Method F's judge is not independent of the generator. I sharpen this by noting a specific evidential gap: the design mandates a final deterministic scan, but the Method F evaluation description omits any mention of that scan, making it ambiguous whether the 100% recall is a property of the self-verdict or of the deterministic scanner. If the scanner was not run, the result is expected under correlated failure; if it was run, the result is not attributable to the single-call pattern alone. Either way, the headline is overstrong relative to the evidence. I also note the table/text inconsistency for Method D, which the reader flagged and which reinforces the need for a corrected evaluation. None of this invalidates the architecture's promise: the threat model and governance mapping are useful, and the code release is a genuine asset. The paper's own limitations already concede most of these risks, so I do not recommend moving from CONDITIONAL to REJECT; instead, the authors should run a quick ablation to clarify what actually blocks the malicious prompts, then adjust the claim to match the measured component contribution.","tokens_in":10054,"tokens_out":3275,"duration_ms":39776,"concrete_test":"Re-run Method F on the provided 50 malicious prompts under three configurations: (i) self-verdict only, exactly as described in the evaluation (no final deterministic scan); (ii) self-verdict followed by the paper's own lexical/numeric scanner from §Output Moderation; and (iii) a cross-family judge model (different model family) instead of the self-verdict. Record per-prompt which layer blocked each query. If (i) drops below 100%, or (iii) differs materially from (i), then the perfect recall cannot be attributed to the single-call JSON pattern and the paper should re-scope its claim. Also correct the Table 3/text discrepancy for Method D before relying on any aggregate numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that Single-Call JSON Verdict (Method F) achieves perfect interception (50/50, 100%) at near-baseline cost—depends on the assumption that an LLM can reliably judge the safety of its own generated answer via the trailing JSON verdict. This is not an independent check: the judge is the same model, in the same call, with the same weights and context that produced the unsafe content. Any failure mode causing the model to output dieting advice (e.g., policy drift, euphemism, obfuscation) can equally cause the model to set is_safe=true, because both the answer and the verdict come from the same autoregressive distribution. The paper's Limitations explicitly state that reusing the same model family 'risks correlated failures and limits external validity'; for Method F the correlation is exact, not merely familial.\n\nThe design section mandates a 'final numeric or lexical re-scan of the buffered output, even in cases where is_safe=true' (§ Single-Call JSON Mode). Yet the Method F evaluation description says only that 'the answer is discarded if is_safe=false or if parsing fails (fail-closed).' It does not state that the deterministic scanner was actually applied in the F pipeline. If the scanner was not applied, then the 100% recall is entirely attributable to the self-verdict, which is precisely the failure regime the concern targets. If the scanner was applied, then the effect of the self-verdict is confounded with the deterministic scan, and the paper does not report which component caught each of the 50 malicious prompts. In either case, the current evidence does not establish that the single-call JSON pattern itself 'closes the gap.'\n\nAdditionally, Table 3 lists Method D as 50/50 (100%), while the text says D blocks 47/50 (94%) and that E 'reaches perfect recall'; this internal inconsistency further weakens confidence in the reported numbers. The architecture is plausible and the open-source code is a positive, but the headline empirical result is not","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper analyzes the 2023 NEDA 'Tessa' chatbot incident as a case of absent safety engineering in a health-adjacent AI assistant. It proposes a modular safety middleware that combines deterministic lexical gates, an LLM policy filter, and a fail-closed gate, including a 'Single-Call JSON Mode' in which the generator emits a trailing is_safe verdict. The authors map Tessa's failure patterns to OWASP LLM Top 10 and NIST SP 800-53 controls, and they evaluate six deployment patterns (A–F) on 100 self-labeled synthetic prompts using Gemma 2 2B. The central empirical claim is that Method F achieves perfect recall (50/50, 100%) on malicious prompts at 1.2× time and 1.5× token cost relative to an insecure baseline, and thus that lightweight, auditable checks can prevent 'another Tessa.' The paper states clear limitations: small synthetic dataset, single backbone, same model family for adjudication, and no wild precision estimate.","tokens_in":10316,"tokens_out":5337,"duration_ms":59241,"significance":"If the empirical claim held, the contribution would be practically useful: a simple, auditable middleware that achieves high recall on the targeted Tessa-like failure modes at near-baseline cost, with a governance mapping that makes the safeguards actionable. The paper is commendable for releasing prompts, code, and audit logs, and for explicitly designing fail-closed behavior and a deterministic post-hoc scan. However, the central result rests on a single run over 100 synthetic prompts whose labels are derived from the same policy the system implements, and on a self-verdict from the same model that generates the answer. These concerns are acknowledged in the Limitations section but are not resolved by the reported experiments. With additional ablations and independent evaluation, the approach could be a useful engineering case study; in its current form, the strength of the conclusion exceeds the evidence.","major_comments":[{"comment":"The narrative and the table disagree on Method D's recall. The text states 'The input-side judge (D) blocks 47/50 malicious queries (94%)' and that the unblocked prompts were caught only by output-side adjudication, while Table 3 reports D as 50/50 (100%). Because the paper's comparative claim is that post-generation adjudication is necessary for full recall, this contradiction is load-bearing. Please correct the inconsistency and verify the counts for every row.","section":"Results, Table 3"},{"comment":"The design mandates a final deterministic numeric/lexical re-scan of the buffered output 'even in cases where is_safe=true', but the Method F evaluation description says only that 'the answer is discarded if is_safe=false or if parsing fails.' It is not reported whether the deterministic scanner was applied in the F pipeline. If it was not applied, the 100% recall is attributable entirely to the self-verdict, which is exactly the failure regime of concern; if it was applied, the contribution of the self-verdict is confounded with the scanner. Please report Method F with and without the deterministic scan, including per-prompt verdicts and scan outcomes.","section":"Single-Call JSON Mode; Evaluation Protocol"},{"comment":"Method F's judge is the same model, in the same call, as the generator. The paper itself states that reusing the same model family 'risks correlated failures and limits external validity'; for Method F the correlation is exact, not merely familial. Any failure mode that makes the model produce dieting advice (e.g., euphemism, obfuscation, policy drift) can also make it set is_safe=true, because both the answer and the verdict come from the same autoregressive distribution. The current experiment cannot rule out that Method F's perfect recall is an artifact of this correlation. Please evaluate Method F with a judge model of a different family (or at least a different prompt/profile) and report disagreement rates.","section":"Environment; Limitations"},{"comment":"The protocol records the number of non-malicious prompts flagged, but the results do not report this quantity for any method, and no precision or false-positive analysis is given. For a safety middleware in a health-adjacent setting, false positives are as practically important as recall: blocking a large fraction of safe queries would be unacceptable even with perfect recall. Please report the full confusion matrix (or at least safe-prompt flag rates) and, ideally, threshold/calibration analysis for the classifier components.","section":"Evaluation Protocol; Table 3"},{"comment":"The 100-prompt test set is labeled 'known by construction' from the same policy criteria that the middleware enforces. This means the evaluation measures consistency with the policy specification, not real-world safety. The paper should be explicit that the 100% recall is an upper-bound/self-consistency result, and should obtain independent human labels or a held-out naturally occurring corpus (even a small one) to ground the central claim.","section":"Dataset; Limitations"}],"minor_comments":[{"comment":"Typographical errors: 'OW ASP' should be 'OWASP'; 'readon' in Background should be 'reason'; 'na¨ıve' has an encoding issue.","section":"Throughout"},{"comment":"The deployment patterns are labeled A–F, but the text refers to 'Section 1–6' without numbered sections; please add consistent numbering or cross-references.","section":"Methodology"},{"comment":"The table would benefit from a precision column and explicit error bars or repeated-trial ranges for time/token multipliers. Currently these are single measurements with no variance information.","section":"Table 3"},{"comment":"The sentence saying the same model was used for 'Input-Judge and Output-Judge' is slightly ambiguous for Method F, which uses a single call; please clarify how the 'different prompting configurations' were realized within that call.","section":"Environment"},{"comment":"The reference list would benefit from page/venue details for conference papers (e.g., OWASP Top 10, ToxiGen) so readers can locate the sources; several entries are only arXiv identifiers.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful engineering case study, but the 'perfect interception' claim is stronger than the current evidence supports. The table inconsistency and the unaddressed self-verdict confound are the main blockers. If the authors can fix these and temper the conclusion, it could be acceptable as a systems/position paper. I would also encourage the editor to weigh whether the contribution is substantive enough for a full-length archival venue given the small synthetic evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper proposes a sensible modular safety layer for health chatbots and makes a good-faith effort to connect Tessa to OWASP/NIST controls, but its headline claim — that the single-call JSON verdict achieves perfect interception at near-baseline cost — is not supported by the evidence as reported.\n\nFirst, what's worth keeping. The architecture is clearly described: a lexical fast path, an in-line LLM policy filter, and a fail-closed gate with a final deterministic scan. The decision to scope the threat model to non-adversarial Tessa-style failures is reasonable, and the mapping to OWASP LLM Top 10 and NIST SP 800-53 controls is genuinely useful for governance audiences. The authors cite prior work on LLM-as-judge and self-refinement, and they ship code and prompts in the repo. The Limitations section is unusually candid; it flags the correlated-failure risk, the small synthetic dataset, and the lack of precision measurement.\n\nBut there are real problems with the empirical centerpiece. Most immediately, the paper contradicts itself: the text says the input-side judge (D) blocks 47/50 (94%), while Table 3 lists D at 50/50 (100%). That's not a nitpick; it affects which method is actually on the Pareto frontier. Second, Method F's verdict comes from the same model, in the same call, with the same weights and context that produced the response. The paper's own limitation that reusing a model family 'risks correlated failures' gets worse when the judge and generator are literally the same autoregressive distribution. Third, the design section mandates a final numeric/lexical re-scan even when is_safe=true, but the Method F evaluation description never says that scan was applied. If it wasn't, the 100% recall is entirely from the self-verdict. If it was, the scan's contribution is confounded and unreported. Either way, the claim that 'the single-call JSON verdict closes the gap' is not established.\n\nThese are fixable. A proper ablation that reports which component caught each of the 50 malicious prompts, plus a cross-family judge or a held-out set, would strengthen the paper considerably. The architecture itself is plausible, and the honesty of the limitations is a point in the authors' favor.\n\nThis paper is for practitioners building health-adjacent chatbots and for people who need a concrete mapping from a real incident to framework controls. It is not a research breakthrough, but it's a serious engineering writeup. I would send it to peer review — it deserves a referee — but I'd require the internal consistency fix and the ablation before publication. For my own work, I wouldn't cite it as evidence of the single-call pattern's safety, but I might mention it as an example of the architecture.","headline":"Plausible middleware with an honest limitations section, but the headline empirical claim is undermined by an internal inconsistency and by relying on a self-verdict that isn't an independent check.","tokens_in":10979,"tokens_out":3712,"would_cite":false,"duration_ms":38610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight middleware that asks a chatbot to append its own JSON safety verdict to each answer—then discards anything unsafe—blocks every malicious prompt in a 100-prompt synthetic test at near-baseline cost.","keywords":["AI safety","eating disorder chatbot","safety middleware","LLM-as-judge","single-call JSON verdict","fail-closed moderation","Tessa chatbot","health-adjacent AI"],"falsifier":"Run Method F on a larger and independently labeled set of eating-disorder prompts, or swap the verdict model to a different family than the generator, and measure recall; if a different judge family or independent labels drop the 100% interception rate, the perfect recall was an artifact of the same model judging itself on a dataset whose labels came from the same policy. A second check: measure the false-positive rate on safe prompts, which the paper does not report.","tokens_in":9880,"feed_emoji":"🛡️","tokens_out":6985,"duration_ms":69172,"temperature":0.7,"pith_summary":"This paper argues that the 2023 NEDA chatbot Tessa incident—where the assistant gave eating-disorder users calorie-deficit and weigh-in advice—was a failure of absent safety engineering, and that a small, modular middleware layer could have prevented it. The proposed design combines a fast keyword/regex gate with an in-line language-model policy filter that emits a strict, fail-closed JSON verdict in the same call that generates the answer. On a synthetic set of 100 prompts (50 malicious, 50 safe), the single-call method blocked all 50 unsafe prompts while running at about 1.2x the time and 1.5x the tokens of an insecure baseline. The paper's point is that health-adjacent safety does not require heavyweight infrastructure: explicit, testable checks at the last mile can close the gap that let 'another Tessa' happen, and governance mappings make those checks auditable.","feed_headline":"One-call safety filter blocks all 50 unsafe chatbot prompts","feed_subtitle":"A single JSON self-verdict plus a keyword scan matches multi-stage safeguards at 1.2x time cost.","key_machinery":"The load-bearing object is the Single-Call JSON Verdict (Method F): the model returns the full answer text followed by a trailing JSON verdict such as {\"response\": ..., \"is_safe\": false, \"violations\": [...]}. The verdict follows the answer so the model has full autoregressive context when judging; at the API boundary only the trailing JSON is parsed, and the buffered answer is dropped if the verdict is unsafe or unparsable (fail-closed). A final deterministic numeric/lexical scan re-checks the rendered text before delivery. The same middleware also includes a pre-generation keyword/regex fast path. These pieces work together: the lexical gate cheaply blocks obvious redlines, the self-verdict","core_discovery":"The paper's central claim is that a modular safety middleware—a lexical fast path plus an in-line LLM policy filter that returns a strict, fail-closed JSON verdict in a single model call—can intercept every unsafe eating-disorder prompt in its synthetic evaluation without the latency and token cost of multi-stage judge pipelines. In the comparison, methods without post-generation adjudication blocked only 4%–52% of malicious prompts, while both the output-judge method and the single-call JSON method reached 100% recall (50/50). The single-call method did so at 1.2x time and 1.5x token overhead, placing it on the Pareto frontier against ensembles that reached the same recall at 1.6–1.7x time","pith_inferences":["Because the paper's perfect recall was measured with the same 2B model as both generator and judge, a natural next test is to replace the judge with a different model family; if recall drops, the 100% figure is partly an artifact of self-consistency rather than an independent safety check.","The synthetic dataset's labels are known by construction from the same policy being enforced, so the true false-positive rate on safe, real-world queries remains unmeasured; a deployment trial with independent clinical review would be the informative extension.","The single-call self-verdict pattern is not limited to eating-disorder redlines: it could be re-specified for medication dosing, self-harm, or other health-adjacent domains, with the deterministic scan re-targeted to domain-specific numerics and lexicon.","The architecture's safety guarantee is only as strong as the final deterministic scan's coverage, so writing the scan as a formal allowlist/denylist specification with test coverage would let the claim be checked automatically rather than by spot evaluation."],"forward_implications":["Post-generation adjudication is necessary for full recall: methods without an output gate (A, B, C) blocked only 4%, 22%, and 52% of malicious prompts, while both output-judge methods (E, F) reached 100%.","Stacking multiple safeguards adds cost without adding recall: the B+C+D+E and D+E ensembles also reached 100% but at 1.6–1.7x time and 4.8x tokens, versus Method F's 1.2x time and 1.5x tokens.","A fail-closed single-call verdict places the safety guarantee at the API boundary: unparsable or unsafe verdicts are treated as refusals, so the system does not depend on the model's goodwill.","Mapping Tessa's failure patterns to OWASP LLM Top 10 and NIST SP 800-53 controls makes the safeguards testable and auditable in governance terms, not just prompt-level intent.","The same modular primitives—input gating, buffered output, deterministic scanners, escalation—can be extended to adversarial classes like prompt injection, but the current prototype does not claim to cover them."],"supporting_citations":[{"why":"Documents the Tessa suspension and frames the incident as the motivating case the middleware must prevent.","marker":"Hoover 2023"},{"why":"Supplies the reported harmful outputs (calorie deficits, weigh-ins) that define the redlines being tested.","marker":"Aratani 2023"},{"why":"Establishes LLM-as-a-judge pipelines as a reliable way to audit and filter model drafts, the basis for the output judge.","marker":"Zheng et al. 2023"},{"why":"Shows self-refinement loops let a model critique its own drafts, supporting the single-call self-verdict design.","marker":"Madaan et al. 2023"},{"why":"Demonstrates prompt injection can bypass policy-as-prompt setups, motivating the deterministic post-generation scanner and fail-closed gate.","marker":"Greshake et al. 2023"},{"why":"Provides the LLM-applications Top 10 failure taxonomy used to map Tessa's gaps to mitigations.","marker":"OW ASP 2024"},{"why":"Supplies NIST SP 800-53 control families that make the technical safeguards auditable and governance-ready.","marker":"Joint Task Force 2020"},{"why":"Adds the AI RMF risk-management framing that situates the middleware within broader governance and escalation requirements.","marker":"NIST 2023"}],"fun_headline_variants":["Single-call JSON verdict intercepts 100% of unsafe prompts","Single-call middleware: perfect recall at 1.2x cost","Single-call safety filter: 100% interception, no multi-stage overhead","One-shot JSON filter: perfect recall, no multi-stage pipeline","Single-call safety check beats multi-stage at 100% recall"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"That the language model's self-reported safety verdict on its own generated answer is trustworthy enough that, combined with the final keyword/numeric scan, it catches everything unsafe—if the model is wrong in the same way about its answer and its verdict, unsafe content can still pass.","fun_headline_variants_meta":{"raw":{"variants":["Single-call JSON verdict intercepts 100% of unsafe prompts","Single-call middleware: perfect recall at 1.2x cost","Single-call safety filter: 100% interception, no multi-stage overhead","One-shot JSON filter: perfect recall, no multi-stage pipeline","Single-call safety check beats multi-stage at 100% recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001208,"raw_usage":{"total_tokens":4813,"prompt_tokens":749,"completion_tokens":4064,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":493,"completion_tokens_details":{"reasoning_tokens":3971}},"tokens_in":493,"tokens_out":4064,"duration_ms":34299,"temperature":1.0,"reasoning_tokens":3971,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:41:42.439453+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Method F on a larger and independently labeled set of eating-disorder prompts, or swap the verdict model to a different family than the generator, and measure recall; if a different judge family or independent labels drop the 100% interception rate, the perfect recall was an artifact of the same model judging itself on a dataset whose labels came from the same policy. A second check: measure the false-positive rate on safe prompts, which the paper does not report.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the Tessa suspension and frames the incident as the motivating case the middleware must prevent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the reported harmful outputs (calorie deficits, weigh-ins) that define the redlines being tested."},{"cited_title":"P.; Zhang, H.; Gonzalez, J","cited_arxiv_id":null,"evidence_quote":"Establishes LLM-as-a-judge pipelines as a reliable way to audit and filter model drafts, the basis for the output judge."},{"cited_title":"P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P","cited_arxiv_id":null,"evidence_quote":"Shows self-refinement loops let a model critique its own drafts, supporting the single-call self-verdict design."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the LLM-applications Top 10 failure taxonomy used to map Tessa's gaps to mitigations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NIST SP 800-53 control families that make the technical safeguards auditable and governance-ready."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Adds the AI RMF risk-management framing that situates the middleware within broader governance and escalation requirements."}],"review_version":1}