Pith. sign in

REVIEW 5 major objections 5 minor 22 references

Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight middleware that asks a chatbot to append its own JSON safety verdict to each answer—then discards anything unsafe—blocks every malicious prompt in a 100-prompt synthetic test at near-baseline cost.

desk verdict Plausible middleware with an honest limitations section, but the headline empirical claim is undermined by an internal inconsistency and by relying on a self-verdict that isn't an independent check. read the letter →

arxiv 2509.07022 v1 pith:C2ET7TBN submitted 2025-09-07 cs.CY cs.AI

Pavan Reddy , Nithin Reddy This is my paper
classification cs.CYcs.AI
keywords AIsafetyeatingdisorderchatbotmiddlewareLLM-as-judgesingle-callJSONverdictfail-closedmoderationTessahealth-adjacent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the 2023 NEDA chatbot Tessa incident—where the assistant gave eating-disorder users calorie-deficit and weigh-in advice—was a failure of absent safety engineering, and that a small, modular middleware layer could have prevented it. The proposed design combines a fast keyword/regex gate with an in-line language-model policy filter that emits a strict, fail-closed JSON verdict in the same call that generates the answer. On a synthetic set of 100 prompts (50 malicious, 50 safe), the single-call method blocked all 50 unsafe prompts while running at about 1.2x the time and 1.5x the tokens of an insecure baseline. The paper's point is that health-adjacent safety does not require heavyweight infrastructure: explicit, testable checks at the last mile can close the gap that let 'another Tessa' happen, and governance mappings make those checks auditable.

What carries the argument

The load-bearing object is the Single-Call JSON Verdict (Method F): the model returns the full answer text followed by a trailing JSON verdict such as {"response": ..., "is_safe": false, "violations": [...]}. The verdict follows the answer so the model has full autoregressive context when judging; at the API boundary only the trailing JSON is parsed, and the buffered answer is dropped if the verdict is unsafe or unparsable (fail-closed). A final deterministic numeric/lexical scan re-checks the rendered text before delivery. The same middleware also includes a pre-generation keyword/regex fast path. These pieces work together: the lexical gate cheaply blocks obvious redlines, the self-verdict

What would settle it

Run Method F on a larger and independently labeled set of eating-disorder prompts, or swap the verdict model to a different family than the generator, and measure recall; if a different judge family or independent labels drop the 100% interception rate, the perfect recall was an artifact of the same model judging itself on a dataset whose labels came from the same policy. A second check: measure the false-positive rate on safe prompts, which the paper does not report.

Watch

Extended reading notes

Core claim

The paper's central claim is that a modular safety middleware—a lexical fast path plus an in-line LLM policy filter that returns a strict, fail-closed JSON verdict in a single model call—can intercept every unsafe eating-disorder prompt in its synthetic evaluation without the latency and token cost of multi-stage judge pipelines. In the comparison, methods without post-generation adjudication blocked only 4%–52% of malicious prompts, while both the output-judge method and the single-call JSON method reached 100% recall (50/50). The single-call method did so at 1.2x time and 1.5x token overhead, placing it on the Pareto frontier against ensembles that reached the same recall at 1.6–1.7x time

Load-bearing premise

That the language model's self-reported safety verdict on its own generated answer is trustworthy enough that, combined with the final keyword/numeric scan, it catches everything unsafe—if the model is wrong in the same way about its answer and its verdict, unsafe content can still pass.

Editorial extensions

If this is right

  • Post-generation adjudication is necessary for full recall: methods without an output gate (A, B, C) blocked only 4%, 22%, and 52% of malicious prompts, while both output-judge methods (E, F) reached 100%.
  • Stacking multiple safeguards adds cost without adding recall: the B+C+D+E and D+E ensembles also reached 100% but at 1.6–1.7x time and 4.8x tokens, versus Method F's 1.2x time and 1.5x tokens.
  • A fail-closed single-call verdict places the safety guarantee at the API boundary: unparsable or unsafe verdicts are treated as refusals, so the system does not depend on the model's goodwill.
  • Mapping Tessa's failure patterns to OWASP LLM Top 10 and NIST SP 800-53 controls makes the safeguards testable and auditable in governance terms, not just prompt-level intent.
  • The same modular primitives—input gating, buffered output, deterministic scanners, escalation—can be extended to adversarial classes like prompt injection, but the current prototype does not claim to cover them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper's perfect recall was measured with the same 2B model as both generator and judge, a natural next test is to replace the judge with a different model family; if recall drops, the 100% figure is partly an artifact of self-consistency rather than an independent safety check.
  • The synthetic dataset's labels are known by construction from the same policy being enforced, so the true false-positive rate on safe, real-world queries remains unmeasured; a deployment trial with independent clinical review would be the informative extension.
  • The single-call self-verdict pattern is not limited to eating-disorder redlines: it could be re-specified for medication dosing, self-harm, or other health-adjacent domains, with the deterministic scan re-targeted to domain-specific numerics and lexicon.
  • The architecture's safety guarantee is only as strong as the final deterministic scan's coverage, so writing the scan as a formal allowlist/denylist specification with test coverage would let the claim be checked automatically rather than by spot evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper analyzes the 2023 NEDA 'Tessa' chatbot incident as a case of absent safety engineering in a health-adjacent AI assistant. It proposes a modular safety middleware that combines deterministic lexical gates, an LLM policy filter, and a fail-closed gate, including a 'Single-Call JSON Mode' in which the generator emits a trailing is_safe verdict. The authors map Tessa's failure patterns to OWASP LLM Top 10 and NIST SP 800-53 controls, and they evaluate six deployment patterns (A–F) on 100 self-labeled synthetic prompts using Gemma 2 2B. The central empirical claim is that Method F achieves perfect recall (50/50, 100%) on malicious prompts at 1.2× time and 1.5× token cost relative to an insecure baseline, and thus that lightweight, auditable checks can prevent 'another Tessa.' The paper states clear limitations: small synthetic dataset, single backbone, same model family for adjudication, and no wild precision estimate.

Significance. If the empirical claim held, the contribution would be practically useful: a simple, auditable middleware that achieves high recall on the targeted Tessa-like failure modes at near-baseline cost, with a governance mapping that makes the safeguards actionable. The paper is commendable for releasing prompts, code, and audit logs, and for explicitly designing fail-closed behavior and a deterministic post-hoc scan. However, the central result rests on a single run over 100 synthetic prompts whose labels are derived from the same policy the system implements, and on a self-verdict from the same model that generates the answer. These concerns are acknowledged in the Limitations section but are not resolved by the reported experiments. With additional ablations and independent evaluation, the approach could be a useful engineering case study; in its current form, the strength of the conclusion exceeds the evidence.

major comments (5)
  1. [Results, Table 3] The narrative and the table disagree on Method D's recall. The text states 'The input-side judge (D) blocks 47/50 malicious queries (94%)' and that the unblocked prompts were caught only by output-side adjudication, while Table 3 reports D as 50/50 (100%). Because the paper's comparative claim is that post-generation adjudication is necessary for full recall, this contradiction is load-bearing. Please correct the inconsistency and verify the counts for every row.
  2. [Single-Call JSON Mode; Evaluation Protocol] The design mandates a final deterministic numeric/lexical re-scan of the buffered output 'even in cases where is_safe=true', but the Method F evaluation description says only that 'the answer is discarded if is_safe=false or if parsing fails.' It is not reported whether the deterministic scanner was applied in the F pipeline. If it was not applied, the 100% recall is attributable entirely to the self-verdict, which is exactly the failure regime of concern; if it was applied, the contribution of the self-verdict is confounded with the scanner. Please report Method F with and without the deterministic scan, including per-prompt verdicts and scan outcomes.
  3. [Environment; Limitations] Method F's judge is the same model, in the same call, as the generator. The paper itself states that reusing the same model family 'risks correlated failures and limits external validity'; for Method F the correlation is exact, not merely familial. Any failure mode that makes the model produce dieting advice (e.g., euphemism, obfuscation, policy drift) can also make it set is_safe=true, because both the answer and the verdict come from the same autoregressive distribution. The current experiment cannot rule out that Method F's perfect recall is an artifact of this correlation. Please evaluate Method F with a judge model of a different family (or at least a different prompt/profile) and report disagreement rates.
  4. [Evaluation Protocol; Table 3] The protocol records the number of non-malicious prompts flagged, but the results do not report this quantity for any method, and no precision or false-positive analysis is given. For a safety middleware in a health-adjacent setting, false positives are as practically important as recall: blocking a large fraction of safe queries would be unacceptable even with perfect recall. Please report the full confusion matrix (or at least safe-prompt flag rates) and, ideally, threshold/calibration analysis for the classifier components.
  5. [Dataset; Limitations] The 100-prompt test set is labeled 'known by construction' from the same policy criteria that the middleware enforces. This means the evaluation measures consistency with the policy specification, not real-world safety. The paper should be explicit that the 100% recall is an upper-bound/self-consistency result, and should obtain independent human labels or a held-out naturally occurring corpus (even a small one) to ground the central claim.
minor comments (5)
  1. [Throughout] Typographical errors: 'OW ASP' should be 'OWASP'; 'readon' in Background should be 'reason'; 'na¨ıve' has an encoding issue.
  2. [Methodology] The deployment patterns are labeled A–F, but the text refers to 'Section 1–6' without numbered sections; please add consistent numbering or cross-references.
  3. [Table 3] The table would benefit from a precision column and explicit error bars or repeated-trial ranges for time/token multipliers. Currently these are single measurements with no variance information.
  4. [Environment] The sentence saying the same model was used for 'Input-Judge and Output-Judge' is slightly ambiguous for Method F, which uses a single call; please clarify how the 'different prompting configurations' were realized within that call.
  5. [Related Work] The reference list would benefit from page/venue details for conference papers (e.g., OWASP Top 10, ToxiGen) so readers can locate the sources; several entries are only arXiv identifiers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported results are empirical evaluations, not derivations that reduce to their inputs.

full rationale

The paper's central claims (blocking recall, latency, token overhead) are empirical measurements on a synthetic dataset, not outputs of a derivation chain that depends on fitted parameters or self-cited theorems. The evaluation labels are 'known by construction, based on policy criteria' (Dataset), and the middleware implements the same policy, so the test set does not provide independent external grounding. This is a validity limitation, which the paper itself acknowledges (e.g., 'does not estimate precision on safe prompts', 'small, synthetic dataset', 'risks correlated failures and limits external validity' in Limitations). It does not make the reported 100% recall definitionally forced, because the model's verdicts are generated responses, not read off the label set. The self-citation to SecuRAG (Reddy 2025) is used as an implementation framework with public code and audit logs, not as evidence that substitutes for the evaluation, so it is not load-bearing. The concern that Method F's self-verdict is correlated with generation is a substantive correctness/validity risk, not a circularity in the sense of the derivation being equivalent to its inputs. Therefore, no circular step is identified.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The load-bearing assumptions are domain-specific: the clinical redlines are taken as authoritative, the single-call self-verdict is assumed reliable enough, and the synthetic dataset is assumed representative. The only numeric free parameters are thresholds for an optional classifier path that was not exercised in the evaluation.

free parameters (1)
  • Policy classifier thresholds t_lo, t_hi = not fitted in this evaluation
    Defined in 'Input Filtering' as thresholds for the continuous risk score p; the evaluation does not implement this classifier path, so these numbers are design placeholders rather than fitted values.
assumptions (3)
  • domain assumption The clinical redlines (no calorie targets, no weigh-ins, no BMI coaching, no restrictive-diet framing) are the correct standard for eating-disorder support.
    The entire evaluation and policy design treat these redlines as ground truth; they are asserted in the policy section and dataset construction, not derived from clinical evidence.
  • domain assumption A single LLM call can produce both a faithful answer and an accurate trailing JSON is_safe verdict for that answer.
    Single-call JSON Mode (Method F) relies on this; the paper notes correlated-failure risk from reusing the same model as judge, but the design assumes the self-verdict is usually trustworthy enough when combined with final scans.
  • domain assumption The 100 synthetic prompts are representative of user behavior around eating-disorder support, and their constructed labels are correct.
    Dataset section says labels were 'known by construction' and near-duplicates were removed; no real user logs or external validation sets are used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants." pith.science (2026). https://pith.science/paper/C2ET7TBN

@misc{pith2026250907022,
  author       = {Pith},
  title        = {Pith review of: Preventing Another Tessa: Modular Safety Middleware For Health-Adjacent AI Assistants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C2ET7TBN}},
  note         = {Machine review of arXiv:2509.07022}
}
read the original abstract

In 2023, the National Eating Disorders Association's (NEDA) chatbot Tessa was suspended after providing harmful weight-loss advice to vulnerable users-an avoidable failure that underscores the risks of unsafe AI in healthcare contexts. This paper examines Tessa as a case study in absent safety engineering and demonstrates how a lightweight, modular safeguard could have prevented the incident. We propose a hybrid safety middleware that combines deterministic lexical gates with an in-line large language model (LLM) policy filter, enforcing fail-closed verdicts and escalation pathways within a single model call. Using synthetic evaluations, we show that this design achieves perfect interception of unsafe prompts at baseline cost and latency, outperforming traditional multi-stage pipelines. Beyond technical remedies, we map Tessa's failure patterns to established frameworks (OWASP LLM Top10, NIST SP 800-53), connecting practical safeguards to actionable governance controls. The results highlight that robust, auditable safety in health-adjacent AI does not require heavyweight infrastructure: explicit, testable checks at the last mile are sufficient to prevent "another Tessa", while governance and escalation ensure sustainability in real-world deployment.

Figures

Figures reproduced from arXiv: 2509.07022 by the authors.

Figure 1
Figure 1. Safety middleware pipeline with input mediation, secure generation, output moderation, and a fail-closed gate that [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

22 extracted references · 16 canonical work pages

  1. [1]

    Aratani, L. 2023. US eating disorder helpline takes down chatbot after it teaches harmful diet tips

    work page 2023

  2. [2]

    Bai, Y.; et al. 2022. Constitutional AI: Harmlessness from AI Feedback

    work page 2022

  3. [3]

    Biggio, B.; Nelson, B.; and Laskov, P. 2012. Poisoning Attacks against Support Vector Machines. arXiv preprint arXiv:1206.6389

    arXiv 2012

  4. [4]

    Carlini, N.; Tramer, F.; Wallace, E.; Jagielski, M.; Herbert-Voss, A.; Lee, K.; Roberts, A.; Brown, T.; Song, D.; Erlingsson, U.; Oprea, A.; and Raffel, C. 2021. Extracting Training Data from Large Language Models. arXiv preprint arXiv:2012.07805

    arXiv 2021

  5. [5]

    Fredrikson, M.; Jha, S.; and Ristenpart, T. 2015. Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security (CCS) — arXiv preprint

    work page 2015

  6. [6]

    Greshake, K.; Abdelnabi, S.; Mishra, S.; Endres, C.; Holz, T.; and Fritz, M. 2023. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. arXiv preprint arXiv:2302.12173

    arXiv 2023

  7. [7]

    Hartvigsen, T.; et al. 2022. ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection. In Proceedings of ACL 2022, 3309--3326

    work page 2022

  8. [8]

    Chan School of Public Health

    Harvard T.H. Chan School of Public Health . 2023. Artificial intelligence tools offer harmful advice on eating disorders

    work page 2023

Show all 22 references
  1. [9]

    Hoover, A. 2023. An Eating Disorder Chatbot Is Suspended for Giving Harmful Advice

    2023

  2. [10]

    Joint Task Force, N. 2020. Security and Privacy Controls for Information Systems and Organizations (SP 800-53 Rev.5). Technical report, National Institute of Standards and Technology

    2020

  3. [11]

    P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P

    Madaan, A.; Tandon, N.; Gupta, P.; Hallinan, S.; Gao, L.; Wiegreffe, S.; Alon, U.; Dziri, N.; Prabhumoye, S.; Yang, Y.; Gupta, S.; Majumder, B. P.; Hermann, K.; Welleck, S.; Yazdanbakhsh, A.; and Clark, P. 2023. SELF-REFINE: iterative refinement with self-feedback. In Proceedi...

    2023

  4. [12]

    Nguyen, B. 2023 a . Eating disorder helpline that reportedly planned to fire its human staff disables its chatbot after a woman said it told her to count calories, weigh, and measure herself weekly. Accessed 2025-08-20

    2023

  5. [13]

    Nguyen, B. 2023 b . Helpline workers for the National Eating Disorder Association say they are being replaced by AI

    2023

  6. [14]

    NIST. 2023. Artificial Intelligence Risk Management Framework (AI RMF 1.0). Technical report, National Institute of Standards and Technology

    2023

  7. [15]

    Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; Schulman, J.; Hilton, J.; Kelton, F.; Miller, L.; Simens, M.; Askell, A.; Welinder, P.; Christiano, P.; Leike, J.; and Lowe, R. 2022. Training language model...

    2022

  8. [16]

    OWASP . 2024. OWASP Top 10 for Large Language Model Applications

    2024

  9. [17]

    Reddy, P. 2025. SecuRAG: Modular Guardrails for Secure LLM Applications

    2025

  10. [18]

    Shokri, R.; Stronati, M.; Song, C.; and Shmatikov, V. 2017. Membership Inference Attacks Against Machine Learning Models. arXiv preprint arXiv:1610.05820

    2017 arXiv

  11. [19]

    Wells, K. 2023. What Does a Chatbot Know About Eating Disorders? Users of a Help Line Are About to Find Out

    2023

  12. [20]

    P.; Zhang, H.; Gonzalez, J

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E. P.; Zhang, H.; Gonzalez, J. E.; and Stoica, I. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Inf...

    2023

  13. [21]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.