{"id":"79fa0a28-4a44-4cae-8b85-45663ed6aca4","arxiv_id":"2507.14207","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"The paper reports simulated vulnerabilities of GPT-3.5 and GPT-4 to educational prompt-injection chains, but the supporting code reveals the data were generated by a random simulation rather than real model interactions.","lead":"This paper claims that students can Trojanize prompts to make GPT-3.5 and GPT-4 bypass safety filters, and reports a 500-trial experiment with high bypass rates. It also sketches a detection tool called TrojanPromptGuard, though no working implementation or evaluation is provided.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The empirical core rests on Appendix A's synthetic data generator rather than real GPT API logs; if Table 1 is reproducible from that code, the claimed 50% bypass finding and all model-comparison conclusions lack an evidential basis.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing issue: Table 1 is presented as experimental results from GPT-3.5 and GPT-4, but the only supporting artifact in Appendix A is a synthetic data generator. That is not a prosecution of the authors; it is a description of the evidence actually supplied. The paper's conclusion, 'Trojanized prompt chains can bypass existing safeguards when masked in moral, pedagogical, or literary forms,' stands or falls on the reality of the 500 trials. If the table is generated by the appendix code, the central empirical result is not an observation at all but a restatement of the code's priors. Several secondary issues reinforce rejection: the chi-square test is non-significant yet interpreted as showing practical relevance; the claimed 50% overall high-risk bypass rate does not match a weighted average of Table 1's high-risk rows (~46.7%); and the two illustrative 'unsafe' responses are arguably benign cautionary or fictional text, weakening the classification of safety violations. The TrojanPromptGuard section is explicitly a design with an evaluation plan, not an implementation, so it cannot independently support the empirical claims. None of these points rely on questioning author intent; they are all checkable from the paper's own text and code. I therefore see no reason to change the reader's REJECT verdict.","tokens_in":7157,"tokens_out":4800,"duration_ms":53423,"concrete_test":"Run Appendix A's script verbatim with np.random.seed(42) and compare its printed summary to Table 1, row by row. Also recompute the overall high-risk bypass rate from Table 1's Trials and Bypass Rate columns. If the script reproduces Table 1 exactly, or if no timestamped API logs or raw prompt/response records can be produced for the 500 trials, the paper's empirical claims should be withdrawn or re-labeled as a simulation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that real GPT-3.5/GPT-4 sessions produced Table 1, and Section 4.3 states a '50% overall bypass rate for high-risk prompts.' The only code provided (Appendix A) never calls an LLM API. It samples model, chain, turn, and risk with random choices, then draws moderation and bypass flags from fixed per-risk probabilities: Low is always no/no, Medium is 10% moderation/15% bypass, High is 20% moderation/50% bypass. Running that script with seed 42 will generate a summary that should match Table 1 row-for-row if the table came from this script. If it matches, the statement 'Experiments were executed on both GPT-3.5 and GPT-4' is unsupported, and the 50% figure is simply the hard-coded High-risk bypass probability, not an observed outcome. Even taking Table 1 as given, the reported chi-square (0.450, p=0.930) provides no statistical basis for claims that GPT-4 is more susceptible or that high-risk bypass behavior is elevated. The two supplied 'unsafe' examples are ordinary cautionary or satirical text and do not independently evidence guardrail failure. Since the tool design, recommendations, and conclusions all lean on this empirical bypass phenomenon, the absence of real interaction logs is the decisive weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims to show experimentally that multi-turn 'Trojanized' prompts, framed as Simulated Child Confusion (SCC) or Prompt Chain Escalation via Literary Devices (PCELD), can bypass moderation in GPT-3.5 and GPT-4 in educational settings. It reports a 500-trial experiment with a 50% overall bypass rate for high-risk prompts, presents a chi-square analysis, and proposes a modular detection tool called TrojanPromptGuard (TPG). The paper concludes that educational LLM use is vulnerable to such attacks and that TPG is a viable mitigation architecture.","tokens_in":7418,"tokens_out":6982,"duration_ms":80270,"significance":"The topic is timely and important: LLM safety in K-12 education is a substantive concern, and a validated detection middleware would be valuable. The paper identifies a plausible threat model and offers a reasonable high-level architecture for TPG. However, the empirical core is not trustworthy: the only code supplied is a synthetic-data generator, the reported chi-square test is non-significant, and the two supplied 'unsafe' examples do not clearly violate content policy. Because the conclusions about model bypass behavior and guardrail failure rest on these unsupported findings, the paper does not currently provide reliable evidence for its central claims.","major_comments":[{"comment":"The only supporting code, Appendix A, simulates a 500-trial dataset; it never calls an LLM API, and it samples moderation and bypass flags from fixed per-risk probabilities (Low: 0/0, Medium: 10%/15%, High: 20%/50%). The code comment explicitly labels this a 'Simulated 500-trial experiment.' This directly contradicts Section 4.1's statement that 'Experiments were executed on both GPT-3.5 and GPT-4.' Unless the authors provide the raw API logs or a real execution script, Table 1 must be treated as synthetic, and every reported bypass rate is an artifact of the chosen simulation probabilities rather than an observed model behavior.","section":"4.1 and Appendix A"},{"comment":"The claim of a '50% overall bypass rate for high-risk prompts' is not supported by Table 1. Summing the four high-risk rows gives 71 bypasses out of 154 trials, or 46.1%, and the unweighted average of the four row rates is 46.7%. The 50% figure appears to be the hard-coded high-risk bypass probability from the Appendix A simulation, not a derived statistic from the reported data. This figure should be corrected or the underlying experiment logs supplied.","section":"4.3 and Table 1"},{"comment":"The chi-square test reported in Table 2 yields p = 0.930 with 3 degrees of freedom, which is far from significant and provides no statistical evidence for an association between model type or chain category and bypass outcomes. Yet Section 4.3 says GPT-4 'was also more susceptible' and Section 6 claims GPT-4 showed 'greater vulnerability.' These conclusions are not derivable from a non-significant test, and the high-risk row rates are inconsistent in direction (GPT-4 SCC 34.88% vs GPT-3.5 SCC 45.45%, while GPT-4 PCELD 62.86% vs GPT-3.5 PCELD 43.75%). The caveat about 'practical differences' cannot carry the paper's conclusions.","section":"4.2, 4.3, and 6"},{"comment":"The two representative failure examples are not clearly unsafe or unintended. The bleach/ammonia response is a factual safety warning, and the 'Totalitaria' story is a generic satirical narrative. The paper does not identify the specific content-policy category violated or explain why these responses constitute a safety failure under OpenAI's moderation guidelines. These examples therefore do not independently demonstrate a guardrail bypass.","section":"4.4 and 4.5"},{"comment":"TPG is described as a 'prototype tool,' but no implementation, code, or evaluation is presented. Section 5.5 explicitly states that deployment is future work and lists only planned evaluation metrics. The conclusion in Section 8 that TPG is a 'viable architecture' and the claim of 'documenting real-world bypasses' are therefore unsupported by the manuscript's evidence.","section":"5 and 8"}],"minor_comments":[{"comment":"There are repeated typos: 'guardrils' in the abstract, 'practicitoners' in Section 2, and 'occuring,' 'guards againts,' and 'Excalation' in Section 5. These should be corrected.","section":"Abstract and Section 2"},{"comment":"Figure 1 is referenced in the text but no figure appears in the manuscript; please include the figure or remove the reference.","section":"Section 4"},{"comment":"The chain-type labels in Appendix A ('SimulatedChildConfusion' and 'Prompt Chain Escalation') do not match the labels in Table 1 ('SCC' and 'PCELD'); clarify the mapping and ensure the code and table use consistent terminology.","section":"Appendix A and Table 1"},{"comment":"Several references are incomplete or appear misattributed; for example, the GPT-4 system card URL points to an o3/o4-mini system card, and the 'Ziegler et al. (2023)' citation lacks a verifiable source. Please check all citations.","section":"References"},{"comment":"The claim that 'AI applications are doubling in power every 5.7 months' is unsupported by a citation and is not needed for the paper's argument; consider removing it or citing a specific source.","section":"Section 1"}],"recommendation":"reject","confidential_remarks":"The decisive issue is the mismatch between the claimed real experiments and the only provided code, which is a synthetic-data generator. If the authors cannot supply the actual GPT-3.5/GPT-4 interaction logs or scripts that call the API, the empirical section of this paper should not be published as an experimental study. I recommend rejection, though the paper could be reconsidered if rebuilt around genuine logged experiments with a proper statistical analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: don't send this to review. The paper reads as a red-team report, but its load-bearing result is not supported by the evidence it provides. The only code in Appendix A never calls an API; it randomly samples outcomes with fixed per-risk probabilities. The claimed 50% bypass rate for high-risk prompts is exactly the hardcoded probability in that script, not an observed value. Section 4.1 says experiments were \"executed on both GPT-3.5 and GPT-4,\" but there are no raw logs, no API traces, and the only supporting code contradicts that statement. That internal contradiction is decisive.\n\nTo be fair, the educational framing is a useful lens. The two chains—Simulated Child Confusion and Prompt Chain Escalation via Literary Devices—are plausible variants of well-known role-play and moral-framing jailbreaks, and the paper maps them to K-12 contexts sensibly. The recommendations (turn-aware moderation, role-consistency checks) are reasonable and align with current thinking, though none are new. The TPG design is a sketch with no implementation, so it can't be evaluated.\n\nThe statistical analysis is also weak: the chi-square p-value is 0.930, non-significant, but the text then highlights \"practical differences.\" That's fine as a caveat, but the 50% figure in Section 4.3 is presented as a result when it is just a simulation parameter. The two example \"unsafe\" outputs are benign safety explanations, which suggests the coding of safety violations is unreliable.\n\nBottom line: this is a position paper, not an empirical study. If the authors had presented it as a design exploration with illustrative simulated data, it could be a modest contribution. As written, the claim of real experiments is unsupported and contradicted by the appendix. I would desk reject, and I would not cite it.","headline":"The paper's central empirical claim is undone by its own appendix: the '500-trial experiment' is a synthetic data generator, not real GPT-3.5/GPT-4 logs.","tokens_in":8007,"tokens_out":2165,"would_cite":false,"duration_ms":24404,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Guardrails in GPT-3.5 and GPT-4 can be bypassed by multi-turn prompts that hide dangerous requests inside moral or literary framing, a 500-trial test claims.","keywords":["prompt injection","LLM safety","jailbreak","K-12 education","multi-turn dialogue","content moderation","trojan prompt","adversarial prompting"],"falsifier":"Search the paper's supporting materials for the raw session logs from the 500 trials; if no logs exist and the table is reproducible from the random-seed script in Appendix A, the reported bypass rates would be artifacts rather than observed behavior.","tokens_in":6881,"feed_emoji":"🛡️","tokens_out":5765,"duration_ms":61135,"temperature":0.7,"pith_summary":"This paper argues that LLM safety guardrails can be defeated in K-12 educational settings by 'trojanized' prompt chains, where a benign-sounding multi-turn dialogue gradually escalates into a request for restricted information. It reports experiments on GPT-3.5 and GPT-4 using two exploit patterns: simulated child confusion and literary device escalation. The headline finding is that for high-risk prompts, neither model flagged the transition from educational to dangerous territory, producing an overall 50% bypass rate. The paper also proposes a detection tool, TrojanPromptGuard, that scores each prompt turn for role inconsistency, semantic drift, and trojan phrases.","feed_headline":"High-risk prompts bypass LLM guardrails 50% of the time","feed_subtitle":"Multi-turn moral and literary framing slips past GPT-3.5 and GPT-4 moderation in simulated K-12 dialogues.","key_machinery":"The load-bearing devices are two multi-turn prompt chains: Simulated Child Confusion (SCC), which impersonates an innocent student asking ethically framed questions about dangerous topics, and Prompt Chain Escalation via Literary Devices (PCELD), which frames restricted requests as satire, metaphor, or fictional narrative. Each chain uses 3-4 turns of increasing semantic ambiguity, with the third turn as the reported critical escalation stage. The paper's proposed mitigation, TrojanPromptGuard, is a six-component middleware that uses role detection, cosine-similarity semantic drift tracking, regex and named-entity pattern matching, and a weighted risk scorer to flag or block suspicious prompts before they reach the LLM.","core_discovery":"The paper's central claim is that content moderation in GPT-3.5 and GPT-4 can be bypassed when harmful requests are disguised in moral, pedagogical, or literary forms over multiple dialogue turns. In both exploit chains, the third turn was the inflection point where benign framing became risky, and neither model raised a moderation flag, yielding a 50% bypass rate for high-risk prompts. The paper further reports that GPT-4, despite greater semantic nuance, was more susceptible to framing exploits. On the strength of these findings, it proposes that moderation move from isolated prompt analysis to session-level, turn-aware evaluation, and it offers TrojanPromptGuard as a prototype middleware design to implement that shift.","pith_inferences":["A natural next test is whether the third-turn inflection appears in live experiments with current frontier models, which would confirm that chain length itself, not a specific model version, drives the effect.","The same TPG component design could be benchmarked against public jailbreak datasets to see whether its risk scores correlate with human red-team judgments before any classroom deployment.","Because the paper's chi-square test found no statistically significant model difference, the meaningful effect is the aggregate 50% bypass rate; future work should focus on that rather than on GPT-4 versus GPT-3.5."],"forward_implications":["If the reported bypass rates hold, schools relying on the tested models' built-in moderation cannot assume that a benign first turn guarantees safe output later.","Moderation systems that evaluate only single prompts will miss the gradual escalation this study identifies; session-level analysis becomes the recommended target.","The proposed TPG architecture, though not yet evaluated with real user traffic, is a concrete blueprint for an intercepting layer in learning-management systems.","The failure modes named in the paper—unsafe chemical mixing advice and satire embedding extremist ideology—give content-moderation teams specific patterns to test against."],"supporting_citations":[{"why":"Supplies the adversarial red-team method that the experiment adapts to educational contexts.","marker":"Ziegler et al. (2023)"},{"why":"Demonstrates that smaller models can generate effective adversarial prompts, informing the design of the prompt chains.","marker":"Zou et al. (2023)"},{"why":"Documents rapid student adoption and misuse of AI tools, establishing the educational threat context.","marker":"Holmes et al. (2023)"},{"why":"Catalogs prompt injection as a top LLM security risk, framing the vulnerability class under study.","marker":"OWASP (2023)"},{"why":"Describes the content moderation systems that the paper tests against GPT-3.5 and GPT-4.","marker":"OpenAI (2023)"},{"why":"Provide publicly shared exploit recipes that the two prompt chains are modeled on.","marker":"HackGPT (2023) and JailbreakGPT (2023)"}],"fun_headline_variants":["Trojanized prompts bypass LLM guards 50% in classroom tests","Multi-turn moral framing slips past GPT-4 and GPT-3.5","Third turn is the breaking point for LLM safety in dialogues","TrojanPromptGuard detects hidden malicious prompt chains in classrooms","Session-level moderation needed to stop LLM prompt hacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central results rest on the assumption that Table 1 records real API interactions with GPT-3.5 and GPT-4; if those numbers came from the synthetic-data generator in Appendix A instead, the empirical claims do not stand.","fun_headline_variants_meta":{"raw":{"variants":["Trojanized prompts bypass LLM guards 50% in classroom tests","Multi-turn moral framing slips past GPT-4 and GPT-3.5","Third turn is the breaking point for LLM safety in dialogues","TrojanPromptGuard detects hidden malicious prompt chains in classrooms","Session-level moderation needed to stop LLM prompt hacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4033,"prompt_tokens":801,"completion_tokens":3232,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":417,"completion_tokens_details":{"reasoning_tokens":3143}},"tokens_in":417,"tokens_out":3232,"duration_ms":29753,"temperature":1.0,"reasoning_tokens":3143,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:17:51.147926+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search the paper's supporting materials for the raw session logs from the 500 trials; if no logs exist and the table is reproducible from the random-seed script in Appendix A, the reported bypass rates would be artifacts rather than observed behavior.","supporting_citations":[{"cited_title":"Training Compute of Frontier AI Models Grows by 4-5x per Year","cited_arxiv_id":null,"evidence_quote":"Documents rapid student adoption and misuse of AI tools, establishing the educational threat context."}],"review_version":1}