{"id":"e749582a-9e93-479a-8f43-8c03468d1156","arxiv_id":"2606.28514","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPTNT benchmark demonstrates that state-of-the-art multimodal models cannot perform real-time collaborative bomb defusal in Keep Talking and Nobody Explodes, unlike human players.","lead":"This paper introduces GPTNT, a benchmark using the cooperative game Keep Talking and Nobody Explodes to test real-time collaboration between multimodal agents under time pressure and information asymmetry. Current AI models fail to defuse any bombs in the setup while humans succeed, exposing gaps in state tracking and communication.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Withholding the manual may fail to isolate derivation if models have internalized standard KTANE rules from pretraining","rationale":"The reader's weakest_assumption directly identifies the isolation-from-memorization issue as the key unverified premise; the full-text placeholder does not supply additional evidence (such as training-data decontamination checks or rule-altered ablations) that would remove the concern, so the UNVERDICTED verdict and LOW confidence are unaffected.","tokens_in":1795,"tokens_out":343,"duration_ms":35556,"concrete_test":"Re-run the withheld-manual condition on the same models after replacing the standard manual with a version that inverts or alters two core module rules (e.g., swap the wire-cut order on the Wires module and change the button-press condition on the Button module); if success rate remains zero and error patterns are unchanged, the original failures are not explained by memorized manual knowledge.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (zero model successes vs. human successes) depends on the claim that withholding conditions (manual, partner, or both) plus procedural generation cleanly separate in-the-moment reasoning and collaboration from memorized knowledge. Because the KTANE manual is a fixed public document, any model whose training corpus includes game wikis, rule PDFs, or playthrough transcripts could still apply the correct module-solving procedures even when the text is withheld at runtime. Procedural generation randomizes module layouts and configurations but does not alter the underlying rule semantics, so it does not block this pathway. If this leakage occurs, the observed failures cannot be attributed solely to deficits in state tracking, real-time action, or ambiguity handling.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces the GPTNT benchmark, built on the cooperative game Keep Talking and Nobody Explodes, to evaluate real-time multimodal agent collaboration under time pressure, information asymmetry, and imperfect communication. Two agents must coordinate asynchronously: one manipulates the bomb without instructions, the other holds the manual without visual access. The design includes withholding conditions (manual, partner, or both) plus procedural generation to isolate in-the-moment derivation and collaboration skills from memorized knowledge. The central empirical claim is that none of the tested closed- or open-source models defuse any bomb in real time (a bar humans clear), with controlled experiments identifying weaknesses in state tracking, efficient action under time pressure, ambiguity handling, and error recovery. The benchmark is released to evolve with the modding community.","tokens_in":1913,"tokens_out":516,"duration_ms":33597,"significance":"If the isolation claim holds and the zero-success result is robust, GPTNT would provide a valuable, hard-to-game benchmark for collaborative capabilities that existing evaluations do not measure. Strengths include the use of a real game with procedural generation (preventing static solution memorization) and the potential for the benchmark to scale rather than be retired. The identification of concrete failure modes offers actionable directions, though the significance is tempered by the need to confirm that observed failures stem from the targeted skills rather than confounds.","major_comments":[{"comment":"Abstract and benchmark design: the claim that withholding the manual (or partner) isolates 'what a model derives in the moment from what it already knows' is load-bearing for attributing the zero-success result to deficits in state tracking, real-time action, and collaboration. Because the KTANE manual is a fixed public document, models whose pretraining includes wikis, rule PDFs, or transcripts could still apply correct procedures even when the text is withheld at runtime. Procedural generation randomizes module layouts and configurations but leaves rule semantics unchanged, so it does not block this pathway. Without additional controls (e.g., explicit tests of rule recall when the manual is absent), the headline result cannot be cleanly attributed to the intended skills.","section":"Abstract / Benchmark Design"},{"comment":"Experimental results section: the abstract reports that 'none of the closed- or open-source models we test defuses a single bomb in real time' yet supplies no information on the number of trials per condition, specific models and versions tested, success criteria, error bars, or statistical tests. This absence prevents verification that the zero-success rate is reliable rather than an artifact of small sample size or particular prompting setups.","section":"Experimental Results"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the detailed and constructive report. We address each major comment below and outline revisions to improve clarity and robustness of the claims.","responses":[{"response":"We agree this is a valid concern: because the KTANE manual is a fixed public document, pretraining exposure could allow rule recall even under withholding. Procedural generation varies module instances but not core semantics. To strengthen the isolation claim, we will add a dedicated control experiment measuring rule recall accuracy when the manual is withheld at runtime (comparing against conditions where it is provided). This will be reported in a new subsection of the benchmark design and results, allowing clearer attribution of failures to state tracking and collaboration rather than memorized knowledge.","revision_made":"yes","referee_comment":"[Abstract / Benchmark Design] The claim that withholding the manual (or partner) isolates 'what a model derives in the moment from what it already knows' is load-bearing... Without additional controls (e.g., explicit tests of rule recall when the manual is absent), the headline result cannot be cleanly attributed to the intended skills."},{"response":"The full manuscript provides these details in the Experimental Results section (including trial counts, model versions, exact success criteria based on bomb defusal within the timer, and any applicable statistics). However, we acknowledge the abstract is too terse for standalone verification. We will revise the abstract to include a concise statement on the number of trials, models evaluated, and success definition.","revision_made":"yes","referee_comment":"[Experimental Results] the abstract reports that 'none of the closed- or open-source models we test defuses a single bomb in real time' yet supplies no information on the number of trials per condition, specific models and versions tested, success criteria, error bars, or statistical tests."}],"tokens_in":1625,"tokens_out":396,"duration_ms":25038,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The punchline is that GPTNT introduces a benchmark for asynchronous multimodal agent collaboration in Keep Talking and Nobody Explodes, where one agent sees the bomb and the other holds the instructions, and no tested models succeed while humans do. The design tries to isolate on-the-spot reasoning through procedural generation and withholding conditions.\n\nThe paper does a reasonable job framing why existing benchmarks fall short by studying time pressure, asymmetry, and communication in isolation. Tying the test to a real game with mods and procedural elements gives it longevity, and the reported weaknesses in state tracking, efficient action, ambiguity handling, and error recovery line up with what the setup would expose.\n\nThe soft spot is the isolation claim. The stress-test concern is worth taking seriously: the KTANE manual is a fixed public document, so pretraining on wikis or transcripts could let models apply known procedures even when the text is withheld at runtime. Randomizing layouts does not change the underlying rule semantics. The abstract provides no details on trial counts, error bars, model outputs, or checks for this leakage, so it is hard to tell how much the failures reflect collaboration deficits versus other factors. Without that, the headline result is difficult to interpret cleanly.\n\nThis is for people building or evaluating collaborative agents who need a more integrated test than component benchmarks. A reader focused on evaluation design would find the setup useful. It deserves peer review because the benchmark idea is fresh and the environment is well-chosen, even if the results section will need more evidence on confounds.","headline":"GPTNT is a new real-time collaboration benchmark with a clever game setup, but the zero-success claim for models rests on an unverified assumption that withholding the manual blocks memorized rules.","tokens_in":2390,"tokens_out":389,"would_cite":false,"duration_ms":22911,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"No tested multimodal models defuse even one bomb in real time under the benchmark conditions, while human players succeed.","keywords":["multimodal agents","real-time collaboration","benchmark","information asymmetry","cooperative games","state tracking","procedural generation","error recovery"],"falsifier":"A model that completes at least one procedurally generated bomb defusal in real time when either the instruction manual or the partner is withheld.","tokens_in":2702,"feed_emoji":"💣","tokens_out":626,"duration_ms":19647,"temperature":0.7,"pith_summary":"The paper presents GPTNT, a benchmark built on the cooperative game Keep Talking and Nobody Explodes, where two agents must collaborate in real time against a countdown. One agent sees and manipulates the bomb without instructions; the other has the instructions but cannot see the bomb. The setup uses procedural generation and conditions that withhold the manual, the partner, or both to force genuine derivation and communication rather than recall of known solutions. Current models from both closed and open sources achieve zero successes, exposing shortfalls in state tracking, action under time pressure, ambiguity handling, and error recovery.","feed_headline":"AI models record zero successes on real-time bomb defusal","feed_subtitle":"New benchmark from Keep Talking and Nobody Explodes shows models fail where humans succeed under time pressure and split information.","key_machinery":"The GPTNT benchmark, which places agents in asymmetric roles within a live countdown game and withholds the manual or partner to isolate real-time collaboration from memorized knowledge.","core_discovery":"GPTNT requires two agents to coordinate asynchronously in real time to defuse procedurally generated bomb modules, with success depending on effective communication across information asymmetry. None of the tested models completes even a single defusal, a threshold human players meet, and controlled experiments trace the failures to specific weaknesses in maintaining shared state, acting efficiently under countdown pressure, resolving ambiguous messages, and recovering from mistakes.","pith_inferences":["Similar real-time asymmetry tests could be applied to other domains where agents must act without full shared information.","Models that improve on GPTNT might transfer better to dynamic human-AI teaming scenarios than those tuned only on static benchmarks.","The zero-success result suggests current training may under-emphasize efficient communication under strict time limits."],"forward_implications":["Models exhibit specific shortfalls in state tracking and handling time pressure that prevent collaborative success.","Error recovery and ambiguity resolution remain insufficient for tasks requiring ongoing coordination.","The benchmark can continue to evolve through the game's procedural generation and modding community instead of becoming fixed and retired.","Withholding conditions allow direct measurement of derivation versus recall in collaborative settings."],"fun_headline_variants":["Zero AI successes in real-time bomb defusal benchmark","Models achieve no defusals under time pressure in GPTNT","AI fails real-time coordination on bomb defusal game","New benchmark shows no model defuses bomb asynchronously"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The game setup with procedural generation and withholding conditions accurately isolates in-the-moment derivation and collaboration skills from memorized knowledge or other confounds.","fun_headline_variants_meta":{"raw":{"variants":["Zero AI successes in real-time bomb defusal benchmark","Models achieve no defusals under time pressure in GPTNT","AI fails real-time coordination on bomb defusal game","New benchmark shows no model defuses bomb asynchronously"]},"model":"grok-4.3","cost_usd":0.004492,"raw_usage":{"total_tokens":2199,"prompt_tokens":751,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":44915500,"prompt_tokens_details":{"text_tokens":751,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1386,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":751,"tokens_out":62,"duration_ms":12631,"temperature":1.0,"reasoning_tokens":1386,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-30T01:11:18.321286+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A model that completes at least one procedurally generated bomb defusal in real time when either the instruction manual or the partner is withheld.","supporting_citations":[],"review_version":1}