{"id":"6fc1c9c1-d558-4fba-a8fc-dbd64693956a","arxiv_id":"2411.11929","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"An LLM-guided IoT HTTP fuzzer reports 103 vulnerabilities in 14 devices, 68 previously undisclosed, using GPT-4o for field annotation, seed enrichment, and scheduling.","lead":"ChatHTTPFuzz uses large language models to read HTTP request structure and device firmware code, then generates protocol-aware test cases to fuzz IoT web interfaces. In tests on 14 real devices, the authors report 103 vulnerabilities, including 23 assigned CVEs, and claim more findings than three existing fuzzers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'more vulnerabilities' claim may rest on unequal oracles: ChatHTTPFuzz's command-injection detection and LLM-seeded dictionaries are not given to baselines, and Snipuzz/Mutiny lack full per-device totals.","rationale":"The central claim has two parts: raw discovery counts (103/68/23) and superiority over Boofuzz/Snipuzz/Mutiny. The strongest evidence is the CVE list, which is externally checkable, and the ablation (LSTE contributes 2.38x); those support a useful system. My concern is that the comparative superiority claim is not yet supported as stated because the experiments do not hold the detection oracle fixed. Section 5's command-injection ID embedding is a detection mechanism built into ChatHTTPFuzz; Section 6.3 says Boofuzz lacks command-injection mutation methods, so the comparison embeds the very contribution being evaluated into the counting rule. Table 7 has no Snipuzz or Mutiny columns, and Table 8 lists only named CVEs, so there is no per-device denominator for those baselines. The manuscript also contains numeric inconsistencies (59 vs 68 unique in the contributions/abstract; 22 vs 23 CVEs in Section 6.4/abstract; Table 10 lists 21 CVE rows; Table 6 has 15 devices while the abstract says 14). These reinforce the need for raw logs and a unified table but do not by themselves prove the result false. A controlled re-run with shared oracle and published raw logs is the decisive check; if it confirms ChatHTTPFuzz's advantage, the concern is settled. Therefore I keep the reader's conditional verdict rather than moving to rejection.","tokens_in":22723,"tokens_out":10330,"duration_ms":97580,"concrete_test":"Re-run Section 6.3 on the same devices with all four tools sharing a single detection harness: the same command-injection ID embedding, the same connection-reset/buffer-overflow classifier, the same seed templates, and the same payload dictionary including command-injection payloads. Publish per-device totals for every tool plus raw logs. If ChatHTTPFuzz no longer exceeds Boofuzz/Snipuzz/Mutiny on per-device counts, the central claim is an artifact of the oracle rather than the fuzzer; if it still does, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 ('Anomaly Monitoring') gives ChatHTTPFuzz a command-injection oracle that the baselines are not described as having: every mutated packet embeds a unique seed-template ID into the command, and device-side detection correlates executed commands with template IDs; buffer overflow is flagged via 'Connection reset'. Section 6.3 then credits Boofuzz's failure to find DNS-320L command injection to Boofuzz's lack of mutation methods for command injection, but that is exactly the kind of capability that LSTE and the custom oracle supply to ChatHTTPFuzz. The comparison in Table 7 (Boofuzz 6, Boofuzz-LSTE 25, ChatHTTPFuzz 81) therefore conflates fuzzer quality with oracle and seed-dictionary advantages. The Snipuzz and Mutiny comparisons are even less specified: no per-device vulnerability totals are given, and the text cites Table 7 for Mutiny even though Table 7 has no Mutiny column. Unless all tools are run with identical detection code, seeds, and mutation dictionaries, the headline superiority claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ChatHTTPFuzz, an LLM-guided black-box fuzzer for HTTP services in IoT devices. It uses GPT-4o to annotate variable fields in HTTP packets into seed templates, to generate and enrich seed templates from statically analyzed backend code, and to schedule templates via a dual-factor Thompson sampling algorithm. The authors evaluate the system on real IoT devices and report 103 discovered vulnerabilities, 68 previously undisclosed, and 23 assigned CVEs, claiming that ChatHTTPFuzz finds more vulnerabilities than Boofuzz, Snipuzz, and Mutiny.","tokens_in":22951,"tokens_out":6683,"duration_ms":61562,"significance":"If the empirical claims are substantiated, ChatHTTPFuzz would be a meaningful advance in automated IoT HTTP fuzzing: it demonstrates a concrete way to use LLM protocol understanding and code-guided seed generation to reduce invalid mutations and expand seed coverage. The approach is not circular: the scheduling formulas are defined a priori, and vulnerability detection is based on device behavior and CVE confirmation rather than on the LLM's own output. The categorized protocol-parsing study in Table 3 and the use of embedded template IDs for command-injection verification are concrete contributions. However, the current reporting does not yet allow independent verification of the headline superiority claim, so the paper's significance depends on fixes to the experimental presentation.","major_comments":[{"comment":"The comparative claim that ChatHTTPFuzz 'finds more vulnerabilities than SNIPUZZ, BOOFUZZ, and MUTINY' is not supported by the reported data. Table 7 contains columns only for Boofuzz, Boofuzz-LSTE, and ChatHTTPFuzz, yet the text states 'As shown in Table 7' when comparing Mutiny; Table 8 lists only 11 named CVEs with binary checkmarks and no per-tool per-device totals. No table or text reports total vulnerabilities found by Snipuzz and Mutiny per device, nor the time budget, seed set, number of runs, and detection criteria used for each tool. Without a unified per-device comparison under identical conditions, the superiority claim is unverifiable.","section":"§6.3, Tables 7 and 8"},{"comment":"The comparison conflates the fuzzer with its detection and seed-generation advantages. ChatHTTPFuzz detects command injection by embedding a unique seed-template ID into each command and correlating executed commands on the device, and flags buffer overflow via 'Connection reset'; the paper does not state that Boofuzz, Snipuzz, or Mutiny were given the same detection harness, the LLM-generated seed templates, or the mutation dictionaries. The discussion in §6.3 credits Boofuzz's failure to find the DNS-320L command injection to Boofuzz's lack of mutation methods, but that capability is supplied to ChatHTTPFuzz by LSTE and the custom oracle. The experiment should either provide the same oracle and seeds to all tools or be framed explicitly as an end-to-end comparison, not a fuzzer-quality comparison.","section":"§5, Anomaly Monitoring; §6.3"},{"comment":"The headline vulnerability counts are internally inconsistent. The abstract reports 103 vulnerabilities, 68 unique, and 23 CVEs; Section 1 reports 103 vulnerabilities, 59 unique, and 23 CVEs; Section 6.4 states 92 vulnerabilities, 59 undisclosed, and 22 CVEs; Table 9 totals 103, 68, and 23. Device lists also disagree: Table 6 lists 15 devices including Netgear WNAP320, while Table 9 lists 14 devices, replaces Cisco RV110W with RV100W, and omits WNAP320. These inconsistencies must be reconciled before the empirical claims can be assessed.","section":"§6.4 and Table 9"},{"comment":"The LLM capability evaluations rely on small self-constructed samples (140 packets in Table 3, 60 generated packets across three devices in Table 4, and 30 code samples in Table 5) with no confidence intervals, no description of how the samples were selected, and no independent test-set construction. If these numbers are intended to motivate the design, the paper should say so explicitly; if they are formal evaluations, the sampling procedure and variance should be reported.","section":"§3.1–3.2, Tables 3–5"}],"minor_comments":[{"comment":"The abstract contains the typo 'resent significant security risks'; it should read 'present significant security risks.'","section":"Abstract"},{"comment":"The text uses the spelling 'Snipfuzz' while the rest of the paper uses 'Snipuzz'; please make the tool name consistent throughout.","section":"§6.1"},{"comment":"The top-template fraction 'Top0.1N' and the Beta prior parameters are introduced without justification or sensitivity analysis; at minimum, report the values used in the experiments and test whether the qualitative results depend on them.","section":"Algorithm 2 and Equation (3)"},{"comment":"The text states that CVE-2024-7828 was rated high severity, but Table 10 lists it as 9.8/Critical; please reconcile this inconsistency.","section":"§6.4"},{"comment":"The Tenda AC15 version is listed as 'V15.03.05.19 multi', which is not a version identifier; please clarify what 'multi' refers to.","section":"Table 9"},{"comment":"The paper says '15 different IoT device models' in Section 6.2 and Table 6, while the abstract and conclusion say 14; please align the device count and the listed devices.","section":"§6.1 and §8"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope and the underlying idea is promising. The main obstacle is empirical reporting: the missing per-tool per-device data and inconsistent totals are fixable, but they are too central for acceptance now. I found no evidence of circularity; the scheduling formulas and detection criteria are defined independently of the claimed results. The authors should also state whether the LLM evaluation samples and experimental artifacts are available for independent inspection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on ChatHTTPFuzz: the system itself is plausible and the 23 CVEs on 14 real devices is genuinely useful evidence that LLM-guided seed generation helps in IoT HTTP fuzzing. But the paper's headline claim—that it finds more vulns than Snipuzz, Boofuzz, and Mutiny—is not actually established by the comparison as reported. The gaps are fixable, but they're real.\n\nWhat's new: the LVA (LLM variable annotation) plus LSTE (seed template enrichment from code) plus STSA (dual-factor Thompson sampling) combo is a reasonable engineering contribution. The ablation shows LSTE matters a lot (2.38x vs NoLSTE), and the scheduling factor halves time to find the same vulns. The parsing checks (140 packets, 98.58% field recognition on 60 packets) are small but demonstrate the LLM can do the job. The CVEs, including critical command injections on D-Link DNS-320L, are externally verifiable and give the work credibility.\n\nSoft spots, in order of severity:\n\n1. Unequal oracle. Section 5 gives ChatHTTPFuzz a command-injection detection mechanism: it embeds a seed template ID in every mutated command and correlates execution. Boofuzz is then blamed for not finding command injection because it lacks mutation methods. That's not a fair comparison—the oracle and the seed dictionaries are part of the system under test, and baselines don't get them. The superiority claim conflates fuzzer quality with oracle and seed advantages.\n\n2. Missing per-device totals for Snipuzz and Mutiny. Table 8 only shows ✓/× for a handful of CVEs, and the text cites Table 7 for Mutiny even though Table 7 has no Mutiny column. There's no unified table of total vulns per tool per device, so the \"13 times more\" claim can't be checked.\n\n3. Internal number inconsistencies: abstract says 103 vulns / 68 unique / 23 CVEs; intro says 103 / 59 / 23; Section 6.4 says 92 / 59 / 22; conclusion says 68 zero-day / 23 CVEs. This looks like sloppy restatement, not necessarily fraud, but it needs correcting.\n\n4. No artifacts released. For a tool that's meant to be used, that's a setback; the CVEs are nice but reproducibility is limited.\n\nOverall: this is a real system with real findings, and the direction is worth pursuing. As it stands, the comparative evaluation needs rework before the main claim holds. I'd send it to peer review, but with a clear ask: provide full per-tool per-device numbers, equalize oracles across tools, and reconcile the totals. A serious referee could make this a solid paper.\n\nRecommendation: accept for review, expect major revision.","headline":"Solid LLM-fuzzing system with real CVE finds, but the superiority claim over baselines is undercut by unequal oracles and sloppy reporting.","tokens_in":23466,"tokens_out":2319,"would_cite":false,"duration_ms":22344,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A large language model can guide HTTP fuzzing of IoT devices, reporting 103 vulnerabilities, 68 previously undisclosed, 23 with CVEs.","keywords":["IoT fuzzing","HTTP protocol","large language models","seed template generation","vulnerability discovery","command injection","memory corruption","CVE"],"falsifier":"Run the proposed fuzzer and the three comparison fuzzers from the paper on the same devices with the same session setup, time budget, seed traffic, and criteria for counting a vulnerability. If the comparison fuzzers collectively find at least as many unique vulnerabilities, or if the reported CVEs cannot be reproduced, the headline outperformance claim is not supported.","tokens_in":22548,"feed_emoji":"🐛","tokens_out":8541,"duration_ms":73738,"temperature":0.7,"pith_summary":"The paper tries to show that large language models can remove the two bottlenecks of black-box HTTP fuzzing on IoT devices: understanding the HTTP protocol structure and having enough good starting seeds. It proposes ChatHTTPFuzz, which uses an LLM to label mutable fields in captured HTTP traffic, read decompiled backend code to generate additional packets and field values, and schedule seed templates with a modified Thompson sampling algorithm. Tested on 14 real IoT devices, the system reports 103 vulnerabilities, 68 previously undisclosed, and 23 assigned CVEs, and says it finds more than three existing fuzzers. If true, this means a fuzzer can be pointed at a device's web interface and produce protocol-valid, code-aware test cases without hand-written templates.","feed_headline":"LLM-guided fuzzing finds 103 IoT HTTP flaws","feed_subtitle":"ChatHTTPFuzz parses HTTP fields and backend code to generate protocol-compliant test cases; 23 flaws already have CVEs.","key_machinery":"The seed template is the central object: an HTTP packet skeleton with variable fields marked, typed, and paired with a mutation space of LLM-inferred legal values. Three mechanisms carry the argument: LLM-guided variable annotation identifies mutable fields with zero false negatives on the paper's 603-parameter corpus; the LLM-guided seed template enrichment algorithm analyzes backend code to generate new packets and expand field value sets; and the dual-factor gain scheduling algorithm combines Thompson sampling with an exploration-balance factor and a mutation-potential factor to choose templates. The load-bearing capability is the LLM's translation of code logic into valid HTTP values, which the paper measures at 98.58% field recognition.","core_discovery":"ChatHTTPFuzz is an LLM-guided black-box fuzzer for IoT HTTP services. It treats an LLM as a protocol-aware seed generator: given captured HTTP requests, it annotates every variable field with type and encoding, builds seed templates that preserve fixed protocol elements, then uses decompiled service code to infer new routes, branch-triggering values, and format constraints such as sscanf patterns and Base64 encoding. Mutations are applied only to labeled variable fields, with type-aware operators and context-specific values. A dual-factor gain Thompson sampling scheduler scores templates by historical reward, invocation frequency, and mutation-space size. The paper's claim is that this combination finds 103 vulnerabilities across 14 devices, 68 of them undisclosed and 23 CVE-assigned, and outperforms the three tested black-box fuzzers.","pith_inferences":["If the advantage is concentrated in the seed-enrichment step, attaching LLM-guided code-aware seed generation to existing template fuzzers could reproduce much of the gain; the paper's augmented-baseline experiment points in that direction.","The same field-annotation and code-branch prompts should transfer to other structured protocols with similar backend routing, such as RTSP, SNMP, or Matter, so the method is not inherently HTTP-specific.","The concentration of critical CVEs in a few device families suggests the fuzzer surfaces systematic classes of command-injection bugs in a vendor's codebase, not only isolated flaws.","A cost question the paper leaves open is how much of the result depends on the specific LLM and on manual reverse-engineering of routing tables; substituting a cheaper local model or automating that extraction would be a testable variation."],"forward_implications":["On the 14 devices tested, ChatHTTPFuzz reports 103 vulnerabilities, of which 68 are previously undisclosed and 23 have been assigned CVE identifiers.","The seed-enrichment step is the main driver: removing it drops discovered vulnerabilities from 57 to 24 across four devices, about a 2.38x reduction.","The scheduling algorithm does not add many vulnerabilities but halves the time needed to reach the same number of detections over 24 hours.","LLM annotation of HTTP fields can be automated with near-zero false negatives on standard and custom parameter formats, removing the need to hand-write protocol templates.","Protocol-aware mutation avoids corrupting fixed HTTP elements, so more generated packets remain parseable by the target service."],"supporting_citations":[{"why":"Supplies the GPT-4o API that performs all HTTP field annotation, code analysis, and packet generation.","marker":"OpenAI, 2024"},{"why":"Defines RFC 2616, the HTTP/1.1 specification from which the 140 parameter-transmission test packets are constructed.","marker":"Hypertext Transfer Protocol, 1995"},{"why":"Describes Snipuzz, the black-box IoT fuzzer whose message-snippet inference is a baseline and a contrast for mutation effectiveness.","marker":"Feng et al, 2021"},{"why":"Describes Boofuzz, the template-based mutation fuzzer used as a baseline and as the base for the augmented-seed comparison.","marker":"Jtpereyda, Last updated 2024"},{"why":"Describes Mutiny, the replay-and-mutate fuzzer used as a baseline.","marker":"Cisco, 2017"},{"why":"Provides the Thompson sampling formulation that the dual-factor seed scheduling algorithm extends.","marker":"Karamcheti et al, 2018"},{"why":"Prior LLM-guided protocol fuzzing that motivates using LLM parsing and generation to guide mutations.","marker":"Meng et al, 2024"}],"fun_headline_variants":["LLM-powered fuzzer finds 103 IoT HTTP vulnerabilities","ChatHTTPFuzz: LLM-guided IoT fuzzing yields 103 bugs, 23 CVEs","103 IoT HTTP flaws exposed by LLM-assisted fuzzing","LLM-based fuzzer uncovers 103 IoT vulnerabilities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that ChatHTTPFuzz beats the comparison fuzzers depends on those fuzzers being tested on the same devices, with the same time budget, seeds, and rules for counting a bug; the paper publishes a full device-by-device count for only one of the three.","fun_headline_variants_meta":{"raw":{"variants":["LLM-powered fuzzer finds 103 IoT HTTP vulnerabilities","ChatHTTPFuzz: LLM-guided IoT fuzzing yields 103 bugs, 23 CVEs","103 IoT HTTP flaws exposed by LLM-assisted fuzzing","LLM-based fuzzer uncovers 103 IoT vulnerabilities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2326,"prompt_tokens":1026,"completion_tokens":1300,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":1220}},"tokens_in":642,"tokens_out":1300,"duration_ms":31971,"temperature":1.0,"reasoning_tokens":1220,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:29:38.134877+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the proposed fuzzer and the three comparison fuzzers from the paper on the same devices with the same session setup, time budget, seed traffic, and criteria for counting a vulnerability. If the comparison fuzzers collectively find at least as many unique vulnerabilities, or if the reported CVEs cannot be reproduced, the headline outperformance claim is not supported.","supporting_citations":[{"cited_title":"https://platform.openai.com/docs/models/gpt-4o, accessed: 28-Oct-2024","cited_arxiv_id":null,"evidence_quote":"Supplies the GPT-4o API that performs all HTTP field annotation, code analysis, and packet generation."},{"cited_title":"Available at: https://www.w3.org/Protocols/rfc2616/rfc2616.html","cited_arxiv_id":null,"evidence_quote":"Defines RFC 2616, the HTTP/1.1 specification from which the 140 parameter-transmission test packets are constructed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes Snipuzz, the black-box IoT fuzzer whose message-snippet inference is a baseline and a contrast for mutation effectiveness."},{"cited_title":"https://github.com/jtpereyda/boofuzz","cited_arxiv_id":null,"evidence_quote":"Describes Boofuzz, the template-based mutation fuzzer used as a baseline and as the base for the augmented-seed comparison."},{"cited_title":"https://github.com/Cisco-Talos/mutiny-fuzzer","cited_arxiv_id":null,"evidence_quote":"Describes Mutiny, the replay-and-mutate fuzzer used as a baseline."},{"cited_title":"Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security pp 37--47","cited_arxiv_id":null,"evidence_quote":"Provides the Thompson sampling formulation that the dual-factor seed scheduling algorithm extends."},{"cited_title":"In: Proceedings 2024 Network and Distributed System Security Symposium","cited_arxiv_id":null,"evidence_quote":"Prior LLM-guided protocol fuzzing that motivates using LLM parsing and generation to guide mutations."}],"review_version":1}