{"id":"bec1b57b-7708-4753-aae4-6eb7c74bf753","arxiv_id":"2607.07007","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":3,"one_line_summary":"SynapseFlow uses dataflow-based function grouping and a staged rollback LLM workflow to auto-generate fuzz harnesses, achieving higher coverage and bug detection than prior tools on 25 C projects.","lead":"This paper presents SynapseFlow, a tool that automatically generates fuzz harnesses for C programs by combining dataflow-aware function grouping with a staged, rollback-capable LLM generation workflow. It matters because automated, high-quality harness generation reduces the manual effort required to find memory-safety bugs in widely-used open-source software.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"Coverage comparison confounds harness count with harness quality: SynapseFlow generates 3–13× more harnesses than baselines per project, inflating aggregate coverage ratios.","rationale":"The reader's weakest assumption (rollback sufficiency) is a valid methodological concern but is addressed by the ablation study (-SDR causes 46.7% drop) and the low average stage count (7.78). The more load-bearing concern is the coverage comparison confound: the headline coverage ratios aggregate across vastly different numbers of harnesses (up to 13× more for SynapseFlow), and the baselines operate over narrower function scopes by design. This means the '3.07× higher branch coverage' claim conflates scope expansion with harness quality improvement. However, this concern does not change the overall verdict because: (1) the bug detection comparison (RQ2) is properly controlled (10 harnesses per bug function per tool) and SynapseFlow still wins 1.77×/1.51×/1.36×; (2) the discovery of 7 previously unreported bugs with 5 CVEs is strong independent evidence of practical utility that cannot be explained by harness count alone; (3) the ablation study demonstrates both SFG and SDR contribute independently within SynapseFlow's own framework. The concern weakens the precision of the coverage headline but does not invalidate the core contribution. The paper would be strengthened by reporting per-harness normalized coverage or explicitly disclosing the scope difference in the headline numbers. Verdict remains ACCEPT, but confidence in the coverage-specific claims should be tempered.","tokens_in":32833,"tokens_out":3423,"duration_ms":110601,"concrete_test":"Compute per-harness average branch coverage (total covered branches / #harnesses from Table 11) for each tool on each project. Then re-compute the coverage ratios. Additionally, run a controlled experiment: cap each tool to generate the same number of harnesses (e.g., match the minimum across tools per project) and re-measure aggregate coverage. If SynapseFlow's per-harness coverage is not significantly higher than baselines, the '3.07×' headline is primarily a scope artifact, not a quality improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim of '3.07×, 1.71×, 4.26× higher branch coverage' aggregates branch coverage across ALL harnesses each tool generates per project (§4.1.1: 'we aggregated the coverage data from all harnesses'). Table 11 reveals that SynapseFlow generates far more harnesses than baselines—e.g., sqlite3: 603 vs 45/112/104; cjson: 150 vs 58/40/64; libtiff: 353 vs 128/153/128. This is structural: SynapseFlow targets all functions via FT extraction (Table 3: 'All Funcs'), while OSS-Fuzz-Gen targets single functions and CKGFuzzer/PromeFuzz target API-only functions. The coverage ratio is therefore confounded by scope: more harnesses over a broader function set mechanically yields higher aggregate coverage regardless of per-harness quality. The paper acknowledges this scope difference (§4.1.3: 'CKGFuzzer's API-only focus and OSS-Fuzz-Gen's single-function approach further restrict their achievable coverage') but the headline numbers do not disclose it. Per-harness coverage efficiency may actually be lower for SynapseFlow: for sqlite3, 28063 branches / 603 harnesses ≈ 47 branches/harness vs OSS-Fuzz-Gen's 16879/45 ≈ 375 branches/harness. The ablation (Table 9, SF-PF) shows SFG contributes 29.6% of coverage, but this isolates SFG vs heuristic grouping within SynapseFlow's own scope—it does not control for the scope difference against baselines. The bug detection comparison (RQ2) IS controlled (10 harnesses per bug function per tool), and SynapseFlow still wins there, which partially mitigates the concern. But the coverage headline remains the most prominent quantitative claim and is the least controlled comparison in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper presents SynapseFlow, an LLM-based automatic fuzz harness generator for C projects. The approach has two phases: (1) dataflow-aware function aggregation, which constructs a Structural Flow Graph (SFG) and extracts Function Triplets (FTs) — minimal, coherent function groups anchored by Input Stream Functions (ISFs); and (2) harness generation via a four-stage decomposition (documentation, snippet stitching, rough assembly, optimization) governed by a staged rollback algorithm. The evaluation covers 25 real-world C projects, comparing against OSS-Fuzz-Gen, CKGFuzzer, and PromeFuzz on branch coverage, bug detection, and efficiency, with an ablation study isolating the SFG and staged rollback contributions. The paper reports 7 previously unreported bugs (5 CVEs). The core technical ideas — dataflow-grounded function grouping and decomposed generation with rollback — are well-motivated and the experimental design is largely rigorous, including the controlled bug-detection comparison (RQ2) and the ablation (RQ6). However, the headline coverage comparison (RQ1) conflates per-harness quality with harness quantity, which undermines the central comparative claim and must be addressed.","tokens_in":33879,"tokens_out":1596,"duration_ms":2006271,"significance":"Automated harness generation is a practically important problem, and the paper's combination of lightweight static dataflow analysis with LLM-driven semantic reasoning and staged rollback is a reasonable and novel architectural contribution. The discovery of 7 previously unreported bugs (5 CVEs) on extensively fuzzed projects is a strong practical result. The ablation study (Table 9) properly isolates the SFG and staged decomposition components within SynapseFlow's own scope. The theoretical proof in Appendix A, while simple, provides a principled (if idealized) justification for the rollback mechanism. The artifact availability (encrypted executable, scripts, generated harnesses) supports reproducibility, though full source code is withheld pending acceptance.","major_comments":[{"comment":"§4.1.1–4.1.3, Table 4, Table 11: The headline coverage comparison aggregates branch coverage across ALL harnesses each tool generates per project. Table 11 reveals that SynapseFlow generates 3–13× more harnesses than baselines (e.g., sqlite3: 603 vs. 45/112/104; cjson: 150 vs. 58/40/64; libtiff: 353 vs. 128/153/128). This is structural: SynapseFlow targets all functions via FT extraction (Table 3), while OSS-Fuzz-Gen targets single functions and CKGFuzzer/PromeFuzz target API-only functions. The coverage ratio is therefore confounded by scope. The paper acknowledges this scope difference in §4.1.3 but the headline numbers ('3.07×, 1.71×, 4.26× higher branch coverage') do not disclose it. Per-harness coverage efficiency may actually be lower for SynapseFlow: for sqlite3, 28063 branches / 603 harnesses ≈ 47 branches/harness vs. OSS-Fuzz-Gen's 16879/45 ≈ 375 branches/harness. The authors' R","section":null},{"comment":"§4.1.3: The paper states that baselines 'achieve high function counts but fail to exercise internal control-flow paths' because they perform 'superficial invocations without properly routing external fuzz inputs.' This is a strong claim about per-harness quality. If true, it should be demonstrable via a per-harness or per-function-coverage-normalized metric. The current presentation does not provide this. Adding a normalized comparison (e.g., median branches covered per harness, or coverage restricted to the intersection of functions targeted by all tools) would either confirm the authors' explanation or reveal that the aggregate advantage is primarily a scope effect. This is load-bearing because the paper's framing positions SynapseFlow as generating higher-quality harnesses, not merely more harnesses.","section":null}],"minor_comments":[{"comment":"The title in the manuscript body ('Thinking More, Harnessing Better: Automatic Harness Generation with Dataflow Aggregation and Workflow Decomposition') differs from the title in the paper_id metadata ('State Machine Guided Harness Automatic Generation with Project Digestion and Workflow Decomposition'). These should be reconciled.","section":null},{"comment":"Abstract states '5 assigned CVEs' and §1 states '4 assigned CVEs'; Table 6 lists 5 CVE IDs. The inconsistency in §1 should be corrected.","section":null},{"comment":"Appendix A, Eqs. (1)–(3): The proof assumes stage independence (P_i) and a compensation success probability c that is introduced without derivation. The result P_rollback > P_seq > P_o is straightforward under these assumptions but the independence assumption is strong. A brief acknowledgment that real stages are not independent would improve honesty.","section":null},{"comment":"§3.2.1: The three-prompt voting scheme is described, but no empirical validation of its accuracy (e.g., precision/recall on a labeled subset) is provided. A sentence or two of evidence would strengthen the claim that it 'significantly improves accuracy.'","section":null},{"comment":"Table 4: The 'Aver. Rate(%)' row shows 46.9, 55.0, 61.3, 83.3. It is unclear whether this is a simple mean across projects, a coverage-weighted average, or a harness-count-weighted average. The caption should specify.","section":null},{"comment":"Figure 6: The x-axis labels appear to be corrupted/garbled (unicode escape sequences). Project names are unreadable. This figure needs to be regenerated.","section":null},{"comment":"§5.3: 'Limitations for Stateful Targets' and the preceding paragraph on 'Semantic Validation' overlap significantly. These should be merged.","section":null},{"comment":"Table 3: The 'Feedback' row lists SynapseFlow as 'Iterative' and baselines as 'None.' The staged rollback is iterative in the sense of retrying stages, but it is not feedback-driven in the sense of incorporating runtime coverage feedback into generation. The label could be more precise (e.g., 'Retry-based' or 'Compilation-guided').","section":null},{"comment":"§4.4.3: The claim that 'excessive prompt length induces quadratic scaling in Time-To-First-Token' is stated without citation. A reference or a brief empirical justification (e.g., measured TTFT vs. prompt length) would suffice.","section":null}],"recommendation":"major_revision","confidential_remarks":"The skeptic's concern about the coverage comparison is well-founded and is the primary reason for the major revision recommendation. The bug detection comparison (RQ2) is properly controlled (10 harnesses per bug function per tool) and SynapseFlow wins there, which partially mitigates the concern — but the headline coverage numbers are the most prominent claim in the abstract and §1, and they are confounded by harness count. The fix is straightforward: add a normalized per-harness or per-target-function coverage metric. If SynapseFlow still wins on that metric, the paper's central claim is strengthened; if not, the framing should be adjusted to emphasize bug detection and breadth-of-scope rather than per-harness quality. Either outcome is publishable, but the current framing is not fully supported by the evidence as presented."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The paper you sent me—SynapseFlow—is worth your time. The core idea is to replace monolithic LLM harness generation with a two-phase pipeline: first, build a Structural Flow Graph from source code to group functions into coherent triplets (input handler, processors, lifecycle managers), then generate harnesses through a four-stage decomposition with rollback. They found 7 previously unreported bugs, 5 with CVEs assigned. That's the real evidence of practical utility, and it's hard to fake. The ablation study (RQ6) properly isolates the SFG and staged rollback contributions, and the bug detection comparison (RQ2) is controlled—10 harnesses per bug function per tool, same backend LLM everywhere. That's good experimental hygiene. The staged rollback idea is sensible and the low average stage count (7.78 vs ideal 4) shows rollbacks stay shallow in practice. The theoretical proof in Appendix A is lightweight but not wrong—just a probability argument that decomposition with retries beats single-shot generation, which is straightforwardly true. What's genuinely new is the specific integration: dataflow-driven function grouping feeding into a decomposed generation workflow with rollback. Neither piece is novel in isolation, but the combination is well-motivated and the execution is thorough. Now the soft spot, and it's a real one. The headline coverage numbers—3.07×, 1.71×, 4.26× over baselines—are confounded. SynapseFlow generates far more harnesses per project than any baseline (sqlite3: 603 vs 45/112/104; cjson: 150 vs 58/40/64). They aggregate branch coverage across all harnesses each tool produces, so broader scope mechanically inflates the ratio. Per-harness, SynapseFlow may actually cover fewer branches than OSS-Fuzz-Gen on some projects. The paper acknowledges the scope difference in passing but the headline numbers don't disclose it. The bug detection comparison is properly controlled and SynapseFlow still wins there, which partially mitigates the concern. But the coverage headline is the most prominent claim and the least controlled comparison. This is fixable in revision—report per-harness coverage or normalize by harness count—but it needs to be addressed. The reproducibility situation is also not great: encrypted executable for review, partial prompt listing, no full source code yet. Minor relative to the confound. Overall: a solid paper with a real contribution and real bugs found, but one load-bearing quantitative claim needs reframing before publication. Deserves a serious referee.","headline":"Solid LLM harness generator with real bugs found, but headline coverage numbers are confounded by harness count","tokens_in":33924,"tokens_out":591,"would_cite":true,"duration_ms":134097,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Decomposed LLM harness generation finds 7 new bugs in fuzzed C projects","keywords":["fuzz harness generation","LLM code generation","dataflow analysis","structural flow graph","staged rollback","function triplet","gray-box fuzzing","C program security"],"falsifier":"If one replaced the SFG-based function grouping with random or call-graph-based grouping while keeping the staged rollback pipeline identical, and coverage dropped by the ~30% the ablation reports, then the SFG — not the rollback — is the primary coverage driver. Conversely, if one kept the SFG but replaced the four-stage pipeline with monolithic generation and coverage dropped by ~47%, then the staged decomposition is the primary driver. The ablation already tests both, and the results suggest both contribute substantially, with staged decomposition contributing more.","tokens_in":32919,"feed_emoji":"🐛","tokens_out":1640,"duration_ms":105751,"temperature":0.7,"pith_summary":"The paper claims that the quality bottleneck in LLM-generated fuzz harnesses is not the model's raw coding ability but two upstream failures: picking the wrong functions to test together, and asking the model to write an entire harness in one shot. SynapseFlow addresses both. First, it builds a Structural Flow Graph (SFG) — a directed graph whose nodes are data-structure types and whose edges are functions that consume one structure and emit another. By traversing this graph, the system extracts Function Triplets: minimal, self-contained groups anchored by a single input-stream function, plus the process functions and lifecycle helpers that share its data structures. This replaces ad-hoc function selection with dataflow-grounded grouping. Second, it decomposes harness synthesis into four sequential stages — API documentation, local snippet stitching, rough assembly, and final optimization — each individually verifiable. A staged rollback algorithm retries failed stages from earlier checkpoints rather than restarting the whole generation, which the authors prove raises compound success probability above both single-shot and linear-retry strategies. Evaluated on 25 real-world C projects, harnesses generated this way achieve 1.7–4.3× higher branch coverage and 1.4–1.8× higher bug-detection rates than three state-of-the-art tools, and surfaced 7 previously unreported bugs (5 CVE-assigned) in projects already under continuous fuzzing.","feed_headline":"Dataflow-guided LLM harness generation finds 7 new bugs in fuzzed C projects","feed_subtitle":"By grouping functions through shared data structures and decomposing generation into rollback-enabled stages, harnesses cover 1.7–4.3× more","key_machinery":"Structural Flow Graph (SFG): directed graph where nodes are structure types (or a null node) and edges are functions transforming one structure into another. Function Triplet (FT): an ordered triplet (I, P, H) — one Input Stream Function, a set of Process Functions, a set of Helper Functions — extracted per ISF from the SFG via forward/backward dataflow reachability. Staged rollback algorithm: a four-stage pipeline (documentation, snippet stitching, rough assembly, optimization) where stage-4 failures trigger incremental rollback to stage 3, then stage 2, then stage 1, with a theoretical proof that compound success probability exceeds both single-shot and linear-retry generation.","core_discovery":"The central object is the Structural Flow Graph: a representation of inter-procedural data dependencies via shared structure types, not call edges. The paper shows that when function grouping for harness generation is grounded in this graph rather than in call-graph proximity or API-level heuristics, the resulting harnesses correctly channel fuzzer inputs through deep data-transformation chains that prior methods systematically miss. Combined with staged decomposition and rollback — which the authors formally show improves compound success probability over monolithic generation — this yields both broader coverage and lower hallucination rates. The 7 new bugs found in continuously fuzzed, CVE","pith_inferences":["The SFG's reliance on tree-sitter for syntax parsing (rather than full dataflow analysis) means the approach may degrade on projects with heavy macro usage or indirect function dispatch — the paper acknowledges this for lcms and dropbear. A natural extension would be to substitute a more precise parser for the SFG construction step while keeping the downstream pipeline unchanged, potentially recov","The rollback algorithm assumes intermediate-stage outputs are not themselves the source of semantic error. If the SFG or FT extraction contains a structural flaw (e.g., misclassifying an ISF), rolling back to regenerate code from the same flawed triplet will not escape the error — the system would need a feedback path that revisits the grouping phase, which the current architecture does not provid","The three-function-category model (ISF/PRF/HPF) is sufficient for C data-processing libraries but may not generalize cleanly to C++ object-oriented initialization patterns or stateful protocol implementations where initialization is spread across constructors and state machines — the paper explicitly flags these as out of scope, suggesting the taxonomy itself is a load-bearing design choice that l"],"forward_implications":["If dataflow-grounded function grouping is the key driver of coverage gains, then any fuzz-harness generator — LLM-based or not — should benefit from constructing an SFG before selecting targets, suggesting the graph construction step is separable and composable with other generation backends.","The staged rollback formalism (Equations 1–3) is domain-agnostic: any multi-step LLM code-generation pipeline with per-stage validation could adopt the same rollback strategy to improve compound success rates, potentially extending beyond fuzzing to test generation, refactoring, or migration tasks.","The finding that 7 new bugs were found in projects under years of continuous fuzzing suggests that existing fuzzing infrastructure has systematic blind spots around functions outside standard API target sets — and that dataflow-guided target selection can systematically surface them."],"fun_headline_variants":["Structural flow graphs guide LLM harness generation to 7 new bugs in fuzzed projects","Dataflow-aware harness generation with staged rollback outperforms three SOTA fuzz tools","Grounding harness generation in structural flow graphs yields 3–4× broader branch coverage","Staged rollback in LLM harness synthesis cuts hallucination and finds 5 CVEs in fuzzed C","Shared-structure function grouping for LLM harnesses beats call-graph heuristics on covera"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The staged rollback mechanism assumes that when a final harness fails compilation or basic execution, the error originates in a later generation stage and can be fixed by regenerating from an earlier stage's output. If the root cause is a structural flaw in the dataflow graph or function triplet itself — for example, a function misclassified as an input-stream function — then rolling back and regenerating code from the same flawed grouping will reproduce the same class of错误.","fun_headline_variants_meta":{"raw":{"variants":["Structural flow graphs guide LLM harness generation to 7 new bugs in fuzzed projects","Dataflow-aware harness generation with staged rollback outperforms three SOTA fuzz tools","Grounding harness generation in structural flow graphs yields 3–4× broader branch coverage","Staged rollback in LLM harness synthesis cuts hallucination and finds 5 CVEs in fuzzed C","Shared-structure function grouping for LLM harnesses beats call-graph heuristics on coverage"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":717,"prompt_tokens":602,"completion_tokens":115,"prompt_tokens_details":null},"tokens_in":602,"tokens_out":115,"duration_ms":45585,"temperature":1.0,"reasoning_tokens":null,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T21:43:45.070962+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If one replaced the SFG-based function grouping with random or call-graph-based grouping while keeping the staged rollback pipeline identical, and coverage dropped by the ~30% the ablation reports, then the SFG — not the rollback — is the primary coverage driver. Conversely, if one kept the SFG but replaced the four-stage pipeline with monolithic generation and coverage dropped by ~47%, then the staged decomposition is the primary driver. The ablation already tests both, and the results suggest both contribute substantially, with staged decomposition contributing more.","supporting_citations":[],"review_version":1}