{"id":"761762ed-3bfc-4dcc-ad9a-6a032b31c2f6","arxiv_id":"2505.06177","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Using eight years of OSS-Fuzz data, the study shows that C/C++ fuzz harnesses keep stable coverage and bug-finding ability while they build, with about 5% of harness versions dropping by 5% coverage within six months.","lead":"This paper measures how well fuzzing harnesses in Google's OSS-Fuzz keep working as the software projects they test evolve. It finds that coverage stays surprisingly stable for most harnesses that still build, while about 5% degrade noticeably within half a year, and it ships detection metrics and a dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no explicit updates' longevity result hinges on a harness-change heuristic with an unmeasured false-negative rate; missed updates could masquerade as unmaintained harnesses.","rationale":"Good-faith reading: the paper is a careful, large-scale empirical study with public artifacts, manual case studies, and an unusually honest threats-to-validity section. The central claim is explicitly scoped to harnesses that still build, so excluding completely broken harnesses is not internally inconsistent; the paper also cites prior work on build failures, making the limitation transparent. The coverage-denominator and OOM/timeout artifacts are real, but the authors manually categorize 308 degradation instances and their RQ4 taxonomy shows that many apparent drops are measurement artifacts; if anything, those artifacts inflate the observed degradation rate rather than the observed stability. The bug-finding analysis is supported by crash-revision data and is consistent with prior work on regression bugs. The weakest point is construct validity of the independent variable: 'explicit update' as detected by the §3.1.4 heuristic. The heuristic is conservative by design, but the false-negative rate is never measured. The manual check for suspicious gaps only detects long silent periods in relatively active projects; it cannot detect frequent, small missed updates in projects with regular commit activity. Because RQ2 and RQ3 define harness lifetime and 'time since last update' entirely from this heuristic, a systematic false-negative bias would directly produce the paper's headline: unmaintained harnesses would appear to survive longer, keep coverage longer, and keep finding bugs longer than they actually do. This is a one-directional threat to the central claim, unlike the coverage artifacts which cut both ways. The concern does not invalidate the paper's contributions—the dataset, the RQ4 case studies, and the proposed detection metrics are useful regardless—but it should be settled before the 'surprising longevity' statement is taken as established. Conditional acceptance remains the right verdict; the required validation is a sensitivity analysis or manual annotation sample of the harness-update heuristic, which is feasible and low-cost given the released artifacts.","tokens_in":19860,"tokens_out":5362,"duration_ms":52356,"concrete_test":"Select a stratified random sample of about 50 projects and about 200 intervals of at least four weeks that the paper classifies as containing no harness update. Manually inspect every commit in the project repository and the OSS-Fuzz repository during those intervals, and classify each commit as harness-relevant if it changes (a) build configuration affecting fuzzer targets, flags, or linked dependencies; (b) dependency versions or vendored code used by the harness; (c) signatures or semantics of any function reachable from LLVMFuzzerTestOneInput; or (d) the harness file itself under a name or path the §3.1.4 heuristic would miss. Compute the false-negative rate of the heuristic against this manual annotation, ideally with two annotators and inter-rater agreement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—surprising longevity of bug-finding capability even without explicit updates, as long as harnesses still build—is an argument about unmaintained harnesses, so the definition of 'explicit update' in §3.1.4 is load-bearing. That heuristic counts a commit as a harness change only if it touches an OSS-Fuzz file other than project.yaml/Dockerfile/Jenkins, or a C/C++ file whose path contains 'fuzz' (excluding 'fuzzy') or that defines LLVMFuzzerTestOneInput/LLVMFuzzerInitialize. This can miss update modes that keep coverage and bug-finding alive: changes to project-repo build files (CMakeLists.txt, Makefile) that add or alter fuzz targets; dependency version bumps or vendored-code updates that change what the harness links against; changes to headers or function signatures reachable from the harness but whose filenames do not contain 'fuzz'; harnesses moved to files without 'fuzz' in the path or using non-standard entry-point macros; and updates committed to separate fuzzing repositories beyond the two manually detected projects. The manual check for 'suspicious gaps' only catches long silent periods in relatively active projects; it does not quantify false negatives in projects with frequent commits. If such missed updates are common, periods labeled 'since last harness update' actually contain maintenance, so the flat coverage and sustained bug rates in Figures 3–5 would be expected even if unmaintained harnesses do degrade. Since RQ2 and RQ3 define harness lifetime entirely by this heuristic, this is the least secure link in the argument. The coverage-measurement artifacts of §3.5.3–3.5.4 are a second reliability concern, but they bias in both directions and are partially addressed by manual categorization; the update-detection bias is one-directional and directly targets the 'without explicit updates' clause.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale longitudinal study of fuzz harness degradation in Google's OSS-Fuzz platform. Using coverage reports, commit histories, Monorail bug data, and Fuzz Introspector reports for 433 C/C++ projects and 29,019 harness versions over 2016–2024, the authors address five research questions on the immediate effects of harness updates, the rate of coverage degradation over time, the degradation of bug-finding capability, the causes of coverage drops, and practical detection metrics. The central findings are that overall coverage in OSS-Fuzz remains stable, that harness updates have only a small mean effect on coverage (0.26%), that bug-finding capability does not automatically decline for harnesses that still build, that approximately 5% of harness versions lose at least 5 percentage points of coverage within half a year, and that manual analysis of 308 coverage-drop cases yields four main causes: partial build failures, added project code or churn, third-party code polluting coverage reports, and corpus-size decreases caused by OOM/timeout measurement artifacts. The authors also report contributions to OSS-Fuzz and Fuzz Introspector in the form of new detection metrics.","tokens_in":20120,"tokens_out":7259,"duration_ms":72836,"significance":"This is, to my knowledge, the first systematic longitudinal study of fuzz harness degradation at the scale of a major continuous fuzzing platform. If the findings hold, they have practical value: they suggest that harnesses in mature OSS-Fuzz projects are more resilient than often assumed, while also identifying concrete, actionable failure modes. The paper is unusually transparent: the dataset, scraping code, analysis notebook, and case-study notes are public, the data-cleaning steps are documented in detail, and the 308 degradation cases were manually reviewed with category counts provided. The proposed metrics, if integrated, could help maintainers detect silent degradation. However, the significance is conditional on two methodological pillars: the heuristic that identifies harness-update events, and the integrity of the coverage reports used to measure stability. Both are explicitly acknowledged as imperfect in the manuscript, but the magnitude of their impact on the central claims is not quantified.","major_comments":[{"comment":"The harness-change heuristic is load-bearing for the 'no explicit updates' longevity claim, and its false-negative rate is unmeasured. The heuristic counts a commit as a harness change only if it touches an OSS-Fuzz file other than project.yaml/Dockerfile/Jenkins, or a C/C++ file whose path contains 'fuzz' (excluding 'fuzzy') or that defines LLVMFuzzerTestOneInput/LLVMFuzzerInitialize. This misses several plausible update modes: changes to project build files (CMakeLists.txt, Makefile) that add or alter fuzz targets; dependency version bumps or vendored-code updates that change what the harness links against; header or signature changes in files whose names do not contain 'fuzz'; harnesses moved to files outside the matching paths; and commits to separate fuzzing repositories beyond the two projects manually detected. The manual 'suspicious gaps' check described in §3.1.4 can only catch long silent periods in relatively active projects; it does not provide a quantitative false-negative rate for projects with frequent commits. Because RQ2 and RQ3 define 'time since last harness update' based entirely on this heuristic (Figures 3–5), a non-trivial false-negative rate would mean that many periods labeled 'unmaintained' actually contain maintenance, in which case the observed flat coverage and sustained bug rates would be expected even if unmaintained harnesses do degrade. I request a validation of the heuristic against a statistically sampled subset of manually labeled commits, or an alternative confirmation using OSS-Fuzz build logs to establish when harness files were actually built or changed.","section":"§3.1.4, Figs. 3–5"},{"comment":"Coverage measurement artifacts are not excluded from the RQ2 stability analysis, even though the paper's own evidence indicates they affect the coverage metric on a large scale. The paper shows that third-party code entering the coverage report can cause a 93-percentage-point relative coverage drop (grpc-httpjson-transcoding, §3.5.3) and that OOM/timeout runs are recorded as successful coverage measurements (§3.5.4), an issue stated in §3.6 to affect 'at least 10% of harnesses'. These artifacts are included in the coverage data used to compute the 'surprisingly stable' coverage result and the roughly 5% degradation rate in Figure 3. The manuscript does not provide a sensitivity analysis that excludes known artifact patterns (e.g., harnesses with corpus-size drops or repeated OOM/timeout behavior) or corrects the denominator for external third-party code. Without such an analysis, the central claim that coverage is stable over time could be partly a property of the measurement infrastructure rather than of the harnesses. I ask the authors to rerun Figure 3 and the associated degradation statistics after removing or adjusting for these artifact-prone observations, and to report whether the qualitative conclusions change.","section":"§3.5.3, §3.5.4, §3.3"},{"comment":"The exclusion of harnesses that no longer build is a scope limitation that should be stated as part of the central claim, not only in the methodology. The abstract and §3.3 conclude 'surprising longevity' of bug-finding capability, qualified by 'as long as they still build'. This is a legitimate conditional claim, but the practical implication is weakened because a harness that fails to build finds zero bugs and is the clearest form of degradation. The paper cites Nourry et al. [41] for the prevalence of broken builds, but does not combine that information with the new data to give an overall picture of how often unmaintained harnesses actually fail. I would like the authors to state explicitly, in the abstract or conclusion, the proportion of harness versions that stop building entirely during the study period, so that the conditional 'as long as they still build' is placed in context.","section":"§3.3, §3.4"}],"minor_comments":[{"comment":"The text says 'In total, we investigate four research questions' immediately before listing RQ1 through RQ5; this should be 'five'.","section":"§3"},{"comment":"The function name is misspelled as 'LLVMFuzzerInititalize'; it should be 'LLVMFuzzerInitialize'.","section":"§3.1.4"},{"comment":"There is a missing space in 'This results in267 instances'.","section":"§3.5"},{"comment":"The caption describes coverage changes as '>5% coverage increase, >5% coverage decrease and everything in between', but the legend in the figure shows '< -5%', '-5% to +5%', and '< +5%'. Please reconcile the notation.","section":"Fig. 4 caption"},{"comment":"The sentence 'we cut off the tail end where#Harnesses falls below 100' contains a rendering issue with the '#' character; please fix the typography.","section":"§3.3"},{"comment":"The phrase 'after which our data starts to be too sparse' is ambiguous; it is unclear whether 'after which' refers to half a year or to the cutoff point in the figure.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically transparent and the dataset release is a genuine asset. The main risk to the central claim is not circularity or fabrication but the under-validation of the harness-change heuristic and the unquantified influence of coverage-measurement artifacts. Both issues are addressable with additional analyses that fit within the scope of the manuscript. I would be willing to re-review a revised version that includes a sampling-based validation of the heuristic and a sensitivity analysis for coverage artifacts."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is the first large-scale, longitudinal look at fuzz harness degradation, based on eight years of OSS-Fuzz data covering 29k harness versions across 433 C/C++ projects. The authors ship data, analysis code, and case-study notes, and manually examined 308 coverage-drop events to build a root-cause taxonomy. That alone makes it worth reading.\n\nWhat it does well: the taxonomy is genuinely useful—partial build failures, added/churned project code, external code polluting coverage reports, and corpus-size artifacts, including an unreported OOM/timeout bug in OSS-Fuzz's coverage measurement. They also contributed practical metrics to Fuzz Introspector/OSS-Fuzz and are transparent about pending PRs. The bug-finding analysis tied to coverage changes is plausible and consistent with prior work.\n\nThe soft spot that matters is the definition of 'no explicit updates.' The central longevity result hangs on the harness-change heuristic in §3.1.4: changes to files with 'fuzz' in the path, functions named LLVMFuzzerTestOneInput, plus all non-config changes in the OSS-Fuzz repo. That misses build-file updates (CMakeLists, Makefile), dependency bumps, header-only changes reachable from the harness, custom entry points, and separate-fuzzing-repo commits. The manual 'suspicious gaps' check is not a false-negative measurement. If missed updates are at all common, 'unmaintained' periods actually contain maintenance, and flat coverage is exactly what you'd expect. This bias is one-directional and directly targets the abstract's headline.\n\nSecond, the analysis excludes harnesses that no longer build; the abstract says 'as long as they still build,' so it's not hidden, but the stability result is conditional on survival, and there's no sensitivity analysis counting broken builds as zero. Third, coverage artifacts (external code, OOM/timeout) are documented but their aggregate impact on the stability plots is unquantified. RQ5's 'will detect all cases' is an overclaim while several PRs are still pending.\n\nThe citation pattern is clean: prior OSS-Fuzz studies (Ding/Le Goues, Zhu/Böhme, Nourry et al.) are distinct and openly discussed.\n\nWho's it for: fuzzing researchers, OSS-Fuzz maintainers, CI/CD infrastructure folks. A good reading-group paper for debating measurement validity.\n\nRecommendation: send to peer review. It deserves a serious referee. The data and taxonomy are a real contribution; the headline needs validation of the update-detection heuristic plus sensitivity analyses before the longevity claim is fully supported.","headline":"First large-scale longitudinal study of harness decay in OSS-Fuzz; valuable taxonomy and open artifacts, but the 'no explicit updates' longevity claim depends on an unvalidated update-detection heuristic.","tokens_in":20729,"tokens_out":6305,"would_cite":true,"duration_ms":51606,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fuzz harnesses in OSS-Fuzz keep finding bugs for years without updates, with coverage staying stable on average, as long as they still build.","keywords":["fuzz harness","harness degradation","OSS-Fuzz","code coverage","bug-finding capability","continuous fuzzing","empirical study","coverage measurement"],"falsifier":"Recompute the coverage-change curves using only coverage days whose Fuzz Introspector report shows no external-library files in the denominator and whose run completed without timeout or out-of-memory; if the median coverage since the last harness update then declines by several percentage points within a year, the surprising-longevity conclusion collapses to a measurement artifact.","tokens_in":19657,"feed_emoji":"🐛","tokens_out":7712,"duration_ms":66091,"temperature":0.7,"pith_summary":"This paper asks whether the fuzz harnesses that feed inputs into software projects decay as the projects themselves evolve, and answers with a longitudinal study of the OSS-Fuzz continuous-fuzzing platform: 29,019 harness versions across 433 C/C++ projects, with daily coverage reports and bug records spanning about eight years. The central finding is that coverage stays stable on average and that harnesses keep finding bugs for long periods even without explicit updates, as long as they still build. Only about 5% of harness versions lose at least 5 percentage points of coverage within half a year, and manual examination of 308 coverage-drop cases attributes these losses to partial build failures, newly added project code, external libraries entering the coverage report, and corpus-size artifacts rather than to gradual harness rot. The paper proposes new monitoring metrics, implemented in OSS-Fuzz and Fuzz Introspector, that would alert maintainers when a harness quietly degrades.","feed_headline":"Unmaintained fuzz harnesses keep finding bugs for years","feed_subtitle":"Analyzing 29,019 harness versions across 433 OSS-Fuzz projects finds stable coverage; ~5% degrade within half a year.","key_machinery":"The central object is the harness version, defined by a heuristic that detects harness-changing commits: any change to a C/C++ file in the project whose path contains 'fuzz' (excluding 'fuzzy'), any change to a file defining the standard entry point LLVMFuzzerTestOneInput or LLVMFuzzerInitialize, and any OSS-Fuzz repository change outside project metadata and Docker/Jenkins files. Each daily Clang line-coverage report from OSS-Fuzz is then attached to the currently active harness version, and coverage is measured as relative line-coverage percentage, using the maximum of the first three days after an update as the baseline to compensate for slow saturation. The argument that degradation is rare rests on the resulting coverage-change curves over time and over changed lines, while the causal taxonomy is built from 308 manually examined coverage drops filtered for a 5-percentage-point sustained decline.","core_discovery":"The authors establish a counter-intuitive result: fuzz harnesses in OSS-Fuzz have a surprisingly long effective lifetime. Coverage measured from the days after each harness update shows only a slight downward trend in the mean and a neutral median, and bug-finding per changed line of code stays roughly flat over at least the first half year of a harness version's life. Harness updates themselves have only a modest average effect — a mean coverage increase of 0.26% and a median of exactly 0% — yet the minority of updates that raise coverage by more than 5 percentage points produce a clear burst of newly found bugs. When coverage does drop sharply, the categorized root causes are partial build failures (such as a dependency that stops compiling), project code added or changed without corresponding harness maintenance, external library code that inflates the coverage denominator, and corpus-size collapses caused by out-of-memory or timeout runs being recorded as successful coverage measurements. The paper concludes that the real risk is not slow rot but discrete, silent events that break or distort coverage, and it contributes metrics to detect those events.","pith_inferences":["If the stability result generalizes beyond OSS-Fuzz's mature, security-critical projects, automatic harness-generation and maintenance tools (including LLM-based ones) should target discrete breakage events rather than continuous re-synthesis; keeping builds green and the coverage denominator clean may matter more than frequent regeneration.","A testable extension follows from the taxonomy: projects that pin their dependencies or exclude third-party libraries from coverage reports should show fewer spurious 'degradation' alarms, which the proposed metrics could verify by comparing alarmed versus non-alarmed projects across such policy changes.","The harness-change heuristic likely undercounts updates made in separate harness repositories or in code paths that do not match the pattern; cross-checking the heuristic against per-harness Fuzz Introspector reports could raise or lower the longevity estimate.","Because the paper itself shows that out-of-memory and timeout runs are recorded as successful coverage and that external code can inflate the denominator, a re-analysis that excludes those artifacts could shift the stable mean and median downward; the magnitude of the shift would quantify how much of the 'surprising longevity' is real."],"forward_implications":["A harness that still builds can be treated as a continuing asset: its bug-finding rate per changed line stays roughly flat for at least half a year, so maintainers can prioritize other work over routine harness refresh.","The benefits of harness maintenance are concentrated in discrete events: updates that lift coverage by more than 5 percentage points are followed by a burst of newly found bugs, while the average update changes coverage by almost nothing.","Roughly one harness version in 20 loses at least 5 percentage points of coverage within half a year, and the cause is usually a build failure, newly added code, a distorted coverage report, or a shrunken corpus — so monitoring for those specific events is the practical response.","Projects whose coverage stays below about 35% find roughly one bug per 14,474 changed lines, versus one per 3,985 lines for higher-coverage projects, making coverage level a first-order driver of fuzzing return.","The proposed metrics — tracking removed or renamed harnesses, coverage stability over time, harness statefulness, and corpus size relative to covered complexity — would in principle detect all of the categorized causes of coverage drops and are being contributed to OSS-Fuzz and Fuzz Introspector."],"supporting_citations":[{"why":"The OSS-Fuzz platform itself: supplies the daily coverage reports, the Monorail bug tracker, and the build infrastructure that the entire longitudinal dataset is drawn from.","marker":"[22]"},{"why":"Prior study of OSS-Fuzz build failures: provides the baseline that roughly 12% of builds are broken over time, which the paper's exclusion of fully broken harnesses builds on.","marker":"[41]"},{"why":"Establishes the correlation between coverage and bug discovery that justifies using coverage percentage and bugs per changed line as harness-health metrics.","marker":"[13]"},{"why":"Supplies the finding that about 77% of newly discovered bugs are regressions, which the paper uses to interpret its crash-revision data.","marker":"[58]"},{"why":"Prior empirical study of OSS-Fuzz bug life cycles that this paper contrasts with by focusing on harness maintenance rather than bug dynamics.","marker":"[16]"}],"fun_headline_variants":["Fuzz harnesses keep finding bugs even without updates","Harness coverage stable for half a year, bugs persist","Gradual decay is rare; silent events hit fuzz coverage","Unmaintained harnesses age well, but discrete failures lurk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The longevity result is computed from OSS-Fuzz's daily line-coverage reports, and the paper shows those reports can be distorted — third-party libraries can enter the coverage denominator and out-of-memory or timeout runs are recorded as successful measurements; if those artifacts are widespread, the apparent stability is partly a measurement artifact rather than genuine harness longevity.","fun_headline_variants_meta":{"raw":{"variants":["Fuzz harnesses keep finding bugs even without updates","Harness coverage stable for half a year, bugs persist","Gradual decay is rare; silent events hit fuzz coverage","Unmaintained harnesses age well, but discrete failures lurk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1837,"prompt_tokens":997,"completion_tokens":840,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":771}},"tokens_in":613,"tokens_out":840,"duration_ms":8528,"temperature":1.0,"reasoning_tokens":771,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:16.717959+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the coverage-change curves using only coverage days whose Fuzz Introspector report shows no external-library files in the denominator and whose run completed without timeout or out-of-memory; if the median coverage since the last harness update then declines by several percentage points within a year, the surprising-longevity conclusion collapses to a measurement artifact.","supporting_citations":[{"cited_title":"OSS-Fuzz: Continuous Fuzzing for Open Source Software","cited_arxiv_id":null,"evidence_quote":"The OSS-Fuzz platform itself: supplies the daily coverage reports, the Monorail bug tracker, and the build infrastructure that the entire longitudinal dataset is drawn from."},{"cited_title":"My fuzzers won’t build: An empirical study of fuzzing build failures","cited_arxiv_id":null,"evidence_quote":"Prior study of OSS-Fuzz build failures: provides the baseline that roughly 12% of builds are broken over time, which the paper's exclusion of fully broken harnesses builds on."},{"cited_title":"On the reliability of coverage-based fuzzer benchmarking","cited_arxiv_id":null,"evidence_quote":"Establishes the correlation between coverage and bug discovery that justifies using coverage percentage and bugs per changed line as harness-health metrics."},{"cited_title":"Regression greybox fuzzing","cited_arxiv_id":null,"evidence_quote":"Supplies the finding that about 77% of newly discovered bugs are regressions, which the paper uses to interpret its crash-revision data."}],"review_version":1}