{"id":"5715f5be-fc08-4ab5-a4ec-13acc6bd2954","arxiv_id":"2502.08341","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Simple code metrics, especially Leopard's vulnerability scores, retrieve crash-relevant functions more accurately than sanitizer heuristics, recently-changed code, or deep learning models on a corpus of 1,621 real crashes.","lead":"This paper compares ways to pick which parts of a program a directed fuzzer should focus on, using a new dataset of more than 1,600 real crashes from Google's OSS-Fuzz project. It finds that simple software metrics, such as Leopard's vulnerability scores, rank crash-prone functions better than most advanced heuristics and machine learning models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The stack-trace oracle makes Leopard-V's measured NDCG advantage a retrieval result, not a fuzzing result; without end-to-end validation, the 'only viable candidate' claim is unsupported.","rationale":"The paper is a solid SoK contribution: it separates target selection from fuzzing mechanics, provides a reusable corpus of 1,621 reproduced OSS-Fuzz crashes, and models selection as an information retrieval problem in a way that allows direct comparison of scoring functions. The in-vitro ranking result—Leopard-V outperforming other methods on NDCG under the stated oracle—is internally consistent and supported by the experiments as scoped. The load-bearing concern is external validity: the central practical recommendation ('default to simple metrics; Leopard-V is the only viable candidate for discrete selection') requires the stack-trace oracle to be a faithful proxy for fuzzing utility. The paper acknowledges the semantic gap but does not close it, and the NDCG+/NDCG− bounds do not address label validity, only label count. Because Leopard-V's metrics include connectivity features that may align with incidental traceback membership, the measured advantage could be inflated. The proposed end-to-end fuzzing experiment would settle whether the retrieval advantage translates into faster crash discovery. Given that the core measurement is sound but the headline practical claim rests on an unvalidated proxy, a conditional accept is appropriate: retain the in-vitro analysis, but require either end-to-end validation or a scaled-back claim about real fuzzing deployments.","tokens_in":24125,"tokens_out":7785,"duration_ms":89045,"concrete_test":"Select a stratified sample of 50–100 crashes from the corpus across projects, sanitizers, and crash types. For each crash, run a directed greybox fuzzer (e.g., AFLGo) with identical seeds and budget, using target sets of fixed size k (e.g., k=1, 5, 25) generated by Leopard-V, sanitizer instrumentation, recent-change, and random ranking. Measure time-to-first-crash and crashes found per unit time. If Leopard-V's top-k targets do not reproduce its NDCG advantage in time-to-crash, the stack-trace oracle is not a valid proxy for target-selection utility and the 'only viable candidate' claim should be withdrawn or heavily qualified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.1 defines relevance exclusively by stack-trace membership: O(f)=1 iff f appears in the crash trace, with main and sanitizer functions zeroed. Both NDCG− and NDCG+ inherit this labeling, so every reported comparison is a measure of 'can the method rank traceback members high,' not directly 'would targeting these functions help a fuzzer find the crash.' The paper's own Section 6, 'False positives,' concedes the semantic gap: root causes can be absent from the trace (data-dependent or multi-threaded crashes), and many stack frames are incidental. The optimistic and pessimistic matching policies bracket how many trace functions count, but they do not question whether trace membership is the right label in the first place. This is particularly relevant for Leopard-V because its vulnerability metrics include call-graph connectivity features (number of parameters and callee parameters, Section 5.1). Such hub-like functions are over-represented in stack traces regardless of whether targeting them helps reach a crash, so the measured 13% top-1 advantage may partly reward ranking functions that are frequent trace members rather than good fuzzing targets. The citation in Section 6 that traceback-based targets beat undirected fuzzing shows that trace-based targeting can work, but it does not validate the relative ranking of methods or equal relevance of all trace frames. Without an end-to-end fuzzing check, the conclusion that Leopard-V is 'the most natural and really only viable candidate' for discrete target selection is an extrapolation from a retrieval metric to fuzzing behavior.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This SoK paper presents the first systematic study of target selection methods for directed fuzzing. The authors review 25 papers from top venues, distill target selection into four characteristics (information source, scoring type, granularity, scoring mechanism), and model target selection as an information retrieval problem. They assemble a corpus of 1,621 reproducible OSS-Fuzz crashes across 97 C/C++ projects, label functions as relevant if they appear in the crash stack trace, and compare ten selection methods (Leopard-C, Leopard-V, sanitizer instrumentation, recent-code-change heuristic, Rats, Cppcheck, ReVeal, Linevul, CodeT5+, and random) using NDCG with optimistic and pessimistic matching policies. The central finding is that simple software metrics, especially Leopard-V, significantly outperform all other methods, with the strongest claim being that Leopard-V is 'the most natural and really only viable candidate' for fuzzing approaches requiring a discrete selection method. The paper also reports breakdowns by sanitizer and crash type, and provides public artifacts.","tokens_in":24362,"tokens_out":4043,"duration_ms":44547,"significance":"If the central claim holds, the paper would provide a practically important and somewhat surprising result: decades of research on sophisticated target selection in directed fuzzing has not surpassed simple code metrics. The work is also valuable as a reproducible benchmark: a corpus of over 1,600 real-world crashes, an explicit information-retrieval formulation, two matching policies that bracket label noise, a random baseline that anchors significance, and statistical tests across retrieval sizes. The systematic review itself (Table 1) is a useful contribution for future work. The paper is careful in many design choices, including using post-preprocessor source code to avoid preprocessor-directive confounds and reporting results broken down by sanitizer and crash type. However, the headline practical claim is currently supported only by a retrieval-based evaluation, not by end-to-end fuzzing experiments, which creates a gap between the measured NDCG performance and the paper's conclusions about fuzzing practice.","major_comments":[{"comment":"The relevance oracle O in Section 4.1 labels a function as relevant if and only if it appears in the crash's stack trace, and Section 6 'False positives' concedes that the root cause may be absent from the traceback. The cited prior work [7, 15, 25] shows that traceback-based targets beat undirected fuzzing, but it does not establish that every trace frame is a good fuzzing target or that the relative ordering of selection methods transfers to time-to-crash. Because Section 5.2 uses this retrieval measure to conclude that Leopard-V is 'the most natural and really only viable candidate' for discrete selection, the paper's central practical claim goes beyond what the measurement actually supports.","section":"Section 4.1 and Section 6"},{"comment":"Leopard-V's vulnerability metrics include the number of parameters of a function and the number of parameters to its callees. These are call-graph connectivity features, and hub-like functions are over-represented in stack traces independently of whether they are good fuzzing targets. The reported 13% top-1 NDCG− result may therefore partly reward ranking trace-frequent functions rather than ranking functions whose targeting would help a directed fuzzer find the crash. I ask for a control: compare Leopard-V against a simple baseline that ranks functions by call-graph degree or by historical trace frequency, or restrict the oracle to the deepest stack frame that is not a sanitizer or main function.","section":"Section 5.1, Leopard-V"},{"comment":"The optimistic and pessimistic matching policies bound the label noise that arises from uncertainty about which trace functions are defective, but they do not bound the semantic gap identified in Section 6, where the relevant function may be missing from the trace entirely. Consequently, NDCG− and NDCG+ are not lower and upper bounds on true fuzzing utility; they are bounds only relative to the trace-membership labeling. This should be stated explicitly in Section 4.1 so that readers do not interpret the reported intervals as bracketing end-to-end fuzzing performance.","section":"Section 4.1, matching policies"}],"minor_comments":[{"comment":"The first two sentences of the abstract are repeated verbatim, which appears to be a formatting error.","section":"Abstract"},{"comment":"The step numbering in Figure 3 lists '1, 2, 44, 43'; the last two labels should likely be '3' and '4'.","section":"Figure 3"},{"comment":"There is a duplicated word in 'for for k > 1 retrieved functions'; please fix the typo.","section":"Section 5.2"},{"comment":"The legend in Figure 8 groups methods by class using color; adding distinct line styles or markers would improve readability for color-blind readers and in grayscale printouts.","section":"Figure 8"},{"comment":"The literature-filtering heuristic (selecting papers containing the word 'directed' at least three times) is coarse; a brief discussion of how many papers were discarded and whether any known directed-fuzzing papers were missed would strengthen confidence in the systematization.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid SoK with reproducible artifacts and a careful in-vitro evaluation design. The main gap is the lack of end-to-end fuzzing validation for the headline claim about Leopard-V being the only viable candidate. If the authors either add a small end-to-end experiment (e.g., running a directed fuzzer with Leopard-V versus sanitizer-based and recent-change targets on a subset of the corpus) or carefully reframe the conclusions to be about retrieval performance rather than fuzzing efficacy, the paper would be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper. It does something genuinely new: it isolates the target-selection step from the fuzzer and compares ten methods on a 1,621-crash OSS-Fuzz corpus as an information-retrieval task. That is a useful contribution, and the authors do it carefully. The two matching policies bracket stack-trace label noise, the random baseline anchors significance, and the breakouts by sanitizer and crash type are sensible. The public artifacts (crash corpus, code) are a plus. As a measurement that simple software metrics — Leopard-V in particular — rank traceback members ahead of sanitizer instrumentation, recency heuristics, SAST tools, and learned models, the result is well-supported.\n\nThe soft spot is exactly what Section 6 concedes: relevance is defined by stack-trace membership, not by whether targeting a function helps a fuzzer reach a crash. NDCG is a retrieval metric, and every comparison inherits the oracle. Leopard-V's best features include call-graph connectivity (parameter counts, callee parameter counts), which plausibly correlates with appearing in traces regardless of fuzzing value. So the 13% top-1 figure is a retrieval result, and the leap to “the most natural and really only viable candidate” for discrete selection is unsupported without an end-to-end fuzzing benchmark. That is an extrapolation, not a measured fact. I would ask the authors to either soften the claim to the retrieval setting or add a small end-to-end check (e.g., run one directed fuzzer with the top-k targets from Leopard-V versus sanitizer-based targets on a subset of crashes) to show the NDCG gap translates.\n\nOne smaller quibble: the cluster analysis via Mann-Whitney-U dominance over every k is a bit coarse; the curves cross for low k, so the sanitizer method beating CodeT5+ for k<5 is buried in a cluster statement. That is a presentation issue, not a flaw.\n\nBottom line: for fuzzing and software-security researchers, this is a solid SoK plus benchmark that deserves referee time. If it were my call, I would accept with revisions that are mostly about calibrating the conclusion. The central retrieval result stands.","headline":"Useful SoK-with-artifact: clean isolation of target selection as an IR task, solid benchmark, honest limitations; but the 'only viable candidate' conclusion overreaches what NDCG can support without an end-to-end fuzzing check.","tokens_in":24962,"tokens_out":2533,"would_cite":true,"duration_ms":26204,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that choosing where to fuzz is an under-studied bottleneck, and that a simple program metric, Leopard-V, outperforms every advanced alternative at picking crash-relevant functions.","keywords":["directed fuzzing","target selection","software metrics","information retrieval","OSS-Fuzz","crash reproduction","vulnerability prediction","large language models"],"falsifier":"Run a controlled head-to-head where a directed fuzzer is given targets from Leopard-V and, separately, an equal-sized set drawn from random functions on the same OSS-Fuzz projects, measuring time-to-first-crash over many repetitions; if the Leopard-V targets do not hit crashes faster than random targets, the paper's central ranking claim would not translate into fuzzing performance.","tokens_in":23901,"feed_emoji":"🎯","tokens_out":7855,"duration_ms":75788,"temperature":0.7,"pith_summary":"The paper asks a question fuzzing research mostly skips: once a fuzzer is directed, who decides where to point it? It argues that target selection is an independent, under-studied component of directed fuzzing, and that it can be measured in isolation by treating selection as an information-retrieval task. Using more than 1,600 reproduced crashes from 97 real-world projects as ground truth, it compares metric-based, pattern-based, static-analysis, and machine-learning selection methods. The paper's central finding is that simple software metrics—above all the vulnerability metric Leopard-V—significantly outperform every other tested method, with the top-ranked Leopard-V function landing in the crash stack trace 13% of the time. A sympathetic reader should take away that 'where to fuzz' is itself a decisive performance factor, and that current default heuristics like recently changed code or sanitizer locations are not the best choices.","feed_headline":"Simple code metrics outperform advanced heuristics for fuzzing targets","feed_subtitle":"Leopard-V's top-ranked function lands in the crash stack trace 13% of the time across 1,600+ real crashes.","key_machinery":"The machinery is a formal analogy between target selection and information retrieval. A selection method is treated as a scoring function $\\rho: \\mathcal{F} \\to \\mathbb{R}^+$ that ranks functions in a project; the top-$k$ functions form a retrieval, and quality is measured by NDCG, a ranking-aware metric that rewards relevant functions placed high. Relevance is assigned by an oracle $\\mathcal{O}$ that labels a function relevant if it appears in the reproduced crash's stack trace, with ubiquitous frames like main and sanitizer helpers zeroed out. To absorb label noise, the paper uses two matching policies: NDCG$^-$ treats every stack-trace function as relevant and NDCG$^+$ only the first retrieved one. The evaluation corpus comes from reproducing OSS-Fuzz crashes at a pinned commit and extracting post-preprocessor functions, giving 1,621 labelled crashes across 97 C/C++ projects.","core_discovery":"On the paper's own terms, the discovery is that target selection for directed fuzzing can be evaluated as ranked retrieval, and that when it is, the ranking is dominated by a classic program-metric method. The authors distill 25 directed-fuzzing papers into scoring mechanisms, then test representative methods in isolation: Leopard's complexity and vulnerability metrics, sanitizer-instrumentation counts, recency of code change, two SAST tools, three deep-learning vulnerability predictors, and random scoring. Across 1,621 reproduced OSS-Fuzz crashes, Leopard-V—a function-level score built from pointer use, control-flow nesting, and dependency-related metrics—significantly outperforms all competitors for nearly every retrieval size and crash class, and Leopard-C is close behind. The paper states that Leopard-V's highest-ranked function matches a crash-stack-trace function in 13% of cases across the whole corpus, making it 'the most natural and really only viable candidate' for directed fuzzers that need a discrete target set. It also reports that a fine-tuned code language model, CodeT5+, approaches the metric-based methods' performance, identifying learned models as the most promising direction for improving selection.","pith_inferences":["Beyond the paper's retrieval results, if Leopard-V's top-1 hit rate carries over to a real fuzzing loop, then simply trying the top handful of metric-ranked functions in order is a discrete selection strategy that should beat current heuristics end-to-end; the paper measures ranking, not end-to-end fuzzing, so this is my extrapolation.","The stack-trace oracle labels functions implicated in crashes, not functions whose targeting shortens time-to-crash; feeding Leopard-V targets into an existing directed fuzzer and measuring time-to-exposure would test the transfer.","The same retrieval framing could be reused for ranking suspicious functions in patch review or static-analysis triage, where the bottleneck is also 'where to look first'.","A practical yardstick falls out of the corpus: future target-selection methods can be compared against Leopard-V's top-1 hit rate on the released crash set."],"forward_implications":["For directed fuzzers that require a discrete target set, defaulting to Leopard-V's top-ranked functions should outperform the field's common heuristics.","Continuous target selection can afford to use methods that only shine at larger retrieval sizes; discrete selection should be chosen only from methods strong at small $k$.","Sanitizer-instrumentation counts and recently modified code, two widely used heuristics, are measurably worse than simple metrics and should not be assumed safe defaults.","Machine-learned vulnerability predictors, particularly CodeT5+, are close enough to metrics to justify further work on learned target selection.","Target-selection quality varies strongly by crash type, so a single selection method may need to be tuned per bug class."],"supporting_citations":[{"why":"Supplies Leopard-C and Leopard-V, the program-metric scoring functions whose dominance is the paper's central finding.","marker":"[24]"},{"why":"OSS-Fuzz is the source of the public crash reports and fuzzing configurations from which the 1,621-crash ground-truth corpus is built.","marker":"[62]"},{"why":"Supplies the 'recently modified code' heuristic and the continuous-scoring regression-fuzzing baseline that the comparison shows to be weaker than metrics.","marker":"[81]"},{"why":"Supplies the sanitizer-instrumentation target-selection heuristic the paper evaluates and finds inferior to Leopard.","marker":"[83]"},{"why":"Supplies CodeT5+, the code language model whose fine-tuned vulnerability scoring nearly matches the metric-based methods.","marker":"[71]"},{"why":"Supplies the graph-neural-network vulnerability predictor used as a machine-learning baseline.","marker":"[14]"},{"why":"Supplies Linevul, the transformer-based line-level vulnerability predictor used as a second learned baseline.","marker":"[30]"},{"why":"Supplies Rats, a rule-based SAST tool used to test whether static-analysis findings make good fuzzing targets.","marker":"[29]"},{"why":"Supplies Cppcheck, the mature open-source SAST tool used as the other static-analysis baseline.","marker":"[50]"}],"fun_headline_variants":["Simple metrics beat advanced heuristics for fuzzing targets","Fuzzing target selection: classic metrics win over fancy heuristics","Simple program metrics outperform deep learning for fuzz target selection","Leopard-V metric wins for choosing fuzzing targets in OSS-Fuzz","Target selection for fuzzing: simple metrics beat sanitizer and recency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a function appearing in a crash's stack trace is a good target for a directed fuzzer, so the whole ranking evaluation inherits that labelling; if the real crash-triggering code is missing from the trace, the measured rankings mislabel what good target selection means.","fun_headline_variants_meta":{"raw":{"variants":["Simple metrics beat advanced heuristics for fuzzing targets","Fuzzing target selection: classic metrics win over fancy heuristics","Simple program metrics outperform deep learning for fuzz target selection","Leopard-V metric wins for choosing fuzzing targets in OSS-Fuzz","Target selection for fuzzing: simple metrics beat sanitizer and recency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000551,"raw_usage":{"total_tokens":2678,"prompt_tokens":1046,"completion_tokens":1632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":1540}},"tokens_in":662,"tokens_out":1632,"duration_ms":12343,"temperature":1.0,"reasoning_tokens":1540,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T05:28:30.723299+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled head-to-head where a directed fuzzer is given targets from Leopard-V and, separately, an equal-sized set drawn from random functions on the same OSS-Fuzz projects, measuring time-to-first-crash over many repetitions; if the Leopard-V targets do not hit crashes faster than random targets, the paper's central ranking claim would not translate into fuzzing performance.","supporting_citations":[{"cited_title":"Ali Babar","cited_arxiv_id":null,"evidence_quote":"Supplies Leopard-C and Leopard-V, the program-metric scoring functions whose dominance is the paper's central finding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OSS-Fuzz is the source of the public crash reports and fuzzing configurations from which the 1,621-crash ground-truth corpus is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 'recently modified code' heuristic and the continuous-scoring regression-fuzzing baseline that the comparison shows to be weaker than metrics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the sanitizer-instrumentation target-selection heuristic the paper evaluates and finds inferior to Leopard."},{"cited_title":"Bui, Junnan Li, and Steven C","cited_arxiv_id":null,"evidence_quote":"Supplies CodeT5+, the code language model whose fine-tuned vulnerability scoring nearly matches the metric-based methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the graph-neural-network vulnerability predictor used as a machine-learning baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Linevul, the transformer-based line-level vulnerability predictor used as a second learned baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Rats, a rule-based SAST tool used to test whether static-analysis findings make good fuzzing targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Cppcheck, the mature open-source SAST tool used as the other static-analysis baseline."}],"review_version":1}