{"id":"56b782c3-e733-461a-8733-9c29b425b574","arxiv_id":"2608.06641","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"PDFuzzer combines LLM-based API specification and relationship inference with SMT solving to fuzz PDF JavaScript engines, finding 31 zero-day vulnerabilities and up to 48% higher coverage than prior tools.","lead":"This paper introduces PDFuzzer, a fuzzer that uses a large language model to read PDF JavaScript API documentation and infer how API calls depend on each other, then applies a constraint solver to generate realistic sequences of calls. On three popular PDF readers it reports up to 48% higher code coverage than existing fuzzers and 31 vendor-confirmed zero-day vulnerabilities, from information leaks to potential arbitrary code execution.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Baseline comparison strips Cooper/Favocado to generator-only mode under TypeOracle's wrapper, so the headline 'outperforms state-of-the-art fuzzers' is not established for the actual released fuzzers.","rationale":"I agree with the reader's CONDITIONAL posture but identify a different load-bearing assumption. The reader's weakest_assumption—that Adobe's API manual may not faithfully specify Foxit and PDF-XChange semantics—is reasonable, yet the paper's own empirical results partially undercut it: the Foxit use-after-free described in Section 4.2/6.2 is an Adobe-manual value-constraint (cName=cQuery) that actually triggered in Foxit, and multiple XChange-specific findings exist. The baseline-fairness issue in Section 5.2 is more direct: every headline percentage is computed against stripped generators, not the released fuzzers whose names appear in the abstract. This does not invalidate the systems contribution; comparing test-case generators under an identical wrapper is a legitimate and clearly scoped scientific comparison, and the paper explicitly acknowledges it. But the claim wording outruns the experiment, so the conditional posture is justified. My concern is addressable by a head-to-head rerun: if PDFuzzer still wins, the central claim holds; if not, the overstatement is real. I therefore keep the verdict unchanged rather than moving it, and I mark agreement as 'partial' because the reader's rationale mentions comparability concerns but their weakest_assumption field does not.","tokens_in":25141,"tokens_out":5510,"duration_ms":53469,"concrete_test":"Run the unmodified released Favocado and Cooper fuzzers (their own generation + mutation + corpus scheduling, without the TypeOracle wrapper) for five 24-hour runs per target under the same VM spec, and measure coverage and crash counts with the same DynamoRIO/werfault triage. If either full tool comes within roughly 10% of PDFuzzer's coverage or finds a comparable number of vulnerabilities, the headline comparison overstates generator-only gains; if PDFuzzer retains its margin and vulnerability lead, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('significantly outperforming state-of-the-art traditional and LLM-based fuzzers') depends on a fair comparison against the released tools. Section 5.2 explicitly says the evaluation does not compare online fuzzing loops: 'we use TypeOracle's public fuzzing wrapper uniformly for all methods,' and Cooper/Favocado are evaluated only as test-case generators, not as complete fuzzers. However, the abstract and contributions claim superiority over 'state-of-the-art PDF fuzzers (TypeOracle, Favocado, and Cooper)', not over their generator components. Favocado and Cooper's shipped systems include mutation and corpus-management logic that the wrapper discards; Fuzz4All is similarly reduced to generated inputs passed through a non-LLM wrapper. If these full tools were allowed their own loops, they might close or even invert the 15-48% coverage gap and could find additional vulnerabilities. This is not an internal inconsistency—the paper is transparent about the scoped comparison—but it is load-bearing for the headline because the 'up to 48% higher coverage' and '31 vs at most 6 vulnerabilities' numbers are produced under an abstraction that isolates sequence generation. The Adobe-manual-as-reference issue is plausible but partially mitigated by the empirical finds in Foxit and XChange (e.g., the Listing 4 value-constraint bug actually triggered in Foxit); the baseline abstraction directly shapes the reported comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents PDFuzzer, a fuzzer for the JavaScript engines embedded in PDF readers. PDFuzzer extracts API specifications from Adobe's JavaScript API manual and from execution traces, uses LLMs to generate parameter-level context-free grammars and to infer three classes of inter-API relationships (producer-consumer, value-constraint, and implicit), and then uses the Z3 SMT solver to generate concrete API-call sequences that satisfy the inferred constraints. The generated JavaScript is embedded into PDF documents and executed against Adobe Acrobat Reader, Foxit PDF Reader, and PDF-XChange Editor. The evaluation reports up to 48% higher basic-block coverage than prior tools, 31 zero-day vulnerabilities over a two-week campaign, and ablations showing that specification inference, parameter-level grammars, strong symbolic relationships, and PDF-object integration each contribute to coverage.","tokens_in":25412,"tokens_out":4161,"duration_ms":41500,"significance":"If the results hold, this is a useful contribution to API-level fuzzing: it automates specification extraction from natural-language documentation, models relationship types beyond producer-consumer dependencies, and demonstrates concrete security impact with 31 reported vulnerabilities, including CVEs and vendor bounties. The paper's strengths include a broad evaluation across three commercial targets, a two-week vulnerability campaign, a component-wise ablation study, spot-check evaluations of LLM accuracy (93-98%), and a public artifact link. The main caveat is that the comparison against prior fuzzers is scoped to test-case generation under a common wrapper rather than to the released full fuzzing systems, so the headline claims need to be tempered or supplemented.","major_comments":[{"comment":"The evaluation compares only test-case generators, not the released fuzzing systems. Section 5.2 states that TypeOracle's public wrapper is used uniformly for all methods and that the evaluation does not compare online fuzzing loops. Cooper and Favocado, however, ship with their own mutation, corpus, and scheduling logic, which is discarded here. The abstract and Section 1 claim that PDFuzzer 'significantly outperforms state-of-the-art PDF fuzzers' and report '31 vs at most 6' vulnerabilities; these claims are not established for the full tools. Please either add a comparison that runs Cooper, Favocado, and TypeOracle with their native loops, or revise the contribution and headline claims to state explicitly that the comparison is among test-case generators under a common execution harness.","section":"§5.2, Figure 2, Table 1"},{"comment":"Coverage results are reported only as means over five runs, without standard deviations, confidence intervals, or per-run data points. The claim that PDFuzzer 'significantly outperforms' baselines by 15-48% is therefore not statistically supported, and some of the smaller gaps in Figure 2 (for example between PDFuzzer and TypeOracle+Favocado on Adobe) could be within run-to-run variation. Please report the variance or provide the per-run coverage values.","section":"§6.1, Figure 2"},{"comment":"The pipeline uses Adobe's JavaScript API manual as the reference specification for Foxit PDF Reader and PDF-XChange Editor, with the justification that both implement Adobe's de facto standard. This assumption is load-bearing for two of the three targets: if those engines diverge in parameter semantics, value-constraint, or implicit relationships, the inferred grammars and constraints would produce invalid sequences and the coverage and vulnerability results for Foxit and XChange would be affected. The empirical finds in Foxit partially mitigate this concern, but the paper should validate API compatibility more directly, for example by checking a random sample of inferred invocations against each engine's runtime behavior or by documenting known divergences.","section":"§4.3"}],"minor_comments":[{"comment":"The phrase 'requiresequencesof API calls' is missing spaces; similar formatting spacing issues occur throughout (e.g., 'PDFuzzercan', 'PDFuzzeris', 'TypeOracle’srandom'). A copyediting pass would improve readability.","section":"Abstract"},{"comment":"The sentence 'the constraints can be broader that equality' should read 'broader than equality'.","section":"§3.2"},{"comment":"The phrase 'Because our targets are closed-sourced' should be 'closed-source'.","section":"§5.2"},{"comment":"The coverage plots would be more informative with error bars or shaded confidence bands, especially since coverage is reported as a mean over five runs, and with a consistent legend across all six panels.","section":"Figure 2"},{"comment":"The table uses '...' to denote omitted rows in several places. Please ensure the published version contains the complete table with all 31 entries, or clearly indicate the truncation.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution, but the framing in the abstract and introduction overreaches relative to the generator-only evaluation. The authors should be pushed to either add full-tool comparisons or soften the 'outperforms state-of-the-art fuzzers' language. There is also notable self-overlap with the authors' own TypeOracle, which supplies both the execution-trace inference and the evaluation wrapper; this is not a correctness problem, but the editor may want to ensure the novelty relative to that prior system is clearly delineated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know up front: this is a solid systems paper, likely the strongest LLM-for-fuzzing result I've seen this cycle. The core claim — that modeling value-constraint and implicit inter-API relationships lets a generator produce sequences that trigger real bugs — is supported by 31 disclosed zero-days, 26 confirmed or fixed, with bounties. That is real evidence, and the paper earns it with a clean pipeline: two-stage RAG+LLM relationship inference, parameter-level CFG generation, SMT solving for constraint instantiation, and a genuine ablation study isolating each component. The 93–98% spot-check accuracies on LLM steps are believable and useful. The methodology section is transparent and the writing is unusually easy to follow.\n\nThe main soft spot is the one the stress-test flagged: the abstract and contributions say PDFuzzer \"significantly outperforms state-of-the-art PDF fuzzers,\" but Section 5.2 scopes the comparison to test-case generators running under TypeOracle's uniform wrapper. Favocado and Cooper lose their own scheduling, mutation, and corpus-management loops; Fuzz4All is similarly reduced. That is a legitimate way to isolate generation quality — and the paper says so explicitly — but it does not establish superiority over the full released tools. The 48% coverage gap and the 31-vs-6 vulnerability count are numbers for generator comparison, not fuzzer-vs-fuzzer comparison. This needs a framing fix, not new experiments.\n\nMinor issues: coverage plots show means only; five runs per tool presumably give variance, but it isn't plotted. The Adobe-manual-as-reference assumption for Foxit and XChange is less concerning than the reader's weakest-assumption worried: prior work already treats Adobe's JS API as de facto standard, and the fact that PDFuzzer's value-constraint bug in Listing 4 actually triggers in Foxit suggests the reference semantics transfer well enough. Also, the artifact link is bare — no documented contents or reproduction instructions for the 31 findings, which matters for a paper this empirical.\n\nThe ablation table is a real strength: each dimension moves coverage in the expected direction, and the PC-only versus Strong comparison (up to 15.5% on Foxit) directly demonstrates that value-constraint and implicit relationships add value beyond producer-consumer. No circularity concerns; the constraints are LLM-inferred before execution, not fitted to the crashes.\n\nWho is this for? Fuzzing, LLM-for-security, and PDF/scripting-engine researchers. It deserves a serious referee; the baseline framing and missing variance are addressable in revision. I'd send it to review and would cite it once the artifact is properly released.\n\nVerdict: accept with revisions, conditional on the comparison claims being scoped honestly.","headline":"A serious, well-evidenced fuzzing paper whose 31 zero-days and coverage gains are credible, but the headline comparison overstates what is shown because competing fuzzers run only as generators under one shared wrapper.","tokens_in":25969,"tokens_out":1367,"would_cite":true,"duration_ms":15363,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PDFuzzer claims that complex, value-coupled JavaScript call sequences produced from LLM-inferred API relationships and SMT solving find far more vulnerabilities in PDF readers than prior fuzzers.","keywords":["fuzzing","PDF readers","JavaScript engines","LLM-driven test generation","API relationship inference","SMT constraint solving","zero-day vulnerability discovery","closed-source software testing"],"falsifier":"Take the 31 minimized crash-triggering test cases and re-run each under PDFuzzer with Z3 constraint solving disabled but the same API call sequences; if most crashes survive, value-constraint solving is not the decisive ingredient. Separately, probe a sample of Foxit and PDF-XChange APIs to see whether the parameter constraints from Adobe's manual match the values those engines actually accept; a large mismatch would invalidate the cross-engine reference-specification assumption.","tokens_in":24959,"feed_emoji":"🛡️","tokens_out":9512,"duration_ms":76905,"temperature":0.7,"pith_summary":"This paper argues that existing fuzzers for JavaScript engines inside PDF readers leave vulnerabilities undiscovered because they generate mostly standalone API calls and only recognize a narrow, type-based dependency between calls. PDFuzzer is a fuzzer that uses an LLM to turn API manuals and execution traces into full specifications for both documented and undocumented APIs, infers three kinds of relationships between API calls (producer-consumer, value-constraint, and implicit shared-state), and uses an SMT solver to produce concrete call sequences that satisfy those relationships. The authors report that this approach reaches up to 48% higher code coverage than prior fuzzers on Adobe Acrobat Reader, Foxit PDF Reader, and PDF-XChange Editor, and that over a two-week campaign it found 31 zero-day vulnerabilities while the best baseline found 6. The contribution is the claim that complex, value-coupled sequences, not individual calls, are what expose serious bugs in these closed-source engines.","feed_headline":"LLM-built API sequences find 31 zero-days in PDF readers","feed_subtitle":"PDFuzzer turns API manuals into constrained call sequences, beating prior fuzzers by up to 48% coverage.","key_machinery":"The central mechanism is a two-tier relationship model plus a four-stage generation pipeline. The two tiers are candidate relationships (co-occurrence only) and strong symbolic relationships, which carry SMT-LIB2 constraints and an optional ordering requirement. Strong symbolic relationships are classified as producer-consumer (a return value feeds another call's parameter), value-constraint (parameters must satisfy equality, set membership, range, or ordering constraints), or implicit (calls interact through shared reader state). PDFuzzer's pipeline extracts documented API specifications with a regex parser, recovers undocumented API signatures via differential analysis and enriches them with an LLM, builds a per-parameter context-free grammar for each API, uses retrieval-augmented generation plus a two-stage LLM pass to infer relationships, then instantiates calls and uses Z3 to solve the symbolic constraints into concrete sequences. The paper's key design choice is to keep LLM semantic reasoning separate from SMT constraint solving, and to generate grammars parameter-by-parameter so fine-grained value constraints are not flattened out of long API specifications.","core_discovery":"On the paper's own terms, the discovery is that value-constraint and implicit relationships between JavaScript API calls can be extracted automatically and turned into executable test cases, and that doing so changes fuzzing outcomes. Of the 1,019 strong symbolic relationships PDFuzzer inferred, 87% were value-constraint, 6.2% implicit, and only 6.8% producer-consumer, which is why the paper judges producer-consumer-only tools insufficient. The paper demonstrates this with the Foxit use-after-free test case in which app.popUpMenuEx's cName and search.query's cQuery must be the same string, a coupling that name-similarity and return-value matching both miss. PDFuzzer's claim is that with this relationship model, 31 zero-day vulnerabilities can be found in three mainstream readers, 11 with potential for arbitrary code execution, and 26 already fixed by vendors at the time of writing.","pith_inferences":["An untested consequence is that most of the benefit may come from simple equalities: since 75.2% of value-constraint relationships are direct equalities, a lightweight generator that reuses concrete strings across similarly named parameters could capture a large share of the reported gains without an LLM or SMT solver.","The 64% instantiation success rate for implicit relationships suggests shared-state preconditions are the bottleneck; explicitly modeling event and state machines inside the reader could raise that rate and yield further coverage gains.","The 93-98% LLM accuracy figures rest on 60 sampled outputs per stage, so they are estimates with wide uncertainty; a follow-up that degrades relationship-inference accuracy artificially would show how much of the 31-vulnerability count depends on inference quality.","The poor output quality of the general-purpose LLM fuzzer (mostly C, Java, or plain text) indicates that generic LLM fuzzing without domain-specific specification extraction is ill-suited to this target, but it also implies improvements to generic harnesses could one day close the gap."],"forward_implications":["Fuzzers for other closed-source, scriptable applications should shift from isolated-call generation to relationship-aware sequence generation; the paper's ablation attributes up to 15.5% coverage gain to value-constraint and implicit relationships beyond producer-consumer-only settings.","Undocumented APIs become fuzzable targets rather than gaps: combining trace-derived signatures with LLM specification inference yields up to 28% higher coverage than type-only inference.","Parameter-level grammar generation is load-bearing; the paper reports that function-level grammars lose enumerated value sets and cost 5-18% coverage.","The method generalizes beyond JavaScript: the authors extract 2,925 VBA APIs from Microsoft Word's documentation and generate grammars for all of them, pointing toward fuzzing Office macros with the same pipeline.","Coverage is stable across LLMs: swapping GPT-4o for o3-mini or Claude-3.7-Sonnet changes coverage by under 1% on Adobe Acrobat Reader, so the approach does not depend on one particular model."],"supporting_citations":[{"why":"Supplies the operand-variation differential analysis that PDFuzzer uses to recover raw signatures and parameter types for undocumented APIs, and serves as a coverage and vulnerability baseline.","marker":"[24]"},{"why":"Favocado is a baseline producer-consumer fuzzer and the main example of manual, documentation-heavy specification extraction that PDFuzzer automates.","marker":"[16]"},{"why":"Cooper contributes the cooperative mutation and PDF-object integration used in Stage 3 of test-case generation, and is also a comparison baseline.","marker":"[63]"},{"why":"Adobe's JavaScript API manuals are the reference specification from which documented API specs and relationship semantics are parsed.","marker":"[3, 4]"},{"why":"Z3 is the SMT solver that instantiates symbolic relationship constraints into concrete parameter values during sequence generation.","marker":"[41]"},{"why":"The retrieval-augmented generation mechanism makes candidate relationship extraction linear in the number of APIs and avoids context-window overflow.","marker":"[44]"},{"why":"The chain-of-thought capability is cited as the basis for the LLM's multi-step reasoning over documentation during relationship inference.","marker":"[59]"},{"why":"TitanFuzz is the inspiration for the naive LLM baseline that prompts a model directly for test cases without specification extraction.","marker":"[14]"},{"why":"Fuzz4All is a general-purpose LLM fuzzer baseline whose generated test cases were mostly invalid, providing a comparison point for PDFuzzer.","marker":"[60]"},{"why":"DynamoRIO is the dynamic instrumentation tool used to measure basic-block coverage on the three closed-source readers.","marker":"[17]"}],"fun_headline_variants":["LLM-built API sequences expose 31 PDF reader zero-days","PDFuzzer: LLM-guided fuzzing nets 31 zero-days","LLM-driven fuzzing finds 31 zero-days in PDF readers","From docs to zero-days: LLM fuzzing uncovers 31 PDF reader bugs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Adobe's JavaScript API manual accurately describes the JavaScript engines in Foxit PDF Reader and PDF-XChange Editor; if those engines diverge from Adobe's semantics, the inferred grammars and constraints would generate invalid sequences for two of the three targets.","fun_headline_variants_meta":{"raw":{"variants":["LLM-built API sequences expose 31 PDF reader zero-days","PDFuzzer: LLM-guided fuzzing nets 31 zero-days","LLM-driven fuzzing finds 31 zero-days in PDF readers","From docs to zero-days: LLM fuzzing uncovers 31 PDF reader bugs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00032,"raw_usage":{"total_tokens":1827,"prompt_tokens":990,"completion_tokens":837,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":760}},"tokens_in":606,"tokens_out":837,"duration_ms":7013,"temperature":1.0,"reasoning_tokens":760,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:10:40.210098+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the 31 minimized crash-triggering test cases and re-run each under PDFuzzer with Z3 constraint solving disabled but the same API call sequences; if most crashes survive, value-constraint solving is not the decisive ingredient. Separately, probe a sample of Foxit and PDF-XChange APIs to see whether the parameter constraints from Adobe's manual match the values those engines actually accept; a large mismatch would invalidate the cross-engine reference-specification assumption.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the operand-variation differential analysis that PDFuzzer uses to recover raw signatures and parameter types for undocumented APIs, and serves as a coverage and vulnerability baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Favocado is a baseline producer-consumer fuzzer and the main example of manual, documentation-heavy specification extraction that PDFuzzer automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cooper contributes the cooperative mutation and PDF-object integration used in Stage 3 of test-case generation, and is also a comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Z3 is the SMT solver that instantiates symbolic relationship constraints into concrete parameter values during sequence generation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The retrieval-augmented generation mechanism makes candidate relationship extraction linear in the number of APIs and avoids context-window overflow."},{"cited_title":"Chi, Quoc V","cited_arxiv_id":null,"evidence_quote":"The chain-of-thought capability is cited as the basis for the LLM's multi-step reasoning over documentation during relationship inference."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TitanFuzz is the inspiration for the naive LLM baseline that prompts a model directly for test cases without specification extraction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fuzz4All is a general-purpose LLM fuzzer baseline whose generated test cases were mostly invalid, providing a comparison point for PDFuzzer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DynamoRIO is the dynamic instrumentation tool used to measure basic-block coverage on the three closed-source readers."}],"review_version":1}