{"id":"307e1fe2-8b68-45d5-8fc6-334898cc16ea","arxiv_id":"2412.15991","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"APIRL, a deep Q-network with a pre-trained transformer embedding of HTTP responses, finds more 5XX bugs than six baseline fuzzers across 26 REST APIs using fewer requests.","lead":"APIRL is a reinforcement learning fuzzer that mutates REST API requests and uses a transformer model to read HTTP response text as feedback. On 26 real-world APIs it reported more server-side bugs than six existing fuzzers while sending fewer test requests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unique-bug oracle is undefined for closed-source BitBucket, so the headline bug-count comparison—and the 'at least 6.4% more bugs' margin—is not yet interpretable; this should be settled before accepting the central claim.","rationale":"The central claim is empirical: APIRL finds significantly more bugs with fewer requests. Both halves depend on the bug-count oracle. The paper's own definition of unique bugs (5XX responses originating from different source lines, Section 5.2) is only implementable when source code is available. BitBucket is closed-source, and the paper acknowledges coverage cannot be collected there, yet reports unique bugs for it without describing any substitute deduplication. Since the closest margin in the headline comparison is on BitBucket (6.6 vs 6.2), and request volumes differ by an order of magnitude (APIRL 10,562.8 vs EvoMaster 1,315.8; RestTestGen 1,194), a per-operation deduplication would make the comparison meaningless. The '49 unique bugs' paragraph in Section 5.4 is also not reproducible from Table 2 (which lists average counts summing to 34.5 for APIRL, with no union/intersection arithmetic), and the percentages do not reconcile; this strengthens the need for a precise, reproducible bug-report log. The reader already conditioned acceptance on the oracle definition; the concrete check above would settle it. I do not see a separate concern that would overturn the paper's other contributions (the ablation and SHAP analysis are useful), so the verdict remains conditional.","tokens_in":16391,"tokens_out":16175,"duration_ms":132456,"concrete_test":"Inspect the published APIRL repository or request the per-request BitBucket logs and the bug-deduplication code; verify that every 5XX is mapped to a distinct source line, or provide the exact alternative oracle used. Then recompute Table 2's BitBucket Unique Bugs row with one consistent oracle applied to all tools. If the 6.6 vs 6.2 margin changes by more than one bug, or if the oracle cannot be made consistent, the 'significantly more bugs' claim loses its largest-service support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2 defines a unique bug as a 5XX response whose originating source line differs from all previously counted bugs. That oracle requires source-code/line-coverage access. The paper explicitly says coverage cannot be collected for BitBucket because it is closed-source, yet Table 2 reports BitBucket unique bugs for every tool. No alternative deduplication rule is stated. BitBucket is the largest test service (518 operations), and the paper's 'at least 6.4% more bugs' claim is anchored to BitBucket (APIRL 6.6 vs EvoMaster 6.2) plus smaller margins in VAmPI and WordPress that overlap within one standard deviation. If BitBucket bugs were deduplicated per operation rather than per source line, the counts are not comparable across tools with very different request volumes (APIRL sends ~10.6k requests vs EvoMaster's ~1.3k on BitBucket), so the central claim rests on an unverified metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes APIRL, a black-box REST API fuzzer based on Deep Q-Networks. The agent mutates HTTP request templates using a fixed action space of 23 actions, and its state representation combines four handcrafted request features with a 768-dimensional embedding produced by a RoBERTa transformer pre-trained on HTTP responses. The reward is primarily status-code-based, giving +10 for 5XX responses, +1 for 2XX, and -1 otherwise. The policy is trained on the Generic University API and then evaluated without further training on 26 APIs (823 operations), comparing against EvoMaster, RestTestGen, DeepREST, ARAT-RL, MINER, and a random-action variant. The main claims are that APIRL finds significantly more bugs than state-of-the-art tools while using fewer test cases, and that ablations show the value of the transformer embedding, the reward function, and the DQN choice.","tokens_in":16592,"tokens_out":5483,"duration_ms":47779,"significance":"If the main claims hold, APIRL is a useful contribution to black-box REST API testing: it packages a transferable RL policy with a pre-trained transformer feedback mechanism, and the evaluation is unusually broad for this line of work, covering 26 real-world APIs with five repetitions per experiment, an ablation over seven reward functions, an RL-algorithm comparison, and a SHAP-based feature analysis. The public release of the tool on GitHub is also a strength. However, the headline bug-count comparison is currently not fully interpretable because the unique-bug oracle for the closed-source BitBucket service is unspecified, and the claim that the policy generalizes to unseen APIs depends on the pre-training corpus not overlapping the evaluation services, which the paper does not explicitly establish. These issues are local and fixable, but they are load-bearing for the central claim.","major_comments":[{"comment":"The paper defines a unique bug as a 5XX response whose originating source line differs from all previously counted bugs, but it also states that coverage cannot be collected for BitBucket because BitBucket is closed-source. Table 2 nevertheless reports BitBucket unique-bug counts for every tool, with no alternative deduplication rule stated for BitBucket. Because APIRL sends about 10,563 requests on BitBucket versus about 1,316 for EvoMaster, per-operation or response-text deduplication would not be comparable across tools. The 'at least 6.4% more bugs' claim in Section 5.4 is anchored to BitBucket (APIRL 6.6 vs EvoMaster 6.2), so the authors must either specify and validate a deduplication procedure for closed-source services, or exclude BitBucket from the bug-count comparison and re-derive the headline margin.","section":"Section 5.2, Table 2"},{"comment":"The transformer is pre-trained on 2,566 HTTP responses from 103 REST APIs found on SwaggerHub, but the paper never states whether the 26 evaluation services (VAmPI, vAPI, c{api}tal, Spree, BitBucket, WordPress) were excluded from that pre-training corpus. The abstract and Section 5.4 claim generalization to 'unseen API endpoints'; this requires the evaluation services to be absent from pre-training, otherwise the measured transfer could partly reflect familiarity with the response structure of the test services. Please provide the full list of pre-training APIs or an explicit statement of exclusion.","section":"Section 5.1, Appendix C"},{"comment":"The textual percentage claims are inconsistent with Table 2. The paper states that EvoMaster and MINER find 117.6% and 99.2% fewer bugs than APIRL, respectively, but summing the per-service averages in Table 2 gives APIRL 34.5 bugs, EvoMaster 15.4, and MINER 16.65, which correspond to APIRL finding about 124% more bugs than EvoMaster and about 107% more than MINER. Likewise, 'at least 6.4% more bugs' appears to be computed from BitBucket (APIRL 6.6 vs EvoMaster 6.2), even though the best non-APIRL baseline on BitBucket is MINER with 6.3, a 4.8% margin. Please report the exact derivation of these percentages or replace them with effect sizes computed directly from the raw data.","section":"Section 5.4"},{"comment":"The claim that APIRL finds 'significantly more bugs' is not backed by any statistical test. On BitBucket, APIRL's margin over EvoMaster is 0.4 bugs and its margin over MINER is 0.3 bugs, both smaller than the reported standard deviations (1.1 versus 1.3 and 1.5); on VAmPI and WordPress the differences from Rand-APIRL also overlap within one standard deviation. With five runs per tool, paired tests or confidence intervals should be reported for the headline comparisons, or the word 'significantly' should be replaced by a more cautious formulation.","section":"Section 5.4, Table 2"}],"minor_comments":[{"comment":"The sentence 'using semi-supervised approach the loss is computed... using the Bellmen equation' is imprecise: DQN training is better described as regression onto bootstrapped targets using the Bellman equation, and 'Bellmen' should be 'Bellman'.","section":"Section 2.1"},{"comment":"Figure 2 does not clearly show the feedback path from the HTTP response back to the agent, even though the text and reward definition rely on it; a loop arrow from 'HTTP response code / Coverage / Bugs' back to the state would improve readability.","section":"Section 3, Figure 2"},{"comment":"The text says 'APIRL-cov finds the most bugs of the ablations (89% more than the next APIRL-cov variant)', but Table 3 does not contain a bug-count column, only error requests; please state explicitly that 'error requests' is the proxy used for bugs in this ablation.","section":"Section 5.5, Table 3"},{"comment":"The sentence 'DeepRESTs multi-learning approach doesn't impact the rate of requests it can send as it achieves a high number of requests when testing endpoints' is unclear; if the intended meaning is that DeepREST is request-inefficient, please rephrase.","section":"Section 5.3"},{"comment":"There are two Kim et al. 2023 entries in the reference list (the NLP-enhanced REST testing paper and the ARAT-RL paper); please disambiguate them with distinct citation keys in the text and bibliography.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the undefined BitBucket deduplication oracle. If the authors cannot provide a valid oracle for closed-source services, the bug-count comparison should be restricted to the open-source services, which would still support a useful contribution but would weaken the headline margin. The pre-training overlap issue should also be resolved with a concrete exclusion list before the 'unseen APIs' claim is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a genuinely new state representation for RL-based REST API fuzzing: a pre-trained RoBERTa embedding of HTTP response text concatenated with four functional features, fed to a DQN. That is a real step beyond ARAT-RL's tabular Q-learning and DeepREST's multi-armed bandits. The authors release code, evaluate on 26 APIs including production-grade ones, and the ablation of seven reward functions plus the transformer ablation gives useful engineering insight. The SHAP analysis is a nice touch. The paper is also honest about its limitations, including the constrained action set.\n\nThe soft spot is exactly what the stress-test note flags. Section 5.2 defines a unique bug as a 5XX whose originating source line differs from all previously counted bugs. That oracle requires source-code access, which the paper says is impossible for closed-source BitBucket. Yet Table 2 reports BitBucket unique bugs for every tool, and no alternative deduplication rule is stated. APIRL's margin over EvoMaster on BitBucket is only 6.6 vs 6.2; the other margins on VAmPI and WordPress overlap within one standard deviation. So the \"at least 6.4% more bugs\" claim is anchored to an unverified metric. That is a load-bearing gap, not a nitpick.\n\nTwo smaller issues. The transformer pre-training set came from public APIs on SwaggerHub, and the paper never states whether the 26 test services were excluded. If any of them overlap, the embedding could have memorized response patterns, and the claim of generalization to unseen APIs weakens. Also, Section 5.4 has sloppy percentage language (\"117.6% and 99.2% fewer bugs\" is presumably meant the other way around) and a minor contradiction about whether Rand-APIRL occasionally finds more bugs. These are text-level fixable problems.\n\nMy overall take: the method is plausible and the ablation study gives real evidence that the transformer embedding helps. The evaluation design has a concrete hole, but it is a fixable one. I would send this to peer review with a request for major revision: the authors need to state how BitBucket bugs were deduplicated, run a sensitivity analysis (e.g., per-operation dedup), and confirm that the test services were excluded from transformer pre-training. Once that is done, the headline claim can be properly evaluated. The paper deserves serious referee time.","headline":"A genuinely new state representation for RL-based REST API fuzzing, but the headline bug-count claim rests on an undefined oracle for BitBucket and needs clarification before it can be accepted.","tokens_in":17170,"tokens_out":2914,"would_cite":true,"duration_ms":26312,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep reinforcement learning agent trained on one API can find more bugs in unseen REST APIs than state-of-the-art fuzzers, using fewer test cases.","keywords":["REST API fuzzing","deep reinforcement learning","Deep Q-Network","RoBERTa transformer embedding","black-box testing","HTTP status code reward","mutation-based fuzzing","API bug detection"],"falsifier":"Re-run the BitBucket experiments counting unique bugs per operation rather than by distinct source lines, since line attribution is impossible for that closed-source service; if APIRL's 6.6 bugs no longer exceed the attention-based baseline's 5.0 (or the ordering changes), the headline bug-count advantage is an artefact of the deduplication oracle.","tokens_in":16115,"feed_emoji":"🐛","tokens_out":9625,"duration_ms":72517,"temperature":0.7,"pith_summary":"The paper tries to establish that a single deep reinforcement learning agent, trained once on one deliberately buggy open-source API, can mutate HTTP requests to find more server-side bugs in unseen real-world REST APIs than six established black-box fuzzers, while sending fewer test cases. APIRL frames REST API testing as a Markov decision process over a fixed 23-action mutation menu, and feeds the agent a state built from the API's JSON and natural-language response via a pre-trained transformer. The authors' central argument is that this response embedding gives the policy feedback that status codes alone cannot provide, which is what lets it generalise to endpoints it has never seen. If the claim is right, black-box API fuzzing becomes substantially cheaper: one pre-trained policy can be deployed directly on a new API instead of running per-API search heuristics. The paper also reports an ablation study showing that the choice of reward function and the transformer embedding both materially change the policy's behaviour.","feed_headline":"RL fuzzer finds more REST API bugs in fewer requests","feed_subtitle":"One policy trained on a single API transfers to unseen endpoints, beating six baselines on 26 services.","key_machinery":"The load-bearing object is the transferable mutation policy: a Deep Q-Network over a fixed 23-action mutation space whose actions include authorisation-token changes, parameter switching, type changes, duplication, removal, file-extension appends, wildcards, default and related values, and admin flags. The state is a 772-vector formed by concatenating four functional features (HTTP method, HTTP response code, current parameter type, and normalised parameter index) with a 768-dimensional embedding from a RoBERTa transformer pre-trained by masked language modelling on 2,566 HTTP responses from 1,283 operations across 103 public APIs. The reward is the HTTP status code alone: +10 for a 5XX server error, +1 for a 2XX success, and −1 otherwise. The transformer embedding is the mechanism the paper credits for letting the agent read the content of responses, not just their status codes, and thereby generalise across endpoints.","core_discovery":"APIRL's central claim is that a Deep Q-Network, whose state concatenates four functional features with a 768-dimensional embedding produced by a RoBERTa transformer pre-trained on HTTP responses from 103 public APIs, learns a mutation policy that transfers to unseen endpoints. The policy is target-agnostic: the same state-action space applies to any OpenAPI-described REST API, so after training the agent runs black-box without further rewards or coverage feedback. In evaluation across 26 APIs and 823 operations, APIRL reports at least 6.4% more unique bugs than each of six baselines, with larger average margins, while using fewer requests than most baselines. Ablations attribute part of the gain to the transformer embedding: removing it reduces coverage by 8.4%, and feature-importance analysis ranks the embedded response features as the most influential input. A status-code reward that gives the largest incentive to 5XX responses outperforms coverage-based, uniform, ratio-based, and tabular-Q-learning reward variants.","pith_inferences":["Editorial extension: a decisive test of the transfer claim would be to swap the RoBERTa embedding for a fixed non-contextual vector of the same length and measure the coverage and bug gap on the same 26 APIs.","Editorial extension: because the bug oracle counts 5XX responses by distinct source lines, the closed-source BitBucket numbers rest on an unstated deduplication rule; re-analysing that service by counting 5XX responses per operation would show whether the headline bug margin is robust.","Editorial extension: the paper's own limitation section notes that APIRL cannot add arbitrary key-value pairs beyond schema parameters; coupling the learned policy with a generative key-value proposal module is a natural extension that would likely recover the seven missed bugs the paper attributes to that restriction."],"forward_implications":["One policy trained on a single didactic API can be deployed black-box on unseen REST APIs without retraining or live coverage feedback.","Testing budgets shrink: APIRL finds at least 6.4% more unique bugs than each baseline while using fewer requests than most, and the per-request coverage efficiency is on average 64.8% higher.","The transformer's response embedding is doing real work: ablation shows an 8.4% coverage drop without it, and feature-importance analysis ranks embedded response features as the most influential state component.","Reward design materially changes the learned behaviour: a sparse status-code reward with a 5XX bonus outperforms coverage-based, uniform, ratio-based, and tabular-Q-style rewards, and a PPO variant falls into local optima, reinforcing the choice of DQN."],"supporting_citations":[{"why":"Supplies the Deep Q-Network algorithm that APIRL's agent is built on.","marker":"Mnih et al. 2015"},{"why":"Supplies the RoBERTa transformer used to embed HTTP responses into the 768-dimensional state.","marker":"Liu et al. 2019"},{"why":"Supplies prioritised experience replay, which the DQN training uses to stabilise learning.","marker":"Schaul et al. 2016"},{"why":"Provides the mutation action set and the Python coverage hook APIRL reuses, and motivates the hardness of REST API bugs.","marker":"Atlidakis, Godefroid, and Polishchuk 2019"},{"why":"Provides Generic University, the open-source API with known bugs used to train the policy.","marker":"Paxton-Fear 2023"},{"why":"Defines the ARAT-RL tabular-Q baseline and the reward scheme that the ablation study compares against.","marker":"Kim et al. 2023"},{"why":"Defines the MINER attention-based neural fuzzing baseline that APIRL must beat.","marker":"Lyu, Xu, and Ji 2023"},{"why":"Defines the DeepREST multi-learning baseline, combining deep RL with multi-armed bandits.","marker":"Corradini et al. 2024"}],"fun_headline_variants":["RL fuzzer with transformer embedding finds more REST API bugs","APIRL: Deep RL fuzzing that transfers to unseen REST APIs","Fewer test cases, more bugs: RL fuzzer for REST APIs","Transformer-boosted RL fuzzer outperforms six baselines on REST APIs","Deep RL fuzzer for REST APIs: more bugs, fewer requests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline comparison rests on the bug oracle: a 5XX response counts as a new unique bug only when it is attributed to a source line not seen before, and for closed-source BitBucket the paper gives no alternative deduplication rule, so the claimed margins presuppose that line attribution (or an unstated substitute) works consistently for every service under test.","fun_headline_variants_meta":{"raw":{"variants":["RL fuzzer with transformer embedding finds more REST API bugs","APIRL: Deep RL fuzzing that transfers to unseen REST APIs","Fewer test cases, more bugs: RL fuzzer for REST APIs","Transformer-boosted RL fuzzer outperforms six baselines on REST APIs","Deep RL fuzzer for REST APIs: more bugs, fewer requests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00031,"raw_usage":{"total_tokens":1756,"prompt_tokens":922,"completion_tokens":834,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":736}},"tokens_in":538,"tokens_out":834,"duration_ms":7204,"temperature":1.0,"reasoning_tokens":736,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:53:24.178891+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the BitBucket experiments counting unique bugs per operation rather than by distinct source lines, since line attribution is impossible for that closed-source service; if APIRL's 6.6 bugs no longer exceed the attention-based baseline's 5.0 (or the ordering changes), the headline bug-count advantage is an artefact of the deduplication oracle.","supporting_citations":[{"cited_title":"A.; Veness, J.; Bellemare, M","cited_arxiv_id":null,"evidence_quote":"Supplies the Deep Q-Network algorithm that APIRL's agent is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RoBERTa transformer used to embed HTTP responses into the 768-dimensional state."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the mutation action set and the Python coverage hook APIRL reuses, and motivates the hardness of REST API bugs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Generic University, the open-source API with known bugs used to train the policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MINER attention-based neural fuzzing baseline that APIRL must beat."}],"review_version":1}