{"id":"0f14176b-c16e-4d66-a105-a8ad92c3fe79","arxiv_id":"2501.18145","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ASTRA learns constraints from 4xx response messages using LLMs, then regenerates API test cases, cutting invalid requests and achieving high coverage with far fewer API calls than prior tools.","lead":"This paper presents ASTRA, a tool that reads the error messages returned by web APIs and uses them to automatically fix its test inputs, reducing invalid requests. It reports high API coverage with far fewer requests than existing search-based testing tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline efficiency claim is contradicted by the paper's own Table 4: ASTRA totals 925 requests while EvoMaster totals 343, so 'lesser number of API requests' holds only against MOREST and ARAT-RL, not against all state-of-the-art search-based tools as claimed.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing problem is not the one identified in the reader's weakest_assumption. The reader focused on external validity of the 14-category taxonomy, which is a genuine generalization threat. However, the paper contains a more direct internal contradiction: its own Table 4 shows EvoMaster making 343 total requests versus ASTRA's 925, so the abstract's 'lesser number of API requests, as compared with the state-of-the-art search-based API Testing tools' is not supported across the full baseline set. The hit-limit experiments cap baselines at ASTRA's request count, which tests performance under ASTRA's budget but does not demonstrate that ASTRA is the most request-efficient tool. This does not destroy the paper's contribution—ASTRA may still be valuable relative to MOREST and ARAT-RL—but it requires a scoped claim and a verification of request-count equivalence. Because the paper can be revised to address this, I keep the conditional verdict rather than moving to reject.","tokens_in":17711,"tokens_out":7235,"duration_ms":72332,"concrete_test":"Re-run the four tools on the same 11 benchmarks with a network-level proxy such as mitmproxy, recording every HTTP request per tool including authentication, retries, and setup traffic, and recompute Table 4 from those logs. If EvoMaster's corrected request count remains at or below ASTRA's—as the current table already suggests—then the claim of 'lesser number of API requests' compared with state-of-the-art search-based tools is false as stated and must be scoped to MOREST and ARAT-RL. If the corrected counts reverse the comparison, the efficiency claim would be restored and the concern resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section 1 assert that ASTRA achieves high coverage with a lesser number of API requests than state-of-the-art search-based API testing tools. Table 4 reports ASTRA total operation hits as 925, MOREST as 188,676, and ARAT-RL as 429,204, but EvoMaster—which is explicitly one of the three baseline tools and is search-based—makes only 343 total requests. On 9 of the 11 benchmarks ASTRA makes more requests than EvoMaster (e.g., gestao 186 vs 60, market 132 vs 26, user 134 vs 60). The hit-limit experiments do not repair this problem because they cap the baselines at ASTRA's observed request count, showing how the baselines behave under ASTRA's budget rather than establishing that ASTRA is the minimal-request tool. The central contribution is request-efficiency plus coverage, so the comparison set and the precise definition of a counted request (for EvoMaster, whether 'action evaluations' equal HTTP calls) are load-bearing. The reader's taxonomy-overlap concern is a real external-validity threat, but Table 4 is an internal inconsistency in the reported evidence for the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASTRA, a black-box REST API testing tool that iteratively learns constraints from 4xx response messages using an LLM-based agent, updates an extended specification model, and generates new test cases. The evaluation compares ASTRA with MOREST, EvoMaster, and ARAT-RL on 11 public APIs, reporting operation coverage, 2xx coverage, request counts, and defect detection. The central claims are that ASTRA reduces 4xx responses, achieves high coverage, and does so with fewer API requests than state-of-the-art search-based tools.","tokens_in":17926,"tokens_out":6909,"duration_ms":56866,"significance":"If the claims hold, ASTRA would be a valuable contribution to black-box API testing, particularly for cost-sensitive industrial settings where request budgets are limited. The paper contributes a specification model that can represent constraints not expressible in OpenAPI, a taxonomy of 14 constraint categories, and an iterative greedy refinement algorithm. The evaluation is based on live APIs and the artifacts appear to be released. However, the significance is currently weakened by an overgeneralized efficiency claim and by the overlap between the corpora used to build the taxonomy and the evaluation benchmarks.","major_comments":[{"comment":"The abstract and Section 1 claim that ASTRA obtains high coverage with 'a lesser number of API requests' compared with state-of-the-art search-based API testing tools. Table 4 reports ASTRA total operation hits of 925 against EvoMaster's 343, and on 9 of the 11 benchmarks EvoMaster makes fewer requests than ASTRA (e.g., gestao 60 vs. 186, market 26 vs. 132). The hit-limit experiment in Section 3.2.2 does not repair this discrepancy because for EvoMaster the cap binds on only two benchmarks (scs and ncs) where EvoMaster's unprotected hits exceed ASTRA's; elsewhere EvoMaster is already under ASTRA's budget. The request-efficiency claim must be scoped to the two baselines for which it holds, or the paper must discuss why EvoMaster's lower request count does not undermine the headline comparison.","section":"Abstract, Section 1, Table 4"},{"comment":"The 14-category taxonomy and the blank-response heuristics (Section 2.3) were derived from a corpus of responses that includes 17 APIs listed in Table 1, and the 11 evaluation benchmarks in Table 3 overlap with this corpus substantially (petstore, gestao, market, user, problem, langtool, scs, ncs, restcountries, person). Consequently, the statement in Section 3.4 that 'we could map each failure in the benchmark to one of the defined categories' partly reflects the fact that the categories were constructed from these very benchmarks. The 95.5% BERT classification accuracy is reported on a random split of the same corpus and is not a measure of performance on genuinely unseen APIs. The external validity of the taxonomy and the classifier should be evaluated on a held-out set of APIs that were not used in the construction.","section":"Section 2.3.1, Tables 1 and 3"},{"comment":"For EvoMaster, the hit limit is imposed via 'maxActionEvaluations', but the paper compares 'operation hits' across tools without establishing that an EvoMaster action evaluation corresponds one-to-one with an HTTP request. If action evaluations include internal or filtered actions that do not result in HTTP calls, the request counts in Table 4 are not comparable across tools. The authors should define the counting unit precisely and, if needed, instrument both tools to count actual HTTP requests.","section":"Section 3.2.2, Table 4"}],"minor_comments":[{"comment":"In the ProducerConsumer example, 'consParam=deleteorder.path.orderId' is missing a capital 'O' in 'deleteOrder'; this is inconsistent with the rest of the text.","section":"Section 2.1.3"},{"comment":"The BERT classifier accuracy of 95.5% is reported from a single 80:20 split without the number of seeds or standard deviation; please report the variance or use repeated stratified splits.","section":"Section 2.3.1"},{"comment":"The word 'significant' is used without a statistical test; the reader cannot tell whether the changes in Fig. 6 are due to random variation.","section":"Section 3.3.2"},{"comment":"Please clarify how the average of three ASTRA runs is converted into an integer hit budget for the baseline tools.","section":"Section 3.2.2"},{"comment":"The threat to validity section acknowledges the non-exhaustive category list but does not address the overlap between the taxonomy-construction corpus and the evaluation benchmarks; a sentence on this would be helpful.","section":"Section 3.4"},{"comment":"The anonymous repository link is good; please ensure the version of the code and the exact tool configurations are pinned for reproducibility.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The paper's central idea is promising, but the abstract currently overstates the request-efficiency result. The EvoMaster comparison must be handled honestly, either by rephrasing the claim or by providing a separate analysis. The taxonomy overlap is a common issue in this line of work, but given that the paper explicitly reports the category mapping as evidence, the authors should be asked to address it. If the authors can re-scope the claims and add a held-out evaluation, the paper would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new piece here is ASTRA's mechanism: an LLM parses natural-language 4xx response messages, maps them onto a fixed taxonomy of 14 constraint types, and feeds those constraints back into test generation. That is a real step beyond MOREST (which used only status codes), ARAT-RL (RL rewards from response data), and Kim et al. (NLP on specification text). The evaluation supports the core idea on most benchmarks: ASTRA gets higher 2xx operation coverage than MOREST, EvoMaster, and ARAT-RL on 9 of 11 APIs, and it does so with two to three orders of magnitude fewer requests than MOREST (188,676) and ARAT-RL (429,204). The hit-limit experiments are a fair way to show that those search-based tools collapse under ASTRA's budget. The artifact is runnable, and the authors are honest about several limitations.\n\nNow the soft spots, in proportion. The headline claim in the abstract and Section 1—that ASTRA achieves high coverage with a lesser number of API requests compared with state-of-the-art search-based tools—is contradicted by their own Table 4. EvoMaster, which is search-based and one of the three baselines, makes only 343 total requests versus ASTRA's 925; on 9 of 11 benchmarks ASTRA makes more requests than EvoMaster. The stress-test note is right. The efficiency claim holds only against MOREST and ARAT-RL, and the paper should say so.\n\nSecond, the constraint taxonomy was derived from 1,663 unique responses collected from 17 APIs that overlap heavily with the 11 evaluation benchmarks. That is a genuine external-validity threat: the evaluation cannot separate whether ASTRA generalizes to unseen APIs or simply remembers the response styles it was built on. The authors flag this as a threat, but the current design cannot rule out the stronger concern.\n\nThird, results are averages of 3 runs with no variance or statistical testing. With benchmarks where request counts are small (person: 42 vs 23), that matters. Fourth, the defect counts in Table 6 are compared only against MOREST and not manually triaged; a 5xx response is not necessarily a true API defect. Fifth, JaCoCo coverage instrumentation failed on 4 of 11 APIs, which weakens the line/branch coverage comparison.\n\nThese are real but not fatal. The central mechanism—learning from error text to refine tests—holds up as plausible and useful. The paper deserves a serious referee. I would send it out with a clear ask: fix the EvoMaster request comparison, add held-out APIs or otherwise separate taxonomy discovery from evaluation, report variance or significance, and triage the defects. This is for API testing researchers and practitioners with rate-limited APIs. I would cite the constraint-extraction idea in future work.","headline":"The core idea—extracting constraints from natural-language 4xx messages—is genuinely new, but the paper's own Table 4 undermines the 'fewer requests than all SOTA tools' claim, since EvoMaster uses fewer requests than ASTRA.","tokens_in":18525,"tokens_out":1912,"would_cite":true,"duration_ms":19302,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A black-box REST API tester can learn constraints from 4xx response messages and reach high operation coverage with one to three orders of magnitude fewer requests than search-based tools.","keywords":["REST API testing","black-box testing","test refinement","response message analysis","constraint inference","large language models","search-based testing","OpenAPI"],"falsifier":"Take ASTRA and one benchmark API, and rewrite the API's error responses to a fixed generic string such as 'Bad request' while keeping the status codes unchanged. If the valid-test ratio and request count stay roughly the same, then response-message content is not the source of ASTRA's learning; if they degrade sharply, the central claim is confirmed.","tokens_in":17424,"feed_emoji":"🧪","tokens_out":7558,"duration_ms":71112,"temperature":0.7,"pith_summary":"Most automated REST API testing either trusts the OpenAPI specification, which is often incomplete, or learns by making tens of thousands of requests, which is too expensive for industry use. This paper argues that the error message itself carries the missing information: a 4xx response names the parameter, operation, or data value that caused rejection. The proposed tool, ASTRA, reads each failure message, classifies it into one of 14 constraint categories, and adds the inferred constraint to an extended specification model before generating the next round of tests. The result, the paper reports, is a testing loop that converges after 6 to 186 requests per benchmark API and produces a higher share of valid 2xx tests than state-of-the-art search-based tools while reaching comparable or better operation coverage.","feed_headline":"API tester reads 4xx errors, hits coverage on far fewer calls","feed_subtitle":"ASTRA turns client-error text into constraints, beating three search-based tools with a tiny fraction of the calls.","key_machinery":"The load-bearing mechanism is the feedback loop around an extended specification model $Spec := \\langle O,S\\rangle$: execute generated tests, log unique 4xx responses, classify each response into one of 14 constraint categories using an LLM, have an agent extract the constrained entities and values, add the resulting constraint object to the model, and regenerate tests until an iteration introduces no new failure. Parameter-selection constraints are compiled into SMT constraints so the generator can enumerate compliant parameter scenarios, and data constraints are turned into natural-language prompts for a small language model that proposes realistic values, verified by a constraint solver. This loop is what lets ASTRA learn from a handful of responses instead of tens of thousands.","core_discovery":"ASTRA's central claim is that natural-language 4xx responses can be converted into machine-readable test constraints, so that each failed request teaches the tester how to pass. The paper derives a taxonomy of 14 response categories from 1,663 unique messages, covering operation-level prerequisites, parameter-selection rules, data-value restrictions, and nested dependencies. An LLM classifies each failure and an agent extracts the entities and values needed to form a constraint, which is stored in an extended specification model. Test generation then re-runs with the new constraints, producing operation sequences, parameter scenarios, and data values that avoid the previously rejected request. On 11 public benchmark APIs, the paper reports that ASTRA achieves the highest operation coverage on 9 APIs and the highest successful (2xx) operation coverage on 9 APIs, and that it does so with 925 total requests versus 188,676 for MOREST, 343 for EvoMaster, and 429,204 for ARAT-RL across the benchmark set.","pith_inferences":["An implication the paper leaves implicit is that its request budget makes functional API testing economical for pay-per-call or rate-limited services, where tens of thousands of requests would be infeasible; the paper frames the cost motivation but does not quantify the savings.","A natural extension is to feed ASTRA's learned constraints into a white-box or coverage-guided phase, since the paper notes its branch coverage trails EvoMaster when no request limit is imposed.","The approach's dependence on informative error text could be tested head-to-head by running ASTRA against a mirrored API whose messages are replaced with generic 'Bad request' strings; this would isolate how much of the learning comes from the message content rather than the status code.","The 14-category taxonomy was built from responses collected from APIs that overlap heavily with the 11 evaluation benchmarks, so transfer to unseen API domains with novel error phrasing is an open question rather than an established property."],"forward_implications":["ASTRA reaches 100% operation coverage on most of the 11 benchmark APIs and matches or exceeds the best baseline on 9 of them, while its total request count is 925 compared with 188,676 for MOREST, 343 for EvoMaster, and 429,204 for ARAT-RL.","When the baselines are given the same request budget as ASTRA, their operation coverage drops sharply, which the paper reads as evidence that reinforcement-learning and search-based exploration need much more data to learn the same constraints.","The iterative loop shifts the response mix away from 4xx toward 2xx and 5xx over time; the 5xx responses expose defects, and ASTRA reports 33 unique defects, 13 with stack traces, versus 29 for MOREST without a hit limit.","Blank or uninformative 4xx responses, about 22% of benchmark requests, are handled by fallback heuristics such as treating a 404 on an identifier parameter as a missing producer operation."],"supporting_citations":[{"why":"Provides MOREST, the main model-based baseline whose response-code feedback ASTRA extends, and one of the tools used to collect failure responses.","marker":"[27]"},{"why":"Provides ARAT-RL, the reinforcement-learning baseline that motivates ASTRA's request-efficiency goal.","marker":"[25]"},{"why":"Provides EvoMaster, the search-based baseline used in the coverage and hit-limit comparisons.","marker":"[10]"},{"why":"Supplies the producer-consumer sequence-generation idea that ASTRA's operation-level constraints build on.","marker":"[12]"},{"why":"Supplies the inter-parameter dependency data and analysis that feed into the constraint taxonomy.","marker":"[28]"},{"why":"Shows that natural-language specification text can be used to infer constraints, the prior step ASTRA extends to response text.","marker":"[24]"},{"why":"Supplies the transformer base for the fine-tuned classifier used when the LLM cannot identify a category.","marker":"[17]"},{"why":"Supplies the language model used to propose realistic data values satisfying the learned constraints.","marker":"[40]"},{"why":"Defines the OpenAPI specification format that the extended specification model parses and augments.","marker":"[33]"}],"fun_headline_variants":["4xx errors teach API tester to pass with fewer calls","Turn failed API calls into better tests with ASTRA","API testing that learns from its own 4xx mistakes","ASTRA: Fewer requests, higher coverage via error learning","Stop guessing API input—let 4xx responses guide you"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline collapses if API error messages cannot be mapped to the 14 predefined constraint categories or the language model cannot identify the parameter or operation the message refers to.","fun_headline_variants_meta":{"raw":{"variants":["4xx errors teach API tester to pass with fewer calls","Turn failed API calls into better tests with ASTRA","API testing that learns from its own 4xx mistakes","ASTRA: Fewer requests, higher coverage via error learning","Stop guessing API input—let 4xx responses guide you"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000202,"raw_usage":{"total_tokens":1416,"prompt_tokens":1012,"completion_tokens":404,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":628,"completion_tokens_details":{"reasoning_tokens":322}},"tokens_in":628,"tokens_out":404,"duration_ms":5024,"temperature":1.0,"reasoning_tokens":322,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T00:29:50.399706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take ASTRA and one benchmark API, and rewrite the API's error responses to a fixed generic string such as 'Bad request' while keeping the status codes unchanged. If the valid-test ratio and request count stay roughly the same, then response-message content is not the source of ASTRA's learning; if they degrade sharply, the central claim is confirmed.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides EvoMaster, the search-based baseline used in the coverage and hit-limit comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the producer-consumer sequence-generation idea that ASTRA's operation-level constraints build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the inter-parameter dependency data and analysis that feed into the constraint taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides ARAT-RL, the reinforcement-learning baseline that motivates ASTRA's request-efficiency goal."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the OpenAPI specification format that the extended specification model parses and augments."}],"review_version":1}