{"id":"9598ac92-78be-46e0-933c-7b2846759bf2","arxiv_id":"2606.10417","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An empirical study extracts 20,729 expected behaviors from ten Java libraries and finds 17.5% remain untested, independent of line coverage and mutation scores.","lead":"This paper introduces an automated approach to extract expected method behaviors from documentation and code, then map them to tests to find untested behaviors. Smart generalists might read it to understand a potential new way to judge if test suites are truly complete beyond traditional coverage numbers.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Behaviour extraction precision (93.1%) and test-mapping correctness are the load-bearing assumptions for the 17.5% gap claim","rationale":"The reader's weakest_assumption correctly isolates the single point whose failure would invalidate the quantitative claim. No other internal inconsistency is visible from the abstract; the independence from coverage/mutation scores is a secondary observation that inherits the same extraction dependency.","tokens_in":1815,"tokens_out":330,"duration_ms":11022,"concrete_test":"Take the 100 methods with the largest number of extracted behaviours; have two independent annotators re-extract expected behaviours from the Javadoc + source using the paper's own definition; recompute the fraction of behaviours that have no corresponding test case under the paper's mapping rule; if the gap changes by >5 percentage points the 17.5% figure is sensitive to extractor error.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (17.5% of extracted behaviours untested) is computed directly from the set of 20,729 behaviours obtained by the automated extractor. The paper states this extractor achieves 93.1% precision on a (presumably sampled) validation set, then maps behaviours to tests to decide which remain unvalidated. If the true precision on the full corpus is lower, or if the mapping rule (whatever it is) systematically misclassifies behaviours as \"mapped\" when the test does not actually exercise the documented expectation, the gap figure is not reliable. The claim that the estimate is \"conservative\" does not address directional bias in either step.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper claims to introduce an automated proof-of-concept approach that extracts expected method-level behaviours from natural language documentation and source code, maps them to test cases, and identifies behavioural gaps in test suites. Evaluating on ten Java libraries (8,922 methods), it extracts 20,729 behaviours with 93.1% precision, conservatively estimating a 17.5% behavioural gap (untested expected behaviours). It further shows that automated test generators like EVOSUITE and ASTER leave similar or larger gaps (20.6%/27.1%), and that these gaps are not predicted by line coverage or mutation kill scores, positioning behavioural coverage as an independent adequacy dimension.","tokens_in":2004,"tokens_out":450,"duration_ms":17227,"significance":"If the extraction precision and mapping accuracy hold, this work provides the first large-scale empirical evidence that traditional structural metrics miss a substantial portion of expected behaviours, suggesting a new complementary metric for test suite assessment. The scale of the study and the comparison to automated tools are strengths that could influence both research and practice in software testing.","major_comments":[{"comment":"Abstract: The 17.5% behavioural gap is computed directly from the 20,729 extracted behaviours whose extraction is reported at 93.1% precision; however, no details are supplied on the validation-set size, sampling procedure, or how false positives were adjudicated, leaving the reliability of the gap figure and the 'conservative' qualifier unsupported.","section":"Abstract"},{"comment":"Evaluation section: The mapping rule that decides whether an extracted behaviour is 'validated' by a test case is described only at high level; without an explicit definition or error-rate measurement for the mapping step, it is impossible to rule out systematic bias that would inflate or deflate the reported 17.5% gap.","section":"Evaluation section"}],"minor_comments":[{"comment":"Abstract: The phrase 'conservatively estimates' is used without explaining which sources of under-counting (e.g., missed behaviours or extraction false negatives) are being treated conservatively.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments that identify areas where additional methodological transparency is needed. We address each point below and will revise the manuscript to supply the requested details.","responses":[{"response":"We agree that the current manuscript does not report the validation-set size, sampling procedure, or adjudication criteria for the 93.1% precision figure. In the revised version we will add these details (including the number of behaviours manually inspected, how the sample was drawn, and the process used to classify false positives) so that readers can assess the reliability of the precision and the conservative behavioural-gap estimate.","revision_made":"yes","referee_comment":"[Abstract] Abstract: The 17.5% behavioural gap is computed directly from the 20,729 extracted behaviours whose extraction is reported at 93.1% precision; however, no details are supplied on the validation-set size, sampling procedure, or how false positives were adjudicated, leaving the reliability of the gap figure and the 'conservative' qualifier unsupported."},{"response":"The referee is correct that the mapping rule is presented at a high level and lacks an explicit definition or measured error rate. We will revise the evaluation section to supply a formal definition of the validation mapping (including decision criteria and illustrative examples), report any available error-rate or agreement statistics, and discuss potential systematic biases and their possible effect on the 17.5% gap figure.","revision_made":"yes","referee_comment":"[Evaluation section] Evaluation section: The mapping rule that decides whether an extracted behaviour is 'validated' by a test case is described only at high level; without an explicit definition or error-rate measurement for the mapping step, it is impossible to rule out systematic bias that would inflate or deflate the reported 17.5% gap."}],"tokens_in":1460,"tokens_out":399,"duration_ms":14680,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core contribution is an empirical measurement showing that 17.5% of expected behaviors pulled from documentation stay untested across ten Java libraries, and that this holds even in methods with high line coverage or mutation kill scores. They also check that EvoSuite and Aster leave similar or larger gaps.\n\nThe work does a few things right. Running the extractor on 8,922 methods and 20,729 behaviors gives a concrete scale. The comparison to two automated test generators and the check against structural metrics provide direct evidence that behavioral gaps are not just a side effect of weak testing or low coverage. That part of the argument lands.\n\nThe soft spot is the extraction and mapping process. The 93.1% precision figure comes from a sampled validation set, but the paper gives little detail on how the sample was chosen, what kinds of errors occurred, or whether the same precision holds on the full corpus. The mapping from extracted behaviors to test cases is described at a high level; if that step systematically over-credits tests or if documentation itself is incomplete, the 17.5% gap shifts. The stress-test note is correct here—the headline result sits directly on those two steps, and calling the estimate conservative does not fix possible directional bias.\n\nThis paper is for researchers working on test adequacy metrics who want to explore dimensions beyond structural coverage. A reader already familiar with mutation testing and documentation analysis will see the value in the independence claim, but will want the validation data before treating the exact percentage as settled.\n\nIt deserves peer review so the extraction validation and mapping rules can be examined in detail.","headline":"The paper extracts expected behaviors from docs to measure a 17.5% untested gap and shows it is independent of coverage and mutation scores, but the extraction precision and mapping steps need more validation to support the number.","tokens_in":2424,"tokens_out":415,"would_cite":false,"duration_ms":14735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Test suites leave 17.5% of documented expected behaviors untested even when line coverage and mutation scores are high.","keywords":["test suite adequacy","behavioural coverage","test gaps","mutation testing","code coverage","automated test generation","Java libraries"],"falsifier":"A replication study on the same or similar libraries that uses different extraction rules and finds the share of untested behaviours to be near zero or strongly predicted by line coverage and mutation scores.","tokens_in":2714,"feed_emoji":"🧪","tokens_out":594,"duration_ms":18397,"temperature":0.7,"pith_summary":"The paper introduces an automated approach to pull expected method behaviors from documentation and code, then checks which ones existing tests actually cover. Across ten Java libraries it finds that 17.5% of these behaviors have no matching test at all. The same gaps appear in methods that already achieve high line coverage and high mutation kill scores. Automated test generators produce similar shortfalls. The results position behavioral coverage as a distinct dimension of test adequacy that structural metrics do not capture.","feed_headline":"Test suites miss 17.5% of expected behaviors","feed_subtitle":"Ten Java libraries show the gap persists in methods with high line coverage and mutation scores.","key_machinery":"An automated extraction method that pulls expected method-level behaviours from documentation and source code and maps them to test cases to reveal untested behaviours.","core_discovery":"The approach extracts expected method-level behaviours from natural language documentation and source code, maps them to existing test cases, and identifies gaps between expected and validated behaviours. Our empirical analysis conservatively estimates that 17.5% of detected expected behaviours remain entirely untested, which we term as the test suite's behavioural gap. Behavioural coverage acts as an independent dimension of test suite adequacy that can complement traditional structural metrics.","pith_inferences":["Teams could add documented-behaviour checks to existing test reports without replacing coverage tools.","The extraction technique might surface mismatches between documentation and implementation that developers would otherwise miss.","The same measurement could be applied to non-Java codebases or to evolving documentation over time."],"forward_implications":["17.5% of extracted expected behaviours remain untested in the ten libraries studied.","EvoSuite and ASTER leave at least 20.6% and 27.1% of behaviours unvalidated respectively.","Most untested behaviours occur inside methods that already have high line coverage.","Over half of the gaps remain inside methods that have high mutation kill scores."],"fun_headline_variants":["17.5% expected behaviors go untested","Gaps in 17.5% of method level behaviors","High coverage fails to close behavioral gaps","Mutation testing misses 17.5% behaviors","Behavioral coverage independent of kill scores"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Expected behaviours can be extracted from documentation and source code at 93.1% precision and mapped reliably to test cases.","fun_headline_variants_meta":{"raw":{"variants":["17.5% expected behaviors go untested","Gaps in 17.5% of method level behaviors","High coverage fails to close behavioral gaps","Mutation testing misses 17.5% behaviors","Behavioral coverage independent of kill scores"]},"model":"grok-4.3","cost_usd":0.008961,"raw_usage":{"total_tokens":4056,"prompt_tokens":728,"num_sources_used":0,"completion_tokens":67,"cost_in_usd_ticks":89612000,"prompt_tokens_details":{"text_tokens":728,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3261,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":728,"tokens_out":67,"duration_ms":24179,"temperature":1.0,"reasoning_tokens":3261,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T12:44:13.729772+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A replication study on the same or similar libraries that uses different extraction rules and finds the share of untested behaviours to be near zero or strongly predicted by line coverage and mutation scores.","supporting_citations":[],"review_version":1}