{"id":"5d7b0d77-04bf-411a-bc1d-3aaacf319621","arxiv_id":"2504.12790","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using bytecode rather than test source text as the diversity artefact in two static test-prioritisation algorithms yields modest APFD gains (2.3-7.8%) and large runtime savings, with a hand-selected bytecode filter improving speed further.","lead":"This paper tests whether Java bytecode, instead of test source text, can serve as the diversity measure for test case prioritisation. Experiments on seven Defects4J projects report large runtime savings and modest fault-detection gains, with some gains not statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 reports 107 real faults for only 97 selected Defects4J versions, suggesting error-revealing tests may have been counted as distinct faults.","rationale":"The Reader's weakest assumption concerns generalisability of the manually chosen filter set. That is a legitimate external-validity threat, but the manuscript itself contains a more immediate and checkable internal inconsistency: 107 real faults cannot come from 97 single-fault Defects4J versions. Because the paper's stated contributions include effectiveness in 'mutant and real-fault detection' and the abstract's headline improvement is framed as 'fault detection', this issue directly threatens a load-bearing part of the empirical support. It is concrete: the replication package and Defects4J metadata can settle it. If the count is indeed error-revealing tests, the real-fault results must be recomputed at fault level; if the count is a typo, the correction may still change reported numbers. The mutation-APFD and efficiency results remain credible, and the paper is already CONDITIONAL, so my recommendation does not move the verdict. I disagree with the Reader's choice of weakest assumption because the real-fault accounting is a more severe, falsifiable threat to the paper's central empirical claims.","tokens_in":18442,"tokens_out":15270,"duration_ms":156212,"concrete_test":"Open the replication package and Defects4J metadata; for each of the 97 selected versions, list the set of tests that fail on the buggy version and pass on the fixed version. Count distinct versions with at least one such test (this should be 97, not 107). Recompute Table 3 and the RQ2 real-fault comparisons with exactly one data point per version, taking the minimum position among that version's error-revealing tests in each prioritised order. If the FAST-Bytecode versus FAST-Text real-fault advantage (median 32 vs 125, p=0.007, reported as 3.9x) and the filtered-bytecode 'doubling' are no longer significant, those claims should be removed or downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1 lists 97 selected Defects4J versions but 107 'real faults' (project sums: Cli 5/7, Compress 7/7, Csv 10/10, Jsoup 16/16, Lang 15/15, Math 42/50, Time 2/2; total 97/107). Defects4J versions are single-fault subjects, so 107 distinct real faults cannot come from 97 versions. The paper states in Section 4.4.1 that the tool 'runs the test suite against the buggy versions of Defects4J to record all the error-revealing tests', and Section 4.2 defines a fault-revealing test as one that fails on the buggy version and passes on the fixed version. The likely explanation is that the '#Real Faults' column actually counts error-revealing tests, not distinct faults. If so, Tables 3 and 6 aggregate the position of every failing test as if each were a separate fault: a single fault exposed by k tests is counted k times, whereas the correct fault-level detection position is the earliest position among those k tests. This can inflate the sample size and shift medians and p-values, so the reported real-fault gains (FAST-Bytecode 3.9x better than FAST-Text; filtering 'doubled real fault detection') may be artifacts of double-counting. The central mutation-APFD result in Table 2 is not directly affected, but the paper's real-fault contribution and the abstract's generic 'improves fault detection' wording would need re-analysis and rephrasing.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces bytecode of test cases as a diversity artefact for static test case prioritisation, arguing that bytecode is more compact and execution-focused than test text. It instantiates Ledru-TCP and FAST-TCP over textual, full-bytecode, and filtered-bytecode representations, and compares them with two coverage-based greedy techniques. The study uses seven Defects4J projects (97 versions), mutation-based APFD, real-fault detection positions, and wall-clock runtimes. The headline findings are that bytecode diversity improves APFD by 2.3--7.8% over text, is 2--3 orders of magnitude faster for Ledru-TCP and 2.5--6 times faster for FAST-TCP, and that filtering bytecode instructions yields further speedups while maintaining effectiveness.","tokens_in":18749,"tokens_out":6286,"duration_ms":65861,"significance":"If the real-fault analysis is repaired, the paper makes a useful contribution to diversity-based TCP: it is, to my knowledge, the first study of bytecode as a diversity artefact, and the reported runtime gains are large and practically relevant. The authors validate their Ledru-TCP implementation against the original case studies and reuse the FAST replication package, and they provide a replication package of their own; these are concrete strengths. The effectiveness advantage is more modest than the abstract suggests, because the Ledru-Bytecode APFD gain is not statistically significant and the FAST filtering gain on real faults is also not significant.","major_comments":[{"comment":"Table 1 reports 107 real faults for 97 Defects4J versions, which is arithmetically impossible because Defects4J versions are single-fault subjects (the project totals are Cli 5, Compress 7, Csv 10, Jsoup 16, Lang 15, Math 42, Time 2, summing to 97). The likely explanation is that the '#Real Faults' column counts error-revealing tests rather than distinct faults, since Section 4.4.1 says the tool records 'all the error-revealing tests' and Section 4.2 defines a fault-revealing test as failing on the buggy version and passing on the fixed version. If so, Tables 3 and 6 aggregate the position of every error-revealing test as if each were a separate fault, whereas the fault-level detection position should be the earliest position among those tests for each version. This can inflate the sample, shift medians, and distort p-values, so the real-fault claims (e.g., FAST-Bytecode 3.9x better, filtering 'doubled real fault detection') need to be recomputed at fault level or explicitly re-scoped. If the column is merely mislabelled, the table and all dependent conclusions must be corrected.","section":"§4.3, Table 1; §4.4.1 and §4.4.3"},{"comment":"The abstract's claim that 'bytecode diversity improves fault detection by 2.3–7.8% over text-based TCP' is not supported for Ledru-Bytecode: Table 2 gives p=0.453 and A12=0.53 for Ledru-Bytecode versus Ledru-Text, and Table 3 shows Ledru-Bytecode's median real-fault detection position (124) is worse than Ledru-Text's (110). Only the FAST-Bytecode APFD gain is statistically significant (p=0.000, A12=0.67). The RQ1 conclusion should distinguish the significant FAST result from the non-significant Ledru result, and the abstract should not present the 2.3–7.8% range as a uniform improvement.","section":"Abstract; §5.1, Table 2"},{"comment":"The claim that filtering bytecode 'doubled real fault detection' compared with text is not statistically significant: Table 3 reports p=0.297 and A12=0.46 for FAST-Bytecode-Filter versus FAST-Text. The body of Section 5.2 includes this caveat, but the abstract and the RQ2 conclusion present the point estimate without it. Please carry the uncertainty into the abstract and conclusions, or restrict the claim to the APFD results where the comparisons are significant.","section":"Abstract; §5.2, Table 3"},{"comment":"The bytecode filter set (constant pushes, field accesses, method invocations) is an author-design choice rather than a derived or independently validated artifact, and it is evaluated only on the seven projects that were selected because JaCoCo and PIT ran without issues. The external-validity discussion acknowledges this, but the RQ2 conclusion that filtering 'maintains effectiveness' should be scoped to the selected projects rather than stated as a general property of the filtered instruction set.","section":"§3.2, §4.3, §5.2"}],"minor_comments":[{"comment":"The caption of Table 6 says 'APFDs for FAST-bytecode TCP and coverage-based TCP', but the table reports real-fault detection positions; the caption should be corrected.","section":"§5.3, Table 6"},{"comment":"The sentence 'there is a medium effect size in favour of bytecode diversity' is inconsistent with the direction of the result, since coverage-based TCP has the lower (better) median detection position; the effect is in favour of coverage-based TCP.","section":"§5.3"},{"comment":"Multiple pairwise comparisons are made across Tables 2, 3, 5, and 6 without correction for multiple testing. This should at least be noted, because borderline p-values such as Ledru-Bytecode-Filter's APFD p=0.040 would not survive a simple Bonferroni correction.","section":"§4.4.3, Tables 2–6"},{"comment":"The runtime comparison in RQ3 compares FAST-Bytecode's total time (preparation plus prioritisation) with coverage-based approaches' prioritisation-only time. Section 4.2 justifies excluding coverage preparation time, but the asymmetry of the comparison should also be noted at the point where 'up to 6.5 times faster' is stated.","section":"§4.2, Table 4"},{"comment":"There is a typo in 'stubborn mutants (those hardest to detec/t/kill)' where 'detect' is split by a slash; this should be cleaned up.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the 107-versus-97 real-fault discrepancy. I do not see evidence of intentional distortion; the likely cause is counting error-revealing tests as faults. If the authors can supply fault-level detection positions, the real-fault section should be reanalysed; if not, the real-fault contribution should be removed or explicitly demoted. I would ask for the fault-level data in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2504.12790. The genuinely new thing is the bytecode representation for diversity-based TCP, and the efficiency story is real. Bytecode is far more compact than source text, so the reported speedups (124–814x for Ledru, 2.5–6x for FAST) are mechanistically plausible and not a surprise. The authors also ship a replication package, which earns credit. The FAST-Bytecode APFD gain over FAST-Text (7.8%, A12=0.67, p<0.001) is the strongest result. The filter instruction set is a reasonable heuristic, and the runtime reductions from filtering are large.\n\nBut there are two soft spots that matter. First, the real-fault analysis appears to count error-revealing tests as faults. Table 1 reports 107 real faults for 97 Defects4J versions; Defects4J is single-fault per version, so 107 distinct faults can't come from 97 versions. The text says the tool records 'all the error-revealing tests,' which suggests each failing test is being treated as a separate fault. For a single fault exposed by k tests, you'd count k data points instead of one position (the earliest). That inflates the sample and can shift medians and p-values; the reported 3.9x real-fault improvement for FAST-Bytecode and the 'doubled' filtered detection could be artifacts. This needs re-analysis, and the replication package should clarify the unit of analysis.\n\nSecond, the abstract and conclusions overstate the uniform picture. The 2.3–7.8% range mixes Ledru's non-significant APFD gain (p=0.453, A12=0.53) with FAST's significant gain. And the 'maintaining effectiveness' claim about filtering is contradicted for FAST, where the filtered version is 2.7% lower in APFD than all-bytecode (though still better than text). Also, the subject selection is narrow—seven projects where JaCoCo/PIT ran without issues—so generalisation is limited, though the authors do note this.\n\nThe mutation-APFD core is not affected by the real-fault counting issue, and the efficiency results are credible. The paper is worth a serious referee with the expectation of a major revision: fix the real-fault unit of analysis, rebalance the claims, and show runtime variability or at least acknowledge it. If the real-fault results survive correction, this is a solid contribution to the TCP literature.\n\nFor you: I'd bring it to a reading group focused on empirical testing, and I'd cite the bytecode representation and efficiency numbers with the caveat about the real-fault data.","headline":"Bytecode representation for TCP is new and the efficiency gains are credible, but the real-fault analysis counts error-revealing tests as faults and the abstract overstates the mixed significance.","tokens_in":19292,"tokens_out":4430,"would_cite":true,"duration_ms":39065,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that using test bytecode instead of test source text for diversity-based test prioritisation improves fault detection while cutting computation time, and that filtering routine instructions preserves or improves those…","keywords":["test case prioritisation","diversity-based testing","bytecode","textual diversity","bytecode filtering","Levenshtein distance","regression testing","static analysis"],"falsifier":"Run Ledru-Bytecode-Filter and FAST-Bytecode-Filter on held-out Java projects with real faults, using only the paper's filter, and compare median APFD to their text-based counterparts; if the APFD advantage shrinks or reverses on several projects, the filter is project-specific rather than a general property of bytecode.","tokens_in":18209,"feed_emoji":"⚡","tokens_out":7270,"duration_ms":70409,"temperature":0.7,"pith_summary":"The paper introduces bytecode as the diversity artefact for test case prioritisation, replacing the test source text that previous static approaches compare. On seven Java projects covering 97 versions, it reports that bytecode-based prioritisation finds more faults earlier than text-based prioritisation, with median APFD gains of 2.3% to 7.8% depending on the algorithm. The bigger payoff is speed: preparation time drops by 124 to 2798 times in a similarity-matrix method and by 2.5 to 17.9 times in the FAST method. Filtering bytecode down to constant pushes, field accesses, and method calls makes the method up to four times faster still while keeping or improving fault detection. The practical consequence is that static prioritisation, which needs no coverage instrumentation, becomes feasible for large regression suites.","feed_headline":"Bytecode test prioritisation runs 2798x faster, finds faults earlier","feed_subtitle":"Swapping verbose test source for compiled bytecode beats text diversity by up to 7.8%.","key_machinery":"The central object is the bytecode of each compiled test method, obtained with a bytecode-reading library and compared with the same distance metrics used for text: Levenshtein distance for the pairwise similarity matrix in Ledru-TCP, and hexadecimal bytecode fed into MinHash and locality-sensitive hashing signatures in FAST-TCP. The second mechanism is the filter, which keeps only instructions that push constants onto the stack, access fields, or invoke methods while discarding stack loads and stores, variable operations, and returns. That filter carries the argument because it shrinks each test's representation, concentrating similarity on the test's substantive actions and turning bytecode's compactness into both a large speed gain and, in the Ledru variant, a small effectiveness gain over full bytecode.","core_discovery":"The paper's central claim is that a test's compiled bytecode carries enough execution-relevant information for diversity-based prioritisation, without the comments, variable names, and formatting that inflate and distort textual similarity. Implementing Ledru-TCP and FAST-TCP over text, full bytecode, and filtered bytecode, the authors find that bytecode versions achieve higher median APFD (89.9 versus 87.6 for Ledru, and 93.2 versus 85.4 for FAST), and that FAST-Bytecode finds real faults after a median of 32 tests compared to 125 for FAST-Text. Filtering bytecode to instructions that push constants, access fields, or invoke methods raises Ledru's median APFD to 92.1 and makes it 455 to 2798 times faster than text, while FAST-Bytecode-Filter stays 5.1% above text in APFD and is 4 to 17.9 times faster. Coverage-based dynamic TCP still beats bytecode TCP by 2.8% to 3.5% in APFD, but bytecode TCP is 6.5 to 8.3 times faster and avoids program instrumentation.","pith_inferences":["My inference: the specific filter set, chosen by the authors on the same seven projects used for evaluation, is plausible but under-tested; a transfer experiment on new projects would show whether the filter is a general principle or an artefact of the chosen subjects.","My inference: bytecode's advantage probably grows with suite size and with the presence of automatically generated tests, which tend to add textual repetition without adding behavioural diversity; a size-stratified replication could quantify this.","My inference: normalising bytecode to remove compiler-version differences could close part of the remaining gap to coverage-based TCP, because raw bytecode still carries some implementation artefacts unrelated to test behaviour."],"forward_implications":["Large regression suites can use static prioritisation without coverage instrumentation at a small fraction of the previous computation cost.","Text-based Ledru-TCP becomes impractical at scale, while bytecode versions make full pairwise similarity feasible even for suites of tens of thousands of tests.","Filtering gives practitioners a tuning knob: dropping mechanical instructions can sacrifice little or no effectiveness while multiplying speed.","Bytecode TCP remains less effective than coverage-based TCP, so projects with existing coverage pipelines may keep dynamic methods, but bytecode offers a cheaper fallback when coverage data is unavailable.","Because the bytecode artefact is independent of the prioritisation algorithm, the benefit is likely to carry over to other similarity-based TCP methods beyond the two studied."],"supporting_citations":[{"why":"supplies the greedy similarity-matrix TCP algorithm and the textual baseline it is built on","marker":"[21]"},{"why":"supplies the FAST scalable similarity-based TCP approach and the text-based version used as a baseline","marker":"[24]"},{"why":"defines diversity-based testing and justifies the choice of Levenshtein distance for string data","marker":"[8]"},{"why":"supplies the Java benchmark subjects with real faults and multiple versions used in the evaluation","marker":"[17]"},{"why":"provides the bytecode-reading library used to extract test bytecode","marker":"[9]"},{"why":"defines the Levenshtein distance metric applied to both textual and bytecode representations","marker":"[22]"},{"why":"defines test case prioritisation and provides the coverage-based greedy algorithms used as dynamic baselines","marker":"[29]"},{"why":"supplies the Vargha-Delaney effect size measure used for comparing approaches","marker":"[32]"}],"fun_headline_variants":["Bytecode diversity speeds up test prioritisation 2798x","Bytecode test prioritisation: 2798x faster, up to 7.8% more faults","Swap source for bytecode: test prioritisation 2798x faster","Bytecode-based TCP: 2798x speedup, 7.8% better fault detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that similarity over bytecode, and specifically over the authors' hand-picked filtered instruction set, tracks how differently tests exercise the program at least as well as similarity over test text; if that premise fails on other projects, the effectiveness gains will not generalise.","fun_headline_variants_meta":{"raw":{"variants":["Bytecode diversity speeds up test prioritisation 2798x","Bytecode test prioritisation: 2798x faster, up to 7.8% more faults","Swap source for bytecode: test prioritisation 2798x faster","Bytecode-based TCP: 2798x speedup, 7.8% better fault detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001022,"raw_usage":{"total_tokens":4327,"prompt_tokens":978,"completion_tokens":3349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":594,"completion_tokens_details":{"reasoning_tokens":3259}},"tokens_in":594,"tokens_out":3349,"duration_ms":25169,"temperature":1.0,"reasoning_tokens":3259,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:22:04.021704+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Ledru-Bytecode-Filter and FAST-Bytecode-Filter on held-out Java projects with real faults, using only the paper's filter, and compare median APFD to their text-based counterparts; if the APFD advantage shrinks or reverses on several projects, the filter is project-specific rather than a general property of bytecode.","supporting_citations":[{"cited_title":"Ledru, A","cited_arxiv_id":null,"evidence_quote":"supplies the greedy similarity-matrix TCP algorithm and the textual baseline it is built on"},{"cited_title":"Elgendy, R","cited_arxiv_id":null,"evidence_quote":"defines diversity-based testing and justifies the choice of Levenshtein distance for string data"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the bytecode-reading library used to extract test bytecode"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the Levenshtein distance metric applied to both textual and bytecode representations"},{"cited_title":"Vargha and H","cited_arxiv_id":null,"evidence_quote":"supplies the Vargha-Delaney effect size measure used for comparing approaches"}],"review_version":1}