{"id":"e4e22607-b5b3-4ec8-95cf-21ddff6e77f4","arxiv_id":"2412.11073","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"Ba-ZebraConf claims 67% fewer tests and zero false positives and negatives versus ZebraConf, but the evaluation is circular and lacks a defined ground truth.","lead":"A new system finds broken settings in large computer networks faster by testing groups of settings together. It reports 67% fewer tests and zero mistakes, but the tests were tuned on the same data used to measure success.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0% FPR/FNR claim is unverifiable because ground-truth labels are never defined, and BO's objective optimizes FPR/FNR on the same 1,000 files; held-out evaluation with disclosed labels is needed.","rationale":"The reader's rejection is based on the unverifiability of the central accuracy claim, and my independent stress-test points to the same load-bearing weakness. The paper's headline is two-fold: 67% cost reduction and 0% classification errors. The efficiency part is at least concretely measured (test counts and wall-clock time on 1,000 files), though it lacks code/data for reproduction. The accuracy part is not even well-defined: false positive and false negative rates require ground-truth labels for heterogeneous-unsafe parameters, and the paper never states where those labels come from. Section V-F mentions simulated environments with known failure modes, but does not tie them to the 1,000-file evaluation, so the reader is left to guess whether the reported 0% values are computed against real labels, injected faults, or some internal classification criterion.\n\nThe in-sample nature of the evaluation strengthens the concern. The BO objective function includes FPR and FNR directly (Section V-D), and the framework's own description says BO continues to refine hyperparameters during the Exploitation Phase on the same unit test files. Reporting results at the best-performing iteration (X = 179) is therefore a form of post hoc selection. This does not prove the numbers are wrong, but it means the 0% error rates are not an out-of-sample prediction; they are an optimized outcome on the evaluation set. A held-out evaluation with frozen hyperparameters and disclosed ground truth would settle the issue.\n\nI also note the 'Algorithm ??' placeholder and the absence of code or data. These are reproducibility gaps that reinforce the need for a concrete test, but the primary scientific concern is the missing ground truth, not the artifacts alone. I am not arguing that the Bayesian framework is internally inconsistent or that the efficiency gain is impossible; the integration of BGT, BO, and BRR is plausible and could be valuable. My concern is specifically that the strongest claim—perfect classification—is not currently supported by the evidence as presented. Therefore the reader's REJECT verdict remains appropriate, and I would not change it without the additional validation described in the concrete test.","tokens_in":13151,"tokens_out":2280,"duration_ms":21848,"concrete_test":"Split the 1,000 unit test files into a strict 200-file exploration set and an 800-file held-out set. Run BO only on the exploration set, freeze the chosen hyperparameters, then execute Ba-ZebraConf on the held-out set and compare its classifications against independently established ground-truth labels (e.g., injected faults with known unsafe parameters, or manually verified labels). Report FPR and FNR on the held-out set only, and do not select the BO iteration based on held-out performance. If 0% FPR/FNR still holds on held-out data with disclosed ground truth, the concern is resolved; otherwise, the central accuracy claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy claim—0% false positives and false negatives—depends on knowing which parameters are truly heterogeneous-unsafe for each of the 1,000 unit test files, yet the paper never defines how this ground truth is obtained. Section V-F mentions simulated environments with known failure modes, but does not connect them to the 1,000-file evaluation in Section VI. Without ground truth, FPR and FNR cannot be computed; the reported 0% values are therefore not meaningful as stated.\n\nThis is compounded by an in-sample selection problem. The BO module's custom objective (Section V-D) explicitly includes FPR and FNR: Objective = 0.4·TSN + 0.4·CSN − FPR − FNR. The Exploration Phase uses the first 200 unit test files to build the surrogate model, and the Exploitation Phase continues to update hyperparameters using additional files, with the Bayesian Optimization Daemon occasionally testing new combinations on subsets of unit files. The final results are then reported at the best-performing BO iteration (X = 179), chosen because of its large time/test savings and 0% error rates. This is effectively model selection on the same data used for reporting, so the 0% error rates may be fitted rather than predictive.\n\nAdditionally, the implementation section contains an 'Algorithm ??' placeholder, and no code or data artifacts are provided, so the reported numbers cannot be independently reproduced. The efficiency result (67% reduction in tests and time) is a concrete, potentially meaningful claim, but the accuracy claim is the load-bearing part of the headline: if the 0% FPR/FNR are not genuinely obtained on out-of-sample data with defined labels, the framework's advantage over ZebraConf reduces to a cost saving whose correctness is unvalidated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Ba-ZebraConf, a framework that combines Bayesian Group Testing (BGT), Bayesian Optimization (BO), and Bayesian Risk Refinement (BRR) for identifying heterogeneous-unsafe configuration parameters in distributed systems. The authors claim that, compared with the ZebraConf baseline, Ba-ZebraConf reduces test execution time and test count by 67% while achieving 0% false positive and false negative rates, based on experiments over 1,000 unit test files at selected Bayesian optimization iterations.","tokens_in":13528,"tokens_out":3415,"duration_ms":29287,"significance":"If the reported results were valid, Ba-ZebraConf would be a useful contribution to configuration troubleshooting: the three-way integration of probabilistic group testing, cumulative risk refinement, and automated hyperparameter tuning is conceptually interesting and directly targets known limitations of binary-splitting group testing. The paper also gives some credit for explicitly addressing noise and dilution effects through likelihood tuning. However, the accuracy evidence presented is not currently trustworthy: the reported 0% error rates are obtained through an in-sample selection procedure, and no ground-truth definition is provided. The efficiency result (67% reduction in tests and time) is a concrete, potentially meaningful comparison, but it is also under-specified and not reproducible from the manuscript. The central claims therefore need substantial additional validation before the results can be accepted.","major_comments":[{"comment":"The 0% false positive and false negative rates reported in §VI-D are fitted quantities rather than predictive estimates. The BO objective in §V-D is 'Objective = 0.4·TSN + 0.4·CSN − FPR − FNR', and the same 1,000 unit test files used to report the final results are also used to tune hyperparameters during the Exploration and Exploitation phases (§V-D). Results are then reported at the best-performing BO iteration X=179, selected based on its improvements (§VI-A). This in-sample selection can drive FPR/FNR to zero by construction. A held-out evaluation (e.g., separate tuning/validation/test splits) with error bars is required to support the accuracy claim.","section":"§V-D, §VI-A, §VI-D"},{"comment":"False positive and false negative rates are only meaningful if there exists a ground-truth label of 'heterogeneous-unsafe' for every parameter in every one of the 1,000 unit test files. The paper never states how these labels are obtained; §V-F mentions simulated environments with known failure modes, but no connection is made between those simulations and the 1,000-file evaluation in §VI. Without a defined ground-truth protocol, the reported 0% FPR/FNR cannot be verified or interpreted.","section":"§VI-D, §V-F"},{"comment":"The implementation section contains an 'Algorithm ??' placeholder (§V-A), and no code, data, or configuration artifacts are provided. The evaluation also omits basic reproducibility information: what constitutes a 'unit test file', how many parameters are involved, the ranges of the six BO-tuned hyperparameters, and the random seeds. As a result, the 67% reduction in tests/time and the accuracy numbers cannot be independently reproduced or checked for fairness of comparison with ZebraConf.","section":"§V-A, §VI"}],"minor_comments":[{"comment":"The framework is called Ba-ZebraConf in the abstract, but §VI repeatedly refers to 'Bayesian-ZebraConf'; please unify the naming.","section":"§VI"},{"comment":"The abstract claims reduction of average test count, while §VI-C reports total executed test counts (35,426 vs 107,962); clarify whether the reported percentages refer to totals or per-file averages.","section":"§VI-C"},{"comment":"Figure 6 and Figure 7 captions and text report false positive and false negative rates as percentages, but the axes and units are not described; also the sentence 'with Rates of 4% at X=2and 0% for Subsequent Iterations' contains a formatting typo.","section":"§VI-D"},{"comment":"Table I lists 'Two Intel Xeon Silver 4114' and '10 cores per node', then states scale as 'Single-node (10 cores)'; clarify whether both sockets (20 cores total) or only one socket (10 cores) is used.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The reader's report is broadly aligned with my reading: the accuracy claims are not supported because of circularity and the absence of a defined ground truth. I am choosing major_revision rather than reject because the framework itself is coherent and the efficiency claim might survive a properly designed evaluation. If the authors cannot provide ground-truth labels or a held-out evaluation, then a rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ming,\n\nShort version: this is a real systems paper with a concrete efficiency claim, but the accuracy claim is the load-bearing part and it does not hold up as presented.\n\nWhat is actually new: the three-way integration of Bayesian Group Testing (the lattice machinery from their own SBGT), Bayesian Optimization, and Bayesian Risk Refinement applied to heterogeneous-unsafe parameter identification. That combination is not in the cited literature, and the engineering is serious—snapshot-based recovery, container isolation, filtering, parallelized BO. The 67% reduction in test count and execution time over ZebraConf is a concrete, possibly useful result if it is real.\n\nThe soft spot is exactly where the reader put it. The 0% false positive and false negative rates are not meaningful as reported. FPR and FNR require ground-truth labels for which parameters are truly heterogeneous-unsafe in each of the 1,000 unit test files, and the paper never says where those labels come from. The simulated environments in Section V-F with known failure modes are not connected to the 1,000-file evaluation. On top of that, the BO objective in Section V-D includes FPR and FNR explicitly, the surrogate is built on the first 200 files and updated on the rest, and the reported iteration X=179 is chosen post hoc because it looked best. That is model selection on the reporting set, so the 0% numbers are fitted quantities, not out-of-sample predictions.\n\nThe efficiency result is more defensible: comparing executed test counts and wall-clock time on the same 1,000 files is a fair apples-to-apples comparison against ZebraConf, and the 67% savings are large. But without a validated accuracy floor, a 67% test reduction is just a cost saving with unverified correctness. The missing ground-truth definition, the 'Algorithm ??' placeholder, and no code or data make independent verification impossible.\n\nOverall, the framework is coherent and the application is plausible, but the central empirical claim needs a redesign: define ground truth, use a held-out set for reporting, give variance, and release artifacts.\n\nRecommendation: send to peer review rather than desk reject, because the efficiency result and the integration are worth engaging with, and a serious referee could force the evaluation to be done properly. If the authors can supply out-of-sample numbers with disclosed labels, this could be a solid systems paper.","headline":"A plausible engineering integration of Bayesian group testing, optimization, and risk refinement for configuration debugging, but the headline 0% error rates are fitted, not predictive, because ground truth is undefined and hyperparameters are selected on the same 1,000 files.","tokens_in":14071,"tokens_out":1594,"would_cite":false,"duration_ms":15539,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ba-ZebraConf replaces binary-splitting group testing with a three-module Bayesian framework that cuts test count and execution time by 67% and drives false positives and false negatives to 0% on its 1,000-file evaluation.","keywords":["Bayesian group testing","configuration troubleshooting","heterogeneous configurations","distributed systems","misconfiguration detection","Bayesian optimization","group testing"],"falsifier":"Run Ba-ZebraConf at its converged iteration on a new, independently labeled set of configuration test files and count mismatches against the known labels; any false positive or false negative above 0%, or a test-count reduction below 67%, would refute the central claim.","tokens_in":12944,"feed_emoji":"⚙️","tokens_out":6744,"duration_ms":54523,"temperature":0.7,"pith_summary":"The paper argues that configuration testing in distributed systems can be made faster and more trustworthy by replacing the binary-splitting group testing used in ZebraConf with a three-module Bayesian framework. The framework combines Bayesian Group Testing, which uses a probabilistic lattice model and the Bayesian Halving Algorithm to choose which configurations to test; Bayesian Optimization, which tunes pool sizes and thresholds; and Bayesian Risk Refinement, which accumulates evidence across tests to handle parameter interdependencies. On 1,000 unit test files, Ba-ZebraConf reports a 67% reduction in both number of tests and execution time compared with ZebraConf, with false positive and false negative rates falling to 0% by the later optimization iterations. If true, the approach would let operators find unsafe heterogeneous configurations with roughly one-third of the testing budget and with fewer classification errors.","feed_headline":"Bayesian framework cuts config-test time and count 67%","feed_subtitle":"Three Bayesian modules find unsafe settings faster while eliminating misclassification on 1,000 test files.","key_machinery":"The load-bearing machinery is the three-dimensional Bayesian loop itself: a probabilistic lattice model over parameter subsets, updated by the Bayesian Halving Algorithm (a test-selection rule that splits the lattice into two nearly equal posterior-probability halves); a Gaussian-process Bayesian optimization module with Expected Improvement acquisition that tunes hyperparameters including pool sizes, priors, and classification thresholds; and Bayesian Risk Refinement, which applies Bayes' rule to accumulate per-parameter risk evidence across tests until thresholds $\\tau_{\\text{safe}}$ and $\\tau_{\\text{unsafe}}$ terminate testing. The lattice shrinking and risk-threshold termination are what convert accuracy gains into test-count savings.","core_discovery":"On its own terms, the central discovery is that three Bayesian ingredients—probabilistic group testing, automated hyperparameter optimization, and iterative risk refinement—can be composed into a single troubleshooting pipeline that outperforms the binary-splitting baseline. The claimed result is quantitative: across 1,000 unit test files, Ba-ZebraConf reduces executed tests from 107,962 to 35,426 and execution time from 472,503 to 153,417 seconds, a 67% reduction in both, while the false positive rate falls from 14% at the first evaluated iteration to 0% and false negatives fall from 4% to 0%. The mechanism credited for the improvement is probabilistic prioritization of high-risk configurations, cumulative risk evidence that captures parameter interdependencies, and dynamically tuned test thresholds.","pith_inferences":["Editorial inference: the same three-module stack could be pointed at other pooled-diagnosis problems where interdependencies mask faults, such as network link fault localization or hardware component screening, because the machinery does not depend on file-based configuration semantics.","Editorial inference: the paper's fixed objective weights (0.4 for time savings, 0.4 for cost savings, minus error rates) invite a sensitivity check; different weights could shift where the optimization converges and how large the reported gains are.","Editorial inference: a deployment would need an explicit ground-truth labeling procedure, since without one the 0% error rates cannot be audited outside the original test set."],"forward_implications":["At iteration $X = 179$, Ba-ZebraConf executes 35,426 tests instead of ZebraConf's 107,962 and finishes in 153,417 seconds instead of 472,503 seconds, a 67% reduction on both metrics.","False positives fall to 0% by $X = 179$, and false negatives reach 0% from $X = 13$ onward, showing that accuracy improves as Bayesian optimization refines the hyperparameters.","Because parameters are removed from the lattice once risk thresholds are crossed, the per-parameter testing cost shrinks as classifications become confident, which is the mechanism behind the count reduction.","Hyperparameter tuning shifts from exploration in the first 200 unit test files to exploitation with a background Bayesian Optimization Daemon, allowing later tests to run under the best-known settings."],"supporting_citations":[{"why":"Defines the ZebraConf baseline and the heterogeneous-unsafe parameter problem that Ba-ZebraConf targets; all efficiency gains are measured against it.","marker":"[6]"},{"why":"Supplies the Bayesian Halving Algorithm and lattice-based scaling (SBGT) that the BGT module adapts for configuration testing.","marker":"[21]"},{"why":"Provides the Bayesian group testing formulation with dilution effects that underlies the noise-resilient likelihood models.","marker":"[24]"},{"why":"Establishes sequential classification on partially ordered sets, the basis for the lattice model used to prioritize tests.","marker":"[25]"},{"why":"Supplies the Bayesian optimization tutorial and the Gaussian process/acquisition machinery used by the BO module.","marker":"[23]"},{"why":"Documents software configuration dependencies, motivating the BRR module's cumulative risk updates.","marker":"[22]"},{"why":"Provides the experimental hardware used for the reported 1,000-file evaluation.","marker":"[30]"}],"fun_headline_variants":["Bayesian trio cuts config-test time and count 67%","Zero-error troubleshooting: Bayesian method trims tests 67%","Three Bayesian modules beat config testing, 67% faster","Ba-ZebraConf: 67% fewer tests, zero false outcomes","Probabilistic group testing: 67% less config testing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 0% false-positive and false-negative rates assume that reliable ground-truth labels exist for the 1,000 unit test files, yet the paper does not state how those labels were obtained.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian trio cuts config-test time and count 67%","Zero-error troubleshooting: Bayesian method trims tests 67%","Three Bayesian modules beat config testing, 67% faster","Ba-ZebraConf: 67% fewer tests, zero false outcomes","Probabilistic group testing: 67% less config testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001228,"raw_usage":{"total_tokens":5058,"prompt_tokens":969,"completion_tokens":4089,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":4002}},"tokens_in":585,"tokens_out":4089,"duration_ms":25581,"temperature":1.0,"reasoning_tokens":4002,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:19:07.986802+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Ba-ZebraConf at its converged iteration on a new, independently labeled set of configuration test files and count mismatches against the known labels; any false positive or false negative above 0%, or a test-count reduction below 67%, would refute the central claim.","supporting_citations":[{"cited_title":"Finding heterogeneous- unsafe configuration parameters in cloud systems,","cited_arxiv_id":null,"evidence_quote":"Defines the ZebraConf baseline and the heterogeneous-unsafe parameter problem that Ba-ZebraConf targets; all efficiency gains are measured against it."},{"cited_title":"Sbgt: Scaling bayesian- based group testing for disease surveillance,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian Halving Algorithm and lattice-based scaling (SBGT) that the BGT module adapts for configuration testing."},{"cited_title":"Bayesian Group Testing with Dilu- tion Effects,","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian group testing formulation with dilution effects that underlies the noise-resilient likelihood models."},{"cited_title":"Sequential Classification on Partially Or- dered Sets,","cited_arxiv_id":null,"evidence_quote":"Establishes sequential classification on partially ordered sets, the basis for the lattice model used to prioritize tests."},{"cited_title":"Understanding and Discovering Software Configuration Dependencies in Cloud and Datacenter Systems,","cited_arxiv_id":null,"evidence_quote":"Documents software configuration dependencies, motivating the BRR module's cumulative risk updates."},{"cited_title":"CloudLab,","cited_arxiv_id":null,"evidence_quote":"Provides the experimental hardware used for the reported 1,000-file evaluation."}],"review_version":1}