{"id":"ae6ed611-9a6e-435c-a038-3d6698b755b5","arxiv_id":"2501.16751","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HiBug2 discovers error slices in vision models via structured GPT-generated attributes, efficient enumeration, and prediction of unseen failure patterns, improving model repair over prior methods.","lead":"HiBug2 automatically finds coherent failure groups (error slices) in computer vision models by using GPT-4V to generate structured visual attribute tags, then efficiently enumerating combinations of tags that have low accuracy. It also predicts failure patterns not seen in the validation set and uses them to select new training data that repairs the model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's 'unseen slice' evaluation does not specify which images are used, so the headline prediction-beyond-validation claim may not be measured on held-out data.","rationale":"The reader correctly flags that GPT-4V tag accuracy is unmeasured; however, the paper's own limitation discussion in Section 6 addresses only random errors, and systematic tag bias is a plausible but not demonstrated failure mode. The more decisive gap is in the paper's most novel claim. Section 4.2 introduces prediction of error slices beyond the validation set, and Table 1 reports declines up to 64.6%, but Section 5.4 omits the evaluation pool. Since every validation-set slice with count >= 10 is already enumerated by Algorithm 3, a result computed on validation images would not demonstrate extrapolation. A result computed on the Section 5.5 query pools would need sample sizes and confirmation that those images were not simultaneously used for repair selection. The released code can resolve this directly. Other components, such as the 510x enumeration speedup, runtime scaling, and overlap analyses, are well specified and not in question; the repair improvements in Table 2 lack error bars and are smaller than the headline, but they are secondary to the unseen-slice claim. Therefore the appropriate verdict remains conditional: accept the algorithmic and efficiency contributions, but require the Table 1 protocol to be disclosed and, if needed, recomputed on a disjoint pool. If the code confirms a disjoint external pool with adequate per-slice counts, the concern is resolved and the claim stands.","tokens_in":19316,"tokens_out":6014,"duration_ms":55950,"concrete_test":"Inspect the released repository (github.com/cure-lab/HiBug2) to identify the exact data used for Table 1: (1) Are the 'predicted error slices' evaluated on the same validation images used for slice enumeration, on the repair query pools of Section 5.5, or on a disjoint hold-out set? (2) If validation images are used, recompute Table 1 restricted to slices with at least 10 matching validation images that were not already identified as error slices; if the reported declines shrink or disappear, the 'beyond validation set' claim is unsupported. (3) If external images are used, report per-slice counts and confirm those images were not also used to select repair data in Section 5.5. Any of these checks will settle whether Table 1 validates unseen-slice prediction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2's headline capability is 'predicting error slices beyond the validation set,' and Section 5.4/Table 1 reports performance declines up to -64.6% on those predicted slices. However, the paper does not state which images are used to compute those declines. The only pools described are the validation set used for enumeration and the query/test pools introduced in Section 5.5 for repair. If Table 1 uses validation images, then any predicted slice with count >= 10 was already enumerated by Algorithm 3, so it is not 'beyond the validation set'; predicted slices with count < 10 would rest on very small, unreported samples. If Table 1 uses the external query pools, sample sizes and the exact selection protocol are missing, and those images may overlap with the data used for repair in Section 5.5. This is more immediately load-bearing than GPT-4V tag accuracy, which the paper acknowledges in Section 6 but only defends against random, not systematic, tagging errors. The claim would stand if the code shows a disjoint held-out pool with adequate per-slice counts; as written, the experiment cannot be audited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents HiBug2, an automated framework for error slice discovery and model repair. It generates task-specific visual attributes and tags using GPT-4V through a structured prompting process, enumerates data slices via a breadth-first tree search with pruning and intersection, and proposes two methods (tag substitution and instruction-based) to predict error slices beyond the validation set. The authors evaluate HiBug2 on image classification, pose estimation, and object detection, reporting improved attribute quality, a 510x speedup over naive enumeration, performance degradation on predicted unseen slices, and gains in model repair relative to HiBug and random selection.","tokens_in":19564,"tokens_out":6508,"duration_ms":51513,"significance":"If substantiated, the HiBug2 framework would be a valuable contribution to the model-debugging literature, as it directly addresses the combinatorial explosion in slice search and extends the scope of slice discovery to non-validation data. The paper's strengths include its clearly described enumeration algorithms (with pseudo-code in Appendix A.7), the breadth of tasks evaluated, and the concrete speedup claims. However, the current evidence is not fully convincing because the unseen-slice experiment is not auditable, the correctness of the GPT-4V-based tags is not validated against systematic errors, and the quantitative comparisons lack uncertainty estimates. These issues are fixable within the scope of a revision by adding a clear hold-out protocol, tag-quality checks, and error bars.","major_comments":[{"comment":"The claim that HiBug2 predicts error slices beyond the validation set is not auditable because the paper does not specify which images are used to compute the performance degradations in Table 1. If the validation set is used, any predicted slice with at least M=10 images would already have been enumerated by Algorithm 3 and is therefore not 'unseen'; if an external pool is used, the per-slice sample sizes and the selection protocol are missing, and the pool may overlap with the repair data of Section 5.5. Please provide a clear description of the data source, per-slice counts, and a disjoint hold-out set.","section":"Section 5.4, Table 1"},{"comment":"The entire pipeline relies on GPT-4V to assign tags to all images, but the paper only verifies that tags conform to predefined formats (Appendix A.1.1), not that they are semantically correct. The defense in Section 6 that 'a few misclassified data points do not alter the average performance of a slice' addresses random tagging noise but not systematic bias (e.g., consistent mislabeling of a visual subgroup). A systematic bias could create spurious slices or conceal real ones, directly affecting the discovered error slices and the repair gains in Table 2. Please report a quantitative tag-accuracy evaluation on a labeled subset, or otherwise show that tagging errors for any subgroup are bounded.","section":"Section 3.2.2, Appendix A.1.1, Section 6"},{"comment":"The model repair results are averaged over five runs, but no standard deviations, confidence intervals, or significance tests are reported. Without this information, the differences between HiBug2, HiBug, and random selection (e.g., +7.6% vs +6.3% vs +4.7% in image classification) could be within run-to-run variability. Please add error bars and a significance test or a clear statement of variance.","section":"Section 5.5, Table 2"},{"comment":"The key hyper-parameters C (error threshold) and M (minimum slice count) are set to 0.2 and 10 without sensitivity analysis. These values directly determine the number and composition of error slices, and thus affect all downstream results. An ablation varying C and M would demonstrate that the reported findings (e.g., the 510x speedup, the slice overlap statistics, and the repair improvements) are not sensitive to these arbitrary choices.","section":"Section 4.1.4, Section 5.3"},{"comment":"The user study has only four participants, all described as machine-learning and computer-vision practitioners, and it reports unanimous preference for HiBug2 on all four criteria. With n=4, this result has very low statistical power, and the paper does not state whether the participants were independent of the authors. This evidence is only suggestive; please expand the study or qualify the claims accordingly.","section":"Appendix A.6, Table 6"}],"minor_comments":[{"comment":"The introduction mentions 'approximately 500 distinct error slices for CLIP', while Section 5.3 reports 499; please make these numbers consistent.","section":"Section 1 vs Section 5.3"},{"comment":"The caption of Figure 4 does not report the dataset size, number of attributes, or tag set sizes used in the timing experiment; please add these details so the speedup comparison is reproducible.","section":"Section 5.2, Figure 4"},{"comment":"The sentence 'we consider slices with three attributes' is a design choice; please justify this choice or state that higher-depth slices were not used for the analysis.","section":"Section 5.3"},{"comment":"The number of predicted slices (100, 20, 40) is given, but not the number of attributes per slice; please specify the composition of the predicted slices.","section":"Section 5.4"},{"comment":"The manual check for no overlap between the web-collected data and the validation set is not described; please explain the procedure or state that it was verified with code.","section":"Appendix A.5"}],"recommendation":"major_revision","confidential_remarks":"The paper provides a GitHub link but no reviewable code in the submission, so the key auditability concerns (especially around Table 1) cannot be resolved by inspecting the manuscript alone. The private pose dataset further limits independent reproducibility. The methodological ideas are promising, and the issues raised should be addressable in a revision; the paper is currently not ready for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [name],\n\nShort version: this is a real contribution to error slice discovery, but the paper oversells the 'unseen slice' prediction. Read it for the enumeration algorithm and the structured attribute generation; treat the Table 1 results with suspicion until the authors say what data those numbers come from.\n\nWhat's actually new: the three-category attribute generation (main object, background, global) with paired-image prompting and task-specific queries is a sensible step beyond the ad-hoc tag lists in HiBug and Domino. The BFS tree enumeration with monotonicity-based pruning and matched-pair intersection is a genuine algorithmic improvement—the 510x speedup over brute force at 4 attributes is not surprising, but it's the right way to make multi-attribute slices searchable. The repair experiments show consistent gains over HiBug and random selection across classification, pose, and detection, which is a useful result even if the margins are modest.\n\nSoft spots, in order of severity. Section 5.4/Table 1 is the load-bearing one: the paper never says which images are used to compute the performance of the 'predicted' slices. If it's the validation set, then slices with count ≥ M were already enumerated by Algorithm 3 and are not 'beyond the validation set'; slices with count < M can't support a -64.6% claim on a few samples. If it's an external query pool, the paper needs to say where it came from, how it was selected, and whether it overlaps the data used for repair in Section 5.5. As written, the headline claim can't be audited. This is worse than the GPT-4V tagging issue, which the paper acknowledges but only defends against random errors—systematic tag bias could create spurious slices, and format checks don't catch semantic errors.\n\nOther issues are real but minor: C=0.2, M=10, and depth 3 are arbitrary, with no sensitivity analysis; Table 2 has no error bars despite five runs; the user study is four people. None of these sink the core tag-then-slice story.\n\nBottom line: the central mechanism is plausible, the code is public, and the efficiency results give readers something concrete to build on. The paper deserves a serious referee, but the unseen-slice section needs an experiment on a properly held-out pool with per-slice counts before that claim is credible.\n\nBest.","headline":"A solid slice-discovery method with real algorithmic improvements, but the headline 'unseen slice' evaluation is unauditable as written.","tokens_in":20047,"tokens_out":2773,"would_cite":true,"duration_ms":24855,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HiBug2 claims that error slice discovery should start from a structured vocabulary of visual attributes—main object, background, global—and that a breadth-first tree enumeration over that vocabulary finds coherent, interpretable failure…","keywords":["error slice discovery","model debugging","visual attributes","slice enumeration","model repair","multimodal tagging","interpretability","computer vision"],"falsifier":"Take a validation set with known ground-truth subgroup labels, systematically corrupt the tag assignment for one subgroup (e.g., all low-light images tagged as 'bright'), run HiBug2, and check whether the discovered error slices shift to follow the corrupted tags; a second check is comparing HiBug2's slice membership against a human-annotated tag set on a random sample of images and measuring how often a discovered 'low-performance' slice would be spurious.","tokens_in":19135,"feed_emoji":"🔍","tokens_out":7419,"duration_ms":61177,"temperature":0.7,"pith_summary":"HiBug2 is a fully automated pipeline for discovering systematic failure groups (error slices) in computer vision models and then repairing the models with data drawn from those groups. The paper argues that generating a structured visual-attribute vocabulary before any slicing—rather than clustering failure cases and tagging them afterwards—yields slices that are more coherent, interpretable, and actionable. Its breadth-first tree enumeration exploits the monotonic decrease of slice data count to prune the search, and the paper reports a 510x speedup over brute-force enumeration for four-attribute slices. The same vocabulary is used to predict error slices beyond the validation set, and targeted data querying from the worst slices improves accuracy, keypoint AP, and mAP by 7.6, 2.1, and 4.9 percentage points respectively over random and prior baselines. If correct, this makes slice-based debugging practical for multi-attribute and multi-task settings.","feed_headline":"Structured attributes find model blind spots 510x faster","feed_subtitle":"Automated pipeline surfaces coherent error slices across classification, pose, and detection, and claims to repair models using them.","key_machinery":"The machinery is the attribute-tag vocabulary plus a breadth-first tree-structured slice enumeration. Attributes are generated by prompting a large multimodal model with comparative image pairs and task-specific failure queries; tags are refined in a multi-stage process with a yes/no convention for binary attributes and a 'not visible' tag for absent elements. The tree is built so that each node adds one attribute-tag pair to its parent, and because the data count of a slice is monotonically non-increasing as depth grows, the algorithm prunes any subtree whose parent has fewer than ten examples and only forms new candidates by intersecting matched slice pairs that share k−1 attributes. This replaces a combinatorial scan over all attribute combinations with a linear scan over surviving candidates; hash tables speed up matched-pair search and matrix multiplication accelerates data counting and accuracy computation.","core_discovery":"The paper's central claim is that error slice discovery should be reordered: first build a task-specific vocabulary of visual attributes—grouped into main object, background, and global categories—with consistent, unambiguous tags, and only then enumerate slices over that vocabulary. HiBug2 shows that this tag-then-slice order, combined with a breadth-first tree enumeration that prunes low-count subtrees and intersects surviving parent slices, identifies hundreds to thousands of low-performance slices across image classification, pose estimation, and object detection. The authors further claim that two cheap heuristics—substituting a tag with the nearest alternative in a contrastive embedding space and asking a large multimodal model to predict confusable attribute combinations—reliably predict slices the validation set never contained, with observed performance drops up to 64.6% on those predicted slices. The experiments also support the claim that error slices are shared across models on the same task (86% overlap among the worst detection slices for four detectors) and that querying data from the worst-discovered slices improves model performance more than random data selection or the prior HiBug method.","pith_inferences":["If tag quality is the bottleneck, the repair gains may partly measure the multimodal model's ability to describe images rather than the slicing algorithm itself; an ablation varying the attribute vocabulary size and tagging source would separate the two.","The monotonicity-based pruned enumeration is not limited to error slices: it could apply to any subgroup analysis over categorical metadata, such as intersectional fairness audits over demographic tags.","The paper's 'a few misclassified tags do not change slice averages' claim is untested for systematic tagging bias; injecting correlated tag noise into a known subgroup would show whether discovered slices remain stable.","The high cross-model slice overlap in detection suggests the framework could pick one representative model for debugging a whole family of detectors, reserving per-model analysis for tasks like classification where overlap is only 31%."],"forward_implications":["Multi-attribute error slices can be enumerated in minutes rather than hours: the reported 510x speedup makes four-attribute slices practical, and runtime scales linearly with the number of images.","Slices become directly interpretable, so an engineer can see at a glance that a classifier fails on white teddy bears not holding an item, or that a detector fails on underexposed high-clutter scenes.","Error slices can be predicted beyond the validation set, which matters when the labeled pool is small: the paper reports performance drops up to 64.6% on predicted slices.","The framework transfers across tasks—classification, pose estimation, and object detection—and the discovered slices generalize across model architectures, with 86% overlap among top detection slices.","Targeted data querying from the worst slices repairs models better than random augmentation: gains of 7.6%, 2.1%, and 4.9% on the three tasks, with random selection yielding 4.7%, 0.9%, and 0.6%."],"supporting_citations":[{"why":"The prior HiBug method that HiBug2 directly compares against for attribute quality, slice discovery, and model repair; its tag-then-slice design is the starting point.","marker":"Chen et al., 2024"},{"why":"AdaVision is the main attribute-centric baseline using iterative scenario generation that HiBug2 contrasts with for structured, comprehensive attribute coverage.","marker":"Gao et al., 2023"},{"why":"Domino represents the slice-then-tag approach whose coherence HiBug2 claims to improve by generating attributes before slicing.","marker":"Eyuboglu et al., 2022"},{"why":"Spotlight is another slice-then-tag baseline used to motivate the need for coherent visual attributes rather than embedding-space clusters.","marker":"d'Eon et al., 2022"},{"why":"The contrastive vision-language model supplies the text embeddings used in tag substitution for predicting unseen error slices, and it is one of the classification models being debugged.","marker":"Radford et al., 2021"},{"why":"The large multimodal model that generates all attributes and tags in HiBug2's pipeline; the entire method's quality depends on its outputs.","marker":"OpenAI, 2023"},{"why":"ImageNet supplies the bear-classification subset used in the classification experiments and model repair.","marker":"Deng et al., 2009"},{"why":"KITTI supplies the object-detection data and models for the car and pedestrian slices.","marker":"Geiger et al., 2012"},{"why":"RTMPose is the pose-estimation model family whose different sizes are debugged with HiBug2.","marker":"Jiang et al., 2023"}],"fun_headline_variants":["Blind spots found 510x faster with structured tags","Predicting model failures beyond the test set","Structured attributes expose hidden error slices","Find and fix model blind spots faster with HiBug2","Error slice discovery with interpretable attributes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes the multimodal model assigns tags to images accurately enough that the statistics computed over each slice reflect true subgroups; the paper verifies tag format and attribute names but not semantic correctness, and asserts without measurement that a few misclassified points do not alter a slice's average performance.","fun_headline_variants_meta":{"raw":{"variants":["Blind spots found 510x faster with structured tags","Predicting model failures beyond the test set","Structured attributes expose hidden error slices","Find and fix model blind spots faster with HiBug2","Error slice discovery with interpretable attributes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000477,"raw_usage":{"total_tokens":2350,"prompt_tokens":920,"completion_tokens":1430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1360}},"tokens_in":536,"tokens_out":1430,"duration_ms":9770,"temperature":1.0,"reasoning_tokens":1360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T10:57:55.602710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a validation set with known ground-truth subgroup labels, systematically corrupt the tag assignment for one subgroup (e.g., all low-light images tagged as 'bright'), run HiBug2, and check whether the discovered error slices shift to follow the corrupted tags; a second check is comparing HiBug2's slice membership against a human-annotated tag set on a random sample of images and measuring how often a discovered 'low-performance' slice would be spurious.","supporting_citations":[{"cited_title":"Hibug: on human-interpretable model debug","cited_arxiv_id":null,"evidence_quote":"The prior HiBug method that HiBug2 directly compares against for attribute quality, slice discovery, and model repair; its tag-then-slice design is the starting point."},{"cited_title":"The spotlight: A general method for discovering systematic errors in deep learning models","cited_arxiv_id":null,"evidence_quote":"Spotlight is another slice-then-tag baseline used to motivate the need for coherent visual attributes rather than embedding-space clusters."},{"cited_title":"Are we ready for autonomous driving? the kitti vision benchmark suite","cited_arxiv_id":null,"evidence_quote":"KITTI supplies the object-detection data and models for the car and pedestrian slices."}],"review_version":1}