{"id":"19710882-05cd-4800-b634-d6c1d11504ed","arxiv_id":"2505.03618","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"cVIL shifts interactive labeling from per-instance class assignment to per-class instance assignment, and a 16-person study finds higher final label accuracy and user preference than an instance-centric interface, though not faster completion time.","lead":"This paper introduces cVIL, a visual labeling workflow that works class by class instead of instance by instance, having users decide whether images belong to a focus class rather than choosing among many classes. In a small user study, cVIL produced more accurate final labels and was preferred by most participants, but it did not significantly reduce labeling time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The O(n) vs O(nm) scalability argument in §3.1 assumes the class-centric advantage instead of deriving it; the many-class scalability claim is therefore unsupported by either theory or measurement.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the average-case complexity advantage in §3.1 relies on an input assumption about batchable subsets and asymmetric iVIL search costs, rather than on a consequence of the model. My independent read confirms that this is the most load-bearing issue for the central claim of scalability to many classes. The user study is real but limited: it shows a significant accuracy improvement (median 98.4% vs 96.05%, p<.001) and strong preference (13/15), but uses a binary task with m=2, so it does not test the many-class regime. The Caltech-101 walkthrough is qualitative and includes the authors' own examples where batch labeling fails, which directly undermines the assumption behind the complexity advantage. The paper does have merits: the cVIL workflow is clearly described, the interface is novel, and the user study is a legitimate within-subjects comparison with randomized order. The excluded participant is a minor methodological concern, but the paper's own justification is plausible. Overall, the verdict should remain CONDITIONAL: the paper should be accepted only if the authors either replace the informal complexity model with a rigorous one or provide quantitative evidence of many-class scaling. My stress-test does not change the reader's conditional verdict; it confirms that the condition is necessary.","tokens_in":24212,"tokens_out":8085,"duration_ms":80610,"concrete_test":"Re-derive the effort comparison in §3.1 under a symmetric interaction model: an iVIL user may lasso any region in the 2D projection and label each visible instance with its correct class in one pass, while a cVIL user must also inspect and remove false positives from each suggested batch. Compute expected total effort over a range of classifier precisions per class. If the ratio of expected efforts does not scale with m as claimed, the §3.1 asymptotic advantage is an artifact of the asymmetric modeling.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central scalability claim for large class counts rests on the complexity comparison in §3.1, which concludes that cVIL is O(n) 'on average' while iVIL is O(nm). This conclusion is not a mathematical consequence of the class-centric paradigm; it depends on two unstated and questionable assumptions. First, the 'average case' assumes that within each focus class the user can identify and batch-confirm a subset of n_c/10 instances after only O(n_c) examination, with no meaningful search or error-correction cost. Second, it assumes the iVIL user must re-scan 10% of the full dataset for every one of the m classes. The authors' own walkthrough (§5.2) contradicts the first assumption: for the 'Gerenuk' and 'Butterfly' classes, predictions are so poor that batch labeling collapses, forcing instance-wise labeling and reverting to the O(nm) worst case. The second assumption is also asymmetric: an iVIL user can lasso a spatial cluster in the t-SNE projection and assign instances to multiple classes in a single interaction, which the analysis does not account for. Since the user study uses only m=2 classes, it cannot validate many-class scaling, and the Caltech-101 walkthrough reports no time or effort measurements. Thus the scalability advantage is an artifact of asymmetric effort accounting, not an established property of class-centric labeling; the authors' own Section 6 limitation statement concedes that 'cVIL's strengths diminish' in difficult cases.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a class-centric visual interactive labeling (cVIL) workflow, in which the user focuses on one class at a time and assigns instances to that class, contrasting with traditional instance-centric VIL (iVIL). The authors formalize the workflow, present a visual analytics interface with four linked views (Class Label View, Instance Property View, Instance Similarity View, Instance Labeling View), and evaluate it in two complementary settings: a within-subjects user study on binary CelebA classification with 15 participants, and a qualitative usage scenario on the Caltech-101 dataset with 100 classes. The user study reports a statistically significant improvement in final accuracy for cVIL (median 98.4% vs. 96.05%, t(14)=5.784, p<.001) and a preference by 13 of 16 participants, while completion time and workload differences were not significant. The paper claims that class-centric labeling reduces labeling effort and scales better than instance-centric labeling for large numbers of instances and classes.","tokens_in":24472,"tokens_out":4830,"duration_ms":48975,"significance":"If the claims are supported, the class-centric paradigm is a genuinely useful addition to visual interactive labeling: it converts the labeling decision from a multi-class choice to a binary membership decision and enables natural batch operations, which can reduce cognitive load and improve accuracy. The user study is a real, randomized within-subjects comparison and shows a large and statistically significant accuracy effect, with most participants preferring the cVIL interface. The qualitative walkthrough provides anecdotal evidence that the approach can be applied to a 100-class dataset. However, the theoretical scalability argument in Section 3.1 is not a derivation but an assumption-laden comparison, and the empirical evidence for many-class scalability is absent: the user study uses only two classes and the walkthrough reports no time or effort measurements. The paper's central claim of scalability for large class counts is therefore currently supported more by design narrative than by evidence, though the underlying idea and the reported single-task result remain promising.","major_comments":[{"comment":"The O(n) vs. O(nm) average-case comparison is not derived from the class-centric model; it builds in the conclusion it claims to prove. Specifically, it assumes that a cVIL user can, for each focus class, examine O(nc) instances and batch-confirm 10% of the class in one action, while it assumes an iVIL user must re-scan 10% of the full dataset for each of the m classes. Both premises are arbitrary: an iVIL user can select spatially coherent clusters and label multiple classes in one interaction, and the walkthrough in Section 5.2 shows classes (Gerenuk, Butterfly) where cVIL collapses to instance labeling, which the analysis does not model. Because the user study uses only m=2 and the walkthrough reports no time or effort data, the many-class scalability claim is unsupported by both theory and measurement. Please either present the analysis as explicitly conditional on stated assumptions or remove the formal O() claims.","section":"Section 3.1, average-case complexity"},{"comment":"The post-hoc exclusion of one participant who 'mistakenly selected the wrong focus class for batch labeling' is not justified by a pre-registered or principled outlier criterion. This error is a realistic failure mode of the interface, and the argument that it 'cannot easily happen when following the full cVIL workflow' is contradicted by the fact that the interface in the study made it possible. Excluding this participant inflates the reported accuracy effect. Please report the statistical analysis with and without this participant, or justify the exclusion with an explicit, pre-defined rule for excluding user errors.","section":"Section 5.1.1, participant exclusion"},{"comment":"The cVIL and iVIL interfaces differ in several design dimensions simultaneously: cVIL uses a KDE plot with a min-margin property measure and class bars, while iVIL uses a t-SNE scatter plot with color-coded predicted classes. The observed accuracy difference and user preference could therefore be caused by the visualization idioms or the uncertainty guidance rather than by the class-centric paradigm itself. To support the paradigm-level claim, the study should compare cVIL with an instance-centric interface that shares the same property-view idiom, or the conclusions should be explicitly restricted to the concrete interface comparison. As written, the paper over-generalizes from a comparison of two specific interfaces.","section":"Section 5.1, interface confound"}],"minor_comments":[{"comment":"The text says 15 of 16 participants completed the study, but later reports '13 out of 16 participants expressed a preference for cVIL'; since one participant was excluded, this should be 13 of 15, or the discrepancy should be clarified.","section":"Section 5.1.2, preference count"},{"comment":"The arrow notation n→S m is not defined precisely enough: the subscript S is used both as the selection set and as part of the arrow subscript, which makes the formalization hard to follow. Please use clearer notation for the sets that the user works with at each step.","section":"Section 3.1, notation"},{"comment":"The paper does not state which tests were used for the accuracy and time comparisons (paired t-test is implied but not named), and no effect sizes or confidence intervals are reported. Adding these would improve reproducibility and allow readers to judge the practical size of the accuracy benefit.","section":"Section 5.1.2, statistical reporting"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of Computers & Graphics and the class-centric idea is worth publishing if the scalability claims are tempered and the analysis issues are addressed. The main risk is that the title and abstract promise 'scalable' labeling for many classes, but the evidence is limited to a 2-class user study and a qualitative walkthrough; the complexity section states assumptions as if they were consequences. Please also verify the novelty disclosure relative to the authors' earlier EuroVA paper [11]; the present paper should clearly state what the new interface adds over that prototype and why the prior evaluation does not already support the claims made here."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version. This is a solid paper, not a breakthrough, but it does something real: it formalizes a class-centric labeling workflow (cVIL), implements an interface, and runs a within-subjects study against an instance-centric baseline. The accuracy effect is large and significant (median 98.4% vs 96.05%, p<.001), and 13 of 15 participants preferred cVIL. That's worth taking seriously.\n\nWhat's new: the four-phase workflow (bootstrap, class selection guidance, class-based labeling, residual labeling) and the specific interface ideas, particularly the KDE property plot plus a class-rank cutoff for pulling in ambiguous neighbors. The related work is thorough, and the connection to property measures gives the work a principled grounding. The paper is also honest about its limitations in Section 6.\n\nThe soft spots are real but not decisive. The complexity analysis in Section 3.1 is more a back-of-the-envelope argument than a proof. The O(n) vs O(nm) 'average' case assumes a cVIL user can batch-select 10% of a class after scanning only that class, while an iVIL user must re-scan 10% of the whole dataset for every class. Neither assumption is derived; the iVIL user can lasso spatial clusters and assign several classes at once. The authors' own walkthrough shows classes (Gerenuk, Butterfly) where the model is too poor for batch selection, so the advantage collapses toward the worst case. So the many-class scalability claim is under-supported: the user study only uses two classes, and the Caltech-101 scenario reports no time or effort measurements.\n\nThe study itself has a confound: the cVIL and iVIL interfaces differ in many design dimensions (KDE vs t-SNE, class-rank cutoff, multi-instance labeling interactions), so the measured difference might come from the interface rather than the class-centric paradigm. The post-hoc exclusion of one participant who mis-selected a focus class is a bit convenient; the authors should justify it more carefully. The non-significant time and workload results also mean the 'efficiency' claim should be softer than the abstract implies.\n\nBottom line: it's a legitimate contribution to visual interactive labeling, and it deserves peer review. The authors should be asked to defend the exclusion and confound, and to either temper the scalability claim or add evidence. I'd probably not cite it in my own work, but I'd bring it to a reading group.","headline":"A serious class-centric labeling workflow with a significant accuracy win in a small study, but the scalability proof and interface confound mean the efficiency claims get ahead of the evidence.","tokens_in":25085,"tokens_out":4186,"would_cite":false,"duration_ms":40567,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that data labeling becomes faster, more accurate, and more scalable when the task is inverted: instead of assigning one of many classes to each instance, the user focuses on a single class at a time and labels whole…","keywords":["class-centric labeling","visual interactive labeling","batch labeling","property measures","labeling scalability","human-in-the-loop machine learning","visual analytics"],"falsifier":"Run a controlled comparison on a dataset with many classes (for example, 20 or more) using a deliberately weak classifier that produces poor predictions for most classes, and measure both final accuracy and the actual size of the batches users manage to confirm per class. If the median batchable fraction per class is far below 10% for a representative dataset, or if cVIL's accuracy advantage disappears in a many-class setting, the average-case O(n) claim would be falsified.","tokens_in":23941,"feed_emoji":"🎯","tokens_out":9562,"duration_ms":86879,"temperature":0.7,"pith_summary":"The paper is trying to establish that interactive data labeling should be reversed: rather than asking 'which class does this instance belong to?', the annotator picks a class and asks 'which instances belong to this class?'. This class-centric inversion turns each labeling decision into a simple yes/no judgment about a single focus class, enables batch selection of dozens to hundreds of instances in one action, and lets the interface show only instances relevant to the current class, reducing visual clutter and cognitive load. The authors formalize this as the cVIL workflow (bootstrap, class selection guidance, class-based labeling, residual labeling), implement it in a visual analytics prototype, and report that in a user study it reached significantly higher final labeling accuracy than an instance-centric interface (median 98.4% vs 96.05%, p < .001) with no significant penalty in time or workload, and that 13 of 16 participants preferred it. They also provide a complexity argument that in typical cases cVIL reduces labeling effort from O(nm) to O(n), which is what would make labeling large, class-rich datasets practical.","feed_headline":"Swap instance labeling for class labeling, accuracy climbs to 98.4%","feed_subtitle":"By labeling batches per class instead of per instance, users hit 98.4% accuracy and 13 of 16 prefer it.","key_machinery":"The central mechanism is the class-centric inversion of the labeling problem, turning a many-way class assignment into a binary membership decision per focus class. The argument is carried by the cVIL workflow: a Bootstrap phase seeds each class with at least one label; Class Selection Guidance uses property measures—quantities such as class size, imbalance, separation, collision, and uncertainty—to choose the next class to focus on; Class-Based Labeling lets users select a subset of instances predicted for that class and either confirm them all with a single click or remove false positives; and Residual Labeling switches back to instance-centric handling for outliers. The prototype interface makes this concrete through four linked views: a stacked class-label bar chart, a kernel density plot of a property measure for the focus class, a similarity scatterplot with adjustable class-rank cutoffs, and an instance labeling panel supporting batch, drag-and-drop, and per-instance actions.","core_discovery":"On its own terms, the central discovery is that the labeling decision itself is the bottleneck, and it can be restructured: when users operate class-by-class, each labeling step is a binary check ('does this batch belong to the focus class?') instead of a multi-class choice, so effort depends on the number of instances per class rather than the product of instances and classes. The paper formalizes this as the cVIL workflow, whose core phases are Class Selection Guidance (which class to focus on next, informed by property measures such as class size, imbalance, collision, and uncertainty) and Class-Based Labeling (where users select and batch-label subsets of the instances predicted for the focus class, using a kernel density plot of a property measure to find prototypical samples and a similarity scatterplot to handle borderline cases). The interface replaces categorical color encoding—which fails above about a dozen classes—with a class bar chart that scales to hundreds of classes, and supports both batch and individual labeling in the same view. Evidence comes from a user study on two binary CelebA tasks, where all 15 analyzed participants achieved higher accuracy with cVIL (median 98.4% vs 96.05%), completion time and workload did not differ significantly, and 13 of 16 preferred cVIL; a walk-through on 100 Caltech-101 classes (6,198 instances) illustrates the multi-class scenario, with manual labels at 97% accuracy and batch labels at 96.1% after labeling nearly 90% of the data.","pith_inferences":["An immediate testable extension: measure the 'batchable fraction'—the median share of a class's instances that a user can confirm in a single selection—for a given dataset and classifier; the paper's O(n) claim predicts this fraction is large, so a dataset where it is small for most classes would narrow cVIL's advantage.","The binary decision structure suggests cVIL should generalize beyond images to text, audio, or tabular data as long as a class-level summary and a property ranking can be visualized; the paper only demonstrates the mechanism on images.","The class-centric framing may also change model-training dynamics: because batch labels are confirmed in groups, the label noise structure differs from per-instance labeling, which could be exploited or accounted for in loss weighting—the paper already down-weights batch labels by a factor of 0.1, an implicit acknowledgment of this.","A controlled comparison with many classes (not just binary) would test whether the accuracy advantage persists when class selection guidance becomes a real decision, since the study's quantitative evidence is limited to two classes per task."],"forward_implications":["If cVIL's average-case complexity argument holds, labeling effort scales with the number of instances per class rather than the product of instances and classes, making datasets with dozens to hundreds of classes practical to label interactively.","Higher label accuracy at equal or lower effort implies that downstream models trained on cVIL-labeled data should be more reliable, or require less human time to reach the same accuracy.","Batch labeling shifts the human role from scrutinizing individual instances to verifying class-level predictions, which could change how annotation work is organized and how quality is audited.","Class selection guidance based on imbalance and uncertainty could be combined with active learning strategies to focus human effort on the classes that most improve the model."],"supporting_citations":[{"why":"Taxonomy of 15 property measures; underpins the property-measure guidance that makes class-based batch selection efficient.","marker":"[1]"},{"why":"Experimental comparison showing user-driven VIL can outperform active learning; motivates the VIL approach cVIL builds on.","marker":"[7]"},{"why":"Self-supervised vision transformer embeddings used as the feature representation for both interfaces in the user study and usage scenario.","marker":"[10]"},{"why":"Prior cVIL prototype this work extends; provides the interface basis and the property-measure implementation.","marker":"[11]"},{"why":"VIAL unified visual interactive labeling process; the instance-centric reference process cVIL contrasts with in the complexity comparison.","marker":"[24]"},{"why":"NASA-TLX workload measure as aggregated by Rubio et al.; supplies the workload comparison metric in the user study.","marker":"[67]"},{"why":"t-SNE projection used for the iVIL baseline interface; the instance-centric visualization cVIL is compared against.","marker":"[68]"},{"why":"CelebA dataset used for the user study's binary labeling tasks.","marker":"[69]"}],"fun_headline_variants":["Label by class, not instance: cVIL improves accuracy to 98.4%","Class-centric labeling: 98.4% accuracy, 13 of 16 prefer it","Shift labeling paradigm: cVIL scales, accuracy hits 98.4%","cVIL: label classes, not instances: 98.4% accuracy, 13/16 prefer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed efficiency gain assumes that for most classes the classifier's predictions are good enough that a user can select and confirm a large subset (the paper's example is 10% of a class) in a single batch, while an instance-centric user must re-examine a fixed fraction of the full dataset for every class; if per-class prediction quality is low, the advantage shrinks toward the same worst case as instance-centric labeling.","fun_headline_variants_meta":{"raw":{"variants":["Label by class, not instance: cVIL improves accuracy to 98.4%","Class-centric labeling: 98.4% accuracy, 13 of 16 prefer it","Shift labeling paradigm: cVIL scales, accuracy hits 98.4%","cVIL: label classes, not instances: 98.4% accuracy, 13/16 prefer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001384,"raw_usage":{"total_tokens":5653,"prompt_tokens":1042,"completion_tokens":4611,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":658,"completion_tokens_details":{"reasoning_tokens":4514}},"tokens_in":658,"tokens_out":4611,"duration_ms":33583,"temperature":1.0,"reasoning_tokens":4514,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:47:48.573524+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison on a dataset with many classes (for example, 20 or more) using a deliberately weak classifier that produces poor predictions for most classes, and measure both final accuracy and the actual size of the batches users manage to confirm per class. If the median batchable fraction per class is far below 10% for a representative dataset, or if cVIL's accuracy advantage disappears in a many-class setting, the average-case O(n) claim would be falsified.","supporting_citations":[{"cited_title":"van der Maaten, G","cited_arxiv_id":null,"evidence_quote":"t-SNE projection used for the iVIL baseline interface; the instance-centric visualization cVIL is compared against."}],"review_version":1}