{"id":"7b4a2c34-973a-47b4-9cf9-87b1c80249a5","arxiv_id":"2508.07169","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CLARITY uses interactive inductive-logic rules to group static-analysis warnings that share structural or code-level features, and reports modest user-study gains in sensemaking.","lead":"This paper presents CLARITY, a tool that lets developers group and summarize static-analysis warnings interactively by labeling examples and highlighting code expressions. A 14-person user study and a simulation suggest the approach helps developers articulate why groups of warnings can be ignored, though the simulation's headline speedup is largely built into the rule-level feedback design.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulation's headline speedup (11.8 vs 17.8 interactions) is not well-defined: rule-level feedback labels many warnings per interaction and ground-truth labels are unspecified, so faster 'alignment' may be a labeling-volume artifact.","rationale":"The reader's weakest_assumption concerned the expressiveness of the structural predicate vocabulary. That is a legitimate generalizability threat, but the more immediate load-bearing issue is that the simulation—the main quantitative evidence for faster alignment—is under-specified and its headline comparison is confounded by labels-per-interaction. The reader's rationale does mention that the simulation 'rests on unspecified ground-truth labels' and is 'partly definitional,' so there is partial agreement. However, because this specific confound is not identified as the weakest assumption, I mark partial rather than agree. The user study provides modest direct evidence that CLARITY helps participants articulate commonalities and report higher confidence, so I would not reject the paper outright. The conditional verdict remains appropriate: the manuscript should be revised to remove or re-baseline the 11.8 vs 17.8 claim and to correct the 'faster' wording, but the core sensemaking idea retains some empirical support.","tokens_in":17412,"tokens_out":7658,"duration_ms":77444,"concrete_test":"Re-run the Section V simulation after (1) specifying the ground-truth label for every Nacos Infer warning (e.g., from the replication package's human labels or fixed/unfixed history) and (2) plotting alignment against cumulative number of warnings labeled, not interactions. If rule-level feedback's advantage over instance-level feedback disappears or shrinks when controlling for labels provided, the 11.8 vs 17.8 result is a labeling-volume artifact; the paper should either retract that quantitative claim or present the per-label comparison. Separately, verify the abstract's 'align all inferred rules' against the body's 80% threshold metric.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative support for faster alignment is Section V's simulation, but its result is not interpretable as currently reported. Section V-A says the simulated user 'makes a decision for each of the selected warnings,' yet the paper never defines the ground-truth label used to make that decision. Without knowing which warnings are truly interesting/uninteresting, 'alignment with the user's feedback' (Section V-B) cannot be reproduced or given meaning. Worse, rule-level feedback labels every warning entailed by a rule in one interaction, while instance-level feedback labels exactly one warning per interaction. Comparing alignment on an interaction count then conflates the number of labels provided: a rule that matches n warnings receives n labels in one interaction under rule-level feedback, so reaching 80% alignment in fewer interactions is partly definitional, not evidence that the ILP inference is better. The abstract's 'align all inferred rules' also contradicts Section V-C, which reports reaching the 80% threshold (alpha=0.8) at 11.8 interactions, while the Introduction claims 100% alignment within 18 iterations. The user study still gives some support for better articulation and confidence, but the 'faster' claim is additionally contradicted by Section VI, which says participants took longer and inspected 24% fewer warnings. Since the simulation and the 'faster' language are prominent in the abstract and central claim, this is a load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CLARITY, an interactive tool that uses active learning and inductive logic programming to summarize static-analyzer warnings into customizable rules. The evaluation comprises a within-subject user study with 14 participants, comparing CLARITY to a one-by-one warning-list baseline, and a simulation comparing rule-level feedback with instance-level feedback. The authors report that CLARITY yields higher confidence and better articulation of commonalities, lower perceived mental demand, and that rule-level feedback aligns inferred rules in fewer interactions (11.8 vs. 17.8).","tokens_in":17770,"tokens_out":6354,"duration_ms":60664,"significance":"If the simulation result were interpretable, the paper would make a useful contribution to interactive warning triage by combining active learning with ILP over code structure. The user study is a reasonable within-subject design with a replication package and produces statistically significant effects on confidence and mental demand; the observation of high individual variation in rules is a useful empirical finding. The main quantitative support for faster alignment is, however, not currently interpretable, which caps the significance of the paper.","major_comments":[{"comment":"The simulation's ground-truth labels are never defined. Section V-A says the simulated user 'makes a decision for each of the selected warnings,' but the paper does not say what the correct label is for each warning. The alignment metric in V-B is defined relative to 'the user's labeling,' so the simulation cannot be reproduced and the alignment curves in Figures 6-7 are not interpretable. Please specify the oracle used to label warnings as interesting/uninteresting (e.g., labels from the user study, historical fixes, or a fixed rule) and explain how the simulated user's decisions are generated.","section":"V-A, V-B"},{"comment":"The comparison between rule-level and instance-level feedback conflates interaction count with the number of labels provided. In the rule-level condition (p=1), one interaction labels every warning entailed by the rule; in the instance-level condition (p=0), one interaction labels exactly one warning. Since alignment measures the fraction of a rule's matched warnings consistent with user labels, the rule-level condition has a large labeling-volume advantage by construction. The 11.8 vs. 17.8 result is therefore not evidence that the ILP inference is better. To support the claim, control for the number of labels (e.g., compare performance after the same number of labeled warnings) or report per-label alignment efficiency.","section":"V-C"},{"comment":"The abstract claims users 'articulated root causes ... faster' and that '11.8 interactions are needed to align all inferred rules.' No articulation-time measurement is reported; Section IV-B reports a 13% increase in number of common symptoms and longer written descriptions, and Section VI explicitly states participants took longer and inspected 24% fewer warnings. The simulation in V-C reports reaching the 80% alignment threshold (alpha=0.8) at 11.8 interactions, not 'all inferred rules.' The Introduction also says 'within 18 iterations, rule alignment becomes 100%.' These statements are inconsistent and should be reconciled; 'faster' should be removed unless time-to-articulation is measured.","section":"Abstract and Section V-C / VI"},{"comment":"The background knowledge is initialized only with containment, type, and API-signature predicates; code expressions are added only when the user highlights them. The paper's claim that CLARITY 'surfaces common symptoms' is therefore limited by this vocabulary. If the actual reason a warning is uninteresting is not captured by package/class/return type/fields/subtyping or a highlighted expression, the rules cannot group those warnings. The user study covers only null-pointer dereference warnings, which may be unusually well matched to these features; the generalization statement in Section VI is not directly supported. This should be presented as a limitation and, if possible, tested with other warning types or an expanded vocabulary.","section":"III-B"}],"minor_comments":[{"comment":"Section IV-B reports '13% more common symptoms' (1.53 vs. 1.35), while the Introduction says '10% more'; please harmonize the numbers.","section":"IV-B / Introduction"},{"comment":"The probability variable p is rendered as '�' in several places (e.g., '� � �', '� � � ��'). Define p with a readable symbol and ensure figures use consistent notation.","section":"V-A"},{"comment":"Axis labels and some caption text are garbled in the typeset version; provide high-resolution figures with legible labels.","section":"Figures 6-7"},{"comment":"Reference [66] appears to duplicate [28] (same title) but with a different year ('2014' vs. '2024'); verify the citation.","section":"References"},{"comment":"Section VI states participants 'took longer' but no time measurements are reported; report average task completion times or qualify the statement.","section":"VI"},{"comment":"The baseline is described as a 'downgraded version of CLARITY' rather than an unmodified existing tool. State more explicitly how the one-by-one baseline differs from CLARITY's interface and whether this could bias the comparison.","section":"IV-Baseline"}],"recommendation":"major_revision","confidential_remarks":"The simulation result is central to the paper's headline and to contribution 3; without defining ground truth and controlling for label volume, the 'faster alignment' claim should not be presented as established. If the authors cannot fix this, the speedup claim should be weakened to a pilot observation and the paper should focus on the user-study findings. The inconsistency between the abstract's 'align all inferred rules' and the actual 80% threshold should be corrected by the authors before any decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"If you're deciding whether to spend time on this, read it for the tool and the user study. CLARITY is a sensible answer to a well-documented problem: static-analysis warnings are triaged one-by-one, and developers struggle to articulate why many are uninteresting. The idea of using ILP to infer summary rules over containment, subtype, and API-signature predicates, with user-highlighted code expressions as extra cues, is a real new combination. The user study, though only 14 participants, is a proper within-subject design and shows genuine benefits: participants articulated more commonalities (13% more), felt more confident (3.4 vs 2.6), and reported lower mental demand (4.8 vs 5.4), with the latter two statistically significant at p<0.05. Each participant also formulated at least one unique rule, reinforcing the configurability argument.\n\nThe paper overreaches in its central claim. The abstract says users articulated root causes 'faster' with CLARITY, but the user study actually reports they inspected 24% fewer warnings and took longer—the authors explain this honestly in Section VI, so the abstract contradicts their own results. The bigger issue is the simulation in Section V. The headline result (11.8 vs 17.8 interactions to align rules) is not well-defined. The ground-truth labels the simulated user uses are never described, so 'alignment' is uninterpretable. And rule-level feedback by construction labels every warning entailed by a rule in one interaction, while instance-level feedback labels exactly one warning; comparing interaction counts then mostly measures label volume, not inference quality. The paper also mixes thresholds: the abstract says 'align all inferred rules' but the simulation actually reports reaching 80% alignment. That's a real numeric inconsistency.\n\nThere is a softer concern too: the background knowledge is seeded with a fixed vocabulary (package, class, return type, fields, subtyping) and code expressions only if the user thinks to highlight them. If the real reason a warning is uninteresting isn't in that vocabulary, the rules can't capture it. The highlighting mechanism is a partial remedy, but the paper doesn't test how much it helps when users don't know what to highlight.\n\nBottom line: read it for the user study and the tool design; take the simulation with a grain of salt. I'd send it to review with a request for major revisions—the simulation needs proper formalization or removal, and the speed language needs to align with the evidence. It's a serious paper, but it overclaims in places.","headline":"Worth reading for the tool and user study; the simulation's headline speedup is largely definitional and the abstract overstates speed.","tokens_in":18186,"tokens_out":5186,"would_cite":true,"duration_ms":38932,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLARITY turns static-analysis warning review into an interactive, rule-learning sensemaking process.","keywords":["static analysis warnings","interactive sensemaking","inductive logic programming","active learning","warning triage","summary rules","code expressions","user study"],"falsifier":"Give CLARITY a corpus of warnings where the uninteresting ones share a common cause invisible to its predicates—for example, a comment marker, a project configuration flag, or a dataflow property such as 'this value comes from a deserializer'—and measure rule alignment under a user who labels by that cause. If alignment cannot be driven above the 80% threshold no matter how many rule-level interactions are allowed, the vocabulary premise fails.","tokens_in":17366,"feed_emoji":"🧩","tokens_out":7302,"duration_ms":69547,"temperature":0.7,"pith_summary":"Static-analysis tools flood developers with warnings, many of them uninteresting, and current interfaces show them one at a time. CLARITY proposes to make warning triage a sensemaking activity: as a developer marks a few warnings as interesting or uninteresting and optionally highlights a code expression, a rule-inference engine builds summary rules that group similar warnings by package, class, subtype, return type, used fields, invoked methods, and highlighted expressions. The paper's claim is that this rule-level abstraction is learnable from a handful of labels and genuinely changes how developers reason about alerts. In a within-subject study with 14 participants, CLARITY users articulated more common symptoms of uninteresting warnings, wrote longer explanations, and reported higher confidence and lower mental demand than a one-by-one baseline; a simulation on null-pointer warnings from one of the study's Java projects indicates that rule-level feedback reaches 80% rule alignment in 11.8 interactions on average versus 17.8 with instance-level feedback only. The authors read these results as evidence that interactive, user-guided summarization—not just filtering or ranking—is a viable way to make bug-finding tools digestible.","feed_headline":"CLARITY turns warning lists into user-tailored summary rules","feed_subtitle":"Developers grasp warning root causes better, and rule feedback aligns in 11.8 interactions vs 17.8.","key_machinery":"The mechanism is inductive logic programming (ILP) over a background knowledge base. CLARITY seeds the knowledge base with containment, subtyping, and API-signature facts for each warning's implicated code, and augments it with facts about expressions the user highlights. An off-the-shelf logic-program solver searches for a hypothesis: an assignment of predicates to rules such that every rule matches as many labeled uninteresting warnings as possible while matching no labeled interesting warning, breaking ties toward generality. The rules are presented to the user with per-rule warning distributions, and the user can either label individual warnings or apply a rule to all matched warnings, c","core_discovery":"The central claim is that warnings from a bug-finding tool can be made sense of by learning a small set of summary rules from the user's own judgments, rather than by presenting each warning in isolation. A rule is a conjunction of logic predicates drawn from a seeded background knowledge of containment (package, class), typing (subtype/interfaces), and API signatures (return type, used fields), plus predicates for code expressions that the user explicitly highlights. The learner is required to cover the warnings the user has marked uninteresting, avoid warnings marked interesting, and maximize the number of still-uninspected warnings matched. The paper reports that this approach yields fast","pith_inferences":["Beyond the paper's null-pointer and resource-leak settings, the same rule-induction loop should apply to any warning class whose false positives are localized by package, type, or API usage; the paper conjectures this but does not test it.","The design implicitly bets on the user's ability to notice salient expressions: expression predicates are harvested only from highlighted code, so a user who does not highlight misses those groupings. A tool could test this by prompting users for highlights or mining frequent expressions automatically.","Rule alignment in the simulation assumes the simulated user's labels are themselves expressible as conjunctions of the available predicates; in practice some users may group warnings by criteria no predicate can state, which would make perfect alignment unattainable.","The user-study result that participants examined fewer warnings under CLARITY suggests that measuring success by warnings-per-minute would miss the point; a better productivity metric may be number of distinct root causes understood per unit time."],"forward_implications":["Warning review can shift from per-alert inspection to group-level reasoning: one rule can stand in for many warnings that share a package, subtype, or API call.","Rule-level feedback is a real efficiency lever: aligning inferred rules with a simulated user's labels takes 11.8 interactions on average with rule-level feedback versus 17.8 with instance-level feedback only.","Because 14 participants produced 55 rules with at least one unique rule each, a one-size-fits-all suppression policy is unlikely to match individual developers' mental models.","CLARITY-style rules can serve as codified, shareable suppression policies that capture why a team ignores certain warnings, not just which warnings to ignore.","Sensemaking improvements come with a cost: participants took longer and examined fewer warnings, suggesting the benefit is in understanding, not raw throughput."],"supporting_citations":[{"why":"Establishes the adoption problem: developers resist static-analysis tools because of warning overload.","marker":"[3]"},{"why":"Shows developers want and need program analysis features, motivating interactive review.","marker":"[4]"},{"why":"Supplies the open-source Java project whose null-pointer warnings drive the simulation and motivating analysis.","marker":"[8]"},{"why":"Provides the variational theory of learning that frames summary rules as a way to form concepts.","marker":"[11]"},{"why":"Shows uninteresting warnings correlate with code locality and organization, justifying the containment predicates.","marker":"[12]"},{"why":"Provides the standard command-line warning inspection interface used as the baseline.","marker":"[19]"},{"why":"Supplies the inductive logic programming formulation used to infer rules from labels.","marker":"[23]"},{"why":"Provides the solver used to search for candidate rule hypotheses.","marker":"[24]"},{"why":"Supplies the second Java project whose warnings form the other user-study task.","marker":"[38]"}],"fun_headline_variants":["CLARITY learns your warning grouping rules from a few marks","Mark warnings as noise, CLARITY finds the common threads","11.8 interactions to align warning rules with CLARITY","Group warnings your way: CLARITY's active learning summaries","Interactive summaries turn warning noise into knowledge"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the things that make a warning uninteresting can be expressed with the small fixed vocabulary CLARITY starts from—package, class, return type, used fields, subtyping, and any code expression a user happens to highlight. If the real common cause lives in a feature outside that vocabulary, CLARITY's rules cannot group those warnings, and the improvement disappears. The paper acknowledges part of this in its design by seeding only containment/typ","fun_headline_variants_meta":{"raw":{"variants":["CLARITY learns your warning grouping rules from a few marks","Mark warnings as noise, CLARITY finds the common threads","11.8 interactions to align warning rules with CLARITY","Group warnings your way: CLARITY's active learning summaries","Interactive summaries turn warning noise into knowledge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00134,"raw_usage":{"total_tokens":5271,"prompt_tokens":723,"completion_tokens":4548,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":4477}},"tokens_in":467,"tokens_out":4548,"duration_ms":32643,"temperature":1.0,"reasoning_tokens":4477,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T22:16:46.045278+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give CLARITY a corpus of warnings where the uninteresting ones share a common cause invisible to its predicates—for example, a comment marker, a project configuration flag, or a dataflow property such as 'this value comes from a deserializer'—and measure rule alignment under a user who labels by that cause. If alignment cannot be driven above the 80% threshold no matter how many rule-level interactions are allowed, the vocabulary premise fails.","supporting_citations":[{"cited_title":"What developers want and need from program analysis: an empirical study,","cited_arxiv_id":null,"evidence_quote":"Shows developers want and need program analysis features, motivating interactive review."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the open-source Java project whose null-pointer warnings drive the simulation and motivating analysis."},{"cited_title":"Marton, Necessary conditions of learning","cited_arxiv_id":null,"evidence_quote":"Provides the variational theory of learning that frames summary rules as a way to form concepts."},{"cited_title":"Correlation exploita- tion in error ranking,","cited_arxiv_id":null,"evidence_quote":"Shows uninteresting warnings correlate with code locality and organization, justifying the containment predicates."},{"cited_title":"Infer explore,","cited_arxiv_id":null,"evidence_quote":"Provides the standard command-line warning inspection interface used as the baseline."},{"cited_title":"Inductive logic programming at 30,","cited_arxiv_id":null,"evidence_quote":"Supplies the inductive logic programming formulation used to infer rules from labels."},{"cited_title":"gringo, clasp, clingo, and iclingo,","cited_arxiv_id":null,"evidence_quote":"Provides the solver used to search for candidate rule hypotheses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the second Java project whose warnings form the other user-study task."}],"review_version":1}