{"id":"8e29a584-2370-47eb-bca2-f69f56bf41b9","arxiv_id":"2505.05326","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"TS-Detector implements five feature toggle smell detectors for six languages, but its reported detection rates are contradicted by the paper's own evaluation table.","lead":"TS-Detector is a new tool that automatically finds five feature toggle usage patterns across six programming languages. The paper's manual evaluation is internally inconsistent, so the reported accuracy rates do not hold up to scrutiny.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground truth for the manual evaluation is generated from the same tool logic, so the reported TP/FP/FN rates measure self-consistency, not detection accuracy.","rationale":"This is not a claim of dishonesty; the paper may be a useful tool paper. But as an empirical validation, the manual evaluation is not an independent oracle. The explicit phrase 'based on the logic built' (Sec. 3.1) makes the circularity concrete. A tool can be self-consistent while being wrong about what counts as a smell. Independent labels are the only way to separate these. Table 1's arithmetic inconsistencies (e.g., Dawn Spread: TP+FN=120, Manual=123; Sentry Spread: TP+FP=203, TS=200) reinforce that even the reported rates are not reproducible as stated. Because the central claim is quantitative accuracy, and that quantity lacks a valid reference standard, the reader's REJECT verdict is appropriate. The tool itself may still be useful and the repository should not be dismissed, but the empirical claims should not be relied upon until independently validated.","tokens_in":6138,"tokens_out":4897,"duration_ms":50212,"concrete_test":"Create an independent gold standard: randomly sample 100 toggle instances across the five manually evaluated projects (or all instances if feasible). Have two annotators, blind to TS-Detector's output and to the tool's internal regex/heuristics, label each instance as Spread, Nested, Dead, Mixed, Enum, or Not-a-toggle using only the published pattern definitions from the authors' prior work [14,15]. Compute TP/FP/FN against TS-Detector's output and report inter-annotator agreement. If the recomputed true positive/negative rates differ materially from 80%, 86.4%, 66.6%, and 100%, or if the annotators cannot agree or cannot apply the published definitions, the reported rates are not externally valid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that TS-Detector reliably detects five toggle usage patterns with the reported rates—rests on the manual evaluation in Section 4 being an independent ground truth. Section 3.1 states that 'we also employ a team of human developers to find toggle usage patterns based on the logic built' in the tool, and Section 4 says the comparison is between TS-Detector's output and those manual findings. If the human evaluators were given the tool's own detector definitions, regexes, and heuristics, then TP/FP/FN only measure whether the implementation matches its own rules; they do not establish that the detected instances are real toggle smells. No protocol, number of evaluators, annotation instructions, or inter-rater agreement is reported, so independence cannot be checked. Independent support is also weakened by internal arithmetic: in Table 1, Dawn Spread has TP+FN=120 vs Manual=123 and Sentry Spread has TP+FP=203 vs TS-Detector=200; the rates in the text (e.g., 99.3% for Temporal Spread) require an unstated denominator. The core accuracy claim therefore lacks validated support.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TS-Detector, a Python-based static analysis tool that detects five feature-toggle usage patterns (dead, nested, spread, mixed, enum) in Java, C/C++, Python, Go, and C#. The authors describe the tool's architecture, command-line interface, and per-pattern detectors, and report an evaluation on ten open-source projects, with manual evaluation on five of them. The abstract and conclusion claim true positive rates of 80% for Spread, 86.4% for Nested, and 66.6% for Dead, and a 100% true negative rate for Mixed and Enum usages. The paper also contributes lessons learned from the tool development and points to a public GitHub repository.","tokens_in":6323,"tokens_out":6697,"duration_ms":67083,"significance":"If the reported accuracy were established, TS-Detector would fill a real gap: no existing tool appears to detect these toggle usage patterns across multiple languages, and the public availability of the implementation could support follow-up research and practitioner adoption. The paper's strengths include the open-source release, the multi-language support, the explicit architecture with separated detectors and unit tests, and the concrete input/output interface. However, the evaluation is not currently sufficient to validate the central accuracy claims: the manual ground truth is constructed from the tool's own logic, the reported counts contain internal inconsistencies, and the metric terminology is ambiguous. The work is best viewed as a promising tool demonstration whose empirical evidence requires substantial revision.","major_comments":[{"comment":"The manual evaluation does not provide an independent ground truth. Section 3.1 states that 'we also employ a team of human developers to find toggle usage patterns based on the logic built' in the tool, and Section 4 compares TS-Detector's output against those manual findings. Because the human evaluators were instructed using the tool's own detector definitions, heuristics, and regular expressions, the reported TP/FP/FN values measure whether the implementation conforms to its own rules, not whether the detected instances correspond to real toggle smells in the projects. The paper does not report the number of evaluators, the annotation instructions, or any inter-rater agreement measure, so the independence of the ground truth cannot be assessed. This issue is load-bearing because every accuracy figure in the abstract and conclusion derives from this comparison.","section":"Sections 3.1 and 4"},{"comment":"Table 1 contains arithmetic inconsistencies that prevent reproduction of the stated results. For Sentry Spread, TS-Detector reports 200 detected instances but TP+FP=124+79=203; for Sentry Dead, TS-Detector reports 131 but TP+FP=97+134=231; and for Dawn Spread, Manual reports 123 but TP+FN=116+4=120. These contradictions mean the counts are unreliable, and the aggregate percentages quoted in the text and abstract cannot be verified from the table. The table must be corrected and the per-project rates should be recomputed from consistent numbers before any accuracy claim can be evaluated.","section":"Table 1 and Section 4"},{"comment":"The accuracy terminology is inconsistent between precision, recall, and true positive rate. The abstract reports 'true positive rates' of 80%, 86.4%, and 66.6% for Spread, Nested, and Dead respectively, but the text in Section 4 computes these as TP/(TP+FP) (e.g., OpenSearch Spread 4/5=80% and Sentry Nested 70/81=86.4%), which is precision, not the true positive rate. Some sentences use other denominators, such as 'TP 99.3%' for Temporal Spread, which equals TP/(tool total) rather than TP/(manual total). In addition, the claimed '100% true negative rate' for Mixed and Enum usages is not meaningful because no Mixed or Enum instances were found in the evaluated projects; a true negative rate cannot be established when the negative class is effectively untested.","section":"Abstract and Section 4"},{"comment":"The manual evaluation protocol is underspecified, which undermines reproducibility. The paper does not state how many human evaluators participated, what instructions they were given, whether they had access to the tool's source code and regex definitions, or how disagreements among evaluators were resolved. The sentence that the projects were selected 'randomly, regardless of any special consideration' is not backed by a description of the random selection process or its seed. Without this information, the reader cannot check for bias in the selection or the labeling, and the reported TP/FP/FN values cannot be independently verified.","section":"Section 4"}],"minor_comments":[{"comment":"The sentence 'One additional usage was identified (false negative FN) by the tool which was not actually a toggle usage and wasn't identified manually' describes a false positive, not a false negative, because the tool found something that was not a real toggle usage; the prose contradicts the table's FN column.","section":"Section 4, OpenSearch Spread paragraph"},{"comment":"The text lists 'six popular languages (i.e. Java, C/C++, Python, Go, and C#)' but only five language categories are named; please clarify how C and C++ are counted or correct the language count.","section":"Section 3.1"},{"comment":"The caption repeats 'Spread Usage Detector' twice; the second occurrence should likely be 'Enum Usage Detector' or another detector name.","section":"Figure 2 caption"},{"comment":"The word 'gobals' should be 'globals' in the sentence about Sentry and Server having many regular constants and global variables.","section":"Section 4"},{"comment":"The paper references the GitHub repository but does not provide a version tag or commit hash, making the artifact less reproducible for readers who want to confirm the reported results.","section":"GitHub repository"}],"recommendation":"reject","confidential_remarks":"The manuscript's central accuracy claims rest on an evaluation whose ground truth is generated from the tool's own logic, and the reported tables contain arithmetic errors. The paper may be more suitable as a tool demonstration without quantitative claims, or after a full external evaluation with an independent annotation protocol and corrected data. I would recommend inviting a resubmission after such an evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this is a short FSE Companion tool paper. The tool is real and plausibly useful — it supports six languages and is on GitHub with a demo video. But the evaluation section has load-bearing problems, and I would not trust the reported true-positive rates as evidence of detection quality.\n\nWhat is actually new: the authors extend their earlier C++-only toggle detector to C, Java, Python, Go, and C#, apply it to ten open-source projects, and manually evaluate on five. The architecture is sensible: separate detectors per pattern, config-file-based toggle extraction, and unit tests are mentioned. They also openly discuss why Sentry and Server produce many false positives (config files mixing toggles with regular constants), which is honest reporting.\n\nThe soft spots are serious. Most importantly, the manual ground truth is not independent. Section 3.1 says human evaluators were asked to find toggle usage patterns “based on the logic built” in the tool. That makes the evaluation a self-consistency check, not a validation against real toggle smells. No protocol, number of evaluators, or inter-rater agreement is given. Table 1 also has internal arithmetic contradictions — e.g., Sentry Dead has TP+FP=231 while TS-Detector supposedly found 131, and Dawn Spread has TP+FN=120 while manual found 123. The text occasionally mislabels false negatives. The claimed 100% true negative rate for Mixed and Enum is vacuous because those patterns were absent from the manually checked projects. And the abstract's “no tool as of today” claim sits oddly next to the statement that this tool was initiated in a previous study.\n\nThese flaws are not minor copy-editing issues; they undermine the empirical contribution. Still, the underlying engineering might be salvageable. If the authors rerun the evaluation with independently constructed ground truth, fix the table arithmetic, report per-project precision/recall with clear definitions, and tone down the novelty claim, the paper could become a useful data point for the feature-toggle community.\n\nWho this is for: researchers and practitioners working on feature flags, configuration options, or code-smell detection. The paper deserves a serious referee in the sense that the topic is relevant and the artifact is concrete, but as submitted I would expect major revision or rejection. If I were refereeing, my recommendation would be: revise and resubmit after the evaluation is redone properly.","headline":"A real multi-language toggle-smell detector, but the evaluation as reported is circular and the headline numbers don't survive close reading of the paper's own table.","tokens_in":6871,"tokens_out":3185,"would_cite":false,"duration_ms":31835,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces TS-Detector, a static analyzer that scans configuration files and source code to detect dead, nested, spread, mixed, and enum feature toggle usage patterns across six programming languages, reporting true-positive…","keywords":["feature toggle","toggle smell","static analysis","dead toggle","nested toggle","spread toggle","mixed usage","enum usage"],"falsifier":"Have a fresh team of developers who have never seen the tool's regex logic independently label all spread, nested, and dead toggle usages in Sentry and Server directly from the source code, then compare their labels with TS-Detector's output; if agreement is no better than the tool's match with its own logic-trained labels, the reported accuracy figures are not supported.","tokens_in":5890,"feed_emoji":"⚙️","tokens_out":5687,"duration_ms":52465,"temperature":0.7,"pith_summary":"Feature toggles are conditional switches that teams use to roll out features gradually, but when used carelessly they leave behind code that is hard to maintain. The paper presents TS-Detector, a static analysis tool that reads a project's toggle configuration file and searches source code for five toggle usage patterns: dead, nested, spread, mixed, and enum. Tested on ten open-source projects in Java, C/C++, Python, Go, and C#, with manual review of five, the tool reports true-positive rates of 80% for spread, 86.4% for nested, and 66.6% for dead usage, and no false positives for mixed or enum usage in the manually checked projects. The authors claim this is the first tool of its kind for detecting toggle usage patterns.","feed_headline":"TS-Detector flags feature-toggle smells in six languages","feed_subtitle":"Static analysis catches spread, nested, dead, mixed, and enum toggles with up to 100% true positives.","key_machinery":"The central machinery is a pipeline that reads toggle names from a project's configuration file, extracts source-code contexts using language-specific regular expressions, and applies five specialized detectors: Dead, Nested, Spread, Mixed, and Enum. The Spread detector uses a component boundary, such as a class, a Go package, or a C# namespace, to decide how widely a toggle is used; the Nested detector tracks toggles inside nested structures and assignments; and the Mixed and Enum detectors target restricted C and C++ patterns. This architecture is what lets one tool span six programming languages.","core_discovery":"The paper claims that TS-Detector is the first tool able to detect toggle usage patterns directly from source code, and that its multi-language design generalizes the algorithms previously developed for C++ and Chromium. The tool identifies toggle variables from configuration files, filters them with naming and keyword rules, then applies pattern-specific detectors; the reported evaluation, with true positives of 80% for spread, 86.4% for nested, and 66.6% for dead toggles, plus a 100% true-negative rate for mixed and enum usages, is presented as evidence that the detection is reliable enough for practical use. The authors also report that mixing regular constants and global variables with toggles in the same configuration file is the main cause of false positives.","pith_inferences":["The paper leaves implicit that its accuracy figures measure consistency with its own definitions: because the human evaluators labeled patterns 'based on the logic built' into the tool, an independent blind evaluation could report different true-positive rates.","A concrete next test is running TS-Detector on Google Chromium, where the prior Combinatorial pattern was observed, to see whether the mixed and enum detectors generalize beyond the five manually checked projects.","A testable implication for teams is that separating feature toggles from ordinary constants and globals in configuration files should improve precision, since Sentry and Server show that mixed configuration files are the main source of false positives."],"forward_implications":["A project that keeps toggles in a dedicated configuration file can run TS-Detector with a single command and receive a JSON list of spread, nested, and dead toggle locations to review and refactor.","The five detector functions can be extended to a new language by supplying language-specific regex patterns and a component boundary such as Go packages or C# namespaces.","Detected spread and nested toggles can be paired with the earlier finding that toggles increase code complexity, giving maintainers a prioritized refactoring list.","For projects like Sentry and Server that share a configuration file among toggles, globals, and constants, practitioners should expect false positives and may need to split the configuration first."],"supporting_citations":[{"why":"Defines feature toggle usage patterns and reports practitioner practices and a case study; it is the origin of the five patterns the tool detects.","marker":"[14]"},{"why":"Provides the toggle usage pattern algorithms and the finding that Combinatorial usage appears only in Chromium, shaping which patterns TS-Detector implements.","marker":"[15]"},{"why":"Earlier tool version and study linking feature toggles to code complexity; TS-Detector extends its C++-only detection logic to other languages.","marker":"[16]"},{"why":"Fowler's article defines feature toggles and their lifecycle concerns, motivating why detecting toggle smells matters.","marker":"[5]"},{"why":"Classifies feature toggles into five types from a practitioner study, supporting the premise that toggle usage is widespread and varied.","marker":"[10]"}],"fun_headline_variants":["TS-Detector: first tool to detect toggle smells in code","TS-Detector flags five toggle patterns in six languages","Spot feature-toggle usage patterns with TS-Detector","TS-Detector catches spread, nested, dead, mixed toggles","New tool detects toggle usage patterns from source"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy claim depends on the manual labels being an independent ground truth, but the evaluators were asked to find patterns 'based on the logic built' into the tool, so the comparison may just confirm that the tool follows its own rules.","fun_headline_variants_meta":{"raw":{"variants":["TS-Detector: first tool to detect toggle smells in code","TS-Detector flags five toggle patterns in six languages","Spot feature-toggle usage patterns with TS-Detector","TS-Detector catches spread, nested, dead, mixed toggles","New tool detects toggle usage patterns from source"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":2005,"prompt_tokens":854,"completion_tokens":1151,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":1068}},"tokens_in":470,"tokens_out":1151,"duration_ms":11153,"temperature":1.0,"reasoning_tokens":1068,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:06:25.038943+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a fresh team of developers who have never seen the tool's regex logic independently label all spread, nested, and dead toggle usages in Sentry and Server directly from the source code, then compare their labels with TS-Detector's output; if agreement is no better than the tool's match with its own logic-trained labels, the reported accuracy figures are not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines feature toggle usage patterns and reports practitioner practices and a case study; it is the origin of the five patterns the tool detects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the toggle usage pattern algorithms and the finding that Combinatorial usage appears only in Chromium, shaping which patterns TS-Detector implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Earlier tool version and study linking feature toggles to code complexity; TS-Detector extends its C++-only detection logic to other languages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Fowler's article defines feature toggles and their lifecycle concerns, motivating why detecting toggle smells matters."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Classifies feature toggles into five types from a practitioner study, supporting the premise that toggle usage is widespread and varied."}],"review_version":1}