Pith. sign in

REVIEW 4 major objections 5 minor 21 references

TS-Detector : Detecting Feature Toggle Usage Patterns

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces TS-Detector, a static analyzer that scans configuration files and source code to detect dead, nested, spread, mixed, and enum feature toggle usage patterns across six programming languages, reporting true-positive…

desk verdict A real multi-language toggle-smell detector, but the evaluation as reported is circular and the headline numbers don't survive close reading of the paper's own table. read the letter →

arxiv 2505.05326 v1 pith:ZX2SXV45 submitted 2025-05-08 cs.SE

Tajmilur Rahman , Mengzhe Fei , Tushar Sharma , Chanchal Roy This is my paper · ORCID
classification cs.SE
keywords featuretogglesmellstaticanalysisdeadnestedspreadmixedusageenum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Feature toggles are conditional switches that teams use to roll out features gradually, but when used carelessly they leave behind code that is hard to maintain. The paper presents TS-Detector, a static analysis tool that reads a project's toggle configuration file and searches source code for five toggle usage patterns: dead, nested, spread, mixed, and enum. Tested on ten open-source projects in Java, C/C++, Python, Go, and C#, with manual review of five, the tool reports true-positive rates of 80% for spread, 86.4% for nested, and 66.6% for dead usage, and no false positives for mixed or enum usage in the manually checked projects. The authors claim this is the first tool of its kind for detecting toggle usage patterns.

What carries the argument

The central machinery is a pipeline that reads toggle names from a project's configuration file, extracts source-code contexts using language-specific regular expressions, and applies five specialized detectors: Dead, Nested, Spread, Mixed, and Enum. The Spread detector uses a component boundary, such as a class, a Go package, or a C# namespace, to decide how widely a toggle is used; the Nested detector tracks toggles inside nested structures and assignments; and the Mixed and Enum detectors target restricted C and C++ patterns. This architecture is what lets one tool span six programming languages.

What would settle it

Have a fresh team of developers who have never seen the tool's regex logic independently label all spread, nested, and dead toggle usages in Sentry and Server directly from the source code, then compare their labels with TS-Detector's output; if agreement is no better than the tool's match with its own logic-trained labels, the reported accuracy figures are not supported.

Watch

Extended reading notes

Core claim

The paper claims that TS-Detector is the first tool able to detect toggle usage patterns directly from source code, and that its multi-language design generalizes the algorithms previously developed for C++ and Chromium. The tool identifies toggle variables from configuration files, filters them with naming and keyword rules, then applies pattern-specific detectors; the reported evaluation, with true positives of 80% for spread, 86.4% for nested, and 66.6% for dead toggles, plus a 100% true-negative rate for mixed and enum usages, is presented as evidence that the detection is reliable enough for practical use. The authors also report that mixing regular constants and global variables with toggles in the same configuration file is the main cause of false positives.

Load-bearing premise

The accuracy claim depends on the manual labels being an independent ground truth, but the evaluators were asked to find patterns 'based on the logic built' into the tool, so the comparison may just confirm that the tool follows its own rules.

Editorial extensions

If this is right

  • A project that keeps toggles in a dedicated configuration file can run TS-Detector with a single command and receive a JSON list of spread, nested, and dead toggle locations to review and refactor.
  • The five detector functions can be extended to a new language by supplying language-specific regex patterns and a component boundary such as Go packages or C# namespaces.
  • Detected spread and nested toggles can be paired with the earlier finding that toggles increase code complexity, giving maintainers a prioritized refactoring list.
  • For projects like Sentry and Server that share a configuration file among toggles, globals, and constants, practitioners should expect false positives and may need to split the configuration first.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that its accuracy figures measure consistency with its own definitions: because the human evaluators labeled patterns 'based on the logic built' into the tool, an independent blind evaluation could report different true-positive rates.
  • A concrete next test is running TS-Detector on Google Chromium, where the prior Combinatorial pattern was observed, to see whether the mixed and enum detectors generalize beyond the five manually checked projects.
  • A testable implication for teams is that separating feature toggles from ordinary constants and globals in configuration files should improve precision, since Sentry and Server show that mixed configuration files are the main source of false positives.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents TS-Detector, a Python-based static analysis tool that detects five feature-toggle usage patterns (dead, nested, spread, mixed, enum) in Java, C/C++, Python, Go, and C#. The authors describe the tool's architecture, command-line interface, and per-pattern detectors, and report an evaluation on ten open-source projects, with manual evaluation on five of them. The abstract and conclusion claim true positive rates of 80% for Spread, 86.4% for Nested, and 66.6% for Dead, and a 100% true negative rate for Mixed and Enum usages. The paper also contributes lessons learned from the tool development and points to a public GitHub repository.

Significance. If the reported accuracy were established, TS-Detector would fill a real gap: no existing tool appears to detect these toggle usage patterns across multiple languages, and the public availability of the implementation could support follow-up research and practitioner adoption. The paper's strengths include the open-source release, the multi-language support, the explicit architecture with separated detectors and unit tests, and the concrete input/output interface. However, the evaluation is not currently sufficient to validate the central accuracy claims: the manual ground truth is constructed from the tool's own logic, the reported counts contain internal inconsistencies, and the metric terminology is ambiguous. The work is best viewed as a promising tool demonstration whose empirical evidence requires substantial revision.

major comments (4)
  1. [Sections 3.1 and 4] The manual evaluation does not provide an independent ground truth. Section 3.1 states that 'we also employ a team of human developers to find toggle usage patterns based on the logic built' in the tool, and Section 4 compares TS-Detector's output against those manual findings. Because the human evaluators were instructed using the tool's own detector definitions, heuristics, and regular expressions, the reported TP/FP/FN values measure whether the implementation conforms to its own rules, not whether the detected instances correspond to real toggle smells in the projects. The paper does not report the number of evaluators, the annotation instructions, or any inter-rater agreement measure, so the independence of the ground truth cannot be assessed. This issue is load-bearing because every accuracy figure in the abstract and conclusion derives from this comparison.
  2. [Table 1 and Section 4] Table 1 contains arithmetic inconsistencies that prevent reproduction of the stated results. For Sentry Spread, TS-Detector reports 200 detected instances but TP+FP=124+79=203; for Sentry Dead, TS-Detector reports 131 but TP+FP=97+134=231; and for Dawn Spread, Manual reports 123 but TP+FN=116+4=120. These contradictions mean the counts are unreliable, and the aggregate percentages quoted in the text and abstract cannot be verified from the table. The table must be corrected and the per-project rates should be recomputed from consistent numbers before any accuracy claim can be evaluated.
  3. [Abstract and Section 4] The accuracy terminology is inconsistent between precision, recall, and true positive rate. The abstract reports 'true positive rates' of 80%, 86.4%, and 66.6% for Spread, Nested, and Dead respectively, but the text in Section 4 computes these as TP/(TP+FP) (e.g., OpenSearch Spread 4/5=80% and Sentry Nested 70/81=86.4%), which is precision, not the true positive rate. Some sentences use other denominators, such as 'TP 99.3%' for Temporal Spread, which equals TP/(tool total) rather than TP/(manual total). In addition, the claimed '100% true negative rate' for Mixed and Enum usages is not meaningful because no Mixed or Enum instances were found in the evaluated projects; a true negative rate cannot be established when the negative class is effectively untested.
  4. [Section 4] The manual evaluation protocol is underspecified, which undermines reproducibility. The paper does not state how many human evaluators participated, what instructions they were given, whether they had access to the tool's source code and regex definitions, or how disagreements among evaluators were resolved. The sentence that the projects were selected 'randomly, regardless of any special consideration' is not backed by a description of the random selection process or its seed. Without this information, the reader cannot check for bias in the selection or the labeling, and the reported TP/FP/FN values cannot be independently verified.
minor comments (5)
  1. [Section 4, OpenSearch Spread paragraph] The sentence 'One additional usage was identified (false negative FN) by the tool which was not actually a toggle usage and wasn't identified manually' describes a false positive, not a false negative, because the tool found something that was not a real toggle usage; the prose contradicts the table's FN column.
  2. [Section 3.1] The text lists 'six popular languages (i.e. Java, C/C++, Python, Go, and C#)' but only five language categories are named; please clarify how C and C++ are counted or correct the language count.
  3. [Figure 2 caption] The caption repeats 'Spread Usage Detector' twice; the second occurrence should likely be 'Enum Usage Detector' or another detector name.
  4. [Section 4] The word 'gobals' should be 'globals' in the sentence about Sentry and Server having many regular constants and global variables.
  5. [GitHub repository] The paper references the GitHub repository but does not provide a version tag or commit hash, making the artifact less reproducible for readers who want to confirm the reported results.

Circularity Check

1 steps flagged · score 7.0 of 10

Manual evaluation ground truth is generated from the tool's own detection logic, so reported TP/FP/FN rates measure self-consistency, not independent detection accuracy.

  1. self definitional [Section 3.1 (Overview), with evaluation methodology in Section 4]
    "Once the tool is developed we test it on ten open source projects. We also employ a team of human developers to find toggle usage patterns based on the logic built"

    The paper's central accuracy claims rest on a manual evaluation in Section 4 in which human evaluators' findings are treated as the reference for True Positive, False Positive, and False Negative counts. However, the evaluators were instructed to find toggle usage patterns 'based on the logic built' in the tool, i.e., using the same regexes, heuristics, language rules, and pattern definitions that TS-Detector implements. Agreement between the manual labels and the tool output therefore only demonstrates that the implementation is consistent with its own specification; it does not show that the detected instances are real toggle smells.

full rationale

The paper's claimed derivation chain is: define five toggle usage patterns from prior work by the same authors, implement detection logic, run the tool on ten projects, and manually evaluate on five. The load-bearing step is the manual evaluation, and Section 3.1 explicitly says the human evaluators were asked to identify patterns 'based on the logic built' in the tool. This makes the ground truth labels dependent on the very implementation whose accuracy is being measured, so the reported TP/FP/FN rates and percentages from Section 4 are not independent evidence of detection quality. The tool may still be a useful engineering artifact, and unit tests or application to real projects could show internal behavior, but the published accuracy numbers reduce to a self-consistency check by construction. No external benchmark, independent oracle, or machine-checked specification is provided to offset this. The score is 7 because the central claim partially collapses into a self-defined ground truth, while the paper does contain independent engineering content such as multi-language support, algorithms, and project applications.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the five pattern definitions from the authors' earlier papers, a set of hand-tuned regexes and filter rules, and a manual ground truth produced with the same logic the tool implements. No new physical entities are introduced; instead, the free parameters are the heuristic choices that gate toggle identification.

free parameters (3)
  • Toggle variable filter rules (duplicates, name length, language keywords)
    Section 3.2 states helper functions filter toggle variables 'based on rules such as duplicates, name length, and specific language keywords.' These thresholds and rules are hand-chosen and directly affect which variables are treated as toggles, hence the detection output.
  • Component granularity for spread toggles (class vs package vs namespace)
    Section 3.2 says 'in most cases, we choose a class to break up the code' but for Go, a package is used. This choice defines what counts as 'spread' and is a tuning parameter without justification.
  • Language-specific regex patterns
    Section 3.2 mentions 'language-specific regex for toggle patterns.' The exact regexes are not given in the paper; they are artifacts that determine detection behavior.
assumptions (4)
  • domain assumption The five usage patterns (dead, nested, spread, mixed, enum) defined in prior studies are the correct and sufficient set of toggle smell patterns.
    Section 2 and Section 3.2 rely on the authors' previous papers [14,15,16] for the pattern definitions without independent validation or comparison to alternative taxonomies.
  • domain assumption Regex-based matching of configuration-file variables can reliably identify feature toggles without semantic analysis.
    Section 3.2 describes extraction 'by matches all variables with a set of predefined general regular expressions' plus filtering. This is a syntactic heuristic, and the evaluation admits confusion with global constants in Sentry and Server.
  • ad hoc to paper Manual evaluation using the same logic as the tool produces an independent assessment of detection quality.
    Section 4 says human developers find toggle usages 'based on the logic built', making the ground truth dependent on the tool's own definitions.
  • domain assumption The five selected projects for manual evaluation are representative of toggle usage across the six languages.
    Section 4 states projects were selected randomly 'regardless of any special consideration', with no justification of representativeness or coverage of language-specific toggle idioms.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TS-Detector : Detecting Feature Toggle Usage Patterns." pith.science (2026). https://pith.science/paper/ZX2SXV45

@misc{pith2026250505326,
  author       = {Pith},
  title        = {Pith review of: TS-Detector : Detecting Feature Toggle Usage Patterns},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZX2SXV45}},
  note         = {Machine review of arXiv:2505.05326}
}
read the original abstract

Feature toggles enable developers to control feature states, allowing the features to be released to a limited group of users while preserving overall software functionality. The absence of comprehensive best practices for feature toggle usage often results in improper implementation, causing code quality issues. Although certain feature toggle usage patterns are prone to toggle smells, there is no tool as of today for software engineers to detect toggle usage patterns from the source code. This paper presents a tool TS-Detector to detect five different toggle usage patterns across ten open-source software projects in six different programming languages. We conducted a manual evaluation and results show that the true positive rates of detecting Spread, Nested, and Dead toggles are 80%, 86.4%, and 66.6% respectively, and the true negative rate of Mixed and Enum usages was 100%. The tool can be downloaded from its GitHub repository and can be used following the instructions provided there.

Figures

Figures reproduced from arXiv: 2505.05326 by the authors.

Figure 1
Figure 1. Development method. 8 -t Toggle usage pattern . Optional . 9 -l Language of the project . Optional . Listing 1. Parameters Users can use the following command and parameters to use TS-Detector. 1 python3 tsd . py -p </ source / path > -c </ config / path > -o < output / path > -t pattern Listing 2. Command template to use ts-detector Output: The output of the tool is a JSON object consists of toggle usage pattern, a… view at source ↗
Figure 2
Figure 2. TS-Detector Architecture. Chromium has a few Mixed, and Enum usage patterns, none of the projects that we manually checked showed any pres￾ence of such usage patterns. Temporal, Dawn, and Sentry are relatively larger projects than the two others OpenSearch and Server, as a result, the toggle usages in these projects are higher in number. 0 25 50 75 100 OpenSearch Temporal Dawn Sentry Server Spread Nested Dead True P… view at source ↗
Figure 3
Figure 3. True-Positives of detecting toggle smells using TS-Detector [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 19 canonical work pages

  1. [1]

    Feature Flags

    2022. Feature Flags. https://apptimize.com/docs/featureflags.html

    work page 2022

  2. [2]

    Matt Campbell. 2020. Uber Open-Sources Tool to Automatically Clean Up Stale Code. https://www.infoq.com/news/2020/06/uber-piranha/. (January 2020)

    work page 2020

  3. [3]

    Wolfram Fenske, Sandro Schulze, Daniel Meyer, and Gunter Saake

    work page

  4. [4]

    Flickr. 2009. Flipping Out. https://code.flickr.net/2009/12/02/flipping- out. (December 2009)

    work page 2009

  5. [5]

    Martin Fowler. 2017. Feature Toggles (aka Feature Flags). https:// martinfowler.com/articles/feature-toggles.html. (2017)

    work page 2017

  6. [6]

    Martin Fowler. 2018. Refactoring: Improving the Design of Existing Code (2 ed.). Addison-Wesley Professional

    2018

  7. [7]

    Gene Kim, Jez Humble, Patrick Debois, John Willis, and Nicole Fors- gren. 2021. The DevOps handbook: How to create world-class agility, reliability, & security in technology organizations . It Revolution

    work page 2021

  8. [8]

    Duc Le, Eric Walkingshaw, and Martin Erwig. 2011. # ifdef con- firmed harmful: Promoting understandable software variation. In 2011 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 143–150

    work page 2011

Show all 21 references
  1. [9]

    Jörg Liebig, Christian Kästner, and Sven Apel. 2011. Analyzing the discipline of preprocessor annotations in 30 million lines of C code. In Proceedings of the tenth international conference on Aspect-oriented software development. 191–202

    2011

  2. [10]

    Rezvan Mahdavi-Hezaveh, Jacob Dremann, and Laurie Williams. 2021. Software development with feature toggles: practices used by practi- tioners. Empirical Software Engineering 26, 1 (2021), 1–33

    2021

  3. [11]

    Flávio Medeiros, Márcio Ribeiro, Rohit Gheyi, Sven Apel, Christian Kästner, Bruno Ferreira, Luiz Carvalho, and Baldoino Fonseca. 2017. Discipline matters: Refactoring of preprocessor directives in the# ifdef hell. IEEE Transactions on Software Engineering 44, 5 (2017), 453–469

    2017

  4. [12]

    Jens Meinicke, Chu-Pan Wong, Bogdan Vasilescu, and Christian Käst- ner. 2020. Exploring differences and commonalities between feature flags and configuration options. In Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering: Software Engineering in ...

    2020

  5. [13]

    Netflix. 2018. Preparing the Netflix API for Deployment. https://netflixtechblog.com/preparing-the-netflix-api-for- deployment-786d8f58090d. (November 2018)

    2018

  6. [14]

    Md Tajmilur Rahman, Louis-Philippe Querel, Peter C Rigby, and Bram Adams. 2016. Feature toggles: practitioner practices and a case study. In Proceedings of the 13th international conference on mining software repositories. 201–211

    2016

  7. [15]

    Tajmilur Rahman. 2023. Feature Toggle Usage Patterns: A Case Study on Google Chromium. In 2023 IEEE/ACM 20th International Conference on Mining Software Repositories (MSR) . IEEE, 142–147

    2023

  8. [16]

    Tajmilur Rahman, Imran Shalabi, and Tushar Sharma. 2024. Exploring Influence of Feature Toggles on Code Complexity. InProceedings of the 28th International Conference on Evaluation and Assessment in Software Engineering. 363–368

    2024

  9. [17]

    Doug Seven. 2014. Knightmare: A DevOps Cautionary Tale. https://dougseven.com/2014/04/17/knightmare-a-devops- cautionary-tale/. (April 2014)

    2014

  10. [18]

    Tushar Sharma and Diomidis Spinellis. 2018. A survey on software smells. Journal of Systems and Software 138 (2018), 158 – 173. https: //doi.org/10.1016/j.jss.2017.12.034

    2018 doi

  11. [19]

    sillysaurus3. 2017. Feature flags are a nice idea, but nobody outside of Facebook seems to be embracing them. https://news.ycombinator. com/item?id=15377207. (October 2017)

    2017

  12. [20]

    Thomas Steiner. 2021. How to set browser flags in Chromium. https: //developer.chrome.com/blog/browser-flags/. (May 2021)

    2021

  13. [2015]

    In 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM)

    When code smells twice as much: Metric-based detection of variability-aware code smells. In 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM) . IEEE, 171–180

    2015

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.