REVIEW 3 major objections 4 minor 27 references
PSASpotter: A Tool to Detect the Usage of Platform-Specific APIs in Python
T0 review · 3 major / 4 minor · reviewed 2026-08-01 · deepseek-v4-flash
Pith's one-line read A tool named PSASpotter automatically detects calls to platform-specific APIs in Python and classifies whether each call is protected by defensive code.
desk verdict Useful tool and dataset, but the evaluation section contradicts itself: the reported precision/recall are not for defensive-code detection, so the headline result is unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is an AST-based static analyzer paired with a curated detection universe: a list of 1,841 platform-specific APIs extracted from the Python Standard Library, each tied to an availability platform. Around that list, PSASpotter recognizes defensive-code contexts by four syntactic patterns: calls inside try/except; calls inside if blocks that inspect platform identifiers such as sys.platform or os.name; calls inside if blocks that check for existence via hasattr; and test calls guarded by one of five skip decorators. This pattern set is what lets the tool move from 'this API is used' to 'this use is protected,' and the evaluation measures exactly that classification.
What would settle it
Construct a corpus of Python snippets in which platform-specific calls are protected by project-specific helpers rather than raw try/except or if blocks (examples the paper's divergence analysis identifies: internal decorators such as @t.skip.if_win32 and internal APIs like salt.utils.platform.is_windows), plus calls to third-party platform-specific APIs not on the standard-library list. If PSASpotter reports these as unprotected or misses them entirely, the general claim that it detects platform-specific API usage and defensive code is overstated; the paper already reports 47 internal-API mis
Extended reading notes
Core claim
The paper's central claim is that platform-specific API usage in Python can be detected automatically and accurately enough to be useful, and that PSASpotter accomplishes this as the first such tool. The tool's output is per-call: API name, availability platform, file and line, and a boolean flag for whether the call sits inside defensive code. The performance evaluation is specifically about that flag, not about detecting every platform-specific API ever written: with a fixed list of 1,841 APIs from the Python Standard Library, it flags defensive usage with precision 97.49%, recall 83.96%, and accuracy 87.44% on 1,544 manually labeled occurrences from 100 projects. The paper also claims tha
Load-bearing premise
PSASpotter's results depend entirely on a predefined list of 1,841 platform-specific APIs harvested from the Python Standard Library; the paper does not measure whether that list is complete, and it concedes other sources of platform-specific APIs may exist, so if the list misses or misclassifies an API, every usage count and defensive-code verdict built on it inherits that error.
Editorial extensions
If this is right
- Maintainers can scan a repository and get a list of platform-specific calls with a risk flag, making cross-platform failure points visible without manual review.
- The published dataset of 709,191 usages across 9,205 projects gives researchers a baseline for studying how widespread unprotected platform-specific API use is and how it changes over time.
- Library and framework developers can use the tool to check their release code for dependence on OS-specific APIs, reducing the chance that errors propagate to downstream users.
- Because the tool accepts a custom API list in JSON, the same machinery can be pointed at third-party or in-house platform-specific APIs once such lists exist.
- The defensive-code output doubles as a searchable inventory of alternative implementations for unavailable APIs, supporting cross-platform development and testing.
Reading between the lines
- I would expect the true scale of unprotected platform-specific usage to be larger than the paper reports: PSASpotter only knows the 1,841 standard-library APIs on its predefined list, and the paper itself concedes other sources may exist.
- The error analysis hints that most missed defensive code involves in-house OS checks and guard clauses; a follow-up tool that resolves control flow across function boundaries, or recognizes project-specific platform helpers, could close much of the recall gap.
- A natural extension is to turn PSASpotter into a CI linter that fails merges when a production file adds an unprotected platform-specific call, making the paper's core mechanism immediately actionable.
- The 709K-usage dataset, replayed over repository history, could expose which platform-specific APIs are most often called without protection and which projects regress most often.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. PSASpotter is an AST-based tool that detects calls to a predefined list of 1,841 Python Standard Library platform-specific APIs and classifies whether each call occurs in defensive code (try/except blocks, platform-checking if blocks, hasattr existence checks, or test skip decorators). The paper describes the tool's design and usage, reports a large mined dataset of 709,191 platform-specific API usages across 9,205 repositories, proposes three practical applications, and evaluates the defensive-code classification on 1,544 manually labeled occurrences from 100 projects, claiming precision 97.49%, recall 83.96%, and accuracy 87.44%.
Significance. If the quantitative claims were correct, PSASpotter would be the first automated detector of platform-specific API usage in Python and would provide a valuable large-scale dataset. The paper is commendable for shipping the tool, the dataset, and the manual evaluation artifacts with public DOIs, which aids reproducibility. However, the headline evaluation metrics are internally inconsistent with the reported confusion matrix and manual label counts. Correcting the matrix substantially changes the measured recall. The tool and dataset are still potentially useful, but the central performance claim needs to be fixed and re-framed.
major comments (3)
- [IV-C, Table III] The reported confusion matrix is incompatible with the manual counts. Section IV-B states 626 defensive and 918 non-defensive occurrences. Section IV-C reports TP=895, FN=171, FP=23, TN=455. If defensive code is the positive class, TP+FN=1,066, not 626; if non-defensive code is the positive class, TP+FN=1,066, not 918. The only matrix consistent with the stated FN=171 (missed defensive) and FP=23 (false defensive) is TP=455, FN=171, FP=23, TN=895. That matrix yields precision=455/478=95.2%, recall=455/626=72.7%, accuracy=87.4%. The reported values 97.49% precision and 83.96% recall correspond to a different class definition (or swapped precision/recall for the non-defensive class). Therefore the headline metrics in the Abstract, Section I, and Section IV-C are unsupported and must be corrected.
- [IV-A, IV-B; II-A] The evaluation validates only the defensive-code classification of occurrences the tool already detected; it does not validate the API-usage detection stage itself. The 1,544 occurrences in Section IV-B were collected by the tool, so any platform-specific API usage missed by the predefined list of 1,841 APIs or by the AST-based matcher is absent from the ground truth. Since Section V concedes that 'other sources of platform-specific APIs may exist' and the list is taken from the authors' own prior study, the reported dataset counts in Table I and the general claim of detecting platform-specific API usage lack independent validation. The paper should either provide a manual check of API-usage detection or clearly scope the contribution as 'detecting usages of a predefined list of platform-specific APIs.'
- [IV-B] The manual labeling procedure is underspecified. The paper states that occurrences were 'manually examined and classified' but does not report the number of annotators, the labeling protocol, whether disagreements were resolved, or any inter-rater agreement measure. Given that the 626/918 counts serve as ground truth for all reported metrics, the reliability of these labels is a load-bearing concern. A revision should describe the labeling procedure in enough detail to be reproduced.
minor comments (4)
- [IV-B] Typo: 'occcurred' should be 'occurred'.
- [Abstract and I] The phrase 'precision, recall, and accuracy of over 97%, 83%, and 87%' must be updated after correcting the confusion matrix; as reported, these numbers do not match the stated TP/FN/FP values.
- [II-C] The four defensive-code cases are clear, but the examples in Figures 3a–3c are cited as URLs; adding the relevant code snippets inside the figure would improve readability.
- [IV-D] The sentence 'another 26 cases are related to manual flagging' could be expanded: it is unclear whether these are cases where the manual label was likely wrong or where the tool's output was ambiguous. Clarify.
Circularity Check
No circularity found; PSASpotter's defensive-code claim is benchmarked against independent manual labels, and self-citations are provenance rather than load-bearing reductions.
full rationale
The paper's central claim is that PSASpotter detects platform-specific API usage and classifies whether that usage occurs within defensive code. The defensive-code classification is evaluated against an externally and manually constructed ground truth: 'we manually examined and classified based on the usage context of platform-specific API. This process resulted in 626 instances identified as occurring in defensive code, while 918 occurred without defensive code' (Section IV-B). The reported confusion matrix is the tool's output versus those manual labels, so the precision/recall/accuracy numbers are not derived from the tool's own detection rules by construction. The API list in Section II-A is inherited from the authors' prior documentation mining [10]; this is a self-citation for provenance, but it is not circular: the manual labels do not depend on that list, Section V concedes 'other sources of platform-specific APIs may exist,' and the prior study is an externally checkable empirical artifact. The large-scale dataset in Section III-A is an application of the tool, not a prediction, and no fitted parameter is renamed as a prediction. No uniqueness theorem, ansatz-smuggling citation, or definitional equivalence is used to force the result. Separately, the evaluation contains an apparent internal inconsistency (TP=895 exceeds the 626 manual defensive labels, suggesting the metrics may be computed for the non-defensive class), but this is a correctness/methodology concern, not a circularity of the derivation chain.
Assumptions & free parameters
assumptions (4)
- domain assumption The list of 1,841 platform-specific APIs from the authors' prior study [10] correctly and completely enumerates platform-specific Python stdlib APIs.
- domain assumption AST-based static analysis is sufficient to capture the API usages of interest.
- domain assumption The four defensive-code categories (try/except, platform-check if, hasattr if, skip decorators) cover meaningful defensive patterns.
- domain assumption Manual classification by the authors is an acceptable ground truth.
Cite this review
Pith. "Pith review of PSASpotter: A Tool to Detect the Usage of Platform-Specific APIs in Python." pith.science (2026). https://pith.science/paper/FLC3ZNRA
@misc{pith2026260717462,
author = {Pith},
title = {Pith review of: PSASpotter: A Tool to Detect the Usage of Platform-Specific APIs in Python},
year = {2026},
howpublished = {\url{https://pith.science/paper/FLC3ZNRA}},
note = {Machine review of arXiv:2607.17462}
}
read the original abstract
A platform-specific API is implemented for a particular platform (e.g., operating system), thus, it may not work on other platforms than the target one. Detecting the usage of such APIs is important for supporting software maintenance, as it allows maintainers to be alerted about APIs that could pose potential risks. This paper proposes PSASpotter, a tool to detect the usage of platform-specific APIs in Python systems. PSASpotter also identifies whether the platform-specific APIs are used within a defensive code, such as try/except blocks or if blocks that check the current platform. PSASpotter can support the development of novel empirical studies about the usage of platform-specific APIs in the Python ecosystem. Moreover, the defensive code detected by PSASpotter may contain alternative solutions for unavailable APIs, which can provide insights for software development and testing across multiple platforms. PSASpotter is available at: https://github.com/ricardojob/PSASpotter. Tool video: https://youtu.be/d3WyozTAKS8.
Figures
Reference graph
Works this paper leans on
-
[1]
How and why developers migrate python tests
L ´ıvia Barbosa and Andre Hora. How and why developers migrate python tests. InInternational Conference on Software Analysis, Evolution and Reengineering, pages 538–548. IEEE, 2022
2022
-
[2]
You broke my code: Understanding the motivations for breaking changes in APIs.Empirical Software Engineering, 25:1458–1492, 2020
Aline Brito, Marco Tulio Valente, Laerte Xavier, and Andre Hora. You broke my code: Understanding the motivations for breaking changes in APIs.Empirical Software Engineering, 25:1458–1492, 2020
2020
-
[3]
Apidiff: Detecting api breaking changes
Aline Brito, Laerte Xavier, Andre Hora, and Marco Tulio Valente. Apidiff: Detecting api breaking changes. InInternational Conference on Software Analysis, Evolution and Reengineering, pages 507–511. IEEE, 2018
2018
-
[4]
Semdiff: Analysis and recommendation support for api evolution
Barthelemy Dagenais and Martin P Robillard. Semdiff: Analysis and recommendation support for api evolution. InInternational Conference on Software Engineering, pages 599–602. IEEE, 2009
2009
-
[5]
Aexpy: Detecting API breaking changes in python packages
Xingliang Du and Jun Ma. Aexpy: Detecting API breaking changes in python packages. InInternational Symposium on Software Reliability Engineering, pages 470–481. IEEE, 2022
2022
-
[6]
Deprewriter: On the fly rewriting method depreca- tions.The Journal of Object Technology, 21(1):1–23, 2022
St ´ephane Ducasse, Guillermo Polito, Oleksandr Zaitsev, Marcus Denker, and Pablo Tesone. Deprewriter: On the fly rewriting method depreca- tions.The Journal of Object Technology, 21(1):1–23, 2022
2022
-
[7]
How do developers react to API evolution? a large-scale empirical study.Software Quality Journal, 26(1):161–191, 2018
Andre Hora, Romain Robbes, Marco Tulio Valente, Nicolas Anquetil, Anne Etien, and Stephane Ducasse. How do developers react to API evolution? a large-scale empirical study.Software Quality Journal, 26(1):161–191, 2018
2018
-
[8]
apiwave: Keeping track of api popularity and migration
Andr ´e Hora and Marco Tulio Valente. apiwave: Keeping track of api popularity and migration. InInternational Conference on Software Maintenance and Evolution, pages 321–323. IEEE, 2015
2015
Show all 27 references
-
[9]
Repfinder: Finding replacements for missing apis in library update
Kaifeng Huang, Bihuan Chen, Linghao Pan, Shuai Wu, and Xin Peng. Repfinder: Finding replacements for missing apis in library update. In International Conference on Automated Software Engineering, pages 266–278. IEEE, 2021
2021
-
[10]
Availability and usage of platform-specific apis: A first empirical study
Ricardo Job and Andre Hora. Availability and usage of platform-specific apis: A first empirical study. InInternational Conference on Mining Software Repositories, pages 27–31. IEEE, 2024
2024
-
[11]
How and why developers implement OS- specific tests.Empirical Software Engineering, 30(1):1–33, 2025
Ricardo Job and Andre Hora. How and why developers implement OS- specific tests.Empirical Software Engineering, 30(1):1–33, 2025
2025
-
[12]
On the implementation of os-specific tests: The cpython case
Ricardo Job and Andre Hora. On the implementation of os-specific tests: The cpython case. InBrazilian Symposium on Systematic and Automated Software Testing, pages 46–54, 2025
2025
-
[13]
Platform-specific apis, November, 2025
Ricardo Job and Andre Hora. Platform-specific apis, November, 2025
2025
-
[14]
Best principles in the design of shared software
Dino Konstantopoulos, John Marien, Mike Pinkerton, and Eric Braude. Best principles in the design of shared software. InInternational Computer Software and Applications Conference, pages 287–292, 2009
2009
-
[15]
Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration
Raula Gaikovina Kula, Daniel M German, Ali Ouni, Takashi Ishio, and Katsuro Inoue. Do developers update their library dependencies? an empirical study on the impact of security advisories on library migration. Empirical Software Engineering, 23:384–417, 2018
2018
-
[16]
API change and fault proneness: A threat to the success of android apps
Mario Linares-V ´asquez, Gabriele Bavota, Carlos Bernal-C´ardenas, Mas- similiano Di Penta, Rocco Oliveto, and Denys Poshyvanyk. API change and fault proneness: A threat to the success of android apps. InJoint Meeting on Foundations of Software Engineering, pages 477–487, 2013
2013
-
[17]
Android api field evolu- tion and its induced compatibility issues
Tarek Mahmud, Meiru Che, and Guowei Yang. Android api field evolu- tion and its induced compatibility issues. InInternational Symposium on Empirical Software Engineering and Measurement, pages 34–44, 2022
2022
-
[18]
How and Why did developers migrate Android Applications from Java to Kotlin? A study based on code analysis and interviews with developers.arXiv preprint arXiv:2003.12730, 2020
Matias Martinez and Bruno Gois Mateus. How and Why did developers migrate Android Applications from Java to Kotlin? A study based on code analysis and interviews with developers.arXiv preprint arXiv:2003.12730, 2020
2003 arXiv
-
[19]
How are framework code samples maintained and used by developers? the case of android and spring boot.Journal of Systems and Software, 1:1–30, 2021
Gabriel Menezes, Bruno Cafeo, and Andre Hora. How are framework code samples maintained and used by developers? the case of android and spring boot.Journal of Systems and Software, 1:1–30, 2021
2021
-
[20]
The effect of object-oriented frameworks on developer productivity.Computer, 29(9), 1996
Simon Moser and Oscar Nierstrasz. The effect of object-oriented frameworks on developer productivity.Computer, 29(9), 1996
1996
-
[21]
Exploring API Deprecation Evolution in JavaScript
Romulo Nascimento, Andre Hora, and Eduardo Figueiredo. Exploring API Deprecation Evolution in JavaScript. InInternational Conference on Software Analysis, Evolution and Reengineering, pages 169–173. IEEE, March 2022
2022
-
[22]
Measuring soft- ware library stability through historical version analysis
Steven Raemaekers, Arie van Deursen, and Joost Visser. Measuring soft- ware library stability through historical version analysis. InInternational Conference on Software Maintenance, pages 378–387, 2012
2012
-
[23]
Robillard, Eric Bodden, David Kawrykow, Mira Mezini, and Tristan Ratchford
Martin P. Robillard, Eric Bodden, David Kawrykow, Mira Mezini, and Tristan Ratchford. Automated API Property Inference Techniques.IEEE Transactions on Software Engineering, 39(5):613–637, May 2013
2013
-
[24]
On the reaction to deprecation of clients of 4 + 1 popular Java APIs and the JDK.Empirical Software Engineering, 23(4):2158–2197, 2018
Anand Ashok Sawant, Romain Robbes, and Alberto Bacchelli. On the reaction to deprecation of clients of 4 + 1 popular Java APIs and the JDK.Empirical Software Engineering, 23(4):2158–2197, 2018
2018
-
[25]
https://docs.python.org/3/library/index.html, November, 2025
The Python Standard Library. https://docs.python.org/3/library/index.html, November, 2025
2025
-
[26]
Historical and impact analysis of api breaking changes: A large-scale study
Laerte Xavier, Aline Brito, Andre Hora, and Marco Tulio Valente. Historical and impact analysis of api breaking changes: A large-scale study. InInternational Conference on Software Analysis, Evolution and Reengineering, pages 138–147. IEEE, 2017
2017
-
[27]
How Android developers handle evolution-induced API compatibility issues: a large-scale study
Hao Xia, Yuan Zhang, Yingtian Zhou, Xiaoting Chen, Yang Wang, Xiangyu Zhang, Shuaishuai Cui, Geng Hong, Xiaohan Zhang, Min Yang, et al. How Android developers handle evolution-induced API compatibility issues: a large-scale study. InInternational Conference on Software Enginee...
2020
Reviewed August 1, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.