REVIEW 5 major objections 6 minor 40 references
SIREN: Software Identification and Recognition in HPC Systems
T0 review · 5 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read This paper claims that SSDeep fuzzy hashing of executables lets HPC systems identify and recognize software even when builds differ by code changes, versions, or compilers, and shows it working in an opt-in deployment on LUMI.
desk verdict SIREN is a real, deployable framework with an honest first deployment on LUMI; the architecture is sound, but the headline identification claim rests on one anecdotal match, not a measured result. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is SSDeep's Context-Triggered Piecewise Hashing (CTPH): a file is split into content-defined chunks, each chunk is hashed, and the hashes are concatenated; two fuzzy hashes are compared via Damerau–Levenshtein distance into a 0–100 similarity score. SIREN computes three such hashes per executable—raw binary, printable strings, global ELF symbol table—plus hashes of modules, compilers, and shared libraries. The multi-channel comparison lets an unknown executable be matched to a software family, not just to one exact file. Collection rides on an LD_PRELOAD-injected library active at process start and exit, sending UDP messages to a central database.
What would settle it
Rebuild one software package across a range of compilers, versions, and optimization flags, and also build several unrelated packages, then measure the six-channel SSDeep similarity for every pair. The central claim fails if two executables from different packages score at or above the similarity that currently matched UNKNOWN to icon (67–100% average), or if two builds of the same package score below it. The paper reports no false-positive or false-negative rates, and its only validation is the single icon match.
Extended reading notes
Core claim
On its own terms, the paper establishes a mechanism for software identification in HPC: instead of matching exact bytes or trusting user names, SIREN fingerprints each executable with three SSDeep fuzzy hashes—raw binary, printable strings, global ELF symbol names—plus hashes of modules, compilers, and shared libraries. Because fuzzy hashes preserve similarity, executables from the same family cluster together despite differing versions, code changes, or compilers. The demonstration: an executable with a nondescript name, labeled UNKNOWN, matched the icon family, with one instance at 100% similarity on all six channels and others at 67–95% average. The paper reads this as evidence that fuzzy
Load-bearing premise
The approach assumes SSDeep fuzzy hashes of a binary, its strings, and its symbol table stay similar across builds of the same software and stay different across builds of different software; the paper validates this on a single unknown executable, and its known labels come from regex-matching file paths—the very thing SIREN is meant to improve upon.
Editorial extensions
If this is right
- HPC operators can recognize repeated executions of the same application even when it is rebuilt with different compilers, versions, or flags, enabling longitudinal studies of performance variability and software usage.
- Unknown executables with arbitrary names (e.g., a.out) can be identified against a corpus of known software purely from similarity, with no need to store or inspect binary contents.
- The bundled metadata—compiler identification strings, loaded shared libraries, imported Python packages—gives support teams diagnostic signal for troubleshooting and for spotting unexpected behavior such as misuse or cryptomining.
- Application identity preserved across variants supports downstream uses the paper lists: CPU frequency tuning, node sharing among compatible applications, and energy-consumption prediction.
Reading between the lines
- The similarity graph among executable fuzzy hashes could itself generate software labels by clustering, which would remove the paper's reliance on regex-matching file paths—the same unreliable signal SIREN is meant to supersede. The UNKNOWN-to-icon match suggests such clustering is feasible but untested at scale.
- The three hash channels likely have different stability profiles: symbol-table hashes should survive compiler changes best, raw-binary hashes worst. A systematic sweep over compilers, versions, and optimization flags could map where each channel breaks, turning the single-case validation into a measured curve.
- Because only hashes and metadata leave the node, the approach suggests privacy-preserving federated software-usage statistics across multiple HPC sites: sites share similarity scores, not binaries.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. SIREN is an LD_PRELOAD-based data collection framework for HPC systems, deployed in an opt-in campaign on LUMI. It collects process metadata, environment variables, loaded modules, shared libraries, compiler identification strings, Python package imports, and SSDeep fuzzy hashes of the raw executable, printable strings, and ELF symbol table. The paper presents descriptive statistics from 12 users, 13,448 jobs, and 2,317,859 processes, and argues that fuzzy hashing enables both recognition of repeated executions of known software and identification of unknown executables. The central identification claim is made in the abstract and Section 1, and is supported in Section 4.3 by a single case where an executable with a nondescript name was matched to the icon software family (Table 7).
Significance. If the central claim were fully validated, SIREN would be a meaningful step beyond job-name and cryptographic-hash-based tracking: similarity-preserving executable fingerprints could enable identification across versions, compilers, and rebuilds while preserving user privacy. The paper contributes a working open-source artifact, a real opt-in deployment, and a rich set of descriptive analyses (library, compiler, and Python package usage) that are valuable to the HPC operations community. However, the key novel claim—similarity-based identification of unknown software—rests on a single anecdotal match with no error analysis, no negative controls, and no comparison against simpler baselines. The current evidence is not sufficient to establish that SSDeep-based fingerprints are discriminative across software families, and the paper should be revised to supply that evidence.
major comments (5)
- [§4.3, Table 7] The central identification claim is supported by only one positive match (UNKNOWN→icon). The table reports similarity scores only against icon instances; there are no similarity scores against the other derived labels (LAMMPS, GROMACS, miniconda, amber, etc.), no negative controls, no decision threshold, and no precision/recall or false-positive/false-negative analysis. The average similarity even mixes executable-content hashes (FI_H, ST_H, SY_H) with environment/context hashes (MO_H, CO_H, OB_H), which can reflect shared modules or default libraries rather than software identity. A rigorous evaluation needs a full similarity matrix over all labeled executables, a threshold analysis, and per-hash-type accuracy metrics.
- [§4.3, Table 7] The claim in the abstract and Section 1 that fuzzy hashing of executables detects similarity despite code changes, versions, and compilers is not supported by the raw-binary hash. In Table 7, FI_H (raw file hash) is 0 for four of the ten icon rows, while ST_H and SY_H carry most of the signal. Thus the current data contradict the broadest version of the claim, at least for icon. The paper should report which hash types actually drive identification, justify the combination rule, and either narrow the claim (e.g., to strings and symbol tables) or add evidence that raw-binary fuzzy hashing is effective in other families.
- [§4.3, Table 5] The 'known' labels are derived from executable file and path names via regular expression matching. This is a noisy ground truth: the same path patterns can contain renamed or copied executables, and the paper itself notes the unreliability of arbitrary user-provided names. The evaluation needs at least an audit of label quality (e.g., manual inspection or independent confirmation via compiler/shared-library evidence) and a discussion of how label noise affects the reported identification accuracy. Without this, the identification result remains anecdotal even if the matching procedure is internally consistent.
- [§2 and §3.1] SIREN is described as lightweight and as imposing minimal overhead, but no overhead measurements are reported. Since LD_PRELOAD-based instrumentation is a central design choice and can perturb every process, the paper should provide at least startup-latency and runtime-overhead measurements for representative jobs, along with the observed UDP message-loss rate (currently only stated as 0.02% of jobs having missing fields). This is not the main identification claim, but it is load-bearing for the framework's practical utility in production HPC systems.
- [§3.1, Requirements and Limitations] The paper correctly states that statically linked executables and processes inside containers are not collected. Given the prevalence of containers in modern HPC, this scope limitation should be reflected in the abstract and conclusions, not only in the limitations paragraph. The current framing ('improves observability in HPC') overstates coverage without this qualification.
minor comments (6)
- [§4.3] Typo: 'UNKOWN' should be 'UNKNOWN' in the text introducing Table 7.
- [Table 7] The column header 'A vg. Sim.' has an extra space and should be 'Avg. Sim.'.
- [§2.1] The sentence describing the Damerau–Levenshtein distance switches between plural and singular subjects; please rephrase for clarity.
- [§4.3 and Figure 2] The phrase 'derived and filtered libraries' is not defined precisely; state which substring combinations were derived, why they were chosen, and how the filtering affects the reported counts.
- [Abstract] The phrase 'preserving privacy and file integrity' is vague; fuzzy hashes are not cryptographic commitments, and the privacy properties should be stated more carefully (e.g., that raw file contents are not stored).
- [§4.4] The Python package analysis is interesting but the counts of 'unique users' in Figure 3 appear to be at most 3; this should be stated explicitly so the reader can judge the strength of the package-level conclusions.
Circularity Check
No significant circularity; SIREN's fuzzy-hash identification is an empirical matching result, not a derivation that reduces to its own inputs.
full rationale
The paper makes no formal derivation in which an output quantity is defined in terms of the quantity it claims to predict. The central identification claim is empirical: SSDeep fuzzy hashes of raw binaries, printable strings, and symbol tables are computed for collected processes, and the UNKNOWN executable is matched by similarity to labeled icon instances (Table 7). The known labels are indeed derived from file/path name regexes (Section 4.3, 'Derived Labels for User Applications'), which the paper itself acknowledges ('Our labels (for known software) were derived so far from file and path names'), but this does not make the fuzzy-hash match circular: the UNKNOWN instance is not labeled by its path; it is matched by six independent hash similarity scores, and the icon identification is additionally corroborated by shared-library evidence (climatedt, hdf5-cray, netcdf-cray, etc.). No parameter is fitted to a subset and then renamed a prediction; SSDeep scores are computed directly. The only self-citation, [22] (Jakobsche and Ciorba), is used as background stating that fuzzy hashing has 'recently been adopted in HPC' / 'used to classify executable files in HPC systems'; it is not invoked as a uniqueness theorem, does not supply the matching rule, and is not load-bearing for SIREN's architecture or results. Concerns about missing false-positive/negative rates and the reliability of regex-derived labels are validation weaknesses, not circularity. Accordingly, the paper is self-contained against external benchmarks for the claims it actually makes, and the circularity score is 0.
Assumptions & free parameters
free parameters (3)
- System directory list for process categorization
- Library substring filter list
- Regex patterns for software labels
assumptions (4)
- domain assumption LD_PRELOAD interception of dynamically linked processes
- domain assumption SSDeep fuzzy hash reliability
- domain assumption Path/regex-based labels as ground truth
- ad hoc to paper The 12 opt-in users provide sufficient evidence
Cite this review
Pith. "Pith review of SIREN: Software Identification and Recognition in HPC Systems." pith.science (2026). https://pith.science/paper/3HCUQ7CK
@misc{pith2026250818950,
author = {Pith},
title = {Pith review of: SIREN: Software Identification and Recognition in HPC Systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/3HCUQ7CK}},
note = {Machine review of arXiv:2508.18950}
}
read the original abstract
HPC systems use monitoring and operational data analytics to ensure efficiency, performance, and orderly operations. Application-specific insights are crucial for analyzing the increasing complexity and diversity of HPC workloads, particularly through the identification of unknown software and recognition of repeated executions, which facilitate system optimization and security improvements. However, traditional identification methods using job or file names are unreliable for arbitrary user-provided names (a.out). Fuzzy hashing of executables detects similarities despite changes in executable version or compilation approach while preserving privacy and file integrity, overcoming these limitations. We introduce SIREN, a process-level data collection framework for software identification and recognition. SIREN improves observability in HPC by enabling analysis of process metadata, environment information, and executable fuzzy hashes. Findings from a first opt-in deployment campaign on LUMI show SIREN's ability to provide insights into software usage, recognition of repeated executions of known applications, and similarity-based identification of unknown applications.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
https://github.com/Cyan4973/xxHash (2014), gitHub repository
Extremely fast non-cryptographic hash algorithm. https://github.com/Cyan4973/xxHash (2014), gitHub repository
work page 2014
-
[2]
An action plan for high performance computing security (Nov 2016), https://csrc. nist.gov/CSRC/media/Projects/High-Performance-Computing-Security/documents/ nist-high-performance-computing-security_draft.pdf, november 2016
work page 2016
-
[3]
https://github.com/spiffe/spire (2017), gitHub repository
Spire (the spiffe runtime environment). https://github.com/spiffe/spire (2017), gitHub repository
work page 2017
-
[4]
https://destine.ecmwf.int/ climate-change-adaptation-digital-twin-climate-dt/ (nd)
Climate change adaptation digital twin (climate dt). https://destine.ecmwf.int/ climate-change-adaptation-digital-twin-climate-dt/ (nd)
-
[5]
Agelastos, A., Allan, B., Brandt, J., Cassella, P., Enos, J., Fullop, J., Gentile, A., Monk, S., Naksinehaboon, N., Ogden, J., et al.: The lightweight distributed metric service: a scalable in- frastructure for continuous monitoring of large scale computing systems and applications. In: SC’14: Proceedings of the International Conference for High Performan...
work page 2014
-
[6]
In: 2014 First International Workshop on HPC User Support Tools
Agrawal, K., Fahey, M.R., McLay, R., James, D.: User environment tracking and problem detec- tion with xalt. In: 2014 First International Workshop on HPC User Support Tools. pp. 32–40. IEEE (2014)
work page 2014
-
[7]
Ates, E., Tuncer, O., Turk, A., Leung, V.J., Brandt, J., Egele, M., Coskun, A.K.: Taxonomist: Application detection through rich monitoring data. In: Euro-Par 2018: Parallel Processing: 24th International Conference on Parallel and Distributed Computing, Turin, Italy, August 27-31, 2018, Proceedings 24. pp. 92–105. Springer (2018)
work page 2018
-
[8]
BBC News Technology: Us national agency computers misused to mine bitcoins (Jun 2014), https://www.bbc.com/news/technology-27779030, 10 June 2014
work page 2014
Show all 40 references
-
[9]
BBC News Technology: Europe’s supercomputers hijacked by attackers for crypto mining (May 2020), https://www.bbc.com/news/technology-52709660, 18 May 2020
2020
-
[10]
In: Dagstuhl Reports
Benkner, S., Franchetti, F., Gerndt, H.M., Hollingsworth, J.K.: Automatic application tuning for hpc architectures (dagstuhl seminar 13401). In: Dagstuhl Reports. vol. 3. Schloss Dagstuhl- Leibniz-Zentrum fuer Informatik (2014)
2014
-
[11]
Brandt, J., Ciorba, F., Gentile, A., Ott, M., Wilde, T.: Driving HPC Operations With Holistic Monitoring and Operational Data Analytics (Dagstuhl Seminar 23171) (2023)
2023
-
[12]
In: 2015 44th International Conference on Parallel Processing Workshops
Breitbart, J., Weidendorfer, J., Trinitis, C.: Case study on co-scheduling for hpc applications. In: 2015 44th International Conference on Parallel Processing Workshops. pp. 277–285. IEEE (2015) 17
2015
-
[13]
https://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_ chain/ (2025), the Register
Claburn, T.: Llms can’t stop making up software dependencies and sabotaging every- thing. https://www.theregister.com/2025/04/12/ai_code_suggestions_sabotage_supply_ chain/ (2025), the Register. Published Sat 12 Apr 2025
2025
-
[14]
In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
Costa, E., Patel, T., Schwaller, B., Brandt, J.M., Tiwari, D.: Systematically inferring i/o per- formance variability by examining repetitive job behavior. In: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1–15 (2021)
2021
-
[15]
IEEE computational science and engineering5(1), 46–55 (1998)
Dagum, L., Menon, R.: Openmp: an industry standard api for shared-memory programming. IEEE computational science and engineering5(1), 46–55 (1998)
1998
-
[16]
Communi- cations of the ACM7(3), 171–176 (1964)
Damerau, F.J.: A technique for computer detection and correction of spelling errors. Communi- cations of the ACM7(3), 171–176 (1964)
1964
-
[17]
Delwiche, T.R.: Harvard research computing resources misused for ‘dogecoin’ mining operation (Feb 2014),https://www.thecrimson.com/article/2014/2/20/harvard-odyssey-dogecoin/, 20 February 2014
2014
-
[18]
Pro- ceedings of the Cray User Group (2010)
Fahey, M., Jones, N., Hadri, B., Hitchcock, B.: The automatic library tracking database. Pro- ceedings of the Cray User Group (2010)
2010
-
[19]
Scientific american228(5), 15–23 (1973)
Feistel, H.: Cryptography and computer privacy. Scientific american228(5), 15–23 (1973)
1973
-
[20]
Cambridge University Press (2015)
Feitelson, D.G.: Workload modeling for computer systems performance evaluation. Cambridge University Press (2015)
2015
-
[21]
IEEE Access8, 136307–136315 (2020)
Haridas, P., Chennupati, G., Santhi, N., Romero, P., Eidenbenz, S.: Code characterization with graph convolutions and capsule networks. IEEE Access8, 136307–136315 (2020)
2020
-
[22]
In: SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
Jakobsche, T., Ciorba, F.M.: Using malware detection techniques for hpc application classifica- tion. In: SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. pp. 1773–1780. IEEE (2024)
2024
-
[23]
Digital investigation3, 91–97 (2006)
Kornblum, J.: Identifying almost identical files using context triggered piecewise hashing. Digital investigation3, 91–97 (2006)
2006
-
[24]
Lazo, E.G., Team, M..D.R.: Combing through the fuzz: Using fuzzy hashing and deep learn- ing to counter malware detection evasion techniques (Jul 2021),https://www.microsoft.com/ security/blog/2021/07/27/, 27 July 2021
2021
-
[25]
In: Soviet physics doklady
Levenshtein, V.I., et al.: Binary codes capable of correcting deletions, insertions, and reversals. In: Soviet physics doklady. vol. 10, pp. 707–710. Soviet Union (1966)
1966
-
[26]
McLay, R., Fahey, M.: Understanding the software needs of high performance computer users with xalt (2015)
2015
-
[27]
In: 2021 IEEE International Conference on Cluster Computing (CLUSTER)
Netti, A., Shin, W., Ott, M., Wilde, T., Bates, N.: A conceptual framework for HPC operational data analytics. In: 2021 IEEE International Conference on Cluster Computing (CLUSTER). pp. 596–603. IEEE (2021)
2021
-
[28]
Communications of the ACM (CACM) 60(9), 72–80 (September 2017)
Peisert, S.: Security in High-Performance Computing Environments. Communications of the ACM (CACM) 60(9), 72–80 (September 2017)
2017
-
[29]
https://github.com/pyupio/ safety-db (2021), gitHub repository
PyUp.io: A curated database of insecure python packages. https://github.com/pyupio/ safety-db (2021), gitHub repository
2021
-
[30]
IEEE spectrum34(3), 32–38 (1997) 18
Roehle, B.: Channeling the data flood. IEEE spectrum34(3), 32–38 (1997) 18
1997
-
[31]
In: 2024 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops)
Shoga, K., Brandt, J., Schwaller, B., Tucker, T.: Evolving large scale hpc monitoring & analy- sis to track modern dynamic environments. In: 2024 IEEE International Conference on Cluster Computing Workshops (CLUSTER Workshops). pp. 36–43. IEEE (2024)
2024
-
[32]
In: IEEE International
Skinner, D., Kramer, W.: Understanding the causes of performance variability in hpc workloads. In: IEEE International. 2005 Proceedings of the IEEE Workload Characterization Symposium,
2005
-
[33]
Trader, T.: Russian nuclear engineers caught cryptomining on lab supercomputer (Feb 2018), https://www.hpcwire.com/2018/02/12/ russian-nuclear-engineers-caught-cryptomining-lab-supercomputer/ , 12 February 2018
2018
-
[34]
Turner, A.: Software usage data on archer2 (February 2022), https://www.archer2.ac.uk/ news/2022/02/07/software-usage-data.html, ePCC
2022
-
[35]
Supercomputer12, 56–68 (1996)
Walker, D.W., Dongarra, J.J.: Mpi: a standard message passing interface. Supercomputer12, 56–68 (1996)
1996
-
[36]
In: High Performance Computing: 33rd International Conference, ISC High Performance 2018, Frankfurt, Germany, June 24-28, 2018, Proceedings 33
Yamamoto, K., Tsujita, Y., Uno, A.: Classifying jobs and predicting applications in hpc systems. In: High Performance Computing: 33rd International Conference, ISC High Performance 2018, Frankfurt, Germany, June 24-28, 2018, Proceedings 33. pp. 81–99. Springer (2018)
2018
-
[37]
Concurrency and Computation: Practice and Experience36(19), e8141 (2024)
Yokelson, D., Lappi, O., Ramesh, S., Väisälä, M.S., Huck, K., Puro, T., Norris, B., Korpi-Lagg, M., Heljanko, K., Malony, A.D.: SOMA: Observability, Monitoring, and In Situ Analytics for Exascale Applications. Concurrency and Computation: Practice and Experience36(19), e8141 (2024)
2024
-
[38]
In: Workshop on job scheduling strategies for parallel processing
Yoo, A.B., Jette, M.A., Grondona, M.: Slurm: Simple linux utility for resource management. In: Workshop on job scheduling strategies for parallel processing. pp. 44–60. Springer (2003)
2003
-
[39]
In: 2021 IEEE High Performance Extreme Computing Conference (HPEC)
Zhang, Y., Aksar, B., Aaziz, O., Schwaller, B., Brandt, J., Leung, V., Egele, M., Coskun, A.K.: Using monitoring data to improve hpc performance via network-data-driven allocation. In: 2021 IEEE High Performance Extreme Computing Conference (HPEC). pp. 1–7. IEEE (2021) 19
2021
-
[2005]
pp. 137–149. IEEE (2005)
2005
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.