{"id":"8b60f318-4dc5-4718-8e8a-ba57764beebe","arxiv_id":"2508.18950","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A process-level telemetry framework using SSDeep fuzzy hashes of executables, strings, and symbol tables to identify and recognize software on HPC systems, demonstrated in a 12-user deployment on LUMI.","lead":"SIREN is a new framework that hooks into running HPC processes with LD_PRELOAD to collect metadata and fuzzy hashes of executables, so that operators can identify what software is running even when file names are arbitrary. It was tested with 12 volunteer users on the LUMI supercomputer, and results show it can recognize repeated runs and match unknown executables to known applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The identification claim rests on a single positive match (UNKNOWN→icon) with no non-match baseline or accuracy measure; SSDeep similarity is not shown to be discriminative across software families.","rationale":"The paper is a coherent framework description with an honest first deployment, and the reader's CONDITIONAL verdict is appropriate. My stress-test focused on whether the central identification claim has sufficient support. It does not: the only identification evidence is one unknown matched to icon, with no comparison to non-matching software and no accuracy metric. This is the same load-bearing assumption the reader identified: SSDeep hashes must be stable within a family and discriminative across families. The concrete test—adding a non-match baseline and leave-one-out cross-validation—would directly measure that. If the check passes, the concern is resolved and the conditional verdict can be upgraded; if it fails, the identification claim should be substantially weakened. I do not see an internal inconsistency or a flaw requiring rejection; the framework and deployment are plausible, but the key capability is under-validated. Hence UNCHANGED.","tokens_in":15991,"tokens_out":4028,"duration_ms":38582,"concrete_test":"Re-run the §4.3 similarity search for the UNKNOWN executable against all labeled executables in the post-processed database, not just icon, and compute the average SSDeep similarity (or nearest-neighbor rank) for each label. Then perform leave-one-out cross-validation over all labeled executables: for each executable, hide its regex-derived label, run the same matching procedure against the rest, and compute top-1 label accuracy and confusion matrix. If a non-icon label also achieves high similarity (e.g., ≥80 avg) to UNKNOWN, or if LOO accuracy is substantially below the 100% implied by the single example, the discriminativity assumption fails and the identification claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, §1) is that fuzzy hashing enables identification and recognition of unknown executables across versions, compilers, and code changes. The only direct evidence is §4.3's UNKNOWN case: Table 7 lists similarity scores against icon instances only. There is no report of similarity scores to other known software (LAMMPS, GROMACS, amber, miniconda, etc.), no decision threshold, and no false-positive/negative analysis. Consequently, the observed 100% match to one icon instance could be non-unique; many HPC executables share toolchain-generated strings, compiler .comment sections, and common dependency libraries (e.g., libsci, netcdf, hdf5, climatedt), so an unrelated executable might also produce high average similarity. Also, Table 7 shows FI_H (raw binary hash) is 0 for several icon variants while ST_H and SY_H carry the signal; the claim that the raw executable fuzzy hash is stable within a family is false at least for icon, and the methodology's reliance on strings/symbols is unvalidated. The paper's own labels come from regex path matching, so the 'known' set may be noisy, but even granting the labels, no accuracy metric is computed. Thus the strongest claim is currently supported by anecdote, not measurement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SIREN is an LD_PRELOAD-based data collection framework for HPC systems, deployed in an opt-in campaign on LUMI. It collects process metadata, environment variables, loaded modules, shared libraries, compiler identification strings, Python package imports, and SSDeep fuzzy hashes of the raw executable, printable strings, and ELF symbol table. The paper presents descriptive statistics from 12 users, 13,448 jobs, and 2,317,859 processes, and argues that fuzzy hashing enables both recognition of repeated executions of known software and identification of unknown executables. The central identification claim is made in the abstract and Section 1, and is supported in Section 4.3 by a single case where an executable with a nondescript name was matched to the icon software family (Table 7).","tokens_in":16311,"tokens_out":3633,"duration_ms":36976,"significance":"If the central claim were fully validated, SIREN would be a meaningful step beyond job-name and cryptographic-hash-based tracking: similarity-preserving executable fingerprints could enable identification across versions, compilers, and rebuilds while preserving user privacy. The paper contributes a working open-source artifact, a real opt-in deployment, and a rich set of descriptive analyses (library, compiler, and Python package usage) that are valuable to the HPC operations community. However, the key novel claim—similarity-based identification of unknown software—rests on a single anecdotal match with no error analysis, no negative controls, and no comparison against simpler baselines. The current evidence is not sufficient to establish that SSDeep-based fingerprints are discriminative across software families, and the paper should be revised to supply that evidence.","major_comments":[{"comment":"The central identification claim is supported by only one positive match (UNKNOWN→icon). The table reports similarity scores only against icon instances; there are no similarity scores against the other derived labels (LAMMPS, GROMACS, miniconda, amber, etc.), no negative controls, no decision threshold, and no precision/recall or false-positive/false-negative analysis. The average similarity even mixes executable-content hashes (FI_H, ST_H, SY_H) with environment/context hashes (MO_H, CO_H, OB_H), which can reflect shared modules or default libraries rather than software identity. A rigorous evaluation needs a full similarity matrix over all labeled executables, a threshold analysis, and per-hash-type accuracy metrics.","section":"§4.3, Table 7"},{"comment":"The claim in the abstract and Section 1 that fuzzy hashing of executables detects similarity despite code changes, versions, and compilers is not supported by the raw-binary hash. In Table 7, FI_H (raw file hash) is 0 for four of the ten icon rows, while ST_H and SY_H carry most of the signal. Thus the current data contradict the broadest version of the claim, at least for icon. The paper should report which hash types actually drive identification, justify the combination rule, and either narrow the claim (e.g., to strings and symbol tables) or add evidence that raw-binary fuzzy hashing is effective in other families.","section":"§4.3, Table 7"},{"comment":"The 'known' labels are derived from executable file and path names via regular expression matching. This is a noisy ground truth: the same path patterns can contain renamed or copied executables, and the paper itself notes the unreliability of arbitrary user-provided names. The evaluation needs at least an audit of label quality (e.g., manual inspection or independent confirmation via compiler/shared-library evidence) and a discussion of how label noise affects the reported identification accuracy. Without this, the identification result remains anecdotal even if the matching procedure is internally consistent.","section":"§4.3, Table 5"},{"comment":"SIREN is described as lightweight and as imposing minimal overhead, but no overhead measurements are reported. Since LD_PRELOAD-based instrumentation is a central design choice and can perturb every process, the paper should provide at least startup-latency and runtime-overhead measurements for representative jobs, along with the observed UDP message-loss rate (currently only stated as 0.02% of jobs having missing fields). This is not the main identification claim, but it is load-bearing for the framework's practical utility in production HPC systems.","section":"§2 and §3.1"},{"comment":"The paper correctly states that statically linked executables and processes inside containers are not collected. Given the prevalence of containers in modern HPC, this scope limitation should be reflected in the abstract and conclusions, not only in the limitations paragraph. The current framing ('improves observability in HPC') overstates coverage without this qualification.","section":"§3.1, Requirements and Limitations"}],"minor_comments":[{"comment":"Typo: 'UNKOWN' should be 'UNKNOWN' in the text introducing Table 7.","section":"§4.3"},{"comment":"The column header 'A vg. Sim.' has an extra space and should be 'Avg. Sim.'.","section":"Table 7"},{"comment":"The sentence describing the Damerau–Levenshtein distance switches between plural and singular subjects; please rephrase for clarity.","section":"§2.1"},{"comment":"The phrase 'derived and filtered libraries' is not defined precisely; state which substring combinations were derived, why they were chosen, and how the filtering affects the reported counts.","section":"§4.3 and Figure 2"},{"comment":"The phrase 'preserving privacy and file integrity' is vague; fuzzy hashes are not cryptographic commitments, and the privacy properties should be stated more carefully (e.g., that raw file contents are not stored).","section":"Abstract"},{"comment":"The Python package analysis is interesting but the counts of 'unique users' in Figure 3 appear to be at most 3; this should be stated explicitly so the reader can judge the strength of the package-level conclusions.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful systems/deployment contribution with a clear, fixable gap: the identification claim needs a proper evaluation. The current single-match evidence does not justify the abstract's claim. I would encourage the editor to request a revision rather than reject, because the framework, artifact, and deployment data are real and the missing accuracy analysis is within the manuscript's scope to provide. Please also ask the authors to be honest about the raw-binary fingerprint: their own Table 7 shows it is often 0, so the headline claim should either be narrowed or supported with additional data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: SIREN is a coherent, clearly described framework that combines LD_PRELOAD process hooking with SSDeep fuzzy hashes of raw binaries, printable strings, and symbol tables, and it ships a working artifact on Zenodo. The first deployment on LUMI (12 users, 13k jobs, 2.3M processes) is honestly reported, and the paper is transparent about its limitations: no static binaries, no containers, opt-in sample. The descriptive analyses of shared-library variants, compiler strings, and Python imports are genuinely useful for HPC operations. Credit is due for the architecture and the deployment data. The soft spot is exactly where the paper makes its strongest claim. Section 4.3's UNKNOWN case is the only direct evidence that fuzzy hashing can identify an unknown executable across versions and compilers. Table 7 shows strong similarity to icon, but there is no comparison to similarity scores against other known software, no decision threshold, no false-positive or false-negative analysis. The reader's stress-test note is on target: FI_H (raw binary hash) is 0 for several icon variants, so the raw hash is not stable within the family; the signal is carried by strings and symbols, which are heavily toolchain-dependent and not shown to be discriminative across families. The known labels themselves come from regex path matching, which is the very thing SIREN is meant to improve. That does not sink the framework, but it means the identification capability is plausible anecdote, not measurement. A secondary shortcoming is the lack of comparison with XALT or LDMS, despite the related-work framing. The paper would be stronger with a small controlled evaluation on a labeled set of executables (even synthetic versions of the same software built with different compilers and flags) to measure precision/recall. The system-directory analysis, especially the deviating bash library sets, is a solid practical addition. Who is this for? HPC system operators and user support teams, and researchers working on HPC observability and software usage analytics. It deserves a serious referee and likely revision; the framework is real, the deployment data is real, but the central claim needs support that is currently missing. I would bring it to a reading group for the architecture and analysis techniques, and I would cite it as a reference for process-level monitoring with fuzzy hashing. Send it to peer review, not desk reject.","headline":"SIREN is a real, deployable framework with an honest first deployment on LUMI; the architecture is sound, but the headline identification claim rests on one anecdotal match, not a measured result.","tokens_in":673,"tokens_out":1088,"would_cite":true,"duration_ms":26062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that SSDeep fuzzy hashing of executables lets HPC systems identify and recognize software even when builds differ by code changes, versions, or compilers, and shows it working in an opt-in deployment on LUMI.","keywords":["HPC observability","software identification","fuzzy hashing","SSDeep","LD_PRELOAD","executable similarity","operational data analytics","LUMI"],"falsifier":"Rebuild one software package across a range of compilers, versions, and optimization flags, and also build several unrelated packages, then measure the six-channel SSDeep similarity for every pair. The central claim fails if two executables from different packages score at or above the similarity that currently matched UNKNOWN to icon (67–100% average), or if two builds of the same package score below it. The paper reports no false-positive or false-negative rates, and its only validation is the single icon match.","tokens_in":15913,"feed_emoji":"🔍","tokens_out":9334,"duration_ms":69221,"temperature":0.7,"pith_summary":"SIREN is a process-level data collection framework that aims to solve a concrete problem for HPC operators: telling what software is actually running. Job names and file names are unreliable because users can name binaries anything (a.out), and cryptographic hashes are useless once a rebuild changes a single bit. The paper's central claim is that SSDeep fuzzy hashes—computed on the raw executable, its printable strings, and its ELF symbol table—preserve enough structure that different builds of the same software score highly similar, so an unknown executable can be recognized as belonging to a known family. SIREN collects these hashes via an LD_PRELOAD-injected library at process start and exit, along with modules, compilers, shared libraries, and imported Python packages, sending them over UDP to a central database. A first opt-in deployment on LUMI (12 users, 13,448 jobs, 2,317,859 processes) shows recognition of repeated executions and the identification of one nondescript unknown binary as a variant of the icon application, with a perfect 100% similarity match on all hash channels.","feed_headline":"Fuzzy hashes tell HPC operators which software is really running","feed_subtitle":"SIREN fuzzy-hashes binaries, strings, and symbols so rebuilds still match—proven on 2.3M LUMI processes.","key_machinery":"The load-bearing mechanism is SSDeep's Context-Triggered Piecewise Hashing (CTPH): a file is split into content-defined chunks, each chunk is hashed, and the hashes are concatenated; two fuzzy hashes are compared via Damerau–Levenshtein distance into a 0–100 similarity score. SIREN computes three such hashes per executable—raw binary, printable strings, global ELF symbol table—plus hashes of modules, compilers, and shared libraries. The multi-channel comparison lets an unknown executable be matched to a software family, not just to one exact file. Collection rides on an LD_PRELOAD-injected library active at process start and exit, sending UDP messages to a central database.","core_discovery":"On its own terms, the paper establishes a mechanism for software identification in HPC: instead of matching exact bytes or trusting user names, SIREN fingerprints each executable with three SSDeep fuzzy hashes—raw binary, printable strings, global ELF symbol names—plus hashes of modules, compilers, and shared libraries. Because fuzzy hashes preserve similarity, executables from the same family cluster together despite differing versions, code changes, or compilers. The demonstration: an executable with a nondescript name, labeled UNKNOWN, matched the icon family, with one instance at 100% similarity on all six channels and others at 67–95% average. The paper reads this as evidence that fuzzy","pith_inferences":["The similarity graph among executable fuzzy hashes could itself generate software labels by clustering, which would remove the paper's reliance on regex-matching file paths—the same unreliable signal SIREN is meant to supersede. The UNKNOWN-to-icon match suggests such clustering is feasible but untested at scale.","The three hash channels likely have different stability profiles: symbol-table hashes should survive compiler changes best, raw-binary hashes worst. A systematic sweep over compilers, versions, and optimization flags could map where each channel breaks, turning the single-case validation into a measured curve.","Because only hashes and metadata leave the node, the approach suggests privacy-preserving federated software-usage statistics across multiple HPC sites: sites share similarity scores, not binaries."],"forward_implications":["HPC operators can recognize repeated executions of the same application even when it is rebuilt with different compilers, versions, or flags, enabling longitudinal studies of performance variability and software usage.","Unknown executables with arbitrary names (e.g., a.out) can be identified against a corpus of known software purely from similarity, with no need to store or inspect binary contents.","The bundled metadata—compiler identification strings, loaded shared libraries, imported Python packages—gives support teams diagnostic signal for troubleshooting and for spotting unexpected behavior such as misuse or cryptomining.","Application identity preserved across variants supports downstream uses the paper lists: CPU frequency tuning, node sharing among compatible applications, and energy-consumption prediction."],"supporting_citations":[{"why":"Prior work applying fuzzy-hashing malware-detection techniques to HPC application classification; SIREN's direct method precedent.","marker":"[22]"},{"why":"Defines context-triggered piecewise hashing, the algorithm SSDeep implements to produce the fuzzy hashes.","marker":"[23]"},{"why":"Supplies the Damerau–Levenshtein distance SSDeep uses to turn two fuzzy hashes into a 0–100 similarity score.","marker":"[16]"},{"why":"XALT, the established LD_PRELOAD-based software-tracking framework whose process-metadata collection SIREN extends with fuzzy hashing.","marker":"[6]"},{"why":"The regex-on-path practice for deriving software usage labels that the paper relies on to name known applications.","marker":"[34]"},{"why":"Cybersecurity reference establishing fuzzy hashing as a proven similarity technique, motivating its adoption here.","marker":"[24]"}],"fun_headline_variants":["Fuzzy hashes unmask software running in HPC clusters","SIREN: binary, strings, symbols hashed to spot software families","Similarity hashing identifies unknown HPC executables by family","SIREN tracks software via fuzzy hashes, proven on LUMI","Hashing executables reveals real software despite arbitrary names"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The approach assumes SSDeep fuzzy hashes of a binary, its strings, and its symbol table stay similar across builds of the same software and stay different across builds of different software; the paper validates this on a single unknown executable, and its known labels come from regex-matching file paths—the very thing SIREN is meant to improve upon.","fun_headline_variants_meta":{"raw":{"variants":["Fuzzy hashes unmask software running in HPC clusters","SIREN: binary, strings, symbols hashed to spot software families","Similarity hashing identifies unknown HPC executables by family","SIREN tracks software via fuzzy hashes, proven on LUMI","Hashing executables reveals real software despite arbitrary names"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000164,"raw_usage":{"total_tokens":1061,"prompt_tokens":700,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":285}},"tokens_in":444,"tokens_out":361,"duration_ms":3964,"temperature":1.0,"reasoning_tokens":285,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:03:13.986961+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild one software package across a range of compilers, versions, and optimization flags, and also build several unrelated packages, then measure the six-channel SSDeep similarity for every pair. The central claim fails if two executables from different packages score at or above the similarity that currently matched UNKNOWN to icon (67–100% average), or if two builds of the same package score below it. The paper reports no false-positive or false-negative rates, and its only validation is the single icon match.","supporting_citations":[{"cited_title":"In: SC24-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis","cited_arxiv_id":null,"evidence_quote":"Prior work applying fuzzy-hashing malware-detection techniques to HPC application classification; SIREN's direct method precedent."},{"cited_title":"Communi- cations of the ACM7(3), 171–176 (1964)","cited_arxiv_id":null,"evidence_quote":"Supplies the Damerau–Levenshtein distance SSDeep uses to turn two fuzzy hashes into a 0–100 similarity score."},{"cited_title":"In: 2014 First International Workshop on HPC User Support Tools","cited_arxiv_id":null,"evidence_quote":"XALT, the established LD_PRELOAD-based software-tracking framework whose process-metadata collection SIREN extends with fuzzy hashing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The regex-on-path practice for deriving software usage labels that the paper relies on to name known applications."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cybersecurity reference establishing fuzzy hashing as a proven similarity technique, motivating its adoption here."}],"review_version":1}