{"id":"28f768d1-e30e-431f-b0b2-8cfe7801cb91","arxiv_id":"2505.18760","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The paper proposes ARMS, a reputation framework with seven security signals and metrics to help OSS maintainers assess contributor cybersecurity risk.","lead":"ARMS is a proposed system that scores open-source contributors on cybersecurity reputation using seven signals drawn from security standards and GitHub tools. It aims to give maintainers a way to vet unfamiliar contributors, but offers no implementation or validation yet.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"S3–S7 are repository-level metrics with no actor-attribution rule for non-maintainers; as specified, ARMS computes only S1–S2 for typical contributors, so the claimed seven-signal operationalization is not actually delivered.","rationale":"After reading the paper and the reader's verdict, I find the reader's identified weakness (unvalidated predictive proxy and gameability) is real but is a standard limitation that the paper itself discloses and plans to test. The more load-bearing issue is whether the proposed signals are actually computable at the actor level. The paper's headline contribution is seven generic security signals mapped to concrete metrics; however, S3–S7 are defined as repository-governance attributes. The attribution caveat in Table 1 is a single sentence, and Section 4.3 does not integrate it into the scoring formulas. Without a precise rule, a prototype cannot compute scores for S3–S7 for non-maintainers, so the central claim is not just empirically unvalidated but underspecified in a way that affects the design's core. This does not change the reader's CONDITIONAL verdict; the gap could be closed in implementation and evaluated via the proposed prototype. It sharpens the condition: ARMS must first demonstrate that S3–S7 can be attributed to individual actors, or the framework should be honestly presented as a two-signal system with repository context. I therefore set agreement_with_reader to disagree because the reader's weakest assumption is not the same as this attribution gap, and verdict_should_be to UNCHANGED because CONDITIONAL remains appropriate for a vision paper with a fixable design gap.","tokens_in":14825,"tokens_out":9031,"duration_ms":76303,"concrete_test":"Build a minimal ARMS scorer from Table 1 and Section 4.3 and run it on a random sample of GitHub users who contributed at least one pull request to a non-owned repository in the past year (e.g., across 100 popular projects). For each user, attempt to compute S3–S7 using the paper's attribution rule: owner/maintainer status or a configuration-change commit attributable to the user. Report the fraction of users with defined (non-zero) S3–S7 scores. If fewer than 20% of non-maintainer contributors have any attributable S3–S7 evidence, the seven signals collapse to two for the target population, and the composite R0(a) is driven almost entirely by S1–S2 plus W2/W3. A secondary check: score Jia Tan's GitHub history up to his first non-trivial XZ contribution and verify whether the proposed attribution rule even applies to his account.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that ARMS operationalizes actor cybersecurity reputation via seven security signals, each mapped to concrete metrics (Table 1, Section 4.2). This fails for signals S3–S7 for the exact population the system targets: external contributors who are not owners or maintainers. S3–S7 measure repository configuration: dependabot alert status, secret scanning, branch protection, security policy, and automated workflows. These are attributes of repositories, not of an actor's behavior. The paper's only attribution rule is the Table 1 note: S3–S7 reflect repository configuration and are treated as context unless governance control is attributable to the actor (e.g., owner/maintainer). But Section 4.3 asserts that the security signal scorer aggregates those metric values into seven security signal scores (S1–S7) and composites them with weights alpha_s. No algorithm is given for attributing a repository-level metric to a non-maintainer contributor, nor for handling the resulting undefined scores in Score_S3..S7. For a typical contributor, those scores are either mathematically undefined or forced to zero, reducing the seven-signal framework to S1–S2 (artifact vulnerability and vulnerable dependencies). Even S1–S2 require attributing vulnerabilities to specific commits, which the paper does not detail. Thus the central operationalization claim is not merely unvalidated; as specified, it is incomplete for the main use case it targets.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This vision paper proposes ARMS, an Actor Reputation Metric System intended to help open-source maintainers assess the cybersecurity reputation of prospective contributors. The paper motivates the need for actor-centric measurement as a complement to artifact-centric security checking, presents a threat model distinguishing inexperienced contributors, reputation spoofers, and impersonation (with impersonation out of scope), and proposes seven security signals S1–S7 plus three calibration factors W1–W3. These signals are mapped to concrete metrics in Table 1, drawing on standards such as SLSA, CNCF, NIST SSDF, and OpenSSF Scorecard, and on GitHub-ecosystem tools. The trust-engine section gives candidate formulas for per-signal scoring, composite scoring, calibration, and benchmarking, and the paper closes with study designs for effectiveness evaluation and user behavior, plus discussions of privacy, gameability, and future extensions to agents and process metrics. The central claim is that ARMS can operationalize actor cybersecurity reputation from public GitHub interaction histories in a way that supports maintainer triage decisions.","tokens_in":15094,"tokens_out":4105,"duration_ms":37256,"significance":"If the central claim were fully realized, ARMS would be a useful complement to artifact-centric supply-chain security: the paper provides a clear mapping from high-level security standards to measurable signals, an explicit threat model with a defensible out-of-scope boundary (impersonation), and a set of concrete, falsifiable evaluation designs, including difference-in-differences, retrospective incident prediction, ablation, and vignette studies. The paper is also unusually candid about limitations: it acknowledges false positives/negatives, insider threats, privacy harms, gameability, and chilling effects, and it proposes specific mitigations. The main weakness is that the operationalization is incomplete for the exact population the system targets, external non-maintainer contributors, because signals S3–S7 are repository-level attributes without a stated attribution rule. As a vision paper this is not fatal, but the central 'seven-signal' claim is currently overstated; the paper needs either a concrete attribution mechanism or a narrowed claim that treats S3–S7 as repository context rather than actor behavior.","major_comments":[{"comment":"The central operationalization claim is not delivered for the population ARMS targets. Signals S3–S7 (dependabot status, secret scanning, branch protection, security policy, automated workflows) are repository-configuration metrics. The only attribution rule is the Table 1 note: \"S3–S7 reflect repository configuration and are treated as context unless governance control is attributable to the actor (e.g., owner/maintainer).\" For a non-maintainer external contributor, no rule is given for attributing these metrics to the actor, and no algorithm is specified for handling Score_S3..S7 when they are undefined or inapplicable. As written, a typical contributor receives only S1–S2, so the claim that ARMS computes \"seven security signal scores\" is unsupported. Please either specify an attribution rule for contribution events to repository-configuration changes, or explicitly redefine S3–S7 as repository-level context that modulates the actor-level signals, and describe how missing or undefined signal scores enter the composite R0(a) in §4.3.","section":"§4.2/Table 1 and §4.3"},{"comment":"The reputation computation is not yet operational even for signals S1–S2. The event-level extraction function f_s(e), the normalization function Norm(·), the calibration factors W1–W3, and the weights alpha_s are left unspecified; the paper explicitly defers alpha_s to future empirical analysis. For a paper whose contribution is to \"operationalize\" trust, this leaves the central mechanism as a schema rather than an instantiation. Please provide at least one fully specified baseline (e.g., uniform weights, min-max normalization, event counts weighted by package downloads) so that the framework can be instantiated and the ablation studies in §5.1 can be executed.","section":"§4.3 composite equation"},{"comment":"The threat model lists Reputation Spoofing as an in-scope adversary, but the only defenses offered are time-weighted scoring and human oversight (§6.1). The XZ Utils worked example (§4.5.1) is post hoc: it applies the framework after the fact to an actor with sparse public history, but it does not show that the proposed signals would have flagged this actor prospectively, and it does not explain how repository-level signals S3–S7 would be attributed to a non-maintainer contributor. The planned retrospective study in §5.1 is a reasonable start, but it needs to include a concrete test of spoofing resistance, such as measuring how quickly a newly created account can accumulate a high score under the proposed aggregation, or a comparison of pre-incident signal profiles for known spoofing attacks against matched controls.","section":"§3.1, §4.5, §6.1"}],"minor_comments":[{"comment":"The phrase \"ARMS operates considers three kinds of threat actors\" is ungrammatical; please revise.","section":"§3.1"},{"comment":"The displayed equation for Score_S_s(a) contains a rendering artifact (\"Norm ©\" and stray formatting); please fix the equation and ensure all variables are defined in the surrounding text.","section":"§4.3"},{"comment":"Reference [48] contains an editorial placeholder, \"Check the exact CCS year/track formatting used by your bibliography style,\" which must be removed before publication.","section":"References"},{"comment":"The statement \"our work assumes that actors are well-intentioned\" appears to conflict with the threat model in §3.1, which explicitly includes malicious reputation spoofers; please reconcile the scope description.","section":"§6.2"},{"comment":"The Table 1 note says S3–S7 are treated as context \"unless governance control is attributable to the actor (e.g., owner/maintainer)\", but no definition of \"governance control\" is given; please clarify what actions or roles establish attribution.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is explicitly positioned as a workshop vision paper (\"Workshop positioning & Next Steps\", \"For a first journal submission (TSE)\"). If the target venue expects completed empirical work, this is a scope concern; as a vision/position paper, the contribution is appropriate. The main technical gap is the missing attribution rule for S3–S7; without it, the central claim of a seven-signal actor reputation system is overstated. The revision should either supply the missing design details or narrow the claim, and the evaluation designs should include a concrete spoofing-resistance test."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead this before you decide. The stress-test note is on target: ARMS claims to operationalize seven security signals, but S3–S7 are repository-configuration metrics (dependabot status, secret scanning, branch protection, security policy, automated workflows). The paper's own Table 1 says these are treated as context unless governance control is attributable to the actor (e.g., owner/maintainer). For the population the system is meant to vet—external contributors who are not maintainers—there is no attribution rule, so their S3–S7 scores are either undefined or forced to zero. Section 4.3 still writes the composite as a sum over all seven signals without saying what happens when those terms don't exist. That is a load-bearing gap, not just missing empirical validation. Even S1–S2, which do target artifact-level behavior, need a commit-to-vulnerability attribution that is not specified.\n\nCredit where it's due. This is a well-scoped vision piece. Deriving signals from SLSA, CNCF, NIST SSDF, and other frameworks, and mapping them to concrete tool outputs, is a useful synthesis; the table is a decent starting point for the community. The proposed evaluation designs—difference-in-differences on security-practice adoption, retrospective incident prediction, ablation of the trust engine—are sensible and show the authors know the difference between a proposal and a result. They also state plainly that this is a vision paper and list limitations (gameability, insider threats, privacy, chilling effects) without hand-waving. The worked examples are post hoc and acknowledged as such. The self-citations are used as context, not as evidence.\n\nThe reader's conditional verdict is fair, but I'd shift the emphasis. The missing validation is not the real problem for a vision paper; the attribution gap is. A serious referee should ask the authors either to restrict S3–S7 to maintainer/owner candidates, or to propose a concrete way to attribute repository-configuration work to a contributor (e.g., PRs that enable branch protection or add a security policy). This is addressable, and the paper would be stronger for it.\n\nBottom line: worth sending to peer review. It's a serious, honest proposal with a fixable structural gap. I'd bring it to the reading group to argue about the attribution issue. I probably wouldn't cite it in my own work until the gap is closed.","headline":"The stress-test is right: five of the seven signals don't actually operate for external contributors, leaving a load-bearing attribution gap, but this is still a serious vision paper worth refereeing.","tokens_in":15667,"tokens_out":3660,"would_cite":false,"duration_ms":31239,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes ARMS, an Actor Reputation Metric System that would score open-source contributors on seven security signals so maintainers can triage their pull requests.","keywords":["software supply chain security","actor reputation","open-source maintainer vetting","security signals","reputation metrics","contributor triage","decision support"],"falsifier":"Compare pre-incident reputation scores for accounts behind known supply-chain backdoors, such as the XZ Utils attacker, with matched benign contributors; if the malicious accounts' scores are indistinguishable from or higher than those of benign contributors, the signals are not predictive of security outcomes.","tokens_in":14619,"feed_emoji":"🛡️","tokens_out":6528,"duration_ms":45526,"temperature":0.7,"pith_summary":"Open-source maintainers routinely merge code from strangers, and existing checks examine the artifact rather than the actor behind it. This paper argues that the ecosystem needs an Actor Reputation Metric System (ARMS): a trust engine that turns a contributor's public security-relevant history into a reputation score. The authors identify seven security signals—covering vulnerability handling, dependency hygiene, code scanning, integrity guarantees, branch protection, security policies, and automated workflows—and map each to concrete, measurable metrics derived from industry standards and existing security tooling. If such a system worked, maintainers could triage contributions from unknown actors with higher confidence, routing risky submissions into stricter review while letting trusted contributors move faster. The paper also outlines quasi-experimental studies to test whether these signals actually predict downstream security outcomes.","feed_headline":"Seven signals could vet open-source contributors by security history","feed_subtitle":"A proposed ARMS trust engine turns public contribution history into triage advice for maintainers.","key_machinery":"The load-bearing object is the seven-signal reputation model (S1–S7) together with weighting factors W1–W3. Signals S1–S2 capture actor behavior (vulnerabilities introduced through commits or pull requests, and use of vulnerable dependencies), while S3–S7 capture repository governance practices (code scanning, integrity guarantees, branch protection, security policies, and automated workflows) and are treated as context unless control can be attributed to the actor. Each signal aggregates one or more measurable metrics, and the trust engine computes a normalized per-signal score, a weighted linear composite, a calibrated score using W1–W3, and an ecosystem-benchmarked score; the final output is triage guidance rather than automatic acceptance or rejection. The design deliberately consumes best-available outputs from existing security tools and treats them as noisy evidence for decision support rather than ground truth.","core_discovery":"The paper's central claim is that actor reputation can be operationalized for cybersecurity in open-source development. It defines a reference architecture in which the maintainer team is the trustor, the prospective contributor the trustee, and ARMS the missing trust engine. Interaction history is distilled into seven security signals (S1–S7), each aggregated from concrete metrics such as time-to-fix vulnerabilities, the number of projects with vulnerable dependencies, security-alert status, protected branch counts, private vulnerability reporting, and automated workflow presence; three calibration factors (W1–W3) adjust evidence for downstream exposure, community tenure, and network centrality. The trust engine combines per-signal scores into a composite, calibrates and benchmarks that score, and outputs a recommended triage action—faster routing for high scores, stronger safeguards or additional verification for low or sparse histories—while treating missing signals as unknown rather than negative evidence. The paper demonstrates the design on the XZ Utils backdoor, where the attacker's sparse and shallow public history would yield a low reputation score, and on the Dexcom outage, which would depress the responsible engineers' signals; it explicitly leaves impersonation attacks out of scope because they break the stable-identity assumption that any reputation system requires.","pith_inferences":["The same signal architecture could be extended to software agents and bots that increasingly commit code; because agents evolve and are composed as dependencies, their reputation would need to be strongly versioned and lifecycle-aware.","A testable extension beyond the paper's retrospective studies is a randomized field experiment in which matched projects receive ARMS-style triage recommendations for new contributors, with merge decisions and downstream vulnerability incidence measured over time.","If public artifact history proves to be a weak proxy for security behavior, the framework could be strengthened by adding the social and process signals the paper lists as future work, such as code-review feedback quality and explicit threat-model discussions.","The chilling-effect concern implies a deployment boundary: ARMS is most defensible on high-criticality, security-sensitive packages, leaving hobby and low-risk repositories untouched, although the exact criticality threshold remains an open question."],"forward_implications":["Maintainers could route unknown contributors' pull requests by risk: high-scoring actors receive normal review, while low-scoring or sparse-history actors are steered into smaller-scoped pull requests, two-person review, or mandatory signing requirements.","ARMS-style actor metrics would complement artifact-based checks, closing a vetting gap the paper identifies in current supply-chain security practice.","If the proposed retrospective incident prediction study confirms predictive utility, projects could set their own risk thresholds and use benchmarked reputation percentiles before granting contributor or maintainer access.","A deployed ARMS would imply a privacy-preserving disclosure model in which contributors opt in, view dashboards of what will be shared, and maintainers receive only aggregated or pseudonymized scores."],"supporting_citations":[{"why":"Supplies the trustor/trustee/trust-engine model that ARMS instantiates for the open-source context.","marker":"[33]"},{"why":"Establishes the artifact/actor/operation framing of the supply chain and motivates the actor-side gap ARMS addresses.","marker":"[52]"},{"why":"Provides the provenance and integrity recommendations underlying signal 4 on integrity guarantees.","marker":"[71]"},{"why":"Supplies the secure-development practice recommendations from which several security signals are derived.","marker":"[66]"},{"why":"Offers an existing artifact-centric scoring approach and concrete metrics that ARMS complements and reuses.","marker":"[54]"},{"why":"Provides prior contributor-reputation measures, including the network centrality concepts used for calibration factor W3.","marker":"[31]"},{"why":"Contributes trust-decomposition concepts and metric ideas such as time-to-fix and community tenure.","marker":"[7]"},{"why":"Demonstrates composite security scoring with weighted signals, informing ARMS's aggregation design.","marker":"[83]"}],"fun_headline_variants":["Seven security signals could rate open-source contributors","ARMS: a trust engine for open-source contributor security","Reputation metrics to flag risky open-source contributors","XZ backdoor shows need for contributor reputation metrics","A reputation engine to vet open-source contributor risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A contributor's public, artifact-based history on collaborative development platforms is a stable and honest proxy for their future security behavior, and determined attackers cannot cheaply fabricate good signals.","fun_headline_variants_meta":{"raw":{"variants":["Seven security signals could rate open-source contributors","ARMS: a trust engine for open-source contributor security","Reputation metrics to flag risky open-source contributors","XZ backdoor shows need for contributor reputation metrics","A reputation engine to vet open-source contributor risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1374,"prompt_tokens":911,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":390}},"tokens_in":527,"tokens_out":463,"duration_ms":4187,"temperature":1.0,"reasoning_tokens":390,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:25:16.804289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare pre-incident reputation scores for accounts behind known supply-chain backdoors, such as the XZ Utils attacker, with matched benign contributors; if the malicious accounts' scores are indistinguishable from or higher than those of benign contributors, the signals are not predictive of security outcomes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Offers an existing artifact-centric scoring approach and concrete metrics that ARMS complements and reuses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the trustor/trustee/trust-engine model that ARMS instantiates for the open-source context."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the artifact/actor/operation framing of the supply chain and motivates the actor-side gap ARMS addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the provenance and integrity recommendations underlying signal 4 on integrity guarantees."}],"review_version":1}