How Stable Is a PNT Resilience Score? Decision-Instability of Single-Number Resilience Ratings under Framework-Aligned Weighting

Chakshu Baweja

REVIEW 2 major objections 6 minor 14 references

How Stable Is a PNT Resilience Score? Decision-Instability of Single-Number Resilience Ratings under Framework-Aligned Weighting

T0 review · 2 major / 6 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single PNT resilience score or maturity Level is not a stable decision basis: re-weighting and threat choice reorder who wins.

desk verdict Solid, carefully scoped simulation study showing that single-number PNT resilience scores and weakest-link Levels are unstable exactly when designs contend; the open engine and quantified flip rates are the real contribution. read the letter →

arxiv 2607.05415 v1 pith:K4CN6MRV submitted 2026-06-22 cs.CR stat.AP

Chakshu Baweja This is my paper · ORCID

classification cs.CRstat.AP

keywords PNTresiliencecompositeindicatorsdecisioninstabilityself-attestationcommon-modefailurematurityLevelGNSSweightingsensitivity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Authoritative PNT resilience frameworks define what resilience means but give only self-attestation checklists or maturity Levels, with no measurement engine. This paper builds an open, deterministic scoring engine over a PNT simulator that emits per-dimension sub-scores tied to scenarios and oracles, then tests whether collapsing those scores into one composite number or one Level is safe for decisions. Across seven architectures that trade off across dimensions, a broad space of weightings over the seven framework categories, and five threat scenarios, the answer splits. A single composite is stable when one design clearly dominates or under near-equal weights, but re-weighting alone flips the winner in up to 22 percent of draws under ordinary conditions where designs compete. Sharper still, a weakest-link maturity Level depends on which threat is assumed rather than on the architecture itself. Because the composite rewards declared techniques, a paper checklist can outscore a more resilient system, and apparent multi-receiver GNSS redundancy collapses to effective diversity of one under a shared failure domain. The paper therefore argues for reporting provenance-tagged sub-scores and rank ranges instead of a phantom single number.

What carries the argument

An open, deterministic scoring engine that maps each architecture and simulated threat to framework-aligned per-dimension sub-scores (from holdover, availability, detector performance, integrity, and bounded-degradation drivers), then aggregates them under a Dirichlet simplex of weightings and a minimum-over-categories Level rule with a bounded-degradation gate.

What would settle it

Re-run the same seven architectures and five threats with a different detection-AUC threshold or a different family of sub-score mappings; if the Level still changes with threat for the diverse architecture and the composite still flips under nominal re-weighting, the central claim stands; if either instability disappears, it falls.

Watch

Extended reading notes

Core claim

A single composite PNT resilience score is stable under active denial and near-equal weightings but reorders contesting designs under broad weightings, while a weakest-link maturity Level is a function of the threat assumed rather than of the architecture. Self-attestation can be gamed by declaration alone, and apparent multi-source GNSS redundancy reduces to effective diversity of one once a common-mode failure domain is recognized. The remedy is per-dimension sub-scores with provenance and a rank range, not one number.

Load-bearing premise

The simple physical model that turns each architecture and threat into a handful of behaviour drivers, and the fixed detection threshold that decides whether degradation is bounded and therefore what Level is assigned.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The punchline is clean: under a Dirichlet re-weighting of the seven RPCF categories, the composite winner flips in ~22 % of draws under nominal conditions but only ~1 % under active denial, and a weakest-link Level (their operationalization, not RPCF’s) is threat-dependent for one of seven architectures. That is the result a procurement reader needs.

What is new is not the composite-indicator sensitivity itself—they correctly cite Saisana/Saltelli/OECD—but the open, oracle-checked scoring engine that maps architectures and threats onto RPCF-aligned sub-scores with provenance, plus the concrete flip rates, rank ranges, and Kendall-tau numbers on a seven-architecture panel. The paper-tiger declaration-gaming example and the common-mode diversity collapse (4.0 → 1.0) are existence proofs by construction, and they land. The honesty discipline is real: every claim is scoped as simulation-derived self-assessment, not certification; the ±20 % driver-magnitude perturbation holds the qualitative conclusions; code, seeds, and mappings are public.

Soft spots are exactly the ones the authors flag. The physical reduction (holdover, AUC, bounded-degradation flag) and the fixed detection-AUC threshold / Level cutpoints (0.2/0.4/0.6/0.8 plus the Level-2 gate) are untested for functional form. That softens H1b more than the composite flip rates, which rest on re-weighting. The panel is synthetic and deliberately adversarial, so prevalence is not claimed. None of this is hidden, and none of it invents a contradiction.

Math and citation pattern look solid: Dirichlet sampling, inverse-Simpson diversity, Kendall tau, and the composite-indicator literature are used correctly. No load-bearing circularity.

This is for people who write or buy against RPCF / IEEE P1952 and for anyone building measurement layers on top of declarative frameworks. It deserves a serious referee; the engine and the quantified instability numbers are worth the time. I would engage.

Referee Report

2 major / 6 minor

Summary. The paper supplies an open, deterministic scoring engine that maps PNT architectures and simulated threat responses to per-dimension sub-scores aligned with the seven DHS RPCF technique categories (with re-projections to RDRR and Yang), each carrying scenario/oracle provenance and a validated-or-modelled tag. It then tests whether a single composite (weight-normalized mean) or a tentative weakest-link maturity Level (minimum over categories with cutpoints 0.2/0.4/0.6/0.8 and a bounded-degradation gate) is a stable decision basis. Across seven architectures spanning cross-dimension tradeoffs, a Dirichlet simplex over the seven categories, and a five-threat ensemble, composite winner flips under re-weighting reach ~22% in the nominal regime but ~1% under active denial and near zero under near-equal priors; the tentative Level changes with threat for one of seven architectures; a constructed paper-tiger single-band receiver that declares all seven techniques outscores a more resilient GNSS-inertial system; and fourfold GNSS redundancy collapses to effective diversity one under a shared common-mode domain. The authors recommend reporting provenance-tagged sub-scores and rank ranges rather than a single number, and scope the work as simulation-derived self-assessment aligned to RPCF v2.0, not certification.

Significance. If the results hold within the stated simulation scope, the paper is a useful, carefully scoped caution for PNT procurement and standards work (RPCF, IEEE P1952): single-number resilience ratings are unstable precisely where designs contend, and a natural weakest-link Level can be threat-dependent rather than architecture-intrinsic. Strengths that should be credited include the open engine with hand-derived oracle tests, integrity-hashed artifacts, versioned seeds, explicit separation of apparent vs effective diversity (inverse-Simpson / Hill N2 over independence groups), and repeated honest scoping as modelled self-assessment rather than certification. The weighting-instability result is a domain application of established composite-indicator sensitivity analysis; the more framework-specific contributions are the Level threat-dependence under a min-over-categories operationalization, the declaration-gaming existence proof, and the common-mode diversity collapse. Reproducibility is a genuine asset.

major comments (2)

[§III-C, §IV-B, §VI] §III-C, §IV-B, and §VI: The H1b claim (tentative Level is a function of the threat assumed) is load-bearing for the paper’s “sharper, weighting-invariant failure,” yet it is driven by free parameters the robustness study does not vary: the Level cutpoints 0.2/0.4/0.6/0.8 and the detection-AUC threshold that gates the bounded-degradation flag and caps maturity at Level 2. The reported ±20% driver perturbation (40 replicates) holds functional form and those thresholds fixed, as Limitations already notes. Either add a cutpoint/threshold sensitivity sweep, or qualify H1b more tightly in Abstract/Results/Conclusion as demonstrated only under this specific ladder and threshold placement, not as a general property of any RPCF-aligned Level.
[§III-E, Table I, §IV-A] §III-E, Table I, §IV-A: The quantitative flip rates (22% re-weighting-only under nominal; pooled top-1 flip 0.053 at α=1) and the wide middle-rank ranges are computed on a seven-architecture panel that deliberately includes adversarial constructions (paper-tiger for H2; quad-GNSS for H3). The paper correctly states it does not claim the panel is unbiased, but the rates are presented as the central numerical result. Clarify in Results and Discussion that these percentages characterize this designed panel rather than a population of fielded systems, and consider reporting the same metrics on a restricted non-adversarial subset as a control so readers can separate panel design from weighting sensitivity.

minor comments (6)

[Title page] Author line encoding: “Ashforde O ¨U” appears corrupted; fix to the intended affiliation string.
[§III-C] §III-C: “we call the result atentativeLevel” has a missing space/markup glitch; render as “a tentative Level.”
[§III-D / Fig. 1] Fig. 1–5 captions are informative, but axis labels in the text description of Kendall tau quantization (steps of 2/C(7,2)≈0.095) would help readers interpret the reported mean tau of 0.90 without hunting the Method section.
[Abstract, §IV-B] Abstract and §I state “changing for one architecture in seven”; §IV-B reports 14.3%. Keep the wording consistent and note that only the diverse architecture moves (Levels 0–3), so the categorical claim rests on a single architecture’s trajectory.
[§II, Table I] §II: Yang’s accuracy criterion is correctly carried as an unmodelled gap; a one-sentence pointer in Table I or the assurance-report description that position-domain accuracy is absent would make the gap harder to miss for non-specialist readers.
[§IV-A] References [8]–[10] on composite-indicator sensitivity are appropriate; a brief explicit cross-reference in §IV-A to which of Saisana/Saltelli’s diagnostics (e.g., ranking robustness under weight uncertainty) the top-1 flip rate and Kendall tau instantiate would help methodologists map the PNT instance back to the handbook.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: flip rates, Level threat-dependence, and existence proofs are computed or constructed openly, not fitted or definitionally disguised as predictions.

full rationale

The paper is a controlled simulation study of decision-stability of composite scores and a weakest-link Level under Dirichlet re-weighting and a threat ensemble. The load-bearing numerical claims (top-1 flip rates ~22% nominal / ~1% denial, Kendall tau, rank ranges, Level flip for 1/7 architectures) are obtained by scoring seven architectures with a fixed engine and sampling weight vectors; they do not reduce by construction to a fitted target or to a prior theorem of the same authors. H2 (paper-tiger) and H3 (quad-GNSS common-mode collapse) are explicitly framed as constructed existence proofs and as consequences of the independence-group definition, respectively—not as empirical predictions. The Level rule is the authors’ own operationalization of the RPCF ladder, applied uniformly; threat-dependence is an observed output of that rule under different behaviour drivers, not a re-labeling of the rule itself. Self-citation of the Kshana engine [14] is the measurement apparatus and reproducibility artifact, not a load-bearing uniqueness or uniqueness-from-authors step. Composite-indicator sensitivity is attributed to the external literature [8–10] as a domain application. No fitted-input-as-prediction, no self-definitional claim presented as derivation, and no ansatz smuggled via self-citation. Score 0 is therefore appropriate.

Assumptions & free parameters 5 free parameters · 5 assumptions · 3 invented entities

The central instability claims rest on a small set of modelling choices that convert architectures and threats into numeric drivers, plus the authors’ own operationalization of the RPCF ladder. Free parameters are the cutpoints, thresholds and sampling concentrations that turn those drivers into Levels and flip rates. Domain axioms encode standard GNSS common-mode physics and diversity indices. The invented entities are the scoring mappings and the tentative Level rule themselves; they have no independent external validation beyond the open simulator.

free parameters (5)

Level cutpoints (0.2/0.4/0.6/0.8)
Hand-chosen thresholds that map the minimum sub-score onto tentative Levels 0–4; changing them would re-label architectures and alter the reported Level-flip rate.
Detection-AUC threshold for bounded degradation
Fixed but numerically unspecified in the text; gates the bounded-degradation flag that caps maturity at Level 2 and therefore drives the strongest categorical claim.
Dirichlet concentration α and 2000 draws
Sampling prior over the seven-category weight simplex; α-sweep is reported but the headline 22 percent flip rate is for the broad α=1 prior.
Source quality weights and independence-group assignments
Hand-assigned per architecture; they enter the inverse-Simpson diversity and the declaration-scaled procedural sub-scores.
±20 percent driver perturbation range
Chosen robustness envelope; conclusions held inside it but the envelope itself is free.

assumptions (5)

domain assumption Co-located GNSS RF sources share a single common-mode failure domain under wideband denial, so apparent multi-receiver redundancy collapses to effective diversity 1.
Invoked for H3 and Fig. 5; standard RF physics but load-bearing for the diversity claim.
domain assumption Inverse-Simpson (Hill number of order 2) over quality-weighted independence groups is an appropriate proxy for effective diversity.
Used for the Diversify sub-score and H3; paper notes it carries no failure-probability semantics.
ad hoc to paper A weakest-link (minimum-over-categories) ladder with cutpoints 0.2/0.4/0.6/0.8 and a bounded-degradation gate operationalizes the RPCF maturity Levels.
Explicitly the authors’ design choice, not RPCF’s rule; the threat-dependence result is a property of this operationalization.
ad hoc to paper Procedural categories Obfuscate/Limit/Isolate are scored from declared presence scaled by source quality and are therefore scenario-independent.
This is the mechanism that lets the paper-tiger outscore measured systems under the composite (H2).
standard math Symmetric Dirichlet on the seven-category simplex is a defensible broad prior over weightings for sensitivity analysis.
Standard composite-indicator practice; paper also reports the near-equal concentration sweep.

invented entities (3)

Tentative weakest-link RPCF Level with bounded-degradation gate
purpose: Turn seven continuous sub-scores into a single maturity label whose threat-dependence can be measured.
Not defined by RPCF v2.0; the categorical instability result is defined with respect to this entity.
Framework-aligned sub-score mappings (Verify←AUC, Diversify←Hill-N2, Mitigate←availability, Recover←holdover×bounded, procedural←declaration×quality)
purpose: Reduce simulated behaviour to the seven RPCF category scores that feed the composite and Level.
One defensible family among several; paper acknowledges alternative reductions could behave differently.
Paper-tiger single-band receiver declaring all seven techniques
purpose: Existence proof that declaration-driven composite scoring can be gamed (H2).
Deliberately constructed adversarial architecture, not an observed field system.

how reviews work

0 comments

Cite this review

Pith. "Pith review of How Stable Is a PNT Resilience Score? Decision-Instability of Single-Number Resilience Ratings under Framework-Aligned Weighting." pith.science (2026). https://pith.science/paper/K4CN6MRV

@misc{pith2026260705415,
  author       = {Pith},
  title        = {Pith review of: How Stable Is a PNT Resilience Score? Decision-Instability of Single-Number Resilience Ratings under Framework-Aligned Weighting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K4CN6MRV}},
  note         = {Machine review of arXiv:2607.05415}
}

read the original abstract

Authoritative positioning, navigation, and timing (PNT) resilience frameworks (the DHS Resilient PNT Conformance Framework, RPCF, and peers) define what resilience means but supply only self-attestation: a checklist or a maturity Level, with no engine and no measurement. We build the missing measurement layer as an open, deterministic scoring engine over a PNT simulator, emitting per-dimension sub-scores traceable to a scenario and an oracle, and ask whether a single composite score or maturity Level is a stable basis for a decision. Across seven architectures spanning cross-dimension tradeoffs, a Dirichlet simplex over the seven RPCF categories, and a five-threat ensemble, the answer splits in two. The composite winner is stable under active denial and under near-equal weightings (about 1 percent flip rate), so a single number is safe precisely where one design dominates; but re-weighting alone flips it in up to 22 percent of draws under nominal conditions, where designs contend, a known composite-indicator sensitivity. The sharper, weighting-invariant failure is categorical: a weakest-link maturity Level (our minimum-over-categories operationalization of the RPCF ladder, not the framework's rule) depends on the threat assumed, not the architecture, changing for one architecture in seven. Because the composite rewards declared techniques, a constructed single-band receiver declaring all seven outscores a more resilient system: self-attestation can be gamed by declaration. And apparent fourfold GNSS redundancy reduces, by the definition of a shared common-mode failure domain, to an effective diversity of one. Conclusions hold under +/-20 percent perturbation of every driver within the reduction. We report per-dimension sub-scores with provenance and a rank range, not a phantom single number. A self-assessment aligned to RPCF v2.0, not a certification.

Figures

Figures reproduced from arXiv: 2607.05415 by the authors.

**Figure 2.** Left: the pooled top-1 winner flip rate falls monotonically as the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 5.** Effective diversity assuming independence versus accounting for [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references

[1]

Resilient Positioning, Navigation, and Timing (PNT) Confor- mance Framework,

U.S. Department of Homeland Security, Science and Technology Di- rectorate, “Resilient Positioning, Navigation, and Timing (PNT) Confor- mance Framework,” Version 2.0, 2022

2022
[2]

Executive Order 13905: strengthening national resilience through responsible use of positioning, navigation, and timing services,

Executive Office of the President, “Executive Order 13905: strengthening national resilience through responsible use of positioning, navigation, and timing services,”Federal Register, vol. 85, no. 32, p. 9359, Feb. 2020

2020
[3]

A structured approach to achieving system resilience for Position, Navigation and Timing (PNT) systems,

RethinkPNT, “A structured approach to achieving system resilience for Position, Navigation and Timing (PNT) systems,” white paper, 2022. [Online]. Available: https://rethinkpnt.com

2022
[4]

Concepts of comprehensive PNT and related key technologies,

Y . Yang, “Concepts of comprehensive PNT and related key technologies,” Acta Geodaetica et Cartographica Sinica, vol. 45, no. 5, pp. 505–510, 2016

2016
[5]

P1952: Standard for Resilient Positioning, Navigation, and Timing (PNT) User Equipment,

IEEE, “P1952: Standard for Resilient Positioning, Navigation, and Timing (PNT) User Equipment,” standard in development
[6]

GNSS spoofing and detection,

M. L. Psiaki and T. E. Humphreys, “GNSS spoofing and detection,”Proc. IEEE, vol. 104, no. 6, pp. 1258–1270, 2016

2016
[7]

A framework to quantitatively assess and enhance the seismic resilience of communities,

M. Bruneau et al., “A framework to quantitatively assess and enhance the seismic resilience of communities,”Earthquake Spectra, vol. 19, no. 4, pp. 733–752, 2003

2003
[8]

Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators,

M. Saisana, A. Saltelli, and S. Tarantola, “Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators,”J. R. Statist. Soc. A, vol. 168, no. 2, pp. 307–323, 2005

2005

Show all 14 references

[9]

Handbook on Constructing Composite Indicators: Methodology and User Guide,

M. Nardo, M. Saisana, A. Saltelli, S. Tarantola, A. Hoffmann, and E. Gio- vannini, “Handbook on Constructing Composite Indicators: Methodology and User Guide,” OECD Publishing, 2008

2008
[10]

Saltelli et al.,Global Sensitivity Analysis: The Primer

A. Saltelli et al.,Global Sensitivity Analysis: The Primer. Chichester, U.K.: Wiley, 2008

2008
[11]

A new measure of rank correlation,

M. G. Kendall, “A new measure of rank correlation,”Biometrika, vol. 30, no. 1/2, pp. 81–93, 1938

1938
[12]

Measurement of diversity,

E. H. Simpson, “Measurement of diversity,”Nature, vol. 163, no. 4148, p. 688, 1949

1949
[13]

Diversity and evenness: a unifying notation and its consequences,

M. O. Hill, “Diversity and evenness: a unifying notation and its consequences,”Ecology, vol. 54, no. 2, pp. 427–432, 1973

1973
[14]

Kshana: an open, reproducible PNT-resilience simulator,

C. Baweja, “Kshana: an open, reproducible PNT-resilience simulator,” software, version 0.19.0, AGPL-3.0-only, 2026. [Online]. Available: https: //github.com/AshfordeOU/kshana

2026

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.

[1] [1]

Resilient Positioning, Navigation, and Timing (PNT) Confor- mance Framework,

U.S. Department of Homeland Security, Science and Technology Di- rectorate, “Resilient Positioning, Navigation, and Timing (PNT) Confor- mance Framework,” Version 2.0, 2022

2022

[2] [2]

Executive Order 13905: strengthening national resilience through responsible use of positioning, navigation, and timing services,

Executive Office of the President, “Executive Order 13905: strengthening national resilience through responsible use of positioning, navigation, and timing services,”Federal Register, vol. 85, no. 32, p. 9359, Feb. 2020

2020

[3] [3]

A structured approach to achieving system resilience for Position, Navigation and Timing (PNT) systems,

RethinkPNT, “A structured approach to achieving system resilience for Position, Navigation and Timing (PNT) systems,” white paper, 2022. [Online]. Available: https://rethinkpnt.com

2022

[4] [4]

Concepts of comprehensive PNT and related key technologies,

Y . Yang, “Concepts of comprehensive PNT and related key technologies,” Acta Geodaetica et Cartographica Sinica, vol. 45, no. 5, pp. 505–510, 2016

2016

[5] [5]

P1952: Standard for Resilient Positioning, Navigation, and Timing (PNT) User Equipment,

IEEE, “P1952: Standard for Resilient Positioning, Navigation, and Timing (PNT) User Equipment,” standard in development

[6] [6]

GNSS spoofing and detection,

M. L. Psiaki and T. E. Humphreys, “GNSS spoofing and detection,”Proc. IEEE, vol. 104, no. 6, pp. 1258–1270, 2016

2016

[7] [7]

A framework to quantitatively assess and enhance the seismic resilience of communities,

M. Bruneau et al., “A framework to quantitatively assess and enhance the seismic resilience of communities,”Earthquake Spectra, vol. 19, no. 4, pp. 733–752, 2003

2003

[8] [8]

Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators,

M. Saisana, A. Saltelli, and S. Tarantola, “Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators,”J. R. Statist. Soc. A, vol. 168, no. 2, pp. 307–323, 2005

2005