{"id":"6b7dec57-44ed-406a-8efe-80b2fd98228e","arxiv_id":"2608.00997","paper_version":2,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Drift-ranked re-auditing catches only about 20% of changed descriptions at a top-5% budget, because few servers ever rewrite descriptions and half of all changes come from new arrivals.","lead":"The paper follows the official MCP server registry for 89 days and shows that most descriptions never change while a few servers churn constantly. It finds that re-auditing the most volatile servers misses most description changes, so auditors should re-check whenever a description's hash moves instead.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Observation cadence hides revert-within-interval churn; the coverage claim needs a bound under denser sampling.","rationale":"The reader's weakest_assumption is exactly the concern I identify: the four-hour commit-on-change sampling cadence means revert-within-interval changes are invisible. I agree that this is the single most load-bearing unquantified risk. The reader judged it 'acknowledged and does not overturn the policy conclusion.' My analysis agrees, but with a sharper reason: for the survival curve the miss is conservative (lower bound), while for the Section 5 coverage ratio the miss can cut either way, so it is the right place to probe. I also agree with the reader's overall ACCEPT verdict: the paper is unusually transparent, the code is deterministic against a released panel, the v1-to-v2 corrections demonstrate self-audit, and the central policy claim has independent support from the §5.3 exhaustion analysis (only 5.0% of servers have any description-change history, so a top-5% budget consumes the whole signal pool). That structural argument is cadence-independent. The concrete test I propose is a targeted dense-sampling study, which is the direct way to settle whether the unquantified sampling error matters. On the verdict: the concern is real but does not overturn the paper's central claim, so UNCHANGED is the appropriate recommendation rather than CONDITIONAL. The paper already discloses the limitation; a denser re-measurement would either add confidence or force a revised coverage figure, but neither outcome invalidates the policy recommendation that content-binding plus a full-catalog sweep is the right control. I rate agreement as 'agree' because the reader's identified weakest assumption is precisely the four-hour cadence issue; I am not introducing a new concern, only sharpening how it interacts with the coverage ratio.","tokens_in":13798,"tokens_out":2183,"duration_ms":21718,"concrete_test":"Re-run the coverage protocol at a higher observation density for a sub-period: take the 30-day held-out window and replay the panel at the finest granularity available in the git history, or better, add a temporary polling loop against the live registry (e.g., every 10 minutes for 7–14 days) and reconstruct the same rankable-coverage metric. If the top-5% rankable coverage under dense sampling stays within, say, 2 percentage points of the ~20% reported, the cadence concern does not materially change the policy conclusion. If it moves by more than that, the coverage claim needs a caveat or a denser-data revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central policy claim is that at a top-5% re-audit budget, prior-drift ranking catches only ~20% of previously-seen servers whose description changes in a held-out window, making drift-ranking a poor control for description revalidation. The measurement relies on a panel that records a change only when a full snapshot sync (every ~4 hours) differs from the previous committed snapshot. Any description edit that is made and reverted inside one sync interval is invisible, as are edits that oscillate between two hashes within one interval (e.g., A->B->A). The paper openly scopes itself to this cadence (§2.1, §7.3), and for the cohort-survival numbers (11.9% at 30 days) the missed events are conservative: they only lower the measured churn, so treating 11.9% as a lower bound is safe. The problem is different for the Section 5 coverage percentage. That number is a ratio of caught changers to detected changers in a held-out window. If hidden sub-interval churn exists, both numerator and denominator are missing, and the ratio can move in either direction. Suppose the hidden changers are predominantly servers with a prior drift history (the very population the ranking targets): then the ranking's measured ~20% coverage overstates its true coverage, and the paper's conclusion that drift-ranking is a weak control would be strengthened, since measured coverage is already weak. But suppose the hidden churn is mostly among new arrivals or the previously-stable majority: then the true coverage could be higher than ~20%, because the ranking would catch a larger share of the detectable changers than the measurement suggests. The paper's own §4.2, showing that drift is front-loaded on new arrivals, makes the second scenario plausible in direction, but the magnitude is unmeasured. Without a bound on how much population can change and revert inside one 4-hour interval, the headline coverage figure has an unquantified sampling error.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reconstructs a longitudinal panel of the official MCP registry from 120 version-controlled snapshot revisions over 88.6 days, covering 19,099 servers. It measures drift of the full server descriptor and of the description field alone, reports concentration and cohort survival, and evaluates whether re-auditing servers ranked by prior drift can keep description-level findings current. The central empirical claim is that at a top-5% re-audit budget, drift-ranking catches only ~20% of previously-seen servers whose description changes in a held-out window, and is blind to the ~half of changers that are new arrivals. The paper recommends content-binding (revalidate when the description hash moves) plus a periodic full-catalog sweep. A secondary deployment observation on the author's own scanner illustrates instrument lag as a separate staleness producer.","tokens_in":14191,"tokens_out":7658,"duration_ms":90115,"significance":"If the result holds, it is a valuable, actionable measurement for the MCP security community: it is the first longitudinal registry drift panel, and it directly challenges the default assumption that re-auditing the most drifted servers is an efficient control. The paper's artifact discipline is exemplary: pinned snapshot commit, pseudonymized released panel, SHA256SUMS, dependency-free regeneration code, and a v2 that self-corrects five v1 errors. The structural explanation for the under-coverage (small prior-description-history pool, new-arrival blind spot) is compelling and is backed by a test of the ranking against the released panel. The deployment observation is appropriately scoped as non-contributory.","major_comments":[{"comment":"The headline coverage figures (19.3% and 21.5% at the top-5% budget) are computed from changes detected at the ~4-hour snapshot cadence. Changes that occur and revert within one sync interval are invisible. The paper acknowledges the cadence for survival estimation (§7.3) but does not bound its effect on the coverage ratio, which is not a lower bound: both numerator and denominator are affected. If hidden revert-within-interval churn is concentrated among high-drift servers (plausible given the concentration in §4.1), the true coverage of drift-ranking could be higher than 20%; if it is concentrated among previously-stable servers, it could be lower. Please add a sensitivity analysis (e.g., simulate sub-interval churn under different concentration assumptions) or an explicit argument that the qualitative under-coverage conclusion is robust to this uncertainty. As written, the single most load-bearing number in the abstract is not a bound.","section":"§5.2–5.3, §7.3"},{"comment":"The policy evaluation rests on only two training cut-dates (44-day and 60-day), and the reported coverage is the spread across two points, not a confidence interval. The paper is appropriately cautious in §5.1 about the retrospective lift being 'one split of one window,' but the prospective coverage headline is given the same epistemic weight as the structural explanation. A bootstrap over cut-dates, or at least an explicit statement that the 19.3–21.5% range is a two-point sensitivity range rather than an estimate with quantified uncertainty, would help readers calibrate the claim.","section":"§5.2"}],"minor_comments":[{"comment":"The abstract says '120 observations' and §2.1 says '120 revisions' and 'up to six observations per day'; please harmonize the terminology (e.g., '120 snapshot revisions' or '120 observations').","section":"§2.1"},{"comment":"The three non-overlapping 30-day cohorts are the primary uncertainty, but their exact start dates are not given; providing them would let readers reproduce the non-overlap check.","section":"§3.2"},{"comment":"The sentence 'the shape beyond the 5% budget is sort-order dependent' is an important robustness check, but the related numbers (+0.86, +0.84, +0.69) are reported without a table; a small table or a pointer to the deposited figures.json would improve readability.","section":"§5.3"},{"comment":"This limitation could be expanded to state explicitly that the coverage ratios in §5 are not claimed as bounds on true (continuous-time) coverage, and to reference the sensitivity analysis requested above.","section":"§7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is one of the most transparent I have reviewed. The cadence concern is the only substantive obstacle; it is fixable with a sensitivity analysis or a bounded-bias argument. I would be happy to see the revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the first longitudinal panel of an MCP registry, and it does something genuinely useful—it measures how long the text a description-level auditor reads actually stays current. The artifact quality is unusually high: pinned snapshot commit, released pseudonymized panel, dependency-free analysis code, SHA256SUMS, and deterministic regeneration. The v2 self-corrections are substantive and confidence-inspiring, not cosmetic. The central policy claim—that drift-history ranking under-covers description revalidation—holds up. At a top-5% budget the ranking catches only ~20% of previously-seen description changers, and the mechanism (only 5% of servers carry any description history to rank on) is convincingly diagnosed. The content-binding-plus-sweep recommendation follows from the data and is the right control for the measured surface.\n\nThe soft spots are real but not load-bearing. The four-hour/commit-on-change cadence means any change that reverts within an interval is invisible. For the survival numbers that is conservative—lower bounds are safe. But for the Section 5 coverage ratio, the stress-test note is correct: hidden churn among the previously-stable majority could make true coverage higher than the measured ~20%, while hidden churn among drift-history servers would lower it. The paper acknowledges the cadence but does not bound this sampling error. That said, the policy conclusion does not hinge on the exact coverage number—content-binding is recommended regardless, and a sweep is sized from the measured event load, not from the coverage ratio. So this is a limitation to state more precisely, not a flaw that overturns the result.\n\nOther gaps are minor and mostly scoped by the author: single official registry, 100-character description field only, self-reported counts from 861 servers, and the Section 6 deployment observation is explicitly unreplicated. The citation pattern is fair—prior audits are correctly characterized as single-point-in-time, and the author's self-citations are to released datasets, not hidden dependencies.\n\nWho gets value: MCP security researchers deciding whether point-in-time audit findings can be trusted, and operators of description-level registry screens. The paper deserves a serious referee. I would send it to review, and ask the author to either bound the intra-sync hidden churn or rephrase the coverage claim as conditional on the observation cadence. A stated sensitivity analysis under denser sampling would fully close the gap.","headline":"First longitudinal MCP registry drift panel with strong artifact discipline; the drift-ranking under-coverage result holds, though the cadence-induced blind spot deserves a sharper bound before publication.","tokens_in":14640,"tokens_out":1333,"would_cite":true,"duration_ms":17411,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ranking MCP servers by past drift and re-auditing the top 5% catches only about one in five previously seen servers whose descriptions change, so a history-based re-audit budget is the wrong control for keeping description-level findings…","keywords":["Model Context Protocol","MCP registry","description drift","re-auditing","content-binding","heavy-tail distribution","survival analysis","registry snapshot panel"],"falsifier":"Sync the official MCP registry continuously for 30 days, or subscribe to its change events if available, rebuild the cohort survival and top-5% drift-ranked coverage on the resulting records, and compare: if description changes that are invisible to four-hour snapshots materially change the ratios, the paper's coverage and survival numbers would not be a complete description of true churn.","tokens_in":13594,"feed_emoji":"🔍","tokens_out":7961,"duration_ms":81301,"temperature":0.7,"pith_summary":"This paper measures how quickly the official Model Context Protocol (MCP) registry's server descriptions go stale, using 120 snapshots over 88.6 days covering 19,099 servers. Its central finding is about audit policy: if a researcher or operator tries to keep description-level findings current by re-auditing the servers that changed most in the past, a top-5% budget catches only about one in five of the previously seen servers whose descriptions actually change, and only about one in ten of all description changers. The reason is not that description rewrites are unpredictable — prior rewrites do predict later ones — but that only 5.0% of servers have any description-change history to rank on, and roughly half of all description changes land on newly arrived servers that no history-based ranking can reach. This matters because point-in-time audits of MCP registries are common, and no prior study measured how quickly the audited text turns over.","feed_headline":"Drift-based re-auditing catches only ~20% of stale descriptions","feed_subtitle":"Half of changes are to new servers no history can reach; hash-triggered revalidation plus a periodic sweep fits better.","key_machinery":"The load-bearing object is the delta-encoded hash panel: every registry snapshot is reduced to SHA-256 of the canonicalized server descriptor (any-field drift) and SHA-256 of the description alone (the description-revalidation surface), compressed because 75.2% of servers never change. On this panel the paper replays 120 observations to compute cohort survival curves, concentration tiers, and a prospective targeting protocol that ranks servers by training-window drift and measures held-out coverage. The identity doing the conceptual work is the distinction between the rankable population (servers present at ranking time) and the whole population of changers; the paper shows that roughly half of all description changes come from new arrivals that are unrankable by construction, which is why the ranking's lift (~4x) fails to convert into coverage (~20%). The proposed alternative mechanism is content-binding: bind each verdict to the description hash it judged, revalidate when that hash moves, and separately sweep the full catalog on a periodic schedule to catch new arrivals and the long tail.","core_discovery":"The paper's central result is a policy claim: keeping description-level findings current cannot be done by re-auditing the servers that drifted most in the past. On a panel of 120 observations of the official MCP registry over 88.6 days, a top-5% re-audit budget ranked by prior drift catches only ~19-21% of the previously-seen servers whose description will change in the held-out window (about 4x lift), against ~26-29% for any-descriptor drift, and only ~9-12% of all description changers once new arrivals are included. The binding limit is sparsity: only 5.0% of servers carry any prior description-change history at all, so the top-5% budget exhausts the signal pool and every slot beyond it is filled by tie-break; separately, roughly half of all description changes land on new arrivals that a history ranking cannot rank by construction. The paper therefore recommends content-binding — revalidate the moment a description's hash moves — plus a sized periodic full-catalog sweep, and reports direct 30-day description survival of 11.9% against a naive compounded prediction of 35.8%, a heavy-tail overestimate that the paper uses as a diagnostic rather than a result.","pith_inferences":["Beyond MCP, the same 'history-ranking under-covers sparse text surfaces' pattern should hold for any registry where a small fraction of entries ever rewrite the exact field being audited, such as package metadata descriptions; an auditor can test this by re-running the panel protocol on other registry snapshot histories.","The paper's hidden-intra-sync blind spot suggests its coverage numbers are lower bounds; a finer-grained sync or event-stream data could raise the measured churn and change the exact coverage percentages, though the structural limit of new-arrival blindness would likely remain.","A testable extension is to measure whether content-binding plus a monthly sweep catches a larger share of description changers than a top-20% drift ranking at comparable cost, which would give operators a direct cost-coverage comparison."],"forward_implications":["An operator who relies on a point-in-time MCP registry audit should expect description-level findings to apply to a shrinking share of servers: only 11.9% of a cohort's descriptions change within 30 days, but the 89-day survival is only about 19%.","Re-auditing the top 5% of servers by prior drift is not an effective control for description revalidation: it covers only ~20% of previously-seen description changers and ~10% of all description changers, so auditor budgets should not be allocated by churn ranking alone.","Content-binding plus a periodic full-catalog sweep is the control that fits the measured pattern: content-binding handles the roughly 39 description changes per day that need re-screening, and a sweep at cadence C bounds worst-case staleness to C days at a fixed, plannable cost.","The heavy-tail structure means compounding daily change rates into a population share overstates decay by roughly 3x at 30 days and 3.8x at 89 days; registry measurements should report direct cohort survival rather than extrapolated rates."],"supporting_citations":[{"why":"Supplies the largest prior single-snapshot MCP registry audit, the class of point-in-time finding this paper's revalidation policy is designed to keep current.","marker":"[1]"},{"why":"States the snapshot limitation explicitly and calls for the measurement this paper provides.","marker":"[2]"},{"why":"Represents a single-point attack analysis with no temporal discussion, the design the paper shows ages unevenly.","marker":"[3]"},{"why":"Recommends periodic re-scanning for tool poisoning but fixes no interval; the paper's revalidation rate and content-binding recommendation target this gap.","marker":"[4]"}],"fun_headline_variants":["Drift-ranked re-audits catch only ~20% of stale descriptions","Half of MCP description changes hit new servers, so history can't rank them","Hash-triggered revalidation plus a periodic sweep beats drift ranking","Past-drift ranking under-covers: ~20% catch, half changes on new arrivals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The panel only records registry state when a full sync changed it, roughly every four hours, so any description change that appears and reverts between syncs is invisible; all measured survival and coverage figures are lower bounds, and the under-coverage conclusion could weaken if hidden within-interval churn is large and distributed differently.","fun_headline_variants_meta":{"raw":{"variants":["Drift-ranked re-audits catch only ~20% of stale descriptions","Half of MCP description changes hit new servers, so history can't rank them","Hash-triggered revalidation plus a periodic sweep beats drift ranking","Past-drift ranking under-covers: ~20% catch, half changes on new arrivals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000818,"raw_usage":{"total_tokens":3727,"prompt_tokens":1232,"completion_tokens":2495,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":848,"completion_tokens_details":{"reasoning_tokens":2409}},"tokens_in":848,"tokens_out":2495,"duration_ms":21627,"temperature":1.0,"reasoning_tokens":2409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T04:11:26.174854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sync the official MCP registry continuously for 30 days, or subscribe to its change events if available, rebuild the cohort survival and top-5% drift-ranked coverage on the resulting records, and compare: if description changes that are invisible to four-hour snapshots materially change the ratios, the paper's coverage and survival numbers would not be a complete description of true churn.","supporting_citations":[],"review_version":2}