{"id":"e8ae74ea-1922-4466-bd58-eebbe682bd3f","arxiv_id":"2611.17233","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A reproducible case study demonstrates that incomplete annual citation histories can substantially affect longitudinal bibliometric rankings, with perfect rank correlations on a common subset coexisting with large changes in Top-K membership.","lead":"This paper presents a case study framework for auditing missing annual citation data in longitudinal bibliometric rankings, using a single Google Scholar profile. It shows that different assumptions about missing years can change which publications are eligible for ranking and alter Top-K membership, even when the relative order of commonly eligible publications remains unchanged.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"No significant objection identified","rationale":"The reader's verdict is ACCEPT with high confidence, and the weakest assumption identified is that the three missing-data scenarios (A, B, C) are meaningful sensitivity analyses. The paper acknowledges this limitation explicitly in Section 7.4, stating that the scenarios 'do not exhaust the space of plausible missing-data mechanisms.' This is a structural limitation of any sensitivity analysis, not a flaw that undermines the central claim. The paper's central claim—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—is convincingly demonstrated within the scope of the three scenarios. The authors are transparent about all key limitations: single profile, hypothetical assumptions, inability to determine true missingness, tie-breaking dependence, and private dataset. The paper does not overclaim; it explicitly states that results are descriptive and should not be generalized. No internal inconsistency or hidden assumption undermines the argument. The reader's weakest assumption is correctly identified, but it does not constitute a load-bearing concern that would change the verdict. The paper's conclusions are appropriately conditional on this limitation. Therefore, the verdict should remain UNCHANGED (ACCEPT). Agreement with the reader is 'agree' because the reader correctly identified the weakest assumption, and the stress-test confirms that this assumption is acknowledged and does not invalidate the central claim.","tokens_in":17125,"tokens_out":712,"duration_ms":6991,"concrete_test":"As a verification step, independently re-implement the ranking-robustness analysis using the same frozen dataset and scenario definitions, and confirm that the reported Jaccard overlaps and displacement statistics are reproducible. This would test the reproducibility of the computational workflow, which the authors have made publicly available (v1.8.5 on GitHub and Zenodo).","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper is a carefully scoped, methodologically transparent case study that makes a modest but well-supported claim about the sensitivity of longitudinal bibliometric rankings to missing-data assumptions. The authors explicitly acknowledge all key limitations: the single-profile scope, the hypothetical nature of the zero-assignment scenarios, the inability to determine the true missingness mechanism, the dependence on tie-breaking conventions, and the lack of public access to the private dataset. The central argument—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—is convincingly demonstrated through the reported results. The three scenarios (A, B, C) are clearly defined, nested, and used only as sensitivity analyses, not as imputations of ground truth. The distinction between eligibility stability, score stability, and ranking-membership stability is well articulated and operationalized through separate diagnostics (eligibility counts, common-universe correlations, Top-K overlap, full-ranking displacement). The paper does not overclaim; it explicitly states that results are descriptive and should not be generalized. No internal inconsistency or hidden assumption undermines the central claim. The weakest assumption identified by the reader—that the three scenarios span a useful range of plausible missing-data mechanisms—is acknowledged by the authors as a limitation (Section 7.4: 'The three scenarios span explicit policies for missing annual observations but do not exhaust the space of plausible missing-data mechanisms'). This is a structural limitation of any sensitivity analysis, not a flaw in the argument. The paper's conclusions are appropriately conditional on this limitation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents a reproducible case-study framework for auditing citation-history gaps and quantifying the sensitivity of longitudinal bibliometric indicators and rankings to explicit missing-data assumptions. Using a single Google Scholar profile (595 consolidated publications, 2024–2025 window), it compares three scenarios: strictly observed annual counts (A), hypothetical zeros for internal gaps only (B), and hypothetical zeros for all missing years within a valid publication interval (C). The central finding is that while indicator values and relative ordering of publications eligible under both scenarios remain unchanged (Spearman/Kendall = 1.0), complete-ranking positions and Top-K membership can change markedly (e.g., acceleration Jaccard@50 = 0.563 between A and C). The paper emphasizes that these are hypothetical sensitivity assumptions, not evidence of provider semantics, and that results are descriptive and based on a single profile.","tokens_in":17421,"tokens_out":1471,"duration_ms":13953,"significance":"If the central claim holds—that unchanged indicator values and perfect correlations on a common subset do not guarantee stable rankings when eligibility changes—this is a methodologically important demonstration for the bibliometrics community. The paper provides a clear, reproducible framework for separating observation provenance, indicator computability, and ranking robustness. It explicitly acknowledges all key limitations (single profile, hypothetical scenarios, no public dataset). The contribution is not a new indicator or a large-scale empirical finding, but a carefully scoped case study that operationalizes a distinction (eligibility stability vs. score stability vs. ranking-membership stability) that is often conflated in practice. The reusable software workflow (Google Scholar Tracker) is a strength, though exact reproduction requires private data. The paper is well-positioned within the literature on responsible bibliometrics (Leiden Manifesto, Waltman review).","major_comments":[{"comment":"The Revival indicator definition is incompletely specified. Section 3.5 states that 'the implementation's convention for a zero preceding mean, percentage scaling, and any boundary cases must be transcribed from the tested source code before submission; the current experiment report does not specify those details.' This is a load-bearing gap because Revival shows different sensitivity patterns (e.g., Jaccard@10 = 0.667) and the reader cannot assess whether the results are robust to reasonable alternative definitions of the indicator. The paper should either provide the full formula in the text or include a clear reference to the exact code location and version that defines it. Without this, the Revival results are not independently verifiable from the manuscript alone.","section":"§3.5, Eq. (7)–(9); §5.2, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper acknowledges extensive ties (94.4% of Acceleration publications under A) and that Top-K boundaries sometimes cut through tie groups, making membership partly dependent on the deterministic secondary ordering. However, the main ranking results (Tables 2 and 3) are reported without any sensitivity analysis over tie-breaking rules. Given that the central claim concerns ranking stability, and ties are pervasive, the paper should at least report the range of possible Jaccard overlaps under alternative admissible tie resolutions (e.g., best-case/worst-case for membership). The current presentation may overstate the precision of the overlap estimates.","section":"§5.5, Table 3; §6.3","authors_acknowledge":"A deterministic secondary 15 ordering makes the exported Top-K lists reproducible, but it does not eliminate the analytical arbitrariness of selecting only part of a tied group at a cutoff.","authors_acknowledge_p":0.82},{"comment":"The three scenarios (A, B, C) are nested and cover a plausible range, but the paper does not discuss a scenario where missingness is correlated with citation volume (e.g., missing years are more likely for low-citation publications). The authors acknowledge this limitation in §7.4 ('missingness may be associated with publication age, citation volume, or provider coverage'), but they do not test any alternative scenario that addresses this. While I agree that a full missing-data mechanism analysis is beyond the scope, the paper could strengthen its sensitivity analysis by including a scenario where missing years are filled with a small positive constant (e.g., 1) rather than zero, to test whether the results are driven by the zero assumption specifically. This would not require imputing ground truth, but would test the robustness of the ranking changes to the magnitude of the hypothetical fill value.","section":"§7.4; §3.4","authors_acknowledge":"Since complete historical provider responses were not systematically archived, the present dataset cannot establish whether omitted rows correspond to zero suppression, incomplete provider coverage, or other causes.","authors_acknowledge_p":0.67}],"minor_comments":[{"comment":"The abstract states 'fourth' contribution after 'third' (line 'Fourth, we document a reusable Google Scholar Tracker software workflow...'). The numbering should be consistent: either list four contributions or rephrase to avoid the mismatch.","section":"§1, abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Growth indicator is defined as the sum of citations in 2024 and 2025. This is a recent citation volume, not a growth rate. The paper should clarify this earlier to avoid confusion with percentage growth measures, especially since the Revival indicator uses percentage growth.","section":"§3.5, Eq. (8)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The Jaccard overlap values for Acceleration A–B at K=50 is reported as 0.667, but the text in §5.3 says 'A–B acceleration Top-50 comparison has J@50 = 0.667'. However, Table 2 shows Acceleration A–B J@50 = 0.667, which is consistent. But the table also shows Acceleration A–C J@50 = 0.563, while the text says '0.5625' (more precise). Please ensure consistency in decimal places throughout.","section":"§5.3, Table 2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper refers to 'Google Scholar Tracker skill' but earlier it is called 'Google Scholar Tracker software workflow' or 'Google Scholar Tracker framework'. Consider using a consistent term throughout.","section":"§6.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The reproducibility statement mentions that exact reproduction requires frozen study data and configuration, which are not public. This is a significant limitation for a paper that emphasizes reproducibility. While I understand data privacy constraints, the paper could provide a synthetic dataset that mimics the structure of the real data to allow code testing.","section":"§7.7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The audit reports '38 possible inconsistency checks' but it is unclear what constitutes an inconsistency check. Please define this term or provide an example.","section":"§4.3","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The paper makes a modest but clean point: perfect rank correlations on a common eligible subset do not guarantee stable rankings when eligibility changes. The demonstration is methodologically careful—they separate observed zeros from missing observations, use three nested sensitivity scenarios without persisting hypothetical values, and report eligibility counts, Top-K overlap, common-universe correlations, and full-ranking displacement as distinct diagnostics. The software workflow is public and versioned. The authors are transparent about the single-profile scope, the hypothetical nature of the zero assignments, and the inability to determine the true missingness mechanism. The central claim is well supported by the reported results: for example, acceleration shows a Jaccard overlap of 0.563 at K=50 between the strictest and most permissive scenarios, despite perfect Spearman and Kendall correlations on the shared universe.\n\nThe soft spots are real but acknowledged. The three scenarios do not exhaust the space of plausible missing-data mechanisms, and the paper says so (Section 7.4). The private dataset is not publicly available, so exact numerical replication is not possible from the code alone—again, stated. The tie-breaking policy affects Top-K membership, and the paper flags this. These are not hidden flaws; they are stated limitations. The weakest load-bearing assumption is that the three scenarios span a useful range of missing-data mechanisms, but the paper does not overclaim—it presents the results as descriptive sensitivity analyses, not as estimates of ground truth.\n\nWho this is for: bibliometricians, research evaluators, and anyone building or using longitudinal citation rankings. The paper provides a reproducible framework and a concrete warning about interpreting rank correlations. It deserves a serious referee. I would accept the peer review assignment.","headline":"A careful, well-scoped case study that cleanly demonstrates a specific failure mode in longitudinal bibliometric rankings; the central claim holds, the limitations are transparent, and the paper deserves peer review.","tokens_in":17937,"tokens_out":437,"duration_ms":5938},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Perfect correlations can hide unstable rankings when eligibility changes","keywords":["bibliometrics","citation history","missing data","ranking robustness","sensitivity analysis","reproducibility","Google Scholar","longitudinal indicators"],"falsifier":"If a replication using a different researcher profile or citation database found that Top-K membership is stable across all three missing-data scenarios for all indicators (Jaccard overlap = 1.0 at all cutoffs), the paper's claim that eligibility changes can cause ranking instability would be shown to be conditional on the specific portfolio and not a general property of longitudinal rankings.","tokens_in":18605,"feed_emoji":"📊","tokens_out":1520,"duration_ms":13653,"temperature":0.7,"pith_summary":"This paper demonstrates a methodological trap in longitudinal bibliometric rankings: when missing citation years are handled differently, the set of publications eligible for ranking changes, yet the relative ordering of publications that remain eligible in both scenarios can stay perfectly correlated (Spearman and Kendall equal to 1.0). The paper shows that this perfect correlation does not guarantee stable rankings overall, because newly eligible publications can enter the ranking and displace previously ranked ones. Using a case study of 595 consolidated publications from a single Google Scholar profile, the author compares three scenarios for treating missing annual citation counts: using only observed counts, filling internal gaps with hypothetical zeros, and filling all missing years within a valid publication interval with zeros. While the common-subset correlations are perfect, Top-K membership changes markedly for some indicators—acceleration exhibits a Jaccard overlap of only 0.563 at K=50 between the strictest and most permissive scenarios. The paper argues that eligibility, score stability, relative ordering, and ranking membership are four separate dimensions that must be reported independently.","feed_headline":"Perfect correlations can hide unstable rankings when eligibility changes","feed_subtitle":"A case study shows that changing how missing citation years are handled alters Top-K lists even when common-subject ordering stays…","key_machinery":"The central machinery is a provenance-aware, read-only sensitivity framework that separates observed annual citation counts from absent records using an observation indicator $o_{i,t}$ and three nested missing-data scenarios ($A$, $B$, $C$) defined by availability predicates $a^{(s)}_{i,t}$. For each scenario, the framework computes indicator-specific eligibility $e^{(s)}_{i,m}$ and scores, then compares rankings using both common-universe correlations (Spearman's $\\rho$, Kendall's $\\tau_b$) and complete-ranking diagnostics (Jaccard overlap of Top-K sets, full-ranking displacement $\\Delta r_{m,i}$). The key identity is that $\\rho = \\tau_b = 1.0$ on the common eligible set is compatible with large displacements and Top-K turnover when newly eligible publications enter the ranking.","core_discovery":"The paper establishes that unchanged indicator values and perfect rank correlations on a common subset of publications do not guarantee stable rankings when the eligible population changes. Specifically, under three different missing-data scenarios (observed-only, internal zeros, all valid zeros), the relative ordering of publications eligible under both scenarios remains identical (Spearman and Kendall correlations equal to 1.0), yet complete-ranking positions and Top-K membership can change markedly. For example, the acceleration indicator shows a Jaccard overlap of only 0.563 at K=50 between the conservative and most permissive scenarios, and a mean absolute full-ranking displacement of 172 positions. The paper demonstrates that ranking robustness must be assessed along four separate dimensions: coverage (which publications can be scored), score stability, relative-order stability within the common eligible set, and composition/positional stability in complete rankings.","pith_inferences":["The paper's findings suggest that any bibliometric ranking based on annual citation differences is inherently fragile when missing data are handled differently, because the set of eligible publications can shift dramatically. This fragility is not captured by standard correlation metrics and could mislead research evaluation exercises that rely on longitudinal indicators.","The methodology could be extended to other citation databases (Scopus, Web of Science) and to other longitudinal indicators (e.g., percentile-based trajectories, field-normalized growth rates). A multi-provider replication would reveal whether the observed patterns are specific to Google Scholar's coverage semantics or general across sources.","The paper implicitly challenges the practice of imputing zeros for missing citation years without provenance information. If adopted as a reporting standard, the framework would require data collectors to retain raw provider responses and timestamps, which many current workflows do not do.","The extreme displacement observed for acceleration (mean 172 positions) suggests that year-to-year differences are particularly sensitive to eligibility changes, because adding many zero-filled publications creates a dense cluster of low or zero scores that pushes previously mid-ranked publications far down. This mechanism could be tested with synthetic data."],"forward_implications":["Longitudinal bibliometric rankings should report eligibility denominators alongside indicator values, so readers can see how many publications were excluded due to missing data.","A perfect rank correlation on a common subset should not be taken as evidence of overall ranking robustness; complete-ranking displacement and Top-K overlap must be reported separately.","The three missing-data scenarios provide a template for sensitivity analysis that other researchers can apply to their own citation datasets, making assumptions about zero-filling explicit and auditable.","Tie-breaking policies matter: with extensive ties (e.g., 94% of acceleration publications tied under scenario A), Top-K membership can be arbitrary at boundaries, so tie-inclusive or tie-averaged sensitivity analyses are needed before drawing substantive conclusions.","The framework's separation of observation provenance, indicator computability, and ranking robustness could inform a reporting standard for longitudinal bibliometrics, similar to the Leiden Manifesto's transparency principles."],"supporting_citations":[{"why":"Establishes the Leiden Manifesto principles for transparent data and analytical procedures, which the paper argues must be extended to longitudinal indicators.","marker":"Hicks et al. (2015)"},{"why":"Provides a comprehensive review of citation impact indicators and methodological choices, which the paper builds on by focusing on missing-data sensitivity.","marker":"Waltman (2016)"},{"why":"Discusses Google Scholar's potential and limitations for citation analysis, providing the context for why provider semantics of missing years are uncertain.","marker":"Harzing and van der Wal (2008)"},{"why":"Compares Google Scholar coverage with Web of Science and Scopus, supporting the claim that broader coverage does not guarantee complete longitudinal histories.","marker":"Martín-Martín et al. (2018)"}],"fun_headline_variants":[],"cache_read_input_tokens":0,"weakest_assumption_plain":"The three missing-data scenarios (observed-only, internal zeros, all valid zeros) are assumed to span a useful range of plausible missing-data mechanisms, but the paper acknowledges that the true missingness mechanism could be different (e.g., correlated with citation volume) and is not identified by the available data.","fun_headline_variants_meta":{"raw":{"variants":[]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"cost":0.0001705522,"is_byok":false,"cost_details":{"upstream_inference_cost":0.0001705522,"upstream_inference_prompt_cost":8.19182e-05,"upstream_inference_completions_cost":8.8634e-05},"total_tokens":1592,"prompt_tokens":1105,"completion_tokens":487,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":1105,"tokens_out":487,"cost_source":"openrouter_usage","duration_ms":5749,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T08:56:12.196992+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"If a replication using a different researcher profile or citation database found that Top-K membership is stable across all three missing-data scenarios for all indicators (Jaccard overlap = 1.0 at all cutoffs), the paper's claim that eligibility changes can cause ranking instability would be shown to be conditional on the specific portfolio and not a general property of longitudinal rankings.","supporting_citations":[{"cited_title":"Bibliometrics: The Leiden Manifesto for research metrics","cited_arxiv_id":null,"evidence_quote":"Establishes the Leiden Manifesto principles for transparent data and analytical procedures, which the paper argues must be extended to longitudinal indicators."},{"cited_title":"A review of the literature on citation impact indicators","cited_arxiv_id":null,"evidence_quote":"Provides a comprehensive review of citation impact indicators and methodological choices, which the paper builds on by focusing on missing-data sensitivity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Discusses Google Scholar's potential and limitations for citation analysis, providing the context for why provider semantics of missing years are uncertain."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}