{"id":"d3af26a8-943a-42c8-b411-af46367e7677","arxiv_id":"2603.26791","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Joint LLM ranking of all citations within a paper identifies impactful references more accurately than isolated classification, gaining +9.5% accuracy and +8.3% F1.","lead":"Crystal uses LLMs to jointly rank every reference inside a citing paper by relative impact, instead of scoring each citation alone. The method beats prior impact classifiers and scales better, offering a practical way to separate foundational citations from incidental ones.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Evaluation mapping of Crystal's High-only to gold impact-revealing may inflate gains if Medium citations often match human impact notions.","rationale":"The reader's weakest_assumption correctly flags that the prompt definitions plus majority vote must produce labels that match human impact notions, and notes the binary gold vs three-class inference mismatch and the small pilot. That is the right neighborhood. The more precise load-bearing point is the post-hoc evaluation mapping itself (High only = impact-revealing). Ablations already show that joint ranking helps even under a binary prompt (+8.5% acc), so the core idea is not illusory; the concern is whether the full reported +9.5/+8.3 partly rides on how Medium is collapsed. Because the paper is otherwise carefully ablated, efficient, and multi-model, the appropriate stance remains CONDITIONAL rather than REJECT: the joint-ranking claim is supported, but the exact magnitude of the SOTA comparison should be stress-tested against the mapping. No stronger internal inconsistency appears; long-list omissions and ToT correlational nature are secondary. Hence verdict stays CONDITIONAL, with agreement only partial because the mapping step is the sharper single point of failure for the strongest claim.","tokens_in":25147,"tokens_out":665,"duration_ms":7424,"concrete_test":"On the 1,783-context evaluation set, re-score every citation that Crystal labeled Medium under the original UKP binary prompt (or have 2–3 independent annotators re-label those Medium instances as impact-revealing vs other using Arnaout's definition). Recompute Table 1 accuracy/F1 after (a) mapping Medium→impact-revealing and (b) mapping Medium→other. If the Crystal–UKP gap shrinks below ~2×SE under the more inclusive mapping, the headline gain is mapping-sensitive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim rests on Crystal outperforming UKP by +9.5% accuracy / +8.3% F1 after mapping Crystal's three-class outputs (High/Medium/Low) onto the binary gold labels of Arnaout et al. (2025). Per Table 3 and §A, only High is treated as impact-revealing; Medium and Low both become other. Ablation 1 (Table 4) shows that the three-class formulation mainly boosts precision by parking borderline cases in Medium rather than impact-revealing. If a non-trivial fraction of those Medium cases would have been judged impact-revealing by the original human annotators (or by a more graded human protocol), the reported deltas partly reflect an evaluation-time mapping choice rather than pure superiority of joint ranking. The pilot (six same-lab co-authors, Spearman ρ>0.7) validates ranking order under the authors' own High/Medium/Low definitions, not the binary gold standard used for the headline numbers. Thus the load-bearing assumption is that the High-only mapping is faithful to the gold notion of impact; if it is systematically stricter, the magnitude of the claimed improvement is overstated.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"Crystal proposes to characterize citation impact by jointly ranking all references inside a citing paper with an LLM judge, rather than scoring each citation context in isolation. To reduce positional bias the reference list is ranked three times under random permutations and impact labels (High/Medium/Low) are aggregated by majority vote; an optional RRF+ordinal-regression path is also offered. On the human-annotated binary impact dataset of Arnaout et al. (2025), after mapping only High to “impact-revealing,” Crystal improves accuracy by +9.5 % and F1 by +8.3 % on average across GPT-5.1, o4-mini and Qwen3-30B relative to the prior UKP LLM baseline, while using asymptotically fewer LLM calls (O(n) papers vs O(m) edges). Ablations isolate the joint formulation, three-class granularity and permutation voting as the sources of gain. A case study of ACL 1996/2000 papers shows that high-impact citation counts under Crystal place Test-of-Time winners among the top four papers of their years. The authors release Crystal-Bank (46.8 k papers with rankings and labels) and code.","tokens_in":25409,"tokens_out":790,"duration_ms":18114,"significance":"If the reported gains hold under a more carefully validated label mapping, Crystal supplies a practical, open-weight-compatible alternative to edge-level impact classifiers and a large public resource (Crystal-Bank) for downstream bibliometric work. The efficiency argument (O(n) vs O(m) LLM calls) and the demonstration that an open 30 B model already outperforms prior closed-model baselines are concrete engineering contributions. The ACL ToT alignment, while only correlational, offers an external, long-horizon sanity check that raw citation counts lack. These elements together make the work useful for the digital-libraries and science-of-science communities even if the absolute magnitude of the accuracy lift is later revised.","major_comments":[{"comment":"§A / Table 3 and Ablation 1 (Table 4): evaluation maps only High → impact-revealing while Medium and Low both become “other.” Ablation 1 shows that the three-class formulation mainly raises precision by parking borderline cases in Medium. Because the gold labels are binary, the headline +9.5 % / +8.3 % deltas partly reflect this post-hoc mapping choice. A sensitivity analysis that also treats Medium as positive (or a small human re-annotation of the Medium band) is needed to confirm that the claimed superiority of joint ranking is not inflated by a stricter decision threshold.","section":null},{"comment":"§F (pilot study): the prompt is validated only by six co-author annotators from the same lab ranking their own papers (Spearman ρ > 0.7). That experiment checks ranking order under the authors’ High/Medium/Low definitions; it does not establish that the High class coincides with the binary “impact-revealing” notion used by Arnaout et al. A modest external validation (even 50–100 double-annotated contexts drawn from the evaluation set) would substantially strengthen the load-bearing assumption that the prompt produces labels faithful to the gold standard.","section":null},{"comment":"§5 / §C and Figure 11: when reference lists exceed ~40 items, all three models omit substantial numbers of papers; omitted items are scored as errors. Because UKP scores every edge independently, the comparison is not fully apples-to-apples for long-reference papers. Reporting accuracy stratified by reference-list length (or restricting the main table to papers with complete rankings) would clarify how much of the gain survives under a fairer protocol.","section":null}],"minor_comments":[],"recommendation":"minor_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing is that Crystal’s joint ranking of every reference inside a citing paper, with three randomized LLM passes and majority vote, beats UKP by roughly +9.5% accuracy and +8.3% F1 on the Arnaout gold set, and the ablations actually isolate the joint formulation as the main driver. That is a real, usable advance for citation-impact work, not just a re-prompt of listwise ranking.\n\nWhat is new is the paper-level calibration: instead of scoring each citation context alone, they force the model to rank the whole reference list and assign High/Medium/Low under explicit quantity and indispensability criteria. The three ablations (binary vs three-class, independent vs joint, single-pass vs PSC voting) are cleanly done and show the joint setup is what cuts the over-prediction of “impactful.” Efficiency is correctly argued: O(n) calls instead of O(m). The open-weight Qwen numbers are competitive, the pilot Spearman >0.7 with co-authors is decent prompt validation, and Crystal-Bank (46.8k papers) plus code is the kind of resource people will actually use. The ACL ToT case study is correlational but sensible: high-impact counts put the award papers near the top and raw citation counts do not.\n\nSoft spots, in proportion. The evaluation maps only High to the binary gold “impact-revealing” and parks Medium with other. Ablation 1 shows that Medium is doing a lot of the precision work by absorbing borderline cases. If human annotators would have called many of those Mediums impact-revealing, the headline deltas are somewhat inflated by the mapping choice rather than pure ranking superiority. The pilot only checks ranking order under the authors’ own three-class definitions, not the binary gold. Gold set is modest (442 citing papers), long reference lists still drop items, and Semantic Scholar coverage is incomplete without scraping. None of these sink the central result; the joint gains survive even the binary ablation, and the deltas sit well above the reported SE.\n\nThis is for people who build or use citation-impact classifiers, bibliometric signals, or scholarly retrieval. It deserves a serious referee. I would accept it for peer review and expect the mapping and long-list issues to be discussed, not fatal. Worth engaging; I would cite the method and the bank if I were working in this area.","headline":"Clean joint-ranking method for relative citation impact that beats the prior LLM baseline with solid ablations and a useful release; the High-only mapping is a real but secondary caveat, not a collapse of the claim.","tokens_in":25998,"tokens_out":601,"would_cite":true,"duration_ms":34184,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Jointly ranking every reference inside a citing paper lets LLMs distinguish high-impact citations more reliably than scoring each one alone.","keywords":["citation impact","listwise ranking","large language models","positional bias","scientific evaluation","ACL Test-of-Time","Crystal-Bank"],"falsifier":"Run the identical three-shuffle majority-vote pipeline on a larger, multi-lab set of author-ranked reference lists and measure whether Spearman rank correlation with the human rankings stays above 0.7 and whether the accuracy gap over independent scoring remains positive.","tokens_in":26029,"feed_emoji":"📊","tokens_out":813,"duration_ms":8478,"temperature":0.7,"pith_summary":"Most citation-impact tools judge a single citation by looking only at the sentences that mention it. Crystal instead asks a language model to rank every reference that appears in the same paper at once, so each citation is judged relative to its neighbors. To blunt the models’ well-known preference for items that appear early or late in a list, the authors shuffle the order three times, run the ranking, and take a majority vote. On a public set of human-labeled citations the method lifts accuracy by roughly nine and a half points and F1 by eight points over the previous best system, while making far fewer model calls. When the same pipeline is run over tens of thousands of papers that cite classic ACL work, the papers that later won Test-of-Time awards sit near the top of Crystal’s high-impact tallies. The authors release both the code and a 46 800-paper bank of rankings so others can reuse the signal.","feed_headline":"Joint ranking lifts citation-impact accuracy by 9.5 points","feed_subtitle":"Three shuffled LLM passes beat isolated scoring and track Test-of-Time awards","key_machinery":"Crystal’s listwise ranking prompt that simultaneously assigns every reference in a citing paper to High / Medium / Low impact, executed three times under random order and aggregated by majority vote (or by reciprocal-rank fusion plus ordinal regression).","core_discovery":"Evaluating all of a paper’s citations jointly, rather than in isolation, supplies comparative context that lets an LLM separate truly indispensable references from background ones more accurately; three randomized ranking passes with majority vote further stabilize the labels against positional bias.","pith_inferences":["The same joint-ranking idea could be applied inside survey papers or grant proposals to surface the few references that actually drive the argument.","Temporal traces of high-impact citations may help predict which recent papers will still be cited a decade later.","Because the prompt forces an explicit High/Medium/Low partition, the method can serve as a cheap filter before expensive human review of citation importance."],"forward_implications":["Impact tallies can be computed for entire fields at O(number of papers) rather than O(number of citation edges), making large-scale analysis practical.","Open-weight models already match closed models on this task, so the method can be run without proprietary APIs.","High-impact citation counts become a usable, time-resolved signal that can be compared against awards, novelty scores, or retrieval benchmarks.","The released 46.8 k-paper bank supplies ready-made training and evaluation data for future impact models."],"fun_headline_variants":["Joint LLM ranking lifts citation-impact accuracy 9.5 points","Three shuffled ranks beat isolated scoring for key citations","Crystal jointly ranks all cites to separate true impact","Relative ranking outs indispensable papers over background","Majority-vote joint ranks track Test-of-Time awards closely"],"cache_read_input_tokens":7296,"weakest_assumption_plain":"That the three-level impact definitions written into the prompt, plus a three-shuffle majority vote, produce labels that line up with what humans mean by “impact,” even though the gold labels are only binary and the pilot check used only six co-authors from the same lab.","fun_headline_variants_meta":{"raw":{"variants":["Joint LLM ranking lifts citation-impact accuracy 9.5 points","Three shuffled ranks beat isolated scoring for key citations","Crystal jointly ranks all cites to separate true impact","Relative ranking outs indispensable papers over background","Majority-vote joint ranks track Test-of-Time awards closely"]},"model":"grok-4.5","effort":"low","cost_usd":0.004764,"raw_usage":{"total_tokens":1344,"prompt_tokens":729,"num_sources_used":0,"completion_tokens":61,"cost_in_usd_ticks":47640000,"prompt_tokens_details":{"text_tokens":729,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":554,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":729,"tokens_out":61,"duration_ms":5293,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-13T18:48:29.412175+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Run the identical three-shuffle majority-vote pipeline on a larger, multi-lab set of author-ranked reference lists and measure whether Spearman rank correlation with the human rankings stays above 0.7 and whether the accuracy gap over independent scoring remains positive.","supporting_citations":[],"review_version":2}