{"id":"96fbaaa9-7b23-4ba6-8dbb-40dd10bfbf1b","arxiv_id":"2606.24099","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Constructs algorithm co-occurrence networks from full-text NLP papers and applies centrality measures to quantify group influence and temporal changes.","lead":"The paper builds co-occurrence networks of algorithms mentioned in NLP papers to study their collective influence over time using network metrics. A smart generalist might read it to see how bibliometric network methods can track trends in which AI techniques rise or fall in research.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Co-occurrence may reflect topic overlap or extraction noise rather than influence links","rationale":"The reader's weakest_assumption is exactly the load-bearing assumption; the abstract alone supplies no evidence that would falsify it, so the UNVERDICTED verdict is unaffected.","tokens_in":1735,"tokens_out":255,"duration_ms":14112,"concrete_test":"Sample 200 papers, have two domain experts independently label all algorithm mentions, compute precision/recall/F1 of the paper's extraction pipeline on that set; if F1 < 0.75, recompute the reported centrality rankings on the cleaned co-occurrence graph and check whether the 'classic algorithm' and 'decline' patterns survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claims (high centrality for classic algorithms, loss of core position preceding influence decline) rest on the untested premise that algorithm-name co-occurrences extracted by deep-learning models from full-text papers encode meaningful interconnections. Co-occurrences can instead arise from papers addressing the same sub-area, from citation conventions, or from NER false positives/negatives; the abstract provides no accuracy figures, inter-annotator agreement, or controls that would rule these out.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper extracts algorithm entities from the full text of NLP papers spanning more than four decades using deep learning models, constructs overall, cumulative, and annual co-occurrence networks, and analyzes their structural properties (e.g., density, complex-network features) together with multiple centrality measures to quantify collective algorithm influence. It reports that classic high-performing algorithms and those at research-period intersections exhibit high popularity, control, and balanced centrality, while declining influence is typically preceded by loss of core network position.","tokens_in":1808,"tokens_out":562,"duration_ms":19270,"significance":"If the extraction and co-occurrence steps can be shown to reliably encode influence rather than topic overlap or noise, the work supplies the first large-scale temporal network view of algorithm influence in NLP, covering four decades and linking structural position to popularity and decline patterns. The scale and the distinction among overall/cumulative/annual networks are genuine strengths.","major_comments":[{"comment":"Abstract and Methods: the construction of all reported networks rests on deep-learning entity extraction, yet no precision, recall, F1, validation-set performance, inter-annotator agreement, or error analysis is supplied; without these figures the centrality rankings and temporal claims cannot be evaluated.","section":"Abstract / Methods"},{"comment":"Results (centrality and decline analysis): the headline claim that 'when the influence of an algorithm declines, it usually loses its core network position first' is presented as an observational finding, but the manuscript provides no independent measure of influence (e.g., citation counts, task adoption) against which network position can be validated, leaving open the possibility that co-occurrence simply tracks shared sub-topics.","section":"Results"},{"comment":"Results / Discussion: no sensitivity analysis or controls are described for co-occurrence threshold choice, NER false-positive rate, or alternative explanations such as citation conventions; these choices directly affect the reported 'increasingly dense connections' and the identification of 'core positions.'","section":"Results / Discussion"}],"minor_comments":[{"comment":"The abstract states results on 'balanced influence' without defining the term or the exact centrality combination used; a short operational definition would improve clarity.","section":"Abstract"},{"comment":"Figure captions and network-visualization panels should explicitly state the time windows and edge-weighting rule employed for each cumulative/annual snapshot.","section":"Figures"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's fit to cs.AI is reasonable, but the absence of any quantitative validation of the extraction pipeline is a reproducibility concern that should be addressed before acceptance."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed comments. We address each major point below and indicate the revisions we will make to strengthen the manuscript.","responses":[{"response":"We agree that the absence of quantitative validation for the entity extraction model is a limitation. In the revised manuscript we will add a Methods subsection reporting precision, recall, and F1 on a held-out validation set, together with an error analysis of false positives and negatives. This will allow readers to assess the reliability of the extracted entities before interpreting the network results.","revision_made":"yes","referee_comment":"[Abstract / Methods] Abstract and Methods: the construction of all reported networks rests on deep-learning entity extraction, yet no precision, recall, F1, validation-set performance, inter-annotator agreement, or error analysis is supplied; without these figures the centrality rankings and temporal claims cannot be evaluated."},{"response":"The reported patterns are observational and derived from the temporal evolution of network position. We accept that external validation would increase confidence. In revision we will add a supplementary analysis correlating network centrality with citation counts for a representative sample of algorithms. At the same time, the multi-network design (overall, cumulative, and annual) and the observed complex-network properties provide evidence that the co-occurrence relations capture more than simple topical overlap.","revision_made":"partial","referee_comment":"[Results] Results (centrality and decline analysis): the headline claim that 'when the influence of an algorithm declines, it usually loses its core network position first' is presented as an observational finding, but the manuscript provides no independent measure of influence (e.g., citation counts, task adoption) against which network position can be validated, leaving open the possibility that co-occurrence simply tracks shared sub-topics."},{"response":"We acknowledge that sensitivity checks and discussion of alternative explanations are missing. The revised version will include (i) sensitivity analyses varying the co-occurrence threshold and (ii) an assessment of how NER false-positive rates could affect density and core-position findings. We will also add a brief discussion of citation conventions as a potential confounder and explain how the annual-network construction helps isolate genuine temporal shifts.","revision_made":"yes","referee_comment":"[Results / Discussion] Results / Discussion: no sensitivity analysis or controls are described for co-occurrence threshold choice, NER false-positive rate, or alternative explanations such as citation conventions; these choices directly affect the reported 'increasingly dense connections' and the identification of 'core positions.'"}],"tokens_in":1425,"tokens_out":548,"duration_ms":21494,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper extracts algorithm entities from full-text NLP papers using deep learning, then builds overall, cumulative, and yearly co-occurrence networks to measure centrality and track changes. It covers more than four decades and reports that the networks grow denser, that classic algorithms sit at high-centrality positions, and that declining influence often shows up first as loss of core connections.\n\nWhat stands out is the temporal scale and the shift from single-algorithm counts to network views. That descriptive step is straightforward and the patterns they describe (denser links over time, intersection algorithms staying central) are the sort of observations that can feed later work on trend monitoring.\n\nThe soft spot is the missing validation. The abstract and claims rest on the untested idea that co-occurrence of extracted names equals meaningful influence links. No accuracy numbers for the entity extraction, no inter-annotator checks, no controls for papers that simply share a sub-topic, and no error bars on the centrality scores. Without those, the network statistics could be driven by extraction noise or citation habits rather than the influence the authors want to measure.\n\nThis is incremental bibliometric work rather than a new theoretical result. It will mainly interest people already doing network analyses of scientific fields or tracking algorithm adoption in NLP. The central argument holds up as a set of observations but would need the validation steps tightened before it supports stronger claims about influence decline.\n\nI would send it to peer review so referees can press on the extraction quality and possible confounds; the data scale alone makes it worth a look from that angle.","headline":"They build co-occurrence networks from algorithm names in NLP papers over four decades but give no validation that the extraction or links actually track influence instead of topic overlap.","tokens_in":2287,"tokens_out":390,"would_cite":false,"duration_ms":13857,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Co-occurrence networks from full-text papers show classic algorithms at research-period intersections hold high centrality and balanced influence.","keywords":["algorithm co-occurrence","network analysis","academic influence","natural language processing","entity extraction","centrality measures","temporal networks","full-text mining"],"falsifier":"A longitudinal check showing that an algorithm's mention frequency and independent usage metrics (such as benchmark adoption) continue to rise after it has already lost its top centrality rank in the co-occurrence network would falsify the claim that core-position loss precedes influence decline.","tokens_in":2645,"feed_emoji":"🔗","tokens_out":665,"duration_ms":13068,"temperature":0.7,"pith_summary":"The paper builds large-scale co-occurrence networks of algorithm names extracted from the full text of NLP papers spanning more than four decades. These networks exhibit the properties of complex networks and grow denser over roughly two decades. Classic high-performing algorithms and those appearing at the boundaries between successive research periods display elevated popularity, control, and multiple centrality scores while maintaining balanced influence. When an algorithm's influence wanes, its loss of core network position precedes the weakening of its associations with other algorithms.","feed_headline":"Co-occurrence networks show classics at era intersections keep high influence","feed_subtitle":"NLP papers over four decades reveal that algorithms bridging research periods retain core positions and balanced centrality longer than othe","key_machinery":"Algorithm co-occurrence networks extracted via deep learning entity recognition from full-text papers, then analyzed with multiple centrality measures across overall, cumulative, and yearly snapshots.","core_discovery":"By constructing overall, cumulative, and annual co-occurrence networks of algorithms from full-text academic papers in natural language processing, the study establishes that these networks display typical features of complex networks with increasingly dense connections over time, and that classic, high-performing algorithms located at the intersections of different research periods exhibit high popularity, control, centrality, and balanced influence, while declining influence is marked first by loss of core network position followed by weaker associations.","pith_inferences":["The same extraction and centrality pipeline could be applied to track influence shifts in other scientific domains beyond NLP.","Early detection of centrality loss might serve as a leading indicator for when an algorithm is about to lose practical adoption.","Combining these networks with author or task co-occurrence graphs could reveal how scholars and problems drive algorithmic change.","The approach supplies a structural baseline against which future claims of algorithmic impact can be compared without relying solely on citation counts."],"forward_implications":["Classic high-performing algorithms maintain elevated centrality scores across the entire field and across successive time windows.","Algorithms positioned at the boundaries between research periods display more balanced influence than those confined to single eras.","Loss of influence for any algorithm first appears as a drop in core network position before its links to other algorithms weaken.","The networks become progressively denser, reflecting growing interconnections among algorithms over two decades."],"fun_headline_variants":["Co-occurrence networks link classics to high centrality at era intersections","Classics at NLP research era intersections show highest network centrality","NLP algorithm networks place classics at period intersections as most central","Co-occurrence shows classics bridging research eras with balanced centrality"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That the co-occurrence of algorithm names within papers reliably signals meaningful influence connections instead of shared topics, citation practices, or extraction errors.","fun_headline_variants_meta":{"raw":{"variants":["Co-occurrence networks link classics to high centrality at era intersections","Classics at NLP research era intersections show highest network centrality","NLP algorithm networks place classics at period intersections as most central","Co-occurrence shows classics bridging research eras with balanced centrality"]},"model":"grok-4.3","cost_usd":0.011116,"raw_usage":{"total_tokens":4901,"prompt_tokens":694,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":111162000,"prompt_tokens_details":{"text_tokens":694,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":4150,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":694,"tokens_out":57,"duration_ms":30311,"temperature":1.0,"reasoning_tokens":4150,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T00:34:44.017210+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A longitudinal check showing that an algorithm's mention frequency and independent usage metrics (such as benchmark adoption) continue to rise after it has already lost its top centrality rank in the co-occurrence network would falsify the claim that core-position loss precedes influence decline.","supporting_citations":[],"review_version":1}