{"id":"ee6dad22-5950-4986-b744-8038cfe3732f","arxiv_id":"2501.10035","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An open-source pipeline maps millions of French publications into interactive community networks by pre-filtering the strongest co-occurrence links and labeling clusters with an LLM.","lead":"French researchers built a web tool that automatically draws network maps of scientific collaborations from a national publication database, using link filtering to keep maps small and readable. The tool is made to help research administrators see which labs and topics cluster together at a glance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global top-K link selection can miss entire low-frequency communities, not just isolated nodes; no coverage validation is provided.","rationale":"The reader's conditional verdict is appropriate. I looked for the place where the central claim most depends on an unvalidated step and found it in the global top-2000 link selection plus 300-node pruning: the pipeline can systematically omit coherent low-frequency communities, and no evaluation measures this. This is closely related to, but more general than, the reader's isolated-node caveat: even connected communities vanish if their dyad counts fall below a global cutoff. A concrete full-graph comparison on a medium corpus would settle it. The open-source code and reproducible infrastructure are real strengths, but they do not by themselves demonstrate that the filtered map represents the scientific community structure. Thus the conditional verdict remains, with the validation requirement now more specific.","tokens_in":6132,"tokens_out":4174,"duration_ms":44265,"concrete_test":"On a medium-sized perimeter (e.g., all publications of one large research institution where the full graph is computable), build the complete co-topic graph, run Louvain to define reference communities, then run the scanR pipeline with its default top-2000 links and 300-node cap. Measure the fraction of reference communities of small-to-mid size (say 10-100 publications) that appear as nodes or as recognizable communities in the pipeline output, and repeat for top-K = 500, 2000, 5000. If recall of small communities increases sharply with K, the default cutoff is dropping them; if small communities are already well represented, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 aggregates the 2000 most frequent co-occurrence pairs in the whole corpus, and Section 2.4 then prunes to at most 300 nodes using component size and betweenness centrality. This is a global frequency cutoff, not a structure-preserving reduction. In a heterogeneous 4M-publication corpus, a coherent but small research community can have all of its internal dyad counts below the top-2000 threshold, so it never enters the graph even though its authors are not isolated nodes; the paper's own caveat about isolated authors in Section 2.1 is a special case of this broader coverage bias. Since the advertised value is strategic decision-making, silently dropping niche or emerging fields is exactly the kind of blind spot that matters for policy. The paper provides no benchmark showing how well the top-K link selection and 300-node pruning recover the community structure of the full graph, so the central claim of 'mapping scientific communities at scale' is not yet supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a methodology and production web tool for mapping scientific communities from the scanR corpus of roughly four million French publications. The pipeline extracts the most frequent co-occurrence pairs (default top 2000) via Elasticsearch aggregations, builds a graph with Graphology, prunes it to at most 300 nodes using component size and betweenness centrality, applies ForceAtlas2 for layout and Louvain for community detection, labels communities with the Mistral NeMo LLM, and enriches the results with OpenAlex citation counts to flag hot topics. The authors claim this approach overcomes the limitations of node-threshold filtering and supports both national-scale maps and custom perimeters for individual institutions. The paper is a system description: it documents each step, gives a small example of LLM-based community naming, and points to an open-source repository and a live online tool.","tokens_in":6359,"tokens_out":4089,"duration_ms":39869,"significance":"If validated, this would be a practically useful, reproducible, open-source instrument for science policy: it combines PID-based disambiguation, Elasticsearch aggregation, graph algorithms, and LLM labeling in a single deployment, and the iframe-based custom-perimeter feature is a concrete service for French research institutions. The manuscript honestly discloses several limitations, including the isolation assumption and the desirability of a Louvain/Leiden benchmark. However, as a methods paper its significance is currently limited by the complete absence of quantitative evaluation of map quality, coverage, scalability, or the reliability of the LLM labels and citation-based hotspot scores.","major_comments":[{"comment":"The central claim that the pipeline maps scientific communities at scale is not yet supported by any quantitative evaluation. Keeping the top 2000 links globally and then pruning to at most 300 nodes by component size and betweenness is a frequency-based reduction, not a structure-preserving one, and no benchmark shows how well the resulting graph recovers the communities or node set of the full network. The paper itself acknowledges in §2.4 that 'a benchmark... would be desirable.' Please add a validation study, for example comparing the reduced graph against the full graph on a sample perimeter, with metrics such as node/community recall, adjusted Rand index, or modularity preservation, plus a sensitivity analysis of the 2000-link and 300-node thresholds.","section":"§2.3–2.4"},{"comment":"The assumption stated in §2.1 ('there are no large isolated nodes') is too narrow. A coherent but low-frequency community can have all its internal dyad counts fall below the global top-2000 threshold, so it will be absent from the map even though its authors are not isolated nodes. This coverage bias is directly relevant to the stated policy application and to the custom-perimeter use case in §3.2, where a small laboratory's community could be almost entirely omitted. Please provide coverage diagnostics, such as the fraction of entities and disciplines retained as a function of the link threshold, and how this fraction varies across institution sizes or disciplines.","section":"§2.1, §2.3"},{"comment":"The LLM-based community labeling is presented as a key feature, but there is no evaluation of whether the generated names are faithful, unique, or robust to noisy topic lists, nor is there an error analysis for empty or near-empty lists. Similarly, the citation-based hot-topic score is introduced without validation; given the paper's own caveat that OpenAlex citations are incomplete (Alperin et al. 2024), the authors should report how sensitive hotspot detection is to citation coverage, and ideally include a small human or topic-based evaluation of a sample of community labels.","section":"§3.1 (LLM labeling and hot topics)"},{"comment":"The 'at scale' claim lacks supporting measurements. The paper states that Elasticsearch aggregations are 'very efficient' and that the top-2000 limit ensures 'optimal performance,' but no runtime, memory, or scaling data are reported for corpora of different sizes (national corpus versus a single-laboratory perimeter). Please add latency and throughput measurements for representative perimeters, and if possible a scaling plot showing how query time and graph-construction time grow with corpus size.","section":"Title, §2.3"}],"minor_comments":[{"comment":"There are typos in algorithm and tool names: 'Force Atltas2' should be 'ForceAtlas2' and 'infered' should be 'inferred'; also 'vizualization' should be 'visualization' in the abstract and keywords.","section":"Abstract, §2.4"},{"comment":"The section numbering is duplicated: both the LLM labeling subsection and the citation/hot-topic subsection are numbered '3.1.' Renumber the second one as §3.2 and adjust the following custom-perimeter section accordingly.","section":"§3.1"},{"comment":"The example aggregation output shows two entries for 'carbon sequestration---Q623###carbon' with doc_counts 14 and 7 that differ only in capitalization; clarify whether topics are case-normalized before aggregation, since inconsistent normalization would inflate some link weights.","section":"§2.3"},{"comment":"Figure 1 is referenced with subfigures (a)–(d) but does not appear in the version under review; please ensure the figure is included in the final submission.","section":"Figure 1"},{"comment":"The ForceAtlas2 reference has a malformed author field ('Jacomy, Tommaso AND Heymann, Mathieu AND Venturini'); it should list Jacomy, Venturini, Heymann, and Bastian. Also, the Alperin et al. reference is an arXiv preprint; a stable journal or edited-volume version would be preferable if available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main barrier to publication is the absence of any validation or benchmark, which makes the paper read more like a system demonstration than a fully supported methods contribution. Since the code is open source and a live tool is available, adding the requested experiments within the scope of a revision is feasible. I would also ask the authors to frame the novelty carefully: the individual components are mostly existing tools, and the contribution is the integrated, at-scale engineering and the custom-perimeter service, which is valuable but should be presented as such."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is an engineering description of a practical pipeline for mapping scientific communities from the French scanR corpus. The real contribution is the link-first filtering strategy: precomputing pairwise co-occurrence links per publication, aggregating the strongest ones with Elasticsearch, and building a network capped at 300 nodes. That is a sensible way to make a multi-million-publication corpus interactive, and the open-source code plus live tool mean the work is reproducible and immediately usable. The LLM labeling of communities is a small but genuinely useful addition. Those assets are real.\n\nThe soft spot is not that all components are pre-existing—that is normal for a system paper—but that nothing in the paper shows the maps actually correspond to the communities they claim to map. The stress-test note is right: the global top-K link selection is a frequency cutoff, not a structure-preserving reduction. A coherent but low-frequency community can have all its internal dyads below the top-2000 threshold and vanish from the graph entirely, even though its authors are well connected to each other. The paper's own caveat about isolated nodes in Section 2.1 is just the extreme case of this broader coverage bias. Since the advertised value is strategic decision-making for science policy, silently dropping niche or emerging fields is exactly the kind of blind spot that matters. There is no benchmark against a known structure, no comparison with backbone extraction methods like disparity filtering, no sensitivity analysis for the top-2000, 300-node, or betweenness thresholds, and no runtime scaling plot. The central claim is therefore not yet evidenced.\n\nThe parameter choices look arbitrary: the paper says \"by default\" for 2000 links and 300 nodes but never studies how the results change. The citation-hotspot measure is also only loosely described. I agree with the reader that there is no circularity problem, because the LLM labeling is descriptive and involves no fitted parameters. The paper is honest about its assumptions and even flags the missing Louvain/Leiden benchmark, which speaks well of the authors' judgment.\n\nWho is this for? Practitioners building research-intelligence portals, and bibliometricians who want a working reference implementation. As a journal article it is thin on evaluation; as a software paper with a validation section added, it would be publishable. I would send it to peer review if the venue accepts application papers and the authors are pushed to add a benchmark, a coverage analysis, and a sensitivity study. Without that, it is a useful technical report but not a supported scientific claim.","headline":"A useful open-source tool for mapping bibliometric networks, but the paper lacks the evaluation needed to back its central claim of mapping communities at scale.","tokens_in":6831,"tokens_out":2069,"would_cite":false,"duration_ms":21217,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes that the bottleneck in mapping large scientific corpora is link selection rather than node count: keeping only the strongest co-occurrence links yields interpretable community maps from millions of publications.","keywords":["scientific community mapping","co-authorship networks","link filtering","Elasticsearch aggregation","Louvain algorithm","ForceAtlas2","bibliometric metadata enrichment","scanR"],"falsifier":"Take a corpus with a known community of solo or near-solo researchers, run the link-extraction step, and count how many authors from that community appear in no retained edge; if the fraction is large, the map's community structure is missing a real population. A simpler observable check is to compare the community partition obtained from the top 2000 links with the partition obtained from all links on a smaller corpus where full computation is feasible.","tokens_in":5940,"feed_emoji":"🗺️","tokens_out":9521,"duration_ms":83986,"temperature":0.7,"pith_summary":"This paper aims to make network maps of scientific communities from very large bibliographic corpora, the kind a national research system produces, instead of only from small curated samples. Its central idea is to reduce the graph by filtering links rather than nodes, keeping only the strongest co-occurrence interactions between entities such as authors, laboratories, topics, or funders. On a corpus of roughly four million publications, the authors argue, this link-first strategy avoids the constellation of isolated nodes that node-threshold methods produce. The resulting interactive maps are generated automatically, labeled by a large language model, and offered both nationally and for any institution that wants an embedded map of its own perimeter.","feed_headline":"Filter links, not nodes, to map science at scale","feed_subtitle":"Precomputed co-occurrence pairs turn millions of publications into interactive community maps.","key_machinery":"The load-bearing mechanism is link-centric filtering via precomputed pairwise co-occurrence fields. At indexing time, each publication carries fields such as co_topics or co_authors listing every pair of entities that appear together in that publication; an aggregation step then counts these pairs across the whole corpus and returns the top 2000 links. The graph is built from those links, with edge weight equal to the pair's document count, then reduced to at most 300 nodes by keeping the main component and pruning by betweenness centrality. Spatialization with ForceAtlas2 and community detection with Louvain turn this reduced graph into the final map, while a large-language-model prompt converts each community's weighted topic list into a short label.","core_discovery":"The paper's claim is that the scientific-community structure of a country can be mapped at web scale by concentrating on the strongest pairwise interactions in the data. Instead of dropping authors or institutions that publish little, which destroys the paths that connect the network, the pipeline drops weak links: for each publication it precomputes every pair of entities that co-occur, aggregates these pairs across the whole corpus, and keeps only the most frequent pairs. Those retained edges define the graph, and the graph is then cut down to a readable size by keeping the main connected component and the highest-betweenness bridge nodes. The authors report that this method, implemented with an aggregation engine for counting pairs, a graph library for construction and layout, and a community-detection algorithm, yields maps of research collaboration and thematic structure in which communities are visible and can be given automatic names and citation-based hotspot scores.","pith_inferences":["Because the paper leaves the maps unbenchmarked, a testable extension is to compare link-filtered community structures with curated disciplinary classifications.","Because only frequent pairs survive, slow-moving fields with long publication cycles or small author teams may be underrepresented in the same map where high-throughput fields dominate; per-field normalization of link weights would be a testable adjustment.","The same filter-links-not-nodes principle could be applied to citation or semantic-similarity links rather than co-occurrence, extending the method from collaboration structure to intellectual influence."],"forward_implications":["An institution can generate a map for its own publication list without redoing affiliation disambiguation, because the same precomputed links are simply filtered to that perimeter.","The method turns a corpus of millions of documents into a responsive interactive map, something node-threshold tools cannot do without losing connectivity.","Automated community labels and citation-based hotspot scores make the maps directly usable for exploring research fields and for funding or policy discussions.","Because 'strongest interaction' is defined as frequency of co-occurrence, any entity type stored with identifiers, such as authors, laboratories, topics, software, funders, or countries, can be mapped with the same pipeline."],"supporting_citations":[{"why":"Supplies the scanR portal and its enriched corpus of roughly four million French publications that the pipeline maps.","marker":"(Jeangirard 2024)"},{"why":"Documents the French Open Science Monitor corpus and its country-wide coverage, the underlying publication set.","marker":"(Chaignon and Egret 2022)"},{"why":"Provides the Elasticsearch-based affiliation matching method that links publications to institution and laboratory identifiers.","marker":"(L’Hôte and Jeangirard 2021)"},{"why":"Provides the entity-fishing disambiguation service used to attach topic identifiers to publications.","marker":"(Foppiano and Romary 2020)"},{"why":"Defines the Louvain modularity-optimization algorithm used to detect communities in the constructed graph.","marker":"(Blondel et al. 2008)"},{"why":"Supplies the ForceAtlas2 layout algorithm whose parameters are inferred from graph order for spatialization.","marker":"(Jacomy 2014)"},{"why":"Underlies the VOSviewer online visualization and the unified mapping-and-clustering approach the maps build on.","marker":"(Waltman, Eck, and Noyons 2010)"},{"why":"Documents the incompleteness of citation data in the external source used, and therefore the caution applied to the hot-topic citation scores.","marker":"(Alperin et al. 2024)"}],"fun_headline_variants":["Precomputed pairs reveal science communities at scale","Map research by keeping strong links, dropping weak ones","Web-scale community maps from co-occurrence pairs","From millions of publications to interactive community maps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that no substantial research community consists of entities with no strong co-publication links to others, because an entity with no connection simply never appears in the map.","fun_headline_variants_meta":{"raw":{"variants":["Precomputed pairs reveal science communities at scale","Map research by keeping strong links, dropping weak ones","Web-scale community maps from co-occurrence pairs","From millions of publications to interactive community maps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1413,"prompt_tokens":986,"completion_tokens":427,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":602,"tokens_out":427,"duration_ms":5156,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T19:22:36.374856+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a corpus with a known community of solo or near-solo researchers, run the link-extraction step, and count how many authors from that community appear in no retained edge; if the fraction is large, the map's community structure is missing a real population. A simpler observable check is to compare the community partition obtained from the top 2000 links with the partition obtained from all links on a smaller corpus where full computation is feasible.","supporting_citations":[],"review_version":1}