REVIEW 4 major objections 6 minor 8 references
Extracting Information About Publication Venues Using Citation-Informed Transformers
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Venue similarity can be read off from SPECTER embedding distributions: ACL, EMNLP and NAACL are indistinguishable, and some venues converge over 2015–2023.
desk verdict The paper has a genuine new application and a plausible qualitative result, but its variance-weighted KL metric is underspecified and the convergence trend is not yet supported by the numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is SPECTER, a citation-informed transformer encoder that maps a title and abstract into a 768-dimensional vector, paired with a variance-weighted KL divergence as the similarity metric. After PCA reduction to 64 components retaining 82.5% of the variance, the paper computes the KL divergence of Equation 1 between two venues for each component separately, weights each component by its share of the retained variance, and sums the weighted terms; a logistic regression classifier trained on each year's venue pair serves as a second probe of the same question. The identity doing the work is $D_{KL}(P\parallel Q)=\int p(x)\log(p(x)/q(x))\,dx$, with per-feature values summed under variance-based weights rather than estimated jointly.
What would settle it
Recompute pairwise venue distances with a joint multivariate KL estimator on the 64 PCA components, for example a k-nearest-neighbour divergence estimator, and test whether ACL, EMNLP, and NAACL still collapse into one cluster and whether AAAI's year-over-year approach to NLP venues persists; if the cluster or trend disappears, the reported findings depend on the independence assumption behind the variance-weighted sum.
Extended reading notes
Core claim
The central claim is that pairwise similarity between publication venues can be measured by the distributions of the document embeddings they publish, and that under this measure venue boundaries are already blurring. Using a variance-weighted KL divergence computed on the 64 principal components retained from SPECTER embeddings, which keep 82.5% of the variance, the authors find that the NLP venues ACL, EMNLP, and NAACL are effectively co-located, that the programming-language venues OOPSLA and POPL are co-located with each other but distinct from everyone else, and that the large general venues AAAI, NeurIPS, and CHI occupy broad regions. Year-over-year comparisons show AAAI's distribution approaching the NLP venues, and ACL's distribution approaching USS and CHI. Logistic regression validation accuracy corroborates the pattern: classifiers find it hard to tell some venue pairs apart.
Load-bearing premise
The load-bearing premise is that the 64 PCA features can be treated as independent so that the variance-weighted sum of per-feature KL divergences is a valid aggregate divergence, even though PCA only ensures uncorrelatedness and the densities in Equation 1 are never defined from the finite samples.
Editorial extensions
If this is right
- Venue similarity can be mapped automatically from the papers themselves, without hand-engineered features or citation-network analytics, using pre-trained embeddings.
- For venues in the same cluster, such as ACL, EMNLP, and NAACL, a paper's topical fit at one venue is statistically interchangeable with fit at the others.
- If the convergence trend continues, AAAI and NLP venues will become increasingly interchangeable in embedding space, which the authors read as a loss of thematic identity that could justify venue splits or reorganization.
- The same distribution-distance pipeline can be applied year by year to monitor venue health, and the released Mizzium Library offers a document-level map for finding interdisciplinary connections.
Reading between the lines
- The authors measure only accepted papers, so the convergence trend blends changes in what authors submit with changes in what venues accept; tracking the same divergences on submissions or preprints would separate the two.
- Because SPECTER is trained with citation links, some of the convergence could reflect growing citation overlap between fields rather than topical convergence in the texts; ablating the citation signal would test this.
- A practical extension is to use the pairwise divergences to flag papers that fall into the overlap region between two venues as interdisciplinary candidates, which is a short step from the Mizzium Library interface.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper collects approximately 58,749 papers from nine computer science venues (2015–2023), encodes each title and abstract with the SPECTER citation-informed transformer, reduces the 768-dimensional embeddings to 64 PCA components, and defines a venue-similarity metric as a variance-weighted sum of per-feature Kullback-Leibler divergences. It reports that the three NLP venues (ACL, EMNLP, NAACL) are nearly indistinguishable in this embedding space, that OOPSLA and POPL are co-located, and that several venue pairs (e.g., AAAI with NLP venues, ACL with USS and CHI) converge over time. The year-over-year analysis is supplemented by logistic regression validation accuracy on yearly venue-pair document sets, and the authors release a visualization tool, Mizzium Library.
Significance. If the core claims hold, the paper contributes a practical, low-cost way to monitor thematic identity and convergence of publication venues, together with a publicly released dataset and an interactive tool. The use of a state-of-the-art citation-informed encoder and the direct comparison with a supervised classifier are valuable design choices. However, the central quantitative claim—venue convergence over time—rests on a similarity metric whose validity is not established: the aggregation of per-feature KL divergences is not a joint KL divergence unless independence holds, and the density estimates underlying Equation (1) are never specified. The corroborating logistic regression lacks uncertainty intervals and does not address class imbalance. These issues make the convergence trend suggestive rather than demonstrated, though the qualitative finding of co-located NLP venues is plausible and consistent with commonsense.
major comments (4)
- [§2.2.1, Eq. (1)] The metric defined in Section 2.2.1 is presented as a Kullback-Leibler divergence between venues, but the actual computation is a variance-weighted sum of per-feature KL divergences over 64 PCA components. This equals a joint KL divergence only if the 64 features are statistically independent. Principal component analysis guarantees only uncorrelatedness, not independence; dependent features can make the summed marginal divergences arbitrarily different from the true joint divergence. The manuscript should either justify the independence assumption, test it, or explicitly rename the quantity (e.g., 'variance-weighted marginal KL score') and discuss what interpretative weight it carries. In addition, Equation (1) is a continuous integral over densities p(x) and q(x), but the paper never states how these densities are estimated from finite document samples—histograms, kernel density estimation, or parametric fits are not described. Without a density estimator, the numerical values in Figures 1 and 2 are not reproducible, and the convergence trend in Figure 2 could be an artifact of the estimator.
- [§3.2, Figures 2 and 3] The central year-over-year claim of convergence is supported only by plotted curves without any uncertainty quantification. The annual document counts vary substantially across venues (from hundreds to tens of thousands), and Figure 2 applies an unspecified 'within venue normalization' that changes the vertical scale and can visually exaggerate small absolute changes. The manuscript should provide bootstrap confidence intervals or other error bars for the KL scores, and a statistical test for monotone or systematic decrease over time. Without this, the decreasing trends for AAAI and ACL could fall within sampling noise, especially for small venues such as POPL and OOPSLA.
- [§2.2.2, Figure 3] The logistic regression corroboration in Section 2.2.2 and Figure 3 lacks essential details: the train/validation split, hyperparameters, regularization, and the treatment of class imbalance. Venue sizes range from 531 (POPL) to 13,884 (NeurIPS), so a classifier can achieve high validation accuracy simply by predicting the majority class. Low validation accuracy is only interpretable as evidence of distributional overlap if the evaluation uses balanced sampling or balanced metrics (e.g., balanced accuracy, precision/recall). The paper should report these details and, ideally, confidence intervals for the validation accuracy, because the apparent convergence in Figure 3 is used to 'consolidate' the KL-based claim.
- [§2.1, §4] The paper does not address a form of circularity arising from the choice of SPECTER, a citation-informed encoder. SPECTER is pretrained on citation links, and citation links are strongly correlated with venue membership (papers in the same venue often cite one another, and venue-specific citation patterns are common in the pretraining graph). The observed co-location of NLP venues, or of OOPSLA and POPL, could be inherited from the citation graph rather than discovered from the textual content of abstracts. The authors should either acknowledge this as an intended property of the analysis (thereby changing the claim to 'citation-informed embedding distributions are separable by venue') or run a control with an encoder that does not use citation information (e.g., SciBERT or a plain transformer) to determine how much of the similarity is text-based.
minor comments (6)
- [Table 1 and Figures 1–3] The venue is called 'NeurIPS' in Table 1 but 'NIPS' in the figure axes and legends; the notation should be consistent.
- [§2.1] There is a grammatical error: 'SPECTER can encoder titles' should be 'SPECTER can encode titles'.
- [§3.2] There are typos: 'steadly' should be 'steadily', and 'could be hypothetically be admitted' has a duplicated 'be'.
- [Figure 2 caption] The caption says 'We perform within venue normalization for readability,' but the normalization procedure is not described in the text or the caption; please specify how each curve is normalized.
- [§3.3] The claim that LLM-based writing assistants are making writing style more similar between venues is speculative and is not supported by any measurement in the paper; it should be clearly labeled as a hypothesis.
- [§3.1] The phrase 'indistinguishable' is used for venue pairs that have small but nonzero KL values; a threshold for 'indistinguishable' should be defined so the claim is falsifiable.
Circularity Check
No significant circularity: the paper defines a similarity metric over pre-trained embeddings and does not fit any parameter to the venue-similarity values it reports.
full rationale
The paper's derivation chain is self-contained. It uses SPECTER as an external, pre-trained encoder, applies PCA as an unsupervised dimensionality reduction, and then defines variance-weighted KL divergence between per-venue embedding distributions as a similarity metric. No parameter is fitted to the reported similarity values, no 'prediction' is constructed from fitted inputs, and no load-bearing claim is justified by self-citation. The KL metric is a definition, not a derived result, and the logistic-regression analysis is an independent validation method rather than a refit of the same quantity. The fact that SPECTER is citation-informed and that citation patterns correlate with venue membership is a potential confound, but the paper does not claim to isolate venue similarity from the training signal; it explicitly analyzes the embeddings produced by that model. Statistical concerns about the KL estimator's validity are correctness risks, not circularity. Therefore no specific circular step can be exhibited from the text, and the correct score is 0.
Assumptions & free parameters
free parameters (3)
- PCA target dimensionality =
64
- Variance-retention threshold =
82.5%
- KL density estimator settings =
unspecified
assumptions (4)
- domain assumption SPECTER embeddings capture meaningful semantic and scholarly similarity between documents.
- ad hoc to paper The 64 PCA components are independent enough that summing per-feature KL divergences gives a valid aggregate divergence.
- domain assumption Semantic Scholar abstracts are representative of each venue's accepted papers.
- domain assumption Low logistic regression validation accuracy reflects thematic overlap between venues.
Cite this review
Pith. "Pith review of Extracting Information About Publication Venues Using Citation-Informed Transformers." pith.science (2026). https://pith.science/paper/EABRB7HE
@misc{pith2026250608199,
author = {Pith},
title = {Pith review of: Extracting Information About Publication Venues Using Citation-Informed Transformers},
year = {2026},
howpublished = {\url{https://pith.science/paper/EABRB7HE}},
note = {Machine review of arXiv:2506.08199}
}
read the original abstract
Scientific document embeddings contain a variety of rich features which can be harnessed for downstream tasks such as recommendation, ranking, and clustering. We explore which tangible insights can be drawn from scientific document embeddings to understand trends in computer science research featured across nine well-known venues. We collect approximately 60,000 scientific documents published between 2015 and 2023 and analyze their embeddings, which we produce with the SPECTER pre-trained language model. In particular, we examine whether similarity between two venues can be measured using the embeddings of the scientific documents they admit for publication. Our findings indicate that some venues within computer science are indistinguishable when only considering the distributions of their document embeddings. We additionally examine whether any two venues are becoming increasingly similar over time and identify a trend of convergence within some venues in our analysis. We discuss the implications of these results and the potential impact on new scientific contributions.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, et al. 2018. Construction of the literature graph in semantic scholar. arXiv preprint arXiv:1805.02262
arXiv 2018
-
[4]
Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020. Specter: Document-level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180
arXiv 2020
-
[5]
Grettel Monteagudo Garc \' a. 2016. Analyzing, comparing and recommending conferences. Ph.D. thesis, Programa de P \'o s-Gradua c \ a o em Inform \'a tica of the Departamento de Inform \'a tica …
work page 2016
-
[6]
Yaakov HaCohen-Kerner, Avi Rosenfeld, Maor Tzidkani, and Daniel Nisim Cohen. 2013. Classifying papers from different computer science conferences. In Advanced Data Mining and Applications: 9th International Conference, ADMA 2013, Hangzhou, China, December 14-16, 2013, Proceedings, Part I 9, pages 529--541. Springer
work page 2013
-
[7]
Brent Mittelstadt, Chris Russell, and Sandra Wachter. 2019. Explaining explanations in ai. In Proceedings of the conference on fairness, accountability, and transparency, pages 279--288
work page 2019
-
[8]
Amanpreet Singh, Mike D'Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2022. Scirepeval: A multi-format benchmark for scientific document representations. arXiv preprint arXiv:2211.13308
arXiv 2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.