Pith. sign in

REVIEW 4 major objections 6 minor 31 references

ICLR peer-review scores do not predict which papers later redirect research trajectories.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 16:03 UTC pith:6TZFAFV5

load-bearing objection Solid large-scale ICLR study showing review scores are orthogonal to EDM-based trajectory change; the null is real at distribution level even if per-paper ranks are noisy. the 4 major comments →

arxiv 2607.05401 v1 pith:6TZFAFV5 submitted 2026-05-24 cs.DL cs.AI

Catalyst Papers in Artificial Intelligence Research: A Landscape on ICLR from 2017 to 2025

Fan Huang This is my paper
classification cs.DL cs.AI
keywords catalyst papersdisruptiveness measuresEDMICLR peer reviewtopic initiationtopic bridgingscience of sciencecitation embeddings
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

A handful of methods have redirected AI research, yet conference review still decides which submissions enter the record. This paper asks whether ICLR reviewer scores, available for every submission from 2017 to 2025, can identify those trajectory-changing papers at decision time. On 36,113 papers it defines catalysts as submissions whose later descendants measurably reorient topics or citation structure, compares four disruptiveness measures, and shows that a direction-aware embedding measure best recovers highly cited work. Topic-initiating and topic-bridging papers precede large subsequent growth in topic share and cross-topic flow. Review scores, however, correlate essentially at zero with future disruptiveness, and accepted and rejected papers look the same on that measure. The residual mismatch is structured by catalyst type and by topic rather than random noise.

Core claim

On the nine-year ICLR record, peer-review signals are essentially orthogonal to long-run trajectory change measured by direction-aware citation embeddings: Spearman correlations with EDM stay within |ρ| ≤ 0.005, accepted and rejected papers have indistinguishable mean EDM, and residual miscalibration concentrates on topic bridges and recognition-misaligned papers rather than on within-topic work.

What carries the argument

The Embedding Disruptiveness Measure (EDM): past and future vectors learned from direction-aware random walks on the ICLR-internal citation graph; disruptiveness is the cosine distance between those vectors, paired with a five-type operational catalyst taxonomy (topic initiator, topic bridge, within-topic redirector, simultaneous, recognition-misaligned).

Load-bearing premise

That an incomplete ICLR-only citation graph and a pipeline-sensitive embedding score adequately capture multi-generational research redirection for both accepted and rejected papers.

What would settle it

Replicate the same EDM-versus-reviewer-score analysis on a multi-venue corpus (or a denser citation graph that includes arXiv and non-ICLR citers) and find a substantial positive correlation between mean review score and future EDM, or a clear mean-EDM gap between accepted and rejected papers.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies 36,113 ICLR submissions (2017–2025) with public OpenReview scores, defining catalyst papers as those whose descendants redirect later research. It compares four disruptiveness measures (CD, node2vec, direction-aware EDM, and an LLM rater), introduces a five-type multi-label catalyst taxonomy (TI, TB, WR, SC, RM), and links catalysts to subsequent topic-share and cross-topic flow growth. EDM best recovers highly cited ICLR papers (ERS AUC 0.83 vs 0.60/0.49/0.42). Topic initiators and bridges precede large multiplicative growth versus year-matched (and propensity-matched) controls. The headline recognition result is that reviewer scores are essentially orthogonal to EDM-based disruptiveness (|ρ|≤0.005; accepted vs rejected mean EDM indistinguishable, p=0.11), with residual miscalibration structured by catalyst type and topic.

Significance. If the results hold, this is a substantial science-of-science contribution for ML/NLP: it is the first corpus-scale pairing of per-paper OpenReview signals with a direction-aware, multi-generational trajectory measure, and the near-null review–disruption relationship is both surprising and programmatically relevant. Strengths include the head-to-head measure comparison on a common venue, year- and propensity-matched mechanism tests, threshold sweeps for TI/TB, explicit documentation of EDM pipeline sensitivity (Appendix C), and a portable operational taxonomy. The work is carefully instrumented (Firth logistic ORs, Mann–Whitney/Welch tests, multi-vendor LLM agreement) and does not overclaim peer-review “failure.” These are real assets for a cs.DL / science-of-science audience.

major comments (4)
  1. §7.1 / Table 4 and the abstract’s strongest claim (review–EDM orthogonality, |ρ|≤0.005; accepted vs rejected mean EDM p=0.11) rest on EDM as the sole trajectory outcome. Appendix C reports that sequential vs parallel walk generation at matched hyperparameters yields cross-pipeline Spearman ρ ≈ −0.145 for per-paper ranks, while only distribution-level AUC is stable. Because review-gap analyses (§7.2, Table 14) and top-decile membership use EDM percentiles/ranks, the null result needs an explicit robustness check under the multi-seed median-rank ensemble (or a second pipeline) already described in Appendix C: recompute Spearman ρ, Firth ORs, and accepted/rejected mean-Δ tests on ensemble EDM. Without that, the headline claim is only as strong as a pipeline-sensitive ranking.
  2. §3 and Appendix A: the citation graph is ICLR-internal with a 77.1% S2 match rate; unmatched papers are concentrated among rejected and recent submissions. The accepted vs rejected EDM comparison (n_acc=8,586; n_rej=13,716 with Δ) and the claim that rejected papers are slightly over-represented in the top EDM decile therefore condition on being matched and having enough structure for EDM. Please quantify how match failure and 2024–2025 sparsity affect the null: e.g., sensitivity restricted to 2017–2022 cohorts with high match rates, or bounds under alternative external-citation graphs for reappearing rejects. This is load-bearing for RQ3’s interpretation that gatekeeping is orthogonal to trajectory change rather than that unmatched rejects are simply unobserved.
  3. §4.1 / §6 and Tables 8–10: catalyst labels are threshold-defined (TI share growth ≥2.0×; TB top 10% flow and D_i≥2; WR above cluster-median centroid shift; RM Δ≥90th pct plus review boundary). Appendix H shows useful TI/TB sensitivity, but the multi-label co-occurrence claims (e.g., 36.6% of RM also TB; union 22.2%) and the “TB is the largest mechanism” ranking should be reported under the same threshold grid used for growth ratios, not only at the canonical cut. Otherwise the taxonomy risks reading as free-parameter-dependent rather than mechanism-stable.
  4. §5.1 / Tables 1–2 and Appendix E: LAS has n=50 with only 9 union positives and run-to-run κ=0.291 for the LLM judge. The paper correctly frames LAS as cross-model semantic-rubric agreement, not human gold, but still uses LAS AUC/OR to position M4 against EDM. With 9 positives, the M4 LAS OR 1.41 (p=0.03) and the “complementarity” narrative are under-powered. Either enlarge LAS (or add a small human-annotated subset) or demote LAS-based ranking claims in the main text and keep M4 as an exploratory content-only baseline validated mainly by inter-LLM agreement (Table 3).
minor comments (6)
  1. Fig. 1 and §5: state clearly in the main text that EDM covers 62% of papers while CD/node2vec cover 35%/18%; readers can otherwise over-read head-to-head AUCs as same-support comparisons.
  2. Eq. (3) and Appendix B: define past/future vectors and the single-side skip-gram objective earlier in §4.2 so that Δ_i is self-contained without jumping to the appendix for the training loss.
  3. §6.2 simultaneous-discovery funnel: the drop from 483,809 to 162 candidates is important; a one-sentence main-text note that raw cosine≥0.9 is dominated by sparse-neighborhood artifacts would prevent over-reading the rarity claim relative to Kim et al. (2026).
  4. Table 15 / topic bias: clarify whether topic gaps are residualized on year and acceptance; trendy topics (diffusion, ViT) may also differ in citation half-life, which could partially drive EDM percentiles.
  5. References and related work: Tran et al. (2020) is cited for score–citation correlation; a brief quantitative contrast (their weak correlation vs your near-zero score–EDM ρ) in §2 or §7 would sharpen the novelty claim.
  6. Ethics / §8: the caution against using EDM as a direct acceptance input is well taken; consider stating explicitly that ensemble ranks, not single-pipeline ranks, would be the minimum if anyone attempted operational use.

Circularity Check

3 steps flagged

TI and RM catalyst labels are defined from the same post-publication growth ratios and EDM–score gaps later reported as their effects, so those magnitude claims are partly tautological by construction; the orthogonality result itself is not.

specific steps
  1. self definitional [§4.1 TI definition + §6.1 / Table 9]
    "Paper i is a TI if s̄post_i / s̄pre_i ≥ 2.0. … TI papers precede topic-share growth at 7.55 imes the rate of matched controls (Table 9; … Mean growth … TI papers 5.91 … Year-matched controls 0.78 … Ratio of means —7.55 imes—)."

    The TI set is defined exactly as those papers whose own topic-share growth ratio meets or exceeds 2.0. Therefore every member of the set has growth ≥2 by construction; reporting that the set’s mean growth is 5.91 (and 7.55 imes controls) simply restates the selection criterion plus a comparison against the complement. The language “precede … growth” presents a definitional selection as an empirical effect.

  2. self definitional [§4.1 RM definition + §7.2 / Table 14]
    "Paper i is an RM if (i) Δi is at or above the 90th percentile of the full-corpus EDM distribution (Δ≥0.899); and (ii) its reviewer mean score s̄i is at or below the year-conditional acceptance boundary minus 0.5 points … RM papers show a mean gap of +0.59 (t=85.75, p<0.001)."

    The review gap is defined as EDM percentile minus reviewer-score percentile. RM membership requires high EDM percentile and low score percentile; the large positive mean gap for the RM class is therefore guaranteed by the membership rule itself, not an independent discovery.

  3. other [§4.3 / §5.1 ERS definition + Table 1]
    "External Recognition Set (ERS), the top 2% by ICLR-internal citation count (n=739 positives) … The citation-count baseline is excluded from ERS because it is definitionally circular. … EDM achieves ERS AUC 0.827."

    ERS is pure citation count on the same ICLR-internal graph from which EDM’s past/future vectors are learned. While not strictly definitional (EDM is a directional embedding distance, not the count), the shared information source makes high AUC partly expected; the authors correctly flag the pure-count baseline as circular but still treat EDM–ERS agreement as primary structural validation.

full rationale

The paper’s central empirical claim (review scores orthogonal to EDM, | ho|≤0.005, accepted/rejected mean EDM indistinguishable) is an ordinary correlation between submission-time scores and a post-hoc citation-embedding measure; it does not reduce to its inputs by definition and is therefore non-circular. EDM vs. ERS shares the citation graph but is not definitionally identical (direction-aware multi-hop embeddings vs. raw count), and the authors correctly exclude the pure-citation baseline as circular. The circularity that does exist is confined to two of the five operational catalyst types: TI is defined by the very topic-share growth ratio whose magnitude is then reported, and RM is defined by the high-EDM/low-score condition whose gap is then reported. These are self-definitional rather than fitted-parameter or self-citation circularities; the remaining types (TB, WR, SC) and the measure comparison retain independent content. No load-bearing self-citations, uniqueness theorems, or ansatz smuggling appear. Score 4 reflects partial definitional circularity on secondary taxonomy claims while the primary orthogonality result stands independently.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The central orthogonality and catalyst-growth claims rest on a small set of hand-chosen operational thresholds, embedding hyperparameters, and the modeling choice that ICLR-internal citation structure plus text-derived topics are sufficient proxies for trajectory change. No new physical entities are postulated; the free parameters are definitional cut-offs and training settings.

free parameters (6)
  • TI topic-share growth threshold = 2.0
    Paper i is labeled TI if post/pre share ratio ≥2.0; the factor 2.0 is chosen by hand and controls the size of the TI set.
  • TB cross-topic flow percentile and descendant floor = top 10%, Di≥2
    TB requires top-10% cross-topic flow and ≥2 descendant topics; both cut-offs are free design choices (swept in appendix but main-text results use the canonical pair).
  • RM EDM percentile and review-boundary offset = Δ≥0.899, −0.5 pts
    RM requires EDM ≥90th percentile and mean score ≤ year acceptance boundary −0.5 (or high variance); both numbers are free.
  • EDM walk and embedding hyperparameters = T=160,R=80,d=100,c=5
    Canonical T=160, R=80, d=100, c=5, κ_in weighting; sensitivity shows AUC stable but ranks pipeline-dependent.
  • HDBSCAN/UMAP topic clustering settings = min_cluster_size=50
    min_cluster_size=50, UMAP 50-D, n_neighbors=15 produce 113 topics and 33.5% noise; different settings would re-label TI/TB/WR.
  • S2 title-match threshold = ≥0.85
    RapidFuzz token-set ratio ≥0.85 and year ±2 determine the 77.1% matched citation graph.
axioms (4)
  • domain assumption ICLR-internal citation edges plus direction-aware random-walk embeddings (EDM) are a valid proxy for multi-generational research-trajectory redirection.
    Stated throughout §§3–4 and used as the dependent variable for all RQ1–RQ3 claims; external diffusion is only secondary.
  • domain assumption Text-embedding + UMAP + HDBSCAN topics correctly partition the ICLR research landscape for measuring initiation, bridging, and redirection.
    Section 4.3; all TI/TB/WR definitions depend on these clusters.
  • domain assumption Year-matched (and propensity-matched) non-catalyst controls isolate the causal contribution of catalyst status to subsequent topic-share and flow growth.
    Section 6.1; residual confounding by unobserved quality or prestige remains possible.
  • standard math Standard skip-gram / node2vec / CD index definitions and Firth logistic regression are correctly implemented.
    Methods §4.2 and validation §4.3; no novel mathematics.
invented entities (2)
  • Five-type operational catalyst taxonomy (TI, TB, WR, SC, RM) no independent evidence
    purpose: Decompose ‘disruptive paper’ into multi-label mechanism types with explicit thresholds so that growth effects and review miscalibration can be measured separately.
    Defined in §4.1; multi-label co-occurrence is reported but the types themselves are new constructs of this paper.
  • Recognition-Misaligned (RM) catalyst class no independent evidence
    purpose: Capture high-EDM papers that received weak or contested review scores, as a submission-stage analog of sleeping beauties.
    §4.1; 83.7% of RM papers were rejected; used to structure the review-gap analysis.

pith-pipeline@v1.1.0-grok45 · 29968 in / 3609 out tokens · 34435 ms · 2026-07-12T16:03:37.881376+00:00 · methodology

0 comments
read the original abstract

A small number of methodological contributions, including word2vec, the Transformer, large-scale pre-training, and reinforcement learning from human feedback, have reshaped NLP and AI research over the past decade. OpenReview now makes numeric reviewer scores and accept/reject decisions public for every ICLR submission. Whether such review signals identify trajectory-changing papers at submission time, however, remains untested at corpus scale. We answer this question on $36{,}113$ papers from ICLR 2017--2025, identifying \emph{catalysts}: papers whose descendants measurably redirect future research. We compare four disruptiveness measures (the Consolidation/Destabilization (CD) index, node2vec, the direction-aware Embedding Disruptiveness Measure (EDM), and an LLM-based semantic rater) and define a five-type operational catalyst taxonomy (topic initiator, topic bridge, within-topic redirector, simultaneous, and recognition-misaligned). EDM leads at identifying highly cited ICLR papers (AUC $0.83$ vs.\ $0.60$ for CD, $0.49$ for node2vec, and $0.42$ for the LLM rater). Topic initiators precede a $7.55{\times}$ topic-share growth and topic bridges precede an $11.52{\times}$ growth in cross-topic citation flow versus year-matched controls. We found that the peer review scores are essentially orthogonal to future disruptiveness ($|\rho|{\leq}0.005$; accepted and rejected papers have indistinguishable mean EDM, $p{=}0.11$).

Figures

Figures reproduced from arXiv: 2607.05401 by Fan Huang.

Figure 1
Figure 1. Figure 1: Disruptiveness distributions across 36,113 ICLR papers. (a) EDM is smooth and near-Gaussian (∆=0 ¯ .76, std 0.11, range [0.44, 1.18]). (b) CD is bimodally degenerate, with scores concentrated near 0 and near 1. (c) EDM distributions are nearly identical for accepted and rejected papers. reported by Park et al. (2023) for broad science. Extended analysis and the yearly trend figure are in Appendix G. 5.1 Va… view at source ↗
Figure 2
Figure 2. Figure 2: EDM parameter sensitivity. (a) ERS AUC across 5 configurations is tightly clustered in [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Inter-LLM agreement (heatmap counterpart to Table [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Temporal trend of EDM across ICLR 2017–2025. (a) Mean and median [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Spearman correlation between disruptiveness [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Pairwise scatter plots between M3 EDM and M1 CD (a), M4 LLM (b), and citation count (c). Red dots [PITH_FULL_IMAGE:figures/full_fig_p017_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Catalyst co-occurrence. Left: raw counts of [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Cross-topic citation flow among the 10 largest topic clusters. Cell (i, j) counts source-target citation edges from topic i to topic j. Adjacent topics (for ex￾ample, diffusion to vision-language, or molecular to protein) show dense flow, while long-range pairs are sparser. Stage Remaining pairs Raw candidates (cosine ≥ 0.9) 483,809 After no author overlap 482,850 After both have ≥ 3 internal citations 334… view at source ↗
Figure 10
Figure 10. Figure 10: Mean cross-domain citation rate (± SE) within three years of publication, by ICLR catalyst type. Rate = (non-AI) / (total). Dashed line: non-catalyst baseline [PITH_FULL_IMAGE:figures/full_fig_p019_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Fraction of papers with ≥ 1 citing work in each non-AI domain, by catalyst type. Robotics is folded into Other CS (no standalone tag in S2). Two-scale picture. TB and WR are high￾intensity, narrow catalysts: proportionally more cross-domain, consistent with TB connecting topic clusters and WR generalizing methods adjacent fields pick up. TI are low-intensity, wide: absolute non-AI footprint dominates ever… view at source ↗
Figure 12
Figure 12. Figure 12: Distribution of EDM ∆ by reviewer mean￾score quartile (accepted vs. rejected). Distributions are nearly identical across quartiles, consistent with the null correlation results in main-text [PITH_FULL_IMAGE:figures/full_fig_p020_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Mean EDM ∆ by reviewer mean-score bin and score-variance bin. Recognition-Misaligned and Topic-Bridge papers cluster in cells where review scores underestimate future disruption. J Discussion: ICLR Setting and Program-Committee Implications ICLR as a laboratory. Three features of ICLR are essential for science-of-science work and un￾Type n Gap t p Non-catalyst 15,723 −0.103 — — TI (Topic Initiator) 2,129 … view at source ↗
Figure 14
Figure 14. Figure 14: Mean review gap by topic cluster (top-10 over-valued and under-valued). Trendy sub-fields are [PITH_FULL_IMAGE:figures/full_fig_p021_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Rejected ICLR papers: trajectory and external-record analysis. (A) KDE of EDM ( [PITH_FULL_IMAGE:figures/full_fig_p022_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 5 linked inside Pith

  1. [1]

    Science Advances , volume=

    Uncovering simultaneous breakthroughs with a robust measure of disruptiveness , author=. Science Advances , volume=. 2026 , publisher=

    2026

  2. [2]

    Management science , volume=

    A dynamic network measure of technological change , author=. Management science , volume=. 2017 , publisher=

    2017

  3. [3]

    Nature , volume=

    Papers and patents are becoming less disruptive over time , author=. Nature , volume=. 2023 , publisher=

    2023

  4. [4]

    Science , volume=

    Science of science , author=. Science , volume=. 2018 , publisher=

    2018

  5. [5]

    Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

    Specter: Document-level representation learning using citation-informed transformers , author=. Proceedings of the 58th annual meeting of the association for computational linguistics , pages=

  6. [6]

    Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    Scirepeval: A multi-format benchmark for scientific document representations , author=. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing , pages=

    2023

  7. [7]

    Pacific-Asia conference on knowledge discovery and data mining , pages=

    Density-based clustering based on hierarchical density estimates , author=. Pacific-Asia conference on knowledge discovery and data mining , pages=. 2013 , organization=

    2013

  8. [8]

    arXiv preprint arXiv:1802.03426 , year=

    Umap: Uniform manifold approximation and projection for dimension reduction , author=. arXiv preprint arXiv:1802.03426 , year=

    Pith/arXiv arXiv

  9. [9]

    Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

    node2vec: Scalable feature learning for networks , author=. Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining , pages=

  10. [10]

    arXiv preprint arXiv:2205.01833 , year=

    OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts , author=. arXiv preprint arXiv:2205.01833 , year=

    Pith/arXiv arXiv

  11. [11]

    DMLR workshop at ICLR 2024 , year=

    Learning representations of learning representations , author=. DMLR workshop at ICLR 2024 , year=

    2024

  12. [12]

    Research policy , volume=

    How status of research papers affects the way they are read and cited , author=. Research policy , volume=. 2022 , publisher=

    2022

  13. [13]

    Research Policy , volume=

    Bias against novelty in science: A cautionary tale for users of bibliometric indicators , author=. Research Policy , volume=. 2017 , publisher=

    2017

  14. [14]

    Nature , volume=

    Large teams develop and small teams disrupt science and technology , author=. Nature , volume=. 2019 , publisher=

    2019

  15. [15]

    Quantitative Science Studies , volume=

    The disruption index is biased by citation inflation , author=. Quantitative Science Studies , volume=. 2024 , publisher=

    2024

  16. [16]

    Science , volume=

    Atypical combinations and scientific impact , author=. Science , volume=. 2013 , publisher=

    2013

  17. [17]

    Biometrika , volume=

    Bias reduction of maximum likelihood estimates , author=. Biometrika , volume=. 1993 , publisher=

    1993

  18. [18]

    , author=

    The Matthew effect in science: The reward and communication systems of science are considered. , author=. Science , volume=. 1968 , publisher=

    1968

  19. [19]

    Science , volume=

    Data-driven predictions in the science of science , author=. Science , volume=. 2017 , publisher=

    2017

  20. [20]

    2024 , month = jul, note =

    2024

  21. [21]

    2024 , month = jan, note =

    New embedding models and. 2024 , month = jan, note =

    2024

  22. [22]

    Proceedings of the American Philosophical Society , volume=

    Singletons and multiples in scientific discovery: A chapter in the sociology of science , author=. Proceedings of the American Philosophical Society , volume=. 1961 , publisher=

    1961

  23. [23]

    BURT , publisher =

    RONALD S. BURT , publisher =. Structural Holes: The Social Structure of Competition , urldate =

  24. [24]

    Proceedings of the National Academy of Sciences , volume=

    Defining and identifying sleeping beauties in science , author=. Proceedings of the National Academy of Sciences , volume=. 2015 , publisher=

    2015

  25. [25]

    arXiv preprint arXiv:2010.05137 , year=

    An open review of openreview: A critical analysis of the machine learning conference review process , author=. arXiv preprint arXiv:2010.05137 , year=

    Pith/arXiv arXiv 2010

  26. [26]

    arXiv preprint arXiv:2109.09774 , year=

    Inconsistency in conference peer review: Revisiting the 2014 neurips experiment , author=. arXiv preprint arXiv:2109.09774 , year=

    Pith/arXiv arXiv 2014

  27. [27]

    arXiv preprint arXiv:1301.3781 , year=

    Efficient estimation of word representations in vector space , author=. arXiv preprint arXiv:1301.3781 , year=

    Pith/arXiv arXiv

  28. [28]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  29. [29]

    2018 , publisher=

    Improving language understanding by generative pre-training , author=. 2018 , publisher=

    2018

  30. [30]

    2019 , url =

    Alec Radford and Jeffrey Wu and Rewon Child and David Luan and Dario Amodei and Ilya Sutskever , title =. 2019 , url =

    2019

  31. [31]

    Advances in neural information processing systems , volume=

    Training language models to follow instructions with human feedback , author=. Advances in neural information processing systems , volume=