Pith. sign in

REVIEW 4 major objections 4 minor 44 references

exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem

T0 review · 4 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The exHarmony benchmark treats a paper's own authors and the authors it cites as ground-truth reviewers, turning the reviewer assignment problem into a retrieval task that can be evaluated without human labels.

desk verdict Useful large-scale weak-label benchmark for RAP-as-retrieval, but the SimCite ground truth is partly circular and the abstract overclaims scholarly embeddings when BM25 wins on Cite. read the letter →

arxiv 2502.07683 v1 pith:V37TSRPI submitted 2025-02-11 cs.IR cs.CL

Sajad Ebrahimi , Sara Salamat , Negar Arabzadeh , Mahdi Bashari , Ebrahim Bagheri This is my paper · ORCID
classification cs.IRcs.CL
keywords reviewerassignmentproblembenchmarkdatasetweaksupervisioninformationretrievaldensescholarlyembeddingspeerreviewdiversitymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the reviewer assignment problem can be studied as a retrieval problem instead of a matching problem requiring manual ground truth. Its central hypothesis is that a paper's own authors would be the best possible reviewers for that paper had they not written it, so the task becomes: given a paper's title and abstract, retrieve the authors most likely to have written it from their prior publications. To make this testable, the authors build the exHarmony benchmark from a large open scholarly database, creating six gold-standard versions that use the paper's authors, the authors of papers it cites, and the authors of the most similar cited papers, each with and without an 'established author' filter. They evaluate lexical, static-embedding, and contextual-embedding retrievers with new relevance and diversity metrics. The benchmark shows that scholarly contextualized embeddings lead, but all tested methods leave large room for improvement.

What carries the argument

The load-bearing mechanism is the weak-supervision hypothesis: a paper's own authors are treated as its ideal hypothetical reviewers, so reviewer assignment reduces to retrieving those authors from a collection of candidate authors indexed by their prior publications. The benchmark operationalizes this by time-stamping papers so that only work published before the review time $\tau$ enters the candidate pool, building three gold standards (Authors, Cite, SimCite) and filtering each into 'established author' versions (at least 15 publications). Retrieval is performed over individual author-papers, and repeated authors are aggregated into a ranked list of unique reviewers. Evaluation is then two-dimensional: classical IR metrics (nDCG, MAP, Recall) for relevance, and new diversity metrics ($\sigma_{\#\mathrm{cite}}$, $\sigma_{\#\mathrm{paper}}$, $\#\mathrm{Institutions}$) for the makeup of the reviewer set.

What would settle it

Take a random sample of papers from the test set and obtain the reviewers who were actually assigned by a journal or conference for those papers; if high-scoring retrievers on exHarmony show no better overlap with these actual reviewers than a random ranking does, the authorship-as-reviewer premise would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that authorship itself provides a weakly supervised signal for reviewer suitability: given a manuscript, the correct reviewers are, approximately, the people who would be best able to write that manuscript, and their past papers are retrievable evidence of that ability. On this view, no explicit reviewer labels are needed, because the authors of the paper and the authors of the work it builds on function as gold-standard reviewers. The paper operationalizes this through three ground truths—authors, cited-paper authors, and SimCite (authors of the top-10 most similar cited papers, chosen with citation-informed transformer embeddings)—and shows that across all variants and measured at top-10 and top-100, contextualized models trained on scientific text outperform static embeddings and general ranking-tuned models, with BM25 as a strong sparse baseline. The authors also propose diversity metrics—standard deviation of citations, standard deviation of publication count, and number of unique institutions—so that a good reviewer set is not just relevant but heterogeneous. The overall discovery is a reproducible, label-free benchmark for the reviewer assignment problem, along with the finding that the task remains largely unsolved.

Load-bearing premise

The benchmark assumes that the source database correctly tells who wrote which paper and which papers cite which others, and that a paper's own authors and cited authors would in fact make good reviewers if they were not the authors.

Editorial extensions

If this is right

  • Because labels are derived from public metadata rather than reviewer identities, the benchmark can be rebuilt and extended to new fields and time periods at low cost.
  • Retrieval performance on exHarmony gives a standardized, reproducible score for any candidate reviewer-assignment system, so future methods can be compared on the same six test sets.
  • The proposed diversity metrics let system builders explicitly trade relevance against career-stage and institutional heterogeneity in the reviewer pool.
  • The low absolute scores across all baselines imply that current open retrieval methods are not yet adequate for real-world reviewer assignment, pointing to aggregation and fine-tuning as the next bottlenecks.
  • The finding that scientific contextualized embeddings lead suggests that domain-specific pretraining, rather than general ranking fine-tuning, is the more promising direction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • As an extension beyond the paper: because the ground truth is built from authorship, a system that memorizes author names or frequently cited authors could inflate scores without providing genuine review expertise; the benchmark would benefit from a collision test against random author sets.
  • As an extension beyond the paper: the filtering of test papers to those whose authors exist in the candidate pool likely removes exactly the hard cases—new authors and interdisciplinary papers—so absolute performance on the published test set may overstate deployability.
  • As an extension beyond the paper: the diversity metrics treat higher standard deviations as better, but extreme outliers (a single very senior, highly cited author) can inflate them without meaningfully diversifying perspectives; a coverage or entropy based metric would be a natural next step.
  • As an extension beyond the paper: the same weak-labeling recipe could be applied to grant review or hiring decisions, since any domain with paper metadata and citation edges yields the same self-supervised gold standard; whether the proxy holds in those settings is a testable empirical question.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces exHarmony, a large-scale benchmark for the Reviewer Assignment Problem (RAP) reframed as an information retrieval task. Using OpenAlex data, the authors construct six weak-label variants based on three assumptions: the authors of a paper are ideal hypothetical reviewers (exHarmony-Authors), authors of cited papers are suitable reviewers (exHarmony-Cite), and authors of the top-10 most similar cited papers are suitable reviewers (exHarmony-SimCite), with each split into all-author and established-author subsets. They evaluate BM25, Doc2Vec, WMD, and several transformer-based retrievers using nDCG, MAP, and Recall, plus new diversity metrics based on citation count, publication count, and institutional diversity. The paper claims that contextualized embeddings trained on scholarly literature, especially SPECTER and SciBERT, perform best.

Significance. If the label premise is accepted, exHarmony would be a valuable resource: it is large-scale, open, cheap to maintain, and it provides six benchmark variants plus diversity-oriented evaluation metrics. The public release of code and data is a clear strength. However, the benchmark's validity as a reviewer-assignment benchmark rests on an unvalidated assumption that authorship and citation relations proxy reviewer competence; the current evaluation therefore measures authorship and citation recovery rather than established review quality. The SimCite subset is also partially circular because SPECTER is used to construct its gold labels and is then scored on it. These issues do not destroy the dataset's utility as an authorship-retrieval benchmark, but they substantially weaken the paper's stronger claims about RAP.

major comments (4)
  1. [Section 4.2, item 3 and Table 4] The SimCite gold standard is constructed by selecting the top-10 most similar cited papers using SPECTER embeddings, and SPECTER is then evaluated on that same subset. This makes the SimCite comparison circular: SPECTER's apparent advantage on SimCite (e.g., nDCG@10=0.158 vs. BM25's 0.151 in Table 4) could reflect construction overlap rather than retrieval quality. The paper should either exclude SPECTER from the SimCite evaluation, construct the SimCite labels with a retriever that is not part of the baseline set, or explicitly report SimCite results only as a diagnostic for non-constructing methods.
  2. [Section 3 and Section 4.2] The central premise—that the authors of a paper, and by extension authors of cited and similar cited papers, are potentially the best reviewers for that paper—is never validated against actual reviewing behavior. All six gold-standard sets operationalize this hypothesis, but the paper provides no evidence that retrieved authors are competent, unbiased, or willing reviewers. A retriever could exploit self-citation patterns or name co-occurrence and achieve high scores on exHarmony-Authors without solving RAP. The authors should either validate the weak labels against real review outcomes or explicitly reframe the benchmark as measuring authorship/citation recovery rather than reviewer assignment quality.
  3. [Section 5.1, Table 3 and Section 5.3] The abstract's claim that scholarly contextualized embeddings perform best is not consistently supported by Table 3. On exHarmony-Cite with all authors, BM25 achieves nDCG@10=0.187 while SPECTER achieves 0.178 and BERT-R 0.183; on the established-author subset, BM25 again leads with nDCG@10=0.163 versus SPECTER's 0.157. Moreover, no confidence intervals or significance tests are reported for any metric in Tables 2–5. Given the small absolute differences across many systems, the comparative conclusions need statistical support, such as bootstrap intervals or pairwise significance tests over the 7,944 test papers.
  4. [Section 4.1] The dataset construction relies entirely on OpenAlex's author disambiguation, topic classification, and citation edges, but the paper reports no error rates or quality checks for these components. If OpenAlex merges distinct researchers, splits one researcher, or misassigns topics, both the candidate pool and the gold labels are corrupted. In addition, the paper states that test papers were kept if at least one author exists in Cτ, but the benchmark goal is to retrieve all gold authors; the text should clarify whether all gold authors are guaranteed to be in Cτ and, if not, how incomplete gold labels are handled in the evaluation.
minor comments (4)
  1. [Section 4.1] There is a typo, 'fileter', and the reference list in the same paragraph contains a placeholder '[?]' that should be resolved before publication.
  2. [Section 4.1 and Table 1] The text reports 1,212,094 total papers while Table 1 reports 1,204,150 papers in the collection; these numbers should be reconciled.
  3. [Section 5.2 and Tables 2–4] The text states that metrics are reported at top-10, top-20, and top-100, but Tables 2–4 present only @10 and @100 values.
  4. [Section 4.2, item 1] The threshold N=15 for 'established authors' is introduced without justification or sensitivity analysis; since this is a free parameter that changes every gold set, the paper should explain the choice and ideally report results across a range of N.

Circularity Check

1 steps flagged · score 6.0 of 10

SimCite gold labels are generated by SPECTER, and SPECTER is then scored on that same gold standard, making part of the benchmark self-referential.

  1. self definitional [Section 4.2, Gold Standards, item 3 (exHarmony-SimCite); Section 5.1, Retrieval Baselines; Table 4]
    "To mitigatetopic drift, this subset focuses on authors of the most similar cited papers, i.e., those with high similarity to the main paper. To do this, we find top-10 similar cited papers using SPECTER [11] embeddings based on the papers title and abstracts. ... we leveraged two widely used language models for scientific purposes: SciBERT* [8] and SPECTER* [11]."

    The exHarmony-SimCite gold-standard labels are constructed by running SPECTER to select the top-10 most similar cited papers for each test paper. SPECTER is then included as one of the systems evaluated against that same gold standard in Table 4. Because the target set of 'similar cited papers' is defined by SPECTER's own embedding similarity, SPECTER's scores on this subset reflect, in part, its ability to retrieve the very papers it was used to choose, rather than an independent measure of reviewer relevance. The Authors and Cite subsets are not label-generated by an evaluated model, so the circularity is confined to SimCite, but the paper's general claim that scholarly contextualized embeddings perform best is partly based on this self-referential subset.

full rationale

Most of the benchmark is not circular: the exHarmony-Authors and exHarmony-Cite ground truths are defined from OpenAlex authorship and citation metadata, with no evaluated model involved, and the retrieval results on those subsets stand independently. No load-bearing uniqueness theorem or self-citation chain was found; citations to the authors' prior work are contextual, not foundational. The one substantive circularity is exHarmony-SimCite: SPECTER is used to define the top-10 similar cited papers that form the gold reviewers, and SPECTER is then scored against that same gold standard. This makes the SimCite results partially self-referential, and since the paper uses these results as evidence that scholarly contextualized embeddings are best, the evaluation contains a partial reduce-by-construction component. Because independent subsets still support the general ranking conclusion, the overall circularity is partial, not total.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

All core labels and evaluations depend on assumptions that are imported rather than demonstrated: author-as-reviewer is a self-stated hypothesis in Section 3, OpenAlex metadata is trusted in Section 4.1, and the diversity metrics encode a normative preference in Section 5.2. The two threshold values, N=15 and k=10, are chosen by hand and no sensitivity analysis is given.

free parameters (2)
  • established-author threshold N = 15
    Chosen by hand in Section 4.2 to define established-author gold subsets; no sensitivity analysis is reported.
  • SimCite top-k = 10
    Chosen by hand in Section 4.2 to select similar cited papers for the SimCite gold set; no sensitivity analysis is reported.
assumptions (3)
  • domain assumption Authors of a paper are the best hypothetical reviewers for that paper had they not been its authors.
    Section 3 states this hypothesis directly and uses it as the basis for the exHarmony gold standards; no independent validation against real reviewer quality is provided.
  • domain assumption OpenAlex author disambiguation, topic classification, and citation data are accurate enough for constructing author collections and ground truth.
    Section 4.1 relies on the OpenAlex March 2024 snapshot and the openalex-topic-classification model without error analysis.
  • domain assumption Higher standard deviation of citations and papers and higher number of institutions are desired diversity properties.
    Section 5.2 proposes these metrics as normative evaluation criteria; no demonstrated link to actual review quality or fairness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem." pith.science (2026). https://pith.science/paper/V37TSRPI

@misc{pith2026250207683,
  author       = {Pith},
  title        = {Pith review of: exHarmony: Authorship and Citations for Benchmarking the Reviewer Assignment Problem},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V37TSRPI}},
  note         = {Machine review of arXiv:2502.07683}
}
read the original abstract

The peer review process is crucial for ensuring the quality and reliability of scholarly work, yet assigning suitable reviewers remains a significant challenge. Traditional manual methods are labor-intensive and often ineffective, leading to nonconstructive or biased reviews. This paper introduces the exHarmony (eHarmony but for connecting experts to manuscripts) benchmark, designed to address these challenges by re-imagining the Reviewer Assignment Problem (RAP) as a retrieval task. Utilizing the extensive data from OpenAlex, we propose a novel approach that considers a host of signals from the authors, most similar experts, and the citation relations as potential indicators for a suitable reviewer for a manuscript. This approach allows us to develop a standard benchmark dataset for evaluating the reviewer assignment problem without needing explicit labels. We benchmark various methods, including traditional lexical matching, static neural embeddings, and contextualized neural embeddings, and introduce evaluation metrics that assess both relevance and diversity in the context of RAP. Our results indicate that while traditional methods perform reasonably well, contextualized embeddings trained on scholarly literature show the best performance. The findings underscore the importance of further research to enhance the diversity and effectiveness of reviewer assignments.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 35 canonical work pages

  1. [1]

    JAIR (2023)

    Aksoy, M., Yanik, S., Amasyali, M.F.: Reviewer assignment problem: A systematic review of the literature. JAIR (2023)

    work page 2023

  2. [2]

    In: Information Access in the Era of Generative AI, pp

    Alaofi, M., Arabzadeh, N., Clarke, C.L., Sanderson, M.: Generative information retrieval evaluation. In: Information Access in the Era of Generative AI, pp. 135–

    work page

  3. [3]

    In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

    Arabzadeh, N., Bigdeli, A., Hamidi Rad, R., Bagheri, E.: Quantifying ranker cov- erage of different query subspaces. In: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 2298–2302 (2023)

    work page 2023

  4. [4]

    Arabzadeh, N., Ebrahimi, S., Salamat, S., Bashari, M., Bagheri, E.: Reviewerly: Modelingthereviewerassignmenttaskasaninformationretrievalproblem.In:Pro- ceedings of the 33rd ACM International Conference on Information and Knowledge Management. pp. 5554–5555 (2024)

    work page 2024

  5. [5]

    In: CIKM (2021)

    Arabzadeh, N., Khodabakhsh, M., Bagheri, E.: Bert-qpp: Contextualized pre- trained transformers for query performance prediction. In: CIKM (2021)

    work page 2021

  6. [6]

    In: Proceedings of the 2023 ACM SIGIR international conference on theory of information retrieval

    Arabzadeh, N., Kmet, O., Carterette, B., Clarke, C.L., Hauff, C., Chandar, P.: A is for adele: An offline evaluation metric for instant search. In: Proceedings of the 2023 ACM SIGIR international conference on theory of information retrieval. pp. 3–12 (2023)

    work page 2023

  7. [7]

    Information Retrieval Journal25(4), 365–385 (2022)

    Arabzadeh, N., Vtyurina, A., Yan, X., Clarke, C.L.: Shallow pooling for sparse labels. Information Retrieval Journal25(4), 365–385 (2022)

    work page 2022

  8. [8]

    arXiv preprint arXiv:1903.10676 (2019)

    Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 (2019)

    arXiv 2019

Show all 44 references
  1. [9]

    In: Proceedings of the 2024 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region

    Bigdeli, A., Arabzadeh, N., Bagheri, E., Clarke, C.L.: Evaluating relative retrieval effectiveness with normalized residual gain. In: Proceedings of the 2024 Annual In- ternational ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Re...

    2024

  2. [10]

    In: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval

    Buckley, C., Voorhees, E.M.: Retrieval evaluation with incomplete information. In: Proceedings of the 27th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 25–32 (2004)

    2004

  3. [11]

    arXiv preprint arXiv:2004.07180 (2020)

    Cohan, A., Feldman, S., Beltagy, I., Downey, D., Weld, D.S.: Specter: Document- level representation learning using citation-informed transformers. arXiv preprint arXiv:2004.07180 (2020)

    2020 arXiv

  4. [12]

    CoRR abs/0906.4044 (2009), http://arxiv.org/ abs/0906.4044

    Conry, D., Koren, Y., Ramakrishnan, N.: Recommender systems for the conference paper assignment problem. CoRR abs/0906.4044 (2009), http://arxiv.org/ abs/0906.4044

    2009 arXiv

  5. [13]

    arXiv preprint arXiv:1810.04805 (2018)

    Devlin,J.,Chang,M.W.,Lee,K.,Toutanova,K.:Bert:Pre-trainingofdeepbidirec- tional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)

    2018 arXiv

  6. [14]

    Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library (2024)

    2024

  7. [15]

    In: Proceedings of the 15th Annual International ACM SIGIR Con- ference on Research and Development in Information Retrieval

    Dumais, S.T., Nielsen, J.: Automating the assignment of submitted manuscripts to reviewers. In: Proceedings of the 15th Annual International ACM SIGIR Con- ference on Research and Development in Information Retrieval. p. 233–244. SI- GIR ’92, ACM, New York, NY, USA (1992).http...

    1992

  8. [16]

    Decision Sciences (1999)

    Hartvigsen, D., Wei, J.C., Czuchlewski, R.: The conference paper-reviewer assign- ment problem. Decision Sciences (1999)

    1999

  9. [17]

    arXiv preprint arXiv:2305.07887 (2023)

    Jovanovic, J., Bagheri, E.: Reviewer assignment problem: A scoping review. arXiv preprint arXiv:2305.07887 (2023)

    2023 arXiv

  10. [18]

    In: Proceedings of the 18th ACM conference on Information and knowledge management

    Karimzadehgan, M., Zhai, C.: Constrained multi-aspect expertise matching for committee review assignment. In: Proceedings of the 18th ACM conference on Information and knowledge management. pp. 1697–1700 (2009)

    2009

  11. [19]

    In: Proceedings of the 17th ACM conference on Information and knowledge management

    Karimzadehgan, M., Zhai, C., Belford, G.: Multi-aspect expertise matching for review assignment. In: Proceedings of the 17th ACM conference on Information and knowledge management. pp. 1113–1122 (2008)

    2008

  12. [20]

    arXiv preprint arXiv:2004.04906 (2020)

    Karpukhin, V., Oğuz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., Yih, W.t.: Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906 (2020)

    2020 arXiv

  13. [21]

    IETE Technical Review (2011)

    Kolasa, T., Krol, D.: A survey of algorithms for paper-reviewer assignment prob- lem. IETE Technical Review (2011)

    2011

  14. [22]

    In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

    Kou, N.M., U., L.H., Mamoulis, N., Gong, Z.: Weighted coverage based reviewer assignment. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. p. 2031–2046. SIGMOD ’15, Association for Comput- ing Machinery, New York, NY, USA (2015).https://do...

    2015

  15. [23]

    Kou, N.M., U, L.H., Mamoulis, N., Li, Y., Li, Y., Gong, Z.: A topic- based reviewer assignment system. Proc. VLDB Endow. 8(12), 1852–1855 (aug2015). https://doi.org/10.14778/2824032.2824084, https://doi.org/10. 14778/2824032.2824084

  16. [24]

    In: International conference on machine learning

    Kusner, M., Sun, Y., Kolkin, N., Weinberger, K.: From word embeddings to doc- ument distances. In: International conference on machine learning. pp. 957–966. PMLR (2015)

    2015

  17. [25]

    In: International conference on machine learning

    Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In: International conference on machine learning. pp. 1188–1196. PMLR (2014)

    2014

  18. [26]

    In: SIGIR

    Lin, J., Ma, X., Lin, S.C., Yang, J.H., Pradeep, R., Nogueira, R.: Pyserini: A pythontoolkitforreproducibleinformationretrievalresearchwithsparseanddense representations. In: SIGIR. pp. 2356–2362 (2021) exHarmony: Benchmarking the Reviewer Assignment Problem 15

    2021

  19. [27]

    arXiv preprint arXiv:1301.3781 (2013)

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word repre- sentations in vector space. arXiv preprint arXiv:1301.3781 (2013)

    2013 arXiv

  20. [28]

    In: Knowledge Discovery and Data Mining

    Mimno, D., McCallum, A.: Expertise modeling for matching papers with reviewers. In: Knowledge Discovery and Data Mining

  21. [29]

    Mirzaei, M., Sander, J., Stroulia, E.: Multi-aspect review-team assignment using latent research areas. Inf. Process. Manage.56(3), 858–878 (may 2019).https:// doi.org/10.1016/j.ipm.2019.01.007, https://doi.org/10.1016/j.ipm.2019. 01.007

    2019 doi

  22. [30]

    In: IoT-SIU

    Mittal, K., Jain, A., Vaisla, K.S.: Understanding reviewer assignment problem and its issues and challenges. In: IoT-SIU. IEEE (2019)

    2019

  23. [31]

    Pattern Recog- nit

    Nguyen, J., Sánchez, G., Agell, N., Rovira, X., Angulo, C.: A decision support tool using order weighted averaging for conference review assignment. Pattern Recog- nit. Lett. 105, 114–120 (2018), https://api.semanticscholar.org/CorpusID: 13678527

    2018

  24. [32]

    In: NeurIPS (2016)

    Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., Deng, L.: Ms marco: A human generated machine reading comprehension dataset. In: NeurIPS (2016)

    2016

  25. [33]

    In: CUCEN

    Ogunleye, O., Ifebanjo, T., Abiodun, T., Adebiyi, A.: Proposed framework for a paper-reviewer assignment system using word2vec. In: CUCEN

  26. [34]

    Peng, H., Hu, H., Wang, K., Wang, X.: Time-aware and topic-based reviewer as- signment book chapter@ dasfaa 2017 international workshops: Bdms (2017)

    2017

  27. [35]

    Priem, J., Piwowar, H., Orr, R.: Openalex: A fully-open index of scholarly works, authors, venues, institutions, and concepts (2022)

    2022

  28. [36]

    arXiv preprint arXiv:1908.10084 (2019)

    Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert- networks. arXiv preprint arXiv:1908.10084 (2019)

    2019 arXiv

  29. [37]

    Nist Special Publication Sp109, 109 (1995)

    Robertson,S.E.,Walker,S.,Jones,S.,Hancock-Beaulieu,M.M.,Gatford,M.,etal.: Okapi at trec-3. Nist Special Publication Sp109, 109 (1995)

    1995

  30. [38]

    Information Retrieval11, 447–470 (2008)

    Sakai, T., Kando, N.: On information retrieval metrics designed for evaluation with incomplete relevance assessments. Information Retrieval11, 447–470 (2008)

    2008

  31. [39]

    In: 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology

    Tang, W., Tang, J., Tan, C.: Expertise matching via constraint-based optimiza- tion. In: 2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology. vol. 1, pp. 34–41 (2010).https://doi.org/10.1109/ WI-IAT.2010.133

    2010

  32. [40]

    IJITDM 9(04), 645–668 (2010)

    Wang, F., Shi, N., Chen, B.: A comprehensive survey of the reviewer assignment problem. IJITDM 9(04), 645–668 (2010)

    2010

  33. [41]

    Advances in Neural Information Processing Systems33, 5776–5788 (2020)

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems33, 5776–5788 (2020)

    2020

  34. [42]

    In: Proceedings of the 28th International Joint Conference on Artificial Intelligence

    Xu, Y., Zhao, H., Shi, X., Shah, N.B.: On strategyproof conference peer review. In: Proceedings of the 28th International Joint Conference on Artificial Intelligence. p. 616–622. IJCAI’19, AAAI Press (2019)

    2019

  35. [43]

    TOIS (2020)

    Zhang, D., Zhao, S., Duan, Z., Chen, J., Zhang, Y., Tang, J.: A multi-label clas- sification method using a hierarchical and transparent representation for paper- reviewer recommendation. TOIS (2020)

    2020

  36. [44]

    Zhao,S.,Zhang,D.,Duan,Z.,Chen,J.,Zhang,Y.p.,Tang,J.:Anovelclassification method for paper-reviewer recommendation (2018)

    2018

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.