Pith. sign in

REVIEW 1 major objections 19 references

Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review

T0 review · 1 major / 0 minor · reviewed 2026-06-29 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Rejoinder organizes defense of ICML 2023 ranking experiment around four themes

desk verdict Rejoinder organizes replies to discussants around four themes but adds no new data, experiments, or formal analysis. read the letter →

arxiv 2605.25172 v1 pith:BBYYT6KP submitted 2026-05-24 stat.AP cs.DLcs.LG

Buxin Su , Jiayao Zhang , Natalie Collina , Yuling Yan , Didong Li , Kyunghyun Cho , Jianqing Fan , Aaron Roth
show 1 more author
Weijie Su
This is my paper · ORCID
classification stat.APcs.DLcs.LG
keywords peerreviewstatisticalestimationIsotonicMechanismICML2023authorself-assessmentgenerativeAIequityconcernsrejoinder
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This rejoinder addresses practical and theoretical points raised by discussants on the original paper about author self-assessment in ML/AI peer review. It structures the entire response around four core themes to defend the experimental approach. A sympathetic reader would care because the themes clarify how to treat peer review as an estimation task, handle fairness issues, add new signals, and adapt the process to generative AI.

What carries the argument

The four core themes used to structure the rejoinder and address discussants' concerns.

What would settle it

A specific concern raised by one of the discussants that falls outside all four themes and is left unaddressed in the rejoinder.

Watch

Extended reading notes

Core claim

The authors address the discussants' points by organizing their response around four core themes: formulating peer review as a statistical estimation problem; mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; incorporating complementary signals such as reviewer rankings and structured metadata; and exploring a human-centered framework for peer review in the era of generative AI.

Load-bearing premise

That organizing the response around these four themes is sufficient to resolve the discussants' practical and theoretical concerns without requiring new empirical data, formal proofs, or direct rebuttals to specific counter-arguments.

Editorial extensions

If this is right

  • Peer review can be treated as a statistical estimation problem to improve ranking accuracy.
  • The Isotonic Mechanism can be deployed after adding mitigations for equity and strategic behavior.
  • Reviewer rankings and structured metadata can serve as useful complementary signals.
  • A human-centered framework can guide peer review adaptations in the presence of generative AI.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same four-theme structure might be reusable for rejoinders in other statistical studies of conference review processes.
  • Testing the equity mitigations in a follow-up experiment at a different conference would provide direct evidence of their effectiveness.
  • Integrating the human-centered AI framework could connect peer-review research to broader questions of automation in academic evaluation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 0 minor

Summary. The manuscript is a rejoinder to the discussion of "The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review" (to appear in JASA). It states that the response to the discussants' practical and theoretical points is organized around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.

Significance. If the thematic organization maps explicitly to and resolves the discussants' specific concerns, the rejoinder could usefully structure ongoing conversation on statistical framing and deployment issues in peer review. Its contribution is primarily organizational, however, as the abstract indicates no new empirical data, formal proofs, or direct rebuttals to individual counter-arguments.

major comments (1)
  1. [Abstract] Abstract: the claim that organizing the response around the four themes addresses the discussants' points assumes that broad thematic discussion is sufficient in place of targeted mapping to specific counter-arguments, new empirical results, or formal analysis; the abstract provides no indication that such mapping or substantiation occurs, leaving the central claim unsubstantiated.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive comment on the abstract. We address it directly below and agree that a revision to clarify the mapping would strengthen the manuscript.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that organizing the response around the four themes addresses the discussants' points assumes that broad thematic discussion is sufficient in place of targeted mapping to specific counter-arguments, new empirical results, or formal analysis; the abstract provides no indication that such mapping or substantiation occurs, leaving the central claim unsubstantiated.

    Authors: We agree that the abstract could more explicitly signal how the four themes correspond to clusters of discussant comments. The full rejoinder text does organize responses to the practical and theoretical points raised, with each theme addressing groups of related concerns (e.g., statistical estimation framing covers modeling critiques; equity and strategic issues address deployment objections). However, the abstract itself does not detail this correspondence. We will revise the abstract to include a brief sentence noting that the themes are chosen to group and respond to specific classes of discussant feedback. As this is a rejoinder, we do not introduce new empirical data or formal proofs; the contribution remains organizational and synthetic. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: thematic rejoinder contains no derivations or fitted claims

full rationale

The paper is a rejoinder that organizes discussion around four listed themes without any equations, statistical predictions, parameter fitting, or derivation chains. No self-definitional steps, fitted inputs renamed as predictions, or load-bearing self-citations appear, as the content is purely textual response to discussants rather than a mathematical or empirical claim that could reduce to its own inputs by construction. The structure is self-contained as a discussion piece and does not invoke uniqueness theorems or ansatzes.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No new mathematical claims, parameters, or entities are introduced; the document is a discussion response relying on the prior experiment and discussant comments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review." pith.science (2026). https://pith.science/paper/BBYYT6KP

@misc{pith2026260525172,
  author       = {Pith},
  title        = {Pith review of: Rejoinder: The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BBYYT6KP}},
  note         = {Machine review of arXiv:2605.25172}
}
read the original abstract

This article is the rejoinder to ``The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review,'' to appear in the Journal of the American Statistical Association with discussion. To address the practical and theoretical points raised by the discussants, we organize our response around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.

Figures

Figures reproduced from arXiv: 2605.25172 by the authors.

Figure 1
Figure 1. MSE and MAE averaged over ICML 2023 authors who submitted rankings of the same [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 5 canonical work pages

  1. [1]

    Aziz, H., Lev, O., Mattei, N., Rosenschein, J., and Walsh, T. (2016). Strategyproof peer selection: Mechanisms, analyses, and experiments. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 30

    2016

  2. [2]

    Down, A. (2025). Artificial intelligence research has a slop problem, academics say: `it's a mess'. The Guardian . Published December 6, 2025; last modified December 9, 2025

    2025

  3. [3]

    Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., and Shah, N. B. (2025). Peer reviews of peer reviews: A randomized controlled trial and other experiments. PloS one , 20(4):e0320444

    2025

  4. [4]

    He, C., Wang, F., and Zhu, L. (2026). Emerging knowledge trend in statistical research: A content-based analysis using covariate-assisted dynamic topic model. Journal of the American Statistical Association , pages 1--14

    2026

  5. [5]

    Kim, J., Lee, Y., and Lee, S. (2025). Position: T he AI conference peer review crisis demands author feedback and reviewer rewards. In International Conference on Machine Learning , pages 81634--81651. PMLR

    2025

  6. [6]

    Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., McFarland, D., and Zou, J. (2023). Can large language models provide useful feedback on research papers? a large-scale empirical analysis. arXiv preprint arXiv:2310.01783

    work page arXiv 2023

  7. [7]

    Pearson, H., Ledford, H., Hutson, M., and Van Noorden, R. (2025). Exclusive: the most-cited papers of the twenty-first century. Nature , 640(8059):588--592

    2025

  8. [8]

    N., Liang, P., Vaughan, J

    Rastogi, C., Stelmakh, I., Beygelzimer, A., Dauphin, Y. N., Liang, P., Vaughan, J. W., Xue, Z., Daum \'e III, H., Pierson, E., and Shah, N. B. (2022). How do authors' perceptions of their papers compare with co-authors' perceptions and peer-review decisions? arXiv preprint arXiv:2211.12966

    work page arXiv 2022

Show all 19 references
  1. [9]

    Shah, N., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018). Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research , 19:1--34

    2018

  2. [10]

    B., Singh, A., and Daum \'e III, H

    Stelmakh, I., Shah, N. B., Singh, A., and Daum \'e III, H. (2021). A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 4785--4793

    2021

  3. [11]

    Su, B., Collina, N., Wen, G., Li, D., Cho, K., Fan, J., Zhao, B., and Su, W. (2025a). How to find fantastic AI papers: S elf-rankings as a powerful predictor of scientific impact beyond peer review. arXiv preprint arXiv:2510.02143

  4. [12]

    Su, B., Zhang, J., Collina, N., Yan, Y., Li, D., Cho, K., Fan, J., Roth, A., and Su, W. (2025b). The ICML 2023 ranking experiment: E xamining author self-assessment in ML/AI peer review. Journal of the American Statistical Association , pages 1--12

    2023

  5. [13]

    Su, W. (2026). You are the best reviewer of your own papers: T he isotonic mechanism. Operations Research , 74(2):804--824

    2026

  6. [14]

    Su, W. J. (2021). You are the best reviewer of your own papers: A n owner-assisted scoring mechanism. Advances in Neural Information Processing Systems , 34:27929--27939

    2021

  7. [15]

    G., Su, B., Collina, N., Deng, Z., and Su, W

    Wen, G. G., Su, B., Collina, N., Deng, Z., and Su, W. (2026). Recommending best paper awards for ML/AI conferences via the isotonic mechanism. arXiv preprint arXiv:2601.15249

    2026

  8. [16]

    Wu, J., Xu, H., Guo, Y., and Su, W. J. (2023). An isotonic mechanism for overlapping ownership. arXiv preprint arXiv:2306.11154

    2023

  9. [17]

    Xu, Y., Jecmen, S., Song, Z., and Fang, F. (2023). A one-size-fits-all approach to improving randomness in paper assignment. Advances in Neural Information Processing Systems , 36:14445--14468

    2023

  10. [18]

    J., and Fan, J

    Yan, Y., Su, W. J., and Fan, J. (2025). Isotonic mechanism for exponential family estimation in machine learning peer review. Journal of the Royal Statistical Society Series B: Statistical Methodology , 87(5):1422--1456

    2025

  11. [19]

    Yuan, W., Liu, P., and Neubig, G. (2022). Can we automate scientific reviewing? Journal of Artificial Intelligence Research , 75:171--212

    2022

Pith tools

Reviewed June 29, 2026 · model on record in the stance chip above.