{"id":"b0afb378-5171-45cf-a2b3-db04dbd0a0ed","arxiv_id":"2605.25172","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":2.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A rejoinder organizing responses to discussants into four core themes on statistical modeling, equity, signals, and AI in peer review.","lead":"This paper is a rejoinder to discussants of the ICML 2023 Ranking Experiment on author self-assessment in ML/AI peer review. It structures responses around four themes: peer review as statistical estimation, equity in the Isotonic Mechanism, complementary signals like reviewer rankings, and human-centered AI-era frameworks.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Thematic organization assumes it resolves discussant concerns without new data, proofs, or direct rebuttals to specific arguments","rationale":"The reader's weakest_assumption matches the load-bearing point exactly: a rejoinder's claim to have addressed concerns via thematic organization is only as strong as its actual engagement with specifics, and the absence of new primary results makes this the critical untested link.","tokens_in":1672,"tokens_out":300,"duration_ms":16793,"concrete_test":"In the full rejoinder manuscript, extract each of the four theme sections and check for explicit citations to individual discussant comments with direct rebuttals, new calculations, or evidence; if responses stay at the level of general discussion without addressing specific counter-arguments, the sufficiency claim does not hold.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that structuring the rejoinder around the four listed themes (statistical estimation framing, equity/strategy mitigation for the Isotonic Mechanism, complementary signals, and human-centered AI-era framework) addresses the discussants' practical and theoretical points. This rests on the assumption that broad thematic discussion suffices in place of targeted responses to individual criticisms, new empirical results, or formal analysis. In a rejoinder, if the themes do not map explicitly to and counter specific counter-arguments from the discussion, the original concerns remain open; the abstract provides no indication that such mapping or new substantiation occurs.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript is a rejoinder to the discussion of \"The ICML 2023 Ranking Experiment: Examining Author Self-Assessment in ML/AI Peer Review\" (to appear in JASA). It states that the response to the discussants' practical and theoretical points is organized around four core themes: (i) formulating peer review as a statistical estimation problem; (ii) mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; (iii) incorporating complementary signals such as reviewer rankings and structured metadata; and (iv) exploring a human-centered framework for peer review in the era of generative AI.","tokens_in":1782,"tokens_out":314,"duration_ms":25620,"significance":"If the thematic organization maps explicitly to and resolves the discussants' specific concerns, the rejoinder could usefully structure ongoing conversation on statistical framing and deployment issues in peer review. Its contribution is primarily organizational, however, as the abstract indicates no new empirical data, formal proofs, or direct rebuttals to individual counter-arguments.","major_comments":[{"comment":"Abstract: the claim that organizing the response around the four themes addresses the discussants' points assumes that broad thematic discussion is sufficient in place of targeted mapping to specific counter-arguments, new empirical results, or formal analysis; the abstract provides no indication that such mapping or substantiation occurs, leaving the central claim unsubstantiated.","section":"Abstract"}],"minor_comments":[],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comment on the abstract. We address it directly below and agree that a revision to clarify the mapping would strengthen the manuscript.","responses":[{"response":"We agree that the abstract could more explicitly signal how the four themes correspond to clusters of discussant comments. The full rejoinder text does organize responses to the practical and theoretical points raised, with each theme addressing groups of related concerns (e.g., statistical estimation framing covers modeling critiques; equity and strategic issues address deployment objections). However, the abstract itself does not detail this correspondence. We will revise the abstract to include a brief sentence noting that the themes are chosen to group and respond to specific classes of discussant feedback. As this is a rejoinder, we do not introduce new empirical data or formal proofs; the contribution remains organizational and synthetic.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that organizing the response around the four themes addresses the discussants' points assumes that broad thematic discussion is sufficient in place of targeted mapping to specific counter-arguments, new empirical results, or formal analysis; the abstract provides no indication that such mapping or substantiation occurs, leaving the central claim unsubstantiated."}],"tokens_in":1201,"tokens_out":272,"duration_ms":14300,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that this is a rejoinder to the ICML 2023 ranking experiment paper. It structures the authors' responses to the discussants around four themes but introduces no new empirical results, derivations, or validations.\n\nThe authors group their replies by framing peer review as a statistical estimation problem, addressing equity and strategic issues with the Isotonic Mechanism, suggesting complementary signals such as reviewer rankings and metadata, and considering a human-centered approach in the generative AI era. This organization is clear and helps connect their positions back to the original work.\n\nThe paper does a reasonable job of laying out their thinking on these points in one place. It shows they have considered the practical and theoretical angles raised in the discussion.\n\nThe soft spot is that the response relies on thematic discussion without new data or targeted evidence to back up how the themes resolve specific criticisms. If discussants pointed to concrete gaps in the original analysis, this format may leave those points open rather than closing them with additional substantiation.\n\nThere are no new equations or parameters, so issues like circularity or overfitting do not arise. Citations stay within the prior paper and the discussion.\n\nThis is for readers already following the original experiment and the JASA discussion on peer review. It is too narrow and incremental for a general audience or for someone seeking fresh methods.\n\nI would not bring this to a reading group on its own. I would not cite it in my own work. As part of a journal discussion, though, it should go to peer review rather than desk rejection so the exchange can be properly recorded.","headline":"Rejoinder organizes replies to discussants around four themes but adds no new data, experiments, or formal analysis.","tokens_in":2286,"tokens_out":392,"would_cite":false,"duration_ms":26883,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Rejoinder organizes defense of ICML 2023 ranking experiment around four themes","keywords":["peer review","statistical estimation","Isotonic Mechanism","ICML 2023","author self-assessment","generative AI","equity concerns","rejoinder"],"falsifier":"A specific concern raised by one of the discussants that falls outside all four themes and is left unaddressed in the rejoinder.","tokens_in":2575,"feed_emoji":"📋","tokens_out":568,"duration_ms":19079,"temperature":0.7,"pith_summary":"This rejoinder addresses practical and theoretical points raised by discussants on the original paper about author self-assessment in ML/AI peer review. It structures the entire response around four core themes to defend the experimental approach. A sympathetic reader would care because the themes clarify how to treat peer review as an estimation task, handle fairness issues, add new signals, and adapt the process to generative AI.","feed_headline":"Rejoinder frames ICML ranking experiment critiques in four themes","feed_subtitle":"Authors address estimation, equity mitigations, new signals, and AI-era adaptations without new data","key_machinery":"The four core themes used to structure the rejoinder and address discussants' concerns.","core_discovery":"The authors address the discussants' points by organizing their response around four core themes: formulating peer review as a statistical estimation problem; mitigating equity and strategic concerns in the deployment of the Isotonic Mechanism; incorporating complementary signals such as reviewer rankings and structured metadata; and exploring a human-centered framework for peer review in the era of generative AI.","pith_inferences":["The same four-theme structure might be reusable for rejoinders in other statistical studies of conference review processes.","Testing the equity mitigations in a follow-up experiment at a different conference would provide direct evidence of their effectiveness.","Integrating the human-centered AI framework could connect peer-review research to broader questions of automation in academic evaluation."],"forward_implications":["Peer review can be treated as a statistical estimation problem to improve ranking accuracy.","The Isotonic Mechanism can be deployed after adding mitigations for equity and strategic behavior.","Reviewer rankings and structured metadata can serve as useful complementary signals.","A human-centered framework can guide peer review adaptations in the presence of generative AI."],"fun_headline_variants":["Rejoinder splits ICML critiques into four themes","Four themes reply to ICML ranking experiment","ICML ranking rejoinder uses four core themes","Rejoinder to ICML experiment via four response themes","Themes cover estimation equity signals in AI review"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"That organizing the response around these four themes is sufficient to resolve the discussants' practical and theoretical concerns without requiring new empirical data, formal proofs, or direct rebuttals to specific counter-arguments.","fun_headline_variants_meta":{"raw":{"variants":["Rejoinder splits ICML critiques into four themes","Four themes reply to ICML ranking experiment","ICML ranking rejoinder uses four core themes","Rejoinder to ICML experiment via four response themes","Themes cover estimation equity signals in AI review"]},"model":"grok-4.3","cost_usd":0.004659,"raw_usage":{"total_tokens":2246,"prompt_tokens":550,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":46587000,"prompt_tokens_details":{"text_tokens":550,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1626,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":550,"tokens_out":70,"duration_ms":13097,"temperature":1.0,"reasoning_tokens":1626,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T23:32:38.758676+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A specific concern raised by one of the discussants that falls outside all four themes and is left unaddressed in the rejoinder.","supporting_citations":[],"review_version":1}