{"id":"0571503f-d73f-4c99-a4bb-a6b8259fce36","arxiv_id":"2608.07641","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SurveyReview is a dataset of 675 surveys with 1,630 reviews annotated into four quality dimensions, plus a fine-tuned evaluator (SurveyAlign) that reports lower error than GPT-5.2 but only reaches majority-class baseline performance.","lead":"This paper introduces SurveyReview, a dataset of 675 survey papers and 1,630 peer-review reports converted into four scored dimensions (readability, criticalness, comprehensiveness, structure) to test whether AI evaluators agree with human reviewers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported improvement over GPT-5.2 is not evidence of reviewer alignment: on the released score distribution a constant predictor that always outputs -1 achieves AVG MSE 1.44 and AVG MAE 0.68, essentially matching SurveyAlign's 1.38/0.69.","rationale":"The central claim is that SurveyAlign 'substantially improves reviewer alignment' over GPT-5.2. The most direct way to test that claim is to ask whether the reported error values are better than what a trivial predictor achieves on the benchmark's own label distribution. They are not: the always-minus-one predictor matches SurveyAlign on both aggregate metrics. This is load-bearing regardless of whether the annotated labels faithfully represent the original reviewers, because even under perfect annotation the benchmark cannot discriminate an alignment-tuned model from a constant rule. The annotator-fidelity issue raised by the reader is also real, but the majority-class baseline is the more decisive and more easily testable problem. The paper reports no significance testing, so the 0.06 MSE gap could easily be noise. The paper does provide useful materials, such as the dataset construction and reproducibility-oriented split, but the headline empirical result is not supported by the presented evidence.","tokens_in":17460,"tokens_out":5936,"duration_ms":57228,"concrete_test":"Insert a 'Majority (-1)' row into Table 2 computed on the released test split, with per-dimension and average MSE/MAE, and run a paired bootstrap over the 414 test instances comparing SurveyAlign to this baseline on AVG MSE and AVG MAE with 95% confidence intervals. If SurveyAlign's AVG MSE is not significantly below the baseline and its AVG MAE is not significantly below or equal, the paper's claim of improved reviewer alignment should be rejected.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.4 reports the overall label distribution: -2: 4.46%, -1: 66.25%, +1: 24.73%, +2: 4.56%. A constant evaluator that always predicts -1 therefore has expected MSE = 0.0446*1 + 0.2473*4 + 0.0456*9 = 1.44 and expected MAE = 0.0446*1 + 0.2473*2 + 0.0456*3 = 0.68. Table 2 reports SurveyAlign's average MSE/MAE as 1.38/0.69, so SurveyAlign is 0.06 lower in MSE and 0.01 worse in MAE than a predictor that never reads the survey. The claimed 'substantial improvement' over GPT-5.2 (2.28/1.15) is therefore consistent with the model learning to exploit the skewed label distribution rather than achieving reviewer-aligned assessment. Per-dimension baselines are absent; Table 6 shows all dimension means are negative (-0.17 to -0.40), so -1 is plausibly the modal label everywhere, and at least Criticalness (MSE 1.52 in Table 2) is worse than the aggregate majority baseline. The lack of any majority baseline, confidence intervals, or significance tests makes the central empirical claim unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SurveyReview, a benchmark and dataset for evaluating automatic survey-paper evaluators. It collects 675 survey papers with 1,630 peer-review reports from F1000Research, MOPRD, and OpenReview, and annotates each review into four dimensions (Readability, Criticalness, Comprehensiveness, Structure) with numeric scores from {-2,-1,+1,+2} and supporting rationales. The authors also propose SurveyAlign, a Qwen3-32B-based evaluator fine-tuned with LoRA and knowledge-augmented inputs, and report that it reduces average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 compared with zero-shot GPT-5.2. The paper also defines a combined Human-aligned Score (HAS) integrating score error and rationale quality.","tokens_in":17755,"tokens_out":3547,"duration_ms":35132,"significance":"If the central claims hold, the dataset would be a useful resource: it is the first multi-dimensional benchmark derived from authentic peer-review reports, and it ships standardized splits, an evaluation protocol, and an open-source code/data release. The annotation reliability check (Krippendorff's alpha 0.74) and the robustness analyses (source-wise, multi-judge RQS, HAS weighting) are valuable additions that go beyond what many benchmark papers provide. However, the headline result — that SurveyAlign substantially improves reviewer alignment over zero-shot GPT-5.2 — is not supported by the evidence as presented, because no trivial baseline is included. The benchmark's claim to measure 'reviewer alignment' also rests on an unvalidated conversion from free-form review text to four-dimensional scores by third-party annotators.","major_comments":[{"comment":"The annotation space initially includes 0 ('dimension not mentioned') and -3 ('unjudgeable'), which are then excluded from training and evaluation. This exclusion is not justified. If a review does not mention a dimension, that absence may be informative signal for a dimension-specific evaluator, and excluding it arbitrarily removes a part of the review signal. The authors should discuss this decision and, if possible, provide a secondary analysis that includes 0 as a valid label.","section":"Appendix A"}],"minor_comments":[{"comment":"The description of Crit-KA thresholding says 'α and β are fixed hyperparameters' but does not give their values in the main text or in Appendix F. Without concrete values the procedure is not reproducible; please report them.","section":"§5.2.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is potentially valuable, and the reproducibility measures (code, data, standardized splits, reliability analysis) are commendable. However, the central empirical claim is currently unsupported because the improvement over GPT-5.2 is essentially matched by a constant -1 predictor. The authors need to add majority baselines and significance tests; if SurveyAlign does not beat the trivial baseline, the evaluator contribution should be reframed or removed. The 'reviewer-aligned' terminology is also risky without validation against original reviewer judgments. I would only consider acceptance after these issues are resolved; the annotation-validity issue may require new data or a careful reframing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline claim in this paper doesn't survive contact with a trivial baseline. SurveyAlign's average MSE 1.38/MAE 0.69 on the test set is essentially matched by a constant predictor that always outputs -1, which gets 1.44/0.68 given the label distribution (66% -1). So the reported 'substantial improvement' over GPT-5.2 is most plausibly the model learning the label skew, not aligning with reviewers. That's the thing to know.\n\nThat said, the resource itself is genuinely new and useful. The authors collected 675 survey papers with 1,630 real peer-review reports from three sources, converted the free-form comments into four-dimension scores (Readability, Criticalness, Comprehensiveness, Structure) plus rationales, and released standardized splits and an evaluation protocol. That is a real step beyond SurveyScope, SurGE, and DeepSurvey-Bench, which don't use actual reviewer reports. The annotation work looks careful: Krippendorff's alpha 0.74 on 280 units, with most disagreements local. Code and data are promised. I'd use the dataset.\n\nThe soft spots are where the paper's empirical claims live. The missing majority baseline is not a minor omission; it undermines the central conclusion. The ground-truth is also annotator interpretation of reviewer comments, not the reviewers' own scores, and that conversion is never validated against actual reviewers. The RQS judge uses GPT-5.2 against reference rationales from the same rubric used for training, which is mildly circular; the human meta-evaluation on 50 pairs helps, but it's small. There are no confidence intervals or significance tests, so the 0.06 MSE gap to the majority predictor could easily be noise.\n\nIn short: the benchmark is a contribution worth refereeing, but the evaluator claims need major revision. I'd send it to review, but with a clear request for a majority-class baseline, validation of the annotation conversion, and significance testing. As it stands, the paper's main quantitative conclusion is not established.","headline":"A valuable dataset resource whose headline evaluator claim collapses against a trivial majority-class baseline.","tokens_in":18329,"tokens_out":2004,"would_cite":false,"duration_ms":17790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper converts 1,630 authentic peer-review reports into four quality dimensions and shows fine-tuning on them cuts an evaluator's average score error roughly in half.","keywords":["survey evaluation","LLM-as-a-judge","peer review","benchmark dataset","human alignment","knowledge augmentation","automatic evaluation","Qwen3-32B"],"falsifier":"Collect the original reviewers' own dimension-level ratings, or ask a sample of the original reviewers to rate their own reviews with the four-dimension rubric, and compare them with the annotator-derived ground truth. If agreement between the original reviewers and the annotations is no better than agreement between the automated evaluators and the annotations, then the benchmark measures annotator alignment rather than reviewer alignment, and the reported error reductions do not establish the paper's claim.","tokens_in":17227,"feed_emoji":"⚖️","tokens_out":8489,"duration_ms":67192,"temperature":0.7,"pith_summary":"The paper argues that off-the-shelf LLM-as-a-judge evaluation of survey papers is misaligned with what human reviewers actually judge, and that alignment can be measured and improved. It introduces SurveyReview, a dataset of 675 surveys with 1,630 peer-review reports annotated into four dimensions—readability, structure, comprehensiveness, criticalness—each with a numeric score and a supporting rationale. It also introduces SurveyAlign, a Qwen3-32B model fine-tuned with LoRA on this data, which on the test set cuts average MSE from 2.28 to 1.38 and MAE from 1.15 to 0.69 compared with GPT-5.2 zero-shot prompting. If the annotation conversion is faithful, the benchmark offers a reproducible target for future survey evaluators.","feed_headline":"Fine-tuned model beats GPT-5.2 at matching human survey reviewers","feed_subtitle":"A new benchmark turns real peer-review comments into four quality scores for automatic survey evaluators.","key_machinery":"The central object is the four-dimensional score–rationale representation derived from free-form peer reviews: annotators read each review and assign, for every nonzero dimension among Readability, Structure, Comprehensiveness, and Criticalness, a score from {−2, −1, +1, +2} with the most relevant sentences as supporting rationale. This structured ground truth powers two things: supervised fine-tuning of the evaluator, and the alignment metrics themselves—MSE/MAE for scores, a Reason Quality Score for rationales judged by GPT-5.2, and the composite Human-aligned Score. SurveyAlign realizes the mapping with four dimension-specific LoRA-fine-tuned Qwen3-32B sub-models, prepending a reference-expanded citation neighborhood (Comp-KA) for comprehensiveness and a temporally compressed summary (Crit-KA) plus DPO for criticalness, then fusing five sampled outputs by majority vote at inference.","core_discovery":"The central claim is that authentic peer-review reports can be converted into structured, four-dimensional score–rationale annotations, and that supervised fine-tuning on those annotations produces an evaluator whose scores align with human reviewers substantially better than zero-shot prompting. SurveyAlign, built from Qwen3-32B with LoRA, uses dimension-specific sub-models, reference-expansion knowledge augmentation for comprehensiveness and criticalness, and majority-vote aggregation at inference. On the held-out test set it reports average MSE 1.38 and MAE 0.69 across the four dimensions, versus 2.28 and 1.15 for GPT-5.2 prompting, and the highest composite Human-aligned Score (0.74). The paper presents this as the first multi-dimensional reviewer-aligned benchmark and a strong baseline for the task.","pith_inferences":["If the free-form-to-score conversion is accepted, the same annotation pipeline could be applied to other genres—research papers, technical reports—yielding reviewer-aligned benchmarks beyond surveys.","The criticalness result suggests a general principle: giving an evaluator more external context does not help unless the training objective also rewards the target reasoning behavior.","Per-dimension MSE profiles give a diagnostic: an evaluator's error pattern identifies which aspect of reviewing it fails to model, enabling targeted fixes instead of tuning one aggregate score.","A natural next test is whether improving alignment on this benchmark transfers to downstream survey generation, for instance by using the evaluator's scores as a reward signal for iterative revision."],"forward_implications":["SurveyAlign attains the lowest MSE and MAE on every one of the four dimensions and the highest HAS (0.74) among all evaluated methods, including GPT-5.2, Gemini-3-pro, and Claude-Opus-4.5.","Task-specific supervision is necessary for reviewer-level consistency: every zero-shot LLM judge, even the strongest, lags behind on score alignment.","Knowledge augmentation helps comprehensiveness (MSE 1.36 to 1.26 with Comp-KA) but helps criticalness only when combined with DPO (MSE 1.96 to 1.52); Crit-KA alone does not improve the dimension.","RQS is judge-sensitive, yet SurveyAlign remains the top-ranked method under three different rationale judges and across all three data sources, so the system-level conclusion is stable.","The paper-level, source-stratified train/test split and fixed evaluation protocol allow future evaluators to be compared on the same MSE/MAE/RQS/HAS scales."],"supporting_citations":[{"why":"supplies the MOPRD reviews, one of the three data sources for the survey–review pairs.","marker":"[14]"},{"why":"documents the F1000Research open peer review model whose reports form the largest share of the dataset.","marker":"[25]"},{"why":"supports treating OpenReview as a community data source for the review reports.","marker":"[22]"},{"why":"is the Qwen3-32B base model that SurveyAlign fine-tunes.","marker":"[29]"},{"why":"is the LoRA method used for parameter-efficient adaptation.","marker":"[9]"},{"why":"is the GPT-5.2 zero-shot baseline that SurveyAlign is compared against.","marker":"[15]"},{"why":"is the AMiner citation graph used to expand reference neighborhoods for knowledge augmentation.","marker":"[24]"},{"why":"is the DPO algorithm combined with Crit-KA for criticalness training.","marker":"[18]"}],"fun_headline_variants":["SurveyAlign: fine-tuned evaluator beats GPT-5.2 at matching human reviews","Fine-tuned Qwen3-32B outperforms GPT-5.2 in survey evaluation alignment","Benchmark turns peer reviews into training data for survey judges","New dataset and baseline for aligning AI survey evaluators with humans","Fine-tuned evaluator matches human reviewers better than GPT-5.2"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the scores and rationales annotators extract from free-form review comments faithfully represent what the original human reviewers meant on each of the four dimensions; the paper does not validate these annotations against the reviewers' own ratings.","fun_headline_variants_meta":{"raw":{"variants":["SurveyAlign: fine-tuned evaluator beats GPT-5.2 at matching human reviews","Fine-tuned Qwen3-32B outperforms GPT-5.2 in survey evaluation alignment","Benchmark turns peer reviews into training data for survey judges","New dataset and baseline for aligning AI survey evaluators with humans","Fine-tuned evaluator matches human reviewers better than GPT-5.2"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001485,"raw_usage":{"total_tokens":6001,"prompt_tokens":1018,"completion_tokens":4983,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":4881}},"tokens_in":634,"tokens_out":4983,"duration_ms":29309,"temperature":1.0,"reasoning_tokens":4881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:26:44.256711+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect the original reviewers' own dimension-level ratings, or ask a sample of the original reviewers to rate their own reviews with the four-dimension rubric, and compare them with the annotator-derived ground truth. If agreement between the original reviewers and the annotations is no better than agreement between the automated evaluators and the annotations, then the benchmark measures annotator alignment rather than reviewer alignment, and the reported error reductions do not establish the paper's claim.","supporting_citations":[{"cited_title":"Does the use of open, non-anonymous peer review in scholarly publishing introduce bias? Evidence from the F1000 post-publication open peer review publishing model","cited_arxiv_id":"1911.03379","evidence_quote":"documents the F1000Research open peer review model whose reports form the largest share of the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the GPT-5.2 zero-shot baseline that SurveyAlign is compared against."}],"review_version":1}