{"id":"0eb3c257-e361-4d57-ac7d-e1f6a022a739","arxiv_id":"2501.11918","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A RoBERTa ensemble with inverse perplexity weighting achieved 0.826 TPR (non-adversarial) and 0.801 TPR (adversarial) in a cross-domain AI-text detection shared task.","lead":"Researchers combined two fine-tuned RoBERTa models with a perplexity-based weighting scheme to detect AI-generated text across many domains. Their system placed 10th among 23 detectors on a shared task, but the claimed improvement from the weighting appears in only one of the two subtasks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts the abstract's 'both subtasks' claim: the inverse-perplexity ensemble beats the single RoBERTa only in Subtask A (0.826 vs 0.813) and loses in Subtask B (0.760 vs 0.801), so the headline generalization claim is unsupported.","rationale":"The reader's verdict is already CONDITIONAL, and I agree with that verdict. My concern is slightly different in emphasis: the most load-bearing problem is not only the possible test-label leakage in Eqs. 3.4.1-3.4.2, but the direct contradiction in Table 3, where the inverse-perplexity ensemble loses to the single model in Subtask B. This contradiction makes the abstract's 'in both' claim false on the paper's own official numbers, even before considering whether the Subtask A weights were fitted to the test set. The leakage issue remains important because it is the only way the one favorable comparison could be salvaged, and the paper does not specify that perplexity was computed on a held-out split.\n\nI am not changing the verdict because the paper is a shared-task system description with externally evaluated rankings; those rankings are credible evidence of the system's absolute performance. The problem is the causal claim about inverse perplexity weighting, which can be fixed by a restrained revision: report the Subtask A comparison with held-out weights, acknowledge that the ensemble underperformed in Subtask B, and remove the unsupported 'both subtasks' generalization. The absence of code and the unspecified 10% RAID split are secondary but reinforce the need for a conditional acceptance with a reproducibility requirement.","tokens_in":7128,"tokens_out":4892,"duration_ms":48646,"concrete_test":"Compute the Subtask B AGG TPR for the FT RoBERTa + RoBERTa OpenAI ensemble with weights estimated only on the RAID validation split (perplexity from true validation labels, or from training labels if no validation split), using the official evaluation script. Compare with the single FT RoBERTa row's 0.801 in Table 3. If the held-out-weighted ensemble does not exceed 0.801, the abstract's 'both non-adversarial and adversarial' claim is contradicted by the paper's own official numbers. If it does exceed 0.801, the Table 3 discrepancy is evidence that the published ensemble row used test-label perplexity weights, and the Subtask A advantage must be re-estimated on held-out weights as well.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that inverse perplexity weighting (IPW) improves generalization and performance in both non-adversarial and adversarial MGT detection. For that to be true, the weighted ensemble must beat the best single component in both subtasks under a valid weight-estimation procedure. The paper's own Table 3 shows the opposite for Subtask B: the weighted FT RoBERTa + RoBERTa OpenAI ensemble scores AGG TPR 0.760, while the single FT RoBERTa scores 0.801. Thus the adversarial half of the claim is refuted by the submitted official results, independent of any methodological subtlety.\n\nThe only comparison that favors IPW is Subtask A (0.826 vs 0.813). That comparison is itself questionable because Eqs. 3.4.1-3.4.2 define perplexity from p(y_i|x_i) with 'true labels' and 'test samples.' If the weights were fit to the test labels, the 0.826 figure is an in-sample estimate, not evidence of generalization. Section 3.4 does not state that these perplexities were computed on a held-out validation split, and no code or data-split specification is provided to rule this out.\n\nSeparately, the 10% RAID training subset is described without a reproducible seed or sampling script, so the exact training distribution cannot be reconstructed. This is secondary to the main claim but blocks independent replication of the single-model baselines.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes LuxVeri's submission to the GenAI Content Detection Workshop (COLING-2025), Task 3 on cross-domain machine-generated text (MGT) detection. The approach fine-tunes RoBERTa-base models and combines them with a RoBERTa-base OpenAI detector using inverse perplexity weighting, where each model's weight is the normalized inverse of its perplexity on the evaluation data. For Subtask A (non-adversarial), the weighted two-model ensemble achieves an aggregate TPR of 0.826 (rank 10/23); for Subtask B (adversarial), a single fine-tuned RoBERTa achieves 0.801 (rank 8/22). The paper claims that inverse perplexity weighting improves generalization and performance in both subtasks, but the adversarial results in Table 3 show the weighted ensemble underperforming the single model, and the weighting procedure uses true test labels to compute perplexity, which makes the reported ensemble result an in-sample estimate rather than a genuine generalization result.","tokens_in":7449,"tokens_out":6252,"duration_ms":59902,"significance":"If the inverse perplexity weighting were evaluated on held-out data and shown to improve over both single models and unweighted ensembles, the method would be a simple, inexpensive way to combine detectors for cross-domain MGT detection. The paper is transparent about its training configuration, reports per-generator TPRs that reveal failure modes (e.g., Cohere) beyond aggregate scores, and explicitly lists limitations such as computational constraints and restricted multilingual coverage. However, as written, the central methodological claim is not supported: the weighting is fitted to the test labels, the adversarial half of the claim is contradicted by the paper's own Table 3, and the non-adversarial comparison confounds the weighting with the addition of a second model. These issues are load-bearing rather than cosmetic, so the paper requires substantive revision before its contribution can be assessed.","major_comments":[{"comment":"The perplexity P is computed as exp(-1/N Σ_i log p(y_i | x_i)), where the text states that y_i is the true label and N is the number of test samples. The weights w_i in Eq. (3.4.2) are then derived from these test-set perplexities. This means the ensemble weights are fitted to the target labels of the evaluation set, so the reported ensemble AGG TPR (e.g., 0.826 for Subtask A) is an in-sample estimate. The comparison in Table 3 between the weighted ensemble and the single FT RoBERTa is therefore not a valid out-of-sample comparison unless the authors demonstrate that the perplexities were computed on an independent validation split and specify the split exactly. The manuscript must clarify this point and, if the weights were indeed computed on test labels, either recompute the results using held-out perplexities or explicitly reframe the reported numbers as in-sample.","section":"Section 3.4.1–3.4.2, Eq. (3.4.1)"},{"comment":"The abstract claims that inverse perplexity weighting 'enhanc[es] generalization and performance in both non-adversarial and adversarial MGT detection.' This is contradicted by the Adversarial Results block of Table 3: the weighted two-model ensemble FT RoBERTa + RoBERTa OpenAI scores AGG TPR 0.760, and the three-model ensemble scores 0.749, both below the single FT RoBERTa's 0.801. In the Non-Adversarial block, the weighted ensemble beats the single model (0.826 vs. 0.813), but the comparison is confounded because the ensemble adds a second model (RoBERTa OpenAI) in addition to the weighting; the improvement cannot be attributed to inverse perplexity weighting alone. The authors should either revise the claim, provide an ablation with simple averaging of the same two models, or explicitly state that the adversarial Subtask B system is a single model and that the weighting did not help there.","section":"Abstract and Table 3"},{"comment":"The Discussion states: 'For non-adversarial tasks, we explored an inverse perplexity-based ensemble approach. However, the detectors in this ensemble underperformed compared to the fine-tuned RoBERTa model.' This directly contradicts Section 4.1 and the abstract, which report that the weighted ensemble FT RoBERTa + RoBERTa OpenAI achieved the highest AGG TPR (0.826) and outperformed FT RoBERTa (0.813) in the non-adversarial setting. As printed, the paper simultaneously asserts and denies the effectiveness of the ensemble in the same subtask. This internal inconsistency must be resolved by correcting the Discussion, the Results, or both.","section":"Section 6 (Discussion and Conclusion)"},{"comment":"The 10% RAID training subset is described only as 'balanced' across genres, decoding strategies, attacks, and domains, with no random seed or sampling script. The totals in Table 1 sum to 518,469 samples, but the RAID-train set is described as containing 802 million words; the relationship between the 10% subset and the full corpus is not documented precisely. Without the exact sampling procedure, the single-model baselines and the ensemble results cannot be independently reproduced. Provide the seed, the sampling script, or a detailed algorithm, or make the subset available.","section":"Section 2.1 and Table 1"}],"minor_comments":[{"comment":"The citation 'Hu et al., 2020' for the XTREME benchmark lists the authors as 'Xiang Hu, Xuezhe Liu, Peng He, et al.'; the correct first author is Junjie Hu, and the reference entry should be corrected.","section":"Section 2.2 and References"},{"comment":"The text states that 'A fixed random seed was maintained to ensure reproducibility,' but no seed value is provided; specify the actual seed(s) used for data sampling, model initialization, and training.","section":"Section 3.2"},{"comment":"The normalization formula is visually ambiguous; use explicit parentheses, e.g., w_i = (1/(P_i − 1)) / (Σ_j 1/(P_j − 1)), to avoid misreading.","section":"Section 3.4.2, Eq. (3.4.2)"},{"comment":"The phrase 'the fine-tuned RoBERTa + RoBERTa OpenAI model which was fine-tuned on RAID dataset' is ambiguous about which components receive fine-tuning; specify whether the OpenAI detector weights are fine-tuned or kept frozen.","section":"Section 4.1, first paragraph"},{"comment":"The row labeled 'FT RoBERTa + RoBERTa OpenAI' is the inverse-perplexity weighted ensemble, but the caption does not say so; add a note to clarify that this row corresponds to the proposed weighting method.","section":"Table 3 caption"},{"comment":"The entry 'SuperAnnotate. 2024. Ai detector.' is a product webpage rather than a scientific reference; provide a full URL with access date or a more formal citation if one exists.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a shared-task system description in a workshop format, but the central methodological claim of inverse perplexity weighting improving both subtasks is undermined by test-set fitting and by the paper's own adversarial results. The Discussion even contradicts the Results section. I recommend requiring the authors to (a) clarify whether the perplexity was computed on a held-out validation split, (b) recompute or relabel the results if it was not, (c) reconcile the Discussion contradiction, and (d) provide the exact data-sampling procedure. If the results change materially after these corrections, the paper's contribution may be limited to a single-subtask positive result or a negative result, and the abstract should be adjusted accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a shared-task system description whose headline claim is refuted by its own table. The inverse-perplexity ensemble beats a single RoBERTa in Subtask A (0.826 vs 0.813) but loses in Subtask B (0.760 vs 0.801), so “enhancing generalization and performance in both subtasks” is simply not supported.\n\nWhat’s here: a straightforward fine-tuned RoBERTa + OpenAI detector ensemble with inverse perplexity weighting. The official rankings and per-generator TPRs are reported transparently, training hyperparameters are listed, and the limitations section acknowledges computational constraints and excluded model families. That’s the honest skeleton of a workshop system description.\n\nThe soft spots are load-bearing. First, the abstract’s claim about both subtasks is false. The submitted Subtask B system is the single RoBERTa, not the weighted ensemble, and the weighted ensemble scores below the single model on the adversarial set. The conclusion even says “the detectors in this ensemble underperformed compared to the fine-tuned RoBERTa model” in the non-adversarial task, which directly contradicts Table 3. Second, Eqs. 3.4.1–3.4.2 compute perplexity using true labels on test samples. If taken literally, the ensemble weights are fitted to the target, making the 0.826 an in-sample number. The paper doesn’t say the perplexities were computed on a held-out split, and no code or data split is provided to rule it out. Third, the 10% RAID subset has no seed or sampling script, so the training data can’t be reconstructed.\n\nNone of these are minor. But they are fixable with a revision: drop the “both subtasks” claim, clarify where the perplexity weights come from, and add the split details. The official TPR numbers are likely fine; it’s the interpretation that’s broken.\n\nWho this is for: readers of GenAI shared-task proceedings who want a quick record of one team’s configuration. It is not a methodological advance.\n\nRecommendation: send it to peer review, not desk reject, because the test-set-fitting issue and the contradictory claim deserve referee eyes. A competent referee can turn this into an honest system description.","headline":"Shared-task system description whose headline claim is contradicted by its own Table 3, and whose weight equations look like test-set fitting; fixable, but not as is.","tokens_in":7978,"tokens_out":3099,"would_cite":false,"duration_ms":28898,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that inverse perplexity weighting of an ensemble improves cross-domain detection of AI-written text, and reports mixed evidence across its two tracks.","keywords":["machine-generated text detection","cross-domain detection","inverse perplexity weighting","ensemble soft voting","RoBERTa fine-tuning","adversarial detection","RAID benchmark"],"falsifier":"Re-run the same two models on the same 10% RAID subset, choose inverse-perplexity weights using only a held-out validation split, and then score the official test set; if the weighted ensemble no longer beats the single fine-tuned RoBERTa on the non-adversarial track (or improves on the adversarial track), the reported advantage is an artifact of weighting with test labels.","tokens_in":6915,"feed_emoji":"🤖","tokens_out":7755,"duration_ms":72523,"temperature":0.7,"pith_summary":"The paper enters a shared cross-domain machine-generated-text (MGT) detection task and proposes a weighted ensemble of two fine-tuned RoBERTa-base classifiers. Each model's vote is scaled by the inverse of its perplexity on the evaluation data, so the model that assigns the highest probability to the true labels gets the largest say. On the non-adversarial track the ensemble reaches 0.826 aggregate true-positive rate (TPR), beating a single fine-tuned RoBERTa at 0.813 and ranking 10th out of 23 teams. On the adversarial track the single RoBERTa reaches 0.801, ranking 8th out of 22, while the weighted ensemble scores only 0.760. The paper's thesis is that inverse-perplexity weighting improves cross-domain generalization; the reported tables show that benefit in one track and a penalty in the other.","feed_headline":"Inverse perplexity weights help one AI-text track, hurt the other","feed_subtitle":"Paper's own numbers: ensemble scores 0.826 vs 0.813 on non-adversarial text, 0.760 vs 0.801 on adversarial text.","key_machinery":"The carrying mechanism is inverse-perplexity-weighted soft voting. For each model $i$, perplexity is $P_i = \\exp\\left(-\\frac{1}{N}\\sum_{j=1}^N \\log p(y_j \\mid x_j)\\right)$, where $p(y_j \\mid x_j)$ is the model's predicted probability of the true label $y_j$ for sample $x_j$; the weight is $w_i = \\frac{1/(P_i-1)}{\\sum_{k=1}^{M} 1/(P_k-1)}$, and the ensemble score for class $c$ is $\\sum_i w_i \\, p_i(c)$. The adjustment $P_i-1$ and the inverse make lower-perplexity (more confident) models dominant in the final soft vote.","core_discovery":"The central claim, as the authors would state it, is that combining detectors by inverse-perplexity weighting yields better cross-domain MGT detection than any single fine-tuned model. The perplexity of each model is computed from its predicted probability of the true label over the evaluation samples; a lower perplexity is read as higher confidence, and the normalized inverse of (perplexity minus one) is used as the voting weight. The paper demonstrates the mechanism with two fine-tuned RoBERTa-base models, one stock and one trained to function as an AI-text detector, and reports 0.826 aggregate TPR for the ensemble in non-adversarial conditions. In adversarial conditions, the same ensemble underperforms the single fine-tuned RoBERTa (0.760 vs 0.801), which the paper notes in its conclusion while still crediting the weighting scheme with improved generalization.","pith_inferences":["If perplexity were computed from the text alone (a language model's own perplexity) rather than from the detector's true-label confidence, the weighting would become a zero-shot ensemble rule; this variant is directly testable on the same RAID data and would remove the need for labels at weighting time.","The inverse-$(P-1)$ formula is only one of many monotone transforms of confidence; comparing it with equal weights, log-probability weights, or softmax-temperature weights on a held-out validation set would isolate whether the specific formula, rather than ensembling itself, deserves the credit.","The paper does not report a repeatable seed or selection script for its 10% RAID subset, so an independent replication currently cannot separate the method's effect from the particular data draw; publishing that selection would make the reported gains reproducible.","In adversarial settings, the single model's higher score suggests that inverse-perplexity weighting amplified a confidently wrong member; a detector could guard against this by down-weighting models whose confidence and accuracy diverge under attack."],"forward_implications":["If the claim holds, any set of detectors can be combined by this confidence-weighted vote without extra training, making small two-model ensembles competitive with larger systems.","The non-adversarial result implies the weighted ensemble is a safe default when inputs are unmodified; the adversarial result implies the weighting should be disabled or re-tuned when inputs may be attacked.","The two-model ensemble with inverse-perplexity weighting is cheap to compute, so the method can be applied on top of already-fine-tuned checkpoints rather than requiring a new training run.","The per-generator scores show the method is not uniform: strong on ChatGPT, GPT-3, and GPT-4, weak on Cohere and Mistral, so cross-domain robustness still depends on the training distribution."],"supporting_citations":[{"why":"Provides the RAID corpus used to fine-tune both RoBERTa models and to evaluate aggregate TPR; the method is trained and scored on it.","marker":"(Dugan et al., 2024)"},{"why":"Supplies the RoBERTa-base architecture that both ensemble components are fine-tuned from.","marker":"(Liu et al., 2019)"},{"why":"Supplies the released detector that the second RoBERTa ensemble component is built on for non-adversarial detection.","marker":"(Solaiman et al., 2019)"},{"why":"Defines the shared task's subtasks, evaluation splits, and aggregate true-positive-rate metric that the reported ranks are based on.","marker":"(Dugan et al., 2025)"},{"why":"Provides the Binoculars zero-shot detector used as a baseline in the non-adversarial comparison table.","marker":"(Hans et al., 2024)"},{"why":"Establishes the prior multidomain, multimodel MGT detection task whose setup motivates this paper's cross-domain design.","marker":"(Wang et al., 2024)"},{"why":"Supplies evidence that RoBERTa-based detectors transfer across domains with few labels, justifying the fine-tuning choice.","marker":"(Rodriguez et al., 2022)"}],"fun_headline_variants":["Inverse perplexity weighting: helps on one AI-text track, hurts on another","Ensemble's inverse perplexity weighting improves non-adversarial, weakens adversarial","Perplexity-weighted ensemble: gain for clean AI-text, loss for adversarial","Inverse perplexity boosts non-adversarial AI-text detection but degrades adversarial"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the true labels used to compute each model's perplexity for weighting come from a held-out validation set rather than from the test set being scored.","fun_headline_variants_meta":{"raw":{"variants":["Inverse perplexity weighting: helps on one AI-text track, hurts on another","Ensemble's inverse perplexity weighting improves non-adversarial, weakens adversarial","Perplexity-weighted ensemble: gain for clean AI-text, loss for adversarial","Inverse perplexity boosts non-adversarial AI-text detection but degrades adversarial"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2973,"prompt_tokens":937,"completion_tokens":2036,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":1950}},"tokens_in":553,"tokens_out":2036,"duration_ms":17734,"temperature":1.0,"reasoning_tokens":1950,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:42:24.876734+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same two models on the same 10% RAID subset, choose inverse-perplexity weights using only a held-out validation split, and then score the official test set; if the weighted ensemble no longer beats the single fine-tuned RoBERTa on the non-adversarial track (or improves on the adversarial track), the reported advantage is an artifact of weighting with test labels.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RAID corpus used to fine-tune both RoBERTa models and to evaluate aggregate TPR; the method is trained and scored on it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the shared task's subtasks, evaluation splits, and aggregate true-positive-rate metric that the reported ranks are based on."}],"review_version":1}