{"id":"c0e322b8-1304-4e76-80f7-e87ddd5998e4","arxiv_id":"2505.18136","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A single multilingual language model serializes all Wikidata edits into text and outperforms the production ORES system in vandalism detection and fairness.","lead":"Wikidata vandalism detection currently misses many edit types, so the authors built Graph2Text, which converts every kind of Wikidata edit into text and scores suspicious revisions with one multilingual language model. The new system beats the production ORES model on accuracy and fairness, and the code and a large dataset are released openly.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline AUC and fairness advantages over ORES are measured on a filtered 'reverted' proxy that discards 57.7% of reverts; the paper does not demonstrate the comparison is invariant to that filtering, so the claimed superiority may be an artifact of label definition.","rationale":"The reader's weakest assumption is that the filtered 'reverted' signal is a clean proxy for vandalism; I agree that this is the load-bearing point. I add two specifics that make the concern concrete rather than generic: the expert-labeled sample used to validate the proxy is itself selected by score bins from both models, so its AUC estimates are not population estimates and cannot fully de-risk the label definition; and the holdout's position at the end of the collection window introduces possible censoring because recent revisions have had less time to be reverted. The fairness claims have a similar fragility, as they are reported without confidence intervals or significance tests. None of this is a reason to reject the paper: the system is well-engineered, the dataset is released, and the large holdout AUC gap is substantial. But the central 'significantly outperforms' claim is conditional on the filtering choice being neutral, which is not demonstrated. The reader's CONDITIONAL verdict remains the right call, so no verdict change is needed; an unbiased expert-labeled sample would settle whether the concern actually lands.","tokens_in":11530,"tokens_out":13155,"duration_ms":125784,"concrete_test":"Draw a simple random sample of 2,000 revisions from the unfiltered holdout, including self-reverts and edit-war revisions, have blind experts label each as Keep, Revert, or Not Sure, and compute Graph2Text and ORES AUCs with inverse-probability sampling weights. As a secondary check, run both models on the 57.7% of filtered-out reverted revisions and report AUC on that subset. Also report bootstrap confidence intervals for DIR and DAUC on the unbiased sample. If Graph2Text's AUC advantage over ORES persists on the weighted gold sample and is not reversed on the filtered-out subset, the filtering concern is resolved; if the gap narrows, disappears, or flips, the headline superiority is an artifact of the filtered revert proxy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Graph2Text 'significantly outperforms' ORES in both performance and fairness (Section 6). The only large-scale training and evaluation label is a binary 'was reverted' signal, after Appendix B.2 removes self-reverts and edit-war revisions. That filter deletes 57.7% of all initially reverted revisions. Self-reverts are plausibly non-vandalism, but edit wars contain both vandalism and good-faith counter-reverts; dropping the whole class changes the target from 'revert' to 'revert outside conflict contexts.' ORES is a production model trained on its own historical label distribution, while Graph2Text is trained and tuned on the filtered target, so the benchmark can reward fitting the filter rather than detecting vandalism. The expert-labeled evaluation (Table 2) is meant to address label noise, but it is not a simple random sample: 50 revisions are drawn from each score bin for each model separately, so selection probabilities depend on both models' scores and the reported AUCs are not population estimates. The holdout is also the last three months of the collection window, so recent revisions have less opportunity to be reverted; no censoring correction or post-collection observation window is described. Finally, the fairness metrics in Table 3 are point estimates without confidence intervals, and Section 6 calls the difference 'significantly smaller' without a test. If the filter is label-irrelevant, the conclusions stand; if it correlates with edit-war or slow-revert vandalism, the magnitude and even the direction of the Graph2Text-versus-ORES gap could change.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Graph2Text, a Wikidata vandalism detection system that serializes structured triple edits and textual edits into text, processes them with a fine-tuned multilingual BERT (LMC), aggregates per-change scores, and combines them with revision metadata via a CatBoost classifier. Using 'reverted' as a proxy label, the authors filter out self-reverts and edit-war revisions in Appendix B.2, adopt a time-based holdout of the last three months, and evaluate against ORES and two feature-ablation baselines. They report higher holdout AUC (0.924 vs. 0.859), better filter rates, and favorable fairness metrics for anonymous and new users, and they release the code and a ~4.8M-revision dataset. The paper frames the contribution as a next-generation production system that outperforms the current ORES model in both accuracy and fairness.","tokens_in":11871,"tokens_out":4041,"duration_ms":34778,"significance":"If the performance and fairness claims hold, the work offers a practically relevant improvement for a large production knowledge graph and strengthens the evidence that content-based transformer features can complement or replace metadata-heavy classifiers in vandalism detection. The open dataset and code are valuable community assets, and the manuscript includes several methodical elements: a time-based split, bootstrapped confidence intervals, multiple baselines, and an expert-labeled evaluation subset. However, the strength of the central claim—'significantly outperforms ORES in both performance and fairness'—depends on evaluation choices that are not fully validated, particularly the heavy filtering of the revert label, the non-random expert sample, and the absence of statistical tests for the fairness comparison.","major_comments":[{"comment":"The headline comparison against ORES is measured on a 'reverted' label that has been filtered to remove self-reverts and edit-war revisions, which discards 57.7% of all initially reverted revisions. The manuscript does not demonstrate that the relative ranking of Graph2Text versus ORES is invariant to this filtering, and because ORES is a production model trained on its own historical label distribution, the benchmark may reward models that fit the filter rather than detect vandalism. Please add a robustness analysis that evaluates both models on the unfiltered reverted set (or on per-filter-category subsets) and report whether the AUC and filter-rate advantages persist.","section":"§5.2 / Appendix B.2"},{"comment":"The expert-labeled subsample is not a simple random sample from the holdout: revisions are drawn from ten score bins constructed separately for ORES and Graph2Text, so selection probabilities depend on both models' scores. The AUC values in Table 2 are therefore not unbiased estimates of population AUC and cannot be directly compared with the holdout AUCs in Table 1. Additionally, only one expert labeled the 755 revisions with no reported inter-annotator agreement. Please compute the expert evaluation on a random subsample (or re-weight by the true sampling probabilities) and report annotator agreement or at least acknowledge the single-annotator limitation in the main text.","section":"§5.3, Table 2"},{"comment":"The fairness claim that the AUC difference between anonymous and registered users is 'significantly smaller' for Graph2Text is not supported by any statistical test or confidence interval; Table 3 reports only point estimates for DIR and DAUC. Please provide bootstrapped confidence intervals or a paired significance test for the fairness metrics, or soften the 'significantly' wording to a descriptive statement.","section":"§5.4, Table 3"},{"comment":"The holdout set is the last three months of the collection window, so recent revisions have less opportunity to be reverted; no censoring correction or post-collection observation window is described. Because the holdout also shows a lower revert rate than the training period (6.2% vs. 7.9%), the absolute AUC values may be time-dependent. While this could affect all models similarly, it may interact with content-based versus metadata-based features. Please report revert probability as a function of revision age or restrict the holdout to revisions with a minimum observation window, and discuss the distribution shift.","section":"§4 / Figure 5"}],"minor_comments":[{"comment":"There is a typo: 'Morover' should be 'Moreover'.","section":"§2.2"},{"comment":"The text says the 5th and 95th percentiles are reported as the confidence interval, which is a 90% interval, but the paper refers to them as 'CI' without specifying the confidence level; please clarify.","section":"Appendix D.1"},{"comment":"The sentence 'The performance based at FR99 indicates that with Graph2Text (compared to ORES), patrollers will need to analyze nearly half as many revisions...' is grammatically awkward and should be rephrased for clarity.","section":"§5.2"},{"comment":"Some references contain encoding artifacts or spacing errors, e.g., 'Vrandeˇci´c' and 'V orobev' in the Dorogush reference; please use proper Unicode and spacing.","section":"References"},{"comment":"The reference for CatBoost is a blog-style technical report; consider citing a peer-reviewed version or the official documentation for reproducibility.","section":"§3.2.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong fit for an industry-track venue and the open dataset/code are clearly valuable. The main risk is overclaiming superiority over ORES without demonstrating robustness to the label-filtering choice and without valid statistical support for the fairness claims. These are fixable with additional analyses, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: this is a real applied result, not a fireworks paper. The Graph2Text serialization is a neat practical trick: flattening Wikidata triple edits into text so one multilingual BERT handles both textual and structured changes. That is genuinely new for Wikidata, and the released 5M-revision dataset and code make the contribution reproducible. The evaluation is more careful than most industry-track papers: time-based holdout, bootstrapped CIs for the headline AUC, an expert-labeled subsample, and a fairness comparison against the production ORES model. The expert labels are the strongest part—they give you a check on the noisy \"reverted\" proxy.\n\nWhere I'd push back on the stress-test note: it is right that the 57.7% filter changes the target, and the paper does not demonstrate invariance to that filter. But the expert evaluation, imperfect as it is, goes some way toward answering that concern: Graph2Text also beats ORES on expert labels. The weakness there is the sampling design: selecting 50 revisions per score bin per model means the expert AUC is not a population estimate; it depends on the binning. The authors should report a population-weighted number or at least discuss the bias. That is a fixable flaw, not a fatal one.\n\nThe fairness section is the softest spot. Table 3 has point estimates, no CIs, and the text says \"significantly smaller\" without giving a test. With DIR values like 5.69 vs 4.43, that is not obviously significant. They need bootstrap CIs or a paired test.\n\nThe label noise point is real but the paper acknowledges it and includes a limitations section that names several of these issues (qualifiers/rankings coverage, English-label mapping, single LM tested). I read the reverts-as-vandalism proxy as standard practice in this line of work; the paper's filters are conservative and documented.\n\nThe citation pattern is fine—uses prior work by the same group where relevant, but also WDVD, ORES, WSDM Cup, and others.\n\nMy recommendation: this deserves a serious referee. The claims are credible, the dataset is valuable, and the flaws are addressable. If I were the editor, I'd ask for fairness CIs and a sensitivity analysis of the filter (e.g., re-run with a less aggressive filter or on a random sample of the removed 57.7%) before acceptance. The paper should be in the review pipeline, not desk-rejected.","headline":"Practical, reproducible win for Wikidata vandalism detection; the head-to-head vs ORES is credible, but the fairness claims need error bars and the filter sensitivity needs a robustness check.","tokens_in":12416,"tokens_out":1579,"would_cite":true,"duration_ms":13038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that converting every Wikidata edit into text and scoring all changes with one multilingual language model produces a vandalism detector that beats the current production system on performance and fairness.","keywords":["Wikidata","vandalism detection","knowledge graphs","multilingual language models","Graph2Text","revert prediction","fairness","content moderation"],"falsifier":"Hand-label a random sample of the 57.7% of reverted revisions the pipeline filters out and score those revisions with both Graph2Text and the production model; if the excluded reverts are mostly clean or rank differently, the filtering choice, not the method, would explain the reported advantage.","tokens_in":11354,"feed_emoji":"🛡️","tokens_out":11082,"duration_ms":98793,"temperature":0.7,"pith_summary":"The paper proposes Graph2Text, a vandalism-detection system for Wikidata that turns every edit into text before scoring it. Structured changes to factual triples and unstructured changes to labels, aliases, and descriptions are serialized as sentences, tagged with their operation type, and passed to a single multilingual language model, whose per-change scores are pooled and combined with edit metadata by a second classifier. On a three-month held-out set of human edits, the system reports an area-under-the-ROC-curve (AUC) of 0.924 versus 0.859 for the current production model, and the gap persists on a 755-revision expert-labeled sample (0.932 vs. 0.885). The paper also reports that reviewers would need to look at 29% rather than 55% of revisions to catch 99% of vandalism, and that the score gap between anonymous and registered users shrinks to near zero. The authors release the code and a dataset of roughly 4.8 million Wikidata revisions to support further work.","feed_headline":"Single text model beats Wikidata vandalism detector: AUC 0.924","feed_subtitle":"Reviewers would examine 29% of edits instead of 55% to catch 99% of vandalism.","key_machinery":"The load-bearing device is Graph2Text, the serialization of Wikidata edits into text. Each change, whether a triple addition, a label change, or a removal, becomes a sentence built by mapping entity and property IDs to English labels (with unknown for the roughly 9% of IDs lacking labels) and prefixing the operation type, in the style of text-to-text models. That lets a single multilingual transformer, fine-tuned for binary revert prediction, consume structured and unstructured edits alike, with its change-level outputs mean-pooled per revision. A second stage, a gradient-boosted classifier, combines the pooled content score with revision metadata such as account age and time since the last edit, and emits the final revert-risk score.","core_discovery":"The central claim is that a single text-serialization step makes one multilingual language model sufficient for every kind of Wikidata edit, and that this content-rich approach beats the current production system on both accuracy and fairness. Graph2Text maps Wikidata IDs in triples to English labels, prepends an edit-type prefix such as insert or remove, and fine-tunes a multilingual transformer classifier on the resulting text; per-revision probability is the mean of its change-level scores, which is then fed together with revision metadata into a gradient-boosted classifier. In evaluation, this beats the production model on the holdout set (AUC 0.924 vs. 0.859), on expert labels (0.932 vs. 0.885), and at every filter-rate and recall operating point. The paper further claims improved fairness: lower disparate-impact ratios for anonymous and new users and a near-zero difference in AUC between anonymous and registered users, though a content-only variant is fairer still.","pith_inferences":["The same graph-to-text serialization could transfer to other structured knowledge bases, since any triple can be verbalized, but the paper does not test this transfer.","Mapping IDs to English labels likely caps performance on non-English content; using the edit's own language or a more language-diverse encoder is a testable improvement the paper flags as future work.","Because the evaluation uses a time-based holdout, the released dataset can support measuring how quickly detection accuracy drifts as editing behavior changes over time, a question the paper leaves open.","Public release of the code invites adversarial adaptation, so a follow-up study should test whether edits crafted after release evade the detector; the paper acknowledges but does not measure this risk."],"forward_implications":["Patrollers would review 29% of revisions instead of 55% to catch 99% of vandalism, cutting the moderation backlog nearly in half.","One deployed model can cover label, description, triple, and mixed edits, replacing content-type-specific models and simplifying maintenance.","The pipeline is designed for CPU-only inference at Wikidata's roughly ten edits per second, so it can run on existing production infrastructure without GPUs.","The full system reduces the AUC gap between anonymous and registered users to near zero and lowers disparate impact for new editors, so fewer good-faith newcomer contributions should be flagged.","The expert-labeled evaluation showing AUC 0.932 versus 0.885 suggests the gain survives when ground truth is verified by humans rather than inferred from reverts."],"supporting_citations":[{"why":"Introduces the ORES production model used as the main baseline and defines the filter-rate evaluation metric.","marker":"Sarabadani et al., 2017"},{"why":"Proposes WDVD, the 47-feature random-forest vandalism detector whose content-plus-context design this work extends.","marker":"Heindorf et al., 2016"},{"why":"Builds the WDVC-2015 corpus and establishes revision-revert history as the ground truth for Wikidata vandalism detection.","marker":"Heindorf et al., 2015"},{"why":"Supplies the multilingual transformer training recipe and the edit-war filtering heuristic used in data preparation.","marker":"Trokhymovych et al., 2023"},{"why":"Introduces BERT, the pretrained transformer architecture that the multilingual content classifier is fine-tuned from.","marker":"Devlin et al., 2019"},{"why":"Motivates the text-to-text prefix encoding that Graph2Text uses to signal insert, remove, and change operations.","marker":"Raffel et al., 2019"},{"why":"Introduces the gradient-boosted classifier used as the final stage that combines content scores with revision metadata.","marker":"Dorogush et al., 2017"},{"why":"Defines the disparate-impact-ratio fairness metric used to measure bias against anonymous and new editors.","marker":"Bellamy et al., 2018"},{"why":"Provides the bootstrap procedure used to compute the reported confidence intervals for AUC.","marker":"Efron and Tibshirani, 1994"}],"fun_headline_variants":["One LM for all Wikidata edits: beats production vandalism detector","Unified text model catches Wikidata vandalism: AUC 0.924","Graph2Text: single LM outdoes Wikidata's vandalism filter","Text-serialized Wikidata edits improve vandalism detection (AUC 0.924)","Graph2Text: one LM makes Wikidata vandalism detection simpler and better"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats a filtered revert as a clean proxy for vandalism after removing self-reverts and edit-war revisions, which together make up 57.7% of all reverted edits; if the removed cases hide systematic vandalism types, the reported performance gap could be an artifact of that filtering.","fun_headline_variants_meta":{"raw":{"variants":["One LM for all Wikidata edits: beats production vandalism detector","Unified text model catches Wikidata vandalism: AUC 0.924","Graph2Text: single LM outdoes Wikidata's vandalism filter","Text-serialized Wikidata edits improve vandalism detection (AUC 0.924)","Graph2Text: one LM makes Wikidata vandalism detection simpler and better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001075,"raw_usage":{"total_tokens":4454,"prompt_tokens":855,"completion_tokens":3599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":3503}},"tokens_in":471,"tokens_out":3599,"duration_ms":21904,"temperature":1.0,"reasoning_tokens":3503,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:34:04.616060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hand-label a random sample of the 57.7% of reverted revisions the pipeline filters out and score those revisions with both Graph2Text and the production model; if the excluded reverts are mostly clean or rank differently, the filtering choice, not the method, would explain the reported advantage.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the ORES production model used as the main baseline and defines the filter-rate evaluation metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the gradient-boosted classifier used as the final stage that combines content scores with revision metadata."}],"review_version":1}