{"id":"2fb73609-e2de-4a54-a5cf-5b72bcb96df5","arxiv_id":"2411.16285","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DFG-NAS automatically chooses propagation and transformation steps in relational graph networks for bot detection, reaching 85.7 percent accuracy on TwiBot-20.","lead":"This paper uses an automatic search algorithm to design a graph-based neural network for detecting bots on X/Twitter. On a public benchmark it reports 85.7 percent accuracy, a small gain over earlier published bot detectors.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 85.7% accuracy is selected from five test-set results (Section VII), so the headline number is the maximum over test rather than the accuracy of a validation-selected model; this invalidates the surpassing-SOTA claim even if baselines were rerun identically.","rationale":"The reader's weakest assumption was that published baseline numbers come from a different split/protocol, making the comparison unfair. That is a real concern, but the paper has an additional internal problem that is independent of baseline provenance: the final \"ours\" model is chosen after inspecting test-set results. Section VII explicitly says the third architecture is designated as the model because it provides the highest accuracy, and Table 4 reports five architectures whose validation accuracies differ by only about 0.2 percentage points. Selecting the best test performer among five candidates creates an optimistic bias in the reported 85.68%, and no adjustment or significance test is provided for this selection. This directly undermines the strongest claim in the abstract. The concern is fixable: the authors could report the test accuracy of the top-validation architecture, or use a nested hold-out split where the test set is used only once. The paper still has value as an application of DFG-NAS to bot detection, and the top-five architectures all perform reasonably, but the headline superiority claim is not established by the evidence as presented. Because the reader already assigned CONDITIONAL, my read does not change the verdict; it strengthens the conditionality with a distinct internal reason rather than only the baseline-comparison concern.","tokens_in":21142,"tokens_out":4195,"duration_ms":42895,"concrete_test":"Re-run the pipeline with a pre-registered selection rule: choose the architecture with the highest validation accuracy (the 1st architecture, 87.01% validation) as the final model, or use a separate hold-out test partition that is never examined until after the final model is chosen. Retrain BotRGCN under exactly the same 70/20/10 split and preprocessing. Compare the validation-selected model's accuracy, F1, and MCC to same-setting BotRGCN with a paired test (e.g., McNemar or bootstrap). If the validation-selected model does not significantly exceed BotRGCN, the \"surpassing state-of-the-art\" claim should be withdrawn or weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section VI.B states that after the NAS search the authors examine \"the five architectures with the best accuracy in the validation set,\" retrain each for 100 epochs, and then test on the test set. Section VII then says the third architecture \"achieves the best evaluation metrics\" and is designated as \"our model, since it provides the highest accuracy.\" The reported 85.68% is therefore the maximum test accuracy over the five candidates, not the accuracy of the architecture selected by validation alone. Reported as a single \"ours\" number, this is an optimistic selection-on-the-test-set estimate: with five candidates, the expected maximum exceeds the expected accuracy of any fixed candidate, and no multiple-testing correction or significance test is reported. This is load-bearing because the central claim is precisely that the chosen architecture surpasses state-of-the-art models. Even if all baselines were rerun under an identical protocol, the chosen-model accuracy would still be an upward-biased estimate. The reader's baseline-reuse concern remains valid and should be addressed, but this internal selection issue is more direct: it affects the interpretation of the paper's own headline number, not only the fairness of the comparison set.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies DFG-NAS, an evolutionary neural architecture search method, to relational graph convolutional networks for bot detection on the TwiBot-20 dataset. It encodes user descriptions, tweets, and numerical/categorical properties, builds a heterogeneous graph from follower/following relations, and searches over sequences of propagation (P) and transformation (T) operations. The five architectures with highest validation accuracy are retrained and evaluated; the best one reaches 85.68% accuracy, which the authors report as surpassing state-of-the-art baselines. Ablations examine removing metadata features and removing the gate and skip-connection mechanisms.","tokens_in":21337,"tokens_out":8532,"duration_ms":73005,"significance":"If the claim were established, the paper would provide a useful demonstration that evolutionary architecture search can replace hand-designed message-passing stacks in relational GNNs for bot detection, while also showing the value of graph structure and metadata. The paper's own results include five-run means and standard deviations for the proposed architectures, and the limitations section candidly notes the single-dataset evaluation and absence of hyperparameter tuning. However, the central quantitative claims depend on a test-set-selected architecture and on copied baseline numbers, and the ablation results do not support the stated conclusion about feature necessity; these issues currently prevent the significance from being realized.","major_comments":[{"comment":"Section VII states that after retraining the five architectures, \"the third architecture achieves the best evaluation metrics\" and is designated \"our model, since it provides the highest accuracy.\" The reported 85.68% is therefore the maximum over five test-set evaluations, not the accuracy of an architecture chosen by validation alone. Because the five candidates were shortlisted on validation and then the best test performer was selected, the headline estimate is upward-biased by selection. This is load-bearing: the abstract and conclusion assert superiority over state-of-the-art on the basis of this number. Please report the validation-selected architecture's test accuracy (the first row in Figure 3) as the primary result, or average over the five candidates, or provide a multiple-testing correction and significance test.","section":"Section VII, Table IV"},{"comment":"Section VI.A says \"We are using the published results for the comparison,\" and Table V lists baseline metrics without error bars. The authors' own numbers are five-run means on their 70/20/10 split, with their own preprocessing and training budget. There is no evidence that the published baseline results were obtained under the same split, preprocessing, or evaluation protocol. The \"surpassing state-of-the-art\" claim is therefore not statistically established. Please rerun at least the strongest baselines (BotRGCN, SATAR) under the identical protocol and report their variability, or explicitly state that the comparison is approximate and weaken the claim.","section":"Section VI.A, Table V"},{"comment":"The paper's contribution is the automatic search over P/T sequences, but no fixed-architecture RGCN is trained with the same metadata, graph, split, and training settings as a control. Without such a control, the gains over published fixed-architecture results could be due to the feature encoding, training protocol, or random split rather than to the searched architecture itself. Please add a same-pipeline controlled comparison, e.g., BotRGCN's default architecture trained under the same 70/20/10 split and 100-epoch protocol. Additionally, Section III.A states that Ilias et al. [29] \"conducted their experiments on the Cresci'17 dataset,\" yet Table V reports a TwiBot-20 result for [29]; if the published value in that row comes from a different dataset, the row is not a valid baseline for this comparison.","section":"Section VI.A, Section III.A, Table V"},{"comment":"Section VIII concludes that \"all features contribute to the model's performance,\" but Table VI shows that removing the description gives accuracy 0.859 ± 0.004, F1 0.875 ± 0.004, and MCC 0.718 ± 0.008, and removing the numerical properties gives accuracy 0.859 ± 0.003 and MCC 0.716 ± 0.007, both numerically above the full model's 0.857 ± 0.004 / 0.712 ± 0.007. With overlapping standard deviations, this ablation does not support the claim of necessity for those features; at most it shows that the categorical features are important. Please add significance testing or reinterpret the results.","section":"Section VIII, Table VI"}],"minor_comments":[{"comment":"The dataset name is written inconsistently as \"TwiBot-20\", \"Twibot-20\", and \"Twi-Bot20\"; please standardize it.","section":"Section IV and throughout"},{"comment":"The transformation formula appears to contain only self/relation-dependent terms with no explicit neighbor aggregation; please clarify how it differs from propagation in Eq. (4) and whether the same relation weights W_r are shared across both equations.","section":"Section V.B, Eq. (5)"},{"comment":"The text gives training budgets for the search (70 epochs) and final training (100 epochs), but the tables say \"five runs of results are averaged\"; please state in the text that five independent runs were used for the final evaluation and whether the search was performed once or multiple times.","section":"Section VI.B, Table IV"},{"comment":"The sentence \"In Figure 2, the five architectures ... are depicted\" should refer to Figure 3; Figure 2 is the layer-connection schematic from Section V.","section":"Section VII"},{"comment":"The text \"the architecture without the gate has a reduced accuracy by 0.5%\" does not match Table VIII (0.857 vs 0.853, i.e., 0.4 percentage points); please align the text and table.","section":"Section VIII, gate ablation"},{"comment":"The sentence \"We see that the architecture without the gate has a reduced accuracy\" should read \"without the skip-connection\"; the current sentence repeats the gate ablation.","section":"Section VIII, skip-connection ablation"},{"comment":"The reference list entry [12] is cited as \"BotRGCN et al. [12]\" in the text; it should be \"Feng et al. [12]\".","section":"Section VI.A"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward application of a published NAS method to a standard dataset; the novelty is modest, but it could be acceptable as a systems/demonstration paper if the comparisons are made rigorous. The test-set selection issue is the most serious and must be addressed before the SOTA claim can be taken literally. I would encourage requiring the authors to rerun the strongest baselines under the same protocol or to substantially weaken the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one new thing here is the adaptation of DFG-NAS to RGCNs for bot detection on TwiBot-20. That is a legitimate extension, even if the algorithmic novelty is small: DFG-NAS is published, and NAS for bot detection already exists in ROSGAS. The paper is transparent about borrowing baseline numbers, reports mean plus/minus standard deviation over five runs, and runs ablations for the gate, skip-connection, and feature sets. That is more care than many bot-detection papers show, and the authors state their limitations plainly.\n\nThe soft spots are real, though. Section VI.B says the five architectures with the best validation accuracy are retrained and then tested. Section VII then says the third architecture 'achieves the best evaluation metrics' and is designated as the model 'since it provides the highest accuracy.' So the reported 85.68% is the maximum test accuracy over five candidates, not the accuracy of an architecture chosen by validation alone. With five candidates, the expected max exceeds the expected accuracy of any fixed candidate, and no correction or significance test is reported. This is not a minor quibble: the central claim is that the chosen model surpasses state of the art, and this selection-on-test invalidates that claim even if all baselines were rerun identically.\n\nSecond, the comparison reuses published baseline numbers with no error bars and no same-setting rerun. The authors admit this in Section VI.A, but it still means the 'surpassing state-of-the-art' conclusion rests on an apples-to-oranges comparison on top of the selection problem.\n\nThird, the ablation story contradicts itself. Table VI shows that removing descriptions or numerical properties slightly increases accuracy relative to the full model (0.859 vs 0.857), yet the text says 'Obviously, the model trained with all the features has the best performance.' That is not what the table shows. The all-features model does have the best balance across metrics, but the claim as written is wrong and should be corrected.\n\nMinor issues: one dataset, no code release, no significance testing. The paper is a reasonable application study, and the authors are not hiding anything, but the headline result is the product of test-set selection and copied baselines.\n\nI would send this to peer review with a request for major revision: pick the architecture once, on validation, report that model's test accuracy; rerun BotRGCN (and ideally one or two other baselines) in the same pipeline; and fix the ablation interpretation. As it stands, the paper is useful as a demonstration of NAS in bot detection, but the superiority claim is not supported.","headline":"An honest application of DFG-NAS to bot detection, but the headline accuracy is the best of five test-set runs, so the surpassing-SOTA claim is not established as written.","tokens_in":21913,"tokens_out":1730,"would_cite":false,"duration_ms":17491,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an evolutionary search over message-passing layouts in a graph neural network detects Twitter bots at 85.7% accuracy, beating published state-of-the-art detectors.","keywords":["bot detection","graph neural networks","neural architecture search","propagation","transformation","social media platform X","TwiBot-20","evolutionary algorithm"],"falsifier":"Run the strongest published baselines on exactly the 70/20/10 split and preprocessing used in this paper; if any of them reaches or exceeds 85.7% accuracy, the claim of surpassing state-of-the-art is not established. A second check: replace the evolutionary search with random sampling of propagation/transformation sequences under identical training; if random architectures match the searched accuracy, the search itself is not the source of the gain.","tokens_in":20937,"feed_emoji":"🤖","tokens_out":9245,"duration_ms":76300,"temperature":0.7,"pith_summary":"This paper tries to establish that bot detection on X (formerly Twitter) can be improved by automating the design of a graph neural network instead of fixing it by hand. It adapts the evolutionary architecture search method DFG-NAS to relational graph convolutional networks, searching over sequences of propagation and transformation operations in the message-passing pipeline. On the TwiBot-20 dataset, with a graph of 229,580 users linked by follow relationships and enriched with user metadata, the best architecture found reaches 85.7% test accuracy, which the authors report as surpassing published state-of-the-art models. Ablations show that each metadata modality, a gating mechanism, and skip connections each contribute to that performance.","feed_headline":"Search-picked graph nets spot Twitter bots at 85.7%","feed_subtitle":"An evolutionary search over message-passing layouts beats published bot detectors on the TwiBot-20 benchmark.","key_machinery":"The load-bearing machinery is the search space of propagation (P) and transformation (T) operations that make up the message-passing protocol of a relational graph convolutional network. P aggregates messages from a node's neighbors with relation-specific weights; T applies a nonlinear transformation to node representations. DFG-NAS represents an architecture as a sequence of P and T steps and evolves it with four mutations (append P, append T, replace P with T, replace T with P), evaluating each candidate on validation accuracy. To keep deep pipelines trainable, P connections use a gating mechanism and T connections use skip connections. The search selects a P/T sequence that feeds a final MLP classifying each user as bot or human.","core_discovery":"The central claim is that an automatically searched message-passing configuration outperforms hand-designed detectors on a standard bot-detection benchmark. The authors construct a heterogeneous graph whose nodes are users and whose edges encode follower/following relations, and each node carries metadata from the user's description, tweets, numerical properties, and categorical properties. They adapt DFG-NAS's evolutionary search to choose the order and number of propagation (neighbor aggregation) and transformation (node update) operations inside relational graph convolutional layers, with gating on propagation and skip connections on transformation. The five highest-validation architectures from the search all land near 85% test accuracy, and the best reaches 85.7% accuracy, 87.1% F1-score, and 0.712 Matthews correlation coefficient, higher than the published numbers of the state-of-the-art models they compare against on the same dataset. The paper's ablations attribute part of this gain to the gating and skip-connection mechanisms and to the full set of metadata features.","pith_inferences":["If the baseline comparison survives a common evaluation protocol, the practical consequence is that architecture search can absorb much of the manual design work in GNN-based bot detection, making detectors easier to refresh as bots evolve.","The ablation ranking of user features suggests a next step is letting the search choose how to fuse modalities rather than concatenating them; the paper itself lists multimodal fusion as future work.","A direct test of whether the search, rather than model capacity, drives the gain would compare the evolved P/T sequence against randomly sampled or fixed one-block architectures trained under the same budget."],"forward_implications":["If the central claim holds, bot detection on graph-structured social data can be framed as a search problem, so adapting to a new dataset need not require manually redesigning the message-passing architecture.","The top five searched architectures achieve validation accuracies between 86.8% and 87.0%, suggesting a plateau of near-optimal propagation/transformation configurations rather than a single fragile design.","Removing the gate from propagation lowers accuracy by about 0.5 percentage points, and removing skip connections from transformation lowers it by about 0.93 points, so both mechanisms contribute to the searched architecture's result.","The feature ablations imply the reported edge depends on having all four metadata modalities; dropping categorical properties alone drops accuracy to 79.2%, and single-feature models perform markedly worse."],"supporting_citations":[{"why":"Supplies the DFG-NAS evolutionary search over propagation and transformation operations that the paper adapts to relational graph convolutional networks.","marker":"[13]"},{"why":"Provides the relational graph convolutional baseline, the user metadata preprocessing, and the published accuracy the paper aims to beat.","marker":"[12]"},{"why":"Defines the TwiBot-20 dataset and benchmark on which all experiments and comparisons are run.","marker":"[14]"},{"why":"One of the strongest published baselines in the comparison table, with 84.1% accuracy, whose result the model claims to surpass.","marker":"[27]"},{"why":"Supplies a published supervised-learning baseline result used in the comparison table.","marker":"[9]"},{"why":"Supplies a published semi-supervised baseline result used in the comparison table.","marker":"[37]"},{"why":"Supplies a widely used feature-heavy detector baseline result used in the comparison table.","marker":"[23]"}],"fun_headline_variants":["NAS-designed graph nets beat hand-built bot detectors","Auto-searched graph nets spot Twitter bots at 85.7%","Graph search finds bot detector topping state of the art","Evolutionary graph search outdoes manual bot detection models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison to earlier detectors relies entirely on their published accuracy numbers and assumes those numbers came from the same data split, preprocessing, and evaluation protocol as the authors' 70/20/10 split, since no baseline is rerun in the same pipeline.","fun_headline_variants_meta":{"raw":{"variants":["NAS-designed graph nets beat hand-built bot detectors","Auto-searched graph nets spot Twitter bots at 85.7%","Graph search finds bot detector topping state of the art","Evolutionary graph search outdoes manual bot detection models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000482,"raw_usage":{"total_tokens":2394,"prompt_tokens":969,"completion_tokens":1425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":585,"completion_tokens_details":{"reasoning_tokens":1368}},"tokens_in":585,"tokens_out":1425,"duration_ms":20258,"temperature":1.0,"reasoning_tokens":1368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:17:19.048056+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the strongest published baselines on exactly the 70/20/10 split and preprocessing used in this paper; if any of them reaches or exceeds 85.7% accuracy, the claim of surpassing state-of-the-art is not established. A second check: replace the evolutionary search with random sampling of propagation/transformation sequences under identical training; if random architectures match the searched accuracy, the search itself is not the source of the gain.","supporting_citations":[{"cited_title":"Deep and flexible graph neural architecture search","cited_arxiv_id":null,"evidence_quote":"Supplies the DFG-NAS evolutionary search over propagation and transformation operations that the paper adapts to relational graph convolutional networks."},{"cited_title":"Botrgcn: Twitter bot detection with relational graph convolutional networks","cited_arxiv_id":null,"evidence_quote":"Provides the relational graph convolutional baseline, the user metadata preprocessing, and the published accuracy the paper aims to beat."},{"cited_title":"Twibot-20: A comprehensive twitter bot detection benchmark","cited_arxiv_id":null,"evidence_quote":"Defines the TwiBot-20 dataset and benchmark on which all experiments and comparisons are run."},{"cited_title":"Satar: A self-supervised approach to twitter account representation learning and its application in bot detection","cited_arxiv_id":null,"evidence_quote":"One of the strongest published baselines in the comparison table, with 84.1% accuracy, whose result the model claims to surpass."},{"cited_title":"Seven months with the devils: A long-term study of content polluters on twitter","cited_arxiv_id":null,"evidence_quote":"Supplies a published supervised-learning baseline result used in the comparison table."},{"cited_title":"Scalable and generalizable social bot detection through data selec- tion","cited_arxiv_id":null,"evidence_quote":"Supplies a published semi-supervised baseline result used in the comparison table."},{"cited_title":"Botornot: A system to evaluate social bots","cited_arxiv_id":null,"evidence_quote":"Supplies a widely used feature-heavy detector baseline result used in the comparison table."}],"review_version":1}