{"id":"56315489-8839-42bf-8d27-1d241de6935d","arxiv_id":"2501.14785","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A branch-and-bound search guided by information gain and an MLP cardinality predictor selects Twitter keywords for eating-disorder classification.","lead":"ED-Filter is a proposed feature-selection pipeline that ranks Twitter keywords by information gain, then uses branch-and-bound search plus a neural network to pick a compact keyword set for classifying eating-disorder posts. The paper claims accuracy gains over standard filters and wrappers, but the experiments appear to evaluate on the same data used for feature selection and cardinality prediction.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Accuracy is reported without any train/test split, so the claimed gains over InfoGain, Relief, and wrapper baselines may be a selection artifact; a held-out evaluation is required.","rationale":"The reader's verdict is REJECT, and I agree. My primary concern is the missing held-out test set because it directly undermines the empirical comparison that the abstract advertises, and it is checkable by re-running the described pipeline with a proper split. The theoretical optimality claim is also unsound: Eq. 7 applies a Fano-style bound using the number of features n where the number of classes (4 in this dataset) is required, and Algorithm 1 prunes branches of an explicitly non-monotone objective using an upper bound computed from the current partial subset, which cannot bound the accuracy of supersets. I did not make the theory the primary attack only because the paper's stated contribution is the empirical improvement; even if a proper split were supplied and the gains vanished, the theory would still need repair to support the word 'optimal'. The concrete test I propose would settle whether the reported accuracy advantage is real or an artifact of selection on the evaluation data.","tokens_in":20576,"tokens_out":4502,"duration_ms":45564,"concrete_test":"Hold out 20% of the 11,620 Twitter accounts (or a temporal split by tweet date) before any processing. Run IG ranking, train the MLP cardinality predictor, and execute ED-Filter's subset search on the remaining 80% only; evaluate the final classifier exclusively on the held-out 20%. Repeat over at least 5 random splits and compare ED-Filter against InfoGain-Filter, Relief-Filter, Wrapper-BestFirst, and Wrapper-Forward using identical splits and the same Multinomial Naive Bayes classifier. If ED-Filter's accuracy advantage over the best baseline is not consistently positive (e.g., a 95% confidence interval that excludes zero), the claimed significant improvements are a selection artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 8 reports classification accuracy (Figures 3 and 4) without describing any train/test split. The pipeline in Sections 4.1, 7, and 8.3 uses the same Twitter data to (i) rank features by Information Gain, (ii) train the MLP cardinality predictor (whose internal cross-validation does not separate final evaluation), (iii) run the ED-Filter subset search, and (iv) measure the final accuracy. Selecting features and tuning the subset cardinality on the same records that are then scored yields an optimistically biased estimate of out-of-sample accuracy. The claimed 'significant improvements' over InfoGain-Filter, Relief-Filter, Wrapper-BestFirst, and Wrapper-Forward in Section 8.3 are therefore uninterpretable: they may reflect selection bias rather than a genuinely better feature subset. This is load-bearing because the abstract's central empirical claim rests entirely on these accuracy figures.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes ED-Filter, a feature-selection method for classifying eating-disorder-related Twitter users. The pipeline ranks 25 keyword-count features by information gain, searches for a high-accuracy subset with an informed branch-and-bound procedure whose pruning uses an information-theoretic upper bound (Eq. 7), and augments the search with a greedy variant and an MLP that predicts the desired subset cardinality. Experiments on a Twitter dataset of 11,620 user accounts with 37,405 tweets compare ED-Filter with InfoGain, Relief, and two wrapper baselines, and the paper reports that ED-Filter achieves the best precision and selects the fewest features. The abstract claims that ED-Filter iteratively identifies an optimal set of promising features and yields significant improvements in classification accuracy.","tokens_in":20735,"tokens_out":9256,"duration_ms":88711,"significance":"Social-media-based eating-disorder detection is a timely and practically relevant problem, and the paper addresses a real difficulty: high-dimensional, sparse keyword-count features. The proposed combination of filter-based ranking and bounded search is a reasonable design idea, and the authors provide a data-collection description, a public code repository, and comparisons against standard baselines. However, the central claims are not currently supported: the evaluation does not separate feature selection from final accuracy measurement, the upper-bound derivation in Section 5 is mathematically invalid, and the algorithm pseudocode is internally inconsistent. If these issues were corrected with a proper held-out evaluation and a valid admissibility proof, the work could make a useful contribution; as presented, the experimental and theoretical evidence is not sufficient.","major_comments":[{"comment":"The paper does not describe any train/test split or cross-validation for the reported accuracy. The same Twitter data are used to rank features by information gain (Section 4.1), to train the MLP cardinality predictor on data chunks (Section 7), to search for a feature subset, and to compute the final theta values in Figures 3 and 4. With selection and evaluation on the same records, the accuracy figures are optimistically biased and the claimed significant improvements over InfoGain-Filter, Relief-Filter, Wrapper-BestFirst, and Wrapper-Forward cannot be interpreted. A held-out evaluation, for example nested cross-validation in which feature ranking, cardinality prediction, and subset search are performed only on training folds, is required before any effectiveness claim can be made.","section":"Section 8.2-8.3, Figures 3-4"},{"comment":"The derivation of Lemma 1 is not valid. In Eq. (6), n is the number of features in F, but the entropy term log(n) in a mutual-information bound must refer to the cardinality of the class variable Y, since I(Y;F) is bounded by H(Y), which is at most log|Y|; using the feature count in this role mixes two different quantities. Algebraically, solving Eq. (6) for theta gives a bound of the form theta <= 1 - (log n - 1 - IG)/log(n-1), not the expression in Eq. (7), and the proof's step of replacing H2(theta) by 1 and adding 1 to the denominator changes the inequality. Because Lemma 2's admissibility argument depends entirely on this upper bound, the optimality claim for Algorithm 1 is unsupported.","section":"Section 5, Eq. (6)-(7), Lemma 1"},{"comment":"In both algorithms, the measured accuracy is immediately overwritten by the upper bound: line 4 of Algorithm 1 sets e.theta = theta(e.F,Y) and line 5 sets e.theta to the Eq. (7) expression, so the heap ordering and the termination test e.theta < thetamin use only the bound and never the measured accuracy. This makes the behavior of the informed branch-and-bound procedure undefined as written. In addition, Eq. (2) defines theta(F',Y) as TP/(TP+FP), which is precision, not accuracy; the text and figures refer to this quantity as \"classification accuracy\" throughout Section 8, so the reported numbers need to be relabeled or the metric redefined.","section":"Section 5, Algorithm 1; Section 7, Algorithm 2"},{"comment":"The SVD/LDA comparison reports per-class precision values but gives no indication of how these are computed, whether the classes are balanced in the test data, or whether the same evaluation split is used as in Figures 3 and 4. The discussion attributes the pattern to class sizes, but no class-size statistics are reported, so the comparison cannot be independently assessed.","section":"Section 8.3, Table 3"}],"minor_comments":[{"comment":"The text says \"we propose FilterBoost\" in the paragraph beginning \"In this paper, we propose FilterBoost,\" but the method is everywhere else called ED-Filter; this inconsistent name should be reconciled.","section":"Section 1"},{"comment":"The data-processing description does not explain how the 37,405 tweets are mapped to the 11,620 user accounts or how the four class labels y in {0,1,2,3} are assigned; without this, the construction of the classification rows in Table 1 is incomplete.","section":"Section 8.1"},{"comment":"The terms \"worst-case scenario\" and \"best-case scenario\" are not defined, and no number of runs, random seeds, or error bars are given despite the text saying each method was run \"a couple of times.\"","section":"Figure 3(e)-(f)"},{"comment":"The information-gain threshold values that yield 25, 18, 14, and 11 eligible features are never reported, which makes the scalability experiments difficult to reproduce.","section":"Section 8.4, Figure 6"},{"comment":"The getNext(F) calls iterate over the full feature set; the algorithms should state explicitly that only features not already in e.F are considered and that the loops terminate when no such feature remains.","section":"Algorithms 1 and 2"},{"comment":"There are spelling inconsistencies in author names, for example \"Dodzilo\" in the text versus \"Dondzilo\" in references [53] and [54], and \"Sedghir\" in the text versus \"Seghir\" in reference [62]; these should be corrected.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript would need a substantially new evaluation protocol and a corrected upper-bound proof to support its claims. The current figures cannot be used as evidence of effectiveness, and the theoretical derivations in Section 5 are not sound. If the authors can provide a proper held-out evaluation and a valid admissibility argument, the topic and method might be publishable, but that would be a major reworking rather than a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper's main empirical claim is unsupported because the evaluation never holds out data, and Lemma 1 looks like a misapplied Fano inequality. The core idea is a minor engineering variant, so there's not much to salvage without a substantial rewrite.\n\nWhat's actually new: the MLP that predicts subset cardinality is a reasonable way to bound the branch-and-bound search, and I haven't seen that exact combination in the cited papers. The domain is real, and the authors assembled a domain-specific feature set via topic modeling and expert consultation, which is a legitimate asset. They also shipped code, which is more than many papers do.\n\nNow the problems. Section 8 reports accuracy in Figures 3 and 4 without describing any train/test split. The pipeline uses the same Twitter data to rank features, train the MLP, run the subset search, and compute the final accuracy. That's a textbook case of selection bias, so the 'significant improvements' over the baselines are uninterpretable. The stress-test note is correct on this.\n\nThe theoretical part is worse. Lemma 1 starts from something that resembles Fano's inequality but uses log(n) where n is the number of features, not the number of classes. Then it replaces the binary entropy with 1 to get an 'admissible' upper bound, which doesn't follow. Algorithm 1 therefore does not establish global optimality, and the pruning rule in line 9 relies on an upper bound that can be violated. The paper also never cites the classic branch-and-bound feature selection literature (Narendra and Fukunaga), so the novelty is overstated.\n\nOne more thing: the accuracy formula in Eq. 2 is just precision, not accuracy. They call it accuracy but it's TP/(TP+FP). That's a minor but real sloppiness.\n\nSo, proportionate verdict: the paper is not ready for publication. The empirical core is circular, and the theory is unsound. If the authors fixed the evaluation with a genuine held-out test set and corrected the bound, they might have a modest workshop-level contribution. As is, I would not send this to peer review; I'd desk reject with an invitation to resubmit after major revision.","headline":"The paper's central accuracy claims are uninterpretable because no held-out split is described, and the theoretical bound looks like a garbled Fano inequality.","tokens_in":21295,"tokens_out":3838,"would_cite":false,"duration_ms":37732,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that ED-Filter finds small feature subsets that classify eating-disorder tweets more accurately than standard filters and wrappers, and fast enough for streaming data.","keywords":["eating disorder detection","feature selection","branch and bound","information gain","greedy search","deep learning","Twitter analysis","classification"],"falsifier":"Re-run the ED-Filter comparisons with a strict temporal or random split: rank features and train the MLP cardinality predictor only on a training portion, search for the best feature subset there, and report classification accuracy on a held-out test portion. If ED-Filter's advantage over InfoGain, Relief, and the wrappers disappears or reverses on the held-out test set, the central claim is falsified.","tokens_in":20350,"feed_emoji":"🧠","tokens_out":9190,"duration_ms":78651,"temperature":0.7,"pith_summary":"The paper claims that a feature-selection method called ED-Filter, which ranks eating-disorder-related Twitter keywords by information gain and then searches the feature space with a branch-and-bound procedure, can find a small feature subset that classifies eating-disorder-related Twitter users more accurately than standard filters and wrappers. This matters because Twitter eating-disorder data is high-dimensional and changes frequently, so the authors argue that a fast, adaptive feature-selection step is needed to turn raw tweets into reliable classification models. They report that on a corpus of Pro-ED tweets, ED-Filter keeps precision around 82-87% as the dataset grows from 1,000 to 8,000 users, while the comparison filters drop below 75%, and that a hybrid greedy deep-learning variant achieves comparable accuracy in under 90 seconds for up to 11 features.","feed_headline":"ED-Filter reports best precision among four feature selectors","feed_subtitle":"On Twitter ED data from 1,000 to 8,000 users, it keeps precision above 82% as filters fall below 75%.","key_machinery":"The load-bearing object is the upper-bound estimate $\\bar{\\theta}(F,Y) = \\frac{IG(Y;F) - \\log(n) + 1}{\\log(n-1)} + 1$ from Equation 7, which the paper derives by relating classification accuracy to information gain and binary entropy and claims is admissible, meaning it never underestimates the true accuracy. The branch-and-bound search keeps partial feature subsets in a max-heap, expanding the most promising entry and stopping when its upper bound falls below the best accuracy found so far. The hybrid greedy deep-learning variant restricts the search to a seed set of high-scoring features and to subsets whose size is predicted by a two-hidden-layer MLP trained on the Twitter data.","core_discovery":"The central claim is that the optimal feature subset for eating-disorder classification can be identified by an informed branch-and-bound search whose pruning is driven by an admissible upper bound on accuracy derived from information gain, and that when the full search is too slow for streaming Twitter data, a greedy expansion restricted to seed features plus an MLP that predicts subset cardinality finds a near-optimal subset quickly. On the collected Twitter data, the authors report that ED-Filter achieves the best precision among InfoGain-Filter, Relief-Filter, Wrapper-BestFirst, and Wrapper-Forward, and that it selects fewer features than the compared methods. The hybrid method is presented as comparable to the greedy method, with accuracy differences below 5 percent on average, while halving or eliminating the combinatorial search overhead.","pith_inferences":["If the reported accuracy is computed on the same records used to rank features and train the cardinality predictor, the gains may shrink under a proper temporal or random split; a reader should test this before relying on the headline numbers.","The upper-bound identity in Equation 7 is the kind of quantity that could be reused in other high-dimensional text classification tasks, but its admissibility would have to be re-derived for each new accuracy measure and data distribution.","Because the MLP cardinality predictor is trained on Twitter eating-disorder data, applying the hybrid method to a new platform or a new disorder would require re-training; the paper does not specify how often the model must be updated as the stream evolves.","The greedy add/remove local search depends on the order in which seed features are considered; testing multiple random seeds would reveal how stable the selected subset is to feature ordering."],"forward_implications":["On the reported Twitter data, ED-Filter is the only method whose precision stays above 82% across data sizes from 1,000 to 8,000 users, while InfoGain and Relief filters fall below 75%.","ED-Filter selects fewer features than the compared filters and wrappers, which the paper argues makes it more scalable for dynamic Twitter streams.","The hybrid greedy deep-learning variant runs within about 90 seconds for feature sets up to 11 features, whereas the full branch-and-bound search time grows sharply beyond 6 features.","Accuracy differences between the hybrid and greedy methods stay below 5 percentage points on average, so the speed-up is claimed to come at a small effectiveness cost."],"supporting_citations":[{"why":"Supplies the #proana Twitter data source and framing of Pro-ED communities that the dataset collection relies on.","marker":"[8]"},{"why":"Provides the TF-IDF technique used to compute feature weights in Definition 3.","marker":"[63]"},{"why":"CorEx topic modeling is adopted to extract the topics whose keywords become the classification features.","marker":"[65]"},{"why":"Information Gain filter used as the ranking phase of ED-Filter and as a comparison baseline.","marker":"[67]"},{"why":"Relief filter used as a comparison baseline for feature selection effectiveness.","marker":"[68]"},{"why":"Defines the wrapper methodology for feature subset selection; the wrapper baselines are built on this approach.","marker":"[69]"},{"why":"Provides the fast wrapper feature subset selection used as the Wrapper-Forward baseline.","marker":"[70]"}],"fun_headline_variants":["ED-Filter tops four selectors in Twitter ED classification","Informed branch-bound ED-Filter maximizes precision on ED tweets","ED-Filter: fewer features, best precision for ED detection","Hybrid greedy ED-Filter keeps accuracy, cuts search time","Branch-bound feature search wins on eating disorder tweets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the classification accuracy $\\theta(F',Y)$ used to compare feature subsets is an honest measure of out-of-sample performance; the paper reports accuracy computed without describing a train/test split, so if the same records are used to select features and to measure accuracy, the reported gains could be an artifact of fitting the data.","fun_headline_variants_meta":{"raw":{"variants":["ED-Filter tops four selectors in Twitter ED classification","Informed branch-bound ED-Filter maximizes precision on ED tweets","ED-Filter: fewer features, best precision for ED detection","Hybrid greedy ED-Filter keeps accuracy, cuts search time","Branch-bound feature search wins on eating disorder tweets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1497,"prompt_tokens":876,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":492,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":492,"tokens_out":621,"duration_ms":6046,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:14:53.021599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ED-Filter comparisons with a strict temporal or random split: rank features and train the MLP cardinality predictor only on a training portion, search for the best feature subset there, and report classification accuracy on a held-out test portion. If ED-Filter's advantage over InfoGain, Relief, and the wrappers disappears or reverses on the held-out test set, the central claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the #proana Twitter data source and framing of Pro-ED communities that the dataset collection relies on."},{"cited_title":"Logtiw:a log anomaly detection model based on tf-idf weighted semantic features","cited_arxiv_id":null,"evidence_quote":"Provides the TF-IDF technique used to compute feature weights in Definition 3."},{"cited_title":"Discovering structure in high-dimensional data through correlation explanation","cited_arxiv_id":null,"evidence_quote":"CorEx topic modeling is adopted to extract the topics whose keywords become the classification features."},{"cited_title":"A feature selection method based on information gain and genetic algo- rithm","cited_arxiv_id":null,"evidence_quote":"Information Gain filter used as the ranking phase of ED-Filter and as a comparison baseline."},{"cited_title":"Urbanowicz, Melissa Meeker, William La Cava, Randal S","cited_arxiv_id":null,"evidence_quote":"Relief filter used as a comparison baseline for feature selection effectiveness."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the wrapper methodology for feature subset selection; the wrapper baselines are built on this approach."},{"cited_title":"G´ amez, and Jos´ e M","cited_arxiv_id":null,"evidence_quote":"Provides the fast wrapper feature subset selection used as the Wrapper-Forward baseline."}],"review_version":1}