{"id":"a6cd6547-44d2-4c9f-82ef-f744750aa0b8","arxiv_id":"2506.02805","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Reordering attributes before FRRI does not improve performance, while removing 10% of attributes via fuzzy rough feature selection yields slightly better balanced accuracy and shorter rules.","lead":"This paper tests whether changing the attribute order before running Fuzzy Rough Rule Induction (FRRI) affects accuracy, rule count, or rule length. Reordering attributes with three standard feature selection methods does not improve FRRI, but removing about 10% of attributes with fuzzy rough feature selection gives a small gain in balanced accuracy and shorter rules.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation does not state whether FRFS ordering/removal is fitted inside each CV training fold; if fitted on full data, the FRFS-0.9 accuracy gain may reflect test-set leakage.","rationale":"The reader's weakest assumption concerns global attribute ordering versus per-object ordering. That is a fair limitation, but it applies mainly to the negative claim, and the abstract carefully restricts that claim to 'known methods' of ordering, which are exactly what the experiments use. The more pressing threat is to the positive claim: the paper recommends FRFS-based removal of up to 10% of attributes, yet the experimental section does not explicitly state that feature selection is nested inside the cross-validation loop. If preprocessing is fit on the full dataset, the ofrfs-0.9 advantage could be an artifact of test-set leakage, especially since the balanced-accuracy gain is already only weakly significant (p = 0.071). The proposed check would settle this by re-running the comparison with a properly nested protocol. This does not change the need for a conditional verdict, but it sharpens the condition: the authors should either confirm that feature selection is fold-local or provide code demonstrating it.","tokens_in":11169,"tokens_out":6312,"duration_ms":77858,"concrete_test":"Inspect the released implementation, or request the CV harness from the authors, to determine whether the computation of the ordered attribute list and the retained attribute subset is inside the training-fold loop or outside it. Then re-run Table 2 with QuickReduct and the 90% retention rule fit independently on each training fold and evaluated on the corresponding held-out fold. If the ofrfs-0.9 minus control mean balanced-accuracy difference drops below 0.005, or the one-sided Wilcoxon p-value rises above 0.1, the Section 6 recommendation is not supported without a different analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 describes ten-fold cross-validation on the KEEL folds, but it never states where the feature-selection preprocessing is fitted. If the QuickReduct ordering and the 90% retention subset are computed on the full dataset before the folds are separated, then the held-out fold contributes to choosing the attribute order and subset used by FRRI. The control condition receives no such test-set information, so the comparison would be biased in favor of the preprocessed variants. This is load-bearing because the paper's positive recommendation rests on ofrfs-0.9 improving balanced accuracy and shortening rules; the only directly reported paired test for the balanced-accuracy gain is a one-sided Wilcoxon p-value of 0.071, described as weakly significant. A small amount of leakage could plausibly manufacture a difference of that size. The manuscript text is terse enough that this protocol ambiguity cannot be resolved without inspecting the code or a clearer experimental description.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically tests whether preprocessing the FRRI rule induction algorithm with feature selection affects its performance. It compares an unmodified control with variants that reorder attributes only (using QuickReduct, mutual information, and Pearson correlation) and variants that reorder and retain 90% or 80% of the attributes, plus a version of FRFS that stops when the dependency reaches 1. The experiments are run on 18 KEEL datasets with ten-fold cross-validation, measuring balanced accuracy, number of rules, and average rule length. The authors conclude that attribute reordering alone does not significantly affect FRRI, while removing up to 10% of attributes with fuzzy rough feature selection can improve balanced accuracy and shorten rules.","tokens_in":11328,"tokens_out":7669,"duration_ms":82593,"significance":"If the conclusions are robust, the paper offers practical guidance for FRRI users and a useful negative result about greedy attribute ordering. The study uses standard benchmarks and appropriate statistical machinery, and it reports three metrics rather than accuracy alone. However, the positive recommendation rests on a one-sided Wilcoxon p-value of 0.071 for balanced accuracy, and the paper does not release code or data. The main strength is the clarity of the research question and the systematic comparison across 18 datasets; the main weakness is that the experimental protocol and statistical evidence are not yet strong enough to support the abstract's positive claim as stated.","major_comments":[{"comment":"The experimental protocol does not state whether the QuickReduct ordering and the 90%/80% retention subsets are computed on the full dataset before the folds are separated or inside each training fold. If the former is the case, the held-out fold contributes to choosing the attribute order and subset used by FRRI, while the control condition receives no such test-set information, which biases the comparison in favor of the preprocessed variants. This is load-bearing because the positive recommendation for ofrfs-0.9 rests on a one-sided Wilcoxon p-value of 0.071 for balanced accuracy and on the corresponding rule-length result; even a small amount of leakage could plausibly produce such differences. Please clarify the protocol and, if feature selection was fitted on the full data, rerun the experiments with feature selection nested inside each training fold.","section":"Section 5.1"},{"comment":"The evidence for improved balanced accuracy is weaker than the abstract and conclusion suggest. The only directly reported comparison between ofrfs-0.9 and control is a one-sided Wilcoxon p-value of 0.071, which is not significant at the usual 0.05 level, and the 10% retention level was selected after inspecting Figure 1 across many retention levels without any correction for multiple comparisons. The Conover post-hoc tests only establish that ofrfs-0.9 beats pcc-0.9 and pcc-0.8, not that it beats control. Please report two-sided paired tests and multiplicity-corrected p-values for all retention levels, or downgrade the 'positively affects balanced accuracy' claim to an exploratory finding.","section":"Section 5.2, Table 2"},{"comment":"The negative conclusion that attribute order does not matter is based only on three global orderings applied before FRRI runs. Because rule shortening processes each training object independently over the fixed global sequence, a single global ordering is a restricted probe of the order effect; per-object attribute orderings might behave differently. The discussion paragraph before Section 6 partly acknowledges that feature selection removes attributes globally, but the abstract and conclusion state the broader conclusion without this qualification. Please restrict the claim to global orderings or provide experiments that vary per-object order.","section":"Section 3.2 and Section 5.2"}],"minor_comments":[{"comment":"The tables report only mean values; adding standard deviations or per-fold results would help readers assess variability, especially for the p=0.071 result.","section":"Tables 2-4"},{"comment":"The footnote says categorical features were removed from some datasets, but Table 1 does not state which datasets were affected or how many features were removed; please add this information.","section":"Section 5.1, Table 1"},{"comment":"The phrase 'if this amount is 1 or 0, we default to the next method' is unclear; please specify which method is the fallback and whether it was triggered for any dataset.","section":"Section 5.1"},{"comment":"The label 'average balanced accuracy ... over all datasets' does not specify whether the average is over the ten folds first and then over the 18 datasets, or over fold-dataset combinations; please clarify.","section":"Section 5.2, Figure 1"},{"comment":"The conclusion says the rule-length improvement appears in 'the majority of benchmark datasets,' but the reported Wilcoxon test is across datasets and does not directly establish a per-dataset majority; a sign test or per-dataset counts would support this phrasing.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"The protocol ambiguity about whether feature selection is nested inside the cross-validation folds is my main reason for major revision. If the authors confirm that feature selection is fitted within each training fold and the weak p-value remains after appropriate corrections, the paper could be acceptable after revision. I would also encourage the authors to release code and data, since the study is empirical and the key positive effect is borderline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The negative result — reordering attributes via FRFS, mutual information, or correlation does not change FRRI's accuracy, number of rules, or rule length — looks solid and is genuinely useful. The positive result, that dropping 10% of attributes via FRFS improves balanced accuracy and shortens rules, is weaker than the abstract suggests, and one protocol detail could puncture it.\n\nWhat's new: as far as I can tell, this is the first systematic test of attribute ordering in FRRI. The setup is reasonable: 18 KEEL datasets, the same predefined folds across conditions, three feature selectors, Friedman plus Conover post-hoc for global comparisons, and a one-sided Wilcoxon for the specific contrast. The writing is clear, and the explanation for why light removal might help while heavy removal hurts — FRRI prunes per-object, global feature selection can't — is plausible.\n\nSoft spots, in order of importance. First, Section 5.1 never states where the feature-selection preprocessing is fitted. Is QuickReduct and the 90% retention computed inside each training fold, or on the full dataset before the folds are split? If the latter, the held-out fold influences the attribute order and subset, while the control gets no such information. That biases the comparison toward ofrfs-0.9. The text is terse enough that I can't rule it out, and the balanced-accuracy gain is a one-sided Wilcoxon p=0.071, so a small bias could manufacture it. The rule-length decrease is more believable, but it's part of the same pipeline. This is load-bearing, and it's easily fixed in revision.\n\nSecond, the tables report only means over folds — no variance, no per-dataset significance. Third, the 10% retention level is highlighted after inspecting the full curve in Figure 1, so there's a multiple-comparisons issue that isn't addressed. Fourth, no code or data is released, which would otherwise resolve the first concern. Minor: categorical features are dropped from datasets that have them, which narrows the scope.\n\nWho it's for: FRRI users and fuzzy-rough-set researchers. It's a narrow, honest empirical contribution, not a breakthrough. I'd send it to peer review because the negative result is worth recording and the protocol gap is fixable, but I'd ask the authors to state the fitting point and release code. If leakage turns out to be present, the positive recommendation evaporates and the paper becomes a null result with a nice auxiliary finding.","headline":"Useful negative result on attribute ordering in FRRI, but the positive FRFS recommendation rests on an unstated protocol detail that could easily flip it.","tokens_in":11835,"tokens_out":3883,"would_cite":false,"duration_ms":41771,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that reordering attributes does not improve FRRI's performance, while removing a small fraction of attributes with fuzzy rough feature selection improves balanced accuracy and shortens rules.","keywords":["fuzzy rough sets","rule induction","attribute ordering","feature selection","QuickReduct","balanced accuracy","interpretability","FRRI"],"falsifier":"On a small dataset with six attributes or fewer, compute FRRI's balanced accuracy and average rule length under every possible global attribute permutation; if any permutation differs from the default by more than the cross-validation variance, the claim that reordering has no effect is falsified. The same test can be run with per-object random permutations, which the paper does not examine.","tokens_in":11002,"feed_emoji":"📊","tokens_out":6029,"duration_ms":57469,"temperature":0.7,"pith_summary":"This paper asks whether the order in which attributes are presented to a greedy rule-shortening step changes the rules that Fuzzy Rough Rule Induction (FRRI) produces. Across 18 benchmark datasets, it finds that reordering attributes with three different feature-selection methods—fuzzy rough feature selection, mutual information, and Pearson correlation—does not significantly change balanced accuracy, rule count, or rule length. However, using fuzzy rough feature selection to also drop about 10% of the least relevant attributes does improve balanced accuracy and shortens the average rule, with little effect on the number of rules. The practical upshot is that FRRI users can ignore attribute ordering but may gain from a light feature-selection preprocessing step before running the algorithm.","feed_headline":"Reordering attributes won't improve FRRI; pruning 10% of them will.","feed_subtitle":"Dropping a few redundant attributes via fuzzy rough feature selection yields shorter rules and higher balanced accuracy.","key_machinery":"The load-bearing mechanism is the greedy rule-shortening pass of FRRI: for each training object, the algorithm walks through attributes in a fixed order and, for each, tries setting the condition to unused, dominant, dominated, or similar, keeping the first type that leaves the rule's covering set inside the object's decision class. Because it is greedy, the resulting rule can depend on the order visited; the paper probes this dependency by feeding FRRI attribute lists reordered by three feature-selection criteria, and by the hill-climbing QuickReduct algorithm used for fuzzy rough feature selection. The fuzzy B-indiscernibility relation (the t-norm aggregation of per-attribute similarity or dominance relations) and the fuzzy positive region's dependency degree $\\gamma_B$ are what let both FRRI's shortening and QuickReduct's search evaluate attribute subsets.","core_discovery":"FRRI constructs a fuzzy indiscernibility relation over training objects and turns each object into a total rule whose conditions are greedily shortened attribute by attribute; the authors hypothesised that changing this attribute order would produce smaller, shorter, more accurate rule sets, just as sorted input improves bin packing. The experiments show the hypothesis is false: reordering alone, whether by QuickReduct-style fuzzy rough feature selection, mutual information, or Pearson correlation, leaves balanced accuracy, ruleset size, and average rule length statistically unchanged. When the reordering is paired with the removal of a small fraction (about 10%) of attributes by fuzzy rough feature selection, balanced accuracy improves and rules become shorter, and this gain does not appear with the other two selection methods or with deeper pruning, which degrades accuracy and inflates the ruleset. The paper explains the asymmetry by noting that FRRI removes attributes per object, whereas preprocessing removes attributes globally, so dropping more than a few attributes can discard features that are essential for distinguishing particular objects.","pith_inferences":["A testable extension is to vary attribute order per training object rather than through a single global ordering; the paper's null result does not cover that setting.","Because the paper's own explanation says global pruning can discard object-specific attributes, a natural next experiment is to compare global fuzzy rough feature selection against a per-object minimal-redundancy criterion.","The planned reformulation of rule shortening as an exact optimisation problem would make attribute order irrelevant by construction, and this null result is consistent evidence that such a reformulation may be worth the extra cost.","On very high-dimensional datasets with hundreds of attributes, the greedy shortening step may become more sensitive to order, so re-testing across a wider range of feature counts would sharpen the scope of the claim."],"forward_implications":["FRRI's default attribute order can be kept; no ordering preprocessing is needed.","Removing up to about 10% of attributes with fuzzy rough feature selection is a safe preprocessing step that yields shorter rules and slightly higher balanced accuracy.","Aggressive pre-pruning, such as cutting to a minimal superreduct, should be avoided because it sharply lowers accuracy and increases the number of rules.","The two classical scoring methods, mutual information and Pearson correlation, do not reproduce the accuracy gain, so the choice of feature-selection method matters for FRRI.","The results support a division of labour: fuzzy rough feature selection removes globally redundant attributes, while FRRI's shortening handles object-specific redundancy during rule construction."],"supporting_citations":[{"why":"Introduces the FRRI algorithm whose rule-shortening and rule-selection steps are the object of the ordering experiments.","marker":"[2]"},{"why":"Supplies QuickReduct, the fuzzy rough feature-selection method whose ordered output and 10% pruning produce the reported accuracy and rule-length gains.","marker":"[6]"},{"why":"Defines the fuzzy positive region and dependency degree used by QuickReduct to evaluate attribute subsets.","marker":"[5]"},{"why":"Provides the 18 benchmark datasets and cross-validation folds used in the experiments.","marker":"[1]"},{"why":"Used to test whether differences across the algorithm variants are statistically significant.","marker":"[9]"},{"why":"Used for paired comparisons, including the weakly significant gain of the 10%-pruned fuzzy rough feature selection variant over control.","marker":"[18]"},{"why":"Motivates the hypothesis that sorted input improves greedy algorithms, the bin-packing analogy the paper tests and rejects.","marker":"[12]"}],"fun_headline_variants":["FRRI gains from pruning 10% of attributes, not reordering","Attribute order alone won't boost FRRI; a little pruning will","Fuzzy rough pruning beats reordering for FRRI's rule quality","Why FRRI needs a 10% attribute prune, not a reorder","Shorter rules, better accuracy from 10% fuzzy rough prune"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The experiments vary attribute order only through a single global ordering chosen before FRRI runs; the conclusion that order does not matter assumes that this is a faithful test of the per-object attribute order the greedy shortening step actually processes.","fun_headline_variants_meta":{"raw":{"variants":["FRRI gains from pruning 10% of attributes, not reordering","Attribute order alone won't boost FRRI; a little pruning will","Fuzzy rough pruning beats reordering for FRRI's rule quality","Why FRRI needs a 10% attribute prune, not a reorder","Shorter rules, better accuracy from 10% fuzzy rough prune"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000437,"raw_usage":{"total_tokens":2215,"prompt_tokens":935,"completion_tokens":1280,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":1200}},"tokens_in":551,"tokens_out":1280,"duration_ms":8889,"temperature":1.0,"reasoning_tokens":1200,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:14:40.612531+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small dataset with six attributes or fewer, compute FRRI's balanced accuracy and average rule length under every possible global attribute permutation; if any permutation differs from the default by more than the cross-validation variance, the claim that reordering has no effect is falsified. The same test can be run with per-object random permutations, which the paper does not examine.","supporting_citations":[{"cited_title":"Information Sciences686, 121362 (2025)","cited_arxiv_id":null,"evidence_quote":"Introduces the FRRI algorithm whose rule-shortening and rule-selection steps are the object of the ordering experiments."},{"cited_title":"In: Wang, G., Li, T., Grzymala-Busse, J.W., Miao, D., Skowron, A., Yao, Y","cited_arxiv_id":null,"evidence_quote":"Supplies QuickReduct, the fuzzy rough feature-selection method whose ordered output and 10% pruning produce the reported accuracy and rule-length gains."},{"cited_title":"Information Sciences 180(2), 209–224 (2010)","cited_arxiv_id":null,"evidence_quote":"Defines the fuzzy positive region and dependency degree used by QuickReduct to evaluate attribute subsets."},{"cited_title":"Journal of Multiple-Valued Logic and Soft Computing17, 255–287 (01 2010) 14 H","cited_arxiv_id":null,"evidence_quote":"Provides the 18 benchmark datasets and cross-validation folds used in the experiments."},{"cited_title":"Journal of the American Statistical Association32(200), 675–701 (1937)","cited_arxiv_id":null,"evidence_quote":"Used to test whether differences across the algorithm variants are statistically significant."},{"cited_title":"Journal of Computer and System Sciences 8(3), 272–314 (1974)","cited_arxiv_id":null,"evidence_quote":"Motivates the hypothesis that sorted input improves greedy algorithms, the bin-packing analogy the paper tests and rejects."}],"review_version":1}