{"id":"108c1f8f-d725-4a7c-ace4-e0dfdc0be5a6","arxiv_id":"2506.20359","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A taxonomy-based feature selection method that selects whole categories of trajectory features gives comparable or better classification results than forward and backward selection, but the gains are not statistically significant.","lead":"This paper tests a feature selection method for trajectory data that first groups features into four categories, such as speed and curvature, and then picks whole categories instead of individual features. It reports that this category-level approach matches or beats classical forward and backward selection in many cases while cutting computation time, though the authors note the differences were not statistically significant.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Taxonomy 'wins' may be an oracle-selection artifact: the best of 15 category combinations is chosen from test results, while forward/backward baselines are not given that choice.","rationale":"The paper's central claim is that taxonomy-based selection is comparable or superior while drastically faster. For this to hold, the performance comparison must be valid. The most load-bearing weakness is not the taxonomy's semantic validity (though that matters for interpretability) but the comparison protocol: Section 4 reports the best of 15 taxonomy combinations per dataset/model, and Section 3.6 does not state a training-only rule for choosing that combination. Picking the maximum over 15 test-evaluated candidates is an oracle procedure; forward/backward selection cannot do this, so the reported 16/24 advantage may be multiple-comparison bias. The authors' own Section 5.1 says improvements were not statistically significant and results are inconclusive, which further weakens the abstract's 'consistently comparable or superior' phrasing. A nested-CV re-analysis or a stated internal selection rule would settle this. I recommend rejecting the current version, not because the idea is bad, but because the central empirical claim is not yet supported; a revised version with nested selection, significance testing, and measured timings could be a reasonable proof of concept. The reader identified a potential leakage problem in the rationale, which overlaps with this concern, but their stated weakest assumption was taxonomy validity rather than the selection protocol, hence 'partial' agreement.","tokens_in":20633,"tokens_out":5834,"duration_ms":67332,"concrete_test":"Re-run one complete dataset/model case (e.g., Arctic Fox + Random Forest) with nested cross-validation: for each outer training fold, select the taxonomy combination by highest mean Weighted-F1 on inner validation folds; then evaluate on the held-out outer fold. Apply the same nested selection to forward and backward selection. If the taxonomy method no longer wins more often than chance (or its gains fall within paired bootstrap confidence intervals that include 0), the reported advantage is an oracle-selection artifact. Independently, verify whether Section 4's 'best feature subset' was chosen before or after seeing test labels; if after, report a corrected comparison that accounts for the maximum over 15 candidates.","verdict_should_be":"REJECT","load_bearing_attack":"Section 3.5 defines 15 taxonomy combinations and Section 3.6 says each combination produces twenty sets of results. Section 4 then reports 'the best feature subset result from our taxonomy method' and Tables 4.8/4.10/4.12 list the best-performing combination per model. Nothing in the methodology specifies that the best combination is selected on training/validation data; the natural reading is that the combination with the highest median test Weighted-F1 is chosen post hoc. This makes the comparison structurally unfair: the taxonomy method is allowed to pick the maximum over 15 candidate feature sets after seeing test performance, while the forward/backward baselines must commit to a feature subset during training. The 16/24 win count in Tables 4.13/4.14 could therefore be a multiple-comparisons artifact rather than evidence that the taxonomy itself helps. The authors' own Section 5.1 concedes that the improvements 'were not statistically significant' and that the overall results are 'inconclusive,' which is consistent with this concern. Without a stated, data-independent rule for choosing among the 15 combinations, the central claim of consistently comparable or superior performance is not supported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This Bachelor's thesis proposes a taxonomy-based feature selection method for trajectory data, grouping low-level movement features into geometric (curvature, indentation) and kinematic (speed, acceleration) categories and evaluating all 15 non-empty category combinations against forward and backward selection on three datasets (Arctic Fox, AIS, Tropical Cyclone) and four classifiers. The paper reports that the taxonomy method wins 16 of 24 model-dataset comparisons, provides frequency analyses of preferred categories, and claims reduced computational time and improved interpretability, but it also concedes in Section 5.1 that the performance improvements were not statistically significant and that the results are inconclusive.","tokens_in":20851,"tokens_out":5149,"duration_ms":51614,"significance":"The underlying research question is worthwhile: category-level feature selection could reduce the combinatorial search space and provide a more interpretable selection process than greedy wrappers, which is relevant for explainable AI and trajectory analysis. The frequency analyses in Figures 4.8, 4.13, and 4.18 are an interesting way to connect selected feature sets to dataset sensitivity, and the authors are transparent in their own discussion about the lack of statistical significance. However, the current experimental design contains a likely selection-on-the-test-set problem, and the central performance claims in the abstract are not supported by the reported evidence. If the comparison were redone with a principled rule for choosing among the 15 taxonomy combinations and with significance testing, the study could make a modest but useful empirical contribution.","major_comments":[{"comment":"The methodology never specifies how the best taxonomy combination is chosen among the 15 candidates. Section 3.6 states that each combination produces twenty sets of results, and Section 4 reports 'the best feature subset result from our taxonomy method' (e.g., before Figure 4.4). Tables 4.8, 4.10, and 4.12 list the best-performing combination per model and dataset. If, as the natural reading suggests, these combinations are selected by their test-set median Weighted-F1 after the fact, the taxonomy method is allowed to pick the maximum over 15 candidate feature sets on the test set, whereas the forward/backward baselines must commit to a subset during training. The 16/24 win count in Tables 4.13 and 4.14 is then a multiple-comparisons artifact rather than evidence for the taxonomy. The authors must specify a data-independent rule for selecting among the 15 combinations (e.g., selection on validation folds within the training split) or treat the comparison as exploratory.","section":"§3.5–3.6, §4"},{"comment":"The abstract's claim of 'robust evidence' and 'consistently achieved comparable or superior predictive performance' is directly contradicted by Section 5.1, which states that 'the improvements were not statistically significant' and that 'overall the results of this experiment is inconclusive on which feature selection method performs better.' The empirical section should be rewritten to match this admitted inconclusiveness; at most the paper can claim that taxonomy-based selection is competitive in some settings and faster, pending the timing evidence.","section":"Abstract; §5.1"},{"comment":"No statistical significance testing is reported for the win counts in Tables 4.13 and 4.14. The medians in Tables 4.7, 4.9, and 4.11 differ by small amounts (e.g., AIS Random Forest non-tuned: 0.7598 versus 0.7516), and with only 20 runs per condition the 16/24 preference counts are not established as real differences. Report paired tests across the 20 seed/CV iterations (e.g., Wilcoxon signed-rank) or confidence intervals for the median differences, for each model-dataset pair and both tuned and non-tuned settings.","section":"§4.4–4.5, §5.1"},{"comment":"The claim that 'the time taken to select features was drastically reduced' is not supported by any reported measurement. Section 5.1 only says speed was observed qualitatively, and no wall-clock times or complexity analysis are given. Provide actual runtime measurements for the three feature selection methods, or at least a formal complexity comparison (e.g., number of model fits: 15 taxonomy combinations versus the number of evaluations in forward/backward selection).","section":"§5.1; Abstract"}],"minor_comments":[{"comment":"The text says 'using five decimal places of π: 14159, 26535, 89793, 23846'; these are four 5-digit blocks of the decimal expansion, so the phrase 'five decimal places' is inaccurate.","section":"§3.6"},{"comment":"The preprocessing class is called 'Standard Scalar' in the text; the correct scikit-learn name is StandardScaler.","section":"§3.8"},{"comment":"The text refers to a Venn diagram that 'visualizes the expected combinations,' but the figure caption and surrounding text do not explain what the overlaps represent; please clarify or remove the reference.","section":"Figure 3.3"},{"comment":"The tables are titled 'Preferred Feature Selection Method,' but it is not stated that 'preferred' means the highest median Weighted-F1; please state the criterion explicitly.","section":"Tables 4.13–4.14"},{"comment":"The sentence 'That too in two instances as such we can conclude in this case of the Arctic Fox dataset taxonomy-based feature selection was able to produce the better predictor' has grammar and punctuation issues and should be revised.","section":"§4.1"},{"comment":"Reference [27] is an API documentation page; the original source of the AIS dataset should be cited instead.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a Bachelor's degree project and reads as such: the experimental effort is substantial, but the analysis is shallow relative to the claims. The taxonomy source [3] is an unpublished arXiv preprint from the same research group as the supervisor, which is not a problem per se but should be clearly positioned. The main risk for a journal is that the abstract and conclusion overstate what the authors' own Section 5.1 concedes is an inconclusive result. The thesis promises a reproducible pipeline with scripts and algorithms, but no code repository or supplementary material is provided for verification. If the comparison is redone with a proper selection rule and significance tests, the paper could become a modest empirical contribution; as is, the central comparison is not reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this is a legitimate proof of concept, not a demonstrated win. The new piece is applying Yashar et al.'s taxonomy (curvature, indentation, speed, acceleration) as the feature selection search space for supervised trajectory classification, which nobody seems to have done. That is a real, if incremental, contribution. The paper also does a few things well: the pipeline is transparent, the limitations section is honest, and Section 5.1 explicitly concedes that the improvements were not statistically significant. The frequency analysis for dataset sensitivity is a nice interpretability byproduct, though it depends on the flawed selection step below.\n\nThe main soft spot is the one the stress-test flagged, and it lands. The method evaluates all 15 taxonomy combinations, and the results sections report the best combination per model and dataset, with no stated rule for choosing it on training or validation data. So the taxonomy method is effectively allowed to pick its winning subset after seeing test performance, while forward and backward selection have to commit during training. The 16/24 win count is a multiple-comparisons artifact, not evidence that the taxonomy itself helps. The authors' own Section 5.1 saying the results are inconclusive confirms this. A secondary issue is that the abstract claims \"robust evidence,\" which is at odds with the discussion.\n\nEven if the oracle-selection problem were fixed, the comparison would need significance testing to support any claim of comparable or superior performance. The paper also doesn't ship code or data, so the promised reproducible pipeline isn't actually available. None of these flaws kill the idea—a properly designed follow-up with nested cross-validation or a pre-registered rule for choosing among the 15 combos could make a solid paper—but in current form the empirical case is not made.\n\nWho should read it: people working on trajectory feature selection will find the categorization and the efficiency argument worth knowing about; methodologists might use it as a teaching example of test-set selection. It deserves a serious referee because the question is real and the flaws are fixable, but I would not cite it in its current state. Send it to peer review with major revision expected.","headline":"The core idea—using a predefined trajectory taxonomy as the search space for feature selection—is genuinely new and worth discussing, but the evaluation lets the method cherry-pick its best category combination from test results, so the headline performance claim is not supported.","tokens_in":21309,"tokens_out":1679,"would_cite":false,"duration_ms":23217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a taxonomy-based feature selection method, which groups trajectory features into curvature, indentation, speed, and acceleration, matches or beats greedy forward/backward selection while drastically cutting…","keywords":["Trajectory analysis","Feature selection","Taxonomy-based feature selection","High-dimensional data","Spatio-temporal data","Explainable AI","Dimensionality reduction"],"falsifier":"Run the same pipeline on a trajectory dataset where a known single feature within one category carries nearly all predictive signal while the rest of the category is noise. If the best individual-feature selection beats every one of the 15 category combinations by a margin beyond random variation, the claim that taxonomy-based selection achieves comparable or superior predictive performance would be falsified.","tokens_in":20464,"feed_emoji":"🛰️","tokens_out":6598,"duration_ms":67411,"temperature":0.7,"pith_summary":"This paper tries to establish that feature selection for trajectory data can be done at the level of meaningful movement categories rather than individual features, without sacrificing predictive accuracy. It organizes extracted trajectory features into four groups: curvature (distance geometry), indentation (angles), speed, and acceleration, and searches over the 15 nonempty combinations of these groups instead of over individual features. Across three datasets (arctic fox, vessel traffic, tropical cyclones) and four classifiers, the category-level selection matched or beat forward and backward greedy selection in most of the 24 experiments, and the restricted search space made selection substantially faster. A sympathetic reader would care because the approach promises a cheaper, more interpretable route into high-dimensional movement data: the selected categories themselves say which aspects of movement drive classification, rather than a black-box list of feature indices.","feed_headline":"Taxonomy-based feature selection matches or beats greedy selection","feed_subtitle":"Grouping features by curvature, indentation, speed, and acceleration cuts search time and reveals what drives each dataset.","key_machinery":"The central machinery is the four-leaf taxonomy of trajectory features, taken from prior work: curvature (distance-geometry signatures measuring straightness), indentation (statistical summaries of turn angles), speed (statistical summaries of speed magnitudes), and acceleration (statistical summaries of acceleration magnitudes). The method performs feature selection by choosing one of the 15 nonempty subsets of these four category leaves, so the selected 'feature set' is a category combination, not a list of individual variables. This collapses the combinatorial search space and makes the output interpretable as a statement about which movement properties matter; the paper uses frequency counts over the best category combinations across models and seeds to read each dataset's sensitivity.","core_discovery":"The central claim is that a taxonomy-based feature selection method, one that groups trajectory features into geometric and kinematic categories and selects whole categories rather than individual features, consistently achieves predictive performance comparable to or better than classical greedy wrapper methods. Because the lowest taxonomic level has four categories, the search is over $2^4-1=15$ combinations, a drastic reduction of combinatorial space relative to individual feature subsets; this is claimed to account for the large reduction in selection time. The authors also claim that the taxonomy adds a layer of interpretability: frequency analysis over the best category combinations reveals which movement aspects (curvature, indentation, speed, acceleration) each dataset is sensitive to. They note that the numerical improvements over baselines were not statistically significant, and frame the contribution as a step toward explainable and efficient feature selection for trajectory data rather than a categorical win over all baselines.","pith_inferences":["If within-category feature redundancy is low, the same category-level trick should scale to larger taxonomies than four leaves; any hierarchical feature organization for movement data could be searched in $2^k-1$ combinations for $k$ categories, growing only exponentially in the number of categories rather than in the number of features.","The frequency-analysis idea suggests a testable extension: for a new trajectory dataset, pre-screening categories by which ones appear in the best subsets could tell analysts which feature families to collect or discard before expensive data processing.","The paper's admitted lack of statistical significance implies a sharper experiment would compare category-level selection against within-category greedy selection on the same training folds; if within-category selection consistently wins, the taxonomy's groupings would need per-dataset refinement.","One could use the taxonomy as a structured regularizer in embedded or deep models, constraining feature importance to be uniform within categories, rather than as a discrete subset selector."],"forward_implications":["On trajectory datasets with many derived features, practitioners can replace per-feature greedy search with category-level search and expect selection time to drop sharply while keeping accuracy within the same range.","The best category combination per dataset can act as a diagnostic: for instance, the arctic fox results point to curvature, indentation, and acceleration as the informative movement aspects, with speed rarely selected.","Because the improvements over baselines were not statistically significant, the method's practical advantage is primarily computational and interpretive rather than a guaranteed accuracy gain.","The method's success on the small arctic fox dataset suggests category-level selection may be especially useful when sample sizes are too small for per-feature wrappers to generalize."],"supporting_citations":[{"why":"Supplies the geometric/kinematic taxonomy (curvature, indentation, speed, acceleration) that the proposed feature selection method operates on.","marker":"[3]"},{"why":"Defines the classical filter/wrapper/embedded framing and the forward/backward selection baselines the method is compared against.","marker":"[22]"},{"why":"Provides the arctic fox trajectory dataset used as one of the three evaluation domains.","marker":"[26]"},{"why":"Provides the AIS vessel trajectory dataset used as the second evaluation domain.","marker":"[27]"},{"why":"Provides the tropical cyclone trajectory dataset used as the third evaluation domain.","marker":"[28]"},{"why":"Supplies the random forest classifier, the model family on which the taxonomy method showed its most consistent gains.","marker":"[30]"},{"why":"Supplies the gradient-boosted tree classifier used in the cross-model evaluation.","marker":"[31]"}],"fun_headline_variants":["Taxonomy-based feature selection cuts time, matches greedy accuracy","Grouping trajectory features by structure speeds selection","Taxonomy shrinks feature selection search to 15 combos","Interpretable taxonomy cuts feature search time, keeps accuracy","Taxonomic grouping yields fast, interpretable feature selection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the four predefined categories, curvature, indentation, speed, and acceleration, are internally coherent groupings for feature selection; if a category mixes highly predictive and useless features, restricting the search to the 15 category combinations can never find the better individual subset.","fun_headline_variants_meta":{"raw":{"variants":["Taxonomy-based feature selection cuts time, matches greedy accuracy","Grouping trajectory features by structure speeds selection","Taxonomy shrinks feature selection search to 15 combos","Interpretable taxonomy cuts feature search time, keeps accuracy","Taxonomic grouping yields fast, interpretable feature selection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001087,"raw_usage":{"total_tokens":4565,"prompt_tokens":991,"completion_tokens":3574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":607,"completion_tokens_details":{"reasoning_tokens":3497}},"tokens_in":607,"tokens_out":3574,"duration_ms":26027,"temperature":1.0,"reasoning_tokens":3497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:49:42.250336+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a trajectory dataset where a known single feature within one category carries nearly all predictive signal while the rest of the category is noise. If the best individual-feature selection beats every one of the 15 category combinations by a margin beyond random variation, the claim that taxonomy-based selection achieves comparable or superior predictive performance would be falsified.","supporting_citations":[{"cited_title":"An introduction to variable and feature selection,","cited_arxiv_id":null,"evidence_quote":"Defines the classical filter/wrapper/embedded framing and the forward/backward selection baselines the method is compared against."},{"cited_title":"Data from: Movement tactics of a mobile predator in a meta-ecosystem with fluctuating resources: the arctic fox in the high arctic,","cited_arxiv_id":null,"evidence_quote":"Provides the arctic fox trajectory dataset used as one of the three evaluation domains."},{"cited_title":"Ais ship type codes reference,","cited_arxiv_id":null,"evidence_quote":"Provides the AIS vessel trajectory dataset used as the second evaluation domain."},{"cited_title":"Interna- tional best track archive for climate stewardship (ibtracs) project, version 4,","cited_arxiv_id":null,"evidence_quote":"Provides the tropical cyclone trajectory dataset used as the third evaluation domain."}],"review_version":1}