{"id":"44739e88-3f57-4c15-9e02-69e1ebda55b2","arxiv_id":"1908.00671","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An interactive visualization system for selecting hyperspectral features in regression models is presented, with a case study showing expert-selected subsets matching or beating automatic selection for sorghum biomass prediction.","lead":"FeatureExplorer is an interactive visual analytics tool that lets plant scientists and remote sensing experts pick which spectral features to use when predicting crop biomass from hyperspectral images. The paper shows a case study where an expert trimmed 36 features to 10 and matched or beat automatic feature selection.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Case study cannot support the human-in-the-loop improvement claim: no numeric results, no control, and interactive test-set feedback may leak into feature selection.","rationale":"The reader's weakest assumption correctly identifies the absence of a control condition and the unreported numerical values as the core weakness of the case study. I agree with that reading and add a more concrete, technical mechanism: the evaluation panel displays test-set performance during interactive selection, which can leak test information into the feature-selection process. If the same partition is later used for the reported comparison, the manual subset is partially fitted to the test set, and the observed gain over automatic selection may be an artifact. The paper's system description, design goals, and model comparison in Table 1 are credible, and the workflow is plausible; the problem is specifically the evidentiary basis for the strongest claim. This is an evaluation gap that could be fixed with a proper held-out protocol rather than a fundamental flaw in the system, so the CONDITIONAL verdict remains appropriate without further adjustment.","tokens_in":7933,"tokens_out":5259,"duration_ms":53257,"concrete_test":"Obtain the case-study dataset and the recorded 10-feature subsets, then rerun the comparison under nested cross-validation: reserve a test partition that is never displayed during interaction; let the expert or an automated script choose features using only training-fold model performance; lock the final subset; evaluate it on the held-out test partition over 100 random splits. Report mean and 95% CI of R2 and RMSE for manual 10, automatic RFE 10, and all-36 baseline. If manual 10 does not beat automatic 10 outside the CI, the improvement claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 6 — 'This indicates the human-in-the-loop can improve the predictive performance of the regression model' — rests entirely on a single expert's session, with no reported R2/RMSE values for the manual-versus-automatic 10-feature comparison, no confidence intervals, and no control condition. A more specific mechanism compounds this: the evaluation panel shows ground-truth-vs-predicted scatterplots and R2/RMSE for the current feature subset while the user is choosing features. If those displayed metrics are computed on the same test partition used for the final comparison, then the 'improvement' from manual selection may be selection bias — the expert is effectively fitting the feature subset to the test data — rather than a benefit of the interface. In addition, the manual subset was constructed using the system's own SVR+RFE importance rankings and correlation matrix, so the comparison cannot separate the interface's contribution from the expert's prior domain knowledge or from chance among the many possible 10-feature subsets. Without a pre-registered held-out test set, a control condition (automatic selection with equal feature count, or expert selection without the system), and repeated trials with error bars, the case study cannot establish the stated claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents FeatureExplorer, an interactive visual analytics system for feature selection and exploration in regression models for hyperspectral images. The system combines a correlation matrix with hierarchical clustering, KDE-enhanced scatterplots, SVR+RFE feature importance ranking, and a model evaluation panel that reports R2 and RMSE for user-selected feature subsets. The authors describe design goals derived from collaborations with remote sensing experts, the system workflow and interface, and a case study in which an expert reduced 36 hyperspectral indices to 10 and achieved better performance than automatic feature selection. The paper concludes in Section 6 that the human-in-the-loop can improve predictive performance. The contributions claimed are the system itself, a comparison of several regression models, a workflow for tracing indices to underlying wavelengths, and the case study.","tokens_in":8177,"tokens_out":2620,"duration_ms":27030,"significance":"The manuscript addresses a genuine problem: hyperspectral data sets have many derived, correlated features, and domain experts need to identify interpretable, predictive subsets. The design of FeatureExplorer is well motivated, and the combination of automatic SVR+RFE ranking with interactive correlation and distribution views is sensible and potentially useful for remote sensing applications. The paper also provides a design process grounded in expert needs and a concrete workflow for tracing selected indices back to wavelengths, which is valuable for future data collection. However, the central claim that human-in-the-loop interaction improves predictive performance rests on a single case study with no reported numeric results, no control condition, and a potential risk of test-set leakage. The model comparison in Table 1 is also reported without variance or significance measures. These issues are load-bearing for the stated conclusions, though they are fixable with a more rigorous evaluation. If the system is meant as a practical tool, the paper would be strengthened by making the evaluation reproducible and statistically grounded.","major_comments":[{"comment":"The conclusion that \"the human-in-the-loop can improve the predictive performance of the regression model\" is not quantitatively supported. The case study reports only qualitative comparisons (\"slightly better,\" \"improved,\" \"outperformed\") and never gives the actual R2 or RMSE values for the manually selected 10-feature subset versus the automatic 10-feature subset. Without numeric results, the reader cannot assess the magnitude of the improvement or its stability. Please report the exact performance values, ideally with standard deviations over repeated cross-validation trials.","section":"Section 6, final paragraph"},{"comment":"The evaluation panel in Figure 2 displays the scatterplot of ground truth versus predicted values along with R2 and RMSE for the current feature subset while the user is selecting features. If these displayed metrics are computed on the same test partition that is later used for the final comparison, the expert's manual selection is effectively fitting the feature subset to the test data, and the observed improvement may be selection bias rather than a benefit of the interface. The manuscript should clarify which data partition is used for the interactive evaluation metrics and should reserve a truly held-out test set that is never shown during the interactive session.","section":"Section 5.2, evaluation panel; Section 6"},{"comment":"The case study provides no control condition. The manual selection was performed by one expert using the system's own rankings and correlation matrix, so the comparison with automatic selection cannot separate the contribution of the interface from the expert's prior domain knowledge or from chance among the many possible 10-feature subsets. To support the central claim, the authors should add a controlled comparison, such as expert selection without the system, automatic selection with the same feature count, and ideally repeated trials with multiple experts or multiple datasets, with statistical measures of variability.","section":"Section 6, case study"},{"comment":"Table 1 reports average R2 values over 100 trials for six regression models across ten dates, but no standard deviations, confidence intervals, or significance tests are provided. Several rows contain ties (e.g., 0.20, 0.27, 0.51, 0.52), so the claim that \"SVR outperforms other models\" is not supported by the evidence as presented. Either add measures of variance and appropriate significance tests, or soften the claim to a descriptive statement about the observed mean values.","section":"Section 5.3, Table 1"}],"minor_comments":[{"comment":"The text says \"UA Vs\" for unmanned aerial vehicles; this should be \"UAVs\".","section":"Section 2"},{"comment":"The caption contains a typo: \"10 datesets\" should be \"10 datasets\".","section":"Section 5.3, Table 1 caption"},{"comment":"The paper states that R2 and RMSE are \"highly correlated,\" which is imprecise; R2 and RMSE are functionally related for a fixed data set, but the phrase \"highly correlated\" suggests an empirical relationship that is not demonstrated. Consider rephrasing.","section":"Section 5.2"},{"comment":"The notation in Equation 1 could be clarified by explicitly noting that the ranking score maps the most important feature (rank 1) to score 1 and the least important (rank d) to score 0; this is not immediately obvious from the expression.","section":"Section 5.2, Equation 1"},{"comment":"The subfigures in Figure 3 are referenced as Figure 3(1) through Figure 3(6), but the figure as printed does not clearly label each subimage with these numbers; please add visible subfigure labels.","section":"Figure 3"},{"comment":"The paper refers to the 36 hyperspectral vegetation indices listed in reference [16], but the actual list of indices is not provided in the manuscript. Including the index names and formulas in an appendix or supplementary material would make the study more reproducible.","section":"Section 2, feature list"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant topic for the journal and the system design is clearly presented. The main weakness is the evidence for the central human-in-the-loop performance claim, which is currently based on a single anecdotal case study. I believe this is recoverable with additional quantitative evaluation and a more carefully described experimental protocol, so I recommend major revision rather than rejection. I would not require the authors to conduct a full user study, but they should at least report the missing numbers, add a suitable control or baseline, and address the test-set leakage concern explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it describes a genuinely plausible visual analytics system for feature selection in hyperspectral regression, and the integration of correlation matrices, KDE scatterplots, SVR+RFE importance ranking, and wavelength tracing is a real extension of prior work like INFUSE and RegressionExplorer. Second, the headline claim that interactive human selection improves predictive performance is supported by only one expert's anecdotal session, with no numeric results, no control, and a potential leakage issue. The idea is good; the evidence is not yet there.\n\nWhat is actually new: the workflow that lets agronomy experts reduce 36 hyperspectral indices to a smaller set while tracing those indices back to the underlying wavelengths. That traceability is a practical contribution with potential cost savings for future data collection. The system design is sensible, the views are well-integrated, and the case study shows how an expert actually used the tool. The paper is also clearly written and honest about the collaboration.\n\nThe soft spots are about the evaluation, not the system itself. The case study in Section 6 concludes that the human-in-the-loop improves predictive performance, but it reports no R2 or RMSE values for the manual-versus-automatic 10-feature comparison, no confidence intervals, and no repeated trials. The comparison also has no control condition — we do not see expert performance without the system, or automatic selection with the same feature count. More specifically, the evaluation panel shows ground-truth-versus-predicted scatterplots and R2/RMSE while the user is choosing features. If those metrics are computed on the same test partition used for the final comparison, then the expert is effectively fitting the feature subset to the test data, and the 'improvement' is selection bias rather than interface value. The paper does not clarify which split feeds the live panel, so this is a real risk rather than a proven flaw.\n\nTable 1 compares six regression models across ten dates, but reports only average R2 with no variance or significance tests, and several values are tied. The claim that SVR is best is therefore weakly supported, though not central to the system contribution. No code or data are released, which makes the comparison harder to verify.\n\nFor a visual analytics audience, a case study is a normal evaluation format, and the system itself is a legitimate contribution. But the specific claim of human-in-the-loop improvement needs a pre-registered hold-out test set, automatic selection as a control, and repeated trials with error bars. If those were added, the paper would be substantially stronger.\n\nMy recommendation: a serious editor should send this to peer review. The system is worth examining, and the evaluation can be fixed. Just do not let the current case study stand as evidence for the headline claim.","headline":"A useful visual analytics system with a clear workflow, but the claim that human-in-the-loop selection improves prediction rests on a single uncontrolled case study with no reported numbers.","tokens_in":8686,"tokens_out":1759,"would_cite":false,"duration_ms":18100,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FeatureExplorer shows interactive expert feature selection can beat automatic selection and cut 36 hyperspectral indices to 10.","keywords":["hyperspectral imaging","feature selection","visual analytics","support vector regression","recursive feature elimination","biomass prediction","human-in-the-loop"],"falsifier":"Compare the same experts selecting ten features with FeatureExplorer against a non-interactive ranked list on the same images; if $R^2$ improves equally in both conditions, the claim that interactivity improves prediction is falsified. The paper would also be more directly testable if it reported the numeric $R^2$ and RMSE for the manual 10-feature versus automatic 10-feature subsets instead of only stating the manual subset performed better.","tokens_in":7779,"feed_emoji":"🌾","tokens_out":6526,"duration_ms":62061,"temperature":0.7,"pith_summary":"FeatureExplorer is a visual analytics system that lets a plant or remote-sensing expert drive feature selection for regression models instead of trusting an automatic selector alone. The paper demonstrates the workflow on sorghum wet-biomass prediction: hyperspectral images yield 36 vegetation indices, and the expert uses correlation views, scatterplots, and SVR+RFE importance rankings to choose a subset, evaluate its $R^2$ and RMSE on the fly, and trace indices back to the wavelengths that feed them. The central claim is that this human-in-the-loop process can improve predictive performance; in the case study, an expert-selected 10-index subset beat the automatic 10-index selection and matched a 20-index automatic selection. If true, the system offers a practical route to smaller, more interpretable feature sets for high-dimensional remote sensing and guides future data collection to the pertinent wavelengths.","feed_headline":"Interactive feature selection beats automatic picks on 10 features","feed_subtitle":"Case study: expert-guided subset of 10 hyperspectral indices outperformed automatic selection for sorghum biomass prediction.","key_machinery":"The mechanism is the closed loop among four coordinated views. A Pearson correlation matrix with hierarchical clustering groups redundant indices so an expert can diversify the subset; a KDE-augmented scatterplot exposes distributions and outliers; SVR+RFE produces a $[0,1]$ importance score for each feature by averaging normalized RFE ranks across cross-validation folds (Equation 1); and the evaluation panel shows predicted versus ground truth, $R^2$, RMSE, feature importance, and a histogram of underlying wavelengths. Drag-and-drop feature lists connect these views, allowing iterative refinement and immediate retraining.","core_discovery":"The paper's central claim is that a visual analytics system built around jointly viewing correlation structure and model-based importance enables a domain expert to select a smaller, better-performing feature subset than automatic selection alone. In the sorghum case, the expert started from SVR+RFE ranking, used the correlation matrix to choose features from different clusters, and built a 10-index model whose performance was nearly equal to the 20-index automatic model; when directly compared at 10 features, the manual subset outperformed the automatic subset. The system also records the reflectance wavelengths underlying each selected index, letting the expert see which spectral regions matter at each growth stage. The paper concludes from this case that the human-in-the-loop improves predictive performance.","pith_inferences":["My inference: the reported gain is not yet attributable to interface interactivity; a controlled experiment with and without the system is required before 'human-in-the-loop improves predictions' can be separated from expert knowledge.","My inference: the successful heuristic of picking one feature per correlation cluster suggests a promising automated extension that recommends diverse, high-importance subsets rather than only top-ranked features.","My inference: if wavelength tracing proves reliable across more dates and varieties, it could guide sensor design to fewer bands, but the paper does not quantify the cost savings or verify that the traced wavelengths are causally important."],"forward_implications":["A 10-index subset chosen by an expert can match the performance of a 20-index automatic subset, so the workflow can shrink the feature space without sacrificing accuracy.","Features selected from one date partially transfer to a nearby date; the July 18 subset did not collapse on July 30 data, and revising it with new correlation information improved results.","The system's wavelength histogram maps each selected index to its constituent reflectance bands, enabling future data collection to focus on pertinent wavelength regions at each growth stage.","The workflow applies to other high-dimensional, correlated feature sets (multispectral or LiDAR features) and other regression targets, since the regression model is pluggable."],"supporting_citations":[{"why":"Supplies the 36 hyperspectral vegetation indices and their formulas that form the feature space.","marker":"[16]"},{"why":"Supplies the SVR model used throughout for regression and feature ranking.","marker":"[23]"},{"why":"Supplies the RBF-kernel SVM feature selection approach combined with RFE for automatic selection.","marker":"[17]"},{"why":"Describes the UAV remote sensing campaign and sorghum biomass prediction this case study builds on.","marker":"[18]"},{"why":"Provides the time-series UAV image feature pipeline used for preprocessing and biomass prediction.","marker":"[27]"},{"why":"Documents the multi-sensor UAV mapping system and data preprocessing for the hyperspectral images.","marker":"[9]"},{"why":"Explains sorghum growth stages, used to interpret why model performance differs across image dates.","marker":"[12]"},{"why":"Shows spectral reflectance changes with plant growth, used to explain why July 30 features differ from July 18.","marker":"[3]"},{"why":"Supplies the RBF kernel and grid-search hyperparameter method used in each SVR training.","marker":"[1]"}],"fun_headline_variants":["Expert picks 10 features, beats auto selection in hyperspectral","Human insight beats algorithms in hyperspectral feature selection","Interactive tool lets experts choose better features than autopilot","Visual analytics beats automatic feature selection in hyperspectral","Expert-guided 10 features outperform auto picks in hyperspectral"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The case study's conclusion assumes the improvement comes from the interactive system rather than from the expert's existing domain knowledge, from chance, or from the particular features he chose, because the paper reports no control condition in which the same task is done without the system.","fun_headline_variants_meta":{"raw":{"variants":["Expert picks 10 features, beats auto selection in hyperspectral","Human insight beats algorithms in hyperspectral feature selection","Interactive tool lets experts choose better features than autopilot","Visual analytics beats automatic feature selection in hyperspectral","Expert-guided 10 features outperform auto picks in hyperspectral"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000409,"raw_usage":{"total_tokens":2031,"prompt_tokens":766,"completion_tokens":1265,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":382,"completion_tokens_details":{"reasoning_tokens":1187}},"tokens_in":382,"tokens_out":1265,"duration_ms":10578,"temperature":1.0,"reasoning_tokens":1187,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:38:46.330596+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the same experts selecting ten features with FeatureExplorer against a non-interactive ranked list on the same images; if $R^2$ improves equally in both conditions, the claim that interactivity improves prediction is falsified. The paper would also be more directly testable if it reported the numeric $R^2$ and RMSE for the manual 10-feature versus automatic 10-feature subsets instead of only stating the manual subset performed better.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 36 hyperspectral vegetation indices and their formulas that form the feature space."},{"cited_title":"Sch¨olkopf and A","cited_arxiv_id":null,"evidence_quote":"Supplies the SVR model used throughout for regression and feature ranking."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RBF-kernel SVM feature selection approach combined with RFE for automatic selection."},{"cited_title":"Liang, L","cited_arxiv_id":null,"evidence_quote":"Describes the UAV remote sensing campaign and sorghum biomass prediction this case study builds on."},{"cited_title":"Zhang, A","cited_arxiv_id":null,"evidence_quote":"Provides the time-series UAV image feature pipeline used for preprocessing and biomass prediction."},{"cited_title":"Gerik, B","cited_arxiv_id":null,"evidence_quote":"Explains sorghum growth stages, used to interpret why model performance differs across image dates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows spectral reflectance changes with plant growth, used to explain why July 30 features differ from July 18."},{"cited_title":"Alpaydin","cited_arxiv_id":null,"evidence_quote":"Supplies the RBF kernel and grid-search hyperparameter method used in each SVR training."}],"review_version":1}