{"id":"2946c852-5ec9-4abf-9c3b-5fc7d5fe5e70","arxiv_id":"2505.06325","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"An interactive tool lets users reshape a model's latent space during training via a human-teacher loss term, with a 14-person user study reporting modest accuracy improvements.","lead":"HILL lets people drag points in a 2D view of a model's internal representation while it trains, using those edits as extra training signals. A 14-person study reports small accuracy gains over passive training, but the evaluation relies on the same validation set users see during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation uses the same validation split as live feedback and as reported metric, so the generalization claim is not established by the presented data.","rationale":"The reader's weakest assumption precisely identifies the most load-bearing vulnerability: the validation set serves both as the participants' live feedback signal and as the paper's reported evaluation metric. This is not a minor methodological quibble; it strikes directly at the central claim that human guidance improves performance while maintaining generalization. If participants are effectively optimizing the validation split, the presented accuracy curves in Figure 5 are not evidence about generalization, only about fitting that split. The additional absence of a specified L_human equation further weakens reproducibility, but even the empirical HCI evidence cannot rescue the central claim without a properly separated test set and statistical comparison. The reader's REJECT verdict is consistent with this reasoning, so no verdict change is warranted. The proposed concrete test—evaluating on a truly held-out split, either in a fresh study or by replaying logged interactions through the released code—would settle whether the concern lands. This is a single, feasible check that directly targets the weakest link in the argument.","tokens_in":11560,"tokens_out":2568,"duration_ms":27053,"concrete_test":"Re-run the user study (or replay recorded interaction logs using the provided GitHub code) with a held-out test split that is never displayed during interaction: record the validation accuracy shown to participants, then evaluate all final models on the held-out test split. If the average test accuracy of HILL-guided models does not exceed a baseline trained for the same number of epochs, or if the validation-minus-test gap is substantially larger than for baseline models, the generalization claim is refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HILL improves performance 'while maintaining generalization' rests on Section 4.2.1, where the reported accuracy is computed on the validation set. Section 4.1 states that participants see the updated model performance after each training iteration, and Section 4.2.1 explicitly says 'The evaluation was conducted on the unseen validation dataset, the same as during training.' The metric participants interactively optimize is therefore identical to the metric used as evidence of generalization. The reported gains (CIFAR-10 +1.6 pp to 87.3%, PAMAP2 +2.2 pp to 75%) may result from participants effectively tuning to that validation split rather than from human intuition transferring to unseen data. With n=14, no held-out test set, no error bars or inferential statistics, and only best/worst participant ranges shown, the claim that generalization is maintained is unsupported. This is compounded by the absence of any explicit equation for L_human in Section 3.2, which means the mechanism cannot be independently checked even if the empirical claim were secured. The qualitative HCI results are plausible but do not establish the quantitative generalization claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HILL, an interactive framework that lets users inspect and modify the latent space of a deep model during training. The user's drag-and-drop manipulations are encoded in a loss term inspired by knowledge distillation, which is combined with cross-entropy and a scale-regularization term. The authors report a user study with 14 participants on CIFAR-10 and PAMAP2 in which human-guided training improved validation accuracy by 1.6 and 2.2 percentage points respectively, while also claiming faster convergence and maintained generalization. They also present qualitative analyses of user strategies, questionnaires, and interviews, and release the tool as open source.","tokens_in":11731,"tokens_out":2827,"duration_ms":28844,"significance":"If the central quantitative claims were sound, HILL would be a valuable contribution to interactive machine learning and human-AI collaboration, offering a concrete way for users to steer internal representations rather than only labels or data. The paper's strengths include a clear motivation, an open-source implementation, a thoughtful qualitative analysis of user strategies (cluster compactness, cluster distance, merging similar classes), and an explicit discussion of the risks of injecting human bias. The qualitative HCI findings and the proposed loss-insertion paradigm are interesting. However, the quantitative evidence for the headline claims—improved performance and maintained generalization—is not established by the evaluation as presented, and the mechanism itself is not fully specified, so the paper's main contribution cannot currently be verified.","major_comments":[{"comment":"The central mechanism of the paper, the loss function L_human that encodes human guidance, is never written down as an explicit equation. The text describes center alignment, spread, separation, normalization over pairwise comparisons K, and the parameters α and λ, but no concrete formula for L_human or for the scale term |1.0 − scale_model| is provided. Without this definition, the claimed knowledge-distillation-inspired loss cannot be reproduced, checked, or independently evaluated, which undermines the paper's central technical contribution.","section":"Section 3.2, Figure 2"},{"comment":"The claim that HILL improves model performance 'while maintaining generalization' is not supported by the evaluation design. Section 4.1 states that participants see the updated model performance after each training iteration and use it to decide their interactions, and Section 4.2.1 explicitly says 'The evaluation was conducted on the unseen validation dataset, the same as during training.' The validation split is therefore used both as the participants' interactive feedback signal and as the reported evaluation metric. The reported accuracy gains (CIFAR-10 +1.6 pp to 87.3%, PAMAP2 +2.2 pp to 75%) may simply reflect participants tuning to that split rather than human intuition transferring to genuinely unseen data. A held-out test set, never accessed during interaction, is required to support the generalization claim; none is reported.","section":"Sections 4.1 and 4.2.1"},{"comment":"The statement that the best-performing HILL runs 'significantly surpassed' the baseline is not backed by any inferential statistics. With only 14 participants, no error bars or confidence intervals in Figure 5, and only best/worst participant envelopes shown, the reported improvements cannot be distinguished from noise or from random variation across training runs. The authors should provide per-participant results, significance tests or effect sizes, and ideally repeated baseline runs, before claiming statistical or practical significance.","section":"Section 4.2.1"},{"comment":"The claim of faster convergence is presented without quantitative support. No learning curves beyond the validation accuracy of Figure 5 are shown, and no metric such as epochs-to-target-accuracy or computational cost is reported. As the convergence claim is listed as a key outcome, it needs a concrete operationalization and comparison against baseline.","section":"Section 4.2.1"}],"minor_comments":[{"comment":"There are typographical errors: 'Inefficent' in Figure 1 should be 'Inefficient', and the title styling 'Human in the Latent Loop (HILL)' is rendered inconsistently with 'Human In the Latent Loop' in the body.","section":"Abstract and Figure 1"},{"comment":"The symbol K in 'normalized over the total number of pairwise comparisons K' is never defined in the text; please clarify how K is computed and how the normalization is applied.","section":"Section 3.2"},{"comment":"There is a typo: 'In order to to prevent excessive experiment durations' should read 'In order to prevent excessive experiment durations'.","section":"Section 4.1"},{"comment":"The caption mentions 'the light grey area' representing the range of participants, but the figure itself is not fully described in the text; please ensure the figure legend and caption are self-contained.","section":"Figure 5"},{"comment":"Question Q2 has a grammatical error: 'The system distracting me' should be 'The system distracted me'.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The paper has a likeable interactive system and useful qualitative findings, but the central quantitative claims are undercut by the use of the validation set as both the human feedback signal and the evaluation metric, and by the absence of any explicit loss function specification. These are not minor presentation issues; they are load-bearing for the abstract's 'maintaining generalization' claim. A revision could fix the missing L_human formula and add inferential statistics, but the fundamental evaluation-design flaw would require either a new held-out test evaluation (not possible retroactively with the current interaction data, since participants already saw that split) or a reframing of the claims to 'improved validation accuracy during interactive training' rather than generalization. Given the scope of the empirical claims, I cannot recommend acceptance or minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is genuinely new: let users drag points and clusters in a frozen 2D projection of the latent space during training, then feed those edits into the model through an auxiliary loss inspired by knowledge distillation. Prior tools like SpaceEditing are post hoc or offline; HILL works inside the training loop. That is a real gap, and the deterministic projection choice is sensible. The qualitative HCI work is the strongest part. The think-aloud analysis surfaces distinct user strategies (increasing cluster compactness, maximizing distance, merging similar classes) that anyone building interactive ML will find useful. The tool is open-source, and the usability and workload metrics are reported in a straightforward way. But the central mechanism is not actually specified. Section 3.2 says user edits are \"encoded within L_human\" and mentions center alignment, spread, separation, and a scale term with α and λ, but no equation. For a loss-function contribution, this is a major omission. I could not check the mechanism or reproduce the training loop from the text. The generalization claim is also not supported by the reported evaluation. Section 4.1 says participants see \"the updated model performance\" after each training iteration, which is validation accuracy. Section 4.2.1 then reports final accuracy on \"the unseen validation dataset, the same as during training.\" So participants were effectively tuning to the validation split, and that same split is the reported metric. With n=14, no significance tests, and only best/worst participant ranges shown, the accuracy gains are plausibly explained by validation-set optimization rather than human intuition transferring to unseen data. The paper never reports a held-out test set. These are load-bearing issues, not polish items. The qualitative findings and the tool are worth keeping, but the quantitative claim needs an explicit loss equation, an evaluation on a test set participants never see, and at least basic inferential statistics or error bars. I would not desk-reject this. The combination is novel enough and the qualitative work is solid enough that serious referees should see it. But if I were the editor, I would expect major revision and would be skeptical of the current accuracy numbers.","headline":"Fresh combination of online latent-space editing with a KD-style guidance loss, but the loss is never specified and the only quantitative evidence comes from a validation set participants were actively tuning against.","tokens_in":12282,"tokens_out":1910,"would_cite":false,"duration_ms":20919,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human-guided edits to a model's latent space, injected as a knowledge-distillation-style loss, improve accuracy and convergence without sacrificing generalization.","keywords":["latent space","interactive learning","human intuition","knowledge distillation","human-in-the-loop","training-time guidance","user study"],"falsifier":"Run the same user study with two disjoint splits: a visible validation set that participants interact with and receive accuracy feedback on, and a locked test set evaluated only once at the end. If HILL models continue to beat baseline on the locked set by the same margin, the generalization claim holds; if the advantage shrinks or disappears, the reported gains are at least partly validation-set tuning.","tokens_in":11314,"feed_emoji":"🖱️","tokens_out":7585,"duration_ms":68511,"temperature":0.7,"pith_summary":"HILL is an interactive training loop that lets a person pause a deep model mid-training, look at a two-dimensional projection of its latent space, and drag data points or entire class clusters to reshape it. The paper claims these human edits, folded into training as an extra loss term rather than as altered data, make the model converge faster and reach higher validation accuracy. In a user study with 14 participants on CIFAR-10 and PAMAP2, the best guided runs reached 87.3% and about 75% accuracy respectively, ahead of passive baseline training, while the authors report generalization was maintained. The mechanism treats the human as a knowledge-distillation teacher whose geometric preferences — class center position, cluster compactness, and inter-class separation — softly steer the representation instead of overriding it.","feed_headline":"Drag-and-drop edits to a model's inner map lift accuracy","feed_subtitle":"Users dragged latent clusters during training and beat passive baselines on two benchmarks without losing generalization","key_machinery":"The load-bearing object is the human-as-teacher loss $\\mathcal{L}_{\\text{human}}$, a geometric loss computed from class center movement, class spread, and cluster separation between pairs of classes, normalized by the number of pairwise comparisons $K$. It is added to cross-entropy as $\\mathcal{L}_{\\text{global}} = \\mathcal{L}_{CE} + \\alpha \\mathcal{L}_{\\text{human}} + \\lambda |1 - \\text{scale}_{\\text{model}}|$, with the scale term keeping feature magnitudes stable. A frozen deterministic fully-connected projection gives users a stable 2D view of the latent space, so a drag gesture maps directly onto the loss targets. Together these pieces turn human intuition into a soft gradient signal that guides, but does not replace, the data-driven objective.","core_discovery":"The central discovery is that a human can act as a teacher for a deep network by editing its latent geometry during training, and that this improves both optimization and final accuracy. The user's drag operations are translated into three measurable targets — center movement, spread, and separation — and combined with cross-entropy in a global loss, with $\\alpha=0.5$ balancing human guidance against classification and $\\lambda=0.1$ regularizing feature scale. Because the projection to 2D is deterministic and frozen after the first epoch, the edits correspond to real changes in the representation, not to visualization artifacts. Across both datasets the first intervention produced the largest jump, and even the weakest participant run outperformed the baseline; the authors attribute this to the human providing structure that pure data-driven optimization misses.","pith_inferences":["Because participants saw validation accuracy after every interaction and used it to choose edits, the reported gains may partly reflect human search over that specific validation set; a locked hold-out set, shown only at the end, would distinguish genuine generalization from validation-set tuning.","The same geometric loss could be generated automatically, for example from a pretrained teacher's cluster structure, making the benefit testable without a user study and turning 'human intuition' into a tunable prior.","Human strategies such as merging semantically related classes suggest HILL can inject world knowledge absent from the data; a cross-dataset transfer test would show whether that knowledge generalizes or overfits to the dataset's label structure.","The projection is frozen after epoch one, so as the model's features drift the visualization may become stale; an adaptive projection or a freshness warning could change which interventions later interactions produce."],"forward_implications":["Users can steer a deep model's internal representation without changing the dataset, architecture, or inference procedure.","Even inefficient or inconsistent human strategies, as in the worst participant run, still beat passive training, suggesting the guidance term is robust to noisy feedback.","The largest accuracy gain comes from the first interaction, so a single early structuring of the latent space may capture most of the benefit.","Faster convergence means a target accuracy can be reached with fewer training epochs, reducing compute cost.","Because human edits enter only through a weighted loss, the model can partially resist biased guidance; but if a user switches strategy mid-training, accuracy can fluctuate, as in the worst PAMAP2 run."],"supporting_citations":[{"why":"Supplies the knowledge-distillation teacher-student setup that HILL adapts, with the human as teacher.","marker":"[17]"},{"why":"Provides the CIFAR-10 image dataset used as one of the two evaluation scenarios.","marker":"[23]"},{"why":"Provides the PAMAP2 wearable-sensor activity dataset used as the second evaluation scenario.","marker":"[39]"},{"why":"Exemplifies the non-deterministic dimension reduction the authors reject in favor of a frozen deterministic projection.","marker":"[48]"},{"why":"Exemplifies another non-deterministic dimension reduction approach the authors avoid for the same reason.","marker":"[32]"}],"fun_headline_variants":["Humans steer neural nets by reshaping latent space during training","Drag latent clusters to teach a model and lift its accuracy","Interactive latent edits outperform passive baselines in model training","Human intuition guides model training via latent space editing","Edit a model's inner map on the fly to boost learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Participants see validation accuracy after every edit and use it to decide their next move, and that same validation set is then reported as the evaluation metric; the claim that HILL preserves generalization assumes this set faithfully represents how the model will behave on truly unseen data.","fun_headline_variants_meta":{"raw":{"variants":["Humans steer neural nets by reshaping latent space during training","Drag latent clusters to teach a model and lift its accuracy","Interactive latent edits outperform passive baselines in model training","Human intuition guides model training via latent space editing","Edit a model's inner map on the fly to boost learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000774,"raw_usage":{"total_tokens":3400,"prompt_tokens":893,"completion_tokens":2507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2428}},"tokens_in":509,"tokens_out":2507,"duration_ms":16620,"temperature":1.0,"reasoning_tokens":2428,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:55:05.368387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same user study with two disjoint splits: a visible validation set that participants interact with and receive accuracy feedback on, and a locked test set evaluated only once at the end. If HILL models continue to beat baseline on the locked set by the same margin, the generalization claim holds; if the advantage shrinks or disappears, the reported gains are at least partly validation-set tuning.","supporting_citations":[{"cited_title":"Introducing a new benchmarked dataset for activity monitoring","cited_arxiv_id":null,"evidence_quote":"Provides the PAMAP2 wearable-sensor activity dataset used as the second evaluation scenario."}],"review_version":1}