{"id":"ca871e36-9af7-4c8b-b778-fcc266e194e7","arxiv_id":"2607.00969","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"A literature survey of VIS4ML papers develops a coding scheme to identify pathways for human knowledge transfer into ML workflows via interactive visualization and explains them using conceptual models of VA as model building and cost-benefit optimization.","lead":"This paper surveys over 200 VIS4ML papers from IEEE VIS conferences to map how humans use interactive visualizations to inject knowledge into ML workflows such as data labeling and model tuning. A smart generalist might read it to understand practical pathways for combining human expertise with automated ML processes.","discovery_kind":"unclear","skeptic_critique":{"model":"grok-4.3","headline":"Survey of published VIS papers cannot supply 'unequivocal evidence' of VA merits","rationale":"The reader's weakest_assumption directly identifies the same point (corpus + coding scheme representativeness). Because the full text was not supplied in the initial query, the reader correctly withheld a verdict; the survey design itself supplies no additional empirical warrant that would change that stance.","tokens_in":1711,"tokens_out":302,"duration_ms":10988,"concrete_test":"Re-annotate a random 20% subset of the corpus with two independent coders using the published scheme; compute Cohen's kappa on the 'actions' and 'interaction' axes. If kappa < 0.6 on either axis, the derived pathways and subsequent conceptual-model explanation rest on unstable labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim ('provides unequivocal evidence showing the merits of using VA in ML workflows') is supported only by (1) collecting IEEE VIS papers 2013-2022, (2) applying an unvalidated four-axis coding scheme, and (3) qualitatively observing 'pathways' that are then interpreted through two conceptual models. No quantitative outcome data, controlled comparison, or external validation of the coding scheme appears; the corpus is restricted to one venue and therefore inherits publication bias. The weakest link is therefore the unsupported leap from 'we observed these patterns in the literature' to 'this constitutes unequivocal evidence of merit.'","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The manuscript surveys over 200 VIS4ML papers from IEEE VIS conferences (2013–2022), develops a four-axis coding scheme covering ML characteristics, visualization, interaction, and actions, identifies pathways by which humans inject knowledge into ML workflows via interactive visualization, and interprets the results through conceptual models of VA as model building and information-theoretic cost-benefit analysis. It concludes that the survey supplies unequivocal evidence of the merits of VA in ML workflows, with the coded corpus and results released online.","tokens_in":1835,"tokens_out":521,"duration_ms":27254,"significance":"If the coding scheme is shown to be reliable, the work would supply a useful taxonomy of human-in-the-loop patterns in visualization-supported ML, helping to organize an emerging subfield. The public release of the full coded dataset and analysis artifacts is a clear strength that supports reproducibility and secondary use. The significance is nevertheless tempered by the observational character of the study and the absence of validation metrics for the central methodological choices.","major_comments":[{"comment":"Abstract and §6: The assertion that the work 'provides unequivocal evidence showing the merits of using VA in ML workflows' is not warranted by the reported methodology. The study performs qualitative coding of a venue-restricted corpus without reported validation or quantitative outcome measures; this observational design cannot supply unequivocal evidence and the claim should be moderated or supported by additional validation.","section":"Abstract and §6"},{"comment":"§3 (Corpus construction): Restricting the corpus to IEEE VIS papers from a single decade introduces venue and publication bias that is not discussed. The paper should explain why other venues publishing VIS4ML work were excluded and how this choice affects the representativeness of the observed pathways.","section":"§3"},{"comment":"§4 (Coding scheme): No information is given on inter-rater reliability, pilot testing, or external validation of the four-axis coding scheme. Because the pathways, conceptual models, and final claims rest directly on the coded data, the lack of these standard methodological safeguards is load-bearing.","section":"§4"}],"minor_comments":[{"comment":"The supplementary website is referenced but the main text would benefit from a compact summary table reporting basic corpus statistics (e.g., paper counts per year and per coding axis) to allow readers to assess coverage without leaving the PDF.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their detailed and constructive comments. We address each major point below, indicating revisions where appropriate.","responses":[{"response":"We agree that 'unequivocal evidence' overstates what an observational qualitative survey can demonstrate. We will revise the abstract and §6 to moderate the language (e.g., 'provides evidence illustrating the merits' or 'offers insights into the role of VA'), while adding an explicit acknowledgment of the study's observational and qualitative character and the absence of quantitative outcome metrics.","revision_made":"yes","referee_comment":"[Abstract and §6] Abstract and §6: The assertion that the work 'provides unequivocal evidence showing the merits of using VA in ML workflows' is not warranted by the reported methodology. The study performs qualitative coding of a venue-restricted corpus without reported validation or quantitative outcome measures; this observational design cannot supply unequivocal evidence and the claim should be moderated or supported by additional validation."},{"response":"We will expand §3 with a new paragraph explaining the rationale: IEEE VIS is the flagship venue for visualization research, ensuring high-quality, peer-reviewed VIS4ML contributions within a consistent community. We will also acknowledge the resulting venue bias and limited representativeness, noting that VIS4ML work appears in other venues (e.g., EuroVis, PacificVis) and that expanding the corpus in future work would improve generalizability of the observed pathways.","revision_made":"yes","referee_comment":"[§3] §3 (Corpus construction): Restricting the corpus to IEEE VIS papers from a single decade introduces venue and publication bias that is not discussed. The paper should explain why other venues publishing VIS4ML work were excluded and how this choice affects the representativeness of the observed pathways."},{"response":"We will revise §4 to describe the coding process in greater detail: the scheme was developed iteratively by the author team through discussion and refinement; a pilot phase was conducted on a subset of papers to test and adjust categories; and disagreements were resolved via consensus meetings. We will note that formal inter-rater reliability statistics (e.g., Cohen's kappa) were not computed and acknowledge this as a methodological limitation, while arguing that the consensus-based approach provided sufficient consistency for the exploratory goals of the survey.","revision_made":"partial","referee_comment":"[§4] §4 (Coding scheme): No information is given on inter-rater reliability, pilot testing, or external validation of the four-axis coding scheme. Because the pathways, conceptual models, and final claims rest directly on the coded data, the lack of these standard methodological safeguards is load-bearing."}],"tokens_in":1460,"tokens_out":571,"duration_ms":18601,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"This paper surveys more than 200 VIS4ML papers from IEEE VIS over the past decade. The authors built a coding scheme covering ML characteristics, visualization, interaction, and actions, then used it to identify pathways for injecting human knowledge into ML workflows through visual analytics. They also link the observations to two conceptual models: VA as model building and an information-theoretic cost-benefit view.\n\nWhat stands out is the effort to systematize existing work and release the full coded dataset and figures online. That makes it a practical reference for anyone designing or studying visual tools for ML.\n\nThe main limitation is that the headline claim of providing 'unequivocal evidence' of VA merits goes beyond what the method supports. This is a qualitative synthesis of published papers, not a controlled study or validated measurement. The abstract gives no numbers on inter-rater agreement for the coding or tests of the scheme's reliability. Limiting the corpus to one conference series also means it reflects publication patterns in that venue rather than the broader field. The leap from observed patterns to evidence of merit is interpretive rather than demonstrated.\n\nThe weakest part is the assumption that the selected papers and coding scheme fully represent how humans actually inject knowledge. It might capture what has been written about, but not necessarily the complete picture or the effectiveness.\n\nThis work is aimed at researchers in visualization and human-computer interaction who focus on ML workflows. A reader looking for a structured overview of the literature will get value from the pathways and the public data. It is not a new technique or empirical result, so its contribution is in the synthesis.\n\nI think it deserves a serious referee. The survey is substantive enough to warrant review, though the authors should be asked to adjust the strength of their conclusions to match the observational nature of the work.","headline":"A structured survey of VIS4ML papers that maps knowledge-injection pathways but overstates its results as 'unequivocal evidence.'","tokens_in":2334,"tokens_out":433,"would_cite":true,"duration_ms":19158,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"A survey of over 200 visualization papers identifies specific pathways for humans to inject knowledge into machine learning workflows via interactive tools.","keywords":["visual analytics","machine learning workflows","human knowledge injection","VIS4ML","interactive visualization","literature survey","coding scheme","knowledge transfer pathways"],"falsifier":"A search of additional papers outside the IEEE VIS corpus that documents common human knowledge injection methods into ML workflows not covered by the coding scheme would undermine the completeness of the identified pathways.","tokens_in":2623,"feed_emoji":"📊","tokens_out":613,"duration_ms":23034,"temperature":0.7,"pith_summary":"The paper surveys more than 200 VIS4ML papers from IEEE VIS conferences over the past decade to map how humans add knowledge to machine learning through visual analytics. It builds a coding scheme across machine learning traits, visualization, interaction, and actions to trace distinct pathways of knowledge transfer. The results are explained through a model that treats visual analytics as model building and through cost-benefit reasoning that positions it as a way to optimize machine learning workflows. A reader would care because the work shows concrete ways visualization makes machine learning more responsive to human insight rather than fully automatic.","feed_headline":"Survey maps pathways for humans to steer ML with visualization","feed_subtitle":"Analysis of 200+ VIS papers shows how interactive tools let people add knowledge at multiple stages of machine learning.","key_machinery":"The coding scheme applied to four perspectives (characteristics of ML, visualization, interaction, and actions) that traces pathways of human knowledge transfer into ML workflows.","core_discovery":"By coding the collected papers from four perspectives, the analysis reveals different pathways that transfer human knowledge to ML workflows via interactive visualization. These pathways are explained using the conceptual model that views VA as model building and the information-theoretic cost-benefit analysis that reasons VA as for optimizing ML workflows, providing evidence of the merits of using VA in ML workflows.","pith_inferences":["Tool builders could design new visual analytics interfaces that explicitly support the identified pathways at different stages of machine learning.","The coding scheme could be tested on papers from additional conferences to check whether the same pathways appear consistently.","Measuring actual workflow efficiency gains when using these pathways in practice would provide a direct test of the cost-benefit reasoning."],"forward_implications":["Distinct pathways exist for transferring human knowledge to ML workflows through interactive visualization.","Visual analytics functions as model building in these workflows.","Information-theoretic cost-benefit analysis positions visual analytics as a means to optimize ML workflows.","The survey supplies evidence for the merits of visual analytics within machine learning workflows."],"fun_headline_variants":["Survey charts pathways injecting human knowledge into ML via VA","Coding 200 papers maps knowledge transfer in ML workflows","Visual analytics enables human inputs across ML pipeline stages","Pathways reveal VA as model building for ML optimization","Analysis shows human knowledge injection via interactive visuals"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The corpus of IEEE VIS papers from the past decade together with the developed coding scheme sufficiently captures the representative ways humans inject knowledge into ML workflows through interactive visualization.","fun_headline_variants_meta":{"raw":{"variants":["Survey charts pathways injecting human knowledge into ML via VA","Coding 200 papers maps knowledge transfer in ML workflows","Visual analytics enables human inputs across ML pipeline stages","Pathways reveal VA as model building for ML optimization","Analysis shows human knowledge injection via interactive visuals"]},"model":"grok-4.3","cost_usd":0.004679,"raw_usage":{"total_tokens":2316,"prompt_tokens":674,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":46787000,"prompt_tokens_details":{"text_tokens":674,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1571,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":674,"tokens_out":71,"duration_ms":10880,"temperature":1.0,"reasoning_tokens":1571,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T06:14:02.432091+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A search of additional papers outside the IEEE VIS corpus that documents common human knowledge injection methods into ML workflows not covered by the coding scheme would undermine the completeness of the identified pathways.","supporting_citations":[],"review_version":1}