{"id":"0b327263-d388-428c-85ca-f53b885d6f1a","arxiv_id":"2412.09152","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"herakoi is an open-source real-time image sonification tool using webcam hand tracking to map pixel colors and brightness to musical notes.","lead":"A team of astronomers presents herakoi, an open-source tool that turns hand gestures captured by a webcam into sound so people can \"hear\" astronomical images. The paper describes the tool and its use in education and outreach, but the evidence for its effectiveness is largely drawn from a separate, previously published educational program.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Effectiveness claim rests on edukoi data, not on herakoi itself; direct evaluation of herakoi is missing.","rationale":"The paper is a software description, and as such it has real strengths: herakoi is open source, the color-to-pitch and brightness-to-amplitude mapping is explicit, and the tool has been used in outreach. The concern is not that the software is nonfunctional; it is that the paper's headline claim is a statement of demonstrated effectiveness, while the only quantitative evidence cited belongs to a different tool. The reader identified the same weakest assumption, and I agree with that assessment. The transfer from edukoi to herakoi is load-bearing because, without it, the abstract's 'high reliability and effectiveness' has no empirical support in the manuscript. A direct evaluation of herakoi would settle the issue. The conditional verdict is appropriate: the paper should be accepted as a software/outreach announcement only if the effectiveness claim is either backed by herakoi-specific data or explicitly softened.","tokens_in":5500,"tokens_out":3579,"duration_ms":35633,"concrete_test":"Run the edukoi evaluation protocol (Guiotto Nai Fovino et al. 2024a) directly on herakoi with a comparable sample (e.g., 30–50 middle-school students, including BVI participants where feasible): participants use only herakoi to identify colors and shapes in the same astronomical images, with accuracy and time-to-first-successful-identification recorded, alongside webcam-to-sound latency. If herakoi-only accuracy falls below the 86–93% range or the training time is not short by a prespecified criterion (e.g., ≤15 minutes), the abstract's effectiveness claim should be softened to 'promising in outreach settings' pending direct evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central effectiveness claim in the abstract—'swift responsiveness allows users to access information in astronomical images with short training, demonstrating high reliability and effectiveness'—has no direct supporting data in this manuscript for herakoi itself. Section 3 reports the only quantitative results in the paper, but they concern edukoi, a separately developed educational spin-off, not herakoi: color-recognition accuracy of 86% in initial trials and 93% in follow-up tests. The manuscript then treats these results as evidence for herakoi ('herakoi and edukoi demonstrate the potential...') and the abstract attributes the demonstrated effectiveness to herakoi. Footnote 10 explicitly concedes that research applications of herakoi have not been investigated systematically. The load-bearing inference is therefore that edukoi's positive school-test outcomes transfer to herakoi's real-time hand-tracking interface. That inference is unexamined: herakoi and edukoi differ in features and target users; no latency or training-time measurement for herakoi is reported; and no data specifically on BVI users is given, despite the accessibility claim in the abstract.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents herakoi, an open-source, real-time image sonification tool for astronomical data. The authors describe its architecture: a MediaPipe-based hand tracking model detects hand landmarks from a webcam feed, maps them onto image pixel coordinates, and converts the visual properties of the \"touched\" pixels, mainly hue and brightness, into MIDI sound parameters. Section 3 reports prior results from edukoi, an educational spin-off, where middle-school students achieved 86% and 93% color-recognition accuracy, and Section 4 outlines future integration of large language and vision models. The abstract claims that herakoi's \"swift responsiveness\" allows \"access information in astronomical images with short training, demonstrating high reliability and effectiveness.\"","tokens_in":5663,"tokens_out":3656,"duration_ms":37071,"significance":"The software is a practical and openly released contribution to astronomical data accessibility, and the prior edukoi study with roughly 150 students provides some empirical support for the underlying sonification approach. The paper also usefully documents a real-time, gesture-controlled interface that could benefit outreach and inclusive education. However, the central claim of demonstrated effectiveness for herakoi itself is not supported by direct measurements in this manuscript: the only quantitative evidence concerns edukoi, and the manuscript explicitly concedes that systematic research applications of herakoi have not yet been investigated. If the claims are appropriately qualified, the paper is a valuable software description, but in its current form the abstract and conclusions overstate the evidence.","major_comments":[{"comment":"The abstract states that herakoi's \"swift responsiveness\" and \"short training\" demonstrate \"high reliability and effectiveness,\" but the only quantitative results in the manuscript are the edukoi color-recognition accuracies of 86% and 93% reported in Section 3. These results were obtained with edukoi, a separately developed educational spin-off, not with herakoi itself. Footnote 10 explicitly states that research applications of herakoi have not been systematically investigated. The manuscript should either report direct measurements of herakoi's latency, training time, and user accuracy, or rephrase the abstract and conclusions to attribute the demonstrated effectiveness to edukoi and describe herakoi's performance as preliminary and anecdotal.","section":"Abstract and Section 3"},{"comment":"The claim of \"swift responsiveness\" is not operationalized anywhere in the paper. No latency, frame-rate, or interaction-timing measurements are reported for herakoi, despite the fact that real-time hand tracking and immediate auditory feedback are central to the tool's stated value. Without a quantitative or even semi-quantitative description of responsiveness, the abstract's claim is unsupported. At minimum, the authors should report the typical end-to-end latency or frame rate observed in their demonstrations, or soften the claim to describe responsiveness qualitatively.","section":"Section 2 and Abstract"},{"comment":"The paper repeatedly claims that herakoi has proven effective for BVI users and in educational settings, but the evidence presented is anecdotal. Section 3 mentions outreach events and states that the tool \"resonated well\" with BVI users, but no structured observations, interviews, or performance data involving BVI participants are provided. Similarly, Section 4 states that herakoi \"has proven to be an effective educational tool in schools,\" even though Section 3 attributes the school tests to edukoi. These claims need to be clearly separated: edukoi has empirical school-test data; herakoi has only anecdotal outreach reports.","section":"Section 3 and Section 4"}],"minor_comments":[{"comment":"The choice to truncate the hue scale at 80% of its full range is presented without justification; the authors should provide a rationale or reference for this parameter, since it affects the default image-to-sound mapping.","section":"Section 2"},{"comment":"The reported edukoi accuracy figures (86% and 93%) are not accompanied by sample sizes, confidence intervals, or statistical tests; the statement that shape recognition was \"significantly above random guessing\" would be much stronger with the corresponding p-values or effect sizes.","section":"Section 3"},{"comment":"Footnote 10 is important because it directly qualifies the abstract's effectiveness claim, and it would be more transparently placed in the main text near the discussion of herakoi's applications. Footnote 9 would also benefit from a more precise definition of \"color\" in the sonification context.","section":"Footnotes 9 and 10"},{"comment":"The manuscript would benefit from a brief section or table listing the system requirements and setup steps for herakoi, including the webcam resolution and processing hardware used in the reported demonstrations, to help readers reproduce the claimed real-time behavior.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"This is a short proceedings-style contribution that fits the journal's scope as a software and methods presentation. The main issue is not the tool itself but the mismatch between the evidence presented and the strength of the claims in the abstract and conclusions. I recommend major revision primarily to bring the claims in line with the evidence; if the authors prefer not to add new evaluation data, they should temper the abstract and conclusions accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: herakoi is a real, open-source sonification tool, and the paper gives a clear account of how it works. But the abstract's \"high reliability and effectiveness\" claim is not supported by anything in the manuscript that actually measures herakoi. The only quantitative results come from edukoi, a differently scoped spin-off, and footnote 10 concedes that systematic research on herakoi itself hasn't been done.\n\nWhat's genuinely new: the combination of MediaPipe hand tracking with real-time image sonification, mapped to MIDI output. That is a useful integration, and the description of the two interaction modes—index-finger square, and thumb-index rectangle for dynamic zoom—is concrete and sensible. The paper also does the field a service by pointing to the GitHub repository and by documenting its outreach and education use. The default color-to-pitch mapping is standard in the field, so the novelty is in the interaction, not the sonification model.\n\nSoft spots: the load-bearing effectiveness claim. Section 3 reports edukoi's 86% and 93% color-recognition accuracy and then attributes that promise to herakoi. That transfer is unexamined. No latency numbers, no training-time measurement, no direct test with BVI users, even though the accessibility claim is prominent in the abstract. The hue-scale truncation at 80% is a design choice; I would have liked one sentence on why 80%, but that is minor.\n\nProportionally: this is a software/outreach paper, not a validation study. The tool is plausibly functional, and the code is open for others to check. The fix is straightforward—either soften the abstract to \"promising\" with anecdotal evidence, or run a small user study on herakoi itself. Given the educational and accessibility context, a direct test would be worth the effort.\n\nMy take: it deserves a serious referee, not a desk reject. A good referee would push for the claims to match the evidence. I would cite it if I were writing about astronomical sonification tools, mostly for the open-source implementation and the interaction design.","headline":"A genuinely useful open-source sonification tool whose abstract overclaims effectiveness on the basis of edukoi data, not herakoi itself.","tokens_in":6194,"tokens_out":1945,"would_cite":true,"duration_ms":18486,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"herakoi turns astronomical images into real-time sound by tracking hand motion, and the authors argue this lets users access image information after short training.","keywords":["data sonification","astronomical imaging","real-time interaction","hand tracking","accessibility","blind and visually impaired","educational outreach","open-source software"],"falsifier":"A controlled study would give naive participants only the sound output of herakoi while they explore synthetic astronomical images with known features, then ask them to identify properties such as a bright source location or a color-coded region; if identification accuracy stays near chance despite a short training session, the claim of reliable information access through herakoi's real-time sonification would be falsified.","tokens_in":5288,"feed_emoji":"🎵","tokens_out":7327,"duration_ms":74149,"temperature":0.7,"pith_summary":"herakoi is an open-source sonification tool that renders astronomical images audible in real time. A webcam tracks the user's hand, re-projects the hand landmarks onto image pixel coordinates, and converts the visual properties of the touched pixels into MIDI sound: hue becomes pitch and brightness becomes loudness. The paper claims this direct, physical mapping is intuitive enough that users can extract information from astronomical images with short training, making data exploration possible for sighted as well as blind and visually impaired users. The quantitative evidence reported for effectiveness comes from a spin-off educational version, edukoi, tested with schoolchildren, who reached color-recognition accuracies of 86 percent in initial trials and 93 percent in follow-up tests. If the claim holds, herakoi gives astronomy a practical multi-sensory interface and a path toward more inclusive education and outreach.","feed_headline":"Webcam hand-tracking turns astronomical images into sound","feed_subtitle":"Open-source herakoi maps pixel color to pitch and brightness to volume in real time, opening astronomy to all.","key_machinery":"The carrying mechanism is the closed loop of webcam hand tracking, coordinate projection, and MIDI synthesis. A hand-landmark model provides real-time tracking of twenty-one hand keypoints, and the index finger, or the index-finger-to-thumb rectangle, defines the touched pixel region. The image-to-sound mapping uses the hue-saturation-brightness color model, truncating hue at 80 percent of its range to avoid its cyclic wrap, then maps hue to MIDI pitch, with red as low pitch and blue as high pitch, and maps brightness to amplitude. Standard MIDI messages are sent to any virtual or physical instrument, which makes the interaction both customizable and hardware-independent.","core_discovery":"herakoi's central claim is that a real-time image sonification loop can carry useful astronomical information by sound alone. A pretrained machine-learning hand-landmark model detects keypoints on the user's hand from a webcam stream; the detected coordinates are re-projected onto the pixel grid of a selected image; and the color, saturation, and brightness of the touched region are converted into pitch and amplitude following the common astronomical color-to-pitch convention. The authors argue that this loop is responsive and the mapping natural enough for users to access image content after short training. The paper's quantitative evidence is indirect: numerical results come from edukoi, the educational spin-off tested with roughly 150 schoolchildren, while herakoi itself is described through qualitative outreach use and is not the subject of systematic measurement in this paper.","pith_inferences":["A fair reading of the evidence suggests the 86 and 93 percent accuracies belong to edukoi, not to herakoi itself; direct user testing of herakoi would be needed before treating real-time reliability as fully demonstrated.","The same hand-tracking scaffold could be lifted to non-image data such as spectra, light curves, or two-dimensional maps, since the pipeline only requires a 2D coordinate grid and a color-or-intensity to sound mapping.","A testable extension would isolate the value of real-time interaction itself by comparing herakoi-style live sonification with static sonification recordings for the same images; the paper does not currently separate these variables.","If paired with objective performance metrics, her a koi could become a research instrument for comparing sonification mappings, not only an accessibility and outreach tool."],"forward_implications":["Astronomical images become explorable by ear in real time, allowing colors and brightness contrasts to be perceived without looking at a screen.","Blind and visually impaired users gain a concrete route into astronomical imagery, extending prior work on sonification as an accessibility aid.","In educational settings, the tool can support color and shape recognition: the related edukoi tests with roughly 150 schoolchildren reported 86 percent color recognition initially and 93 percent after follow-up.","Because herakoi emits standard MIDI messages, any MIDI synthesizer can render its output, so the sonification interface works with widely available hardware.","The authors propose combining herakoi with language and vision models plus voice synthesis so users could ask questions and receive verbal descriptions while exploring images.","A quantitative pipeline from image coordinates to sound properties is already present, so direct research use of herakoi is a plausible next step rather than a conceptual leap."],"supporting_citations":[{"why":"Supplies the hand-landmark model that detects hand keypoints from webcam frames in real time.","marker":"Lugaresi et al. 2019"},{"why":"Reports the school experiments with about 150 students that yield the 86 percent and 93 percent color-recognition accuracies cited as evidence of effectiveness.","marker":"Guiotto Nai Fovino et al. 2024a"},{"why":"Defines the color-to-pitch convention that herakoi adopts as its default image-to-sound mapping.","marker":"Zanella et al. 2022"},{"why":"Establishes data sonification as a complementary approach to visual analysis and motivates the need for such tools in astronomy.","marker":"Harrison et al. 2022"},{"why":"Underpins the accessibility argument by connecting sonification to data access for blind and visually impaired users.","marker":"Tucker Brown et al. 2022"},{"why":"Identifies the software release that this paper presents and describes.","marker":"Di Mascolo & Ginolfi 2023"}],"fun_headline_variants":["Open-source herakoi turns webcam hand motion into astronomy sounds","Hear the stars: herakoi sonifies astronomical data via hand tracking","Real-time image sonification with herakoi lets you hear astronomy","Webcam hand-tracking converts image color and brightness into sound","Astronomy audibly: herakoi makes space images accessible via sound"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The effectiveness claim rests on positive results from edukoi, a spin-off educational version, being transferred to herakoi itself, whose real-use performance has not been directly tested.","fun_headline_variants_meta":{"raw":{"variants":["Open-source herakoi turns webcam hand motion into astronomy sounds","Hear the stars: herakoi sonifies astronomical data via hand tracking","Real-time image sonification with herakoi lets you hear astronomy","Webcam hand-tracking converts image color and brightness into sound","Astronomy audibly: herakoi makes space images accessible via sound"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001057,"raw_usage":{"total_tokens":4382,"prompt_tokens":835,"completion_tokens":3547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":451,"completion_tokens_details":{"reasoning_tokens":3454}},"tokens_in":451,"tokens_out":3547,"duration_ms":23181,"temperature":1.0,"reasoning_tokens":3454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:13:44.081089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A controlled study would give naive participants only the sound output of herakoi while they explore synthetic astronomical images with known features, then ask them to identify properties such as a bright source location or a color-coded region; if identification accuracy stays near chance despite a short training session, the claim of reliable information access through herakoi's real-time sonification would be falsified.","supporting_citations":[{"cited_title":"M., Lenzi, S., et al","cited_arxiv_id":null,"evidence_quote":"Defines the color-to-pitch convention that herakoi adopts as its default image-to-sound mapping."}],"review_version":1}