REVIEW 3 major objections 4 minor 12 references
herakoi: a sonification experiment for astronomical data
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read herakoi turns astronomical images into real-time sound by tracking hand motion, and the authors argue this lets users access image information after short training.
desk verdict A genuinely useful open-source sonification tool whose abstract overclaims effectiveness on the basis of edukoi data, not herakoi itself. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the closed loop of webcam hand tracking, coordinate projection, and MIDI synthesis. A hand-landmark model provides real-time tracking of twenty-one hand keypoints, and the index finger, or the index-finger-to-thumb rectangle, defines the touched pixel region. The image-to-sound mapping uses the hue-saturation-brightness color model, truncating hue at 80 percent of its range to avoid its cyclic wrap, then maps hue to MIDI pitch, with red as low pitch and blue as high pitch, and maps brightness to amplitude. Standard MIDI messages are sent to any virtual or physical instrument, which makes the interaction both customizable and hardware-independent.
What would settle it
A controlled study would give naive participants only the sound output of herakoi while they explore synthetic astronomical images with known features, then ask them to identify properties such as a bright source location or a color-coded region; if identification accuracy stays near chance despite a short training session, the claim of reliable information access through herakoi's real-time sonification would be falsified.
Extended reading notes
Core claim
herakoi's central claim is that a real-time image sonification loop can carry useful astronomical information by sound alone. A pretrained machine-learning hand-landmark model detects keypoints on the user's hand from a webcam stream; the detected coordinates are re-projected onto the pixel grid of a selected image; and the color, saturation, and brightness of the touched region are converted into pitch and amplitude following the common astronomical color-to-pitch convention. The authors argue that this loop is responsive and the mapping natural enough for users to access image content after short training. The paper's quantitative evidence is indirect: numerical results come from edukoi, the educational spin-off tested with roughly 150 schoolchildren, while herakoi itself is described through qualitative outreach use and is not the subject of systematic measurement in this paper.
Load-bearing premise
The effectiveness claim rests on positive results from edukoi, a spin-off educational version, being transferred to herakoi itself, whose real-use performance has not been directly tested.
Editorial extensions
If this is right
- Astronomical images become explorable by ear in real time, allowing colors and brightness contrasts to be perceived without looking at a screen.
- Blind and visually impaired users gain a concrete route into astronomical imagery, extending prior work on sonification as an accessibility aid.
- In educational settings, the tool can support color and shape recognition: the related edukoi tests with roughly 150 schoolchildren reported 86 percent color recognition initially and 93 percent after follow-up.
- Because herakoi emits standard MIDI messages, any MIDI synthesizer can render its output, so the sonification interface works with widely available hardware.
- The authors propose combining herakoi with language and vision models plus voice synthesis so users could ask questions and receive verbal descriptions while exploring images.
- A quantitative pipeline from image coordinates to sound properties is already present, so direct research use of herakoi is a plausible next step rather than a conceptual leap.
Reading between the lines
- A fair reading of the evidence suggests the 86 and 93 percent accuracies belong to edukoi, not to herakoi itself; direct user testing of herakoi would be needed before treating real-time reliability as fully demonstrated.
- The same hand-tracking scaffold could be lifted to non-image data such as spectra, light curves, or two-dimensional maps, since the pipeline only requires a 2D coordinate grid and a color-or-intensity to sound mapping.
- A testable extension would isolate the value of real-time interaction itself by comparing herakoi-style live sonification with static sonification recordings for the same images; the paper does not currently separate these variables.
- If paired with objective performance metrics, her a koi could become a research instrument for comparing sonification mappings, not only an accessibility and outreach tool.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents herakoi, an open-source, real-time image sonification tool for astronomical data. The authors describe its architecture: a MediaPipe-based hand tracking model detects hand landmarks from a webcam feed, maps them onto image pixel coordinates, and converts the visual properties of the "touched" pixels, mainly hue and brightness, into MIDI sound parameters. Section 3 reports prior results from edukoi, an educational spin-off, where middle-school students achieved 86% and 93% color-recognition accuracy, and Section 4 outlines future integration of large language and vision models. The abstract claims that herakoi's "swift responsiveness" allows "access information in astronomical images with short training, demonstrating high reliability and effectiveness."
Significance. The software is a practical and openly released contribution to astronomical data accessibility, and the prior edukoi study with roughly 150 students provides some empirical support for the underlying sonification approach. The paper also usefully documents a real-time, gesture-controlled interface that could benefit outreach and inclusive education. However, the central claim of demonstrated effectiveness for herakoi itself is not supported by direct measurements in this manuscript: the only quantitative evidence concerns edukoi, and the manuscript explicitly concedes that systematic research applications of herakoi have not yet been investigated. If the claims are appropriately qualified, the paper is a valuable software description, but in its current form the abstract and conclusions overstate the evidence.
major comments (3)
- [Abstract and Section 3] The abstract states that herakoi's "swift responsiveness" and "short training" demonstrate "high reliability and effectiveness," but the only quantitative results in the manuscript are the edukoi color-recognition accuracies of 86% and 93% reported in Section 3. These results were obtained with edukoi, a separately developed educational spin-off, not with herakoi itself. Footnote 10 explicitly states that research applications of herakoi have not been systematically investigated. The manuscript should either report direct measurements of herakoi's latency, training time, and user accuracy, or rephrase the abstract and conclusions to attribute the demonstrated effectiveness to edukoi and describe herakoi's performance as preliminary and anecdotal.
- [Section 2 and Abstract] The claim of "swift responsiveness" is not operationalized anywhere in the paper. No latency, frame-rate, or interaction-timing measurements are reported for herakoi, despite the fact that real-time hand tracking and immediate auditory feedback are central to the tool's stated value. Without a quantitative or even semi-quantitative description of responsiveness, the abstract's claim is unsupported. At minimum, the authors should report the typical end-to-end latency or frame rate observed in their demonstrations, or soften the claim to describe responsiveness qualitatively.
- [Section 3 and Section 4] The paper repeatedly claims that herakoi has proven effective for BVI users and in educational settings, but the evidence presented is anecdotal. Section 3 mentions outreach events and states that the tool "resonated well" with BVI users, but no structured observations, interviews, or performance data involving BVI participants are provided. Similarly, Section 4 states that herakoi "has proven to be an effective educational tool in schools," even though Section 3 attributes the school tests to edukoi. These claims need to be clearly separated: edukoi has empirical school-test data; herakoi has only anecdotal outreach reports.
minor comments (4)
- [Section 2] The choice to truncate the hue scale at 80% of its full range is presented without justification; the authors should provide a rationale or reference for this parameter, since it affects the default image-to-sound mapping.
- [Section 3] The reported edukoi accuracy figures (86% and 93%) are not accompanied by sample sizes, confidence intervals, or statistical tests; the statement that shape recognition was "significantly above random guessing" would be much stronger with the corresponding p-values or effect sizes.
- [Footnotes 9 and 10] Footnote 10 is important because it directly qualifies the abstract's effectiveness claim, and it would be more transparently placed in the main text near the discussion of herakoi's applications. Footnote 9 would also benefit from a more precise definition of "color" in the sonification context.
- [General] The manuscript would benefit from a brief section or table listing the system requirements and setup steps for herakoi, including the webcam resolution and processing hardware used in the reported demonstrations, to help readers reproduce the claimed real-time behavior.
Circularity Check
No significant circularity: the paper is a software/tool presentation with no derivation chain; effectiveness claims rest on external empirical edukoi tests, which is an evidence gap, not circular reasoning.
full rationale
herakoi is a sonification tool paper; it contains no mathematical derivation, no fitted parameter subsequently renamed as a prediction, and no first-principles result whose output is equivalent to its input by construction. The main quantitative evidence cited in Section 3 (color-recognition accuracy of 86% and 93% in edukoi school tests) comes from Guiotto Nai Fovino et al. (2024a), an empirical study with around 150 students. Although those cited papers share authors with the present work, the cited results are empirical measurements, externally falsifiable, and do not presuppose the conclusion that herakoi itself is effective. The paper does assume, without direct herakoi-specific user testing, that edukoi's positive results transfer to herakoi; Footnote 10 explicitly concedes that research applications of herakoi have not been systematically investigated. That is an external-validity or generalization gap, not a circular reduction: no equation, definition, or construction in this manuscript makes the claimed high reliability and effectiveness true by definition. The default color-to-pitch mapping is presented as a design choice consistent with prior sonification literature, not as a theorem derived from the tool's inputs. Therefore, the appropriate finding is no significant circularity, with the noted evidence gap belonging to a correctness or completeness discussion rather than to circularity.
Assumptions & free parameters
free parameters (1)
- Hue scale truncation =
80% of the hue scale
assumptions (3)
- domain assumption MediaPipe Hand Landmarker reliably detects and tracks hand landmarks in real time via a webcam.
- domain assumption The color-to-pitch mapping based on hue is a valid and effective sonification strategy for conveying image content.
- domain assumption Results obtained with edukoi in school settings transfer to herakoi.
Cite this review
Pith. "Pith review of herakoi: a sonification experiment for astronomical data." pith.science (2026). https://pith.science/paper/O4QCY6HP
@misc{pith2026241209152,
author = {Pith},
title = {Pith review of: herakoi: a sonification experiment for astronomical data},
year = {2026},
howpublished = {\url{https://pith.science/paper/O4QCY6HP}},
note = {Machine review of arXiv:2412.09152}
}
read the original abstract
Recent research is revealing data-sonification as a promising complementary approach to vision, benefiting both data perception and interpretation. We present herakoi, a novel open-source software that uses machine learning to allow real-time image sonification, with a focus on astronomical data. By tracking hand movements via a webcam and mapping them to image coordinates, herakoi translates visual properties into sound, enabling users to "hear" images. Its swift responsiveness allows users to access information in astronomical images with short training, demonstrating high reliability and effectiveness. The software has shown promise in educational and outreach settings, making complex astronomical concepts more engaging and accessible to diverse audiences, including blind and visually impaired individuals. We also discuss future developments, such as the integration of large language and vision models to create a more interactive experience in interpreting astronomical data.
Figures
Reference graph
Works this paper leans on
-
[1]
2017, Proceedings of the International Astronomical Union, 14, 251–256 Di Mascolo, L
Hannam, J., & Garcia, B. 2017, Proceedings of the International Astronomical Union, 14, 251–256 Di Mascolo, L. & Ginolfi, M. 2023, her- akoi/herakoi: v0.2.0 Guiotto Nai Fovino, L., Grassi, M., Zanella, A., Di Mascolo, L., & Ginolfi, M. 2023, arXiv e-prints, arXiv:2309.02209 Guiotto Nai Fovino, L., Zanella, A., Di
arXiv 2017
-
[2]
Evaluating the efectiveness of sonifcation in science education using Edukoi
Mascolo, L., et al. 2024a, arXiv e-prints, arXiv:2405.18908 Guiotto Nai Fovino, L., Zanella, A., & Grassi, M. 2024b, AJ, 167, 150
-
[3]
Meredith, K., & Misdariis, N. 2022, Nature Astronomy, 6, 22
work page 2022
-
[4]
Harrison, C. M., Zanella, A., & Girdhar, A. 2023, Communicating Astronomy with the Public Journal, 33, 6
work page 2023
-
[5]
G., Yunus, M., O’Neill, C., et al
Iyer, K. G., Yunus, M., O’Neill, C., et al. 2024, arXiv e-prints, arXiv:2408.01556
arXiv 2024
-
[6]
2019, arXiv e-prints, arXiv:1906.08172
Lugaresi, C., Tang, J., Nash, H., et al. 2019, arXiv e-prints, arXiv:1906.08172
arXiv 2019
-
[7]
2022, Nature Astronomy, 6, 1249
Misdariis, N., ¨Ozcan, E., Grassi, M., et al. 2022, Nature Astronomy, 6, 1249
work page 2022
-
[8]
Peebles, P. J. & Ratra, B. 2003, Reviews of Modern Physics, 75, 559
work page 2003
Show all 12 references
-
[9]
D., et al
Perkowski, E., Pan, R., Nguyen, T. D., et al. 2024, Research Notes of the American Astronomical Society, 8, 7 Planck Collaboration, Aghanim, N., Akrami, Y ., et al. 2020, A&A, 641, A6
2024
-
[10]
R., Moran, J
Thompson, A. R., Moran, J. M., & Swenson, George W., J. 2017, Interferometry and Synthesis in Radio Astronomy, 3rd Edition
2017
-
[11]
W., Harrison, C
Trayford, J. W., Harrison, C. M., Hinz, R. C., et al. 2023, RAS Techniques and Instruments, 2, 387 Tucker Brown, J., Harrison, C. M., Zanella, A., & Trayford, J. 2022, MNRAS, 516, 5674
2023
-
[12]
M., Lenzi, S., et al
Zanella, A., Harrison, C. M., Lenzi, S., et al. 2022, Nature Astronomy, 6, 1241
2022
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.