Multi-Task Multi-Frame Visual Piano Transcription

Alexander Lerch; Hoyeol Sohn; Juhan Nam; Yonghyun Kim

REVIEW 3 major objections 5 minor 31 references

Multi-Task Multi-Frame Visual Piano Transcription

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Video of the keyboard alone—no audio—can now produce complete MIDI piano transcription, including key-release offsets and per-note velocity, a task audio-only systems cannot fully achieve because the sustain pedal masks key release.

desk verdict V2N is the first video-only system to get credible note-level velocity and physical-key-release offsets, and the PianoVAM ablations support it; the R3 SOTA claim rests on an untested equal-impact sync assumption, so treat those numbers as provisional. read the letter →

arxiv 2608.03419 v1 pith:Q4DNCYEB submitted 2026-08-04 cs.SD cs.AIcs.CVcs.MMeess.IV

Yonghyun Kim , Hoyeol Sohn , Juhan Nam , Alexander Lerch This is my paper · ORCID

classification cs.SDcs.AIcs.CVcs.MMeess.IV

keywords visualpianotranscriptionMIDImulti-tasklearningkey-releaseoffsetpredictionvelocityestimationvideo-onlyConformerbackbone

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Video of a piano keyboard contains the physical truth audio hides: a key's press and release are visible even when the sustain pedal keeps the sound ringing. This paper argues that a video-only system, V2N, can transcribe complete MIDI notes—onset, offset, duration, and per-key velocity—by supervising four task heads (onset, offset, key hold, velocity) at every frame of a one-second window rather than only at the window center. If the paper is right, visual piano transcription can deliver note-level results audio systems cannot, using the camera instead of the microphone, and the reported numbers show V2N beating prior video-only baselines on every offset-dependent metric while matching or exceeding onset accuracy. The central design claim is that offset and velocity prediction in vision are not by-products of pressed-key detection but require their own supervision and dedicated decoding logic.

What carries the argument

The load-bearing mechanism is the four-head multi-task architecture with per-frame supervision and offset-guided decoding. Onset, offset, key hold, and velocity heads share a temporal backbone of three Conformer ConvModule blocks that give every output frame a receptive field over the full 1 s input; losses are applied at every frame rather than only at the window center. At inference, a note is started at an onset-head peak and ended at the earlier of an offset-head peak or a key-hold drop, a decoding rule that is what separates physical key-release prediction from mere pressed-key detection.

What would settle it

Re-run the R3 comparison on a version of the test set that excludes the 70 systematically misaligned files (or realigns them); the paper reports that V2N's onset F1 at 100 ms rises from 86.6 to 94.3 when 10 affected R3x test files are removed, so if the baselines also shift non-uniformly under the same exclusion, the state-of-the-art margin on R3x would shrink or vanish.

Watch

Extended reading notes

Core claim

The paper claims that physical key release (offset) and per-key velocity, which audio-based transcription cannot recover because the sustain pedal decouples sound from key motion, are directly observable in video and become predictable when trained as dedicated tasks under per-frame supervision. V2N pairs a shared convolutional temporal backbone with four parallel BiLSTM heads for onset, offset, key hold, and velocity, supervises every frame of a 1 s clip, and decodes each note's end from the earlier of an offset-head peak or a key-hold drop. On PianoVAM this yields 94.7 onset F1, 89.5 key-release F1, 82.8 velocity F1, and 78.3 all-attribute F1 at 50 ms tolerance; on R3 the same recipe sets a new state of the art and makes V2N the first video-only system to report note-level velocity F1. The ablation results are used to argue that multi-task heads improve onset accuracy and that offset accuracy collapses without dedicated offset supervision.

Load-bearing premise

The R3 state-of-the-art reading depends on the assumption that systematic video–MIDI timing errors, present in 70 of 895 files with MIDI always preceding video, degrade every competing system to the same degree and can safely remain in the test split.

Editorial extensions

If this is right

Video alone can now produce complete MIDI notes, including physical key-release offsets and per-note velocity, matching or beating audio-dependent systems on onset while adding attributes audio cannot provide.
A dedicated offset head, trained with key-hold and velocity supervision, turns offset prediction from a near-collapse for prior systems (45.9 +Off F1 at 50 ms for PPAN on PianoVAM) into a usable 89.5.
Longer temporal context (1 s vs. 0.2 s) is decisive on heterogeneous rehearsal data, where short-window onset-only baselines cannot predict key release.
Because transfer between the two datasets collapses in every direction, current VPT models work only for the camera geometry seen in training; robustness to unseen geometry remains unsolved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If key-release offsets become as reliable as onsets, video could be used to create sustain-pedal-independent training targets for audio-based transcription, effectively teaching audio systems what physical key release looks like in sound.
The per-frame multi-task recipe may transfer to other instruments with visible contact states, such as drumsticks on a kit or fretting fingers on a guitar, where physical onset/offset boundaries are visually available.
A testable extension is camera-agnostic keyboard normalization—mapping keys by physical position in the video frame rather than pixel column—which the paper identifies as future work; a positive result would open the same model to piano videos from arbitrary viewpoints.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

You should know two things about this one. First, it is the first video-only system to actually predict note-level velocity and physical key-release offsets, and the ablations back that up. Second, the R3 state-of-the-art claim has a load-bearing assumption that hasn't been tested: the paper assumes that systematic MIDI-before-video timing offsets affect all models equally, but only reports cleaned-split numbers for V2N.

The genuinely new thing is the task formulation: a dedicated offset head supervised on physical key release, a velocity head, and per-frame multi-task training. The PianoVAM ablations are clean and show each head helps, both as supervision and at decoding. The offset-guided decoding beats key-hold-only decoding. Code and checkpoints are released. That's real evidence.

The soft spot is the R3 sync issue. Section 6 discloses that 70 of 895 R3 files have MIDI preceding video by more than 200 ms, and all are in that direction. The paper asserts this affects all models equally and keeps the original split. But V2N uses a 1 s window with a bidirectional LSTM, so it has up to ~0.75 s of future context; the baselines see at most ~0.2 s and almost no future frames. If the video event lags the MIDI label, a model with future context can learn to predict the onset before it visually happens. Short-window models cannot. So the equal-impact assumption is not safe. The paper reports that cleaning the 10 sync-affected R3x test files raises V2N's Onset F1 at 100 ms by +7.7 pp, but it doesn't report the same cleaning for baselines. Without that, the R3 margin could shift materially. This doesn't break the central claim—PianoVAM is clean and the ablations hold there—but the R3 numbers as reported are provisional.

Minor issues: the PianoVAM test set is 9 recordings, no confidence intervals, just paired Wilcoxon. And the baselines have no velocity head and near-zero offsets for Li et al., so the SOTA gaps on those metrics are partly against degenerate baselines. That's not a flaw in the method, but it means the headline margins are less impressive than they look.

Who is this for? MIR researchers who care about multimodal transcription or video-based interfaces. It deserves a serious referee. I'd send it to review and ask for a cleaned-split comparison on R3 for at least one baseline, plus a statement about sync in training files.

Referee Report

3 major / 5 minor

Summary. The paper presents V2N, a video-only piano transcription system that jointly predicts onsets, offsets, key hold states, and per-note velocity from one-second video windows. A shared temporal backbone (Conformer-style blocks) feeds four BiLSTM-based task heads, and training supervises every frame rather than only the window center, with task loss weights onset=2 and others=1. The system is evaluated on PianoVAM and R3 with onset, +offset, +velocity, and +offset+velocity F1 at 50 ms and 100 ms tolerances. The authors report that V2N matches or surpasses prior video-only baselines on onset, substantially improves offset metrics, and is the first video-only system to report note-level velocity F1. Ablations attribute these gains to multi-task heads, multi-frame loss, sequence modeling, and the one-second input window. Cross-dataset transfer between PianoVAM and R3 collapses for all systems, which the authors attribute to fixed-geometry preprocessing.

Significance. If the results hold, V2N demonstrates that physical key release and strike velocity are visually predictable, closing the offset-accuracy gap and enabling complete MIDI transcription from video alone. The main scientific contribution is the evidence in Tables 2 and 3 that dedicated offset supervision, multi-task training, and temporal context jointly produce large gains on offset and velocity metrics. The paper is transparent: code, checkpoints, predicted MIDI, and per-file sync offsets are released, the R3 synchronization issue is disclosed rather than hidden, and the cross-dataset transfer failure is analyzed explicitly. The central functional claims are internally consistent and supported by ablations. The main uncertainty is the robustness of the R3 state-of-the-art numbers, which rest on an untested equal-impact assumption for systematically misaligned test files.

major comments (3)

[Section 6] The statement that the 70/895 R3 files with video–MIDI offsets exceeding 200 ms 'affects all models equally' is not demonstrated. Cleaned-split gains are reported only for V2N (+7.7 pp Onset at 100 ms on R3x), not for any baseline, so the original-split rankings in Table 1 may change if sync errors interact with architecture. This interaction is plausible: V2N's 1 s window with central 0.5 s inference supplies roughly 0.25–0.75 s of future context, whereas the 5–6 frame windows of S2S, V2R, and PPAN supply at most about 0.1 s of future context; with MIDI preceding video, a model with enough future context can anticipate the early ground-truth onset while short-window models cannot. Please re-evaluate all baselines on the cleaned R3x split (or on a properly synchronized alignment), and report whether R3 training files carry the same misalignment, since that could differentially shape learned anticipation biases.
[Section 5.2, Table 3] The claim that the 1 s window 'is decisive on R3' is not backed by a controlled ablation. Table 3 varies the window size only on PianoVAM, where the gain is +0.4 pp Onset and +0.8 pp +Off at 50 ms; the R3 comparison against baselines with different architectures, losses, and training recipes is not an ablation. Provide an R3 version of Table 3 (same model with 0.2 s vs 1.0 s windows) or qualify the claim to state that the R3 benefit is inferred from cross-model comparison rather than from a controlled experiment.
[Section 5.1, Table 1] All significance claims are based on paired Wilcoxon one-sided tests, with n=9 for PianoVAM, and no confidence intervals or run-to-run variance are reported. With a nine-recording test set, even large mean differences can be unstable, and the reported p<0.01 is not a substitute for variance estimates. Please report bootstrap confidence intervals or per-seed standard deviations, especially for the R3 numbers, where the baseline results are not re-evaluated on the cleaned split and the top-line margins are therefore not yet established as robust.

minor comments (5)

[Section 4.3] Frame F1 is computed on a 60 Hz grid while the model and predicted notes are on a 25 fps grid; please clarify how frame-level predictions are resampled or rounded to 60 Hz.
[Section 4.1] The exclusion of the eight special-split PianoVAM recordings should be stated prominently in the table caption or a footnote so that readers can compare with prior work that may use a different test split.
[Section 5.1] The sentence 'Our retrained S2S and V2R baselines reproduce the published R3s/R3x Onset numbers within roughly one percentage point' is vague; please specify which published numbers, which split, and which tolerance are being compared.
[Section 3.3] No sensitivity analysis is reported for the soft-label kernel half-width (ell=2) or the binarization threshold (tau=0.5); if these were fixed across all runs, a sentence stating so would address potential concerns about tuning.
[Table 1] The parenthesized values for degenerate or absent baseline heads are easy to misread as confidence intervals; consider using a separate marker or footnote to distinguish 'not a true head' values from valid scores.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: V2N's claims are empirical results on external benchmarks with fully specified architecture and losses.

full rationale

The paper does not contain a derivation chain that reduces to its inputs. V2N's central claims are empirical: it reports F1 scores on the public PianoVAM and R3 datasets, compares against externally defined baselines (S2S, V2R, PPAN, Li et al.) retrained with the authors' own code, and ablates its design choices (multi-task heads, multi-frame loss, sequence model, 1 s context) with controlled experiments. The loss in Eq. (1) and total loss in Eq. (2) are stated with fixed task weights, and all thresholds (tau=0.5, onset/offset tolerances 50/100 ms) are disclosed rather than fit to test results. The only self-citation is to the PianoVAM dataset paper (ref. [19]), which is a public dataset used with the release's own train/valid/test split; citing the dataset does not make the benchmark circular because the ground-truth MIDI and video content are external to the present model's construction. The Section 6 statement that >200 ms video–MIDI offsets 'affects all models equally' is an unsupported assumption, and the cleaned-split gain is reported only for V2N; this is a robustness and correctness concern about the R3 SOTA claim, but it is not a circularity concern because the models are not fit to the test-set sync errors. No equation or definition in the paper is equivalent by construction to a predicted quantity, and no load-bearing argument relies on a self-citation that is itself unverified. Therefore the circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No physical entities or new theoretical constructs are introduced. The result rests on dataset labels, camera normalization, architecture choices, and evaluation conventions, all stated in the paper but none independently verified here.

free parameters (4)

Task loss weights = onset=2, others=1
Chosen by hand to emphasize onset, following Onsets-and-Frames; no search or fitting reported.
Binarization threshold tau = 0.5
Used for onset, offset, and key hold probability heads during inference; set to a standard value.
Soft-label kernel half-width = 2 frames
Spreads onset and offset gradient labels over +/-2 frames; chosen ad hoc.
Inference window stride = 0.5 s with central 0.5 s retained
Overlap strategy to avoid boundary artifacts; affects final note timestamps.

assumptions (4)

domain assumption Disklavier MIDI and R3 annotations are accurate enough to supervise onset, offset, and velocity.
Used throughout as ground truth in Sections 3.1 and 4.1; no independent verification of label accuracy is provided.
domain assumption Perspective-transformed 800x144 grayscale frames preserve the visual information needed for key state and velocity.
All models share this preprocessing in Section 4.1; the cross-dataset failure documented in Section 5.3 shows sensitivity to this geometry assumption.
domain assumption Systematic R3 sync offsets affect all compared models equally.
Stated in Section 6, but only V2N's scores on the cleaned split are reported, so the equal-impact assumption is not tested.
domain assumption mir_eval metrics, including L2-optimal velocity rescaling, are accepted evaluation tools.
Defines the reported F1 values in Section 4.3, including a velocity tolerance that is lenient for constant-velocity baselines.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Multi-Task Multi-Frame Visual Piano Transcription." pith.science (2026). https://pith.science/paper/Q4DNCYEB

@misc{pith2026260803419,
  author       = {Pith},
  title        = {Pith review of: Multi-Task Multi-Frame Visual Piano Transcription},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4DNCYEB}},
  note         = {Machine review of arXiv:2608.03419}
}

read the original abstract

Audio-based piano transcription performs well on onset, pitch, and velocity, but the sustain pedal lets sound persist long after key release, so audio systems predict pedal-extended offsets rather than physical key release. Yet existing Visual Piano Transcription (VPT) systems focus on onset detection from short video windows, offset accuracy lags onset by a wide margin, and note-level velocity has not been reported. To address these gaps, we present V2N (Video to Notes), the first complete VPT system: a shared temporal backbone feeds task-specific heads for onset, offset, key hold, and velocity, jointly trained with per-frame supervision rather than only at the window center. Ablations show that multi-task supervision enables offset and velocity prediction while improving onset accuracy; longer temporal context yields further improvements. V2N sets new state-of-the-art results on PianoVAM and R3.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 29 canonical work pages

[1]

Multi-Task Multi- Frame Visual Piano Transcription

INTRODUCTION Automatic Music Transcription (AMT) recovers symbolic note attributes such as pitch, onset, offset, and velocity from audio. Audio-based piano transcription has matured in estimating pitch, onset, and velocity [1, 2, 3], butoffset remains systematically confounded by the sustain pedal: sound persists after the key returns to rest, so audio sy...

work page 2026
[2]

Multi-Task Multi-Frame Visual Piano Transcription

RELATED WORK 2.1 Audio-based Piano Transcription Audio-based piano transcription is a mature field [11]. On- sets and Frames [ 1] established multi-task prediction of onsets, sustained audio activity, and velocity, whose com- plementarity dramatically improved transcription. Subse- quent work refined the paradigm with Transformer archi- tectures [2] and h...

work page Pith review arXiv 2026
[3]

frame” exclusively for a video frame (40 ms; all models in this paper operate at 25 fps). This should not be confused with the “frame

METHODOLOGY 3.1 Task Definition A piano note is defined by a key press (MIDINoteOn) and a key release ( NoteOff), together with pitch and veloc- ity. Video capturesmechanicalcues (keys pressing, being held, and returning to rest); audio capturesacousticcues (hammer-strike transient and sound envelope). With the sustain pedal engaged, sound persists long a...

work page
[4]

EXPERIMENTAL SETUP 4.1 Datasets PianoV AM[19] contains 107 top-view piano videos with synchronized MIDI from a Yamaha Disklavier, recorded during amateur practice sessions under naturalistic con- ditions. We follow the proposed splits in PianoV AM v1.1’s metadata.json: train+ext-train (81 recordings) for training, valid (9) for validation, and test (9 rec...

work page
[5]

V2N is our default full-configuration model (onset, offset, key hold, velocity heads)

RESULTS 5.1 Main Results Table 1 compares V2N against the baselines defined in Section 4. V2N is our default full-configuration model (onset, offset, key hold, velocity heads). On PianoV AM, V2N matches Li et al. on Onset (within 0.5 %p at both tolerances) and surpasses all baselines on every offset-dependent metric. The largest gains appear on the physic...

work page
[6]

follows the S2S [6] recipe of supervising only thecenter 0 2 4 6 8 10 Time (s) 50 55 60 65 70 75 80 85MIDI pitch Ground Truth 0 2 4 6 8 10 Time (s) Li et al. 0 2 4 6 8 10 Time (s) PPAN 0 2 4 6 8 10 Time (s) V2N (Ours) 30 40 50 60 70 80 90 Velocity (MIDI) Figure 3.Qualitative piano-roll comparison.10 s PianoV AM excerpt (Yiruma,Kiss the Rain; test recordin...

work page 2024
[7]

Since this affects all models equally, we report results on the original test split

DISCUSSION R3 synchronization issues.Cross-correlating MIDI on- set trains with video audio energy (10 ms resolution,±3 s search) reveals systematic video–MIDI offsets exceeding 200 ms in 70 of 895 R3 files, all with MIDI preceding video. Since this affects all models equally, we report results on the original test split. Excluding the 10 sync-affected R3...

work page
[8]

CONCLUSION We presented V2N, a visual piano transcription system that jointly predicts onsets, offsets, key hold states, and velocity through multi-task, multi-frame modeling. V2N substan- tially improves physical key-release accuracy over prior video baselines, whose offsets either collapse to near-zero duration or are inferred post-hoc from key hold act...

work page

Show all 31 references

[9]

ACKNOWLEDGEMENTS This work was supported by the National Research Founda- tion of Korea (NRF) grant funded by the Korea government (MSIT) under Grant RS-2023-NR077289

2023
[10]

Onsets and frames: Dual-objective piano transcription,

C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dual-objective piano transcription,” inInterna- tional Society for Music Information Retrieval Confer- ence (ISMIR), 2018

2018
[11]

Sequence-to-sequence piano transcription with transformers,

C. Hawthorne, I. Simon, R. Swavely, E. Manilow, and J. H. Engel, “Sequence-to-sequence piano transcription with transformers,” inInternational Society for Music Information Retrieval Conference (ISMIR), 2021

2021
[12]

High- resolution piano transcription with pedals by regressing onset and offset times,

Q. Kong, B. Li, X. Song, Y . Wan, and Y . Wang, “High- resolution piano transcription with pedals by regressing onset and offset times,”IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3707–3717, 2021

2021
[13]

Skipping the frame-level: Event-based piano transcription with neu- ral semi-CRFs,

Y . Yan, F. Cwitkowitz, and Z. Duan, “Skipping the frame-level: Event-based piano transcription with neu- ral semi-CRFs,” inAdvances in Neural Information Processing Systems (NeurIPS), 2021

2021
[14]

Scoring time intervals using non- hierarchical transformer for automatic piano transcrip- tion,

Y . Yan and Z. Duan, “Scoring time intervals using non- hierarchical transformer for automatic piano transcrip- tion,” inInternational Society for Music Information Retrieval Conference (ISMIR), 2024

2024
[15]

Sight to sound: An end-to-end approach for visual piano transcription,

A. S. Koepke, O. Wiles, Y . Moses, and A. Zisserman, “Sight to sound: An end-to-end approach for visual piano transcription,” inIEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020

2020
[16]

Audeo: Audio gen- eration for a silent performance video,

K. Su, X. Liu, and E. Shlizerman, “Audeo: Audio gen- eration for a silent performance video,” inAdvances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020

2020
[17]

Pay attention to the keys: Visual piano transcription using transformers,

U. Zivanovic, I. Pilkov, and C. Cancino-Chacón, “Pay attention to the keys: Visual piano transcription using transformers,” inInternational Joint Conference on Ar- tificial Intelligence (IJCAI), 2025

2025
[18]

A two-stage audio-visual fusion piano transcription model based on the attention mechanism,

Y . Li, X. Wang, R. Wu, W. Xu, and W. Cheng, “A two-stage audio-visual fusion piano transcription model based on the attention mechanism,”IEEE/ACM Trans- actions on Audio, Speech, and Language Processing, vol. 32, pp. 3618–3630, 2024

2024
[19]

Conformer: Convolution-augmented transformer for speech recognition,

A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu, and R. Pang, “Conformer: Convolution-augmented transformer for speech recognition,” inInterspeech, 2020

2020
[20]

Auto- matic music transcription: An overview,

E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Auto- matic music transcription: An overview,”IEEE Signal Processing Magazine, vol. 36, no. 1, pp. 20–30, 2019

2019
[21]

Enabling factorized piano music modeling and generation with the MAESTRO dataset,

C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.- Z. A. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” inInterna- tional Conference on Learning Representations (ICLR), 2019

2019
[22]

Detection of piano keys pressed in video,

P. Suteparuk, “Detection of piano keys pressed in video,” Stanford University, Technical Report, 2014

2014
[23]

claVision: Visual automatic piano music transcription,

M. Akbari and H. Cheng, “claVision: Visual automatic piano music transcription,” inInternational Confer- ence on New Interfaces for Musical Expression (NIME), 2015

2015
[24]

Deep residual learning for image recognition,

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inIEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016

2016
[25]

An image is worth 16x16 words: Transformers for image recognition at scale,

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weis- senborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gellyet al., “An image is worth 16x16 words: Transformers for image recognition at scale,” inInternational Conference on Learning Repre- sentations (I...

2021
[26]

The rach3 dataset: Towards data-driven analysis of piano performance re- hearsal,

C. E. Cancino-Chacón and I. Pilkov, “The rach3 dataset: Towards data-driven analysis of piano performance re- hearsal,” inMultiMedia Modeling: 30th International Conference (MMM), 2024

2024
[27]

Virtual piano us- ing computer vision,

S. Kang, J. Kim, and S.-e. Yoon, “Virtual piano us- ing computer vision,”arXiv preprint arXiv:1910.12539, 2019

1910 arXiv
[28]

PianoV AM: A multimodal piano performance dataset,

Y . Kim, J. Park, J. Bae, K. Kim, T. Kwon, A. Lerch, and J. Nam, “PianoV AM: A multimodal piano performance dataset,” inInternational Society for Music Information Retrieval Conference (ISMIR), 2025

2025
[29]

Decoupled weight decay regularization,

I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” inInternational Conference on Learn- ing Representations (ICLR), 2019

2019
[30]

mir_eval: A trans- parent implementation of common MIR metrics,

C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Ni- eto, D. Liang, and D. P. W. Ellis, “mir_eval: A trans- parent implementation of common MIR metrics,” in International Society for Music Information Retrieval Conference (ISMIR), 2014

2014
[31]

Quantifying the corpus bias problem in automatic music transcription systems,

L. S. Marták, P. Hu, and G. Widmer, “Quantifying the corpus bias problem in automatic music transcription systems,” inInternational Workshop on Sound Signal Processing Applications (IWSSPA), 2024

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Multi-Task Multi- Frame Visual Piano Transcription

INTRODUCTION Automatic Music Transcription (AMT) recovers symbolic note attributes such as pitch, onset, offset, and velocity from audio. Audio-based piano transcription has matured in estimating pitch, onset, and velocity [1, 2, 3], butoffset remains systematically confounded by the sustain pedal: sound persists after the key returns to rest, so audio sy...

work page 2026

[2] [2]

Multi-Task Multi-Frame Visual Piano Transcription

RELATED WORK 2.1 Audio-based Piano Transcription Audio-based piano transcription is a mature field [11]. On- sets and Frames [ 1] established multi-task prediction of onsets, sustained audio activity, and velocity, whose com- plementarity dramatically improved transcription. Subse- quent work refined the paradigm with Transformer archi- tectures [2] and h...

work page Pith review arXiv 2026

[3] [3]

frame” exclusively for a video frame (40 ms; all models in this paper operate at 25 fps). This should not be confused with the “frame

METHODOLOGY 3.1 Task Definition A piano note is defined by a key press (MIDINoteOn) and a key release ( NoteOff), together with pitch and veloc- ity. Video capturesmechanicalcues (keys pressing, being held, and returning to rest); audio capturesacousticcues (hammer-strike transient and sound envelope). With the sustain pedal engaged, sound persists long a...

work page

[4] [4]

EXPERIMENTAL SETUP 4.1 Datasets PianoV AM[19] contains 107 top-view piano videos with synchronized MIDI from a Yamaha Disklavier, recorded during amateur practice sessions under naturalistic con- ditions. We follow the proposed splits in PianoV AM v1.1’s metadata.json: train+ext-train (81 recordings) for training, valid (9) for validation, and test (9 rec...

work page

[5] [5]

V2N is our default full-configuration model (onset, offset, key hold, velocity heads)

RESULTS 5.1 Main Results Table 1 compares V2N against the baselines defined in Section 4. V2N is our default full-configuration model (onset, offset, key hold, velocity heads). On PianoV AM, V2N matches Li et al. on Onset (within 0.5 %p at both tolerances) and surpasses all baselines on every offset-dependent metric. The largest gains appear on the physic...

work page

[6] [6]

follows the S2S [6] recipe of supervising only thecenter 0 2 4 6 8 10 Time (s) 50 55 60 65 70 75 80 85MIDI pitch Ground Truth 0 2 4 6 8 10 Time (s) Li et al. 0 2 4 6 8 10 Time (s) PPAN 0 2 4 6 8 10 Time (s) V2N (Ours) 30 40 50 60 70 80 90 Velocity (MIDI) Figure 3.Qualitative piano-roll comparison.10 s PianoV AM excerpt (Yiruma,Kiss the Rain; test recordin...

work page 2024

[7] [7]

Since this affects all models equally, we report results on the original test split

DISCUSSION R3 synchronization issues.Cross-correlating MIDI on- set trains with video audio energy (10 ms resolution,±3 s search) reveals systematic video–MIDI offsets exceeding 200 ms in 70 of 895 R3 files, all with MIDI preceding video. Since this affects all models equally, we report results on the original test split. Excluding the 10 sync-affected R3...

work page

[8] [8]

CONCLUSION We presented V2N, a visual piano transcription system that jointly predicts onsets, offsets, key hold states, and velocity through multi-task, multi-frame modeling. V2N substan- tially improves physical key-release accuracy over prior video baselines, whose offsets either collapse to near-zero duration or are inferred post-hoc from key hold act...

work page