{"id":"ff49884b-b8cc-4c52-a474-af75a798ac08","arxiv_id":"2608.03419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"V2N uses a one-second keyboard video window and four jointly trained heads to predict onset, offset, key hold, and velocity, achieving state-of-the-art visual piano transcription on PianoVAM and R3.","lead":"This paper presents V2N, a system that watches videos of piano keys and converts them into MIDI notes, including when each key is released and how hard it was pressed. It is the first video-only transcription system to report all four note attributes, and it outperforms earlier visual methods on two public datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The R3 SOTA margin rests on an untested assumption that >200 ms MIDI/video misalignment in 10 R3x test files degrades all models equally; the cleaned-split gain is shown only for V2N, so the original-split ranking is not yet established.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the R3 sync explanation is unsupported, and the large cleaned-split gain for V2N alone leaves the SOTA margins uncertain. I considered other potential concerns, such as the absence of a non-degenerate velocity baseline and the narrowness of the PianoVAM test set, but these do not threaten the internal validity of V2N's reported gains because the ablations provide within-system comparisons and the PianoVAM MIDI is synchronized. The R3 sync issue is different: it is an acknowledged dataset artifact, the equal-effect assertion is an empirical claim with no supporting baseline numbers, and the architecture's extra temporal context makes equal degradation unlikely. The concrete test would settle whether the original-split ranking holds. Since the central contribution is also supported by PianoVAM results and ablations, the appropriate disposition remains conditional acceptance; the concern should be addressed but does not overturn the core finding.","tokens_in":10175,"tokens_out":14611,"duration_ms":138043,"concrete_test":"Recompute the R3x rows of Table 1 after removing the 10 test files whose per-file sync offsets (released with the code) exceed 200 ms, for every baseline (S2S, V2R, PPAN, and Li et al. video-only), not just V2N. Compare the change in Onset/+Off/+Vel/+Off+Vel F1 at 50 and 100 ms against V2N's reported +7.7 pp Onset improvement. If all baselines shift by comparable amounts and pairwise gaps are preserved within the Wilcoxon threshold, the original-split ranking is robust; if deltas differ, report cleaned-split margins and re-run significance tests.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The R3 state-of-the-art claim depends on the assertion in Section 6 that the 70/895 R3 files with MIDI-video offsets exceeding 200 ms 'affects all models equally,' justifying use of the original test split. This is not demonstrated: the cleaned-split gain is reported only for V2N (+7.7 pp Onset at 100 ms on R3x after excluding the 10 sync-affected files), not for any baseline. The equal-effect assumption is questionable because V2N uses a 1 s window with a bidirectional BiLSTM and retains the central 0.5 s at inference, providing roughly 0.25-0.75 s of future context; baselines use 5-6 frame windows (~0.2 s) and have at most ~0.1 s of future context. A model with sufficient future context can partially compensate for MIDI-preceding-video shifts by predicting onsets before the visual press, while short-window models cannot, so the sync-affected files can interact differently with each architecture. The paper also does not report whether R3 training files carry the same misalignment, which could differentially shape learned anticipation biases. This concern is serious but not fatal to the central functional claim, because the PianoVAM ablations independently support multi-task, multi-frame supervision for offset and velocity prediction; however, the R3 numerical contributions are not robust as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents V2N, a video-only piano transcription system that jointly predicts onsets, offsets, key hold states, and per-note velocity from one-second video windows. A shared temporal backbone (Conformer-style blocks) feeds four BiLSTM-based task heads, and training supervises every frame rather than only the window center, with task loss weights onset=2 and others=1. The system is evaluated on PianoVAM and R3 with onset, +offset, +velocity, and +offset+velocity F1 at 50 ms and 100 ms tolerances. The authors report that V2N matches or surpasses prior video-only baselines on onset, substantially improves offset metrics, and is the first video-only system to report note-level velocity F1. Ablations attribute these gains to multi-task heads, multi-frame loss, sequence modeling, and the one-second input window. Cross-dataset transfer between PianoVAM and R3 collapses for all systems, which the authors attribute to fixed-geometry preprocessing.","tokens_in":10469,"tokens_out":6507,"duration_ms":55324,"significance":"If the results hold, V2N demonstrates that physical key release and strike velocity are visually predictable, closing the offset-accuracy gap and enabling complete MIDI transcription from video alone. The main scientific contribution is the evidence in Tables 2 and 3 that dedicated offset supervision, multi-task training, and temporal context jointly produce large gains on offset and velocity metrics. The paper is transparent: code, checkpoints, predicted MIDI, and per-file sync offsets are released, the R3 synchronization issue is disclosed rather than hidden, and the cross-dataset transfer failure is analyzed explicitly. The central functional claims are internally consistent and supported by ablations. The main uncertainty is the robustness of the R3 state-of-the-art numbers, which rest on an untested equal-impact assumption for systematically misaligned test files.","major_comments":[{"comment":"The statement that the 70/895 R3 files with video–MIDI offsets exceeding 200 ms 'affects all models equally' is not demonstrated. Cleaned-split gains are reported only for V2N (+7.7 pp Onset at 100 ms on R3x), not for any baseline, so the original-split rankings in Table 1 may change if sync errors interact with architecture. This interaction is plausible: V2N's 1 s window with central 0.5 s inference supplies roughly 0.25–0.75 s of future context, whereas the 5–6 frame windows of S2S, V2R, and PPAN supply at most about 0.1 s of future context; with MIDI preceding video, a model with enough future context can anticipate the early ground-truth onset while short-window models cannot. Please re-evaluate all baselines on the cleaned R3x split (or on a properly synchronized alignment), and report whether R3 training files carry the same misalignment, since that could differentially shape learned anticipation biases.","section":"Section 6"},{"comment":"The claim that the 1 s window 'is decisive on R3' is not backed by a controlled ablation. Table 3 varies the window size only on PianoVAM, where the gain is +0.4 pp Onset and +0.8 pp +Off at 50 ms; the R3 comparison against baselines with different architectures, losses, and training recipes is not an ablation. Provide an R3 version of Table 3 (same model with 0.2 s vs 1.0 s windows) or qualify the claim to state that the R3 benefit is inferred from cross-model comparison rather than from a controlled experiment.","section":"Section 5.2, Table 3"},{"comment":"All significance claims are based on paired Wilcoxon one-sided tests, with n=9 for PianoVAM, and no confidence intervals or run-to-run variance are reported. With a nine-recording test set, even large mean differences can be unstable, and the reported p<0.01 is not a substitute for variance estimates. Please report bootstrap confidence intervals or per-seed standard deviations, especially for the R3 numbers, where the baseline results are not re-evaluated on the cleaned split and the top-line margins are therefore not yet established as robust.","section":"Section 5.1, Table 1"}],"minor_comments":[{"comment":"Frame F1 is computed on a 60 Hz grid while the model and predicted notes are on a 25 fps grid; please clarify how frame-level predictions are resampled or rounded to 60 Hz.","section":"Section 4.3"},{"comment":"The exclusion of the eight special-split PianoVAM recordings should be stated prominently in the table caption or a footnote so that readers can compare with prior work that may use a different test split.","section":"Section 4.1"},{"comment":"The sentence 'Our retrained S2S and V2R baselines reproduce the published R3s/R3x Onset numbers within roughly one percentage point' is vague; please specify which published numbers, which split, and which tolerance are being compared.","section":"Section 5.1"},{"comment":"No sensitivity analysis is reported for the soft-label kernel half-width (ell=2) or the binarization threshold (tau=0.5); if these were fixed across all runs, a sentence stating so would address potential concerns about tuning.","section":"Section 3.3"},{"comment":"The parenthesized values for degenerate or absent baseline heads are easy to misread as confidence intervals; consider using a separate marker or footnote to distinguish 'not a true head' values from valid scores.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The R3 synchronization issue is the main correctness risk: the equal-effects assumption is untested and the reported cleaned-split gains only for V2N suggest the SOTA margin could shrink after re-evaluation. The PianoVAM results and ablations are convincing enough to warrant revision rather than rejection. No circularity or attribution concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about this one. First, it is the first video-only system to actually predict note-level velocity and physical key-release offsets, and the ablations back that up. Second, the R3 state-of-the-art claim has a load-bearing assumption that hasn't been tested: the paper assumes that systematic MIDI-before-video timing offsets affect all models equally, but only reports cleaned-split numbers for V2N.\n\nThe genuinely new thing is the task formulation: a dedicated offset head supervised on physical key release, a velocity head, and per-frame multi-task training. The PianoVAM ablations are clean and show each head helps, both as supervision and at decoding. The offset-guided decoding beats key-hold-only decoding. Code and checkpoints are released. That's real evidence.\n\nThe soft spot is the R3 sync issue. Section 6 discloses that 70 of 895 R3 files have MIDI preceding video by more than 200 ms, and all are in that direction. The paper asserts this affects all models equally and keeps the original split. But V2N uses a 1 s window with a bidirectional LSTM, so it has up to ~0.75 s of future context; the baselines see at most ~0.2 s and almost no future frames. If the video event lags the MIDI label, a model with future context can learn to predict the onset before it visually happens. Short-window models cannot. So the equal-impact assumption is not safe. The paper reports that cleaning the 10 sync-affected R3x test files raises V2N's Onset F1 at 100 ms by +7.7 pp, but it doesn't report the same cleaning for baselines. Without that, the R3 margin could shift materially. This doesn't break the central claim—PianoVAM is clean and the ablations hold there—but the R3 numbers as reported are provisional.\n\nMinor issues: the PianoVAM test set is 9 recordings, no confidence intervals, just paired Wilcoxon. And the baselines have no velocity head and near-zero offsets for Li et al., so the SOTA gaps on those metrics are partly against degenerate baselines. That's not a flaw in the method, but it means the headline margins are less impressive than they look.\n\nWho is this for? MIR researchers who care about multimodal transcription or video-based interfaces. It deserves a serious referee. I'd send it to review and ask for a cleaned-split comparison on R3 for at least one baseline, plus a statement about sync in training files.","headline":"V2N is the first video-only system to get credible note-level velocity and physical-key-release offsets, and the PianoVAM ablations support it; the R3 SOTA claim rests on an untested equal-impact sync assumption, so treat those numbers as provisional.","tokens_in":10987,"tokens_out":3683,"would_cite":true,"duration_ms":32377,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Video of the keyboard alone—no audio—can now produce complete MIDI piano transcription, including key-release offsets and per-note velocity, a task audio-only systems cannot fully achieve because the sustain pedal masks key release.","keywords":["visual piano transcription","MIDI transcription","multi-task learning","key-release offset prediction","velocity estimation","video-only transcription","Conformer backbone"],"falsifier":"Re-run the R3 comparison on a version of the test set that excludes the 70 systematically misaligned files (or realigns them); the paper reports that V2N's onset F1 at 100 ms rises from 86.6 to 94.3 when 10 affected R3x test files are removed, so if the baselines also shift non-uniformly under the same exclusion, the state-of-the-art margin on R3x would shrink or vanish.","tokens_in":9978,"feed_emoji":"🎹","tokens_out":8086,"duration_ms":60153,"temperature":0.7,"pith_summary":"Video of a piano keyboard contains the physical truth audio hides: a key's press and release are visible even when the sustain pedal keeps the sound ringing. This paper argues that a video-only system, V2N, can transcribe complete MIDI notes—onset, offset, duration, and per-key velocity—by supervising four task heads (onset, offset, key hold, velocity) at every frame of a one-second window rather than only at the window center. If the paper is right, visual piano transcription can deliver note-level results audio systems cannot, using the camera instead of the microphone, and the reported numbers show V2N beating prior video-only baselines on every offset-dependent metric while matching or exceeding onset accuracy. The central design claim is that offset and velocity prediction in vision are not by-products of pressed-key detection but require their own supervision and dedicated decoding logic.","feed_headline":"Video-only piano transcription now reads key releases and velocity","feed_subtitle":"Per-frame multi-task training lets V2N beat prior video-only systems on physical key release and velocity","key_machinery":"The load-bearing mechanism is the four-head multi-task architecture with per-frame supervision and offset-guided decoding. Onset, offset, key hold, and velocity heads share a temporal backbone of three Conformer ConvModule blocks that give every output frame a receptive field over the full 1 s input; losses are applied at every frame rather than only at the window center. At inference, a note is started at an onset-head peak and ended at the earlier of an offset-head peak or a key-hold drop, a decoding rule that is what separates physical key-release prediction from mere pressed-key detection.","core_discovery":"The paper claims that physical key release (offset) and per-key velocity, which audio-based transcription cannot recover because the sustain pedal decouples sound from key motion, are directly observable in video and become predictable when trained as dedicated tasks under per-frame supervision. V2N pairs a shared convolutional temporal backbone with four parallel BiLSTM heads for onset, offset, key hold, and velocity, supervises every frame of a 1 s clip, and decodes each note's end from the earlier of an offset-head peak or a key-hold drop. On PianoVAM this yields 94.7 onset F1, 89.5 key-release F1, 82.8 velocity F1, and 78.3 all-attribute F1 at 50 ms tolerance; on R3 the same recipe sets a new state of the art and makes V2N the first video-only system to report note-level velocity F1. The ablation results are used to argue that multi-task heads improve onset accuracy and that offset accuracy collapses without dedicated offset supervision.","pith_inferences":["If key-release offsets become as reliable as onsets, video could be used to create sustain-pedal-independent training targets for audio-based transcription, effectively teaching audio systems what physical key release looks like in sound.","The per-frame multi-task recipe may transfer to other instruments with visible contact states, such as drumsticks on a kit or fretting fingers on a guitar, where physical onset/offset boundaries are visually available.","A testable extension is camera-agnostic keyboard normalization—mapping keys by physical position in the video frame rather than pixel column—which the paper identifies as future work; a positive result would open the same model to piano videos from arbitrary viewpoints."],"forward_implications":["Video alone can now produce complete MIDI notes, including physical key-release offsets and per-note velocity, matching or beating audio-dependent systems on onset while adding attributes audio cannot provide.","A dedicated offset head, trained with key-hold and velocity supervision, turns offset prediction from a near-collapse for prior systems (45.9 +Off F1 at 50 ms for PPAN on PianoVAM) into a usable 89.5.","Longer temporal context (1 s vs. 0.2 s) is decisive on heterogeneous rehearsal data, where short-window onset-only baselines cannot predict key release.","Because transfer between the two datasets collapses in every direction, current VPT models work only for the camera geometry seen in training; robustness to unseen geometry remains unsolved."],"supporting_citations":[{"why":"supplies the S2S feature extractor (ResNet-18 with a learned slope prior) that V2N reuses as its visual front end and the center-frame supervision recipe that V2N extends.","marker":"[6]"},{"why":"supplies the PPAN transformer baseline, the R3 preprocessing recipes, and the augmentation scheme V2N adopts.","marker":"[8]"},{"why":"provides the closest audio-visual baseline; its video branch is the direct comparison for onset and a baseline for offset/velocity gaps.","marker":"[9]"},{"why":"supplies the multi-task onset/key-hold/velocity paradigm and the task-weighting (onset=2) that V2N ports from audio.","marker":"[1]"},{"why":"supplies the Conformer ConvModule blocks used as the shared temporal backbone.","marker":"[10]"},{"why":"supplies the PianoVAM dataset and its splits for training, validation, and test.","marker":"[19]"},{"why":"supplies the R3 dataset and its R3s/R3x splits used for training and evaluation.","marker":"[17]"},{"why":"supplies the evaluation metrics that define the F1 scores.","marker":"[21]"}],"fun_headline_variants":["Video piano transcription now captures key releases and velocity","V2N: first video-only piano transcription with key release and velocity","Video-to-notes: predicting key releases and velocity from piano video","Per-frame multi-task video model tracks piano key lifts and velocity","Video piano transcription finally gets offset and velocity prediction"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The R3 state-of-the-art reading depends on the assumption that systematic video–MIDI timing errors, present in 70 of 895 files with MIDI always preceding video, degrade every competing system to the same degree and can safely remain in the test split.","fun_headline_variants_meta":{"raw":{"variants":["Video piano transcription now captures key releases and velocity","V2N: first video-only piano transcription with key release and velocity","Video-to-notes: predicting key releases and velocity from piano video","Per-frame multi-task video model tracks piano key lifts and velocity","Video piano transcription finally gets offset and velocity prediction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3130,"prompt_tokens":893,"completion_tokens":2237,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":2154}},"tokens_in":509,"tokens_out":2237,"duration_ms":14697,"temperature":1.0,"reasoning_tokens":2154,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:49:27.644553+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the R3 comparison on a version of the test set that excludes the 70 systematically misaligned files (or realigns them); the paper reports that V2N's onset F1 at 100 ms rises from 86.6 to 94.3 when 10 affected R3x test files are removed, so if the baselines also shift non-uniformly under the same exclusion, the state-of-the-art margin on R3x would shrink or vanish.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the S2S feature extractor (ResNet-18 with a learned slope prior) that V2N reuses as its visual front end and the center-frame supervision recipe that V2N extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the PPAN transformer baseline, the R3 preprocessing recipes, and the augmentation scheme V2N adopts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the closest audio-visual baseline; its video branch is the direct comparison for onset and a baseline for offset/velocity gaps."},{"cited_title":"Multi-Task Multi- Frame Visual Piano Transcription","cited_arxiv_id":null,"evidence_quote":"supplies the multi-task onset/key-hold/velocity paradigm and the task-weighting (onset=2) that V2N ports from audio."},{"cited_title":"Onsets and frames: Dual-objective piano transcription,","cited_arxiv_id":null,"evidence_quote":"supplies the Conformer ConvModule blocks used as the shared temporal backbone."},{"cited_title":"Conformer: Convolution-augmented transformer for speech recognition,","cited_arxiv_id":null,"evidence_quote":"supplies the PianoVAM dataset and its splits for training, validation, and test."},{"cited_title":"Pay attention to the keys: Visual piano transcription using transformers,","cited_arxiv_id":null,"evidence_quote":"supplies the R3 dataset and its R3s/R3x splits used for training and evaluation."},{"cited_title":"Enabling factorized piano music modeling and generation with the MAESTRO dataset,","cited_arxiv_id":null,"evidence_quote":"supplies the evaluation metrics that define the F1 scores."}],"review_version":2}