{"id":"a0f7ef63-1025-436d-8f87-a6d840f0f86e","arxiv_id":"2509.08800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multimodal dataset of 21 hours of amateur piano practice with synchronized video, audio, MIDI, hand landmarks, and fingering pseudo-labels, plus benchmarks for audio-only and audio-visual transcription.","lead":"PianoVAM is a new 21-hour dataset of amateur piano practice, with synchronized top-view video, audio, MIDI, hand landmarks, and fingering labels. It could help researchers train audio-visual piano transcription systems and study fingering, given existing datasets lack one or more of these modalities.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fingering 'complete and accurate' claim conflicts with the algorithm's stated 'leave notes unlabeled' for no-candidate notes; whole-dataset accuracy rests on a tiny, unrepresentative validation.","rationale":"The reader's weakest assumption focuses on the generalizability of the fingering pseudo-labeler to the unvalidated majority of notes; that is a legitimate concern about external validity. My stress-test confirms that concern but identifies a more acute internal inconsistency: the algorithm is described as leaving no-candidate notes unlabeled, yet the paper claims 100% labeled coverage. This is not merely a sampling problem—it is a direct contradiction between the method description and the dataset statistics. If the released data indeed contain unlabeled notes, the dataset is incomplete, and the 'complete and accurate' headline claim is false. I still believe the dataset itself is useful and the paper is worth conditional acceptance, because the multimodal acquisition and benchmarking contributions are real and the limitations section partially acknowledges visual ambiguities. However, the manuscript must either correct the completeness claim, clarify that human annotators fill in no-candidate notes, or release per-note annotation status. The reader's verdict of CONDITIONAL remains appropriate; my concern reinforces it rather than changing it, so I recommend UNCHANGED.","tokens_in":10525,"tokens_out":4103,"duration_ms":48110,"concrete_test":"Download the released PianoVAM dataset and count, for every MIDI file, the number of note events that lack any fingering annotation. If any notes lack a finger label, Table 2's 100% labeled-ratio claim is false. Independently, trace the annotation GUI code path to confirm whether no-candidate notes are subsequently human-labeled; if they are not, the 'complete' claim is directly refuted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—'complete and accurate fingering annotations for the entire dataset' (§5)—is contradicted by the algorithm description. §5.2 states: 'Note that there might be notes with either no candidate or multiple candidates, in which case the algorithm will leave these notes unlabeled.' Yet §5 claims the pipeline 'ensures complete and accurate fingering annotations for the entire dataset,' and Table 2 lists a 100% labeled ratio. Table 3 reports that, for the 10 validation pieces, 3.7–35.1% of notes (weighted average 13.0%) have no candidate. Unless the GUI-based manual step is also used to label no-candidate notes—which the paper never states—those notes remain unlabeled, making the dataset incomplete, not merely slightly inaccurate. Even ignoring completeness, the accuracy claim is overextended: precision is computed only on the first 150 notes of 10 pieces, a contiguous excerpt from each piece's opening, which is likely easier than later, denser passages. No recall is reported, and no-candidate notes are presumably excluded from the precision denominator. If ~13% of notes have no candidate and ~5% have multiple candidates, the fraction of all notes with a correct automatic label is at most ~0.956 × (1 − 0.13 − 0.05) ≈ 78% for those pieces—and this is unmeasured for the remaining ~96 pieces. The paper's own discussion (§7) acknowledges visual ambiguities in Ravel and Schumann, consistent with this concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PianoVAM is a new multimodal piano performance dataset comprising 21 hours (106 recordings) of amateur practice sessions captured with a Yamaha Disklavier, including synchronized top-view video, audio, MIDI, hand landmarks (MediaPipe), fingering pseudo-labels, and metadata. The paper describes the acquisition workflow, audio-MIDI alignment via FluidSynth rendering and DTW, loudness normalization, dataset statistics, and a semi-automated fingering annotation pipeline based on hand landmarks and a rule-based candidate scoring method with GUI-assisted disambiguation. Benchmark experiments cover audio-only piano transcription (Onsets and Frames) and an audio-visual post-processing filter, with statistical tests showing benefits from PianoVAM training data and from visual filtering in noisy/reverberant conditions. The central contribution is the dataset itself, with the fingering labeling being its most novel but least validated component.","tokens_in":10826,"tokens_out":5012,"duration_ms":52488,"significance":"If released as described, PianoVAM would be a valuable community resource: it combines real performance audio/MIDI with top-view video and hand landmarks at a scale comparable to OMAPS2, adds fingering annotations absent from MAESTRO, and provides a reproducible data split and baseline results. The distributional analysis relative to MAESTRO and the use of standard metrics with significance testing are useful. The manuscript also gives a fairly detailed description of the difficult audio/MIDI/video alignment problem. However, the paper's headline claim of 'complete and accurate fingering annotations for the entire dataset' is not supported by the evidence presented; this issue is load-bearing because the fingering modality is a primary differentiator of the dataset.","major_comments":[{"comment":"The statement 'This approach ensures complete and accurate fingering annotations for the entire dataset' is directly contradicted by §5.2's 'there might be notes with either no candidate or multiple candidates, in which case the algorithm will leave these notes unlabeled' and by Table 3, where 3.7–35.1% of notes (weighted average 13.0%) have no candidate. Table 2 nonetheless reports a 100% labeled ratio. If the custom GUI is used to fill in all no-candidate and multiple-candidate notes, that manual step must be stated and quantified; if it is not, the dataset is incomplete. The text also claims multiple-candidate cases affect ~20% of notes, whereas Table 3 gives a 5.1% weighted average, so the description of manual workload is internally inconsistent.","section":"§5 and §5.2 / Table 3"},{"comment":"The reliability estimate is based on the first 150 notes of 10 pieces—about 1,500 notes total, roughly 0.14% of the 1,050,966-note dataset—and the 10 excerpts are contiguous openings, which are unlikely to be representative of dense or technically difficult passages later in pieces. Precision is reported without recall and without specifying whether no-candidate notes are excluded from the denominator. Under the most optimistic interpretation, this gives an upper bound of ~95% × (1−0.13−0.05) ≈ 78% correctly labeled notes on the validation pieces, and this is unmeasured for the remaining 96 pieces. The authors should either evaluate on a random or stratified sample covering the full dataset or explicitly demote the claim to 'best-effort pseudo-labels.'","section":"§5.2.1 / Table 3"},{"comment":"The z-depth and candidate heuristics (28° IWR angle, 0.9 floating-hand threshold, 50%/80% candidate thresholds, and the ±2-key video filtering threshold in §6.3) contain several free parameters, but no sensitivity analysis is provided. Section 7 itself acknowledges failures due to motion blur in Ravel and shadows in Schumann, and Table 3 shows Ravel has 35.1% no-candidate notes. The ~95% precision number should therefore be presented as condition-specific rather than as a dataset-wide reliability certificate; at minimum, per-piece confidence intervals and a breakdown of error types are needed.","section":"§5.2 / §7"}],"minor_comments":[{"comment":"The dataset name is typeset inconsistently ('PianoV AM' vs. 'PianoVAM'); please unify the spelling in the title, abstract, and body.","section":"Title/Abstract"},{"comment":"The notation is confusing: Eq. (1) defines I0W0R0 as a median of |△IW R|, but Eq. (3)–(5) use ||I0W0||, ||W0R0||, ||R0I0|| as distances. Please clarify how the median triangle gives segment lengths and define all symbols. The symbol AR in Eq. (2) is overloaded as both a coordinate bound and the aspect ratio.","section":"§5.2, Eq. (1)–(5)"},{"comment":"Section 2.3 says the dataset is 'improved by manual annotation of incomplete fingering labels,' but the annotation pipeline in §5 never describes this manual step except for GUI-based disambiguation of multiple candidates. Add a cross-reference and a short description of how no-candidate notes are handled.","section":"§2.3 vs. §5"},{"comment":"'Powell's dog leg algorithm' should be 'Powell's dogleg algorithm.'","section":"§5.2"},{"comment":"The table's footer states 'Bold: highest; Underline: significantly higher over the preceding method,' but the table as printed uses no underlining. Please make the formatting consistent with the caption.","section":"§6.3, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The dataset appears to be a legitimate and potentially useful contribution, and the transcription benchmarks are competently executed. The main obstacle to acceptance is the unsupported completeness/accuracy claim for the fingering annotations; this is fixable by either adding the missing manual-labeling statistics and a more representative evaluation, or by reframing the fingering labels as pseudo-labels with known coverage limitations. I do not see this as a reject: the underlying data collection and alignment work is valuable, and the fingering issue is localized to one section of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: PianoVAM is a genuinely useful resource—the first public dataset I know of that pairs top-view video with real MIDI, audio, hand landmarks, and fingering labels from one recording rig. If you work on audio-visual transcription or hand-pose-based analysis of piano performance, this is worth a close look. The acquisition and alignment sections are careful, and the transcription benchmarks use proper statistical checks rather than just point estimates.\n\nThe main soft spot is the fingering annotation claim. Section 5 says the pipeline \"ensures complete and accurate fingering annotations for the entire dataset,\" but Section 5.2 says notes with no or multiple candidates are left unlabeled by the algorithm, and Table 3 reports 13% no-candidate and 5.1% multiple-candidate notes on average. The flowchart suggests a human GUI catches low-confidence cases, but the prose never explicitly says the GUI also labels zero-candidate notes, so the \"complete\" claim is at best under-specified and at worst wrong. The accuracy check is also thin: precision ~95% is computed on the first 150 notes of ten pieces—contiguous openings, which are usually not the hardest passages—and no-candidate notes are presumably excluded from the denominator. The hand-tuned thresholds (the 28-degree angle, the 0.9 floating-hand depth, the 50/80% candidate scores) are not validated on a held-out set, so overfitting is plausible. This is fixable: reword the claim, report precision over a stratified sample across entire pieces, or release the manual validation set.\n\nThe stress-test note is right about the contradiction and the thin validation. Its arithmetic about ~78% correct automatic labels stacks no-candidate and multiple-candidate notes together; the latter are presumably manually resolved, so the bound is pessimistic, but the underlying point stands: whole-dataset accuracy is unmeasured.\n\nWho this is for: MIR researchers building or evaluating piano transcription models, especially audio-visual fusion, and anyone studying hand motion or fingering from video. The dataset is real, the documentation is above average, and the limitations section is honest about performer bias and the practice-session nature. I would send this to review, and the revision should tighten the fingering claims and ideally add a proper validation set. I'd bring it to a reading group and cite it.","headline":"Valuable new multimodal piano dataset, but the fingering 'complete and accurate' claim is contradicted by the algorithm's own description and is validated on only 150 notes per piece.","tokens_in":11399,"tokens_out":4582,"would_cite":true,"duration_ms":43733,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PianoVAM publishes 21 hours of synchronized piano performance data with fingering labels.","keywords":["piano performance dataset","multimodal","audio-visual transcription","fingering annotation","hand landmarks","MediaPipe","MIDI","Disklavier"],"falsifier":"Take a random sample of, say, 500 notes from the remaining ~1.05 million annotated notes, have an expert manually label the fingering from the video, and compare; if the precision falls substantially below the reported ~95%, or if certain pieces with heavy pedal use or fast passagework consistently show errors, the claim of complete and accurate fingering annotations across the dataset is falsified.","tokens_in":10373,"feed_emoji":"🎹","tokens_out":3435,"duration_ms":32392,"temperature":0.7,"pith_summary":"The paper introduces PianoVAM, a multimodal dataset of amateur piano practice sessions recorded on a Disklavier, with synchronized top-view video, audio, MIDI, hand landmarks, fingering pseudo-labels, and metadata. The central claim is that this dataset fills gaps left by prior collections, which either lack video, rely on synthetic audio, or have incomplete fingering annotations. The paper further claims that its hybrid fingering annotation algorithm, combining hand-pose landmarks with manual refinement of ambiguous cases, yields complete fingering labels for all notes with about 95% precision. A sympathetic reader would care because the dataset enables audio-visual piano transcription research and the study of fingering and hand movement in real practice conditions.","feed_headline":"PianoVAM pairs 21 hours of piano video with MIDI and fingering labels","feed_subtitle":"A new resource that could let transcription models use visuals and teach fingering from real practice.","key_machinery":"The key mechanism is the hybrid fingering annotation pipeline. It first runs MediaPipe Hands on each video frame to extract hand landmarks, then estimates the relative z-depth of each hand from the projected 2D skeleton using a model skeleton defined by the wrist, index metacarpal, and ring metacarpal, with a heuristic 28-degree angle for the neutral hand and a 0.9 depth threshold to identify and discard a floating hand. For each MIDI note, a fingering score is computed from the number of frames in which each fingertip lies within the note's key area; fingers scoring above 50% and 80% of the maximum become normal and strong candidates. Notes with a single strong candidate are labeled automat","core_discovery":"The core discovery is the dataset itself: 106 solo piano recordings from 10 amateur performers, roughly 21 hours, captured in realistic practice conditions with a Yamaha Disklavier. Unlike MAESTRO, which lacks video, or OMAPS2 and PianoYT, which have limited or pseudo MIDI, PianoVAM provides aligned top-view video, real audio, ground-truth MIDI from the Disklavier, hand landmark sequences, and fingering pseudo-labels. The paper also demonstrates that the fingering annotation method, a hybrid of automatic hand-landmark-based candidate scoring and human selection when multiple candidates exist, reaches an average precision above 95% on a manually checked subset, and that a simple audio-visual","pith_inferences":["Beyond the paper's own benchmarks, the hand landmarks and fingering labels could be used to study the relationship between fingering choices and expressive timing or dynamics across the same pieces performed by different amateurs.","The validation of the fingering algorithm covers only the first 150 notes of 10 pieces; a natural next step is to extend the manual ground-truth set to a random sample across all 106 recordings to measure the accuracy on the remaining ~1.05 million notes and on rarer techniques.","Because the dataset is dominated by practice sessions with heavy pedal use and a specific recording studio, models trained on it may need domain adaptation before transferring to concert recordings or other pianos; this is a caveat rather than a defect, but testable."],"forward_implications":["If the fingering annotations are accepted, PianoVAM becomes the largest piano dataset with both real performance audio and complete fingering labels, enabling supervised learning of fingering prediction from video or score.","The synchronized top-view video and MIDI enable audio-visual transcription models that can be tested on realistic practice-room conditions, including noise and reverberation, as shown by the benchmark.","The dataset's hand landmarks and per-hand MIDI separation support research on left-right hand assignment, hand posture analysis, and practice behavior analysis.","The reported benchmark indicates that combining audio-only models with video-based candidate filtering can improve onset precision in degraded acoustic conditions, pointing to a practical path for robust transcription."],"fun_headline_variants":["PianoVAM: 21 hours of piano video with aligned MIDI and fingering","New piano dataset adds video to MIDI and fingering labels","PianoVAM captures real practice: video, audio, MIDI, and hand poses","PianoVAM: multimodal piano recordings with hand landmarks and fingerings","First piano dataset with video, real MIDI, and hand landmarks"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The fingering pseudo-labels for the entire dataset rest on the assumption that the hand-pose landmarks and the z-depth heuristics (28-degree angle and 0.9 threshold) correctly identify which hand and finger plays each note from top-view video, an assumption validated only on the first 150 notes of 10 pieces.","fun_headline_variants_meta":{"raw":{"variants":["PianoVAM: 21 hours of piano video with aligned MIDI and fingering","New piano dataset adds video to MIDI and fingering labels","PianoVAM captures real practice: video, audio, MIDI, and hand poses","PianoVAM: multimodal piano recordings with hand landmarks and fingerings","First piano dataset with video, real MIDI, and hand landmarks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1068,"prompt_tokens":691,"completion_tokens":377,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":276}},"tokens_in":435,"tokens_out":377,"duration_ms":3995,"temperature":1.0,"reasoning_tokens":276,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T20:06:05.222984+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 500 notes from the remaining ~1.05 million annotated notes, have an expert manually label the fingering from the video, and compare; if the precision falls substantially below the reported ~95%, or if certain pieces with heavy pedal use or fast passagework consistently show errors, the claim of complete and accurate fingering annotations across the dataset is falsified.","supporting_citations":[],"review_version":1}