{"id":"7f1db7df-8891-43b2-9b72-9227b64048bd","arxiv_id":"2501.16100","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A lightweight ensemble of audio and video classifiers detects football highlights with 89%/83% accuracy on small private datasets, but the method is not reproducible due to NDA and missing baselines.","lead":"The paper trains simple neural networks to spot football highlights from short audio and video clips, reaching 89% and 83% accuracy on its own test sets. It shows that a lightweight audio-video ensemble can be built with small data, but the models, data, and training details are not released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ensemble robustness claim is unmeasured: the paper reports no test-set metrics for Me, and the two illustrative Soccernet clips are anecdotal, with one containing a false positive.","rationale":"The reader's weakest assumption focuses on ground-truth label quality, which is indeed a serious concern because noisy or unrepresentative labels would invalidate all reported accuracies. However, the most immediate load-bearing gap in the central claim is more specific: the paper never quantifies the ensemble's performance at all. The central contribution stated in the abstract is that the ensemble 'shows improved robustness against false positives and false negatives,' yet Section IV-B compares only the audio and video models, and Section V-B offers only two qualitative clips, one of which contains an acknowledged false positive. Even if the labels are perfect, the claimed ensemble benefit is not established by any reported number. This concern is concrete and testable with the authors' existing models and test sets, and it does not depend on external data or on breaking the NDA. The reader's REJECT verdict remains appropriate and is reinforced by this gap, so I recommend no change to the verdict.","tokens_in":10402,"tokens_out":3080,"duration_ms":33614,"concrete_test":"Re-run the trained Ma and Mv on the exact test splits that produced Tables I-III; for each test chunk compute the per-stream scores pHa and pHv, then form pHe per Eq. (5) with epsilon = 0.5. Report the ensemble's accuracy, precision, recall, and F1 alongside the per-stream rows of Table I, plus a paired McNemar test comparing Me against each single stream. If the ensemble is not significantly better than both streams, or at least not worse on the same examples, the claimed robustness improvement is unsupported. Repeat over several random train/validation/test splits to obtain confidence intervals, since the current numbers are single-split point estimates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract and Section V claim that the ensemble model Me, formed by averaging audio and video scores (Eq. 5), improves robustness against false positives and false negatives. However, Section IV-B and Tables I-III report accuracy, precision, recall, and F1 only for the individual audio and video models; no ensemble metrics are reported on the held-out test sets. The only evaluation of Me is qualitative, on two Soccernet clips (Section V-B), and the second clip still yields a brief false highlight ('GK throw-in') that the authors say 'can be easily avoided automatically' rather than being avoided by the ensemble itself. This is a correctness gap, not just a reproducibility gap: averaging two classifiers can reduce variance, but it can also dilute a stronger stream with a weaker one, so the claimed improvement requires a paired comparison on identical test examples. The NDA (Remarks 3.1 and 4.1) prevents independent reimplementation, but it does not prevent reporting ensemble numbers from the already-trained models on the already-available test sets.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dual-stream deep-learning pipeline for detecting football highlights. Audio is encoded as Mel-spectrograms and classified by a 2D CNN; video is encoded as stacked grayscale frames (treated as a multi-channel image) and classified by a transfer-learned 2D CNN whose first-layer filters are adapted by averaging RGB weights across channels and replicating across frames. The authors build two private, balanced datasets of five-second chunks, train the two models, and report approximately 89% audio accuracy and 83% video accuracy on held-out test portions. They then define an ensemble model (Eq. 5) that averages the per-second audio and video scores, claim that it improves robustness against false positives and false negatives, and illustrate this with two qualitative clips from the Soccernet repository. The paper concludes that the approach is lightweight, scalable, and practical for automated highlight detection.","tokens_in":10616,"tokens_out":7918,"duration_ms":75938,"significance":"If the reported numbers are reliable, the work is a modest practical contribution: it provides a clean problem formulation, a simple way to reuse a 2D image-classification network for short video chunks, and evidence that small datasets can support reasonable highlight detection. The audio and video confusion matrices are internally plausible in shape, and the use of a public corpus for the two demonstration clips is a positive feature. However, the scientific significance is strongly limited by the private dataset, the NDA-restricted architecture and training details, the absence of any baseline comparison, and the lack of quantitative evaluation of the ensemble claim. The paper currently reads more like an industrial application note than a fully verifiable methods contribution.","major_comments":[{"comment":"There is a quantitative inconsistency between Table I and the confusion matrices. With a balanced test set and entries normalized to the total test-set size, the audio confusion matrix (TP 43%, FN 7%, FP 4%, TN 46%) yields accuracy 89%, recall 86%, precision 91.5%, and F1 approximately 88.7%, whereas Table I reports recall = accuracy = 89.24% and precision 89.33%. The video matrix (TP 41%, FN 9%, FP 7%, TN 43%) yields accuracy 84%, recall 82%, precision 85.4%, and F1 approximately 83.6%, again disagreeing with Table I. The statement in Section IV-B that accuracy 'coincides with the recall index' is also not generally true for a balanced test set; it requires the true-positive rate to equal the true-negative rate, which the confusion matrices do not satisfy. Please reconcile the tables and report exact test-set sizes and raw counts.","section":"Section IV-B, Tables I-III"},{"comment":"The central claim that the ensemble model Me improves robustness against false positives and false negatives is not quantitatively evaluated. Tables I-III report metrics only for the individual audio and video models; no test-set accuracy, precision, recall, F1, or confusion matrix is reported for the ensemble defined by Eq. (5). The only evaluation of Me is qualitative, on two Soccernet clips, and the second clip still contains a brief false highlight ('GK throw-in') that the authors propose to filter out post hoc rather than having the ensemble suppress it. Because averaging two scores can also dilute a stronger stream with a weaker one, the claimed improvement requires a paired comparison of Ma, Mv, and Me on identical test examples. Please report ensemble metrics on the same held-out test sets, including a per-second or per-chunk comparison and the chosen threshold epsilon.","section":"Section V, Eq. (5), Section V-B"},{"comment":"The experimental protocol is too underspecified to support the generalization claims. The datasets are described only as samples taken from 'available recordings' with known highlights, balanced to 50/50, with no source matches, number of matches, annotation protocol, or temporal split. A single random split is used, with no repeated runs, error bars, or confidence intervals, and no baseline comparison (e.g., majority class, standard feature-based methods, or prior soccer-event-detection systems). The NDA in Remarks 3.1 and 4.1 further prevents independent reproduction, since the architecture family, layer details, hyperparameters, and exact training configuration are withheld. In particular, the filter-adaptation step in Eq. (3) is asserted to preserve useful video features, but no ablation or alternative adaptation is tested. Please provide as much dataset and training detail as possible, add repeated split evaluation or cross-validation, and include at least one baseline comparison.","section":"Sections II-B, II-C, IV-A, Remarks 3.1 and 4.1"}],"minor_comments":[{"comment":"Reference [3] ('J. Smith and J. Doe', Journal of Video Analysis, 2000) appears to be an unverifiable placeholder; please replace it with a complete and accessible citation.","section":"Section I, reference [3]"},{"comment":"The condition Hpaq = H should be written as Hpaq = ∅; as printed, H is used both as the name of the highlight-interval mapping and as a set, which is confusing.","section":"Section II-A, Eq. (1)"},{"comment":"There is a typo, 'made up of of approximately'; more importantly, the approximate dataset sizes (about 700 and 1000 samples) should be replaced by exact cardinalities, and the train/validation/test split sizes should be given precisely.","section":"Section IV-A"},{"comment":"The pipeline does not specify the window stride or how the set of windows containing a given second is constructed; Eq. (4) depends on this choice, so please define the window placement and overlap.","section":"Section V-A, Eq. (4)"},{"comment":"The prediction curves in Figures 3 and 4 are difficult to read in print: please add axis labels, a clear legend, the ground-truth highlight intervals, and markings for the annotated video frames.","section":"Figures 3 and 4"}],"recommendation":"major_revision","confidential_remarks":"I am recommending major revision rather than rejection because the main missing items—ensemble metrics on the existing test sets, reconciliation of the confusion matrices with Table I, and repeated split or baseline results—are within the authors' control to provide. The more serious concern is the NDA: if the authors cannot disclose even the model family and training configuration, the paper will remain largely unverifiable as a methods contribution. I would ask the editor to weigh whether a private, NDA-protected system with qualitative demonstration is within the scope of this journal; if not, rejection may be the more appropriate outcome after the authors have had a chance to respond."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a cleanly written industry report on a lightweight recipe—Mel-spectrogram audio CNN plus frozen 2D image CNN on stacked grayscale frames, averaged together—that gets sensible accuracy on private football highlight data. What it is not is a reproducible research contribution, and more importantly the abstract's central claim that the ensemble improves robustness is not supported by any test-set numbers.\n\nWhat is genuinely useful: the problem formulation in Section II is crisp, and the trick of treating grayscale video frames as channels for a frozen 2D image encoder is a practical way to avoid 3D CNNs on small data. The reported individual accuracies (89% audio, 83% video) are plausible for a balanced binary task, and the confusion matrices are internally consistent. The paper is also honest about the NDA, and the author bios make clear this came out of an industrial lab.\n\nThe soft spots are real and load-bearing. First, the ensemble model Me is praised in the abstract and Section V, but no ensemble accuracy, precision, recall, or F1 is reported on any test set. The only evidence is two SoccerNet clips, and the second clip still contains a brief false highlight that the authors wave away as 'easily avoided automatically.' Averaging two classifiers does not automatically reduce false positives/negatives; it can just dilute a stronger stream. Since the trained models already exist, computing Me's metrics on the same test sets is trivial and should have been done. That is a correctness gap, not a style preference. Second, the NDA hides the architecture and training details, and no code or data are released, so no independent reimplementation is possible. Third, there are no baselines—not even a simple SVM on handcrafted features—and the single random split with no repeated runs means the reported numbers have no error bars. Fourth, the ground-truth labels are described only as 'known' highlights from 'available recordings,' with no match count or annotation protocol. Finally, reference [3] ('J. Smith and J. Doe, Journal of Video Analysis, 2000') looks like a placeholder and should be checked.\n\nOn the positive side, there is no circularity: the hand-set ensemble weights and threshold are not fitted to the test set and do not feed back into the reported individual accuracies. But the missing ensemble evaluation undercuts the main advertised benefit.\n\nIf I were refereeing, I'd ask for the ensemble metrics, repeated splits, a baseline comparison on SoccerNet, and a clarification of the data annotation. The recipe itself is worth knowing, but as it stands the paper does not demonstrate its central claim. For a practice-oriented venue it deserves a serious referee; for a top-tier conference requiring reproducibility, it's a desk reject. I'd probably assign it to a workshop track with major revision.","headline":"A readable, honest industry report on a small-data highlight detection recipe, but the ensemble improvement it claims is never measured.","tokens_in":11122,"tokens_out":3264,"would_cite":false,"duration_ms":30706,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a small ensemble of two 2D CNNs—one on audio Mel-spectrograms, one on stacked greyscale frames—detects football highlights at about 89% audio and 83% video test accuracy, and that averaging the two streams cuts false…","keywords":["sports highlight detection","audio-visual ensemble","Mel-spectrogram","2D convolutional neural network","transfer learning","football video analysis","lightweight deep learning","binary video chunk classification"],"falsifier":"Apply the described pipeline to a fresh set of full football broadcasts with independently annotated highlight intervals, and measure per-chunk classification accuracy with $k=5$ second windows and per-second scores at a threshold of 0.5. If the audio stream does not reach roughly 89% accuracy, the video stream roughly 83%, or if the averaged ensemble does not lower false-positive and false-negative rates below both single streams, the paper's core claim that small datasets and simple 2D models are sufficient would be contradicted.","tokens_in":10153,"feed_emoji":"⚽","tokens_out":11555,"duration_ms":100222,"temperature":0.7,"pith_summary":"This paper aims to show that automated sports highlight detection does not require heavy 3D video models or large annotated corpora. It frames the task as binary classification of $k$-second chunks: does a chunk intersect a labelled highlight interval? A 2D CNN trained from scratch on Mel-spectrograms of match audio reaches about 89% accuracy, and a 2D CNN built from a pretrained image classifier adapted to stacked greyscale frames reaches about 83% accuracy on the authors' private balanced test sets. Averaging the two streams' per-second scores is argued to compensate each modality's characteristic mistakes, so the ensemble is more robust to false positives and negatives than either stream alone. The claim matters because, if correct, fast and cost-effective highlight detection is within reach for small datasets and simple architectures, reducing manual annotation work in sports media.","feed_headline":"Two small 2D neural nets catch football highlights at 89% and 83%","feed_subtitle":"Averaging audio and video scores suppresses each stream's errors, making automated highlight detection cheap and fast.","key_machinery":"The load-bearing machinery is the pair of cheap encodings plus a score-averaging rule. Audio chunks become Mel-spectrograms, which emphasise human-voice frequencies and let a from-scratch 2D CNN capture commentator and crowd reactions. Video chunks become stacked greyscale frames, one channel per frame, so a pretrained image-classification model can be reused: the first convolutional layer's RGB filters are adapted to the multi-frame input by averaging each filter's three channel weights and repeating the result across frames, preserving the pretrained spatial knowledge. The ensemble is the arithmetic mean of per-second audio and video scores, and its claimed work is to cancel complementary false positives and negatives. The exact architectures and training configurations are withheld under a non-disclosure agreement, so the documented public core is this encoding scheme and the averaging rule rather than the numerical training details.","core_discovery":"The central claim is that a dual-stream design can detect football highlights with practical accuracy while staying lightweight. Each 5-second chunk is encoded either as a Mel-spectrogram image or as a stack of greyscale frames treated as a multi-channel image, and a binary classifier learns whether the chunk intersects a highlight interval. On the authors' test sets the audio model reaches approximately 89% accuracy and the video model approximately 83%, with balanced error behaviour across positive and negative labels. The key result is that the simple arithmetic average of the two streams' raw sigmoid scores at each second produces an ensemble whose mistakes are complementary rather than additive: crowd-noise audio false positives are offset by video true negatives, while video misses of spectator reactions are offset by audio detection of the crowd roar. The robustness mechanism is demonstrated on two applied clips, including one where audio extends the detected highlight through celebrations and one where video pins down the start of the action.","pith_inferences":["Beyond the paper, the complementary-error claim predicts a testable asymmetry: masking crowd audio should hurt the ensemble far less than the audio stream alone, because the video stream still detects goalmouth action.","The balanced private test sets probably overstate performance on real broadcasts, where highlight seconds are rare; reporting precision and recall under natural class imbalance would show whether the reported robustness survives deployment conditions.","The filter-averaging trick that converts RGB pretrained filters to stacked greyscale frames could transfer to other multi-channel inputs, such as optical-flow stacks or multi-view video, without retraining the feature extractor.","Varying the window length $k$ while keeping the labels fixed would map the precision-versus-training-difficulty trade-off noted in the paper, giving practitioners an operating curve for choosing $k$."],"forward_implications":["With the reported accuracies, highlight detection can run on modest computational resources and roughly 700 to 1000 labelled chunks per modality, so deployment is feasible where large annotated video corpora are unavailable.","Because the ensemble is just the mean of two independently trained streams, robustness can be improved by adding a modality without retraining either network.","Per-second score averaging over sliding windows naturally makes detected highlights include the build-up before a key event and the celebration after it, matching the structure of human-edited highlight clips.","The same binary-chunk formulation and pipeline applies to other sports and scene types, requiring new labelled data and a suitable choice of window length $k$."],"supporting_citations":[{"why":"shows that combining audio and video streams improves soccer action spotting, the multi-modal result the paper extends with lightweight 2D models.","marker":"[7]"},{"why":"supplies the transfer-learning method on which the video model is built.","marker":"[17]"},{"why":"establishes the 3D-CNN real-time event-detection baseline that the paper contrasts as computationally expensive.","marker":"[5]"},{"why":"demonstrates 3D CNNs for spatiotemporal features, the video approach the paper replaces with 2D CNNs on greyscale frames.","marker":"[4]"},{"why":"defines the Mel scale used to compute the audio spectrogram inputs.","marker":"[13]"},{"why":"provides the two public football clips used to demonstrate the ensemble's behaviour, including the false-positive case the paper analyses.","marker":"[18]"}],"fun_headline_variants":["Audio+video nets hit 89% and 83% on highlight detection","Small DL models find sports highlights at 89% accuracy","Dual-stream AI catches highlights with complementary errors","Audio and video ensemble boosts highlight detection robustness","89% audio, 83% video: lightweight DL for sports highlights"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the ground-truth highlight intervals used to label the audio and video chunks being correct and representative of real football broadcasts; the paper does not specify where the recordings come from, who annotated the highlights, or how many matches are involved.","fun_headline_variants_meta":{"raw":{"variants":["Audio+video nets hit 89% and 83% on highlight detection","Small DL models find sports highlights at 89% accuracy","Dual-stream AI catches highlights with complementary errors","Audio and video ensemble boosts highlight detection robustness","89% audio, 83% video: lightweight DL for sports highlights"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000336,"raw_usage":{"total_tokens":1828,"prompt_tokens":881,"completion_tokens":947,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":864}},"tokens_in":497,"tokens_out":947,"duration_ms":8297,"temperature":1.0,"reasoning_tokens":864,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:43:26.831445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the described pipeline to a fresh set of full football broadcasts with independently annotated highlight intervals, and measure per-chunk classification accuracy with $k=5$ second windows and per-second scores at a threshold of 0.5. If the audio stream does not reach roughly 89% accuracy, the video stream roughly 83%, or if the averaged ensemble does not lower false-positive and false-negative rates below both single streams, the paper's core claim that small datasets and simple 2D models are sufficient would be contradicted.","supporting_citations":[{"cited_title":"Improved Soccer Action Spotting using both Audio and Video Streams","cited_arxiv_id":"2011.04258","evidence_quote":"shows that combining audio and video streams improves soccer action spotting, the multi-modal result the paper extends with lightweight 2D models."},{"cited_title":"A survey on deep transfer learning,","cited_arxiv_id":null,"evidence_quote":"supplies the transfer-learning method on which the video model is built."},{"cited_title":"Using 3d convolutional neural networks for real- time detection of soccer events,","cited_arxiv_id":null,"evidence_quote":"establishes the 3D-CNN real-time event-detection baseline that the paper contrasts as computationally expensive."},{"cited_title":"Learning spatiotemporal features with 3d convolutional networks,","cited_arxiv_id":null,"evidence_quote":"demonstrates 3D CNNs for spatiotemporal features, the video approach the paper replaces with 2D CNNs on greyscale frames."},{"cited_title":"The relation of pitch to frequency: A revised scale,","cited_arxiv_id":null,"evidence_quote":"defines the Mel scale used to compute the audio spectrogram inputs."},{"cited_title":"Soccernet: A scalable dataset for action spotting in soccer videos,","cited_arxiv_id":null,"evidence_quote":"provides the two public football clips used to demonstrate the ensemble's behaviour, including the false-positive case the paper analyses."}],"review_version":1}