{"id":"c795eb11-6593-48f8-a8eb-9a409bc294c7","arxiv_id":"1909.02856","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A per-video SVM decision boundary, learned against a negative bag of noise features, is used as a video descriptor and improves action recognition over average and max pooling on eight benchmarks.","lead":"The authors propose a new way to summarize a video for action recognition: train a support vector machine to separate a video's frame features from a fixed set of unrelated or noise features, then use the separating boundary as the video's signature. This 'SVM pooling' method reports accuracy gains over standard pooling across eight video and image-set benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unproved equivalence between Algorithm 2 and the P1 objective is load-bearing: all experiments test Algorithm 2, while Definition 1 defines SVMP via P1.","rationale":"The reader's weakest assumption (negative-bag proxy) is important but is partly addressed by the paper's negative-bag ablation in §5.2, which compares four choices on validation. The equivalence issue is more fundamental: it determines what object is being evaluated. Definition 1 defines SVMP via P1, a non-convex MIL problem; Algorithm 2 is introduced as an approximation and then used for all main results. The paper asserts equivalence without proof, and the assertion is doubtful because Algorithm 2 lacks any subset-selection decision variable. If the concern lands, the paper can still be salvaged empirically by redefining SVMP as the Algorithm-2 heuristic and softening the theoretical claims, but the current text overstates what has been established. This does not require rejection: the empirical gains across eight datasets are consistent and the method is reproducible in principle (though code is not released). A conditional accept with a request to prove or remove the equivalence claim, add error bars, and release code is appropriate. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":22203,"tokens_out":11231,"duration_ms":133683,"concrete_test":"Run a small exhaustive check in 2D: fix n=6 positive points, m=6 negative points, η=0.5. For every subset of size 3 treated as positives in P1, solve the resulting QP and take the lowest objective; this is the exact P1 value. Run Algorithm 2 over a log-spaced grid of C1 and stop when exactly 3 positives satisfy w^T x+b≥0. Compare the resulting (w,b) and objective values. Repeat for 100 random draws; if any draw differs, the §3.3.3 equivalence is false. A KKT comparison should be reported alongside.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3.3 claims that Algorithm 2 is equivalent to solving P1: 'due to the convexity of SVM, it can be shown that the optimizing objective of P1 will be the same in both cases.' This claim is the only bridge between Definition 1 (SVMP = solution of P1) and the experiments, since the paper states 'we use parameter-tuning solution in the following experiments.' But P1 is non-convex subset selection: θ(x;η) labels an unknown subset X̂+ of size at least η|X+| as positives and forces all other positive-bag points to be negatives. Algorithm 2 instead trains a soft-margin SVM with every positive-bag feature labeled +1 and the fixed negative bag labeled -1, then increases C until ≥η of the positives lie on the positive side. The points that end up on the negative side are not a decision variable; they are simply margin errors. Convexity of each fixed-subset SVM subproblem does not imply that the two optimizers coincide, and the KKT conditions differ (P1 has explicit subset constraints; Algorithm 2 has none). If the equivalence fails, the eight-benchmark results validate Algorithm 2, not Definition 1, and the conceptual story about max-margin selection of useful frames is not what was tested. The paper itself labels Algorithm 2 a 'heuristic,' which makes the unproved equivalence especially important.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SVM pooling (SVMP): given the CNN features of frames or clips in a video as a positive bag and a fixed negative bag (typically CNN features from random white-noise images), it learns an SVM hyperplane that separates at least a fraction η of the positive features from all negatives, and uses the hyperplane parameters (w,b) as a video descriptor. The manuscript presents linear, kernelized, temporally ordered, and end-to-end trainable variants of the idea, and reports experiments on eight benchmarks spanning action recognition, action detection, skeleton-based recognition, image-set verification, and dynamic texture recognition. The central claim is that this discriminative, max-margin pooling filters out uninformative frames and yields state-of-the-art or competitive accuracy, improving over standard average/max pooling by 3–14% and over prior published results by 1–5%.","tokens_in":22560,"tokens_out":5844,"duration_ms":68798,"significance":"If the claims hold, SVMP is a simple, feature-extractor-agnostic pooling layer that could be plugged into many CNN-based video systems; its compact fixed-length descriptor and the breadth of the evaluation are genuine strengths. The paper also makes a useful connection between multiple instance learning and standard SVM solvers, and the temporal and end-to-end variants broaden the potential impact. However, the formal bridge between the paper's Definition 1 and the algorithm actually used in the experiments is not established, and several empirical comparisons lack statistical support. The central idea is credible, but the manuscript needs substantial clarification and additional validation before the claimed results can be accepted at face value.","major_comments":[{"comment":"The claim that Algorithm 2 is equivalent to solving P1 is not established. P1 involves a non-convex subset-selection constraint |X̂+| ≥ η|X+| with unknown labels θ, whereas Algorithm 2 trains a soft-margin SVM with every positive-bag feature labeled +1 and raises C until at least the fraction η of positives lie on the positive side; the points on the negative side are margin errors, not decision variables. Convexity of each fixed-label SVM subproblem does not imply that the two optimizers coincide, and the paper itself calls Algorithm 2 a heuristic. Because all subsequent experiments use Algorithm 2 (and its ordered variant), the reported results validate Algorithm 2, not the Definition 1 SVMP descriptor. The authors should either prove the claimed equivalence under precise assumptions, or redefine SVMP as the output of Algorithm 2 and adjust the conceptual narrative accordingly; a small-scale comparison of Algorithms 1 and 2 against an exact or enumerated P1 solution would clarify the actual gap.","section":"3.3.3 (Eq. (1)–(5) vs. Algorithm 2)"},{"comment":"The choice of negative bag is the defining modeling assumption of the SVMP descriptor, but the comparison across the four negative-bag candidates (ActivityNet, UCF-101, Thumos, white noise) is reported without error bars or repeated trials, and the accompanying TSNE discussion is qualitative. The claim that white-noise CNN features are a good proxy for uninformative frames in real videos needs direct quantitative support: please report mean and variance over several random negative-bag draws on at least a subset of the datasets, and ideally show how validation-selected hyperparameters (η, C, bag sizes) transfer across datasets. Without this, it is difficult to tell how much of the reported gain depends on the single chosen negative-bag distribution.","section":"5.2 (Figure 5, negative-bag selection)"},{"comment":"The end-to-end gradient is derived for a squared-hinge SVM objective with a fixed regularization parameter λ, but the SVMP output used in experiments is obtained by Algorithm 2, which iteratively increases C until the η constraint is satisfied. This iterative search is not the argmin of a fixed objective, so Lemma 3.3 of [80] does not apply directly as written. The end-to-end experiments are therefore not supported by the derivation as stated; the authors should either derive gradients for the actual Algorithm 2 procedure, or present a controlled experiment showing that the approximate gradient behaves as intended (e.g., by comparing E2E training curves against a finite-difference or alternative approximation).","section":"4.2 (Eq. (9))"}],"minor_comments":[{"comment":"The temporally-ordered variant is not fully specified: Eq. (8) lists ordering constraints, but the actual optimization problem minimized by the 'custom solver' is not stated, and reference [78] is Manopt, a manifold optimization toolbox, not an SVM solver. Please provide the complete objective and implementation details, since ordered-SVMP is the variant used in most video experiments.","section":"3.5"},{"comment":"The text and Figure 5 panels are inconsistent: the text says Figure 5(b) shows accuracy versus C, but the caption labels that panel as a bag-size analysis, and the panel labels for negative-bag selection and η are also unclear. Please renumber the panels and fix all references.","section":"5.2"},{"comment":"All experimental numbers are point estimates with no error bars, number of repeats, or significance tests. For claims of 1–3% improvements over state-of-the-art, some measure of variance is needed, especially on the three-fold HMDB/UCF protocols where splits are known to have nontrivial variance.","section":"Tables 6–9"},{"comment":"Please clarify whether the row labeled 'I3D' is the I3D+ model with average pooling and whether the 'SVMP' row uses exactly the same input features, training data, and evaluation protocol; otherwise the comparison is not controlled.","section":"5.7, Table 8"},{"comment":"For non-CNN skeleton features, the paper says white noise with the dataset mean and deviation is used directly as the negative bag; please explain why raw white noise in the feature space is a meaningful proxy for uninformative skeleton configurations, since this is a different construction from the CNN-feature noise negatives used elsewhere.","section":"5.6 (MSR Action3D)"},{"comment":"Please correct typos and spacing errors, including 'Howeer', 'we ues', 'V emulapalli', and 'bgs'.","section":"Throughout"},{"comment":"The phrase 'nonlinear hyperplane' is imprecise; a kernelized SVM still yields a hyperplane in the induced feature space. Please rephrase accordingly.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an extension of a CVPR 2018 paper; the editor should weigh whether the new material (temporal constraints, end-to-end derivation, five additional datasets) is sufficient for a TPAMI submission. The main technical risk is the unproved equivalence in Section 3.3.3, and I would recommend asking the authors to provide either a proof or a clearly redefined descriptor for Algorithm 2. No code is provided, which limits the ability of reviewers to check the empirical claims against the described algorithm."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the empirical work is real: the SVM pooling procedure that the authors actually run (Algorithm 2) gives consistent gains over average and max pooling across eight benchmarks, and the temporal-ordering variant adds a small but repeatable improvement. Second, the paper's formal framing is not what the experiments test. Definition 1 defines the SVMP descriptor as the solution of the subset-selection MIL problem P1. Algorithm 2 is a heuristic: it trains a soft-margin SVM with every positive-bag feature labeled +1, then increases the regularization C until at least η of the positives fall on the positive side. The paper says the two coincide 'due to convexity of SVM,' but no proof is given and I do not think the statement is true. P1 is non-convex (unknown subset), and the KKT conditions for the two problems are different. Since all experiments use Algorithm 2, the results validate the heuristic, not the P1 objective. This is a real soft spot, and the paper itself calls Algorithm 2 a heuristic, which makes the unproved equivalence especially awkward.\n\nWhat is genuinely new here, relative to the authors' CVPR 2018 paper, is the temporal-ordering constraint, the end-to-end training variant, five additional datasets, and a careful analysis of the negative-bag choice. Those are incremental but useful; a journal extension does not need to reinvent the wheel. The parameter analysis is honest—they show that white-noise negatives win on validation and that C must be tuned carefully. The writing is clear, and the authors admit when results are not significant (e.g., on I3D+ features).\n\nOther soft spots: no released code, no error bars, and the state-of-the-art tables are not all apples-to-apples—some rows compare SVMP on different backbone features against baselines with different pipelines. The assumption that random-noise CNN features are a good proxy for in-video background is plausible but under-analyzed; the paper tries four candidates and picks by validation, which is legitimate but shows sensitivity to a somewhat arbitrary choice.\n\nWho is this for? People working on video pooling and action recognition with pre-trained features. It is a useful data point, not a breakthrough. If the authors either prove the equivalence or rewrite the paper around Algorithm 2 as the actual method, the piece would be substantially stronger. As it stands, it deserves serious peer review—a good reviewer can force that revision. My recommendation: send it to review, with the request that Section 3.3.3 be fixed.","headline":"Solid empirical extension of the CVPR SVM-pooling work, but the paper's formal story (P1) is not what the experiments actually test—Algorithm 2 is a heuristic, and the claimed equivalence is unproved.","tokens_in":23004,"tokens_out":2990,"would_cite":false,"duration_ms":35219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the parameters of an SVM separating a video's features from a bag of noise features form a video descriptor that beats standard average/max pooling on eight benchmarks.","keywords":["video representation","action recognition","discriminative pooling","multiple instance learning","support vector machine","SVM pooling","deep learning","temporal ordering"],"falsifier":"On a dataset whose frames carry per-frame action/background labels, test whether the positive support vectors selected by SVMP match the labeled action frames well above chance; if they do not, the claimed mechanism of filtering uninformative frames is not what drives the accuracy gains. A second test: swap the white-noise negative bag for features of background frames from the same videos and see whether accuracy improves, stays flat, or drops.","tokens_in":22024,"feed_emoji":"🎬","tokens_out":9463,"duration_ms":89835,"temperature":0.7,"pith_summary":"Most action-recognition models classify short clips and then pool the per-clip predictions, typically by averaging or max pooling, which weights all frames equally even though many frames are uninformative background. This paper proposes SVM pooling: treat the deep features of a video as a positive bag, take a fixed bag of irrelevant features (usually CNN features of white-noise images) as a negative bag, and learn a max-margin SVM hyperplane that separates a fraction of the positive bag from the negatives. The parameters of that hyperplane become the descriptor for the whole video. Because the SVM's support vectors determine the hyperplane, SVMP is effectively a weighted average pooling that assigns zero weight to non-support vectors, filtering out frames the boundary deems irrelevant. On eight benchmarks spanning action recognition, action detection, skeleton-based recognition, image-set verification, and texture recognition, the authors report that SVMP improves over standard pooling by 3–14% and over prior state-of-the-art results by 1–5%.","feed_headline":"SVM pooling beats max and average pooling on 8 benchmarks","feed_subtitle":"The margin hyperplane, learned against white-noise features, acts as a weighted pooling that ignores irrelevant frames.","key_machinery":"The central object is the SVM Pooling (SVMP) descriptor, defined as $\\mathrm{SVMP}(X) = [w, b]^\\top$ for a sequence $X$, where $(w,b)$ solves the max-margin MIL objective in Eq. (1): separate all negative-bag features from at least a fraction $\\eta$ of the positive-bag features, allowing slacks $\\zeta$ with regularization $C_1$. The paper's practical inference is Algorithm 2, which increases $C$ until the desired fraction of positive features lies on the positive side of the hyperplane; the resulting $w$ is a sparse linear combination of support vectors, giving the descriptor its weighted-pooling interpretation. Temporal extension adds constraints $w^\\top x_j + \\delta \\le w^\\top x_k$ for $j