{"id":"e0fa485b-58b3-4b3e-bd91-e294d82027e6","arxiv_id":"1908.05877","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A context-aware zero-shot learning model predicts novel crowd behavior attributes by weighting known-attribute video classifiers with word-vector co-occurrence estimates, improving over prior zero-shot baselines.","lead":"A computer vision method learns to recognize crowd behaviors never seen in training, using co-occurrence of familiar behaviors as context. It also detects violence in a new video dataset with zero training examples, approaching supervised accuracy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (6) never penalizes zero-co-occurrence pairs, so CoCAZSL's learned 'visual compatibility' is not trained on incompatibility; the M-extrapolation assumption is unsupported.","rationale":"The paper is a good-faith extension of zero-shot learning to multi-label crowd behavior recognition, and the central experiment is reasonably designed. The most load-bearing step, however, is not the feature representation or split but the learned co-occurrence predictor M. The reader's weakest assumption was that M extrapolates to unseen attribute pairs; my concern identifies a concrete mechanism why that extrapolation is unsupported. Because Eq. (7) gives zero weight to c_ij = 0 pairs, and log(0) is undefined, the implementation necessarily skips or zeroes out all negative co-occurrence examples. The regression therefore has no training signal for visual incompatibility, despite the paper's motivating example being exactly an incompatible pair. This is an internal tension between the stated motivation and the objective, not merely a disagreement with prior work. The reader's Eq. (8) normalization point is valid but less decisive for ranking metrics, since the denominator depends only on q and therefore acts as a per-attribute constant for label-wise AUC and AP. The violence-transfer comparison is also softened by the fact that the supervised ViF baseline uses a weaker feature than the ITF used for zero-shot; a supervised ITF reaches 98.72 AUC, so the claim of beating fully supervised learning is feature-dependent. These issues do not by themselves refute the approach, but they mean the central claim should be accepted only conditionally on the negative-pair diagnostic and on fairer supervised baselines. The verdict therefore remains CONDITIONAL rather than moving to ACCEPT or REJECT.","tokens_in":19609,"tokens_out":7245,"duration_ms":78896,"concrete_test":"First run a diagnostic: fit M exactly as in Eq. (6) on known-known pairs, then compare predicted exp(v_i^T M v_j) for held-out known-known pairs with c_ij = 0 versus pairs with c_ij > 0. If the zero-co-occurrence pairs are not systematically ranked below positive pairs, M is not learning visual incompatibility. Then re-run the WWW 50-split zero-shot experiment with a modified regression that includes explicit negative training signal, e.g., a nonzero weight for c_ij = 0 or a margin penalty forcing exp(v_i^T M v_j) below a threshold for observed non-co-occurring pairs. Check whether the Table 2 AUC margin over TexCAZSL (0.69 vs 0.65) and the Table 3 violence AUC (87.22) persist. If they do not, the central claim that visual co-occurrence context drives the gains is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the trained known-to-novel conditional in Eq. (8), produced by M from Eq. (6). The objective in Eq. (6) weights each pair by w(c_ij) from Eq. (7). For c_ij = 0, w(0) = 0 (with alpha > 0), so all non-co-occurring known-attribute pairs are absent from the regression. M is therefore fit exclusively to positive co-occurrence counts. But the paper's motivation for preferring visual co-occurrence over text is precisely that some semantically similar pairs, e.g., 'indoor' and 'outdoor', have zero visual co-occurrence. The training signal contains no examples of such incompatibility, so there is no learned pressure for M to suppress them. At test time, Eq. (8) applies exp(v_q^T M v_p) to every known-novel pair, including many pairs that should be at or near zero. Positive word-vector similarity can therefore leak through, and the claimed 'visual compatibility' correction is not actually learned from negative evidence. If M does not extrapolate to unseen pairs, Eq. (1) marginalizes noise and the CoCAZSL gains in Table 2 and Table 3 are not explained by the stated mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes CoCAZSL, a zero-shot multi-label crowd behavior recognition method. The model first trains probabilistic classifiers for known attributes, then estimates relations from known to novel attributes via a bilinear map M that predicts visual co-occurrence from pairs of word vectors, and finally marginalizes known-attribute predictions through these relations in an IAP-style formulation given by Eq. (1). The authors also present a text-only variant (TexCAZSL). Experiments on the WWW crowd video dataset report improved label-based AUC and AP over four ZSL baselines, and a transfer experiment to Violence Flow suggests competitive zero-shot violence detection. The central claim is that modeling multi-attribute co-occurrence context improves zero-shot crowd behavior recognition and generalizes to novel behaviors cross-domain.","tokens_in":19959,"tokens_out":5978,"duration_ms":53994,"significance":"If the results hold, the paper makes a useful contribution by introducing zero-shot learning to multi-label crowd behavior recognition and by showing that co-occurrence context learned from known attributes can be transferred to novel attributes. The idea of learning a pairwise mapping from text embeddings to visual co-occurrence is interesting and the WWW dataset provides a challenging testbed. The reported AUC gains (0.69 vs 0.65 for TexCAZSL) and the cross-domain violence transfer are compelling. However, the two technical issues detailed below concern the probabilistic validity of the conditioning step and the mechanism by which visual compatibility is learned; both are load-bearing for the stated claims.","major_comments":[{"comment":"The quantity defined in Eq. (8) is not the conditional probability p(y*_q | y_p) used in Eq. (1). The denominator sums over the conditioning variable p (known attributes), producing a distribution over p for each fixed q, whereas a conditional p(y*_q | y_p) should sum over q to 1. This is also inconsistent with Eq. (5), where the normalization is over the conditioned variable j. Consequently, the marginalization in Eq. (1) does not yield calibrated probabilities over novel labels, and the relative scale of scores for different q is arbitrary. This scale arbitrariness affects the example-based metrics (e.g., example-based AP and ranking loss) though not the per-label AUC. Please clarify whether Eq. (8) is intended as an unnormalized affinity; if so, the probabilistic graphical model in §3.1 and the interpretation of Eq. (1) as a marginal conditional probability need to be revised.","section":"§3.2.2, Eq. (8)"},{"comment":"The regression in Eq. (6) is trained only on known attribute pairs with positive co-occurrence because the weight function in Eq. (7) gives w(0)=0 for alpha>0. The stated motivation for using visual co-occurrence is that textually similar pairs such as \"indoor\" and \"outdoor\" have zero visual co-occurrence and should be suppressed, but the objective never presents such incompatible pairs to M. Thus, the learned mapping has no pressure to output low values for non-co-occurring pairs, and the improvement of CoCAZSL over TexCAZSL in Table 2 (AUC 0.69 vs 0.65) cannot be attributed to the \"visual compatibility\" mechanism as described. Please either include zero-co-occurrence pairs with a non-zero penalty in Eq. (6), or provide direct evidence (e.g., predicted co-occurrence for held-out known-known pairs) that M extrapolates to incompatible unseen pairs.","section":"§3.2.2, Eqs. (6)-(7)"},{"comment":"The reported gains of CoCAZSL over TexCAZSL (AUC 0.69 vs 0.65; label-based AP 0.27 vs 0.24) are the central empirical evidence for the contribution, but the paper gives no variance or significance information across the 50 random splits. Since the difference is modest and the baselines are also within 0.02-0.03 of each other, please report per-split mean±std or paired tests across the 50 splits to show that the improvement is systematic rather than due to a few favourable splits.","section":"Table 2, §4.1.2"},{"comment":"The claim that the zero-shot model \"beats the fully supervised Linear SVM with ViF feature in AUC (87.22 vs 85.00)\" compares CoCAZSL using ITF with a supervised baseline using ViF. The same-feature supervised ITF baseline achieves 98.72 AUC, which is substantially higher. Please reframe the claim to acknowledge the feature mismatch and, if possible, add a supervised ViF baseline or a zero-shot model using the same features to make the comparison apples-to-apples.","section":"§4.2, Table 3"}],"minor_comments":[{"comment":"The second line \"p( ˜y ˜p|x)\" appears to be a typo; it should presumably read p(\\tilde{y}_{\\tilde{p}}|x).","section":"§3.1, Eq. (1)"},{"comment":"The gradient in Eq. (9) uses f(c_ij) but the loss in Eq. (6) uses w(c_ij); please make the notation consistent.","section":"§3.2.2, Eq. (9)"},{"comment":"The parameter setting \"λ = 1−3\" should likely be \"λ = 10^{-3}\"; please clarify.","section":"§4.1.1"},{"comment":"The supervised baseline is cited as \"[18]\" in the table, but the text refers to \"the best performance of [19]\"; the citation appears inconsistent.","section":"Table 3"},{"comment":"The caption contains a typo: \"Importantce\" should be \"Importance\".","section":"Fig. 9"},{"comment":"In the phrase \"Indirect Attribute Predici ton (IAP)\", \"Predici ton\" should be \"Prediction\".","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be formatted as a book chapter and contains a number of self-citations in the related work, but the central experimental pipeline is independent of prior work and the zero-shot formulation does not appear circular. The two technical issues in the model formulation (the mis-normalized conditional in Eq. (8) and the absence of negative co-occurrence pairs in Eq. (6)) are the main barriers; if the authors can fix these and provide variance-aware experimental validation, the contribution could be publishable. The lack of released code and data is a limitation but not a reason for rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is the first multi-label zero-shot learning study on crowd behavior, and the transfer experiment to violence detection is a genuine proof of concept. The core idea is sound in spirit: learn a bilinear map from word-vector pairs to visual co-occurrence on known attributes, then reuse it for known-novel pairs. The WWW experiments show consistent gains over standard ZSL baselines on label-based metrics (AUC 0.69 vs 0.65 for the best prior), and the violence transfer beats a supervised ViF baseline. That is real evidence the approach has legs.\n\nBut there are two soft spots that a referee should push on. The first is Eq. (8). As written, the denominator sums over the known attributes p, not over the novel attribute q, so the quantity is not the conditional probability p(y*_q | y_p) that the paper claims. It is an attention weight over p for each q. The marginalization in Eq. (1) might still be a reasonable heuristic, but the paper should say so and normalize correctly if it wants to call it a conditional.\n\nThe second, more serious issue is the training objective in Eq. (6). The weight function w(c_ij) is zero whenever c_ij = 0, so the regression never sees a pair of attributes that do not co-occur visually. The paper's own motivating example is that \"indoor\" and \"outdoor\" have high text similarity but zero visual co-occurrence. Because those zero pairs are excluded from the loss, M has no pressure to suppress them. At test time, the model can therefore emit high compatibility for semantically similar but visually incompatible pairs. The stress-test note is right about this. The method's success on the datasets suggests M extrapolates reasonably for the specific novel attributes used, but the claimed \"visual compatibility correction\" over text is not actually learned from negative evidence. This is a modeling choice, not a derived result, and it should be acknowledged.\n\nThe violence comparison also overstates. Comparing zero-shot CoCAZSL (87.22 AUC, ITF features) to supervised ViF (85.00) is not an apples-to-apples comparison; the supervised ITF baseline is 98.72. The paper should present that context honestly.\n\nAll of this is addressable. The framing is clear, the dataset split is sensible, and the empirical gains are plausible. I would send this to a referee, with the expectation that the normalization issue and the negative-evidence gap would need a response. The paper is for researchers working on zero-shot learning or crowd analysis, and it does give a useful new evaluation setup. Not a breakthrough, but a solid contribution with an honest revision.","headline":"First multi-label ZSL for crowd behavior with a useful violence transfer, but the conditional probability is mis-normalized and the learned co-occurrence map never sees negative pairs.","tokens_in":20389,"tokens_out":3476,"would_cite":false,"duration_ms":34045,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Crowd behaviors with zero training videos become recognizable","keywords":["zero-shot learning","crowd behavior recognition","multi-label classification","attribute co-occurrence","word-vector embeddings","video surveillance","violence detection","bilinear mapping"],"falsifier":"Annotate a fresh set of crowd videos for the held-out novel attributes, compute their empirical co-occurrence with known attributes, and compare these counts against the conditional probabilities predicted by $M$; if the rank correlation is near zero or negative, the extrapolation at the core of the paper collapses.","tokens_in":19418,"feed_emoji":"👥","tokens_out":5810,"duration_ms":50311,"temperature":0.7,"pith_summary":"This paper tries to establish that a video system can recognize crowd behaviors it has never been trained on, such as violence, by combining confidence scores for known behaviors with predicted relations between known and novel behaviors. Since crowd videos are inherently multi-label, the paper argues zero-shot recognition should exploit co-occurrence context rather than treat each behavior independently. Its proposal is a bilinear map $M$, learned from word-vector embeddings and annotated visual co-occurrence of known attribute pairs, that predicts co-occurrence for pairs involving an unseen attribute. If correct, novel-behavior detection becomes possible without new annotations: the paper reports a zero-shot AUC of 0.69 on the WWW dataset versus 0.65 for the best prior zero-shot baseline, and a transferred violence detector reaching 87.22 AUC, above a supervised baseline's 85.00.","feed_headline":"Unseen crowd behaviors recognized with no training videos","feed_subtitle":"Predicting which known and novel attributes co-occur detects violence with 87.22 AUC, above a supervised baseline.","key_machinery":"The load-bearing object is the bilinear map $M$ of Eq. (6), a matrix that takes a pair of word vectors $(v_i, v_j)$ and outputs the expected log visual co-occurrence $\\log c_{ij}$ of the two attributes. It is fit by weighted ridge regression against the co-occurrence matrix $C = Y_S Y_S^\\top$ computed from multi-label annotations of known attributes, with a weighting function that down-weights rare pairs. The same $M$ is then used in Eq. (8) to score conditional probabilities $p(y^*_q \\mid y_p)$ for pairs that include a novel attribute, and those conditionals are marginalized over known-attribute confidences in Eq. (1) to predict novel labels. The map is the mechanism that lets word-vector semantics be corrected by and aligned with visual context, and then extrapolated to unseen attribute pairs.","core_discovery":"On its own terms, the paper claims that multi-label zero-shot crowd behavior recognition can be solved by marginalizing known-attribute predictions through estimated known-to-novel attribute relations. Its model, CoCAZSL, learns a probabilistic classifier for known attributes, then forms conditional probabilities between novel and known attributes from a bilinear mapping $M$ that predicts visual co-occurrence from word vectors. Setting $M$ to the identity recovers text-only relations; learning $M$ from annotated co-occurrence corrects cases where linguistic similarity mismatches visual context, such as indoor and outdoor. Experimentally, the paper reports that this context-aware marginalization improves label-based AUC and AP over four existing zero-shot models on the WWW dataset, and that transferring the learned relations to violence detection yields AUC 87.22, above the supervised ViF baseline's 85.00. The claim is that contextual co-occurrence, transferred through a learned word-pair map, is what makes zero-shot crowd behavior recognition work.","pith_inferences":["A natural extension the paper leaves implicit is learning separate co-occurrence maps for the 'where', 'who', and 'why' attribute groups, which could test whether group-specific context extrapolates better than a single global map.","The same word-pair to co-occurrence transfer could apply to other multi-label zero-shot problems, such as tagging images with novel objects or actions, whenever word vectors and partial label co-occurrence are available.","The indoor/outdoor failure of text-only relations suggests a general caution: any text-based semantic prior should be calibrated against visual annotation statistics before being trusted for visual prediction."],"forward_implications":["Zero-shot crowd attribute prediction is feasible: on the WWW dataset all tested models beat random guessing, and the context-aware model reports the highest label-based AUC and AP.","Adding visual co-occurrence context to a text-only semantic relation improves prediction: CoCAZSL reaches 0.69 AUC versus 0.65 for TexCAZSL.","The learned relations transfer across domains: without any violence training examples, CoCAZSL reaches 87.22 AUC on Violent Flow, above the supervised ViF baseline's 85.00.","Multi-label context is what supplies the missing statistics: because novel attributes cannot have annotated co-occurrence, the bilinear map generates those statistics from word pairs and known-attribute annotations."],"supporting_citations":[{"why":"supplies the WWW crowd video dataset, its 94 attributes, and the multi-label annotations used to train and evaluate the model","marker":"[41]"},{"why":"provides the DAP/IAP probabilistic attribute-transfer formulation that Eq. (1) adapts to multi-label zero-shot prediction","marker":"[23]"},{"why":"supplies the skip-gram word-vector embeddings used to represent attribute names and to form semantic relations","marker":"[29]"},{"why":"defines the multi-label zero-shot baselines ExDAP and DMP that the paper compares against","marker":"[12]"},{"why":"provides the Violent Flow dataset and the supervised ViF baseline used in the transfer violence-detection experiment","marker":"[19]"},{"why":"supplies the co-occurrence-statistics idea and the weighting function w(c_ij) used in Eq. (7)","marker":"[28]"},{"why":"provides the text-corpus semantic inter-class relation approach that the text-only TexCAZSL variant builds on","marker":"[14]"},{"why":"the bilinear word-co-occurrence model that the learned map M in Eq. (6) is related to","marker":"[33]"},{"why":"provides improved dense trajectories with Fisher-vector encoding, the visual feature representation used for all video experiments","marker":"[49]"}],"fun_headline_variants":["Zero-shot model reads crowd behavior without video labels","Co-occurrence predicts unseen crowd behaviors, even violence","No training videos needed: model recognizes novel crowd acts","Zero-shot crowd analysis: context predicts violence and more","Learning crowd behavior co-occurrence enables zero-shot recognition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The single bilinear rule learned from pairs of known attributes keeps predicting co-occurrence accurately for pairs that include a novel attribute.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot model reads crowd behavior without video labels","Co-occurrence predicts unseen crowd behaviors, even violence","No training videos needed: model recognizes novel crowd acts","Zero-shot crowd analysis: context predicts violence and more","Learning crowd behavior co-occurrence enables zero-shot recognition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000752,"raw_usage":{"total_tokens":3385,"prompt_tokens":1021,"completion_tokens":2364,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":2299}},"tokens_in":637,"tokens_out":2364,"duration_ms":15436,"temperature":1.0,"reasoning_tokens":2299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:02:34.647235+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Annotate a fresh set of crowd videos for the held-out novel attributes, compute their empirical co-occurrence with known attributes, and compare these counts against the conditional probabilities predicted by $M$; if the rank correlation is near zero or negative, the extrapolation at the core of the paper collapses.","supporting_citations":[{"cited_title":"Deeply learned attributes for crowded scene understanding","cited_arxiv_id":null,"evidence_quote":"supplies the WWW crowd video dataset, its 94 attributes, and the multi-label annotations used to train and evaluate the model"},{"cited_title":"Lampert, H","cited_arxiv_id":null,"evidence_quote":"provides the DAP/IAP probabilistic attribute-transfer formulation that Eq. (1) adapts to multi-label zero-shot prediction"},{"cited_title":"Distributed Rep- resentations of Words and Phrases and their Compositionality","cited_arxiv_id":null,"evidence_quote":"supplies the skip-gram word-vector embeddings used to represent attribute names and to form semantic relations"},{"cited_title":"Trans- ductive Multi-Label Zero-shot Learning","cited_arxiv_id":null,"evidence_quote":"defines the multi-label zero-shot baselines ExDAP and DMP that the paper compares against"},{"cited_title":"Violent flows: Real-time detection of violent crowd behavior","cited_arxiv_id":null,"evidence_quote":"provides the Violent Flow dataset and the supervised ViF baseline used in the transfer violence-detection experiment"},{"cited_title":"COSTA: Co-occurrence statistics for zero-shot classification","cited_arxiv_id":null,"evidence_quote":"supplies the co-occurrence-statistics idea and the weighting function w(c_ij) used in Eq. (7)"},{"cited_title":"Exploring semantic inter-class relationships (sir) for zero-shot action recognition","cited_arxiv_id":null,"evidence_quote":"provides the text-corpus semantic inter-class relation approach that the text-only TexCAZSL variant builds on"},{"cited_title":"Glove: Global vectors for word representation","cited_arxiv_id":null,"evidence_quote":"the bilinear word-co-occurrence model that the learned map M in Eq. (6) is related to"},{"cited_title":"A robust and efficient video representation for action recog- nition","cited_arxiv_id":null,"evidence_quote":"provides improved dense trajectories with Fisher-vector encoding, the visual feature representation used for all video experiments"}],"review_version":1}