{"id":"fdbd53bf-56aa-4a2a-a98b-3496ee70ef47","arxiv_id":"1908.08527","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Word embeddings trained on visual co-occurrences from Visual Genome and ImageNet complement GloVe and improve several vision-language tasks, while random embeddings match learned ones on those supervised tasks.","lead":"This paper learns word embeddings from visual co-occurrences, using which objects and attributes appear together in annotated images, and shows these vectors improve several vision-language tasks when combined with GloVe. It also reports that random word vectors perform nearly as well as learned ones on supervised vision-language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tab. 5 downstream gains are small and unreplicated; without significance tests, the claim that ViCo outperforms GloVe on all tasks may be seed noise.","rationale":"The reader's weakest assumption concerns synset expansion and annotation completeness in the co-occurrence counting. That is a real limitation, but the paper's w/o WordNet ablation (Tab. 3) directly tests the WordNet component and still finds gains over GloVe, and the residual annotation-incompleteness concern is an inherent property of any visual grounding dataset rather than a hidden flaw in the method. The more load-bearing issue is the lack of statistical support for the paper's broadest downstream claim. The effect sizes in Table 5 are very small, no variance estimates are given, and the random-embedding result shows the evaluated vision-language models are largely insensitive to word embeddings, making it plausible that the observed ViCo advantages are noise. This does not require rejecting the paper; the clustering and zero-shot analyses provide independent support for the core idea. But the universal downstream claim needs a significance check before it can be taken at face value. The reader already noted small effect sizes in the rationale, so this is partial agreement rather than a new objection. Since the appropriate remedy is additional experiments and statistical reporting, the conditional verdict stands.","tokens_in":14240,"tokens_out":8610,"duration_ms":93192,"concrete_test":"Re-run the five downstream tasks from Sec. 4.3 with at least 5 random seeds for GloVe, GloVe+random, and GloVe+ViCo, freezing embeddings exactly as in the paper and keeping all task-specific hyperparameters fixed. For each metric, compute paired per-seed differences between GloVe+ViCo and GloVe, and report a bootstrap 95% confidence interval or a Wilcoxon signed-rank p-value. If fewer than three of the five tasks show a positive median difference with a confidence interval excluding zero, the 'gains on all tasks' claim should be weakened or treated as unverified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The broadest empirical claim—\"Augmenting GloVe with our embeddings yields gains on all tasks\" (abstract, Sec. 4.3)—rests on Table 5 differences that are mostly 0.1–0.3 absolute points. For example, VQA Overall is 67.7 vs 67.5, VQA Other 58.4 vs 58.3, RefCOCO+ TestA 75.5 vs 75.3, BLEU-4 0.291 vs 0.290, and SPICE 0.168 vs 0.167. No standard errors, confidence intervals, or significance tests are reported for any row of Table 5. The paper's own random-embedding condition shows that the same supervised vision-language models are nearly insensitive to word embeddings, which lowers the prior that such small deltas are real rather than run-to-run noise. If those deltas are within noise, the universal downstream improvement claim is not established, and the conclusion that visual co-occurrences help vision-language tasks is correspondingly weaker. This concern is separate from the synset-expansion issue: the w/o WordNet ablation in Table 3 partially addresses the WordNet contribution, but nothing in the paper addresses the reliability of the small downstream differences.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ViCo, a method for learning word embeddings from visual co-occurrences. It formulates a multi-task extension of GloVe's log-bilinear objective (Eq. 2) in which a single shared word vector is projected through type-specific linear transforms into separate spaces for four co-occurrence types: object-attribute, attribute-attribute, context, and object-hypernym, with counts derived from VisualGenome and ImageNet annotations. The authors evaluate GloVe+ViCo against GloVe, random vectors, and vis-w2v on unsupervised clustering over manually annotated word categories, a zero-shot-like CIFAR-100 transfer protocol, five downstream tasks (discriminative attributes, caption-image retrieval, VQA, referring expression comprehension, image captioning), and qualitative analogies. The main claims are that GloVe+ViCo outperforms GloVe across these evaluations and that random embeddings perform comparably to learned embeddings on supervised vision-language tasks.","tokens_in":14564,"tokens_out":6972,"duration_ms":67103,"significance":"If the results hold, the paper offers a simple and scalable way to inject visual statistics into word embeddings and provides one of the first systematic demonstrations that frozen word embeddings have limited impact in several supervised vision-language models. The multi-task log-bilinear formulation is clean and the w/o WordNet ablation is a useful control. The clustering and zero-shot-like results are more substantial than the downstream results, which rest on small, unreplicated differences. The finding about random embeddings is interesting and potentially important, but the evidence for it is entangled with the same statistical reliability issues.","major_comments":[{"comment":"The central claim that GloVe+ViCo outperforms GloVe on all downstream tasks rests on Table 5 differences that are mostly 0.1-0.3 absolute points, with no standard errors or significance tests. For example, VQA Overall is 67.7 vs 67.5, VQA Other is 58.4 vs 58.3, RefCOCO+ TestA is 75.5 vs 75.3, BLEU-4 is 0.291 vs 0.290, and SPICE is 0.168 vs 0.167. The image captioning row is particularly concerning: the random-embedding row alone reports BLEU-4 0.296, CIDEr 0.910, and SPICE 0.170, which are all higher than the corresponding GloVe+ViCo values (0.291, 0.894, 0.168), suggesting that captioning metrics can move around by amounts larger than the claimed gains. The authors should report multiple random seeds or bootstrap confidence intervals for every row of Table 5, or explicitly tone down the all-task claim. This is load-bearing because the abstract and conclusion claim gains on all tasks.","section":"Sec. 4.3, Table 5"},{"comment":"The zero-shot-like protocol is described as 'slightly different' from typical zero-shot because images of unseen CIFAR-100 classes are used to compute the word co-occurrences used for embedding training. The paper then concludes that GloVe+ViCo 'generalizes to unseen classes better than GloVe.' This is not a genuine zero-shot transfer test, since the unseen class names and their visual co-occurrence statistics are available to the embedding. I do not see equation-level circularity, but the wording overstates what is measured. The authors should either re-frame this as a novel-class classification experiment with supervised embeddings, or add an experiment in which the embeddings are trained without any annotations from the unseen classes. At minimum, the limitations paragraph should state explicitly that the zero-shot result does not test whether ViCo can transfer to classes absent from its training data.","section":"Sec. 4.2"},{"comment":"The co-occurrence counting expands each synset to all of its words, so a region annotated with object synset {man, person, adult, mammal} and attribute {smiling} contributes counts for (man, smiling), (person, smiling), (adult, smiling), and (mammal, smiling). This means WordNet structure is injected into the co-occurrence statistics, not only through the explicit Object-Hypernym type but also through synonym expansion in the other types. The w/o WordNet ablation in Table 3 is a good control and shows that WordNet is not the sole contributor, but it is reported only for the clustering analysis. Since the zero-shot-like and downstream claims are among the paper's main contributions, the authors should provide w/o WordNet variants for those evaluations as well, or justify why the clustering ablation is sufficient to bound the effect.","section":"Sec. 3.3, Table 3"}],"minor_comments":[{"comment":"The sentence 'The model is trained with a batch size of0.01 for 50 epochs' appears to contain a typo ('of0.01'); please clarify the batch size and the learning rate.","section":"Sec. 4.2, Learning"},{"comment":"The best model is selected using seen-class accuracy on the test set; a held-out validation split would be cleaner, since the test set is also used to compute the unseen-class accuracy.","section":"Sec. 4.2, Model Selection"},{"comment":"The text says results are averaged over four runs and that 7x4x4=112 models were trained; the description of how the 112 models are distributed across splits, embeddings, and runs is easy to misread and should be made explicit.","section":"Sec. 4.2, Fig. 5"},{"comment":"The abstract states that random embeddings 'perform comparably' to learned embeddings on all supervised vision-language tasks, while the Table 5 caption says GloVe+ViCo outperforms random 'for all tasks except Image Captioning.' These statements are consistent only if 'comparably' is interpreted loosely; the wording should be tightened to avoid the appearance that random embeddings match learned embeddings everywhere.","section":"Abstract and Table 5 caption"},{"comment":"The analogy evaluation is based on only 30 hand-picked pairings and is reported as a count of correct answers with no candidate-set construction details; it should be presented as qualitative evidence rather than a rigorous evaluation.","section":"Sec. 4.4"},{"comment":"The weighting function f(X) is set to 1 for all co-occurrence counts, so rare and frequent co-occurrences contribute equally in the first term; the paper should state whether this choice was evaluated, since GloVe's original weighting is designed to downweight rare pairs.","section":"Sec. 3.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a vision-language venue and the core method is sound and interesting. The main risk is overclaiming: the downstream gains in Table 5 are small and unreplicated, and the zero-shot-like experiment is not a genuine zero-shot test. These issues are fixable by adding statistical tests, re-running with multiple seeds, and adjusting the claims, so I am recommending major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe short version: this is a solid, genuinely useful paper, and the part I trust most is the part that will probably get it into trouble: the finding that random embeddings match learned embeddings on supervised vision-language tasks. The multi-task log-bilinear extension of GloVe is clean, and extracting co-occurrences directly from visual annotations bypasses the image-feature bottleneck of vis-w2v and ViEW. The clustering and zero-shot-like analyses show real, non-trivial gains for GloVe+ViCo over GloVe, and the w/o WordNet ablation largely answers the worry that the gains are just WordNet structure.\n\nWhere it gets softer is Table 5. The downstream gains are mostly 0.1–0.3 absolute points, with no error bars or significance tests, and the paper itself shows the same models are nearly insensitive to the choice of word embedding. So the claim \"gains on all tasks\" is not actually established; it could be seed noise. The zero-shot-like protocol is also not true zero-shot, as they acknowledge: unseen CIFAR-100 classes appear in the co-occurrence data used to train the embeddings. That limits the strength of the generalization claim, but the authors are upfront about it. The synset expansion is a real concern, but the w/o WordNet ablation shows ViCo still beats GloVe, so it's not fatal.\n\nOverall: I'd send this out for review. The method is well-specified, the evaluation is broad, and the random-embedding result is a valuable negative result for the vision-language community. The main revision I'd want is to either get error bars/significance on the downstream numbers or soften the claim to \"no evident harm and occasional small gains.\" Also, releasing the embeddings would make the contribution much more useful.\n\nRecommendation: accept for review with the expectation of a solid revision, not a desk reject.\n\nBest,","headline":"Solid, useful paper with a genuine negative result about random embeddings; the downstream gains are too small and unreplicated to carry the 'gains on all tasks' claim.","tokens_in":15004,"tokens_out":1927,"would_cite":true,"duration_ms":19218,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Visual co-occurrences in annotated images encode word meanings that text-only embeddings miss.","keywords":["visual co-occurrences","word embeddings","multi-task log-bilinear model","GloVe","zero-shot generalization","vision-language tasks","Visual Genome","ImageNet"],"falsifier":"Train ViCo using only directly attested region labels with no synonym expansion and no WordNet hypernyms, and compare its clustering and zero-shot performance against GloVe; if the gains over GloVe disappear, the reported advantage comes from WordNet and annotation structure, not from visual co-occurrence itself. More narrowly, replacing each synset's words with its single most frequent surface form should preserve visual signal while removing synonym-expansion effects.","tokens_in":14086,"feed_emoji":"🖼️","tokens_out":5714,"duration_ms":56209,"temperature":0.7,"pith_summary":"This paper tries to establish that word embeddings can be learned from visual co-occurrence—two words co-occur visually when both apply to the same image or region—and that these embeddings capture visual distinctions that text corpora obscure. From object and attribute annotations in Visual Genome and ImageNet, the authors count four kinds of co-occurrence: object-attribute, attribute-attribute, context, and object-hypernym. A multi-task log-bilinear model learns one compact vector per word shared across all four co-occurrence types. The paper claims that concatenating ViCo with GloVe outperforms GloVe alone on unsupervised clustering, zero-shot-like transfer to unseen CIFAR-100 classes, and all five downstream applications tested. It also claims that on supervised vision-language tasks, random word vectors perform nearly as well as learned ones—a result the authors present as contrary to conventional wisdom.","feed_headline":"Visual co-occurrence embeddings beat GloVe alone on every test","feed_subtitle":"Training word vectors from object-attribute co-occurrences in annotated images adds visual signal absent from text.","key_machinery":"The load-bearing mechanism is a multi-task log-bilinear objective. A shared word vector $w_i\\in\\mathbb{R}^d$ is projected through a co-occurrence-type-specific transformation $\\varphi_t$ into a dedicated space, where a GloVe-style squared loss approximates the log of the observed count $X^t_{ij}$ plus a hinge-style penalty for zero-count pairs. Four count matrices corresponding to object-attribute, attribute-attribute, context, and object-hypernym co-occurrences are computed by expanding synset annotations across all member words. This formulation lets one compact embedding encode several senses of relatedness at once; the paper shows a linear $\\varphi_t$ matches the concatenation of separately trained embeddings at lower dimension.","core_discovery":"ViCo's central claim is that visual co-occurrence statistics carry genuine word-meaning signal beyond text: words that describe the same visible stuff (salad and chopped, glass and clear, fork and spoon) end up closer in the learned space even when they rarely co-occur in prose. The paper demonstrates this by training a multi-task extension of GloVe's log-bilinear objective on four co-occurrence matrices, then showing that GloVe+ViCo beats GloVe alone on clustering, zero-shot generalization, and five downstream applications. A separate claim, supported by the same experiments, is that supervised vision-language models are largely insensitive to the choice of word embeddings: random vectors match learned ones on all four vision-language tasks, while the word-only discriminative-attributes task still strongly favors learned embeddings.","pith_inferences":["If visual co-occurrence signal is as strong as reported, similar counting over other dense annotation sources (video captions, pixel-level segmentations, image tags) could produce embeddings for a much larger vocabulary without synset expansion.","The near-parity of random and learned embeddings on vision-language tasks suggests those benchmarks may be dominated by visual features and model architecture; a clean test would sweep vocabulary coverage or embedding quality while holding everything else fixed.","The type-specific projections could be reused as a cheap sense-disentangling tool: querying which projection gives high similarity reveals whether two words are related by appearance, function, hypernymy, or context."],"forward_implications":["Using ViCo in place of or alongside GloVe should improve any task where word similarity must reflect visible properties, such as attribute-based retrieval or zero-shot object recognition.","Because the multi-task model shares one compact embedding across co-occurrence types, adding new co-occurrence sources need not inflate embedding dimensionality.","For supervised vision-language models, pretrained word embeddings may be replaceable by random vectors without much loss, freeing capacity or simplifying pipelines.","Word embeddings can be obtained from image annotation alone, which matters for domains or languages with little text but available visual labels."],"supporting_citations":[{"why":"Supplies the log-bilinear objective that ViCo extends and the GloVe baseline used throughout.","marker":"[37]"},{"why":"Provides ImageNet synset labels and WordNet ancestors used for Object-Hypernym co-occurrence counts.","marker":"[8]"},{"why":"The prior visually grounded word embedding (vis-w2v) used as the baseline comparison in clustering and combination experiments.","marker":"[18]"},{"why":"Provides CIFAR-100 categories and splits used in the zero-shot-like transfer evaluation.","marker":"[20]"},{"why":"Defines the discriminative attributes task, the word-only downstream benchmark that distinguishes learned from random embeddings.","marker":"[19]"}],"fun_headline_variants":["Visual co-occurrence embeddings beat text-only GloVe on all tasks","Word vectors from images outperform text-based GloVe","Random vectors match learned ones on vision-language tasks","Visual co-occurrence signals improve word embeddings beyond text","GloVe plus visual co-occurrence vectors wins every benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The counting pipeline assumes that every synonym listed for an annotated concept truly applies to every image region carrying that concept label and that the region annotations are exhaustive; if labels are missing or synonyms are used loosely, the co-occurrence counts encode annotation and WordNet structure rather than visual co-occurrence.","fun_headline_variants_meta":{"raw":{"variants":["Visual co-occurrence embeddings beat text-only GloVe on all tasks","Word vectors from images outperform text-based GloVe","Random vectors match learned ones on vision-language tasks","Visual co-occurrence signals improve word embeddings beyond text","GloVe plus visual co-occurrence vectors wins every benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000226,"raw_usage":{"total_tokens":1431,"prompt_tokens":874,"completion_tokens":557,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":476}},"tokens_in":490,"tokens_out":557,"duration_ms":6035,"temperature":1.0,"reasoning_tokens":476,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:36:41.096058+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train ViCo using only directly attested region labels with no synonym expansion and no WordNet hypernyms, and compare its clustering and zero-shot performance against GloVe; if the gains over GloVe disappear, the reported advantage comes from WordNet and annotation structure, not from visual co-occurrence itself. More narrowly, replacing each synset's words with its single most frequent surface form should preserve visual signal while removing synonym-expansion effects.","supporting_citations":[{"cited_title":"Glove: Global vectors for word representation","cited_arxiv_id":null,"evidence_quote":"Supplies the log-bilinear objective that ViCo extends and the GloVe baseline used throughout."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior visually grounded word embedding (vis-w2v) used as the baseline comparison in clustering and combination experiments."},{"cited_title":"Learning multiple layers of features from tiny images","cited_arxiv_id":null,"evidence_quote":"Provides CIFAR-100 categories and splits used in the zero-shot-like transfer evaluation."},{"cited_title":"Semeval-2018 task 10: Capturing discriminative attributes","cited_arxiv_id":null,"evidence_quote":"Defines the discriminative attributes task, the word-only downstream benchmark that distinguishes learned from random embeddings."}],"review_version":1}