{"id":"ef292c59-2882-40be-acf4-213e65c00922","arxiv_id":"2507.07102","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In controlled visual experiments, compositional generalization improves with concept diversity rather than dataset size, and linearly factored representations would in principle need only two observed combinations per concept value.","lead":"This paper asks whether adding more training images makes vision models combine concepts such as color and shape in ways they never saw together. Controlled experiments show that what matters is concept diversity, not raw data scale, and that storing concepts as additive vector parts would in principle allow perfect generalization from two examples per concept.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.1/B.7 is not proven as written: the Eq. (25) system is rank-deficient, the proof assumes U1⊥U2 without stating it, and the stated joint-span dimension 2n−1 contradicts the proof's own dimensions. The k=2 sufficiency claim needs repair or extra hypotheses.","rationale":"I read the paper as making two separable claims: (i) an empirical finding that compositional generalization tracks combinatorial diversity rather than ID data quantity, with three-phase emergence of linear structure, and (ii) a theoretical claim that linear factorization makes k=2 per concept sufficient for perfect generalization. Claim (i) is well supported: the controlled sweeps over n and k, the fixed-dataset-size diversity comparison (Fig. 3b), the dataset-size null result (Fig. 4), and the representation metrics (Fig. 5) are consistent and reproducible. Claim (ii) is the load-bearing theoretical anchor for the abstract's 'we prove' statement and for the pretrained-model interpretation in Section 5.1, but the proof of Proposition B.7 is not valid as written: the linear system is rank-deficient, the orthogonal projection step uses an unstated and non-implied assumption, and the stated dimension 2n−1 contradicts the proof's own 2n−2. These are precise, fixable flaws rather than evidence against the empirical core; adding zero-sum constraints and an explicit connectedness/orthogonality assumption (or proving separability without orthogonality) would repair the proof. The reader's weakest-assumption analysis already identified the same rank and orthogonality problems, and I agree. Since the flaws are localized to the proof and the conditional verdict already reflects them, no verdict change is needed; the paper should be accepted only if the proof is corrected or the theoretical claim is downgraded to a conjecture.","tokens_in":25003,"tokens_out":16022,"duration_ms":192500,"concrete_test":"Run a numerical counterexample search for Proposition 4.1: set n=4 in R^7, choose zero-sum u_i and zero-sum v_j with U1∩U2={0} but u_i and v_j non-orthogonal (e.g., v_j = R u_j for a small rotation), generate the 8 cyclic pair means, and apply the paper's Algorithm 1 both with and without enforcing zero-sum. Then check whether the projection classifier achieves 100% on the 12 unseen pairs and whether the 8×8 design matrix has rank 7. If the non-zero-sum recovery fails while the zero-sum version succeeds, the proof can be fixed by adding Lemma B.4; if the projection classifier fails even with exact recovery, then the unstated orthogonality assumption is essential and the proposition is false as stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical claim is Proposition 4.1 (proved as B.7): with linearly factored embeddings, k=2 combinations per concept value suffice for a perfectly generalizing linear classifier. The proof has three concrete gaps. First, the 2n×2n system in Eq. (25) is not full rank: for the cyclic split (i,i) and (i,i+1 mod n), the sum of the n diagonal rows equals the sum of the n shifted rows, so the row rank is at most 2n−1. The text asserts 'these form distinct patterns that ensure linear independence', which is false. This can likely be repaired by adding the zero-sum constraints of Lemma B.4, which remove the common constant-shift null vector, but that repair is not in the proof. Second, Part 3 assumes that any vector in the joint span decomposes into u1⊥u2 and that orthogonal projections onto U1 and U2 give the needed classifiers. Orthogonality is not implied by Definition 3.2 or by U1∩U2={0}; non-orthogonal subspaces can intersect trivially, and then a linear functional that ignores concept 2 need not exist. Third, Proposition 4.1 states the joint span has dimension 2n−1, while the proof itself has dim U1=n−1, dim U2=n−1, and U1∩U2={0}, forcing dimension 2n−2; the statement and proof are arithmetically inconsistent. The proposition also does not specify which k=2 combinations are observed; recovery only goes through for a connected 2-regular split such as the cyclic one used in the experiments. None of this undermines the controlled empirical diversity-over-scale results, but the abstract's 'we prove this structure is key to efficiency' is not supported by the proof as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether scaling training data volume improves visual compositional generalization. Using a controlled (n,k) framework over synthetic datasets (dSprites, Shapes3D, FSprites, Colored-MNIST, PUG-Animal, plus MPI3D), the authors vary the number of concept values n and the number of observed combinations k per value. Their empirical findings are that in-distribution data scale alone does not close the ID-OOD accuracy gap, whereas increasing concept diversity and combination coverage improves zero-shot accuracy on unseen combinations. They also report a three-phase transition in learned representations: spurious features, decodable but non-linear features, and finally linearly factored, mutually orthogonal concept subspaces. Section 4.3 introduces Proposition 4.1 (proved as Proposition B.7) claiming that under linear factorization, observing k=2 combinations per concept value suffices to learn a linear classifier that perfectly generalizes to all unseen combinations. The paper then evaluates pretrained models (DINO, CLIP, ImageNet ResNet) through linear-factorization recovery and probing, finding partial but imperfect compositional structure. The central empirical diversity-over-scale claim is supported by controlled experiments across several datasets; however, the theoretical proposition as stated is not established by the proof in Appendix B.","tokens_in":25394,"tokens_out":3808,"duration_ms":44861,"significance":"If the empirical claims hold, the paper provides a useful, controlled counterpoint to the broad scaling narrative: for compositional generalization, combinatorial diversity of concept combinations can matter more than raw data volume. The dataset-grid methodology, the consistency of the three-phase feature-learning result across five synthetic datasets, and the public release of code are concrete strengths. The theoretical result, if repaired, would add a crisp mechanism for why linear factorization enables few-shot compositional generalization. The evaluation of pretrained models extends the findings beyond from-scratch training. The significance for the compositional generalization and representation learning community is moderate to high. However, the current proof of Proposition B.7 contains rank, orthogonality, and dimension issues that mean the stated theoretical claim is not yet supported.","major_comments":[{"comment":"The proof claims that the 2n x 2n linear system in Eq. (25) is full rank because the diagonal and cyclic rows 'form distinct patterns that ensure linear independence.' This is false for the cyclic split used in the paper. For the diagonal pairs (i,i) and shifted pairs (i,i+1 mod n), the sum of the n diagonal rows equals the sum of the n shifted rows, so the row rank is at most 2n-1 and the system is underdetermined. The proof does not use the zero-sum constraints of Lemma B.4 to remove the null direction. Consequently, the k=2 sufficiency claim in Proposition 4.1 is not proven as written.","section":"Appendix B, Proposition B.7, Eq. (25)"},{"comment":"Part 3 of the proof assumes that any vector in the joint span of the two concept subspaces decomposes uniquely as u1 + u2 with u1 perpendicular to u2, and that orthogonal projections onto U1 and U2 give optimal classifiers. Orthogonality of the two concept subspaces is not part of Definition 3.2 and does not follow from U1 intersecting U2 only at {0}; non-orthogonal subspaces can intersect trivially. Without a stated orthogonality or annihilator assumption, the existence of a linear classifier for concept 1 that ignores concept 2 is not guaranteed. The empirical orthogonality measurements in Section 4.2 motivate the assumption, but it is not stated in the proposition.","section":"Appendix B, Proposition B.7, Part 3"},{"comment":"The proposition states that the joint span of {u_{c_1}} and {u_{c_2}} has dimension 2n-1, while the proof in Part 3 asserts dim(U1)=n-1, dim(U2)=n-1, and U1 ∩ U2 = {0}, which forces the joint span to have dimension 2n-2. These statements are arithmetically inconsistent unless the proposition refers to uncentered concept vectors and the proof to centered, zero-sum vectors, but that distinction is not made. The statement and proof need to be reconciled, for example by stating which representation (raw or centred) is used for each dimension count.","section":"Proposition 4.1 / Proposition B.7, dimension statement"},{"comment":"The proposition does not specify which k=2 combinations per concept value are observed. The proof only goes through for a specific connected 2-regular split, namely the cyclic split in Eq. (15). For an arbitrary set of two combinations per value, the design matrix in Eq. (25) need not be identifiable even after adding zero-sum constraints. The statement should either restrict the observation pattern explicitly or justify that the result holds for any connected 2-regular split.","section":"Proposition 4.1, generality of k=2 claim"}],"minor_comments":[{"comment":"There is a typo in the sentence introducing pairwise joint embeddings: 'contstructed' should be 'constructed.'","section":"Appendix B, proof of Proposition B.7"},{"comment":"The text refers to 'the detailed algorithm in Appendix 1'; the algorithm is actually in Appendix B.1, so the cross-reference should be corrected.","section":"Section 5.1"},{"comment":"The caption of Figure 3(b) says dataset size is kept fixed while increasing n with k=n-1, but the number of training combinations n(n-1) grows with n. Please clarify how total dataset size is controlled, or adjust the wording to describe what is actually held fixed.","section":"Figure 3 caption"},{"comment":"The notation 'ci, cj ∈ {1,...,n}' overloads concept values with indices; the proposition should state that c_i ranges over the value set C_i and that combinations are indexed by i,j ∈ {1,...,n}.","section":"Proposition 4.1 notation"},{"comment":"The global mean is written as f in the appendix but as \\bar{f} in the main text; unify the notation to avoid confusion between the feature extractor f and the mean embedding.","section":"Section 3.2 / Definition 3.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Net take: the empirical case that diversity beats scale for compositional generalization is solid and worth reading; the theoretical half is not proven as written, and the appendix proof has three concrete holes. I would send the paper to review, but someone needs to fix Proposition 4.1/B.7 or the paper should drop the word \"prove\".\n\nWhat is new: the (n,k) framework for controlling concept-space size against training coverage, the FunnySprites dataset, and the observed three-phase transition from spurious features to decodable to linearly factored representations across five datasets. The diversity-over-scale contrast is consistent across all of them, including MPI3D, and the pretrained model evaluation (DINO, CLIP) is a sensible check on whether the structure identified in controlled settings exists in large-scale models. The self-citations are to metrics and extensions, not to the central result; there is nothing circular there.\n\nThe soft spot is exactly where the reader put it. Proposition 4.1/B.7 is not established as written. The 2n by 2n system in Eq. (25) is not full rank: summing the n diagonal equations gives the same vector as summing the n shifted equations, so the row rank is at most 2n−1, and the claim that the rows \"form distinct patterns that ensure linear independence\" is simply false. The proof then assumes that any vector in the joint span decomposes as u1 + u2 with u1 ⊥ u2; trivial intersection of U1 and U2 does not imply orthogonality, and without it the projection-classifier construction does not go through. There is also an arithmetic inconsistency: the proposition states joint span dimension 2n−1, while the proof's own dimensions (n−1)+(n−1) with trivial intersection give 2n−2. The zero-sum constraints from Lemma B.4 would likely repair the rank issue, and an explicit orthogonality hypothesis would repair the classifier step, but those repairs are not in the paper. The proposition also never says which k=2 combinations are observed; the recovery argument only goes through for the cyclic split used in the experiments.\n\nNone of this damages the empirical core: the controlled comparisons are well designed and the diversity conclusion is supported. The abstract's \"we prove this structure is key to efficiency\" overstates what is actually established.\n\nWho this is for: anyone working on compositional generalization or dataset design for vision; the framework and the negative scaling result are useful. It deserves a serious referee — the empirical contribution is strong enough that the flawed appendix should not sink it, but the proof needs repair or the claim needs to be downgraded to a conjecture.","headline":"Solid controlled evidence that diversity beats scale for compositional generalization, but the k=2 efficiency theorem is not proven as written and needs repair before the paper's central claim can stand.","tokens_in":25897,"tokens_out":4309,"would_cite":true,"duration_ms":41064,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","68T45"],"pacs":[],"model":"deepseek-v4-flash","headline":"Compositional generalization in vision models comes from combinatorial data diversity, not data volume; an ideal linear structure needs only two examples per concept.","keywords":["compositional generalization","data diversity","data scaling","linearly factored representations","visual concepts","out-of-distribution generalization","representation geometry","attribute-object combinations"],"falsifier":"For $n=3$ in the linear system of equation (25), summing the three diagonal rows and the three shifted rows both give the all-ones vector, so the $6\\times 6$ design matrix has rank at most 5 even though Part 2 of the proof of Proposition B.7 claims full rank; directly computing this rank for any $n$ settles whether two examples per concept value can recover the factored vectors.","tokens_in":24814,"feed_emoji":"🧩","tokens_out":8909,"duration_ms":91318,"temperature":0.7,"pith_summary":"Compositional generalization—recognizing an unseen attribute-object pairing by recombining known concepts—is a basic human cognitive ability that state-of-the-art vision models only partially exhibit. This paper asks whether the standard scaling recipe of ever-larger datasets and models will ever deliver it, and answers no: in controlled experiments that vary concept diversity and combination coverage independently of data volume, out-of-distribution accuracy on unseen pairings stays 60-80% below in-distribution accuracy even when training data is quadrupled. What closes the gap is diversity: more distinct concept values and more of their observed pairings push zero-shot accuracy above 90%, and only then do models organize concepts into linearly factored representations where a combined concept is the vector sum of its parts. The paper proves that under this structure, observing just two pairings per concept value is enough to generalize perfectly to all unseen pairings, and shows that pretrained models like CLIP and DINO have partial but imperfect versions of this structure. The practical reading is that dataset design should target combinatorial coverage, not just raw scale.","feed_headline":"Data diversity, not dataset size, drives compositional vision","feed_subtitle":"Scaling in-distribution images leaves a 60-80% generalization gap; diverse concept pairings close it.","key_machinery":"Linearly factored embeddings (Definition 3.2) are the central object: for two concepts, the joint representation of value pair $(c_1,c_2)$ equals $u_{c_1}+u_{c_2}$, the sum of the two concept-value vectors. The load-bearing recovery mechanism is the conditional-mean estimator: average the feature vectors of all training images that share a concept value, centre by the global mean, and the result recovers that value's concept vector up to a shift; for pairwise combinations this yields the additivity identity $u'_{c_i,c_j}=u'_{c_i}+u'_{c_j}$. That identity turns the training set of $k=2$ pairings per value into a $2n$-equation linear system whose solution gives the factored vectors, and the separation of the two concept subspaces lets the paper construct projection-based classifiers that label every unseen pairing. This mechanism is what converts combinatorial coverage into representational geometry, and representational geometry into zero-shot generalization.","core_discovery":"On the paper's own terms, the discovery is that the driver of visual compositional generalization is data diversity—the number of concept values and the fraction of their combinations seen during training—while data scale, measured as the number of images per combination, is not. In the paper's $(n,k)$ framework, models trained from scratch show three phases as combinatorial coverage grows: spurious features that cannot even discriminate the concepts, then discriminative features that are still not linearly factored, and finally, at high diversity, a geometry in which the representation of a combined concept is the sum of concept-value vectors, with near-perfect zero-shot accuracy on unseen pairings. Proposition 4.1 states the efficiency this buys: if representations are linearly factored and the concept subspaces are suitably separated, $k=2$ observed combinations per concept value suffice for a linear classifier to generalize perfectly to all $(n-2)n$ unseen combinations. The paper then tests pretrained models by recovering concept vectors from just two pairings per value and finds above-random performance that is concept-dependent—some attributes are nearly linearly accessible, others are not—so pretraining confers partial structure rather than full compositionality.","pith_inferences":["The paper studies pairs of concepts; if the additive mechanism is the reason diversity works, diversity should matter at least as much for three or more interacting concepts, since the number of combinations grows combinatorially.","A direct testable extension: explicitly regularizing a model's features toward an orthogonal, additive concept structure should yield OOD gains on unseen pairings even when $n$ is small, which would confirm that geometry rather than raw sample count is the causal channel.","Dataset construction for compositional competence could be guided by a coverage metric—how many distinct concept combinations appear—rather than by image count; comparing two datasets matched for diversity but differing in size would sharpen the headline claim.","The proof's reliance on disjoint concept subspaces suggests that concepts that are linearly factored but not orthogonal may need more than two examples; checking whether $k=3$ or $k=4$ restores perfect generalization in non-orthogonal settings would map the boundary of the result."],"forward_implications":["Scaling up in-distribution training data will not by itself produce compositional generalization; closing the ID-OOD gap requires increasing the variety of concept values and the combinations of them that a model sees.","A reliable indicator of readiness for compositional generalization is the linearity and orthogonality of concept representations; decodability alone is insufficient.","Under a perfectly linearly factored representation, a model needs only two seen pairings per concept value to generalize to every unseen pairing, so data efficiency and representational geometry are linked.","Pretrained vision models will remain partially compositional—strong on some concept pairs, weak on others—unless they are trained or probed on diverse combination sets.","Sparse combinatorial coverage in large web-scale datasets is a structural obstacle that raw scale cannot overcome, because the number of possible combinations grows faster than any practical dataset."],"supporting_citations":[{"why":"Supplies the definition of linearly factored embeddings and the conditional-mean recovery method that the proof of Proposition 4.1 builds on.","marker":"Trager et al., 2023"},{"why":"Prior evidence that large vision-language models sometimes represent composite concepts as vector sums, motivating the structure hypothesis.","marker":"Stein et al., 2024"},{"why":"Provides the shortcut-learning account used to interpret the early spurious-feature phase.","marker":"Geirhos et al., 2020"},{"why":"Gives the balanced linear-probe decodability procedure that the paper contrasts with linear structure.","marker":"Kirichenko et al., 2023"},{"why":"Contributes the dSprites dataset used in the controlled training-from-scratch experiments.","marker":"Matthey et al., 2017"},{"why":"Contributes the 3DShapes dataset used in the controlled training-from-scratch experiments.","marker":"Kim & Mnih, 2019"},{"why":"Contributes the PUG photorealistic synthetic dataset used to evaluate pretrained models.","marker":"Bordes et al., 2023"},{"why":"CLIP is one of the pretrained models evaluated for partial linear structure.","marker":"Radford et al., 2021"},{"why":"DINOv2 is one of the pretrained models evaluated for partial linear structure.","marker":"Oquab et al., 2024"},{"why":"Provides the oracle model-selection protocol that gives the models maximally favorable conditions.","marker":"Gulrajani & Lopez-Paz, 2020"}],"fun_headline_variants":["Diverse data beats dataset size for compositional vision","Combinatorial coverage, not scale, unlocks visual generalization","Scaling images isn't enough; diversity drives composition"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The construction assumes that the small set of seen pairings pins down each concept's direction exactly, and that the two groups of concepts point into fully separate directions so that projections can separate them.","fun_headline_variants_meta":{"raw":{"variants":["Diverse data beats dataset size for compositional vision","Combinatorial coverage, not scale, unlocks visual generalization","Scaling images isn't enough; diversity drives composition"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00068,"raw_usage":{"total_tokens":3085,"prompt_tokens":939,"completion_tokens":2146,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":2096}},"tokens_in":555,"tokens_out":2146,"duration_ms":16538,"temperature":1.0,"reasoning_tokens":2096,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:48:04.793761+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For $n=3$ in the linear system of equation (25), summing the three diagonal rows and the three shifted rows both give the all-ones vector, so the $6\\times 6$ design matrix has rank at most 5 even though Part 2 of the proof of Proposition B.7 claims full rank; directly computing this rank for any $n$ settles whether two examples per concept value can recover the factored vectors.","supporting_citations":[{"cited_title":"Linear Spaces of Meanings : Compositional Structures in Vision-Language Models , 2023","cited_arxiv_id":null,"evidence_quote":"Supplies the definition of linearly factored embeddings and the conditional-mean recovery method that the proof of Proposition 4.1 builds on."},{"cited_title":"Towards Compositionality in Concept Learning , 2024","cited_arxiv_id":null,"evidence_quote":"Prior evidence that large vision-language models sometimes represent composite concepts as vector sums, motivating the structure hypothesis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the shortcut-learning account used to interpret the early spurious-feature phase."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the balanced linear-probe decodability procedure that the paper contrasts with linear structure."},{"cited_title":"dSprites : Disentanglement testing sprites dataset, 2017","cited_arxiv_id":null,"evidence_quote":"Contributes the dSprites dataset used in the controlled training-from-scratch experiments."},{"cited_title":"and Mnih, A","cited_arxiv_id":null,"evidence_quote":"Contributes the 3DShapes dataset used in the controlled training-from-scratch experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes the PUG photorealistic synthetic dataset used to evaluate pretrained models."},{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I","cited_arxiv_id":null,"evidence_quote":"CLIP is one of the pretrained models evaluated for partial linear structure."},{"cited_title":"DINOv2 : Learning Robust Visual Features without Supervision , 2024","cited_arxiv_id":null,"evidence_quote":"DINOv2 is one of the pretrained models evaluated for partial linear structure."},{"cited_title":"and Lopez-Paz , D","cited_arxiv_id":null,"evidence_quote":"Provides the oracle model-selection protocol that gives the models maximally favorable conditions."}],"review_version":1}