{"id":"7586fc9c-e21d-47d9-bd0c-9e911a4b465b","arxiv_id":"2506.01970","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Johnny tokenizes RPM images into a learned codebook, adds a self-referential 'sub-enumeration' loss to the reasoning module, and pairs it with a new Spin-Transformer layer; gains over strong baselines are modest, and a metadata-based generalization result is suspicious.","lead":"This paper introduces Johnny, a neural network for Raven's Progressive Matrices that learns a small set of image symbols and uses them to score both the candidate answers and every symbol in its learned space. It also adds a Spin-Transformer block meant to improve positional reasoning, and reports small accuracy gains on standard visual reasoning benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sub-enumeration loss l4 uses a self-referential nearest-codebook target; absent codebook-usage evidence or a target-ablation, the ACT3 gains could be generic regularization rather than true negative-option supplementation.","rationale":"I read the paper as making two intertwined claims: (1) the Johnny architecture with a learnable representation space and the sub-enumeration loss l4 improves abstract reasoning by supplementing the finite option pool, and (2) the Spin-Transformer improves positional reasoning. The strongest and most novel claim is (1), and it depends on l4 being a meaningful enumeration signal. The single most load-bearing weak point is that l4's target is self-referential: the encoder defines the token z_{αj}, the codebook is aligned to that same encoder through l1/l2, and then l4 rewards the codebook component nearest to that token. If the codebook is degenerate or merely reflects the encoder's own clustering, l4 does not add information about incorrect options; it only adds a consistency regularizer. This is not an external-consensus disagreement but an internal-validity risk: the paper gives no evidence that the codebook components are diverse or that the nearest-neighbor target is semantically meaningful. A random-target or farthest-target ablation would settle the question, and codebook-usage statistics would reveal collapse. The Spin-Transformer contribution has some independent support from the module-replacement experiments in Table II, though it is a secondary contribution. The PGM metadata experiments are explicitly acknowledged by the authors to rely on enumerable metadata, so they are self-limited and not the core of the claim. Given that the central mechanism is untested against its most natural confound, the reader's CONDITIONAL verdict is appropriate; I would not change it.","tokens_in":20491,"tokens_out":6620,"duration_ms":77192,"concrete_test":"Re-run the Johnny ACT3 configuration on RAVEN and I-RAVEN with all hyperparameters and losses unchanged, but replace the l4 target k~ = argmin_k ||z_{αj} - T_k||^2 with a component chosen uniformly at random (or, more strongly, with the component farthest from z_{αj}). If average accuracy stays near the reported 99.4/99.6, the specific nearest-neighbor target is not load-bearing and the sub-enumeration claim is not supported. Additionally, after training the original ACT3 model, report the number of codebook components that are ever selected as argmin over the validation set; if fewer than roughly 10% of the 256 components are used, codebook collapse is present and the representation space cannot plausibly enumerate incorrect option configurations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Johnny \"supplements primitive negative option configurations with a learned representation space\" rests on loss l4 (Eq. 8). In l4, the positive target for the Reasoning Module's distribution over codebook components is k~ = argmin_k ||z_{αj} - T_k||^2, where z_{αj} is the token of the correct option produced by the same encoder that is being trained, and T_k are the same learnable codebook vectors that l1/l2 pull toward those encoder outputs. This is a circular target: the model is asked to reward whichever codebook vector the model itself has chosen to be closest to the correct token. Nothing in the loss or in the paper guarantees that the codebook has enough diversity or semantic alignment to serve as an enumeration of incorrect options. A standard failure mode of VQ-style losses without commitment or usage penalties is codebook collapse, where most components are never selected; if collapse occurs, k~ is effectively arbitrary or constant per token position, and l4 can be minimized without providing any signal about which incorrect completions are plausible. The reported ACT2-to-ACT3 gains (e.g., RAVEN average 98.8 to 99.4, PGM 98.2 to 99.0) are then explained equally well by the extra training signal from a regularizing auxiliary loss as by the proposed sub-enumeration mechanism. The paper provides no codebook-usage statistics, no analysis of which components are selected, and no ablation replacing the l4 target with a non-informative one, so the mechanism is unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that end-to-end RPM-solving models are limited by their reliance on the specific incorrect options present in the training option pool, and proposes a learned representation space to 'sub-enumerate' possible incorrect configurations. It introduces the Johnny architecture, which separates a tokenized representation extractor (ViT backbone) from a reasoning module that scores individual tokens and, in the ACT3 configuration, also scores components of a learned codebook under a new loss l4 (Eq. 8). The paper further proposes Spin-Transformer and a lightweight Straw Spin-Transformer variant that modify inter-head communication in the transformer encoder. Experiments are reported on RAVEN, I-RAVEN, and PGM, with claims of state-of-the-art accuracy (e.g., 99.4% on RAVEN, 99.6% on I-RAVEN, 99.0% on PGM), and an additional metadata-conditional experiment on PGM generalization sub-problems is presented in Table V.","tokens_in":20861,"tokens_out":6711,"duration_ms":69518,"significance":"If the reported results and the proposed mechanism are substantiated, the paper would make three contributions: a diagnosis of end-to-end RPM solvers' dependence on option-pool configurations with a concrete remedy; an architecture that couples tokenized representation learning with a reasoning module that scores both options and representation-space components; and an attention architecture variant with a lightweight version that improves positional-relational reasoning. The ACT1-ACT4 decomposition is a useful experimental design because it isolates the representation-space losses from the backbone change, and the paper evaluates on three standard benchmarks against several strong baselines. The claims are falsifiable, and the loss equations are stated in sufficient detail to reimplement the method. However, the central ACT3 mechanism currently lacks direct evidence: the l4 target is self-referential, and the reported gains are small on saturated metrics without repeated-run statistics or code. The contribution is therefore conditional until the mechanism analysis and reproducibility are supplied.","major_comments":[{"comment":"The positive target in the sub-enumeration loss is \\tilde{k} = argmin_k ||z_{\\alpha j} - T_k||^2_2, where z_{\\alpha j} is produced by the same encoder being trained and T_k are the same codebook vectors optimized by l1-l3 (Eqs. (2)-(5)). Final accuracy is measured against external labels, so the overall result is not circular, but the attribution of ACT3 gains to 'sub-enumeration of incorrect options' is not supported as stated. The paper reports no codebook usage statistics, no active-component counts, no analysis of which components are selected by \\tilde{k}, and no ablation replacing the l4 target with a non-informative one (e.g., random components or a fixed target). Because Eqs. (2)-(4) are VQ-style losses without a commitment or usage penalty, codebook collapse is a concrete risk; if only a few components are active, \\tilde{k} is effectively arbitrary and l4 can act as a generic regularizer. The ACT2-to-ACT3 gains (Table I: RAVEN 98.8 to 99.4; Table III: PGM 98.2 to 99.0) are then equally compatible with a regularization effect. Please provide codebook-usage analyses and target ablations, or revise the mechanistic claim.","section":"§VI-D, Eq. (8)"},{"comment":"All experiments are reported as single accuracies without seeds, standard deviations, or confidence intervals, and the experimental setup is delegated to self-citations [29], [32] (\"the same settings and equipment\"). On RAVEN/I-RAVEN many entries are at 99-100%, so a single-run difference of 0.1-0.5 percentage points (e.g., Johnny ACT3 99.4/99.6 vs Triple-CFN 98.9/99.1 in Table I) is within plausible run-to-run variability. Without repeated runs, the state-of-the-art claims and the cross-configuration comparisons ACT1-4 cannot be verified. Please report mean and standard deviation over multiple seeds, and release code or detailed hyperparameters, including the 'sliding window' schedule mentioned in §VI-C and the exact training schedule for adding l4.","section":"§VIII, Tables I-III"},{"comment":"The metadata experiments require clarification of what \"enumerate all possible manifestations of metadata\" means. If the representation space {Y_\\beta | \\beta in [1,L]} is sized to include metadata manifestations that appear only in held-out generalization splits, then construction of the space uses knowledge of the test distribution, and the high accuracies in Table V (92.2% on Interpolation, 98.0% on Held-out Pairs of Triples and Attribute Pairs) may reflect this prior knowledge rather than the method's generalization. Please specify whether L is chosen from training metadata only, whether any test metadata or test instances are used in building or training the space, and report repeated-run statistics for Table V. The candid limitation paragraph at the end of §VIII.C is appreciated, but it does not resolve the test-distribution question.","section":"§VIII-C, Eq. (33), Table V"},{"comment":"The lightweight claim for Straw Spin-Transformer is asserted but never quantified. Table II compares accuracy but gives no parameter counts, FLOPs, or training/inference time for Spin-Transformer versus Straw Spin-Transformer; the only justification is the architectural reduction in the number of mapping matrices (Eqs. (14)-(25)). Please add quantitative complexity measurements to support the 'lightweight' label and to justify the 'computational parity' argument used in the replacement experiments (half the number of layers).","section":"§VII.B, Tables II"}],"minor_comments":[{"comment":"PGM is introduced with citation [16], but the correct reference is [17] (Barrett et al.).","section":"Abstract and §II"},{"comment":"'Patttern Extractor' should be 'Pattern Extractor'; similarly, 'fallows' in §VII.A should be 'follows'.","section":"Figure 8 and §VI.B"},{"comment":"The table title says 'Valen' instead of 'PGM', and the header layout 'Model/Task' is confusing; please clarify which column corresponds to the task.","section":"Table IV"},{"comment":"The placeholder '' with the caption 'on-demand broadcasting' is undefined; please replace it with explicit indexing notation.","section":"Eq. (14)"},{"comment":"The statement that 'when N=1, the Spin-Transformer collapses into a regular Transformer-Encoder' is not justified by Eqs. (10)-(13) unless the pose matrices are set to zero; please explain the collapse explicitly.","section":"§VII.A"}],"recommendation":"major_revision","confidential_remarks":"For the revision, the editor should strongly encourage code release, since the claims rest on small accuracy differences and the key loss uses a model-defined target. It would also be useful to clarify the novelty relationship with the authors' own prior preprints [29], [31], [32], especially where hyperparameters are delegated to those works. The manuscript appears formatted as a journal submission but carries an arXiv preprint number; please confirm the intended venue and ensure the experimental reporting matches its standards."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about Johnny: the Spin-Transformer block is a genuinely new attention variant and the reported gains on RAVEN/I-RAVEN/PGM are plausible; but the paper's central sub-enumeration loss is self-referential, and the PGM metadata experiments look contaminated. The paper deserves a serious referee, but as written the mechanism evidence is not there.\n\nWhat's new and good: the architecture is clean. ViT tokens, a codebook, and a reasoning module that scores all codebook entries; the loss terms are explicit. Spin-Transformer is a real idea — inter-head communication via learned pose vectors — and the module-replacement experiments on RS-TRAN and Triple-CFN are a good sanity check, especially the gains on OI-G and 3×3 Grid where positional reasoning matters. The paper is also honest about where things fail: it couldn't drop Spin-Transformer into RS-TRAN on PGM due to memory, and progressive generalization remains at chance.\n\nSoft spots, in order of importance:\n\n1. The l4 target is circular. Eq. 8 picks k~ = argmin ||z_αj - T_k||, where z_αj comes from the encoder being trained and T_k are the same codebook vectors that l1/l2 pull toward encoder outputs. There is no external ground truth, no codebook-usage statistics, and no ablation with a non-informative target. Without that, the ACT2-to-ACT3 gains could simply be extra regularization. The mechanism claim is unsupported.\n\n2. The PGM generalization experiment (Table V) likely leaks. Eq. 33 enumerates all possible metadata manifestations. If metadata includes the held-out attributes, then those splits are no longer held out. The paper never addresses this. The jump from 87.3 to 92.2 on interpolation and near-chance on extrapolation is consistent with fitting metadata directly, not with sub-enumeration.\n\n3. Reproducibility: no seeds, no error bars, no code, hyperparameters delegated to self-citations. For a paper reporting 99% accuracy, that's a real problem.\n\nThe central critique of option-pool dependence is mostly a reframing of supervised classification, but that doesn't sink the architecture. The math in the Spin-Transformer section appears consistent, and the citations look fine.\n\nWho gets value: people working on RPM and abstract visual reasoning, and anyone designing token-level auxiliary losses. I'd bring it to reading group but wouldn't cite it until code and ablations exist.\n\nRecommendation: peer-review it, but major revision. Ask for an l4 target ablation, codebook usage analysis, a leakage check on Table V, and code or seeds.","headline":"A well-specified architecture with plausible small benchmark gains, but the central sub-enumeration mechanism is self-referential and the PGM metadata experiments look contaminated.","tokens_in":21362,"tokens_out":2554,"would_cite":false,"duration_ms":26532,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Johnny's claim: a learned representation space of wrong-answer stand-ins lifts RPM accuracy to 99.4% on RAVEN, 99.6% on I-RAVEN, and 99.0% on PGM.","keywords":["abstract reasoning","Raven's Progressive Matrices","representation space","sub-enumeration","tokenization","Transformer","inter-head communication","visual reasoning"],"falsifier":"Run Johnny ACT3 identically but replace the target of $\\ell_4$ with a randomly chosen codebook vector, or with the nearest vector from a frozen, randomly initialized codebook, and compare per-subset accuracy on RAVEN D-9/OIG and PGM Neutral against ACT2: if the gain persists, $\\ell_4$ is a regularizer rather than sub-enumeration; if it vanishes, check whether the learned codebook vectors individually correspond to coherent human-annotated attributes, because without such alignment the codebook cannot be supplying the missing negative configurations.","tokens_in":20296,"feed_emoji":"🧩","tokens_out":15414,"duration_ms":140095,"temperature":0.7,"pith_summary":"Johnny is a solver for Raven's Progressive Matrices (RPM), a family of visual multiple-choice puzzles. The paper argues that end-to-end models that pick from an option pool are limited by the wrong answers they are shown: training fits a decision boundary from the observed incorrect options, so an incomplete pool leaves that boundary loose and caps accuracy. To remove the cap, Johnny tokenizes image representations and aligns them to a learned, discrete 'representation space'; an extra loss scores every vector in that space and treats the vector closest to the correct answer's token as a stand-in for missing wrong answers. This 'sub-enumeration' of negative configurations is reported to push accuracy to 99.4% on RAVEN, 99.6% on I-RAVEN, and 99.0% on PGM, above the strongest published baselines, while a second contribution—the Spin-Transformer—adds explicit inter-head communication to capture positional relations, with a lightweight variant that keeps most of the gain. A reader should care because the claim is that abstract reasoning improves by structuring the latent space rather than by scaling the network.","feed_headline":"A learned codebook of wrong answers lifts RPM reasoning to 99.4%","feed_subtitle":"Sub-enumerating the missing wrong answers in a token codebook lifts accuracy on RAVEN, I-RAVEN, and PGM.","key_machinery":"The load-bearing object is the learned representation space, a finite set of $K=256$ optimizable vectors in the same dimension as one image token. Johnny's extraction module maps each of the 16 images of an RPM instance into token vectors $\\{z_{ij}\\}$; losses $\\ell_1$, $\\ell_2$, and $\\ell_3$ align each token to its nearest codebook vector, update the codebook, and reconstruct the original image from the selected vectors through a decoder $D$, making the space a complete coding of observable images. The sub-enumeration term $\\ell_4$ then computes a reasoning score for every codebook vector and applies a softmax cross-entropy whose positive target is the vector $\\tilde{k}=\\arg\\min_k \\|z_{\\alpha j}-T_k\\|_2^2$ nearest the correct option's token; this is the mechanism that manufactures synthetic negative options. The Spin-Transformer is secondary machinery: its Spin Block computes pose vectors $P_{jkl}=H_{jk}W_{jkl}$, sums them with a squash nonlinearity, and adds them to output tokens, giving attention heads a route to communicate local position information; the Straw variant shares the mapping matrices across token positions and folds the weighted sum into a masked cross-attention over a learned anchor vector.","core_discovery":"The paper's central claim has two stages. First, training an end-to-end RPM solver is equivalent to fitting a probability distribution: the mean is set by correct options and the variance by incorrect options, so the model converges its decision boundary only from the negatives it has seen. Second, Johnny replaces exhaustive enumeration of all possible wrong answers with a bounded discrete codebook of $K=256$ learnable vectors $\\{T_k\\}_{k=1}^{K}$, aligned with every image token through three losses (alignment, codebook update, and reconstruction via a decoder). The sub-enumeration loss $\\ell_4$ makes the reasoning module score every codebook vector and drives the probability mass, at temperature $\\tau=0.01$, onto the vector closest to the correct option's token. In the reported experiments, the CE-only baseline achieves 98.6/99.0 on RAVEN/I-RAVEN, adding the representation space reaches 98.8/99.2, and adding $\\ell_4$ reaches 99.4/99.6; on PGM the comparable sequence is 97.9, 98.2, 99.0.","pith_inferences":["If the mechanism is what carries the gain, the nearest-codebook-target loss should transfer to other multiple-choice reasoning formats with sparse negatives, such as diagrammatic reasoning or attribute-based visual question answering; the paper only tests RPM-style benchmarks.","The paper fixes $K=256$; the sub-enumeration story predicts a non-monotonic dependence on $K$, since too small a codebook cannot cover the concepts and too large a one makes the nearest-neighbor target nearly arbitrary, so sweeping $K$ would separate the mechanism from generic regularization.","The Gaussian-boundary argument implies that end-to-end solvers' accuracy should respond systematically to the information content of the option pool; varying the number or similarity of wrong answers across existing RPM datasets would test this prediction, which the paper does not run.","The future-work section's generative extension depends on an external judge for the synthesized answers, and the paper itself notes that current symbolic solvers are not accurate enough to serve as that judge."],"forward_implications":["With the representation space and sub-enumeration loss, RAVEN accuracy rises from 98.6% to 99.4% and I-RAVEN from 99.0% to 99.6%, with the largest subset gains on D-9 and OIG, which require positional rules.","On PGM, the same additions lift overall accuracy from 97.9% to 99.0% and interpolation generalization from 81.0% to 87.3%, while extrapolation remains near 18-19%.","Swapping the standard Transformer encoder for the Spin-Transformer in two strong published solvers, under compute-matched settings, improves their accuracy on 3×3 Grid and OIG subsets; the lightweight Straw variant retains most of the gain with a reduction in trainable mapping matrices from $O(N)$ to $O(1)$.","Because the decoder can reconstruct images from selected codebook components, the trained Johnny can in principle generate a candidate answer image rather than choose one, which the paper proposes for future generative RPM solving."],"supporting_citations":[{"why":"Supplies the RAVEN dataset and its seven configurations used for the main accuracy tables.","marker":"[16]"},{"why":"Supplies the PGM dataset and its generalization subproblems, including interpolation and held-out attribute settings.","marker":"[17]"},{"why":"Provides the visual Transformer backbone whose patch tokenization makes a finite representation space tractable.","marker":"[18]"},{"why":"Supplies a strong visual-reasoning baseline whose subset accuracies motivate the claim that positional rules are hard for standard encoders.","marker":"[25]"},{"why":"Provides the Transformer-based solver whose experimental settings are reused and whose encoder is swapped for the Spin-Transformer in ablations.","marker":"[29]"},{"why":"Provides the strongest prior solver in the comparison tables, used as an accuracy baseline and as another encoder-replacement target.","marker":"[32]"},{"why":"Supplies the I-RAVEN dataset and its rule configurations used for the second benchmark.","marker":"[33]"},{"why":"Supplies the criterion that a correct option is one that lets the completed matrix follow the statement's progressive pattern, which grounds the probability-distribution analysis.","marker":"[34]"},{"why":"Supplies a comparison baseline used on the PGM generalization subproblems.","marker":"[22]"}],"fun_headline_variants":["Johnny: codebook of negatives pushes RAVEN to 99.4%","Learning the space of wrong answers for Raven","Sub-enumerating negatives with a token codebook yields 99.4%","Codebook of negatives lifts RPM reasoning to 99.4%","Johnny: structured representation space for abstract reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method depends on the learned set of 256 representative vectors genuinely separating the concepts in the puzzle images, so that the vector closest to the correct answer is a meaningful stand-in for a missing wrong answer; if the vectors have not captured the concepts, the extra training term only reinforces the model's own habits and the reported gains are regularization, not reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Johnny: codebook of negatives pushes RAVEN to 99.4%","Learning the space of wrong answers for Raven","Sub-enumerating negatives with a token codebook yields 99.4%","Codebook of negatives lifts RPM reasoning to 99.4%","Johnny: structured representation space for abstract reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000561,"raw_usage":{"total_tokens":2671,"prompt_tokens":959,"completion_tokens":1712,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1623}},"tokens_in":575,"tokens_out":1712,"duration_ms":11660,"temperature":1.0,"reasoning_tokens":1623,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:45:29.852658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Johnny ACT3 identically but replace the target of $\\ell_4$ with a randomly chosen codebook vector, or with the nearest vector from a frozen, randomly initialized codebook, and compare per-subset accuracy on RAVEN D-9/OIG and PGM Neutral against ACT2: if the gain persists, $\\ell_4$ is a regularizer rather than sub-enumeration; if it vanishes, check whether the learned codebook vectors individually correspond to coherent human-annotated attributes, because without such alignment the codebook cannot be supplying the missing negative configurations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RAVEN dataset and its seven configurations used for the main accuracy tables."},{"cited_title":"Measuring Abstract Reasoning in Neural Networks","cited_arxiv_id":null,"evidence_quote":"Supplies the PGM dataset and its generalization subproblems, including interpolation and held-out attribute settings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Transformer-based solver whose experimental settings are reused and whose encoder is swapped for the Spin-Transformer in ablations."},{"cited_title":"Triple-CFN: Separating Concepts and Features Enhances Machine Abstract Reasoning Ability","cited_arxiv_id":"2403.03190","evidence_quote":"Provides the strongest prior solver in the comparison tables, used as an accuracy baseline and as another encoder-replacement target."},{"cited_title":"Stratified Rule-Aware Network for Abstract Visual Reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the I-RAVEN dataset and its rule configurations used for the second benchmark."},{"cited_title":"A., Just, M","cited_arxiv_id":null,"evidence_quote":"Supplies the criterion that a correct option is one that lets the completed matrix follow the statement's progressive pattern, which grounds the probability-distribution analysis."},{"cited_title":"Scale-Localized Abstract Reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies a comparison baseline used on the PGM generalization subproblems."}],"review_version":1}