{"id":"e4a70a25-8644-4e83-8c61-e420d0ab0878","arxiv_id":"2506.22389","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A general routing framework lets each token take its own path through a network, matching dense baselines while enabling learned compute savings and interpretable path specialization.","lead":"This paper introduces distributed neural architectures, where each token or patch is routed through its own sequence of modules in any order, and shows they match dense baselines on ImageNet and language modeling. It also finds that the chosen paths follow a power-law distribution and that tokens group into interpretable, specialized routes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5)'s bias-controller set point is dimensionally inconsistent with the reported 25%/30% skip rates, under-specifying the paper's central compute-efficiency claim.","rationale":"The reader's weakest assumption is that the hard top-k plus softmax update in Eq. (3) and the bias dynamics in Eq. (5) provide stable, low-variance gradients. That concern is reasonable but not the most concrete: the paper reports successful training, so the gradient estimator is empirically supported, and the absence of a theoretical guarantee is a limitation rather than a correctness objection. The more load-bearing and testable issue is the bias controller itself: dimensional analysis of Eq. (5) shows that the set point it implements is far smaller than the reported skip ratios given the stated values of N_m, k, and r. This directly affects the reproducibility of the paper's headline claim that compute efficiency can be learnt from data. If the inconsistency is only a typographical error in the equation, the underlying empirical result may still stand; that is why the appropriate verdict remains conditional (as the reader already concluded) rather than reject. I credit the paper for its honest discussion of limitations, its explicit acknowledgement that random models also exhibit power-law path distributions, and its clear separation of proof-of-principle results from claims about scaling. The concrete check proposed above would settle whether the efficiency mechanism is correctly specified.","tokens_in":45461,"tokens_out":12875,"duration_ms":144130,"concrete_test":"Implement the controller of Eq. (5) with the Table 1 top-2 vision DNA settings (N_m=24, k=2, r=0.25, one identity module per step), using the paper's stated definitions of c_i and \\bar c, and measure the equilibrium identity count c_Id. If the update converges to c_Id \\approx T/24 (effective top-k about 1.96) rather than c_Id \\approx T/2 (effective top-k about 1.5), then Eq. (5) as written does not produce the reported 25% skip. The same check should be run for the language model settings (N_m=72, r=0.30). The authors should also be asked to supply the exact update used in code and to confirm whether \\bar c is averaged over all modules including identity modules.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The compute-efficiency claim rests on the bias update in Eq. (5) steering the fraction of skipped modules to a target r. Under the definitions in Section 2.2, c_i is a token count and the average module count is \\bar c^{(s)} = (1/N_m)\\sum_i c_i^{(s)} = T k / N_m, where T is the number of tokens at that step and k is the top-k size. For one identity module per step, the controller's set point is r k \\bar c = r k^2 T / N_m, not r T k or r T. For the top-2 vision DNA (Table 1: N_m=24, k=2, r=0.25), this set point is 0.25*4/24*T = T/24, about 4.2% of tokens, i.e. roughly 2% of the 2T slots, so the effective top-k would be about 1.96 rather than the reported 1.5. For the top-2 language model with 30% skip (N_m=72, k=2, r=0.30), Eq. (5) gives about 0.8% of slots skipped, not 30%. The correct expression for a slot-skip fraction r would be r N_m \\bar c = r T k. As printed, Eq. (5) cannot produce the claimed skip rates; either the formula is misreported or the implementation uses a different target. Since no code or weights are released, the text alone does not specify the mechanism behind the central efficiency result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces distributed neural architectures (DNAs), in which each token or patch is routed by learned routers through a sequence of modules (attention, MLP, or identity), with top-k selection and softmax-weighted updates. The authors train top-1 and top-2 DNAs at ViT-Small scale on ImageNet and at GPT-2-Medium scale on FineWeb-Edu, and report that the models are competitive with dense baselines and can be trained to skip modules under a bias-based controller. The paper also presents interpretability analyses (path/ribbon visualizations, deep-dream reconstructions, module-reuse statistics) and claims that path frequencies follow a power law. The appendices provide hyperparameters, additional models with separate attention/MLP modules, and control experiments with random models.","tokens_in":45816,"tokens_out":11667,"duration_ms":127945,"significance":"If the empirical findings hold, this is a useful proof-of-principle that non-feed-forward, data-dependent routing can be trained end-to-end in both vision and language, and that routing decisions carry interpretable structure. The inclusion of a random-model baseline for the path-distribution analysis and the explicit statement that the current implementation is slower and more memory-hungry than dense baselines are commendable. The paper is clearly positioned as preliminary, not as a benchmark improvement; however, the load-bearing efficiency mechanism has a specification gap, and the strongest efficiency model is dominated by a simple shallower baseline, so the central 'learnt compute efficiency' claim needs revision before the results are fully supported.","major_comments":[{"comment":"The bias-controller set point is inconsistent with the reported skip rates, and this is load-bearing for the compute-efficiency claim. With c_i^{(s)} a token count per module and \\bar c^{(s)} = (1/N_m) \\sum_i c_i^{(s)} = T k / N_m, the target in Eq. (5) is r k \\bar c = r k^2 T / N_m. For the vision top-2 model (N_m=24, k=2, r=0.25) this equals T/24, i.e. about 4.2% of tokens, not 25%; for the language 30%-skip model (N_m=72, k=2, r=0.30) it is T/60, i.e. about 1.7% of tokens, not 30%. Unless the implementation uses a different definition of \\bar c or a different target (e.g. r k T for a slot-skip fraction), the controller as written converges to a skip rate roughly N_m/k^2 times smaller than claimed, and the paper does not state what quantity is actually regulated. Since no code or weights are released, the text alone does not specify the mechanism behind the headline efficiency result. Please correct the equation or the definitions, state whether r is a token-skip or slot-skip fraction, and report measured (not target) skip rates.","section":"§2.2, Eq. (5)"},{"comment":"The strongest efficiency experiment, top-2 DNA with 30% skip, is worse than the corresponding depth-reduced baseline on every reported metric: validation loss 2.784 vs 2.772 for GPT-2 (30% shallower), and downstream accuracies are lower (e.g. LAMBADA 23.8 vs 31.4, RACE 28.1 vs 30.1, Wikitext perplexity 52.6 vs 38.0). The paper does not discuss this comparison in the text, although it is directly relevant to the claim that compute efficiency and parameter sharing can be learnt from data. The claim should be qualified to state that, at this scale, the learned skipping does not provide a better compute-accuracy trade-off than simply training a shallower dense model.","section":"Table 3"}],"minor_comments":[{"comment":"The final sentence '...the routers at the end do not look at' is incomplete; please finish the sentence or remove it.","section":"Appendix C.2, Figure 14 caption"},{"comment":"The text refers to hyperparameters 'reported in Fig. 1', but they appear in Table 1; the cross-reference should be corrected (the same issue appears in Appendix A.1 when comparing with 'Fig. 1').","section":"§3.1 and Appendix A.1"},{"comment":"There are typographical errors: 'it's own path' should be 'its own path', and 'To this en' should be 'To this end'.","section":"§2.1 and §3.3"},{"comment":"The sentence 'The choices,i are fixed at their original values throughout' contains a punctuation error; it should read 'The choices i are fixed at their original values throughout'.","section":"§2.2, Eq. (6)"},{"comment":"The text says 'Note the log-scale of the x-axis in Fig. 10', but the compute distribution with log-scale x-axis appears in Fig. 11; the figure reference should be checked.","section":"§4.3"},{"comment":"The power-law exponents are stated without a fitting procedure or uncertainty; if this is intended as a quantitative claim, describe the fitting method, range, and error bars.","section":"Fig. 1 caption"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main concern is that Eq. (5), as written, cannot produce the reported skip rates; this is fixable but essential. The authors should be asked to provide code or an exact description of the implemented controller, and to contextualize Table 3 against the shallower baseline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good paper to know about. It introduces a genuinely broader proto-architecture—tokens route through any subset of modules in any order, trained end-to-end—and shows, at small scale, that such models can be competitive with dense baselines and learn to allocate compute from data. The power-law path statistics and the emergent specialization are real observations, not just hand-waving. The authors are honest about not chasing SOTA and include random-model baselines for the interpretability. Credit where due: this is a solid proof of concept.\n\nThe main soft spot is the bias-update equation. The stress-test note is correct: Eq. (5) as printed has a dimensionally inconsistent set point. With their definitions, c̄ = T k / N_m, so the target for identity-module counts is r k c̄ = r T k^2 / N_m, not r T k. For the vision model, that gives an effective top-k near 1.96, not the reported ~1.5; for the language model it's even further off. Either the formula is misreported or the implementation uses a different target. Since no code or weights are released, the reader cannot tell, and that directly undermines the reproducibility of the compute-efficiency claim. It's fixable—likely a typo, but a load-bearing one.\n\nSecond, the power-law claim needs uncertainty quantification. They report exponents from a single trained model, no confidence intervals, and note the random model also gives a power law with exponent -1. That's interesting but under-analyzed. Third, the interpretability results are compelling but rely on selected examples; the dreaming reconstructions are beautiful, but the paper doesn't quantify how representative they are.\n\nThe citation pattern is healthy: they build on prior work (DeepSeek bias trick, their own initialization results) and say so. None of the soft spots are fatal; the central empirical finding—that DNAs train and learn to skip compute—is supported by the measured effective top-k and compute distributions, which do not depend on the printed equation.\n\nWho this is for: anyone working on conditional computation, routing, or architecture search. It deserves a serious referee—the idea is new enough, the results are credible enough, and the issues are addressable. I'd send it out with a request for a corrected equation, code or a clear explanation of the actual controller, and uncertainty on the power-law fits.\n\nRecommendation: accept for peer review (with major revision expected).","headline":"A genuine proof of concept for distributed routing, with a load-bearing typo in the efficiency-control equation and under-quantified power-law claims; worth peer review.","tokens_in":46297,"tokens_out":4321,"would_cite":true,"duration_ms":44753,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a neural network built from unglued modules and per-token routers can be trained end-to-end to match dense baselines while learning compute allocation and parameter sharing.","keywords":["distributed neural architectures","conditional computation","token routing","mixture-of-experts","mixture-of-depths","dynamic compute allocation","emergent specialization","routing interpretability"],"falsifier":"Retrain the top-1 DNA on ImageNet with all router choices replaced by random draws matched to the same per-step module frequencies; if test accuracy stays at the trained level, the learned routing is not causally responsible for the result, and the paper's interpretability claims would need re-examination.","tokens_in":45297,"feed_emoji":"🧠","tokens_out":9290,"duration_ms":95617,"temperature":0.7,"pith_summary":"The paper sets out to show that a neural network can be trained without a fixed feed-forward order: a pool of interchangeable modules (transformer blocks, attention-only blocks, MLP-only blocks) plus lightweight routers, where every token or image patch is sent through its own sequence of modules. It reports that such distributed neural architectures (DNAs) reach accuracy and loss close to dense ViT-small and GPT-2-medium baselines on ImageNet and FineWeb-Edu, while learning to skip modules, reuse parameters, and route by content. This matters because it turns compute allocation and architecture connectivity into things that can be learned from data rather than fixed by hand, and because the per-token routing itself becomes a readable trace of what the model is doing. The paper is explicit that this is a feasibility and analysis study, not a benchmark chase: the goal is to show that distributed models are trainable and to understand their emergent structure.","feed_headline":"Letting every token choose its own route still matches dense baselines","feed_subtitle":"Learned routing keeps accuracy, cuts compute per token, and exposes interpretable paths in images and text.","key_machinery":"The central object is the proto-architecture: a collection of modules and routers with no depth ordering, plus a short un-routed backbone that stabilizes training. Routers are linear token classifiers using hard top-k selection, and module outputs are merged by the softmax-weighted residual update of Eq. (3), an update chosen to preserve signal propagation. Identity modules that do nothing to a token, combined with the decoupled bias update of Eq. (5), teach the model to skip work at a targeted ratio. A path records the sequence of modules one token takes; for k>1 a ribbon bundles the parallel k-tuples of choices. This machinery is what lets connectivity, compute allocation, and parameter sharing emerge from the loss rather than from the architecture.","core_discovery":"On the paper's own terms, the central discovery is that end-to-end training of a routed, order-free architecture works: trained DNAs are competitive with their dense baselines in both vision and language, and the models spontaneously develop structured behavior. Paths taken by tokens follow a power-law distribution, with exponents around -1 for random initializations and trained vision, and -1.2 for trained language. Some paths specialize: vision paths group edges, backgrounds, objects, and boundaries, while language paths group verb forms, punctuation, sentence-end tokens, adjectives, and word fragments. The models also learn interpretable compute allocation: images with many object boundaries consume more compute, and low-compute text tends to be links, markup, bibliography material, or characters from languages the model has not learned. When transformer blocks are split into separate attention and MLP modules, the trained models prefer attention early and MLP late, rather than reassembling the usual glued block.","pith_inferences":["The paper does not claim that the power-law path distribution is universal, but if it is, it could serve as a diagnostic for specialization in future routed networks.","A testable extension the paper does not run would add a path-diversity penalty to the objective and measure accuracy and performance under distribution shift.","The paper lists data filtering as future work; the low-compute text patterns suggest a trained DNA could be inverted into a data-quality scorer.","The attention-early, MLP-late preference implies that fixed homogeneous stacks may be suboptimal, though this extrapolation to larger scale is the editor's, not the paper's."],"forward_implications":["Mixture-of-Experts, Mixture-of-Depths, early exit, weight sharing, and sparse attention all become special cases of one trained system rather than separate design choices.","Compute and active parameter counts become learnable objectives: changing the skip target in the objective changes how much work the model does per token.","Routing decisions provide an interpretability channel independent of activations, allowing path statistics to segment images and group tokens by syntax and semantics.","Path frequencies follow a power law, meaning a small set of sub-networks carries most of the processing while a long tail handles rare, specific inputs.","Given separate attention and MLP modules, trained models organize them by depth, with attention near the input and MLP near the output, suggesting that the glued transformer block is not the only natural composition."],"supporting_citations":[{"why":"Supplies the conditional-computation idea that different tokens can take different paths through a network.","marker":"Bengio et al. (2013)"},{"why":"Defines the Mixture-of-Experts routing setup that DNA generalizes to arbitrary module order.","marker":"Shazeer et al. (2017)"},{"why":"Introduces Mixture-of-Depths, the dynamic-compute baseline that DNA extends with content-dependent paths.","marker":"Raposo et al. (2024)"},{"why":"Provides the layer-skip and early-exit baseline for per-token compute savings.","marker":"Elhoushi et al. (2024)"},{"why":"Supplies the decoupled bias trick used in Eq. (5) to target the skip ratio.","marker":"Liu et al. (2024)"},{"why":"Supports the signal-propagation reasoning behind the residual update in Eq. (3).","marker":"Roberts et al. (2022)"},{"why":"Motivates the work by showing that deeper layers are often underutilized in trained transformers.","marker":"Gromov et al. (2024)"},{"why":"Provides the ViT-small dense baseline for the vision experiments.","marker":"Dosovitskiy et al. (2021)"},{"why":"Provides the GPT-2-medium dense baseline and tokenizer for the language experiments.","marker":"Radford et al. (2019)"},{"why":"Provides the FineWeb-Edu training data for the language models.","marker":"Lozhkov et al. (2024)"}],"fun_headline_variants":["Every token picks its own route, still matches dense baselines","Learned routing: tokens choose paths, models stay accurate","Power-law paths emerge in trained routed networks","Attention early, MLP late: learned module order","Tokens learn to allocate compute, paths become interpretable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is empirical: training with hard top-k routing plus the softmax-weighted update stays stable and low-variance enough for end-to-end learning, and the bias dynamics converge to the target skip ratio; the paper offers small-scale evidence but no theoretical guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Every token picks its own route, still matches dense baselines","Learned routing: tokens choose paths, models stay accurate","Power-law paths emerge in trained routed networks","Attention early, MLP late: learned module order","Tokens learn to allocate compute, paths become interpretable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3111,"prompt_tokens":946,"completion_tokens":2165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2088}},"tokens_in":562,"tokens_out":2165,"duration_ms":17534,"temperature":1.0,"reasoning_tokens":2088,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:05:20.084385+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the top-1 DNA on ImageNet with all router choices replaced by random draws matched to the same per-step module frequencies; if test accuracy stays at the trained level, the learned routing is not causally responsible for the result, and the paper's interpretability claims would need re-examination.","supporting_citations":[{"cited_title":"Fineweb-edu: the finest collection of educational content, 2024.https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu","cited_arxiv_id":null,"evidence_quote":"Provides the FineWeb-Edu training data for the language models."}],"review_version":1}