{"id":"7c6ddb0d-2bfb-4af6-946a-251628bc3db1","arxiv_id":"2509.10530","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DASG-MoE combines grouped sliding-window attention, a two-tier expert structure, and adaptive routing, and claims state-of-the-art results, but the experimental support is internally inconsistent.","lead":"The paper introduces a MoE model that splits attention into groups, uses shallow and deep expert modules, and routes tokens to experts based on an attention-derived importance score. It claims better accuracy and efficiency than Switch Transformer and several open-source LLMs on GLUE and other benchmarks.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Empirical validation never tests the claimed long-sequence setting; all reported benchmarks are short-text, so the central SOTA claim is unsupported.","rationale":"The reader identified the implausible compute setup (5T tokens on 8 A100s) as the weakest assumption, and that concern is legitimate. My stress-test instead focuses on an even more direct gap: the paper's central assertion is about long-sequence performance, yet the experiments only use short-text tasks. Even a perfectly reproducible version of the reported GLUE and reasoning results would not support the abstract's headline claim, because the relevant long-sequence evaluations are entirely absent. This makes the missing long-sequence evaluation the single most load-bearing concern. It is not a disagreement with the reader; it is a sharper pinpointing of the same underlying unsupportedness. The verdict remains REJECT, as the empirical foundation for the central claim does not exist. The concrete test would settle whether the architecture has any long-sequence advantage, which the paper currently does not demonstrate.","tokens_in":16573,"tokens_out":2415,"duration_ms":22433,"concrete_test":"Run DASG-MoE and the Switch Transformer baseline on a standard long-context benchmark such as LongBench or Long Range Arena at matched parameter counts and training budgets, reporting performance for sequences of at least 4K tokens. If DASG-MoE does not outperform the Switch Transformer baseline on those long-sequence tasks, the abstract's claim of superior long-sequence modeling is contradicted. This check should be accompanied by public training logs and a compute budget breakdown to verify the 5T-token pretraining claim from Section 5.1.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that DASG-MoE 'outperforms state-of-the-art models' on 'multiple long-sequence benchmark datasets'. The experimental section does not evaluate any long-sequence dataset. Section 5.2 reports GLUE tasks (SST-2, MRPC, MNLI, etc.), whose instances are short sentences and paragraphs, typically under 512 tokens. Section 5.1 and Table 1 evaluate MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, and HumanEval, none of which are long-sequence benchmarks. No LongBench, Long Range Arena, SCROLLS, or similar long-context tasks appear anywhere in the paper. Thus the abstract's central empirical claim is untested even under the paper's own assumptions. Two additional problems compound this gap: Section 5.1 states only 8 NVIDIA A100 GPUs while Section 5.2 reports pretraining an 8B model on 5T tokens, which is not credible with that hardware, and Table 2 and Table 3 report inconsistent GLUE numbers (e.g., SST-2 80.13 vs 95.89) for ostensibly comparable setups. However, the decisive issue is that even if the compute and results were accepted, they would not bear on long-sequence modeling, because no long-sequence task was run.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DASG-MoE, a Mixture-of-Experts Transformer variant that combines grouped multi-head attention (GMHA) with sliding windows, a dual-scale shared expert structure (DSSE) with shallow and deep expert modules, and an adaptive dynamic routing (ADR) mechanism that allocates a variable number of experts per token based on a learned importance score. The authors claim that this architecture outperforms state-of-the-art models on multiple long-sequence benchmarks. The experiments reported in Sections 5.1-5.6 cover GLUE tasks, MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, and HumanEval, together with ablations on the routing mechanism and the number of attention groups.","tokens_in":16831,"tokens_out":6316,"duration_ms":54083,"significance":"Variable expert allocation is a plausible and potentially useful extension of static top-k MoE routing, and the paper identifies a real limitation of fixed expert budgets. However, the significance cannot be assessed from the present manuscript because the central empirical claims are unsupported: no long-sequence benchmark is evaluated, the reported pretraining scale is incompatible with the stated hardware, and the GLUE numbers are internally inconsistent. On the positive side, the paper provides ablations for the number of attention groups and for routing versus random forwarding, and it reports confidence intervals for HumanEval and MBPP; these are useful elements, but they do not compensate for the missing evidence on the core claims. The manuscript also lacks code, data, or sufficient training details to be reproducible.","major_comments":[{"comment":"The central claim of state-of-the-art performance on 'multiple long-sequence benchmark datasets' is not tested anywhere in the paper. Section 5.2 evaluates GLUE tasks (SST-2, MRPC, MNLI, etc.), whose instances are short sentences, and Table 1 lists MMLU, CMMLU, CEval, BBH, GSM8K, MATH500, MBPP, and HumanEval, none of which is a long-sequence benchmark. No LongBench, Long Range Arena, SCROLLS, or similar long-context evaluation appears in the manuscript. The conclusion itself admits that 'conducting extensive evaluations on larger-scale long-sequence datasets' is future work, which confirms that the claimed long-sequence evaluation was not performed.","section":"Abstract, Sections 5.2-5.3, Conclusion"},{"comment":"The reported training setup is not credible. Section 5.1 states that the infrastructure is 8 NVIDIA A100 GPUs, while Section 5.2 reports pre-training an 8B-parameter model on 5T tokens and a 3B model on 2.2T tokens. Even under optimistic assumptions about A100 throughput and MoE sparsity, training an 8B model on 5T tokens requires on the order of 10^23 FLOPs, which would take many months or years on a few A100s. The paper provides no training time, FLOP count, parallelism strategy, or data-mixture details. Without a credible explanation of how this computation was performed, the empirical results cannot be considered reliable.","section":"Sections 5.1 and 5.2"},{"comment":"The GLUE results are internally inconsistent. Table 2 reports an 8B model with SST-2 at 80.13, MRPC at 73.01, and MNLI at 74.79, while Table 3, described as fine-tuning after C4 pre-training, reports 95.89, 88.42, and 90.02 on the same tasks. The text in Section 5.2 says the 8B configuration 'outperforms the ST baseline on 6 out of 8 GLUE tasks, with the exception of the MRPC and SST-2 tasks,' which matches Table 2 but not Table 3, where CoLA and QNLI are the exceptions. These tables appear to describe different models or protocols, and the paper does not reconcile them.","section":"Tables 2 and 3, Section 5.2"},{"comment":"The adaptive-allocation result is largely built into the definitions. Equations (2)-(3) define K_i as a non-decreasing function of L_i, so 'important tokens receive more experts' holds by construction; the empirical question is whether the learned importance L_i from the lightweight evaluator in Eq. (19) tracks semantic importance. The paper does not validate the evaluator against any ground-truth importance measure, and the Table 5 ablation replaces adaptive routing with random forwarding rather than with a static equal-allocation baseline. Thus the experiments do not establish that the importance-based allocation rule, rather than the extra capacity or the routing signal generally, is responsible for the reported gains.","section":"Section 3, Eq. (19), Table 5"},{"comment":"The complexity claim is internally inconsistent. The text says GMHA reduces attention complexity from O(N^2) to O(Nw), but the last paragraph of Section 3 says it reduces complexity to O(N^2/g). If tokens are split into g groups and attention is computed within each group, the cost is O(N^2/g), which is still quadratic in N; only the sliding-window mask of Eq. (10) yields O(Nw). The paper should state which mechanism is responsible for linear scaling and provide actual FLOP counts, because the long-sequence efficiency argument depends on this distinction.","section":"Section 3, Eqs. (9)-(10)"}],"minor_comments":[{"comment":"The caption says 'B and E represent millions, billions, and experts,' but B is used both for billion and for baseline; the caption and column headers should be clarified.","section":"Table 2 caption"},{"comment":"The text refers to 'MiniCPM4' as the model whose benchmarks are being selected, although the paper's model is DASG-MoE; this appears to be a copy-paste error.","section":"Section 5.1"},{"comment":"Section 5.5 says the threshold-refinement results are shown in Table 6, but Table 6 is the grouping ablation; the threshold experiments appear in Table 4, and the cross-references are incorrect.","section":"Section 5.5"},{"comment":"Several references are mismatched: [59] and [69] both cite DenseNet for unrelated claims, and [66] cites a linear-attention theory paper for the source of Hugging Face ST model weights.","section":"References"},{"comment":"Equation (8) uses g(C_i) on the left but Eq. (5) defines g(C_i,T_j), and the right-hand side mixes scalar and vector quantities; the notation should be made consistent.","section":"Section 3, Eq. (8)"},{"comment":"A paragraph about dynamic routing and group ablation is duplicated nearly word-for-word between Sections 5.2 and 5.6; the repeated text should be removed.","section":"Sections 5.2 and 5.6"}],"recommendation":"reject","confidential_remarks":"The manuscript's reported experimental program is not credible as described: the claimed long-sequence evaluation does not exist, the pretraining scale is incompatible with the stated hardware, and the GLUE tables are mutually inconsistent. I see no path to acceptance without a complete experimental redesign and a new round of evaluation at a credible scale, so rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The architecture is a plausible recombination of known pieces: grouped/sliding-window attention, top-k MoE, a two-tier expert structure, and threshold-based routing. The writing is clear enough that a reader can understand what is proposed, and the ablation on the number of attention groups is systematic. The paper also identifies a real limitation of standard MoE models, fixed expert counts per token, and cites the relevant prior work by name, including Switch, GLaM, and expert-choice routing.\n\nThat is where the credit ends. The central claim, that DASG-MoE outperforms SOTA on long-sequence benchmarks, is never tested. Every reported experiment uses short-text tasks: GLUE, MMLU, CMMLU, CEval, BBH, GSM8K, MBPP, HumanEval. There is no LongBench, no LRA, no SCROLLS, nothing with long context. So even if every number in the paper were correct, they would not support the abstract's claim.\n\nThe experimental section has more serious problems. Section 5.1 says the infrastructure is 8 NVIDIA A100 GPUs, and Section 5.2 claims pre-training an 8B model on 5T tokens and a 3B model on 2.2T tokens. That is not credible by a large margin. A 3B model on 2.2T tokens alone would take tens of thousands of A100-hours, likely far more, and 5T tokens for an 8B model is even further from what eight GPUs could do in any reasonable timeframe. Something is wrong with the description or the results did not come from that setup.\n\nThe GLUE results are internally inconsistent. Table 2 shows SST-2 around 80 and MNLI around 70 for the 8B model, which is far below standard fine-tuned baselines, while Table 3 shows SST-2 at 95.89 and MNLI at 90.02 for what appears to be a comparable setup. The paper never explains why the same GLUE tasks produce such wildly different numbers. That alone undermines confidence in the measurements.\n\nThere is also a genuine circularity in the adaptive-routing claim. Equations 2 and 3 define K_i as a non-decreasing function of importance, so \"important tokens get more experts\" holds by construction. The thresholds are manually tuned, and the evaluator is trained jointly, making it hard to separate what is learned from what is imposed. The paper never compares against expert-choice routing or BASE layers, which already allow non-uniform allocation, so the claimed advantage over the fixed-top-k baseline is not evidence of a novel mechanism.\n\nCode, data, and hyperparameters are missing. The reference list is sloppy, with several citations that do not actually support the statements they are attached to. This is not a paper where heavy revision would fix the problems, because the load-bearing evidence is absent and part of it is not plausible. I would desk reject it.\n\nRecommendation: reject, and do not send to reviewers unless you want a documented verdict for a misconduct or integrity file. The paper is a useful negative example of how an architecture paper can fail its own claim, but it does not deserve referee time.","headline":"A well-described recombination of known MoE/attention ideas whose central empirical claim is untested: no long-sequence benchmark anywhere, impossible compute claims, and internally inconsistent GLUE tables.","tokens_in":17418,"tokens_out":2498,"would_cite":false,"duration_ms":23219,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Token importance decides expert count, beating fixed top-k","keywords":["mixture of experts","dynamic routing","token importance","grouped multi-head attention","long-sequence modeling","dual-scale shared experts","adaptive routing","sparse activation"],"falsifier":"Compute the wall-clock time implied by 5 trillion tokens on eight accelerators; if a single 8-billion-parameter run requires many months, the training curves in the paper cannot come from the described setup. A smaller reproduction at the stated 0.6-billion-parameter scale, with the adaptive router enabled versus disabled, should either restore or erase the reported benchmark deltas.","tokens_in":16320,"feed_emoji":"🧠","tokens_out":8854,"duration_ms":76737,"temperature":0.7,"pith_summary":"The paper proposes DASG-MoE, a mixture-of-experts transformer in which the number and depth of experts assigned to a token depend on an importance score extracted from the attention distribution rather than a fixed top-k. The authors claim this changes the efficiency-accuracy tradeoff: low-information tokens use one lightweight expert while significant tokens can use several, and splitting the sequence into sixteen windowed groups cuts the attention cost. The supporting experiments report that the model beats the baseline architecture on six of eight tasks in an established language-understanding benchmark, with a maximum gain of 4.64 points, and that it also posts the highest average in the paper's broad comparison of open models. If correct, this offers a practical way to scale long-context mixture-of-experts models without paying full attention cost or wasting expert capacity on trivial tokens. The main caveat, visible in the setup section, is that the reported pre-training scale is hard to reconcile with the described hardware.","feed_headline":"Token importance decides expert count, beating fixed top-k","feed_subtitle":"A mixture-of-experts model wins 6 of 8 language-understanding tasks by sending important tokens to deeper experts.","key_machinery":"The argument turns on three coupled modules. Grouped Multi-Head Attention (GMHA) splits the input sequence $X \\in \\mathbb{R}^{N\\times d}$ into $G=16$ subgroups, applies sliding-window multi-head attention inside each group with a mask $M_{\\text{window}}$, passes each group's output through its own MLP, and concatenates the results; this is claimed to cut attention cost from $O(N^2)$ to $O(N^2/G)$ with a local window of size $w$. Dual-Scale Shared Experts (DSSE) provide two tiers: eight shallow one-layer-MLP experts for low-dimensional features and eight deep three-layer-MLP experts for complex semantics, with the deep experts' first layer copied from pre-trained shallow weights and frozen. Adaptive Dynamic Routing (ADR) connects the two: a lightweight two-layer evaluator maps each token's attention weights to an importance score $I_i$ and a complexity score $C_i$, a global router selects the shallow or deep module using a softmax decision trained with time and accuracy rewards, and a local router then activates the top two experts inside the chosen module. A threshold rule with $\\theta_s=0.3$ and $\\theta_d=0.7$ decides when a token is simple enough for shallow-only handling or complex enough for the deep module, and the number of experts allocated grows with $I_i$ through $K_i = \\lceil L_i K \\rceil$.","core_discovery":"On its own terms, the paper's central discovery is that routing by token importance is enough to improve both accuracy and efficiency over a fixed-top-k mixture-of-experts baseline. In the pre-training comparison, DASG-MoE outperforms the baseline on six of the eight benchmark tasks, with the largest improvement of 4.64 points in accuracy or F1 at the largest tested size, and the average scores move from 66.72 to 68.12. In the fine-tuning comparison, the model again wins six of eight tasks, losing only on the grammatical-acceptability and question-inference tasks. The paper also presents a broad comparison in which the instruction-tuned model averages 81.6 across eight evaluation suites, above the listed comparison models, and the ablation studies attribute the gains to the adaptive routing module and to the sixteen-group attention setting.","pith_inferences":["The same importance evaluator could plausibly be reused to prune tokens or skip attention heads, since the paper only uses it to choose expert count and depth.","The described compute budget suggests either a much larger cluster than stated or a different experimental lineage; reproducing the smallest reported run at the stated token count would resolve which.","A task-conditional routing prior, such as forcing paraphrase tasks to use mostly shallow experts, would likely close the observed paraphrase gap; the paper's threshold analysis points there but stops short of testing that variant.","The reported $O(N^2/G)$ complexity claim should be checked against the actual sliding-window implementation, because the mask still limits each token to a window of size $w$ and the group count mainly changes parallelism and aggregation."],"forward_implications":["Fixed top-k routing is not required: a token-importance-dependent expert count can raise average benchmark accuracy while cutting computation on low-information tokens, if the reported comparisons hold.","Sixteen sequence groups is the reported sweet spot for grouped windowed attention; using 32 or 64 groups degrades accuracy, with paraphrase detection suffering the most.","The failure on the paraphrase and sentiment tasks is attributed to over-routing simple tokens into deep experts, so routing thresholds must be task-aware rather than global.","Scaling the architecture from 0.6B to 8B parameters both accelerates training convergence and increases the margin over the baseline, supporting the claim that adaptive allocation scales well.","Because the framework is described as generic, it can be dropped into other mixture-of-experts transformer variants, not only the baseline it was tested in."],"supporting_citations":[{"why":"Introduces the sparse expert-layer formulation that the proposed architecture extends.","marker":"[5]"},{"why":"Supplies the baseline mixture-of-experts transformer into which the proposed modules are inserted and against which the main comparisons are made.","marker":"[6]"},{"why":"Exemplifies the fixed top-2 routing strategy the paper argues is inefficient.","marker":"[13]"},{"why":"Provides an alternative expert-choice routing scheme used as a contrast point for dynamic allocation.","marker":"[16]"},{"why":"Supplies the paraphrase corpus whose sensitivity to deep-expert routing the paper analyses.","marker":"[35]"},{"why":"Supplies the large crawled corpus used in the fine-tuning comparison.","marker":"[41]"},{"why":"Defines the eight-task benchmark where the main six-of-eight result is measured.","marker":"[44]"},{"why":"Provides the pretrained baseline weights used as the shared starting point for fine-tuning.","marker":"[66]"}],"fun_headline_variants":["Token importance decides expert depth, beats fixed top-k","Adaptive routing lifts MoE accuracy beyond fixed top-k","Important tokens unlock deeper experts, improving MoE","Dynamic expert routing wins 6 of 8 tasks in MoE","Token-aware expert selection outdoes static top-k"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported pre-training of an 8-billion-parameter model on 5 trillion tokens, and a 3-billion-parameter model on 2.2 trillion tokens, is attributed to a setup of eight high-end accelerators, a scale that does not obviously fit that hardware, and all empirical comparisons rest on that premise.","fun_headline_variants_meta":{"raw":{"variants":["Token importance decides expert depth, beats fixed top-k","Adaptive routing lifts MoE accuracy beyond fixed top-k","Important tokens unlock deeper experts, improving MoE","Dynamic expert routing wins 6 of 8 tasks in MoE","Token-aware expert selection outdoes static top-k"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000823,"raw_usage":{"total_tokens":3607,"prompt_tokens":958,"completion_tokens":2649,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2571}},"tokens_in":574,"tokens_out":2649,"duration_ms":17912,"temperature":1.0,"reasoning_tokens":2571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:27:41.662494+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the wall-clock time implied by 5 trillion tokens on eight accelerators; if a single 8-billion-parameter run requires many months, the training curves in the paper cannot come from the described setup. A smaller reproduction at the stated 0.6-billion-parameter scale, with the adaptive router enabled versus disabled, should either restore or erase the reported benchmark deltas.","supporting_citations":[],"review_version":1}