{"id":"693efce9-f675-4cbe-afa7-c0209b7a75cb","arxiv_id":"2608.11519","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FLARE++ synthesizes input-dependent routing queries with one extra attention call, preserving FLARE's linear low-rank mixer and improving accuracy on PDE surrogates and Long Range Arena.","lead":"FLARE++ makes a low-rank attention model's routing queries depend on each input sample, instead of using fixed learned queries. It reports about 24% lower error on average on PDE surrogate benchmarks and a 2.3 point gain on Long Range Arena, at the same linear O(N by M) cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The attribution of all gains to dynamic routing is unsupported because FLARE++ adds two C×C projections per head; without a parameter-matched or shared-projection control, extra capacity remains a viable explanation.","rationale":"The reader's weakest_assumption identifies exactly the parameter-matching confound, and I agree that this is the single most load-bearing issue. The mathematical derivation of the rank-M factorization and the exact context-parallel reduction are independently sound, and the complexity analysis is careful about the distinction between operation counts and wall-clock cost. Those parts deserve credit. The empirical numbers, however, are used to support a mechanistic conclusion: input-dependent routing is better than fixed routing. That conclusion requires separating the routing mechanism from the added parameters and computation. The paper acknowledges the parameters differ but asserts without evidence that the difference is attributable to route choice alone. Section C.2 even names the missing control, a shared-projection variant, but does not run it. A single ablation of that kind would settle the attribution concern; multiple seeds and error bars would further strengthen the empirical claim. Because the concern is a missing control rather than an observed contradiction, the evidence is sufficient for a conditional acceptance but not for accepting the central attribution at face value. The reader's CONDITIONAL verdict therefore stands, and no verdict change is needed.","tokens_in":20282,"tokens_out":3212,"duration_ms":38811,"concrete_test":"Run a parameter-matched control on Elasticity and Darcy at the Table 5 settings: implement FLARE++ with tied synthesis projections eK_h = K_h and eV_h = V_h, or with eK and eV frozen at identity, so query synthesis retains its input-dependent SDPA form but no extra C×C parameters are introduced. Compare test relative L2 against FLARE and full FLARE++ over at least three seeds. If the tied variant retains most of the 24% margin, dynamic routing is the cause; if it falls to FLARE-level error, the extra projections explain the gains and the central claim must be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is the attribution statement in Section 3.2: because only depth and width are matched, \"any accuracy difference is therefore attributable to how the M routes are chosen.\" This inference is not secured. FLARE++ adds two C×C projection matrices per head in Eq. 7 and one additional SDPA call per mixer. At C=128, H=8, B=8, the extra projections alone add on the order of 2·8·128²·B ≈ 2.1M parameters, before counting the additional attention path. The 24% average PDE gain, the 2.3-point LRA gain, and every cell of the (M,B) grid are therefore consistent with the alternative explanation that the extra parameters, or the additional nonlinear aggregation in query synthesis, improve accuracy while the routing mechanism itself is neutral. Section C.2 explicitly names a shared-projection variant that would remove the extra projections and states that its accuracy cost is unknown; that control is not run. Without it, the central claim that dynamic routing, rather than added capacity, is responsible for the improvements is unsupported, even if the raw numbers reproduce exactly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FLARE++, a low-rank attention mixer that replaces FLARE's learned latent routing queries with input-conditioned queries synthesized by one additional scaled dot-product attention call over learned seeds (Eqs. 8–11). The authors claim this preserves FLARE's rank-M encode–decode factorization and O(NM) complexity while improving average test relative L2 error by 24% on five PDE benchmarks (Table 1) and raising the Long Range Arena average by 2.3 points (Table 6). They also provide an exact token-sharded multi-GPU implementation whose collective payload is independent of N, with measured parallel efficiency near unity over four ranks (Table 2, Appendix A). The paper presents a joint (M,B) ablation on Elasticity and Darcy in which FLARE++ beats FLARE in all 21 cells, and it reports that dynamic routing can reach a given error at half the residual depth of FLARE, with the caveat that wall-clock end-to-end savings are not directly measured.","tokens_in":20453,"tokens_out":2206,"duration_ms":25637,"significance":"If the empirical claims hold, FLARE++ is a useful contribution: it is a clean, well-motivated modification of a known low-rank mixer, preserves the explicit rank-bound interpretation and linear complexity, and the experimental sweep across five PDE benchmarks and five LRA tasks is broader than what is typical for a mixer study. The exact token-sharded encoder with global softmax normalization and its backward pass (Eqs. 13–40) is a solid systems contribution with machine-checkable algebra. The paper also ships quantitative efficiency measurements over three orders of magnitude in N. The main caveat is that the paper's headline attribution—that gains come from dynamic routing rather than added parameters—is not supported by the experiments as reported, because the comparison is not parameter-matched and the paper explicitly identifies a shared-projection control that would resolve the issue but does not run it.","major_comments":[{"comment":"The attribution statement that 'any accuracy difference is therefore attributable to how the M routes are chosen' is not secured. FLARE++ adds two C×C projection matrices per head plus one additional SDPA call per mixer, so at C=128, H=8, B=8 the extra projections alone add on the order of 2·8·128²·8 ≈ 2.1M parameters, before counting the extra attention path. The 24% average PDE gain, the 2.3-point LRA gain, and every cell of the (M,B) grid are consistent with the alternative explanation that the extra capacity, or the additional nonlinear aggregation in query synthesis, is responsible for the improvement while routing itself is neutral. Section C.2 explicitly names a shared-projection variant that would remove the extra projections and states that its accuracy cost is unknown; that control is not run, and without it the central claim that dynamic routing—rather than added capacity—drives the gains is unsupported.","section":"Section 3.2, Eqs. (7)–(11)"},{"comment":"The main accuracy tables report no confidence intervals, seed counts, or standard deviations, and Section C.1 states that 'every number in this comparison is a single seed.' With differences as small as 0.02–0.05 on Airfoil and Pipe (0.52 vs 0.57 and 0.34 vs 0.51), a single seed cannot establish that the improvement is real rather than noise, especially given the run-to-run spread the paper itself cites in Appendix A.7. The claim that dynamic routing wins in all 21 (M,B) cells needs at least repeated-seed means with error bars or a paired-seed analysis before it can be called a consistent effect.","section":"Table 1 and Section C.1"},{"comment":"The LRA comparison is presented as evidence that 'the same backbone gains 2.3 points of average accuracy from dynamic routing alone,' but many baseline rows and the FLARE row are reproduced from the authors' own prior papers rather than re-run in this work, and the protocol states that FLARE++ uses a 'matched configuration' without specifying whether the FLARE row is the same configuration. Since the paper's broader claim is about a mechanism rather than a tuned architecture, the baseline provenance and exact configuration matching should be stated per row, and the FLARE++ numbers should be accompanied by seed-level statistics, particularly on Image and Pathfinder-32 where gains of 5.2 and 3.5 points are the largest.","section":"Section 4.2, Table 6"},{"comment":"The depth-substitution claim ('FLARE++ reaches a lower error than FLARE at a shallower residual depth') is presented as an efficiency argument, but the paper explicitly notes that end-to-end training-time savings are not measured. Because FLARE++ incurs 1.3–1.5× FLARE's step time, the claim that halving depth reduces wall-clock cost is an extrapolation from operation counts plus a per-step-time ratio; this should either be verified with actual end-to-end training-time measurements at matched accuracy or be reworded as a per-block cost claim.","section":"Section 5.2 and Section C.2"}],"minor_comments":[{"comment":"'This removes the restriction withFLARE++' has a missing space; the typo appears in the first sentence of the abstract.","section":"Abstract"},{"comment":"The symbol B is used both for batch size (Eq. 14) and for the number of blocks in Table 4 and Figure 3. The paper warns about this in Appendix A.1, but the main text still switches between the two meanings without explicit visual distinction; consider renaming one of them.","section":"Section 3.3, Eq. (13)"},{"comment":"The benchmark summary reproduces Plasticity from the FLARE paper, but Plasticity is never used in any experiment; either remove the row or state why it is listed.","section":"Table 3"},{"comment":"The precision-sensitivity figure is based on single-seed runs and the text says the comparison against full self-attention is precision-dependent; this is reported honestly, but the figure caption should repeat the single-seed caveat rather than leaving it only in the main text.","section":"Section C.1, Figure 4"},{"comment":"The notation eQ_h, eK_h, eV_h is introduced without a placeholder subscript for the head in the text of Eq. (7)–(8); the head index is clear from context but should be spelled out in the surrounding prose.","section":"Section 3.2"},{"comment":"The table caption says bold and underline mark best and second-best, but the text only discusses the best entry; the second-best entries are not described, which weakens the interpretation of the ranking.","section":"Section 4.1, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is technically sound in its formal claims: the rank-M bound, the O(NM) complexity, and the exact distributed encoder derivation are all correct as far as I can tell. My concern is not correctness of the math but the strength of the empirical attribution. The authors themselves identify the shared-projection control in Section C.2 and do not run it; given that this control directly tests the paper's central claim, I would treat a revision that adds the control, plus repeated-seed statistics for the main tables, as the appropriate path. I would also note that many baseline numbers are recycled from the authors' prior FLARE work; this is acceptable if documented, but the provenance should be explicit in the table caption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The genuinely new thing here is that FLARE++ builds its low-rank routing queries on the fly by running an extra FLARE-style encode call over the input. That is a clean way to make the compression template input-dependent while keeping the explicit rank-M factorization and O(NM) complexity. The empirical picture is consistent: lower relative L2 error than fixed-template FLARE on all five PDE benchmarks (24% average), and a 2.3-point average gain on LRA across all five tasks.\n\nThe paper is also unusually honest about cost. It reports FLOP counts, wall-clock time (1.3–1.5× FLARE), peak memory (1.18×), and explicitly states that the mixers are not parameter-matched. The context-parallel sharding is described in enough mathematical detail to reimplement, and the efficiency measurements are plausible. The (M,B) sweep covers 21 cells and shows a consistent trend.\n\nWhere it wobbles is the causal claim. Section 3.2 says that because depth and width are matched, \"any accuracy difference is therefore attributable to how the M routes are chosen.\" That inference doesn't hold. You add two C×C projections per head and one extra attention call. The stress-test calculation is right: even at modest settings that's over two million extra parameters before counting the attention path. So the gains could in principle come from extra capacity or from the additional nonlinear aggregation, not from dynamic routing per se. The paper names a shared-projection variant that would remove the confound and says its accuracy cost is unknown, but does not run it. That is the key missing control. Also, the main tables have no confidence intervals or seed counts, and there is no code or data release. Those are not fatal, but they keep the headline numbers from being more than plausible.\n\nMy recommendation: the paper deserves peer review. The construction is novel, the evaluation is broad, and the authors' own caveats tell a reviewer exactly what to ask for. The number one request should be a parameter-matched FLARE baseline or the shared-projection ablation, plus a few seeds and error bars. If the attribution survives that control, the method is a solid contribution; if it doesn't, the mechanism is still worth publishing as an efficient way to get input-dependent routing, but the marketing around \"dynamic routing wins\" would need to soften.","headline":"FLARE++ is a genuinely novel dynamic low-rank routing mechanism with consistent empirical gains, but the central attribution of those gains to dynamic routing is unsupported until a parameter-matched control is run.","tokens_in":21021,"tokens_out":3287,"would_cite":true,"duration_ms":31464,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FLARE++ computes its M routing queries from the input itself, keeping FLARE's rank-M factorization and linear cost while cutting relative error by 24 percent on average across five PDE benchmarks.","keywords":["FLARE++","low-rank attention","dynamic attention routing","PDE surrogate models","token mixing","scaled dot-product attention","context parallelism","Long Range Arena"],"falsifier":"Train a fixed-query control: FLARE with the same two extra $C\\times C$ projections and one extra softmax attention call, but with those queries left constant after training, matched in parameter count; if that control matches FLARE++'s error on Elasticity, Darcy, Airfoil, Pipe, and DrivAerML-40K, then dynamic routing itself is not what drives the reported gains.","tokens_in":20014,"feed_emoji":"🔀","tokens_out":10533,"duration_ms":92266,"temperature":0.7,"pith_summary":"FLARE++ sets out to show that the $M$ latent routing queries in FLARE's low-rank attention should be built from the current input rather than fixed as learned parameters. It reuses FLARE's own encode call, driven by learned seed queries, to synthesize input-conditioned queries, then uses those queries in the same encode–decode pair that compresses and redistributes the $N$ tokens. At matched depth and width, this one architectural change lowers relative $L^2$ error by 24% on average across five PDE surrogate benchmarks and raises the Long Range Arena average by 2.3 points. The significance, if the result holds, is that a low-rank attention bottleneck is spent more effectively when the compression template adapts per sample and per layer, without changing the $O(NM)$ scaling.","feed_headline":"Dynamic token routing cuts FLARE's PDE error by 24%","feed_subtitle":"Input-built queries win on all five PDE benchmarks and lift Long Range Arena by 2.3 points.","key_machinery":"The load-bearing object is the synthesized routing-query set $Q_h(X)$, produced by an extra SDPA encode call from learned seed queries $\\tilde Q_h$ and separate key/value projections of the input. With $W_{\\mathrm{enc},h}(X)$ and $W_{\\mathrm{dec},h}(X)$ the softmax routing factors built from $Q_h(X)$, the mixer composes them as $Y_h = W_{\\mathrm{dec},h}(X)W_{\\mathrm{enc},h}(X)V_h$, an explicit rank-at-most-$M$ operator implemented by three fused SDPA calls. This keeps FLARE's structure while making the compression template field-dependent; the mixer costs $O(N(5C^2+3MC))$ time and $O(NC)$ space, and the context-parallel encoder reduction uses only the latent log-normalizers and outputs, with a collective payload independent of $N$.","core_discovery":"On the paper's own terms, the discovery is that replacing fixed learned routing queries with input-conditioned ones makes low-rank attention a better token mixer while preserving the explicit rank-$M$ encode–decode structure. The synthesized queries are $Q_h(X) = \\mathrm{SDPA}(\\tilde Q_h, \\tilde K_h, \\tilde V_h)$, one extra encode call that gathers the input tokens through learned seeds; those queries then define both factors of the routing matrix, so the effective input-to-input attention matrix still has rank at most $M$. Empirically, FLARE++ records the lowest relative $L^2$ error on all five PDE benchmarks, cutting FLARE's error by 9–41% (24% on average) and Transolver-3's by 18–44% (31% on average). It also improves on FLARE on all five Long Range Arena tasks, raising the average from 58.08 to 60.36, and in ablation it wins in every matched $(M,B)$ cell while FLARE saturates or degrades with larger latent budgets on some benchmarks. The paper reads the depth ablation as showing that dynamic routing substitutes for depth: FLARE++ at half the residual depth is more accurate than FLARE at full depth in the configurations swept.","pith_inferences":["If the attribution holds, the same one-extra-encode recipe could be applied to other fixed-template mixers, such as PerceiverIO's latent array or Linformer's projection, turning any learned compression template into an input-dependent one; FLARE++ is the first test of that pattern.","A direct test the paper leaves open is a parameter-matched fixed-query FLARE control: adding the same two $C\\times C$ projections and one extra attention call while keeping the queries constant would separate the effect of dynamic routing from the effect of extra capacity.","The finding that fixed queries saturate in $M$ while input-built queries keep improving suggests that in practice the latent budget $M$ can be set more aggressively with dynamic routing, trading per-block cost for fewer blocks at equal or better accuracy."],"forward_implications":["At matched depth and width, dynamic routing beats fixed-query routing on every measured benchmark, with the lowest error on all five PDE tasks and an average 24% reduction in relative $L^2$ error over FLARE.","Dynamic routing substitutes for depth: in the joint sweep, FLARE++ at four residual blocks is more accurate than FLARE at eight, which is the mechanism the paper proposes for recovering its 1.3–1.5$\\times$ step-time overhead.","Input-conditioned routing uses added latent capacity better: FLARE++ keeps converting latent budget into accuracy where fixed-query FLARE saturates or degrades.","The mechanism transfers outside PDE surrogates: FLARE++ raises the Long Range Arena average from 58.08 to 60.36, above the full self-attention row, with gains on all five tasks.","Token-sharded context parallelism keeps parallel efficiency at or near unity over four ranks, so the low-rank mixer can train on meshes that do not fit on one device without ever gathering the full token sequence."],"supporting_citations":[{"why":"Defines FLARE, the fixed-query encode–decode mixer and the five PDE benchmarks that FLARE++ modifies and compares against.","marker":"Puri et al., 2026"},{"why":"Supplies scaled dot-product attention, the primitive every routing call is built from, and the full self-attention reference.","marker":"Vaswani et al., 2017"},{"why":"Supplies the Long Range Arena benchmark tasks and protocol used for the secondary evaluation.","marker":"Tay et al., 2021b"},{"why":"Supplies Transolver, the latent-workspace baseline family compared under the matched backbone.","marker":"Wu et al., 2024"},{"why":"Supplies PerceiverIO, the latent-workspace architecture compared as a complete model in Table 1.","marker":"Jaegle et al., 2021a"},{"why":"Supplies the fused FlashAttention-style SDPA kernel on which the linear-space and context-parallel complexity claims rely.","marker":"Dao, 2024"}],"fun_headline_variants":["FLARE++ routing with input-built queries cuts errors 24%","Dynamic routing beats fixed latents: 24% lower PDE error","FLARE++ lifts LRA score 2.3 points via dynamic attention","Less depth, more accuracy: FLARE++ dynamic routing wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands or falls on the assumption that the accuracy gain comes from making the routing queries depend on the input, not from the extra parameters and extra computation the method adds, and the paper does not test a fixed-query FLARE with the same added parameters.","fun_headline_variants_meta":{"raw":{"variants":["FLARE++ routing with input-built queries cuts errors 24%","Dynamic routing beats fixed latents: 24% lower PDE error","FLARE++ lifts LRA score 2.3 points via dynamic attention","Less depth, more accuracy: FLARE++ dynamic routing wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000307,"raw_usage":{"total_tokens":1801,"prompt_tokens":1036,"completion_tokens":765,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":689}},"tokens_in":652,"tokens_out":765,"duration_ms":8094,"temperature":1.0,"reasoning_tokens":689,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:36:07.894882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a fixed-query control: FLARE with the same two extra $C\\times C$ projections and one extra softmax attention call, but with those queries left constant after training, matched in parameter count; if that control matches FLARE++'s error on Elasticity, Darcy, Airfoil, Pipe, and DrivAerML-40K, then dynamic routing itself is not what drives the reported gains.","supporting_citations":[],"review_version":1}