{"id":"03ef8efa-b473-4f2e-9168-8fb73dbc47b4","arxiv_id":"2505.10475","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Running P parallel streams of a language model with learned prefixes gives performance comparable to multiplying parameters by about k log P + 1, with k around 0.33 to 0.39.","lead":"Parallel scaling (ParScale) runs the same language model several times in parallel on slightly varied inputs, then combines the results. The paper claims this improves model capability almost as if the model had more parameters, but with much less memory growth.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing control: ParScale gains are not isolated from the added prefix/MLP parameters; a single-stream adapter with equal added parameters could reproduce the O(log P) curve.","rationale":"The reader's weakest_assumption is the adapter/computation attribution, and I agree it is the most load-bearing concern. The paper's central claim is not merely that ParScale improves models—that is well supported by the 24-run fits and downstream tables—but that the improvement is quantitatively equivalent to scaling parameters by O(log P), which requires the active ingredient to be the parallel computation. The implementation bundles P forward passes with P learned prefixes and a learned aggregator, so without a single-stream control the equivalence is underdetermined. The pivot experiments provide partial support: at fixed P, adding LoRA and BitFit changes the loss by only about 0.1% (Table 6), suggesting extra adapter parameters are not by themselves the main driver at fixed P. But this is not the same as varying P at fixed adapter budget, and the PEFT experiment (Figure 6c) shows the adapters can move performance on their own when the backbone is frozen. The theoretical Proposition 1 does not resolve this: it derives a loss formula in terms of a correlation parameter rho, but the empirical O(log P) form is fitted, not derived, and the paper acknowledges in Section 6 that the relationship between diversity and log P remains an open question. Therefore the missing ablation is the decisive gap. Because the authors release code and checkpoints, the proposed control is feasible. The conditional verdict is appropriate: accept with the requirement of this ablation and out-of-sample checks rather than reject, since the empirical phenomenon is real and the concern is about interpretation, not integrity.","tokens_in":36989,"tokens_out":9156,"duration_ms":90092,"concrete_test":"Train a 1.6B single-stream model on Stack-V2-Python with the same 42B tokens and hyperparameters as Section 3.2, but with an adapter whose trainable parameter count and placement match ParScale P=8: for instance, 8 prefix blocks of 48 tokens per layer that are summed before attention (or concatenated into a single stream) plus an output MLP of the same size as the aggregation MLP. Compare the final loss to the P=8 ParScale run and to the P=1 baseline. If the single-stream adapter's loss matches ParScale P=8 within the fitting errors reported in Tables 10 and 11, the observed gains are attributable to added parameters and the O(log P) interpretation collapses; if it matches P=1, parallel computation is the active ingredient and the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that P parallel streams behave like N(k log P + 1) parameters—rests on attributing the loss reduction to parallel computation. But the ParScale implementation adds roughly 0.2% trainable parameters per stream, namely prefix embeddings plus an aggregation MLP (Section A), and the scaling-law fits in Section 3.2 compare P>1 models (with these adapters) only against P=1 models that explicitly have no added parameters. The pivot experiments in Table 6 vary the adapter type at fixed P and show only small differences, which weakens the naive version of this confound: at P=2, adding LoRA and BitFit changes the loss by only about 0.1%. However, these experiments never hold the added-parameter count fixed while varying P, and they never run a single-stream model with the same adapter parameter budget. The PEFT result in Section 4.2 (Figure 6c) shows that tuning only the introduced parameters on a frozen backbone improves code generation substantially, demonstrating that these adapters are not inert. If a single-stream adapter with the same added parameters (same number of prefix tokens and an MLP of matching size) achieves the same loss as ParScale at P=2 or P=8, then the fitted multiplier (k log P + 1) is a property of the adapters, not of parallel compute, and the inference-efficiency comparison in Section 3.3 loses its basis. This is the single most load-bearing assumption because the entire framing—'scaling computation', equivalence to parameter scaling, and the memory/latency claims—depends on computation being the active ingredient.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ParScale, a method that runs P parallel forward passes of a language model on P learnable input transformations (prefix embeddings) and aggregates the outputs with a dynamic weighted MLP. The central claim is a new scaling law, L = (A / (N (k log P + 1)))^alpha + E, which implies that increasing the number of parallel streams P is equivalent to multiplying the model's effective parameter count by a factor k log P + 1. The authors validate this law with 24 from-scratch pre-training runs per dataset on Stack-V2-Python and Pile (42B tokens), report downstream task improvements, extend the method to a two-stage 1T-token training recipe and to off-the-shelf Qwen-2.5 models, and argue that ParScale gives large inference memory and latency savings relative to parameter scaling at equal model capacity.","tokens_in":37295,"tokens_out":8488,"duration_ms":86489,"significance":"If the equivalence claim holds, the paper identifies a genuinely new scaling axis — parallel computation at roughly constant parameters — and supports it with large-scale experiments, public code and checkpoints, and consistent downstream trends. The 24-run fits per corpus have very high R^2, and the two-stage and PEFT results are practically useful. However, the significance is conditional: the attribution of the gains to parallel computation rather than to the newly introduced adapter parameters is not yet established, and the theoretical proposition does not actually derive the logarithmic form that carries the paper's headline claim. The empirical law is therefore a promising observation rather than a validated scaling law.","major_comments":[{"comment":"The loss reduction attributed to parallel computation is not isolated from the parameters introduced by the method. In the scaling-law fits, every P>1 model contains prefix embeddings plus an aggregation MLP (about 0.2% additional parameters per stream), while the P=1 baseline has no added parameters. The pivot experiments in Table 6 vary the transformation type only at fixed P, and Figure 6(c) shows that tuning the introduced parameters on a frozen backbone yields substantial code-generation gains (Pass@1 from 47.4 at P=1 to 53.0 at P=8). A necessary control is a single-stream P=1 model that receives the same number of additional parameters, e.g., the same prefix length and an MLP of matching size. If such a single-stream adapter reproduces the loss curve, the fitted multiplier (k log P + 1) reflects added capacity rather than parallel computation, and the parameter-equivalence interpretation in Section 3.3 loses its basis. The authors should run this control or explicitly restrict the claim to ParScale-with-adapters.","section":"Section 3.2, Table 6, Figure 6(c)"},{"comment":"The logarithmic form is assumed, not derived. Proposition 1 expresses the P-dependent factor as P^{1/alpha} * DIVERSITY = [P / ((P-1)rho + 1)]^{1/alpha}. If rho is constant across P, this factor saturates as P grows, so the approximation loss tends to a nonzero limit; that is inconsistent with the unbounded log growth in Eq. (5). The paper defines DIVERSITY by replacing this factor with k log P + 1 \"based on the finding of the logarithmic trend\" from only the P in {1,2,4,8} points. With 24 runs and four free parameters (A, k, alpha, E), the high R^2 does not discriminate Eq. (5) from saturating or power-law alternatives; indeed, the appendix fit of Eq. (4) with constant rho also achieves R^2 around 0.996. Since the headline O(log P) claim is the empirical form itself, the authors should either measure rho directly, compare several candidate scaling forms and report model selection, or explicitly present the log law as an empirical observation without theoretical support.","section":"Section 3.1, Eq. (4); Section 3.2, Eq. (5)"},{"comment":"The 22x and 6x efficiency numbers are derived from the llm-analysis analytical framework (Li, 2023), not from measured inference runs, and no validation against real GPU measurements is reported. The paper should state this clearly and ideally validate the memory and latency model on at least one hardware configuration. As written, the abstract and Section 1 present these ratios as achieved results rather than as analytical projections.","section":"Section 3.3, Figure 4"}],"minor_comments":[{"comment":"There are typos: \"scenerios\" in Section 1 and \"bottlenect\" in Section 3.3 should be \"scenarios\" and \"bottleneck\".","section":"Section 1, Section 3.3"},{"comment":"The Taylor expansion step drops first-order and higher-order terms with only a heuristic justification and uses \"~\" loosely; please state the formal assumptions needed for the approximation loss to equal the squared relative residual.","section":"Appendix B, Eq. (9)"},{"comment":"The theoretical derivation assumes equal aggregation weights 1/P, while the implementation uses a dynamic weighted sum; the statement that the full version is \"at least not worse\" is not proven. The numerical comparison in Table 6 is reassuring but should be presented as empirical rather than implied by the theory.","section":"Section 3.1, Table 6"},{"comment":"The fitted k values (0.393 vs. 0.334) are reported without confidence intervals or seed variance; please include uncertainties or at least a statement about the number of seeds so readers can assess whether the difference is significant.","section":"Figure 2, Tables 8-9"},{"comment":"Please clarify whether the P=1 condition in the frozen-backbone PEFT experiment includes any introduced parameters (prefix/MLP) or is the unmodified pretrained model; this detail matters for interpreting the control issue.","section":"Figure 6(c)"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the missing adapter control; if a single-stream model with the same added parameters reproduces the gains, the paper's computational-scaling interpretation and the efficiency comparison both collapse. I still recommend major revision rather than rejection because the control is a feasible, well-defined experiment and the empirical scaling fit is valuable regardless of interpretation. The authors should also be encouraged to temper the theoretical claim, since Proposition 1 does not actually imply the logarithmic law."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Chen et al. propose ParScale: run P copies of a model on P learnable input prefixes and aggregate with a learned weighted sum. The paper's new claim is a scaling law L = A/(N(k log P + 1))^alpha + E, i.e. P streams behave like multiplying parameters by O(log P). The empirical package is real: 24 pretraining runs per dataset, R^2 > 0.99, downstream gains that mostly track P, a 1T-token two-stage validation, and public code/checkpoints. That is more evidence than most scaling-law papers ship with.\n\nThe soft spot is exactly where the stress test points. The P>1 models add prefix embeddings plus an aggregation MLP (~0.2% params per stream); the P=1 baselines add nothing. The pivot experiments in Table 6 show that swapping LoRA/BitFit for prefixes at P=2 changes loss only ~0.1%, which weakens the naive 'any adapter helps' story. But they never run a single-stream model with the same added-parameter budget. The PEFT result in Figure 6c shows the introduced parameters are not inert—tuning them alone on a frozen backbone lifts code performance substantially. If a single-stream adapter matches ParScale's loss at P=2 or P=8, the 'scaling computation' interpretation collapses and the law becomes a statement about parameter-efficient adapters. That control is load-bearing and missing.\n\nThe theory is honest but thin. Proposition 1 leaves rho unmeasured and is derived for equal weights, while the implementation uses dynamic weights. The O(log P) form is assumed after seeing a trend, then fitted to the same 24 runs the trend was read from. So the headline equivalence is a compact description of the data, not a tested prediction. I'd want an out-of-sample fit (untrained N or P) before calling it a law. The 22x/6x inference numbers come from llm-analysis simulation, not hardware; reasonable as estimates, but they should be labeled as such.\n\nAll that said, the central finding—P parallel streams reduce loss roughly logarithmically across model sizes and datasets—looks solid within the trained range, and the paper is open about the open questions. This deserves a serious referee. The revision should add the single-stream adapter control, an out-of-sample prediction, and ideally hardware latency/memory measurements. I would not cite the O(log P) equivalence in my own work yet, but I'd follow the thread.","headline":"A serious empirical scaling-law paper with a load-bearing missing control: the O(log P) equivalence is a fit, not yet a law.","tokens_in":37899,"tokens_out":1951,"would_cite":false,"duration_ms":19218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that running a language model in P parallel streams, with the same weights, matches the capacity of scaling parameters by O(log P).","keywords":["parallel scaling law","language models","scaling computation","classifier-free guidance","model ensemble","inference efficiency","prefix tuning","two-stage pretraining"],"falsifier":"Train a single-stream model that contains exactly the same added prefix and aggregation parameters as the $P=8$ ParScale model, on the same corpus and 42B-token budget, and compare final losses; if the single-stream model matches ParScale's loss, the fitted law is about parameter count, not parallel computation.","tokens_in":36751,"feed_emoji":"🔀","tokens_out":11712,"duration_ms":95627,"temperature":0.7,"pith_summary":"The paper proposes a third scaling axis for language models: instead of adding parameters or adding reasoning tokens, run the same model in $P$ parallel streams, each fed a differently prefixed version of the input, and combine the $P$ outputs with learned weights. It claims that this 'parallel scaling' behaves like parameter scaling: a model with $N$ parameters and $P$ streams approximates a model with $N(k\\log P+1)$ parameters, with $k\\approx0.39$ on a code corpus and $0.33$ on a general corpus. The fitted law $L=(A/(N(k\\log P+1)))^\\alpha+E$ matches 24 pretraining runs per corpus with $R^2\\approx0.998$. If the claim holds, models can be made more capable without proportional memory growth; at batch size 1, a 1.6B model with $P=8$ costs 22x less memory increase and 6x less latency increase than the parameter-scaled model of equal capacity, which matters for edge deployment. The paper also argues that parameters mostly buy memorization while parallel computation mostly buys reasoning, a distinction that reframes how model capacity is allocated.","feed_headline":"P parallel streams buy O(log P) extra parameters","feed_subtitle":"A fitted law predicts 8 parallel passes of a 1.6B model match a 4.4B model at 22x lower memory cost.","key_machinery":"The load-bearing mechanism is parameter-reusing parallel computation: $P$ learnable prefixes (implemented as separate KV caches) create $P$ streams through the same Transformer, and a small MLP on the concatenated stream outputs produces softmax aggregation weights, regularized by label smoothing to keep all streams active. The identity that carries the argument is the substitution $N\\mapsto N(k\\log P+1)$ inside the standard power-law loss curve, converting 'more parallel passes' into 'equivalent parameters.' The theoretical Proposition 1 performs the same substitution in terms of stream-residual correlation $\\rho$, so the empirical fit and the theory share one central object: how much extra effective capacity each stream contributes.","core_discovery":"The central claim is that parallel computation can substitute for parameters at a quantitative rate. On the paper's own terms, the discovery is the parallel scaling law: after training to convergence, the cross-entropy loss of a $P$-stream model fits $L=(A/(N(k\\log P+1)))^\\alpha+E$, so the effective parameter count is multiplied by $k\\log P+1$ with $k=0.39$ on Stack-V2-Python and $k=0.33$ on Pile. The paper also derives a theoretical form $L=(A/(N P^{1/\\alpha}\\,\\mathrm{DIVERSITY}))^\\alpha+E$ with $\\mathrm{DIVERSITY}=[(P-1)\\rho+1]^{-1/\\alpha}$, where $\\rho$ is the correlation between streams' relative prediction residuals; the empirical logarithmic law is a fitted characterization of how that diversity grows with $P$. Downstream tasks show the substitution is uneven: reasoning-heavy code and math gain more than general-knowledge tasks, which the paper interprets as computation driving reasoning and parameters driving memorization.","pith_inferences":["If the effective-parameter substitution is literal, parameter count and parallel compute are partially interchangeable inputs to one scaling law; future fits should include a parallel-compute term alongside $N$ and training data, and compute-optimal allocation would trade one against the other under memory and latency budgets.","The logarithmic law implies a saturating ceiling as $P$ grows; Proposition 1 ties the ceiling to stream-residual correlation $\\rho$, so measuring how $\\rho$ changes at $P=16$ or $32$ would test whether the law extends or bends.","The per-token dynamic stream weighting with label smoothing is functionally soft routing, which suggests that combining ParScale with sparse mixture-of-experts architectures could offset each method's weakness, memory-heavy MoE versus compute-heavy ParScale.","Because the method is architecture-agnostic, the same $k\\log P+1$ law should be testable on vision or multimodal models; a cross-domain comparison of $k$ would reveal whether the computation-versus-parameters trade is universal or domain-dependent."],"forward_implications":["Every doubling of $P$ adds a constant parameter-equivalent boost: with $k=0.39$, $P=8$ gives roughly a $1.8\\times$ larger effective parameter count, and the fitted law says this holds across all tested sizes.","Larger models gain more from parallel scaling: because the multiplier multiplies $N$, a 1.6B model at $P=8$ matches a 4.4B model on code tasks while staying far smaller in memory.","ParScale is memory-efficient at small batch: the same capacity as parameter scaling with 22x less memory increase and 6x less latency increase at batch size 1, which suits edge deployment.","Training cost can be amortized: normal pretraining on 1T tokens followed by 20B ParScale tokens recovers most of the gain, and frozen-backbone variants allow $P$ to be switched at deployment time.","Parallel and serial scaling combine: on GSM8K the $P=8$ model improves by 34% relative to the baseline, and the gain persists when chain-of-thought is added."],"supporting_citations":[{"why":"Supplies the parameter-scaling law $L=(A/N)^\\alpha+E$ that ParScale extends to parallel streams.","marker":"(Hoffmann et al., 2022)"},{"why":"Classifier-free guidance, the two-forward-pass idea that motivates scaling parallel computation.","marker":"(Ho & Salimans, 2022)"},{"why":"Power-law relation between ensemble size and loss, the special case of Proposition 1 when stream residuals are uncorrelated.","marker":"(Lobacheva et al., 2020b)"},{"why":"Prefix-tuning, the input-transformation mechanism used to differentiate parallel streams.","marker":"(Li & Liang, 2021)"},{"why":"Soft prompting, the base technique for creating trainable per-stream input transformations.","marker":"(Lester et al., 2021)"},{"why":"Provides the fitting procedure, hyperparameters, and data-repeat result used to fit and validate the scaling law.","marker":"(Muennighoff et al., 2023)"}],"fun_headline_variants":["Parallel scaling: P streams act like log P extra parameters","P parallel passes match O(log P) more parameters","ParScale: P streams yield O(log P) parameter boost","P streams, O(log P) params, 22x cheaper memory"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper attributes the gains to extra parallel computation, but its controls never compare ParScale to a single-stream model that receives the same small set of added trainable parameters, so the interpretation would collapse if those added parameters alone produced the gains.","fun_headline_variants_meta":{"raw":{"variants":["Parallel scaling: P streams act like log P extra parameters","P parallel passes match O(log P) more parameters","ParScale: P streams yield O(log P) parameter boost","P streams, O(log P) params, 22x cheaper memory"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00019,"raw_usage":{"total_tokens":1375,"prompt_tokens":1013,"completion_tokens":362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":629,"completion_tokens_details":{"reasoning_tokens":292}},"tokens_in":629,"tokens_out":362,"duration_ms":3971,"temperature":1.0,"reasoning_tokens":292,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:08:48.913221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a single-stream model that contains exactly the same added prefix and aggregation parameters as the $P=8$ ParScale model, on the same corpus and 42B-token budget, and compare final losses; if the single-stream model matches ParScale's loss, the fitted law is about parameter count, not parallel computation.","supporting_citations":[],"review_version":1}