{"id":"831c0654-f0b5-4ba5-901c-6b12b6340a00","arxiv_id":"2509.23722","paper_version":2,"verdict":"REJECT","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Co-optimizing model partition, placement, and workload scheduling for pipeline-parallel LLM training is claimed to improve throughput by 1.15 to 1.44x (abstract) or up to 2.14x (body).","lead":"A paper submitted under the title OctoPipe actually presents a system named AdaPtis that trains large language models faster on GPU clusters. The reported speedups are 1.15 to 1.44x in the abstract and up to 2.14x in the body, though the two parts of the paper disagree on the system's name.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Beyond the OctoPipe/AdaPtis mismatch, the claim rests on Algorithm 1's T_d = C_d + BubbleTime(d) - OverlapTime(d), with BubbleTime/OverlapTime never specified and fidelity shown only for Nemotron-H at SeqLen=4K; the surrogate is untested in the Gemma/DeepSeek and SeqLen != 4K regimes where speedups","rationale":"The reader's weakest-assumption identification matches the core technical vulnerability: the entire co-optimization pipeline is only as trustworthy as the Pipeline Performance Model, and the paper validates that model only on Nemotron-H at SeqLen=4K while claiming speedups across Gemma, DeepSeek, and a wide range of sequence lengths. The naming inconsistency between OctoPipe and AdaPtis is a separate, serious presentation/attribution problem: a reader cannot tell whether the evaluated artifact is the system named in the title and abstract. Neither issue proves the approach is wrong; the idea is plausible, the implementation is substantial, and the real traces in Figure 11 provide some supporting evidence. But as submitted, the central claim cannot be independently checked: the simulator's bubble/overlap computation is unspecified, the fidelity evidence is narrow, no code is released, and no error bars are reported. These are fixable with clarification and expanded validation, but they prevent accepting the current claim. The reader's REJECT verdict is therefore not changed by this stress-test pass.","tokens_in":959,"tokens_out":940,"duration_ms":86616,"concrete_test":"Run the Section 5.5 fidelity protocol on Gemma and DeepSeek (all three sizes) at SeqLen=2K and 8K, comparing simulated vs real normalized throughput for S-1F1B, I-1F1B, ZB, Mist, and AdaPtis. If the average absolute prediction error exceeds about 5%, or if the surrogate ranks the five methods differently from the real runs, rerun the full tuning for those workloads and re-measure the headline speedups. In the same revision, state explicitly whether OctoPipe and AdaPtis are the same system and reconcile the abstract's 1.15-1.44x numbers with the body's averages.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Two load-bearing problems. First, attribution: the title, abstract, and opening paragraph present \"OctoPipe\" with 1.15-1.44x throughput gains, while every design and experiment section presents \"AdaPtis\" with different average speedups (1.34x/1.42x/1.34x/1.20x over S-1F1B/I-1F1B/ZB/Mist, up to 2.14x). No sentence connects the two names, so the evidence does not clearly evaluate the system announced in the abstract. Second, and more decisive for the technical claim, the Pipeline Performance Model that drives the tuner is under-specified and under-validated. Algorithm 1 Step 3 states T_d = C_d + BubbleTime(d) - OverlapTime(d), but BubbleTime(d) and OverlapTime(d) are not defined by any formula or pseudocode; they are said to come from \"simulating the execution behavior.\" The fidelity check (Section 5.5, Figure 12) covers only Nemotron-H models at SeqLen=4K, reports normalized (relative to S-1F1B) throughput rather than absolute, and gives no error bars or per-configuration variance. The E2E speedups in Figures 8-9 are claimed for Gemma, DeepSeek, and Nemotron-H at 2K/4K, and for Nemotron-H up to 32K; the surrogate's accuracy in those regimes is simply assumed. Because the generator transfers layers, permutes placements, and reorders W based on predicted bubble/overlap differences, an error larger than the difference between candidate pipelines would flip tuning decisions. Thus the claimed average gains are not anchorable without broader fidelity evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript body, titled 'AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models', proposes a pipeline-parallelism system that co-optimizes model partition, model placement, and workload scheduling for heterogeneous LLMs. It introduces a Pipeline Performance Model that estimates per-device runtime from profiled layer costs and simulated bubble/overlap times; a Pipeline Generator that iteratively tunes the three phases using this model; and a unified Pipeline Executor that emits computation/communication instructions with deadlock elimination and communication-computation overlap. Experiments on Gemma, DeepSeek, and Nemotron-H report average speedups of 1.34x, 1.42x, 1.34x, and 1.20x over S-1F1B, I-1F1B, ZB, and Mist respectively, with up to 2.14x over I-1F1B. However, the arXiv-level abstract provided for this submission describes a different system, 'OctoPipe', with different speedup numbers (1.15-1.44x), and no part of the body connects OctoPipe to AdaPtis. The central technical claim is the co-optimization benefit, but the performance-model underpinning is only partially specified and validated in a narrow regime.","tokens_in":19850,"tokens_out":5372,"duration_ms":54149,"significance":"If the reported results are reproducible, the paper's contribution is meaningful for the systems community: it addresses a real gap by co-optimizing three pipeline phases rather than one, and the instruction-based executor is a reasonable design for supporting irregular schedules. The evaluation covers multiple heterogeneous model families and GPU scales, and the core claim is falsifiable. However, the paper ships no code, no raw data, and no error bars, which limits verification. The correctness of the tuning mechanism depends on a performance model whose formal specification and validation coverage are incomplete. The circularity concern raised in the stress-test note does not land: the model uses profiled per-layer costs and simulation, not fitted speedups, so it is not circular in the sense of defining the output as a function of the measured result. The decisive issues are identity inconsistency, under-specified model internals, and insufficient fidelity evidence outside the validated configuration.","major_comments":[{"comment":"The manuscript is internally inconsistent about what system it presents. The title of the provided submission and the opening abstract describe 'OctoPipe' with claimed speedups of 1.15-1.44x over state-of-the-art pipeline approaches, while the body title, body abstract, all sections, and all experiments describe 'AdaPtis' with average speedups of 1.34x/1.42x/1.34x/1.20x over S-1F1B/I-1F1B/ZB/Mist and up to 2.14x over I-1F1B. No sentence connects the two names. A reader cannot tell whether the paper is evaluating the system announced in the abstract. This must be reconciled before the paper can be assessed.","section":"Abstract / Title"},{"comment":"The core performance equation is under-specified. Algorithm 1 Step 3 computes T_d = C_d + BubbleTime(d) - OverlapTime(d), but BubbleTime(d) and OverlapTime(d) are never defined by formulas, pseudocode, or a precise simulation procedure; Section 4.2 only says the model 'simulates the execution behavior' and 'identifies when and where device idle time occurs.' Since the Pipeline Generator makes tuning decisions based on differences in these terms, the method is not reproducible as written. Please provide a formal definition of the simulation, including how per-device execution order, communication events, and overlap are computed.","section":"Section 4.2, Algorithm 1"},{"comment":"The fidelity validation of the Pipeline Performance Model is too narrow to anchor the E2E claims. Figure 12 and the accompanying text validate only Nemotron-H models at SeqLen=4K, report normalized (relative to S-1F1B) throughput rather than absolute predictions, and give no error bars or per-configuration variance. Yet Figures 8-9 claim speedups for Gemma, DeepSeek, and Nemotron-H at SeqLen=2K/4K and up to 32K. The surrogate's accuracy in those regimes is assumed, not shown. A tuner guided by a model that is inaccurate by more than the difference between candidate pipelines could select suboptimal configurations; please add fidelity measurements, or a sensitivity analysis showing that the reported speedups are robust to model error, for the other model families and sequence lengths.","section":"Section 5.5, Figure 12"},{"comment":"The scaling claims are mislabeled or miscalculated. The text says AdaPtis achieves a 'scaling efficiency of 534%' when scaling from 8 to 128 GPUs. Efficiency cannot exceed 100%: with 16x more GPUs, a 5.34x throughput increase corresponds to 33% strong-scaling efficiency, not 534%. If the intended quantity is a speedup factor relative to AdaPtis on 8 GPUs, it should be stated as speedup, not efficiency. The same issue applies to the weak-scaling claim of 519%. Please correct the terminology and provide the actual throughput values or a clear derivation.","section":"Section 5.7, Figures 14-15"}],"minor_comments":[{"comment":"The fidelity numbers are reported inconsistently: the text states an average prediction error of 2.12% and then lists maximum errors that include '2.12% for ZB'. The same value appears twice, and no maximum is given for S-1F1B. Please clarify which number is which and report errors for all baselines.","section":"Section 5.5"},{"comment":"All throughput plots are normalized and single measurements, with no absolute throughput values, standard deviations, or run counts. This makes it difficult to assess whether the reported speedups exceed run-to-run variance. Please include absolute numbers or an artifact with raw data for verification.","section":"Figures 8-15"},{"comment":"The bubble-ratio measurements in Figure 1 have no methodology description. Please state how bubble ratio is computed from traces, and whether the values are from real executions or from the simulator.","section":"Figure 1"},{"comment":"The model name is misspelled as 'Nemontron-H' in the captions of Figures 14 and 15. It should be 'Nemotron-H'.","section":"Figures 14-15"}],"recommendation":"major_revision","confidential_remarks":"The OctoPipe/AdaPtis mismatch looks like the submission was renamed mid-production and the arXiv metadata was not synchronized. The editor should require the authors to confirm which name is the intended system and ensure the title, abstract, and body all refer to the same artifact. The lack of an artifact link or raw data is also a concern for a systems paper making strong empirical claims; if this is a journal submission, data availability may be a requirement. The performance-model fidelity gap is fixable with additional experiments, so I do not recommend rejection at this stage, but the requested additions are not optional polish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: the paper has a genuine idea — jointly optimizing model partition, placement, and scheduling for heterogeneous LLMs — and the taxonomy in Table 2 is genuinely useful. But as submitted, the title and abstract advertise \"OctoPipe\" with 1.15–1.44x speedups, while the body presents \"AdaPtis\" with different averages (1.34x, 1.42x, 1.34x, 1.20x, up to 2.14x) and never connects the two names. That is not cosmetic: a reviewer cannot tell whether the evidence evaluates the system the abstract promises. This must be fixed before publication.\n\nWhat the paper does well: the iterative bottleneck-driven tuning procedure is sensible; the ablation in Figure 10 supports the co-optimization thesis; the strong/weak scaling results are broad; and the framing of prior work as optimizing only one of the three phases is accurate and useful. The simulator-driven approach with profiled per-layer costs is a reasonable methodology, and the fidelity check on Nemotron-H at SeqLen=4K shows average error around 2%, which is respectable.\n\nThe soft spots are real but not fatal. The performance model is under-specified: BubbleTime(d) and OverlapTime(d) in Algorithm 1 are described as outputs of \"simulating execution behavior\" but never defined by formulas or pseudocode. More importantly, the fidelity check covers only Nemotron-H at 4K sequence length, reported as normalized throughput without error bars or per-configuration variance. The E2E speedups for Gemma and DeepSeek, and for other sequence lengths, rest on the assumption that the surrogate generalizes; that assumption is unverified. Since the generator makes decisions based on predicted differences between candidate pipelines, an error larger than the candidate difference flips tuning choices. The missing cross-model fidelity evidence is the biggest technical gap. Also, the abstract's \"without deadlocks\" overstates what the executor implements — it detects and reorders, not a proof of deadlock freedom. No code or raw data is released, which is not disqualifying in systems work but matters given the measurement-only claims.\n\nWho this is for: systems researchers working on pipeline parallelism for heterogeneous models. The framing and taxonomy are valuable, and the system is a plausible engineering contribution. Send it to peer review, but expect major revision: resolve the OctoPipe/AdaPtis mismatch, specify the performance model precisely, and validate the surrogate on at least one more model and sequence length, or narrow the claims.","headline":"A real co-optimization idea buried under an identity mismatch and thin performance-model validation; worth reviewing but not in this state.","tokens_in":20318,"tokens_out":3177,"would_cite":false,"duration_ms":24432,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that on increasingly heterogeneous LLM architectures, pipeline bubbles shrink—and throughput rises—only when model partition, stage placement, and micro-batch scheduling are co-optimized, not tuned one at a time.","keywords":["pipeline parallelism","large language model training","model heterogeneity","model partition","model placement","workload scheduling","bubble reduction","distributed training"],"falsifier":"A concrete check: profile Gemma and DeepSeek at 1K, 16K, and 32K sequence lengths, generate OctoPipe/AdaPtis schedules from the simulator, then run them. If real throughput deviates from the simulated throughput by much more than the measured 2.12% average error (or if the gap to baselines disappears), the additive performance model—and the speedups built on it—would fail.","tokens_in":19248,"feed_emoji":"🧩","tokens_out":7709,"duration_ms":52685,"temperature":0.7,"pith_summary":"The submission, titled OctoPipe but naming its system AdaPtis throughout the body, sets out to show that pipeline-parallel training of heterogeneous LLMs is best treated as one joint optimization problem rather than three separate ones. The target is the idle GPU time (pipeline bubbles) that grows when architectures mix oversized vocabularies, MoE layers, and hybrid attention/state-space blocks. The paper builds a simulator-based performance model, an iterative bubble-aware tuner that co-optimizes model partition, model placement, and workload scheduling, and an executor that runs irregular schedules without deadlock while overlapping communication with computation. On Gemma, DeepSeek, and Nemotron-H, the system reports 1.15–1.44× throughput gains over state-of-the-art pipeline baselines, with the largest gains on the most heterogeneous models. The appeal of the result is practical: if correct, it converts architectural heterogeneity from a pipeline-parallel liability into a tuneable scheduling problem.","feed_headline":"Three-way pipeline tuning lifts LLM training throughput up to 1.44x","feed_subtitle":"Co-optimizing partition, placement, and schedule recovers idle GPU time on heterogeneous LLMs.","key_machinery":"Three pieces carry the argument. The Pipeline Performance Model is a graph-based simulator whose core identity is T_d = C_d + BubbleTime(d) − OverlapTime(d): profiled per-layer compute and memory costs are summed into stages, stages into devices, and the device's wall time is estimated by subtracting communication that can overlap computation from the raw compute plus bubbles. The Pipeline Generator is a heuristic search that starts from known baseline pipelines and each iteration tunes only the current bottleneck phase, rolling back any change that degrades the simulated objective, which keeps a combinatorial search tractable. The Pipeline Executor turns schedules into computation and commu","core_discovery":"The paper's central claim is that the runtime of each device in a pipeline can be modeled as T_d = C_d + BubbleTime(d) − OverlapTime(d), and that this decomposition makes it possible to search the joint space of partition, placement, and schedule. Using profiled per-layer costs, the model aggregates stages into devices, simulates where bubbles appear, and detects where communication could hide under computation. A generator then tunes the bottleneck phase repeatedly—moving layers between stages with the largest and smallest bubbles, permuting stage-to-device mappings for finer granularity, and reordering forward/input-gradient/parameter-gradient work within memory constraints—while rolling b","pith_inferences":["Editorial inference: the same co-optimization loop should transfer to other axes of heterogeneity—variable sequence lengths across micro-batches, mixed expert parallelism, or mixed GPU types—because the model only needs per-layer profiled costs, but the paper only validates architectural heterogeneity.","Editorial inference: since the simulator exposes bubble and overlap breakdowns, it could serve as an online re-tuner when batch size, sequence length, or cluster shape changes during a long run; the paper presents the tuner as offline generation only.","Editorial inference: the instruction-based, deadlock-checked executor is not pipeline-specific in principle, so the scheduling machinery might be reusable for sequence-parallel or disaggregated inference workloads; this is not attempted in the paper."],"forward_implications":["Throughput on heterogeneous LLM families improves by 1.15–1.44× over prior pipeline-parallel methods, with the gap growing on the most heterogeneous architectures (up to 2.14× over the interleaved-1F1B baseline).","Single-phase tuning is not just weaker but can backfire: the ablation shows placement-only tuning slowing Nemotron-H by 16%, evidence that the three phases interact.","Pipeline generation remains practical: the iterative search completes large configurations—large models, many stages, 256 micro-batches—in about 100 seconds, where ILP-based search is impractical.","The performance model is accurate enough to guide search: average throughput prediction error is 2.12% on the Nemotron-H family at 4K sequence length, with simulated traces closely matching real execution traces.","The system scales: from 8 to 128 GPUs it reports 534% strong-scaling and 519% weak-scaling throughput growth, ahead of all compared methods."],"fun_headline_variants":["OctoPipe cuts pipeline bubbles, boosting LLM training up to 1.44x","Joint optimization of pipeline phases yields up to 1.44x faster LLM training","Heterogeneous LLM training sped 1.44x by co-tuning partition, placement, schedule","Pipeline bubbles shrink as OctoPipe co-optimizes three scheduling choices","1.44x throughput gain from co-optimizing pipeline partitioning, placement, scheduling"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a device's true wall-clock time is well approximated by summing profiled per-layer compute costs and then subtracting cleanly overlapped communication from compute-plus-bubble time (T_d = C_d + BubbleTime(d) − OverlapTime(d)); fidelity is verified only on Nemotron-H at 4K sequence length, so the model's accuracy on Gemma, DeepSeek, and other sequence lengths is assumed.","fun_headline_variants_meta":{"raw":{"variants":["OctoPipe cuts pipeline bubbles, boosting LLM training up to 1.44x","Joint optimization of pipeline phases yields up to 1.44x faster LLM training","Heterogeneous LLM training sped 1.44x by co-tuning partition, placement, schedule","Pipeline bubbles shrink as OctoPipe co-optimizes three scheduling choices","1.44x throughput gain from co-optimizing pipeline partitioning, placement, scheduling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1560,"prompt_tokens":735,"completion_tokens":825,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":727}},"tokens_in":479,"tokens_out":825,"duration_ms":5842,"temperature":1.0,"reasoning_tokens":727,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T14:37:48.788661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: profile Gemma and DeepSeek at 1K, 16K, and 32K sequence lengths, generate OctoPipe/AdaPtis schedules from the simulator, then run them. If real throughput deviates from the simulated throughput by much more than the measured 2.12% average error (or if the gap to baselines disappears), the additive performance model—and the speedups built on it—would fail.","supporting_citations":[],"review_version":1}