{"id":"970ff201-2f34-4da4-8d58-127ec441ebc2","arxiv_id":"2505.17548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A framework called H2 combines a unified PyTorch interface, device-direct RDMA, and automatically searched pipeline parallelism to train a 100B model on over 1,000 heterogeneous chips, with up to 16.37% higher aggregate throughput than separate homogeneous runs.","lead":"This paper describes H2, a framework for training large language models on clusters that mix more than a thousand chips from multiple vendors. It combines a unified programming interface, direct RDMA communication, and an automatic pipeline-parallelism search; on a 100-billion-parameter model the system reached up to 16.37% higher aggregate throughput than running the same chips as separate homogeneous jobs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Superlinear claim rests on GBS-mismatched baselines: H2's winning runs use 6–8M-token global batches while Table 6 homogeneous baselines use 2M; matched-GBS runs Exp-A-1 and Exp-B-1 are below 100%.","rationale":"The reader's weakest_assumption correctly noted that the HeteroSpeedupRatio baseline may not transfer across cluster sizes or global batch sizes, and that claim is the right place to look. My stress-test sharpens this into a decisive, testable confound: the only experiments that cross the 100% superlinear threshold are exactly those with GBS 3–4x larger than the Table 6 baseline, while the same-GBS experiments fall short. This is strong internal evidence that the apparent speedup is driven by microbatch count / pipeline bubble reduction rather than by heterogeneous scheduling or communication optimizations. Because the paper's own data contains these matched-GBS control runs, the concern is not about outside-consensus assumptions but about internal consistency of the comparison. The engineering contributions—DiTorch, DiComm, HeteroAuto, activation resharding—are plausibly real and useful; the ablation in Table 9 shows expected monotone effects. However, the central quantitative claim as stated in the abstract and Section 1 is overstated. The reader already returned CONDITIONAL with requests to correct the overclaim and add fair baselines, so my assessment does not move the verdict; it reinforces it and supplies a specific experiment (GBS-matched homogeneous baselines) that would settle whether any genuine superlinearity remains.","tokens_in":19517,"tokens_out":4453,"duration_ms":34987,"concrete_test":"Re-run the homogeneous baseline for Exp-A-2 using exactly the Table 6 parallel configurations for Chips A, B, and C on 256 chips, but with GBS=6M instead of 2M, measuring TGS for each chip type. Then recompute HeteroSpeedupRatio = N*TGS_hetero / (256*TGS_A_6M + 256*TGS_B_6M + 256*TGS_C_6M). If this ratio drops to about 1.0 or below, the reported 109.03% is a batch-size artifact. Repeat the same procedure for Exp-B-2 with GBS=8M on all four chip types and for Exp-C-2 with GBS=8M; if all GBS-matched ratios fall below or near 100%, the superlinear claim should be withdrawn.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, 'consistently achieves a superlinear speedup' (abstract and Section 1), is not supported by the paper's own HeteroSpeedupRatio data, and the apparent superlinearity is confounded by global batch size. Table 6 reports homogeneous per-chip TGS from 256-chip runs at GBS=2M tokens. The heterogeneous runs that exceed 100%—Exp-A-2 (109.03%), Exp-B-2 (104.29%), Exp-C-2 (116.37%), and Exp-D (100.78%)—use GBS=6M, 8M, 8M, and 8M, respectively. With the 1F1B pipeline schedule described in Section 4.3.2, a larger GBS increases the number of micro-batches, reducing the pipeline bubble fraction and raising per-chip throughput even without any heterogeneity benefit. The controlled same-GBS experiments confirm this interpretation: Exp-A-1 (768 chips, GBS=2M) reaches only 89.56% and Exp-B-1 (1,024 chips, GBS=2M) only 77.45% of the summed homogeneous baselines. Thus every configuration crossing 100% does so only when GBS is 3–4x the baseline GBS, so the HeteroSpeedupRatio defined in Section 6.2 does not isolate the effect of the H2 framework. Independently of this confound, the abstract's 'consistently superlinear' wording is directly contradicted by Exp-A-1 and Exp-B-1, so the headline claim must be weakened or re-baselined.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents H2, a framework for training a 100B-parameter LLM on a hyper-heterogeneous cluster of more than 1,000 chips spanning four accelerator types. The system has three main components: DiTorch, a PyTorch-compatible unified interface with precision alignment; DiComm, an RDMA-based communication library with device-direct transfer; and HeteroPP with HeteroAuto, a heterogeneous pipeline-parallelism strategy with an automatic search for layer sharding, tensor-parallel sizes, and recomputation settings. The central evaluation claim, stated in the abstract and Section 1, is that H2 ``consistently achieves a superlinear speedup'' over homogeneous baseline training, by up to 16.37%, measured by the HeteroSpeedupRatio defined in Section 6.2.","tokens_in":19900,"tokens_out":2994,"duration_ms":25772,"significance":"If the headline claim were supported, the paper would make a strong practical contribution: it demonstrates a working path to integrating chips from multiple vendors with incompatible software stacks into one large-scale LLM training run, and it does so at a scale (over 1,000 chips) that is uncommon in the heterogeneous-training literature. The engineering strengths are real and should be credited: open-source components DiTorch and DIOPI are released, the search overhead is very low (Table 8), the ablation in Table 9 isolates the contribution of device-direct RDMA, topology-aware activation resharding, and fine-grained overlap, and the precision-alignment results in Section 3.1.2 provide a concrete, falsifiable acceptance criterion. However, the central performance claim is not supported by the paper's own data: the same-batch-size experiments in Figure 11 fall below 100% of the baseline, and all configurations that exceed 100% use a 3-4x larger global batch size than the baseline. The contribution is therefore defensible as a systems-building and evaluation paper only if the claims are substantially weakened and the baseline is re-measured or re-defined.","major_comments":[{"comment":"The claim that H2 ``consistently achieves a superlinear speedup'' is directly contradicted by the paper's own measurements. Figure 11 reports HeteroSpeedupRatio values of 89.56% for Exp-A-1 and 77.45% for Exp-B-1, both below 100%, and the text in Section 6.2.1 explicitly acknowledges these values. The headline statement must be weakened to apply only to the configurations with enlarged global batch sizes, or the claim of consistency must be removed.","section":"Abstract and Section 6.2 (Figure 11)"},{"comment":"The HeteroSpeedupRatio baseline is not comparable to the heterogeneous runs because the global batch size differs. Table 6 measures per-chip TGS on 256-chip homogeneous runs with a global batch size of 2M tokens, while every configuration that exceeds 100% in Figure 11 uses GBS=6M or 8M (Exp-A-2, Exp-B-2, Exp-C-2, Exp-D) or GBS=4M (Exp-C-1). Since the paper uses a 1F1B pipeline schedule (Section 4.3.2, alpha=1), a larger GBS increases the number of micro-batches and reduces the pipeline bubble fraction even without any heterogeneity benefit. The two controlled same-GBS experiments, Exp-A-1 and Exp-B-1, reach only 89.56% and 77.45%, respectively. Thus the ratio as defined does not isolate the effect of H2; the apparent superlinearity is confounded with the batch-size effect. The authors should either re-measure homogeneous baselines at the same GBS as the heterogeneous runs, or report matched-GBS HeteroSpeedupRatio as the primary result.","section":"Section 6.2, Eq. (HeteroSpeedupRatio), Table 6 and Table 7"},{"comment":"The explanation offered for the superlinear results is descriptive rather than quantitative. The text argues that Chip B's memory limits it to TP>=8 or recomputation, while Chip A has more memory, so H2 places Chip A in early pipeline stages and Chip B in later stages. However, no decomposition is provided to show how much of the measured gain comes from this reallocation versus from the larger global batch size, the different per-stage parallel configurations, or the improved communication stack. To support the central claim, the authors should provide a controlled comparison that holds GBS fixed and, if possible, isolates the contribution of the HeteroAuto-chosen strategy from the contribution of DiComm and the resharding optimizations.","section":"Section 6.2.1 (paragraph on superlinear performance)"},{"comment":"The HeteroSpeedupRatio definition assumes that the per-chip TGS values measured on 256-chip runs are representative and linearly composable when the same chips are placed in 768- to 2,432-chip heterogeneous clusters. This assumption is not justified. Training throughput per chip can change with cluster size, communication topology, memory sharing, and pipeline depth, and the paper provides no evidence that the Table 6 values remain valid at the larger scales. The authors should either validate this linear-composability assumption or define the baseline at the same cluster configuration and GBS as the heterogeneous experiments.","section":"Section 6.2 and Table 6 in relation to Section 6.1.2"},{"comment":"The second search stage treats every set of 128 homogeneous chips as a single heterogeneous group, which is a modeling assumption that prunes the search space. This is reasonable as a heuristic, but the paper does not report the sensitivity of the final strategy or the resulting throughput to this group size. Since the group size is a free parameter of the search, a brief sensitivity study would strengthen the claim that HeteroAuto's output is robust rather than tuned to this particular choice.","section":"Section 4.3.3 (two-stage search)"}],"minor_comments":[{"comment":"The text contains the typo ``DiTrain achieves'' where DiTrain is not introduced elsewhere; this should read ``H2'' or ``HeteroPP.''","section":"Section 6.2.1"},{"comment":"The sentence ``Figure 6 presents the training throughput...'' is incorrect; the throughput data appear in Table 6 and Figure 11, not in Figure 6, which shows communication latency comparisons. Figure and table cross-references should be corrected throughout the paper.","section":"Section 6.1.2, text above Table 6"},{"comment":"The third row of Table 8 contains a corrupted string of private-use Unicode characters with no readable content. This must be fixed before publication.","section":"Table 8"},{"comment":"The paper cites reference [16] and [17] with identical content, and the reference list contains several duplicated entries (e.g., the Ang Li et al. GPU interconnect paper appears twice). The bibliography should be deduplicated.","section":"Section 2.2 and Section 3.1.1"},{"comment":"The bubble coefficient alpha is described as configurable and set to 1 for 1F1B, but the value of alpha depends on the pipeline schedule. Since the cost model is used to compare strategies across schedules, the paper should state explicitly whether all experiments use the same schedule and same alpha, and how alpha is estimated in practice.","section":"Section 4.3.2, Eq. (cost model)"},{"comment":"The comparison of search overhead with Metis and Alpa is presented without elaboration on hardware, model size, or search-space details, so the reported values (600 seconds for Metis, 240 minutes for Alpa) are not directly comparable to the H2 search times in Table 8. A one-sentence caveat about the comparability of these numbers would be appropriate.","section":"Section 6.2.2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this one for the engineering, not the headline. The headline claim — 'consistently achieves a superlinear speedup' — is contradicted by the paper's own controlled experiments. Exp-A-1 and Exp-B-1 match the baseline global batch size of 2M tokens and run at 89.56% and 77.45% of summed homogeneous throughput. Every configuration that clears 100% does so at GBS=6M or 8M. Because the Table 6 baselines come from 256-chip runs at 2M tokens, and a larger GBS reduces the 1F1B pipeline bubble, the HeteroSpeedupRatio as defined does not isolate H2's effect. The paper does disclose the sub-100% numbers in Section 6.2.1, which makes the abstract's 'consistently' harder to defend.\n\nWhat is real: the integration of DiTorch's unified interface, DiComm's device-direct RDMA, topology-aware activation resharding, fine-grained overlap, and HeteroAuto's search, tested on a 100B model across 1,024 chips of four architectures. The Table 9 ablation is informative: RDMA over TCP, HeteroPP over uniform, and the resharding/overlap optimizations all help. Search overhead is strikingly low (0.6–12 seconds). The cost model is explicitly equivalent to Metis's, so the novelty is the combination and scale, not the formulation. DiTorch and DIOPI are open source, but H2 itself is not.\n\nSoft spots: no comparison to Metis, Whale, or HETHUB; no convergence evidence for the 100B training; no released configs. The GBS confound is the main issue. If the authors re-baselined with matched GBS, reported honest ratios, and added convergence data, this could be a solid systems paper. As it stands, the central claim needs correction before anyone should rely on it.\n\nI'd send it to peer review: the integration and scale are useful to the distributed-training community, and a decent referee can push for the re-baselining. Just don't cite the superlinear claim as is.","headline":"Useful heterogeneous-training engineering, but the superlinear claim is undercut by the paper's own same-GBS runs.","tokens_in":20428,"tokens_out":3599,"would_cite":false,"duration_ms":25424,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims its H2 system trains a 100-billion-parameter LLM on a cluster of 1,024 chips spanning four architectures faster than the same chips used as separate homogeneous jobs, with up to 16.37 percent surplus throughput.","keywords":["hyper-heterogeneous cluster","large language model training","pipeline parallelism","device-direct RDMA","automatic strategy search","activation resharding","heterogeneous training","tokens per chip per second"],"falsifier":"Reproduce one reported configuration, such as Exp-C-2, and run the same 100B model for the same number of tokens twice: once as the H2 mixed cluster and once as the constituent homogeneous clusters with each cluster's optimized 3D parallel configuration, matching total global batch size and data. If the summed token throughput of the homogeneous runs equals or exceeds the mixed run's throughput, the superlinear claim is refuted.","tokens_in":19321,"feed_emoji":"⚙️","tokens_out":8431,"duration_ms":67862,"temperature":0.7,"pith_summary":"This paper attempts to establish that a single heterogeneous training system can outperform the homogeneous baseline that treats each chip type as a separate cluster. It proposes H2, which combines a unified tensor-programming interface, a device-direct RDMA communication path, and an automatic search over pipeline stage sizes, layer shards, and recomputation settings. In evaluations on a 100-billion-parameter model, the mixed cluster reaches a HeteroSpeedupRatio above 100 percent in several configurations, up to 116.37 percent, meaning it produces more tokens per second than the sum of its parts run separately. If true, organizations could stop waiting for homogeneous allocations and train large models on whatever diverse chips they already own.","feed_headline":"Mixed-chip training beats homogeneous baseline by 16 percent","feed_subtitle":"H2 trains a 100B-parameter model on 1,024 chips of four architectures and beats running each chip type separately.","key_machinery":"The central object is the HeteroSpeedupRatio, defined as $N\\cdot TGS \\,/\\, \\sum_{i=1}^{C} N_i \\cdot TGS_i$, where $TGS$ is training throughput in tokens per chip per second for the heterogeneous run, $N$ is the total chip count, $N_i$ is the number of chips of type $i$, and $TGS_i$ is that chip type's throughput measured on a 256-chip homogeneous run. A ratio above 100 percent is the claimed superlinear speedup. The mechanism that makes this ratio exceed 100 percent is HeteroAuto's cost model, which minimizes an estimated iteration time over pipeline stage counts per chip type, tensor-parallel degrees, layer assignments, and recomputation flags, subject to per-chip memory limits; combined with the memory-descending pipeline ordering, this lets memory-rich chips absorb early-stage memory pressure while compute-rich but memory-poor chips avoid expensive tensor-parallel widening and CPU offload.","core_discovery":"The central claim is that hyper-heterogeneous training at scale is not only feasible but can be superlinearly faster than homogeneous training on the same chips. H2 achieves this with three coordinated pieces: DiTorch supplies a single tensor-programming interface that accepts an existing training program with a one-line change and aligns numerical precision across chips; DiComm implements device-direct RDMA between chips of different vendors, reducing measured point-to-point latency by an average of 9.94 times relative to CPU-mediated TCP; and HeteroPP is a pipeline-parallel scheme in which each stage uses one chip type, chips are ordered by memory from large to small along the pipeline, layers are distributed unevenly, and stage-level tensor-parallel, data-parallel, and recomputation settings are free. HeteroAuto searches that space with a profiled cost model and depth-first enumeration, and a topology-aware activation resharding plus fine-grained P2P overlap cuts cross-node transfer costs. The quantitative payoff is a HeteroSpeedupRatio above 100 percent, up to 116.37 percent, on 100B-parameter training runs with up to 1,024 chips.","pith_inferences":["Beyond the paper, a real superlinear ratio would change procurement logic: a fleet built from slower, cheaper chips plus a few large-memory chips could match or beat a homogeneous fleet at the same token throughput, but the paper does not measure price or power, so the economic claim is not established.","The ordering principle of memory-rich stages first and memory-poor stages later is a general scheduling heuristic that could apply to any asymmetric memory hierarchy, including single-vendor clusters with different GPU memory sizes, not only the multi-vendor case.","The precision check covers 300 iterations of a smaller model; an extension would be to track mean relative error over a full 100B pretraining run to see whether cross-vendor numerical drift accumulates beyond the 1.5 percent threshold.","A scale study holding global batch size fixed while varying only cluster size would separate the superlinear effect from the effect of using a larger global batch in the mixed run."],"forward_implications":["A mixed 1,024-chip run can deliver more tokens per second than the same chips partitioned into four homogeneous jobs, reaching 89.56 percent to 116.37 percent of the homogeneous sum depending on chip mix and global batch size.","Putting larger-memory chips in early pipeline stages lets memory-limited fast chips drop costly CPU offload or high tensor-parallelism, matching each stage's work to the chip's actual bottleneck.","Replacing CPU-mediated TCP cross-chip transfers with device-direct RDMA reduces measured point-to-point latency by an average of 9.94 times, with the largest gains at large message sizes.","Automatic strategy search completes in tens of seconds (0.62 to 12.29 seconds in the reported configurations), while comparable heterogeneous search tools are cited at minutes to hours for far smaller clusters.","A one-line modification of an existing training program is enough to execute it across all four chip types, because operator-level precision alignment holds each chip's loss within 1.5 percent mean relative error of the reference over 300 training iterations."],"supporting_citations":[{"why":"Supplies the standard tensor-programming interface whose dispatch semantics DiTorch unifies across chips.","marker":"[4]"},{"why":"Provides the optimizer-state memory sharding assumed enabled by default in HeteroPP.","marker":"[30]"},{"why":"Defines the 3D parallel formulation, especially tensor parallelism, that HeteroPP extends to heterogeneous stages.","marker":"[32]"},{"why":"Prior heterogeneous parallel training system limited to same-vendor GPU families that H2 generalizes to arbitrary chip types.","marker":"[15]"},{"why":"Prior heterogeneous parallelism search algorithm whose cost-model formulation and search overhead H2 compares against.","marker":"[37]"},{"why":"Broadcast-based inter-node activation resharding strategy that H2 replaces with topology-aware send/recv plus all-gather.","marker":"[42]"},{"why":"Zero-bubble pipeline schedule whose fine-grained computation-communication overlap inspired H2's overlap design.","marker":"[28]"}],"fun_headline_variants":["H2: 100B LLM training on 1,000+ mixed chips, 16% faster","Heterogeneous 1,000-chip cluster trains 100B LLM superlinearly","H2 mixes AI chips to beat homogeneous training by 16%","Superlinear speedup on 1,024 mixed chips for 100B LLM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The superlinear claim assumes that the tokens-per-second each chip type delivers on its own 256-chip run is a fair baseline for what the same chips contribute inside the 768-to-2,432-chip mixed run; if throughput per chip shifts with cluster size or global batch size, the measured ratio is not a real speedup.","fun_headline_variants_meta":{"raw":{"variants":["H2: 100B LLM training on 1,000+ mixed chips, 16% faster","Heterogeneous 1,000-chip cluster trains 100B LLM superlinearly","H2 mixes AI chips to beat homogeneous training by 16%","Superlinear speedup on 1,024 mixed chips for 100B LLM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2412,"prompt_tokens":990,"completion_tokens":1422,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":1329}},"tokens_in":606,"tokens_out":1422,"duration_ms":11480,"temperature":1.0,"reasoning_tokens":1329,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:44:40.920820+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce one reported configuration, such as Exp-C-2, and run the same 100B model for the same number of tokens twice: once as the H2 mixed cluster and once as the constituent homogeneous clusters with each cluster's optimized 3D parallel configuration, matching total global batch size and data. If the summed token throughput of the homogeneous runs equals or exceeds the mixed run's throughput, the superlinear claim is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the 3D parallel formulation, especially tensor parallelism, that HeteroPP extends to heterogeneous stages."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior heterogeneous parallel training system limited to same-vendor GPU families that H2 generalizes to arbitrary chip types."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Broadcast-based inter-node activation resharding strategy that H2 replaces with topology-aware send/recv plus all-gather."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Zero-bubble pipeline schedule whose fine-grained computation-communication overlap inspired H2's overlap design."}],"review_version":1}