{"id":"cce59ef5-8523-4056-9b55-b7877558d9ce","arxiv_id":"2608.11034","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SCOUT localizes hangs, stragglers, and silent data corruption in LLM pre-training through strict-majority consensus among equivalent replicas, with replay-based checkpoint certification.","lead":"SCOUT is a diagnostic framework that finds which GPU rank or communication path caused a training job to hang, slow down, or silently corrupt data, by comparing each rank against healthy peer ranks. It could cut the hours operators currently spend locating faulty machines in large AI training runs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Checkpoint certification overreaches: the two-cycle gate certifies only the replayed surface, so a persistent SDC in an un-replayed layer is promoted as verified.","rationale":"We read the paper as an honest, well-scoped systems contribution. The majority-consensus principle is explicitly conditional: Section 5.3 and Section 11 state that a faulty value held by a strict majority is indistinguishable and that replay coverage is limited to exercised surfaces. The reader's weakest_assumption about the majority is valid but already conceded in the body. Our concern targets the unqualified 'certifies checkpoint numerical integrity' wording in the abstract, which is the most load-bearing promise for recovery. The evaluation's certification tests are circular because faults are injected exactly at the replayed surfaces, which is the sampled subset; a fault in an un-replayed layer would not be observed. The paper's own limitations paragraph narrows the guarantee, but the abstract does not. We would not reject: the body is candid and the mechanism is plausible. The conditional verdict remains appropriate, with an added condition that the certification claim be scoped to the replayed surface or that coverage be shown exhaustive or representative of likely SDC locations. Hence UNCHANGED, with the reader's majority concern and our coverage concern being complementary rather than identical.","tokens_in":25036,"tokens_out":13410,"duration_ms":122206,"concrete_test":"Run a 16-GPU FSDP2 job with SCOUT's default replay configuration. Inject a persistent bit-flip into the weight tensor of a hidden layer that is not in the sampled replay rotation (or into an optimizer-state slice after its rotation has passed). Execute the checkpoint certification protocol (Algorithm 3) for two clean cycles and inspect whether the corrupted checkpoint is promoted to verified and selected by CheckpointRetrieval. If it is promoted, the certification claim fails for un-replayed surfaces.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SCOUT 'certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC' is stronger than the replay coverage contract. Section 8 promotes a candidate only after two clean recipe cycles, where a cycle covers the K representatives in the replay catalog—one hidden layer at cadence I (Section 6.1.1), selected optimizer slices, and qualified MoE shapes. A persistent SDC affecting any tensor surface outside that rotating sample (an un-sampled hidden layer, an un-replayed optimizer slice, or an un-qualified MoE routing shape) is never observed, so the gate will certify a numerically corrupt checkpoint as verified. Table 3's certification cells inject faults only at the surfaces SCOUT replays (DDP parameter after backward, FSDP2 replay output, HSDP gradient shard), making the evaluation circular with respect to coverage: it demonstrates exclusion of faults at replayed surfaces, not certification of the whole checkpoint. Section 11 concedes 'Replay covers the layers, values, shapes, and communication paths exercised by its rotating checks,' narrowing the guarantee, yet the abstract states the prevention claim without that qualification. The most load-bearing promise—safe recovery—thus depends on coverage that the paper neither defines exhaustively nor measures.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SCOUT, a runtime service that localizes latent failures in synchronous LLM pre-training by comparing each rank's progress, timing, and numerical signatures against those of equivalent replicas and reporting the outlier under a strict-majority rule. It introduces C3, a collective-comparison abstraction; an out-of-band CPU observer for hangs; in-situ replay for straggler and SDC diagnosis; an MoE shape-compression catalog; and a checkpoint gate that promotes a recovery checkpoint only after two clean recipe cycles. The evaluation uses a 16-A100 two-host testbed and fault injection across DDP, FSDP2, and HSDP, plus dedicated harnesses for MoE replay compression and kernel-role SDC. Results report perfect or near-perfect pass rates for the injected scenarios, with the paper explicitly stating that production-scale, end-to-end overhead, and RDMA/multi-rack validation remain future work.","tokens_in":25259,"tokens_out":8308,"duration_ms":77075,"significance":"SCOUT's core principle—using strict-majority consensus among data-parallel replicas as a live reference—is simple and well motivated by production evidence that faults are usually rank-local and rare. The paper's strengths are real: the implementation is open source; the fault-injection results are reported as exact counts with an honest statement of the one-seed, 16-GPU scope; the failure schemas (progress fingerprints for hangs, timed replay for stragglers, deterministic signatures for SDC) are cleanly separated; and the C3 abstraction returns an outlier bitmap while keeping the comparison rule local. The explicit limitation passages in Sections 5.3 and 11 are unusually candid. If the results hold, the framework would give operators a single runtime path to name a faulty rank or peer group for three failure classes and to gate recovery checkpoints; the missing overhead and scale measurements, however, mean the current evidence supports mechanism behavior on a small testbed rather than production deployment claims.","major_comments":[{"comment":"The checkpoint certification claim in the abstract and Section 8 is stronger than the rotating-coverage contract that SCOUT actually implements. Section 8 states that for dense models 'every in-memory checkpoint captured after an accepted SCOUT check is certified by that check' because shapes are static, but static shape does not imply that the values of un-replayed layers or un-replayed optimizer slices were verified. A persistent SDC in any un-sampled hidden layer, un-replayed optimizer slice, or un-qualified MoE shape is never observed, so the two-clean-cycle gate in Algorithm 3 can promote a numerically corrupt checkpoint as verified. Table 3's recovery cells inject faults only at the replayed surfaces (DDP parameter after backward, FSDP2 replay output, HSDP gradient shard), so the experiments demonstrate exclusion of faults at replayed surfaces, not certification of the whole checkpoint. Section 11 narrows the guarantee to 'Replay covers the layers, values, shapes, and communication paths exercised by its rotating checks'; the abstract and Section 8 should carry the same qualification, or the certification contract needs an explicit coverage argument.","section":"Section 8, Algorithm 3, Table 3, Section 11"},{"comment":"The strict-majority assumption is a load-bearing blind spot for the recovery claim. Section 5.3 explicitly concedes that 'a faulty value held by a strict majority is indistinguishable from the expected value,' and that common-mode corruption making every peer agree remains invisible. Yet the abstract and Section 8 present SCOUT as 'preventing recovery from selecting state corrupted by SDC' without that caveat. A deterministic software bug, a shared-library defect, or a cluster-wide environment event that corrupts a majority of replicas would yield an Agree verdict and a certified checkpoint. The paper should either state the healthy-majority assumption in the abstract and Section 8, or add and evaluate a temporal/majority-crossing detection path.","section":"Section 5.3 and Section 8"},{"comment":"The paper's practicality claim rests on an analytical overhead estimate, but no end-to-end measurement is provided. Section 6.1.1 estimates the amortized layer-equivalent replay overhead as V/(IN) ≈ 0.3% for V=3, I=20, N=50, while Section 11 states that the paper 'does not measure end-to-end throughput and resource overhead across replay cadences, or report recovery time and rollback distance.' Since Section 1 identifies minimizing the overhead of continuous diagnosis as a core design challenge, the absence of measured throughput overhead, recovery latency, multi-rack/RDMA validation, and error bars around the one-seed fault-injection counts leaves the production-deployment claim unverified. The revision should either add such measurements or explicitly scope the contributions to mechanism behavior on the small testbed.","section":"Sections 6.1.1, 10, and 11"}],"minor_comments":[{"comment":"There are spacing and formatting errors in the text, such as 'usesitsConsensus Collective Communication(C3)' and 'an O( 10,000)-GPU cluster'; these should be cleaned up.","section":"Abstract and Section 1"},{"comment":"The captions contain missing spaces in expressions like 'a4 ×2DP–FSDP mesh' and in the logical mesh notation; please format the dimension products consistently.","section":"Figures 2 and 3"},{"comment":"Section 9 describes the GEMINI checkpoint coordinator, but Section 10 does not report a recovery experiment that explicitly exercises GEMINI's in-memory checkpoint path; please clarify whether the checkpoint-recovery tests used the GEMINI code path or a stand-in.","section":"Section 9 and Section 10"},{"comment":"The execution-path coverage preorder is evaluated only for uniform per-expert row counts, and Section 11 concedes that arbitrary heterogeneous routing vectors require separately qualified templates; this limitation should be stated at the point where the compression contribution is introduced.","section":"Section 6.2 and Table 4"},{"comment":"The two-clean-cycle promotion logic is described in prose but would be clearer if the pseudocode included an explicit comment that the candidate promoted at the end of a cycle is the checkpoint captured at the previous cycle boundary.","section":"Algorithm 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the journal's scope and the authors are honest about limitations. The central issue is that the abstract and Section 8 overstate the checkpoint certification guarantee relative to the rotating-coverage contract and the strict-majority assumption; this can be fixed by narrowing the claims or by adding coverage arguments and measurements. I see no evidence of misconduct or citation-manipulation concerns; the related-work set is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper deserves a serious referee. The core idea is simple and right: use strict-majority consensus among equivalent replicas (DP/FSDP peers) to turn progress, timing, and numerical signatures into localization evidence for hangs, stragglers, and SDC. The C3 abstraction, the out-of-band observer for hangs, and the in-situ replay design are a coherent package. The implementation is real—it hooks PyTorch/TorchTitan/Megatron/DeepSpeed through public interfaces—and the fault-injection results are clean: 344/344 SDC injections, 120/120 hang launches, 5,960/5,960 MoE injection cells, all on a 16-GPU two-host testbed. The limitations section is unusually candid. That matters.\n\nThe soft spots. The biggest one is the checkpoint-certification claim. The abstract says clean replay coverage 'certifies checkpoint numerical integrity, preventing recovery from selecting state corrupted by SDC.' But the two-cycle gate covers only the rotating replay catalog—one hidden layer at a time, selected optimizer slices, qualified MoE shapes. A persistent SDC in any surface outside that sample never gets observed, so the gate can promote a corrupt checkpoint to verified. Section 11 concedes this in effect ('Replay covers the layers, values, shapes... exercised by its rotating checks') but the abstract and the contribution list don't carry the qualification. Table 3's certification injections are placed exactly at the replayed surfaces, so they don't test coverage. That's a load-bearing mismatch, not a nit: 'certify' should be 'certify the replayed surface' or the claim should be scoped to a statistical guarantee the paper doesn't provide.\n\nThe evaluation also lacks end-to-end overhead numbers, error bars, production-scale multi-rack/RDMA validation, and baselines against Minder, Holmes, SDCHunter, AEGIS. The paper says as much. Those gaps are fine for an initial systems paper if the claims are scoped, but they do mean the headline diagnostic-interval reduction is unmeasured. The fault injections are single-seed and software-only; that's honest but it doesn't establish recall on real hardware faults.\n\nThe strict-majority assumption is a real boundary condition—if a majority shares a faulty value (common bug, bad library, environmental event), SCOUT returns agree. Section 5.3 says exactly this. Not a flaw in the design, but it should be more prominent than a limitations-tail discussion.\n\nWho is this for: anyone building or evaluating training-resilience infrastructure. It's a solid, honest design paper with a working prototype and reproducible fault-injection evidence. I'd engage with it. My recommendation: send it to peer review. It needs revision—scope down the certification claims, add overhead and baseline measurements, and ideally a multi-rack study—but the mechanism is novel enough and the evidence clean enough to warrant referee time.","headline":"A genuinely useful consensus-based failure-localization framework with honest small-scale evidence; the checkpoint-certification claim outruns its replay coverage and needs to be scoped down before publication.","tokens_in":25816,"tokens_out":2141,"would_cite":true,"duration_ms":18338,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that hangs, stragglers, and silent data corruption in LLM pre-training can all be localized online by a single rule: a rank whose behavior disagrees with the strict majority of its equivalent replicas is the outlier.","keywords":["failure localization","strict-majority consensus","silent data corruption","straggler detection","hang detection","LLM pre-training","checkpoint certification","consensus collective communication"],"falsifier":"Inject a fault that affects a strict majority of an equivalent peer group, for example a shared library that corrupts the same parameter on most replicas or a cluster-wide slowdown, and check whether SCOUT returns 'agree' with an empty outlier bitmap while the job remains unhealthy.","tokens_in":24783,"feed_emoji":"🔍","tokens_out":5641,"duration_ms":46320,"temperature":0.7,"pith_summary":"This paper tries to establish that a single runtime service can name the faulty rank, peer group, or machine behind three hard-to-localize failures in large-scale LLM pre-training: hangs, stragglers, and silent data corruption (SDC). It proposes SCOUT, which treats a latent failure as a behavioral outlier: a rank whose progress, timing, or numerical output disagrees with the strict majority of equivalent replicas doing the same work. The payoff is that recovery no longer guesses, because it knows which resource to replace or restart and which checkpoint is numerically safe to restore, all online without a separate offline log-analysis step.","feed_headline":"Majority-vote consensus pinpoints LLM training failures","feed_subtitle":"SCOUT compares equivalent replicas online and flags the outlier, so recovery knows which rank, group, or checkpoint to act on.","key_machinery":"The Consensus Collective Communication (C3) abstraction is the central mechanism: a diagnostic collective that AllGathers compact evidence from each rank, finds the most frequent value for exact comparisons or a robust median for timing comparisons, and marks every rank whose evidence diverges from the strict majority as an outlier. It is supported by an out-of-band CPU observer that reads shared-memory progress and collective fingerprints after a hang, and by in-situ replay that reruns captured layers on live accelerators to expose stragglers and numerical corruption under production conditions. A coverage-compression rule for Mixture-of-Experts shapes keeps replay bounded, and a checkpoint gate promotes a candidate checkpoint to verified only after complete clean replay cycles.","core_discovery":"SCOUT's central claim is that the three latent failure classes that plague synchronous LLM pre-training—hangs, stragglers, and silent data corruption—can all be localized by the same consensus rule: align equivalent replicas, gather compact evidence, and attribute divergence to any rank that a strict majority disagrees with. The paper argues that this converts redundancy already present in data-parallel and FSDP training into live diagnostic evidence, eliminating the need for absolute health thresholds, preselected golden ranks, or post-mortem reconstruction. For hangs, the evidence is a collective fingerprint and progress coordinate read by an out-of-band CPU observer; for stragglers, it is replay timing on controlled equivalent work; for SDC, it is a deterministic numerical signature. Clean replay coverage also certifies which checkpoint is numerically trustworthy for recovery.","pith_inferences":["If the strict-majority rule holds at production scale, the same principle could be extended to other synchronous distributed workloads beyond LLM training, such as HPC simulations, wherever equivalent replicas exist.","The checkpoint-certification logic implies a trade-off: the longer the verification cycle, the more healthy work is discarded on recovery, so tuning the recipe-catalog size and cadence against corruption probability is a natural next step the paper does not quantify.","The assumption that healthy values are identical may break for numerical signatures under legitimate nondeterminism, so a testable extension is to make the comparison tolerate small numerical differences rather than requiring exact equality.","A testable extension is to use SCOUT's per-surface outlier bitmaps to build a failure-type classifier that distinguishes compute, communication, and input stalls, which the current evaluation only partially demonstrates."],"forward_implications":["A hang that stalls a job can be attributed to a specific rank when its published collective fingerprint or progress coordinate diverges from the majority, while equal-progress stalls are reported as group-scoped for fabric diagnosis.","A recurring straggler is identified by controlled replay timing: a rank that remains slower than equivalent peers on identical work, with computation and communication timed separately to distinguish a slow module from a slow collective.","SDC is localized by deterministic numerical signatures: the rank whose output or gradient differs from the healthy majority is named, and the contaminated checkpoint is excluded from recovery.","Clean replay coverage certifies a checkpoint: only after a complete recipe cycle passes without SDC can a candidate checkpoint be promoted to verified, so recovery avoids reintroducing corrupted state.","SCOUT attaches to existing training stacks through public interfaces, so it can provide this diagnosis without modifying training loops or framework source."],"supporting_citations":[{"why":"Supplies the production report that a single faulty GPU can cascade into a cluster-wide NCCL stall, grounding the sparse-fault assumption behind strict-majority consensus.","marker":"[14]"},{"why":"Provides the straggler study quantifying that 42.5% of jobs have stragglers and the what-if analysis that motivates timing-based localization.","marker":"[23]"},{"why":"Documents that manual faulty-machine localization takes over 30 minutes on average, motivating online localization.","marker":"[3]"},{"why":"Shows most NCCL watchdog timeouts stem from rank desynchronization, grounding the collective-fingerprint hang evidence.","marker":"[26]"},{"why":"Supplies lightweight SDC sensing and definitive verification, the baseline against which SCOUT's numerical-signature comparison is positioned.","marker":"[19]"},{"why":"Shows SDC-defective GPUs can fail only for particular training inputs and replays the triggering workload, motivating in-situ replay and value variation.","marker":"[55]"},{"why":"Provides in-memory checkpoint capture and fast retrieval that SCOUT's checkpoint gate builds on.","marker":"[46]"},{"why":"Shows state-sharded ranks execute the same forward and backward computation after parameter AllGather, providing the equivalent state-shard peer groups.","marker":"[54]"}],"fun_headline_variants":["Majority vote outs the bad rank in LLM training","Consensus localizes LLM hangs, stragglers, and data corruption","SCOUT: strict majority reveals the failing replica","One dissent flags the rank that breaks LLM training","Consensus among replicas pinpoints failure origin"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that healthy ranks hold a strict majority in every compared peer group and all report the same healthy value; if a bug or environmental fault affects a majority, SCOUT sees agreement and the failure escapes localization.","fun_headline_variants_meta":{"raw":{"variants":["Majority vote outs the bad rank in LLM training","Consensus localizes LLM hangs, stragglers, and data corruption","SCOUT: strict majority reveals the failing replica","One dissent flags the rank that breaks LLM training","Consensus among replicas pinpoints failure origin"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1936,"prompt_tokens":957,"completion_tokens":979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":898}},"tokens_in":573,"tokens_out":979,"duration_ms":8405,"temperature":1.0,"reasoning_tokens":898,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:37:25.691994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inject a fault that affects a strict majority of an equivalent peer group, for example a shared library that corrupts the same parameter on most replicas or a cluster-wide slowdown, and check whether SCOUT returns 'agree' with an empty outlier bitmap while the job remains unhealthy.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the production report that a single faulty GPU can cascade into a cluster-wide NCCL stall, grounding the sparse-fault assumption behind strict-majority consensus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the straggler study quantifying that 42.5% of jobs have stragglers and the what-if analysis that motivates timing-based localization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that manual faulty-machine localization takes over 30 minutes on average, motivating online localization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows most NCCL watchdog timeouts stem from rank desynchronization, grounding the collective-fingerprint hang evidence."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies lightweight SDC sensing and definitive verification, the baseline against which SCOUT's numerical-signature comparison is positioned."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows SDC-defective GPUs can fail only for particular training inputs and replays the triggering workload, motivating in-situ replay and value variation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows state-sharded ranks execute the same forward and backward computation after parameter AllGather, providing the equivalent state-shard peer groups."}],"review_version":1}