{"id":"4991e505-8345-4e0d-a1c5-d63414eff5f0","arxiv_id":"2611.16330","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Snoopie is an open-source, instrumentation-based multi-GPU communication profiler that captures peer-to-peer transfers and NCCL/NVSHMEM calls with source code and data object attribution.","lead":"Snoopie is a new profiling tool that tracks data movement between multiple GPUs, including peer-to-peer transfers and communication library calls. It helps developers find performance bugs and understand communication patterns in multi-GPU applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"SASS-to-PTX matching for source-line attribution is unvalidated and may fail silently for a large fraction of instructions, undermining a key claimed capability.","rationale":"The reader's weakest_assumption correctly identifies the NVBit limitation as a concern. However, the reader's verdict (ACCEPT) treats this as a minor weakness that 'does not invalidate the central claim.' I argue this is more central: source-line attribution is a primary claimed differentiator (Table 1, §1 'attributing data movement to the source code line'). The paper's fallback SASS-to-PTX matching is described but never validated. The reader's rationale notes 'lack of quantitative accuracy evaluation' but does not connect this specifically to the source-line attribution pipeline. My concern is that the fallback may silently produce wrong or no lines for a large fraction of instructions, making the tool's key feature unreliable. The paper's own limitation (§4.4) says NVBit 'faces limitations in extracting this information from a significant portion of instructions' — this is an explicit warning that the primary method fails often, and the secondary method is unvalidated. I recommend CONDITIONAL: the paper should be accepted only if the authors provide validation of the SASS-to-PTX matching accuracy on a representative set of instructions, or clearly document the expected success rate and failure modes. This is not a rejection; the tool's other contributions (P2P capture, NCCL/NVSHMEM support, visualization) remain valuable. But the source-line attribution claim needs empirical backing to be credible.","tokens_in":22039,"tokens_out":787,"duration_ms":8543,"concrete_test":"Run Snoopie on a small, known multi-GPU kernel (e.g., the BFS ring example from §5.1) where every remote access instruction is manually annotated with its correct source line. Compare Snoopie's reported line numbers against the ground truth for all captured remote accesses. Report: (a) fraction of accesses where nvbit_get_line_info succeeded, (b) fraction where SASS-to-PTX matching was used, (c) fraction of those where the matched line was correct, (d) fraction where no line could be assigned. If (c) is below 90% or (d) is above 10%, the source-line attribution claim is significantly weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim includes attributing inter-GPU memory operations to source code lines (Gap 3, §1). The method (§4.4) relies on NVBit's nvbit_get_line_info, which the authors acknowledge fails for 'a significant portion of instructions' (§4.4). When it fails, Snoopie falls back to matching SASS instructions to PTX instructions using nvdisasm output. This SASS-to-PTX matching is described only at a high level; no validation is reported. The paper provides no quantitative data on what fraction of instructions are successfully matched, how often the matching is ambiguous (multiple PTX instructions map to the same SASS pattern), or how often the matched line is incorrect. Without this validation, the claimed source-line attribution for device-initiated P2P accesses (a key differentiator from tools like Nsight Systems) may be unreliable for a substantial fraction of events. The case studies (BFS, stencil) do not independently verify the line numbers shown; they only demonstrate that the tool produces some output. This is a load-bearing concern because source-line attribution is a headline contribution (Table 1, 'Src Code Line Attribution' column) and the paper's own limitation statement (§4.4) flags the underlying NVBit limitation without showing that the fallback resolves it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents Snoopie, a multi-GPU communication profiler and visualizer built on NVBit. It claims to capture device-initiated peer-to-peer direct memory accesses and communication library calls (NCCL, NVSHMEM) and attribute them to source code lines and data objects. The tool is open-source and includes a multi-view visualizer. Case studies demonstrate its functionality for BFS, stencil, NCCL AllReduce, and CosmoFlow.","tokens_in":22419,"tokens_out":2620,"duration_ms":18885,"significance":"If the claims hold, Snoopie fills a genuine gap in the GPU profiling tool landscape. The ability to attribute device-initiated P2P accesses to source lines and data objects is novel and could aid debugging and optimization of multi-GPU applications. The tool is open-source and the paper provides detailed implementation description. However, the significance is tempered by the acknowledged limitations: the tool is intra-node only, does not support MPI-based NCCL, and the source-line attribution relies on a fallback that is not validated.","major_comments":[{"comment":"The source code line attribution method is a key differentiator (Table 1, Gap 3), but its accuracy is not validated. The paper acknowledges that NVBit's nvbit_get_line_info fails for 'a significant portion of instructions' (§4.4) and falls back to SASS-to-PTX matching using nvdisasm. No quantitative data is provided on: (i) what fraction of instructions are successfully matched, (ii) how often the matching is ambiguous, or (iii) how often the matched line is incorrect. The case studies do not independently verify the line numbers shown; they only demonstrate that the tool produces some output. Without this validation, the claimed source-line attribution for device-initiated P2P accesses (a headline contribution) may be unreliable for a substantial fraction of events. This is a load-bearing concern because source-line attribution is a central claim and the paper's own limitation statement (§4.4) flags the underlying NVBit limitation without showing that the fallback resolves it.","section":"§4.4, §5.5","authors_acknowledge":"However, it should be noted that NVBit faces limitations in extracting this infor- mation from a significant portion of instructions.","authors_acknowledge_p":0.77},{"comment":"The accuracy evaluation is limited. The only quantitative accuracy comparison is for the stencil-p2p code with and without sampling (Figure 12), showing that the communication matrix pattern is similar but volumes are less balanced with sampling. This is a visual comparison, not a statistical measure. The paper claims 'Snoopie is accurate' (§5.5) but provides no ground-truth comparison for the exact profiling mode (e.g., comparing captured transfer counts against known program behavior or against a hardware counter). The accuracy of device attribution, object attribution, and source-line attribution is not quantitatively assessed. The authors should provide a more rigorous accuracy evaluation, at least for one case study, by comparing Snoopie's output against expected behavior derived from the source code or against a trusted baseline.","section":"§5.5, Figure 12","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The overhead measurements are reported for Stencil, BFS, and NCCL, but the numbers are inconsistent. In §5.5, the text says 'when the device-side filtering optimization is applied, the overhead of Snoopie is 16×, 2.04×, 96×, and 16.4× for Stencil, BFS, and NCCL, respectively.' This lists four numbers for three benchmarks. It appears the fourth number (16.4×) may be for NCCL, but the sentence structure is confusing. Additionally, the overhead for NCCL is reported as 96× in one place and 16.4× in another? The paper should clarify which overhead corresponds to which benchmark and ensure consistency.","section":"§4.7, §5.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper acknowledges that Snoopie does not support MPI-based NCCL applications, which includes PyTorch distributed training. However, the CosmoFlow case study (§5.4) uses a single-process port of the model. This is a reasonable workaround for demonstration, but it limits the generality of the findings. The paper should discuss how representative this single-process port is of real distributed training workloads and whether the communication patterns observed would hold in a multi-process MPI setting.","section":"§4.8, §5.4","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"Table 1 lists 'Src Code Line Attribution' and 'Data Object Attribution' as checkmarks for Snoopie, but the paper does not provide a clear definition of what constitutes a data object. In §4.5.1, a data object is defined as 'the call stack of a memory allocation call', which is reasonable but could be clarified earlier.","section":"§1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The description of SASS-to-PTX matching is vague. The paper says 'matching a disassembled SASS instruction with its PTX version, which has been previously mapped to a source code line in the output of nvdisasm.' It is unclear how the matching is performed (e.g., by opcode, by operands, by address?). A brief explanation of the matching algorithm would help reproducibility.","section":"§4.4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 11 shows overhead comparison but the y-axis is log-scaled and the bars are not labeled with exact values. The text provides some numbers, but it would be helpful to include the exact slowdown factors on the figure or in a table.","section":"§5.5, Figure 11","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper mentions 'custom wrapper allocation functions enabling programmers to include the variable name' but does not provide an example or API documentation. This would be useful for users.","section":"§4.5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The support for Python programs is described, but the paper does not evaluate the overhead or accuracy for Python applications. The CosmoFlow case study is in PyTorch, but the overhead measurements in §5.5 do not include a Python-based benchmark.","section":"§4.6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The BFS bug detection demo is interesting, but it is not clear whether the bug was introduced intentionally or was a real bug in the original code. The paper should clarify.","section":"§5.1, Demo 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The stencil bug demo (indexing error) is a useful illustration, but the paper does not explain how Snoopie's output specifically revealed the bug. The object view shows the second row being accessed instead of the first, but the user would need to know the intended behavior to spot the discrepancy. The paper should discuss whether Snoopie can automatically detect anomalies or if it requires manual inspection.","section":"§5.2, Demo 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The device attribution method using memory range tracking is described, but the paper does not discuss potential issues with memory reuse or deallocation. If a memory region is freed and reallocated to a different device, could the stale mapping cause incorrect attribution? The paper mentions that the approach 'persists beyond the lifetime of the memory allocation itself' but does not address this scenario.","section":"§4.2","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 4 illustrates NVSHMEM address space layout but the caption is unclear. The text says 'NVSHMEM 2.7' and 'NVSHMEM 2.8/2.9' but the figure does not distinguish between versions. The figure should be clarified or the caption should explain the difference.","section":"§4.3, Figure 4","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 9 shows the NCCL AllReduce visualization, but the node sizes and edge widths are not explained in the caption. The text mentions that GPU0 and GPU7 have smaller communication sizes for TREE, but the figure should be self-explanatory.","section":"§5.3, Figure 9","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The sampling method is described as using a pseudo-random number generator. The paper should specify the random number generator used and whether it is deterministic for reproducibility.","section":"§4.7","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The overhead for NCCL is reported as 96× in one sentence and 16.4× in another. This inconsistency needs to be resolved. Also, the sentence 'the overhead of Snoopie is 16×, 2.04×, 96×, and 16.4× for Stencil, BFS, and NCCL, respectively' lists four numbers for three benchmarks. Please clarify.","section":"§5.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states 'Timeline information falls outside the scope of Snoopie.' This is a reasonable design choice, but the paper should discuss how users can combine Snoopie's output with timeline tools like Nsight Systems for a complete picture.","section":"§4.8","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Figure 12 shows exact vs. sampling profiling for stencil. The color scale is not explained. The paper should specify what the colors represent (e.g., number of transfers, bytes).","section":"§5.5, Figure 12","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The related work section is comprehensive but could benefit from a more direct comparison with ComScribe, which is the closest prior work. The paper mentions ComScribe but does not discuss specific differences in capability or methodology beyond what is in Table 1.","section":"§6","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper uses a splay tree for address range tracking. The choice of data structure is mentioned but not justified. A brief explanation of why a splay tree was chosen over other interval tree implementations would be helpful.","section":"§4.5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The warp-level aggregation (one thread per warp compiles addresses) is described, but the paper does not discuss the potential for missing accesses if multiple threads in a warp access different remote addresses. The paper should clarify whether all addresses are captured or if there is a limitation.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper reports overhead for SpMV in the text but SpMV is not introduced as a benchmark in the evaluation section. The paper should either include SpMV in the evaluation description or remove the reference.","section":"§5.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper mentions 'custom wrapper allocation functions' for variable names but does not provide an example or API. This would be useful for users.","section":"§4.5.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The overhead measurements include log compression time. The paper should also report the overhead without compression to understand the impact of compression.","section":"§5.5","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper states that 'a single thread compiles memory addresses targeted by memory operations from all other threads in the warp.' This could lead to missing accesses if multiple threads access different remote addresses. The paper should clarify whether all addresses are captured or if there is a limitation.","section":"§4.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The paper reports overhead for SpMV in the text but SpMV is not introduced as a benchmark in the evaluation section. The paper should either include SpMV in the evaluation description or remove the reference.","section":"§5.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"Snoopie: A Multi-GPU Communication Profiler and Visualizer","keywords":["multi-GPU","communication profiling","binary instrumentation","NVBit","NCCL","NVSHMEM","data movement","visualization"],"falsifier":"Run a multi-GPU application that uses device-initiated P2P accesses on a system with NVLink, and compare the communication map produced by Snoopie against a ground-truth obtained by instrumenting the same application with a hardware performance counter that logs remote memory operations; if Snoopie misses more than, say, 10% of the remote accesses or misattributes device IDs for a significant fraction, the tool's accuracy claim would be undermined.","tokens_in":23606,"feed_emoji":"🔍","tokens_out":1533,"duration_ms":16689,"temperature":0.7,"pith_summary":"The paper presents Snoopie, a profiling tool that captures inter-GPU data movement, including device-initiated peer-to-peer direct memory accesses and communication library calls (NCCL, NVSHMEM), which existing tools like Nsight Systems and HPCToolkit handle poorly or not at all. Snoopie attributes each data transfer to the source code line and the data object involved, and provides a multi-view interactive visualization. The authors demonstrate through case studies that Snoopie can locate performance bugs (e.g., incorrect flag allocation in BFS), compare communication patterns (e.g., ring vs. tree all-reduce algorithms), and visualize the communication footprint of different parallelization strategies in PyTorch. The tool is open-source and targets single-node multi-GPU systems.","feed_headline":"New tool reveals hidden GPU-GPU data movement","feed_subtitle":"Snoopie captures device-initiated peer-to-peer transfers and NCCL/NVSHMEM calls, attributing them to source code lines and data objects.","key_machinery":"The central mechanism is dynamic binary instrumentation using NVBit, which allows Snoopie to intercept every load/store instruction executed on the GPU and check whether the accessed address belongs to a remote GPU's memory. Device attribution relies on NVIDIA's Unified Virtual Addressing (UVA) to map pointer addresses to device IDs. Source-code line attribution combines NVBit's line-info function with nvdisasm-based SASS-to-PTX matching when the former fails. Data-object attribution uses LD_PRELOAD to intercept memory allocation calls and records call stacks, then matches runtime addresses against a tree of allocated regions.","core_discovery":"Snoopie is a binary-instrumentation-based profiler that fills a gap in multi-GPU communication profiling: it can detect and attribute device-initiated peer-to-peer direct memory accesses (loads/stores from one GPU to another GPU's memory) and communication library calls (NCCL, NVSHMEM) to source code lines and data objects. This is something existing tools cannot do comprehensively. The paper claims that Snoopie's combination of NVBit instrumentation, on-device filtering, device attribution via UVA memory ranges, and SASS-to-PTX matching for line info enables a level of visibility into inter-GPU data movement that was previously unavailable.","pith_inferences":["Snoopie's ability to attribute communication to source code lines and data objects could be integrated into continuous integration pipelines to automatically flag unexpected communication patterns or regressions.","The SASS-to-PTX matching approach for line attribution, while acknowledged as imperfect, suggests a general strategy that other binary-instrumentation tools could adopt for GPU code attribution.","The paper's focus on single-node multi-GPU leaves a natural next step: extending to multi-node setups (e.g., NCCL with MPI) would make Snoopie applicable to the majority of distributed training workloads.","The tool's reliance on NVBit ties it to NVIDIA GPUs, but the conceptual approach (instrument memory operations, attribute via UVA-like address spaces, match to objects) could be ported to AMD's ROCm or Intel's oneAPI as those ecosystems develop equivalent instrumentation frameworks."],"forward_implications":["Application developers can use Snoopie to locate communication bottlenecks and bugs in multi-GPU codes that use device-initiated P2P or NCCL/NVSHMEM, which were previously opaque.","The tool enables direct comparison of communication patterns between different parallelization strategies (e.g., data vs. model parallelism in deep learning) at the level of individual data objects and source lines.","Snoopie's visualizations (system, object, code, device views) provide a multi-granularity view that can reveal unexpected communication patterns, such as non-neighbor transfers in a ring algorithm or memory over-allocation.","The overhead reduction techniques (on-device filtering, sampling, log compression) make the tool practical for real applications, with reported slowdowns of 2×–96× depending on the workload and sampling rate.","The open-source release allows the community to extend the tool to multi-node setups and other GPU vendors."],"supporting_citations":[{"why":"NVBit is the dynamic binary instrumentation framework that Snoopie is built on; it provides the ability to instrument GPU instructions and capture memory addresses.","marker":"[67]"},{"why":"Nsight Systems is the primary existing tool that Snoopie compares against; the paper identifies its limitations in capturing device-initiated P2P and attributing to source lines/objects.","marker":"[48]"},{"why":"HPCToolkit is another existing profiler that Snoopie compares against; it lacks support for device-initiated transfers and object attribution.","marker":"[1]"},{"why":"ComScribe is a prior tool from the same group that captures intra-node GPU communication but only presents communication matrices and lacks source line/object attribution.","marker":"[8]"},{"why":"NVSHMEM is one of the communication libraries that Snoopie supports; the paper describes how Snoopie handles NVSHMEM's symmetric address space for device attribution.","marker":"[44]"},{"why":"NCCL is the other communication library that Snoopie supports; the paper shows how Snoopie visualizes NCCL's all-reduce algorithms.","marker":"[53]"},{"why":"nvdisasm is used for SASS-to-PTX matching to obtain source code line information when NVBit's built-in function fails.","marker":"[45]"},{"why":"CosmoFlow is the deep learning application used in the case study to demonstrate Snoopie's ability to compare data vs. model parallelism communication patterns.","marker":"[36]"}],"fun_headline_variants":["Snoopie spies hidden GPU conversations","See every byte move between GPUs","Multi-GPU data leaks no longer secret","Snoopie instruments GPU-GPU transfers","Communication profiler exposes GPU chatter"],"cache_read_input_tokens":0,"weakest_assumption_plain":"NVBit can reliably instrument all relevant GPU instructions and the SASS-to-PTX matching for source line attribution works for a sufficient fraction of instructions to be useful; the paper acknowledges that NVBit fails to extract line info from a significant portion of instructions.","fun_headline_variants_meta":{"raw":{"variants":["Snoopie spies hidden GPU conversations","See every byte move between GPUs","Multi-GPU data leaks no longer secret","Snoopie instruments GPU-GPU transfers","Communication profiler exposes GPU chatter"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":9.8e-05,"raw_usage":{"cost":9.7552e-05,"is_byok":false,"cost_details":{"upstream_inference_cost":9.7552e-05,"upstream_inference_prompt_cost":8.6268e-05,"upstream_inference_completions_cost":1.1284e-05},"total_tokens":1010,"prompt_tokens":948,"completion_tokens":62,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":0}},"tokens_in":948,"tokens_out":62,"cost_source":"openrouter_usage","duration_ms":9363,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":0,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-11-13T06:34:29.315669+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run a multi-GPU application that uses device-initiated P2P accesses on a system with NVLink, and compare the communication map produced by Snoopie against a ground-truth obtained by instrumenting the same application with a hardware performance counter that logs remote memory operations; if Snoopie misses more than, say, 10% of the remote accesses or misattributes device IDs for a significant fraction, the tool's accuracy claim would be undermined.","supporting_citations":[{"cited_title":"Chamberlain, Romain Cledat, H","cited_arxiv_id":null,"evidence_quote":"NVBit is the dynamic binary instrumentation framework that Snoopie is built on; it provides the ability to instrument GPU instructions and capture memory addresses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Nsight Systems is the primary existing tool that Snoopie compares against; the paper identifies its limitations in capturing device-initiated P2P and attributing to source lines/objects."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"HPCToolkit is another existing profiler that Snoopie compares against; it lacks support for device-initiated transfers and object attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ComScribe is a prior tool from the same group that captures intra-node GPU communication but only presents communication matrices and lacks source line/object attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NVSHMEM is one of the communication libraries that Snoopie supports; the paper describes how Snoopie handles NVSHMEM's symmetric address space for device attribution."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NCCL is the other communication library that Snoopie supports; the paper shows how Snoopie visualizes NCCL's all-reduce algorithms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"nvdisasm is used for SASS-to-PTX matching to obtain source code line information when NVBit's built-in function fails."},{"cited_title":"Pennycook, Kristyn Maschhoff, Jason Sewall, Nalini Kumar, Shirley Ho, Michael F","cited_arxiv_id":null,"evidence_quote":"CosmoFlow is the deep learning application used in the case study to demonstrate Snoopie's ability to compare data vs. model parallelism communication patterns."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}