{"id":"a0867bdb-5922-4f2a-b12e-ef3b5850e363","arxiv_id":"2412.08466","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Fault injection at the application level and at the instruction level ranks DNN software hardening techniques differently, sometimes reversing the winner entirely.","lead":"This paper compares two ways of injecting hardware faults into neural networks: corrupting weights and activations in software (APP) or corrupting GPU instructions and registers (ISA). It finds the two methods rank three software hardening techniques differently, and argues the cheaper APP method can overstate how well hardening works.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conclusion that APP-level FI overestimates hardening benefits and that ISA FI is 'closer to real' rests on a comparison that varies fault model, injection target, and GPU simultaneously; the observed ranking reversal cannot be attributed to abstraction level.","rationale":"The reader's weakest_assumption identifies the same core issue: the equivalence step in Section III.C does not isolate abstraction level because the two FI campaigns differ in fault model, injection target, and GPU platform. I agree that this is the most load-bearing concern. The paper's headline claim is precisely that APP FI overestimates hardening benefits and that ISA FI changes the ranking because it is at a more detailed abstraction level. That causal attribution requires the comparison to differ only in abstraction, which is not the case. The internal contradiction in the conclusion (whether APP or ISA identifies Adaptive Clipper as best for MobileNet V2) further weakens the presentation but is not as fundamental as the confound. The proposed concrete test would separate abstraction level from fault model and target by holding the GPU constant and varying the fault model independently at each abstraction level. Since the paper can be revised to reframe the claim as a sensitivity result and to provide a controlled experiment, the conditional verdict remains appropriate; no change to the reader's verdict is needed.","tokens_in":10100,"tokens_out":6023,"duration_ms":62798,"concrete_test":"Perform a matched experiment on the same GPU (V100) with four arms: (1) APP independent bit-flips in weights; (2) APP permanent stuck-at faults in weights with the same BER; (3) ISA transient bit-flips in the destination register of weight-loading instructions with the same activation count; (4) ISA permanent stuck-at faults in that register. The central claim predicts a rank reversal between APP and ISA that is invariant to the fault-model change; if the rank changes between (1) and (2) or between (3) and (4), the original comparison was confounded by fault model, not abstraction level.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III.C claims a 'fair comparison' by computing the BER from ISA stuck-at fault excitation and reusing it for APP multiple bit-flips, but the two campaigns differ on several axes at once. ISA injection (Section III.A) targets register-file destinations and FU inputs/outputs with permanent stuck-at faults on an RTX 3060 Ti; APP injection (Section III.B) corrupts weight tensors and neuron output feature maps with single or multiple random bit-flips on V100 GPUs. A single stuck-at bit produces correlated, repeated errors whenever the register/FU is used, whereas APP bit-flips are independent, one-shot perturbations. The BER equalization only matches the average activation rate, not the spatial/temporal error structure, the fault polarity, or the affected data type (addresses, control, and weights vs activations). Thus the rank reversal in Fig. 2 and Table I could be caused by the fault model, the target, or the platform rather than by the abstraction level. Additionally, the statement that ISA results are 'closer to the real ones' (Section IV.A) is an assertion: no hardware or microarchitectural ground truth is provided. The paper can validly claim that FI methodology affects hardening evaluation, but it has not shown that APP FI overestimates beneficial impact or that ISA FI is more accurate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper compares two fault injection (FI) abstraction levels—application-level (APP) and instruction-level (ISA)—for evaluating software-based hardening strategies on DNNs. The authors adapt PyTorchFI for APP injections (bit-flips on weights and neuron outputs) and an NVBitFI-based tool for ISA injections (permanent stuck-at faults on register files and functional units) on GPUs. They apply both to LeNet5, MobileNetV2, and ResNet18, with and without three hardening techniques (Ranger, Swap ReLU6, Adaptive Clipper), and compare fault-class distributions and accuracy. The key claim is that APP-level FI overestimates the beneficial impact of hardening techniques and yields a different—and, in the authors' view, misleading—ranking of those techniques compared with ISA-level FI.","tokens_in":10334,"tokens_out":5313,"duration_ms":55378,"significance":"If the central claim were rigorously supported, the paper would be a useful methodological warning for the DNN reliability community, where APP-level FI is widely used to evaluate hardening techniques. The raw comparative data, including exhaustive ISA fault counts and the fault-class distributions in Fig. 2, are valuable and appear internally consistent. Strong points are the concrete fault counts (10,496 register and 1,536 FU injections per DNN) and the attempt to equalize BER across abstraction levels. However, as detailed below, the comparison varies multiple factors simultaneously, and the claim that ISA results are 'closer to the real ones' lacks any ground-truth reference, so the paper's strongest conclusions are not currently supported.","major_comments":[{"comment":"The claimed 'fair comparison' between ISA and APP is undermined by confounds. ISA injection (Section III.A) places permanent stuck-at faults on register-file destinations and FU inputs/outputs on an RTX 3060 Ti, while APP injection (Section III.B) applies single or multiple random bit-flips to weight tensors and neuron output feature maps on V100 GPUs. A stuck-at fault produces correlated, repeated errors whenever the target register or FU is exercised, whereas an APP bit-flip is an independent one-shot perturbation. The BER equalization described in Section III.C matches only the average activation rate; it does not control for error polarity, spatial or temporal error structure, affected data type (addresses and control vs. weights and activations), or GPU microarchitecture. Therefore the rank reversal in Fig. 2 and Table I cannot be attributed uniquely to the abstraction level, which is the paper's central conclusion.","section":"Section III.C, Table I"},{"comment":"The statement that ISA FI results are 'closer to the real ones' is an unsupported assertion. The paper provides no ground-truth measurement—such as hardware-level FI, radiation testing, or RTL/microarchitectural simulation—against which either abstraction can be validated. Disagreement between two proxies does not privilege one as more realistic. The authors should either remove this claim or substantiate it with an external reference, since it is load-bearing for the conclusion that APP FI is 'misleading.'","section":"Section IV.A and Section V"},{"comment":"The key example illustrating the ranking reversal is internally contradictory. Section IV.A states that for MobileNetV2, under Regs and FUs corruptions, 'Adaptive Clipper outperforms the other hardening strategies,' while 'APP FI highlights that Swap ReLU6 is the best-performing technique.' The Conclusion (Section V) inverts this: 'APP FI identifies Adaptive Clipper as the best solution ... while for ISA FI the best solution is Swap ReLU6.' Table I does not straightforwardly support either statement: under ISA FUs for MobileNetV2, Ranger attains the highest accuracy (68.47%), whereas under APP Weights at BER=10^-6, Swap ReLU6 is best (84.12%). Because the ranking reversal is the main result, this contradiction must be resolved.","section":"Section V vs. Section IV.A"},{"comment":"The cross-abstraction fault-distribution comparison is not controlled. APP SBFs corrupt only weight tensors, while ISA stuck-at faults target registers and FUs. The ISA campaign exhaustively injects faults on one SM and a single thread, whereas the APP campaign uses statistical sampling at 95% confidence over the whole execution. The large differences in Critical-SDC and DUE rates (e.g., DUEs up to 47.5% in Regs) may therefore reflect the choice of fault location and campaign scope rather than the abstraction level. The conclusion that 'APP FIs report a limited percentage of Critical SDCs' and thus overestimate hardening effectiveness depends on comparing experiments that differ in coverage and target, not only in abstraction.","section":"Section IV.A (Fig. 2)"}],"minor_comments":[{"comment":"The fault-class definitions are unclear: Masked is defined as 'if they match' and Safe-SDC as 'when the fault does not change the prediction,' which appear to describe the same outcome. Clarify the distinction, for example whether Safe-SDC refers to a change in confidence while the top-1 label remains correct.","section":"Section III.C"},{"comment":"There are minor typos: 'understimate' should be 'underestimate' (Section V), 'crush' should be 'crash' (Section III.A), and 'As illustrated in 1' should read 'in Figure 1' (Section III.A).","section":"Section V and Section III.A"},{"comment":"The phrase 'for the first time' is used without a systematic comparison to prior work beyond the cited references; consider softening to 'to our knowledge, no prior study has directly compared...'.","section":"Section I"},{"comment":"The color scale used to rank results may be ambiguous when printed in grayscale; adding numerical ranks or arrows would improve readability.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short conference-style manuscript from a group that has already published extensively on ISA-level FI for GPUs and DNNs (refs [22]–[24]). The novel contribution is the direct comparison with APP FI for hardening evaluation, which is timely. However, the comparison design has serious confounds that currently prevent the strong conclusions. The internal contradiction between Section IV.A and Section V must also be fixed. I believe the raw data and the methodological message ('FI abstraction affects hardening assessment') are salvageable if the authors reframe the conclusions, remove the unvalidated 'closer to reality' claim, and either control the confounds or explicitly discuss them as limitations. For these reasons I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine first: a direct comparison of application-level and instruction-level permanent fault injection for ranking three DNN software hardening strategies, and it shows the ranking flips depending on abstraction. That is a real and useful observation. Prior work used one abstraction or the other; this is the first head-to-head on hardened DNNs, and the reported fault-class distributions (Masked, Safe-SDC, Critical-SDC, DUE) across three DNNs and three hardening methods are worth knowing. The BER equalization effort—computing how often a stuck-at fault is excited and using that to pick APP bit-flip rates—is a reasonable attempt at fairness, even if it only matches an average rate.\n\nThe load-bearing problem is the inference from \"different results\" to \"APP is misleading and ISA is accurate.\" No hardware or microarchitectural ground truth is provided; \"closer to the real ones\" is an assertion. Also, the two campaigns differ in injection target (weights/activations vs register files/FUs), fault model (single/multiple bit-flips vs stuck-at), timing, and GPU (V100 vs RTX 3060 Ti). A single stuck-at fault produces correlated, repeated errors; APP bit-flips are independent and one-shot. So the rank reversal in Fig. 2 and Table I could be due to any of these confounds. The BER equalization only matches activation frequency, not error structure, fault polarity, or affected data types. There is also an internal contradiction: Section IV.A says APP FI on MobileNet V2 ranks Swap ReLU6 best, while the conclusion says APP ranks Adaptive Clipper best for MobileNet V2. That needs fixing.\n\nMinor: no error bars or statistical significance are reported, even though the campaign sizes are large. The paper would be stronger if it reframed the conclusion as \"FI methodology can change hardening rankings\" rather than \"APP FI is wrong.\"\n\nWho is this for? Researchers doing DNN reliability evaluation, especially in safety-critical contexts. It deserves a serious referee because the central observation is plausible and the empirical setup, while confounded, is not fundamentally broken. My recommendation: send it to peer review, with a request for major revision—reframe the claim, fix the contradiction, add error bars, and either add a ground-truth comparison or at least acknowledge the confounds explicitly.","headline":"Useful head-to-head comparison of APP vs ISA fault injection for ranking DNN hardening techniques, but the strong claim that APP-level FI is misleading and ISA-level FI is accurate goes beyond what the confounded setup can support.","tokens_in":10873,"tokens_out":1876,"would_cite":false,"duration_ms":19338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fault-injection abstraction level changes DNN hardening rankings.","keywords":["deep neural networks","fault injection","application-level fault injection","instruction-level fault injection","software hardening","permanent faults","GPU reliability","silent data corruption"],"falsifier":"Run the same application-level multiple-bit-flip campaigns on the same GPU that was used for instruction-level injection, with bit-flip locations and timing matched to the instruction sites where stuck-at faults were injected; if the hardening ranking then matches the instruction-level ranking, the original difference came from platform or fault-model mismatches rather than from abstraction level, whereas if the ranking still flips, the abstraction-level interpretation is supported.","tokens_in":9877,"feed_emoji":"🧪","tokens_out":5857,"duration_ms":56442,"temperature":0.7,"pith_summary":"The paper claims that the usual way of evaluating software hardening techniques for deep neural networks — perturbing weights or neuron outputs at the application level — can rank the techniques in the wrong order. It compares this application-level fault injection with instruction-level injection that models permanent stuck-at faults in a GPU's register files and functional units. Across three DNNs and three hardening methods, the two abstraction levels give different fault distributions, different accuracy losses, and a different winner. The authors argue that instruction-level results, being closer to the hardware, are the trustworthy ones and that application-level campaigns overstate how much hardening helps.","feed_headline":"Fault-injection level flips DNN hardening rankings","feed_subtitle":"Application-level tests overstate hardening gains; instruction-level faults tell a different story.","key_machinery":"The key mechanism is a BER bridge between the two abstraction levels. After each instruction-level stuck-at injection, the authors count how many times the fault is excited — the target bit assumes the opposite polarity during kernel execution — and divide by the total uses of that register to obtain a Bit Error Rate. They then use the observed BER range to configure application-level multiple bit-flips at the same error rate, holding the error-activation rate roughly constant so differences can be attributed to the abstraction level. The evaluation is organized around a four-way fault classification (Masked, Safe-SDC, Critical-SDC, DUE) and the Relative Accuracy Degradation metric.","core_discovery":"The central discovery is that the abstraction level used for fault injection is not a neutral experimental detail: it changes the measured reliability of the same DNN and the same hardening method. Application-level injection underestimates Critical SDCs (no more than 10% of faults, versus up to 31% at instruction level) and misses detectable unrecoverable errors entirely, although register faults can produce DUEs in up to 47.5% of cases. As a direct consequence, the ranking of hardening techniques flips: in the paper's example, application-level injection picks Adaptive Clipper as the best hardening for MobileNetV2 while instruction-level injection picks Swap ReLU6. The authors conclude that instruction-level results are closer to what happens on real hardware and therefore APP-based conclusions about hardening effectiveness are misleading.","pith_inferences":["A testable extension is to repeat the comparison with a transient (single-event upset) fault model; the ranking flip may be smaller or larger, which would delimit how much of the effect is specific to permanent faults.","The paper's BER-matching step could be strengthened by matching fault excitation timing and data-dependent activation between levels, not just average rates; without that, part of the observed difference may be an artifact of the calibration.","If the effect generalizes, published comparisons of DNN hardening that rely solely on application-level fault injection are not portable across GPUs, and reliability evaluation standards should mandate the injection abstraction and hardware target.","The implication for practice is that cheap application-level fault injection may still be useful for screening many candidates, with instruction-level injection reserved for the top-ranked few — a two-stage evaluation worth testing."],"forward_implications":["If the central claim holds, reliability rankings produced by application-level fault injection alone should not be used to choose a hardening technique for deployment.","Hardening methods that look neutral or harmful at instruction level, such as Adaptive Clipper on ResNet-18, would be missed or mis-ranked by APP-only studies.","Because instruction-level injection reveals DUEs that application-level injection cannot produce, safety-critical systems need hardware-aware fault injection to budget for crashes and hangs, not just accuracy loss.","The same hardening technique can be the best for one DNN and ineffective or harmful for another, and this interaction only becomes visible at instruction level."],"supporting_citations":[{"why":"Supplies the application-level fault injector used to corrupt weights and neuron outputs via forward hooks.","marker":"[17]"},{"why":"Supplies the instruction-level GPU fault injection framework that the paper adapts for permanent stuck-at faults.","marker":"[20]"},{"why":"Defines the Swap ReLU6 hardening technique evaluated in both fault injection campaigns.","marker":"[7]"},{"why":"Defines the Ranger range-restriction hardening technique evaluated in both fault injection campaigns.","marker":"[11]"},{"why":"Defines the Adaptive Clipper hardening technique evaluated in both fault injection campaigns.","marker":"[12]"},{"why":"Motivates the claim that application-level fault injection is hardware-agnostic and can mislead reliability assessment.","marker":"[16]"},{"why":"Provides the prior approach for injecting permanent faults in a GPU running a DNN, which the instruction-level campaign extends.","marker":"[22]"},{"why":"Supplies the error-list format (tail size, Neuron Error Rate, bit locations) used to configure application-level neuron-output corruptions.","marker":"[29]"}],"fun_headline_variants":["Fault-injection depth changes DNN hardening winner","Application-level FI understates DNN faults, flips rankings","ISA-level faults overturn DNN hardening leaderboard","Choosing FI abstraction reshapes DNN defense ranking","Instruction-level injection reveals true DNN hardening order"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that matching the Bit Error Rate makes the application-level and instruction-level campaigns equivalent; in reality the two campaigns also differ in what they corrupt, in the fault model (stuck-at vs bit-flip), and in the GPU used, so those differences could cause the ranking change instead.","fun_headline_variants_meta":{"raw":{"variants":["Fault-injection depth changes DNN hardening winner","Application-level FI understates DNN faults, flips rankings","ISA-level faults overturn DNN hardening leaderboard","Choosing FI abstraction reshapes DNN defense ranking","Instruction-level injection reveals true DNN hardening order"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000128,"raw_usage":{"total_tokens":1050,"prompt_tokens":806,"completion_tokens":244,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":422,"completion_tokens_details":{"reasoning_tokens":168}},"tokens_in":422,"tokens_out":244,"duration_ms":3834,"temperature":1.0,"reasoning_tokens":168,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:44:30.796792+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same application-level multiple-bit-flip campaigns on the same GPU that was used for instruction-level injection, with bit-flip locations and timing matched to the instruction sites where stuck-at faults were injected; if the hardening ranking then matches the instruction-level ranking, the original difference came from platform or fault-model mismatches rather than from abstraction level, whereas if the ranking still flips, the abstraction-level interpretation is supported.","supporting_citations":[{"cited_title":"Pytorchfi: A runtime perturbation tool for dnns,","cited_arxiv_id":null,"evidence_quote":"Supplies the application-level fault injector used to corrupt weights and neuron outputs via forward hooks."},{"cited_title":"Nvbitfi: Dynamic fault injection for gpus,","cited_arxiv_id":null,"evidence_quote":"Supplies the instruction-level GPU fault injection framework that the paper adapts for permanent stuck-at faults."},{"cited_title":"Transient-fault-aware design and training to enhance dnns reliability with zero-overhead,","cited_arxiv_id":null,"evidence_quote":"Defines the Swap ReLU6 hardening technique evaluated in both fault injection campaigns."},{"cited_title":"A low-cost fault corrector for deep neural networks through range restriction,","cited_arxiv_id":null,"evidence_quote":"Defines the Ranger range-restriction hardening technique evaluated in both fault injection campaigns."},{"cited_title":"Enhancing the reliability of split computing deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Defines the Adaptive Clipper hardening technique evaluated in both fault injection campaigns."},{"cited_title":"Pros and cons of fault injection approaches for the reliability assessment of deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Motivates the claim that application-level fault injection is hardware-agnostic and can mislead reliability assessment."},{"cited_title":"Evaluating the impact of permanent faults in a gpu running a deep neural network,","cited_arxiv_id":null,"evidence_quote":"Provides the prior approach for injecting permanent faults in a GPU running a DNN, which the instruction-level campaign extends."},{"cited_title":"Evaluating the reliability of supervised compression for split computing,","cited_arxiv_id":null,"evidence_quote":"Supplies the error-list format (tail size, Neuron Error Rate, bit locations) used to configure application-level neuron-output corruptions."}],"review_version":1}