{"id":"b59cde21-8653-4ab7-aeba-1f7c3a5e1b16","arxiv_id":"1909.02190","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"DeepMorph claims to locate the root cause of poor deep network performance by comparing internal layer outputs, called data flow footprints, against patterns learned from training data.","lead":"DeepMorph is a proposed tool that looks inside a neural network's layers to explain why it performs badly, choosing among bad architecture, insufficient data, and wrong labels. The idea is useful for debugging models, but the paper is too thin to verify that it works.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DeepMorph's reference patterns are learned from the very model and data under diagnosis, so the reported dominant ratios may reflect the injected defect's confound rather than an independent footprint signature.","rationale":"The reader's weakest assumption is that internal layer outputs, after auxiliary softmax training, carry a stable and separable signature per defect and that reference patterns are not contaminated. My concern is the same condition, sharpened: the references are trained on the defective model and defective data, and the paper provides no control for this confound. This is load-bearing because the entire empirical case is Table I, and the table's reported ratios are uninterpretable without a definition of the footprint/ratio computation or an independence argument. I also agree with the reader that the paper is REJECT rather than CONDITIONAL: the missing definitions and controls are the core of the method, not a minor presentation issue. In good faith, the underlying idea of reading internal activations to triage model failures is plausible, and the paper should be credited for a concrete falsifiable design; but as written, the central claim is not established, and no independent verification artifacts (code, derivations, baselines) are provided.","tokens_in":5151,"tokens_out":6849,"duration_ms":71928,"concrete_test":"Run the pipeline on a defect-free version of one configuration (e.g., CIFAR-10 DenseNet) and confirm whether any defect type is reported as largest; then, for the UTD-injected model, retrain only the auxiliary softmax layers on a clean holdout set that excludes relabeled examples and recompute the ratios on the same faulty cases. If the clean model yields a dominant defect, or if the UTD ratio drops below the largest when reference patterns are clean, the reference patterns were contaminated by the defect under diagnosis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the ratio values in Table I identify the injected defect. That requires the per-class execution patterns used as references to be stable, separable signatures that are not themselves contaminated by the defect being diagnosed. The paper never establishes this. Section III says the softmax-instrumented model 'is used to learn the execution pattern of the training cases for each target class' from the same target model, and Section IV says it is trained on the training data. In the UTD setting those training labels are deliberately wrong; in the ITD setting the per-class training distribution is deliberately skewed; in the SD setting the auxiliary layers are fitted inside the already-defective architecture. In every case the reference patterns are extracted from the defective system, not from an independent healthy baseline. Section IV then states only that DeepMorph 'produces the ratio of each type of defects,' without defining footprint specifics, the comparison rule, or the ratio computation. Consequently Table I cannot distinguish a genuine diagnostic signal from an artifact: for a UTD model, the auxiliary softmax layers may simply memorize the corrupted labels, and the high UTD ratio would reflect that memorization rather than an intrinsic activation signature. The single table, with no clean-model control, no separability analysis, and no repeated runs, does not support the 'always largest' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeepMorph, a white-box diagnostic tool that aims to locate the root cause of poor performance in deep neural network classifiers. The method instruments a target model with auxiliary softmax layers at hidden layers, learns per-class 'execution patterns' from the training data, and compares 'data flow footprint specifics' of faulty test cases against these patterns to compute a ratio for each of three defect types: structure defect (SD), insufficient training data (ITD), and unreliable training data (UTD). The paper reports in Table I that for four architectures (LeNet, AlexNet, ResNet, DenseNet) and two datasets (MNIST, CIFAR-10), the injected defect always receives the highest ratio, and concludes that DeepMorph can effectively locate defects.","tokens_in":5388,"tokens_out":2807,"duration_ms":29825,"significance":"If the core mechanism is sound, DeepMorph would address a real need: distinguishing model defects from inherent model limitations and guiding developers toward targeted fixes. The paper identifies a meaningful problem, and the diagonal pattern in Table I is suggestive. However, the significance cannot be assessed as submitted, because the central concepts ('footprint specifics', 'execution pattern', the comparison rule, and the ratio computation) are never defined, no algorithm or equations are given, and the experimental protocol is too coarse to rule out artifacts. The manuscript has no machine-checked proofs, no reproducible code, and no formal falsifiable prediction beyond a single table.","major_comments":[{"comment":"The core mechanism of the method is undefined. The text states that DeepMorph builds a 'softmax-instrumented model', learns 'the execution pattern of the training cases for each target class', extracts 'data flow footprint specifics from the intermediate outputs of hidden layers', and 'compares the footprints against the execution pattern of each target class', but none of these terms is given a formal definition. There is no equation, algorithm, or pseudocode describing how intermediate activations are mapped to the three ratios reported in Table I. Without this definition, the experimental table cannot be reproduced or checked, so the paper's central claim is not verifiable as written.","section":"Section III"},{"comment":"The ratio values in Table I are presented as the only evidence for the central claim, but the experimental setup is severely underspecified. The text does not state how many faulty cases are analyzed, how the auxiliary softmax layers are trained (architecture, loss, optimizer, hyperparameters, epochs), how the injected defects are quantified (e.g., how many training examples are removed for ITD or relabeled for UTD, which layers are removed for SD), or whether the reported ratios are averaged over multiple runs. No error bars, standard deviations, or statistical significance tests are provided. In several cells the diagonal margin is small (e.g., ResNet SD row: 0.433 vs 0.481; ResNet UTD row: 0.577 vs 0.285), so without variance information the 'always the largest' claim is not supported.","section":"Section IV"},{"comment":"There is a potential circularity in the reference-pattern construction. The auxiliary softmax layers are trained on the very training data being diagnosed, and in all three defect settings this training data is defective: UTD contains corrupted labels, ITD has a skewed class distribution, and SD means the auxiliary layers are trained inside the already-defective architecture. The per-class 'execution pattern' used as the reference is therefore derived from the defective system, not from a healthy baseline. The paper does not provide a clean-model control or an argument that the reference patterns are stable and separable despite being built from contaminated data. Consequently, the high ratio for the injected defect could reflect memorization of the defect in the reference pattern rather than an independent diagnostic signature. This must be addressed with a controlled experiment comparing defective and non-defective models.","section":"Section III and Section IV"},{"comment":"The claim that 'for all cases, DeepMorph is able to locate the injected defect effectively' is strong, but the table provides no comparison against any baseline or null model. There is no experiment showing what the ratios look like for a healthy model, no ablation isolating the effect of each defect, and no comparison with existing DL debugging techniques such as DeepMODE or DeepGauge. Without these controls, the possibility that the reported ratios are dominated by a trivial property (for example, the overall accuracy drop) is not excluded.","section":"Section IV, Table I"}],"minor_comments":[{"comment":"The word 'tomography' appears in the title and in the tool name DeepMorph (DNN Tomography) but is not used or explained anywhere in the body; the paper should either define the analogy or drop the term.","section":"Abstract and Section I"},{"comment":"The related-work section is very brief and does not discuss how DeepMorph differs operationally from existing debugging tools such as MODE, DeepDebug, or the model-repair approach cited in [14]; a concrete comparison of inputs, outputs, and assumptions would help position the contribution.","section":"Section II"},{"comment":"The description of the DL models is incomplete: the paper cites a GitHub repository for CIFAR-10 architectures but does not specify the exact configuration of the LeNet and AlexNet models used for MNIST, nor the training procedure (optimizer, learning rate, epochs). The paper should list the model parameters and training details to enable reproduction.","section":"Section IV"},{"comment":"There is a grammatical error: 'DeepMorph can successfully identified the injected defects' should be 'can successfully identify'. Also, the phrase 'which aims at degrading the models via a weaker network structure' in the SD injection description is redundant, since removing layers already implies a weaker structure.","section":"Section IV"},{"comment":"Figure 1 is never described in the text; the boxes and arrows are not explained, and the reader cannot infer the flow of information or the location of the ratio computation from the figure alone.","section":"Figure 1"}],"recommendation":"reject","confidential_remarks":"This manuscript is a three-page extended abstract rather than a complete research paper. The central claim rests entirely on Table I, but the method is not specified well enough to be reproduced, and the experimental protocol leaves too many degrees of freedom. The circularity concern about reference patterns built from defective training data is not merely a presentation issue; it is a correctness risk that would require additional controlled experiments to resolve. Even with major revisions, the paper would need to be substantially expanded with a formal algorithm description, a complete experimental setup, and a clean-model control. The current submission does not meet the standard for publication in a rigorous venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis short paper makes one empirical claim: by instrumenting a network with auxiliary softmax layers and comparing 'internal data flow footprints' of misclassified cases against per-class reference patterns, DeepMorph can tell apart three defect types — structure defects, insufficient training data, and unreliable training data. The table reports the ratio of each defect type and the diagonal dominates, so the injected defect is always the largest. If the method worked, it would be a nice addition to the ML debugging toolbox.\n\nWhat's actually new: I haven't seen anyone use auxiliary classifiers at intermediate layers to triage these three failure causes. That's a reasonable extension of existing work on intermediate representation inspection. The paper is honest that it's a first step, and the related work section is adequate.\n\nNow the soft spots — and they are substantial. The core method is never defined. We are never told what a 'footprint specific' is, how the comparison to an 'execution pattern' is computed, or what the ratio in Table I actually measures. No equations, no pseudocode, no code release. The auxiliary softmax training is described in one sentence. Without that, the central claim is not verifiable. The stress test concern is on target: the reference patterns are learned from the same training data whose defects are being diagnosed. In the UTD case, those labels are intentionally wrong; in the ITD case, the per-class distribution is skewed; in the SD case, the auxiliary layers are fit inside the already-defective architecture. So the ratios may just reflect the auxiliary layers memorizing the defect, rather than a separable activation signature. A clean-model control, repeated runs with error bars, and an ablation showing the footprints are stable across healthy models would help. None are present. Also, the margins in Table I are sometimes thin (e.g., ResNet SD row: 0.481 vs 0.433 ITD), so 'always largest' is not as strong as it sounds.\n\nWhere does that leave us? The idea is worth pursuing, but this paper is a sketch, not a validated tool. It would be a good starting point for a longer paper with details, reproducibility, and proper statistical evaluation. As written, I wouldn't rely on the result, and I wouldn't cite it as evidence. But I'd send it to peer review rather than desk reject: the problem is real, the approach is plausible, and a referee could push the authors to supply the missing material.\n\nHope this helps.","headline":"A plausible debugging idea, but the paper is too under-specified to support the claim that DeepMorph can reliably detect the injected defect.","tokens_in":5888,"tokens_out":3026,"would_cite":false,"duration_ms":29119,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a deep neural network's internal data flow footprints, read through auxiliary softmax layers, reveal which of three defect types—bad network structure, unrepresentative training data, or unreliable labels—is behind…","keywords":["deep neural network defects","data flow analysis","model debugging","root cause analysis","softmax instrumentation","structure defect","training data quality","misclassification diagnosis"],"falsifier":"Run DeepMorph on a model where two defects are injected together (for example, dropping part of one class and mislabeling another class). If the reported highest ratios do not correspond to the two injected defects, the largest-ratio rule fails. A second direct check is to corrupt the training set used to build reference patterns and see whether the injected defect still yields the largest ratio.","tokens_in":4907,"feed_emoji":"🔬","tokens_out":6841,"duration_ms":62961,"temperature":0.7,"pith_summary":"The paper sets out to separate intrinsic model quality from fixable defects in deep neural networks, a distinction that developers currently struggle to make. It claims that the intermediate outputs of hidden layers, collected as data flow footprints and passed through auxiliary softmax layers, carry enough information to tell apart three defect types: an improper network structure, training data that underrepresents the production distribution, and training labels that are unreliable. DeepMorph reports a ratio for each defect type, and the paper's experiments on four image classifiers show the injected defect always receives the largest ratio. If that holds, a developer who sees a low-accuracy model can run DeepMorph, read off the dominant defect, and target the fix.","feed_headline":"DeepMorph reads layer traces to name a DNN's dominant defect","feed_subtitle":"A highest-ratio rule on internal footprints tells developers where to fix the model.","key_machinery":"The load-bearing mechanism is the softmax-instrumented model: the target network with auxiliary softmax layers inserted after hidden layers and trained on the training data. These auxiliary layers convert hidden representations into class probabilities, so DeepMorph can describe each training case's classification path as an execution pattern and each faulty case as a set of footprint specifics. Comparing footprint specifics to per-target-class execution patterns produces the three defect ratios; the largest ratio is the reported root cause.","core_discovery":"The central claim is that defects in a deep neural network can be located by treating the network's layer-by-layer execution as data flow and comparing each faulty input's footprint against reference patterns learned from the training data. Concretely, DeepMorph adds an auxiliary softmax layer after hidden layers of the target model, trains those layers on the training set, and uses their outputs as a footprint of how the model classifies a case. For each faulty case it computes a ratio for structure defect, insufficient training data, and unreliable training data, and declares the defect with the highest ratio to be dominant. The experimental section states that for all injected-defect cases across LeNet, AlexNet, ResNet-34, and DenseNet-40 on MNIST and CIFAR-10, the injected defect always yields the largest ratio.","pith_inferences":["A natural but untested extension is combined defects: the injection experiments place one defect at a time, while real models may suffer from multiple problems simultaneously; whether the highest-ratio rule still points to a dominant cause is an open question the paper does not address.","Because the reference execution patterns are learned from the training data, a training set that already contains the defect under diagnosis could contaminate the baseline; one testable extension is to build reference patterns from a trusted validation split instead.","The reported ratios are all below 1 and the structure-defect row has the lowest diagonal values (roughly 0.48-0.63), so an operator may need a calibration step or confidence threshold for low-margin cases; the paper does not provide one.","The same footprint idea could be adapted to other failure modes, such as optimization problems or class imbalance, by defining their own reference patterns from the same layer outputs."],"forward_implications":["A developer can use DeepMorph before retraining: run the faulty cases, take the highest ratio, and choose between fixing network structure, collecting more representative data, or cleaning labels.","Because the reference patterns are built from training data alone, DeepMorph does not require prior knowledge of which defect exists or a labeled set of defective models to start diagnosing.","The method's consistency across four architectures and two datasets suggests it could serve as a general first pass for image classifiers before deeper per-layer debugging is applied.","The ratio ordering rather than an absolute threshold means the tool points at a defect category even when the model's overall accuracy is poor; the exact magnitude is secondary."],"supporting_citations":[{"why":"Provides the CIFAR-10 benchmark dataset used for the ResNet and DenseNet evaluations.","marker":"[29]"},{"why":"Supplies the LeNet architecture used for the MNIST experiments.","marker":"[31]"},{"why":"Supplies the AlexNet architecture used for the MNIST experiments.","marker":"[32]"},{"why":"Supplies the ResNet-34 architecture used for the CIFAR-10 experiments.","marker":"[1]"},{"why":"Supplies the DenseNet-40 architecture used for the CIFAR-10 experiments.","marker":"[33]"},{"why":"Supplies the deep learning framework in which DeepMorph is implemented.","marker":"[5]"},{"why":"Supplies the open-source model implementations adapted for the experimental study.","marker":"[30]"}],"fun_headline_variants":["DeepMorph pinpoints DNN defects via data-flow ratios","Data flow analysis reveals a DNN's dominant defect","Layer footprints expose the root cause of DNN flaws","New method names a DNN's top defect from traces","DeepMorph: data flow footprints rank DNN defects"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"DeepMorph assumes that each defect type leaves its own recognizable trace in the layer outputs, and that these traces stay distinct when compared with patterns built from the training data.","fun_headline_variants_meta":{"raw":{"variants":["DeepMorph pinpoints DNN defects via data-flow ratios","Data flow analysis reveals a DNN's dominant defect","Layer footprints expose the root cause of DNN flaws","New method names a DNN's top defect from traces","DeepMorph: data flow footprints rank DNN defects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000364,"raw_usage":{"total_tokens":1880,"prompt_tokens":783,"completion_tokens":1097,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":399,"completion_tokens_details":{"reasoning_tokens":1017}},"tokens_in":399,"tokens_out":1097,"duration_ms":8410,"temperature":1.0,"reasoning_tokens":1017,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:57:38.325812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DeepMorph on a model where two defects are injected together (for example, dropping part of one class and mislabeling another class). If the reported highest ratios do not correspond to the two injected defects, the largest-ratio rule fails. A second direct check is to corrupt the training set used to build reference patterns and see whether the injected defect still yields the largest ratio.","supporting_citations":[{"cited_title":"The cifar-10 dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the CIFAR-10 benchmark dataset used for the ResNet and DenseNet evaluations."},{"cited_title":"Handwritten digit recognition with a back-propagation network,","cited_arxiv_id":null,"evidence_quote":"Supplies the LeNet architecture used for the MNIST experiments."},{"cited_title":"Imagenet classification with deep convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the AlexNet architecture used for the MNIST experiments."},{"cited_title":"Deep residual learning for image recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies the ResNet-34 architecture used for the CIFAR-10 experiments."},{"cited_title":"Densely connected convolutional networks,","cited_arxiv_id":null,"evidence_quote":"Supplies the DenseNet-40 architecture used for the CIFAR-10 experiments."},{"cited_title":"Tensorflow: A system for large-scale machine learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the deep learning framework in which DeepMorph is implemented."},{"cited_title":"Convolutional neural networks for cifar-10,","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source model implementations adapted for the experimental study."}],"review_version":1}