{"id":"e8adda62-cd90-4659-9321-7e5f1733c262","arxiv_id":"2509.08717","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A CNN classifies Bewick's wren song variants at 94.8% accuracy, and ensembling Grad-CAM and DeepLIFT saliency maps highlights complementary song features.","lead":"A team recorded Bewick's wren songs, trained a neural network to tell two regional song variants apart, and compared four AI explanation methods on the resulting spectrograms. The paper says combining two of those methods, Grad-CAM and DeepLIFT, gives fuller explanations and hints at hidden sub-populations in the songs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The model's Eastern/Mexican discrimination is not shown to generalize beyond recording sites; no group-wise split or provenance control is reported.","rationale":"The reader's weakest-assumption identification matches the most load-bearing issue: the absence of any control for recording provenance leaves the central biological claim underdetermined. This is not merely outside-consensus disagreement; it is a correctness risk internal to the paper's argument, because the classification target (Eastern vs Mexican variant) is geographically defined and the recordings come from two states. Without a split that isolates recording site or individual, the reported accuracy and the saliency maps could be driven by non-song cues. I would keep the verdict CONDITIONAL rather than REJECT: the concern is concrete and testable, but it has not been shown to fail—the authors may have collected metadata or split appropriately without documenting it. If the proposed group-wise split test were run and accuracy collapsed, the verdict would move to REJECT; if accuracy and explanation patterns persisted on held-out sites, the concern would be resolved. The secondary point about the max ensemble being definitionally larger than either component is worth noting but does not by itself change the verdict, since the paper's qualitative claims about interpretability rest mainly on visual inspection rather than on Figure 6 alone.","tokens_in":9874,"tokens_out":2595,"duration_ms":374861,"concrete_test":"Request the recording metadata (site, date, individual identifier if available) and rerun the full pipeline with a group-wise split that keeps all clips from the same recording session or individual in the same fold, ideally with a leave-one-site-out evaluation (e.g., train on New Mexico recordings, test on Arizona recordings). If accuracy drops toward chance, or if t-SNE cluster labels align with site rather than song variant, then the model and explanations cannot be attributed to song-content differences.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central biological interpretation—that the CNN separates Eastern and Mexican Bewick's wren song variants based on song content, and that the saliency maps therefore reveal song features—depends on the classifier not exploiting recording-site, individual, or ambient-acoustic cues. Section III-A states that all recordings were made in Arizona and New Mexico, and Section III-B reports 1854 training and 806 test samples but never states that clips from the same recording session, individual, or location were kept together when splitting. If clips were randomly assigned, the same bird or same recording conditions can appear in both train and test, inflating the 94.8% accuracy and making the saliency maps reflect site or recording artifacts rather than biologically meaningful song features. The t-SNE sub-population claim in Section IV-C is similarly vulnerable: clusters could be produced by recording site, background noise, or individual identity rather than song-type groups. Table I's background-color comparison only varies spectrogram background and does not control for recording provenance. A related but secondary issue is that the quantitative support for the ensemble-max claim (Figure 6) is partly definitional: an element-wise maximum will always exceed or equal each component heatmap at every pixel, so showing that it activates more pixels above a threshold does not independently establish that it captures more meaningful features.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains a CNN to classify spectrograms of Eastern and Mexican Bewick's wren songs, reaching 94.8% accuracy on a mixed-background dataset, and applies four XAI methods (LIME, SHAP, Grad-CAM, DeepLIFT) to interpret predictions. The authors propose ensemble saliency maps combining Grad-CAM and DeepLIFT via weighted averaging and element-wise maximum, reporting that the max ensemble highlights more regions above importance thresholds. They also apply PCA and t-SNE to the latent space and claim that t-SNE reveals sub-populations within each song variant, supported by qualitative inspection of Grad-CAM/DeepLIFT heatmaps for selected cluster samples.","tokens_in":10095,"tokens_out":3617,"duration_ms":34992,"significance":"If the evidence were sound, the paper would offer a useful comparative evaluation of XAI techniques in bioacoustics and a practical ensemble strategy for generating more complete saliency maps, as well as a hypothesis-generating latent-space analysis. The work addresses an under-explored application of XAI to bird song classification, and the qualitative biological observations (e.g., emphasis on terminal trills) are interesting. However, the current manuscript does not yet substantiate its central claims because key quantitative support is partly definitional and several analyses rely on very small sample sizes without control for recording provenance.","major_comments":[{"comment":"The manuscript never describes how the 1854 training and 806 test samples were split with respect to recording sessions, individual birds, or geographic locations. Since all recordings were collected in Arizona and New Mexico (Section III-A), a random clip-level split can place the same bird or the same ambient recording conditions in both train and test, which would inflate the reported 94.8% accuracy and make the saliency maps reflect recording-site or background-acoustic artifacts rather than song content. This is load-bearing for the biological interpretation of the explanations and for the t-SNE sub-population claim. The authors should clarify the split procedure and, if possible, perform a group-wise split (e.g., by recording session or location) and report the resulting metrics, or provide control analyses ruling out site and background cues.","section":"III-B"},{"comment":"The quantitative support for the max ensemble is definitional. Since Hmax(i,j) = max(H_CAM(i,j), H_LDF(i,j)) by Equation 2, the fraction of pixels above any threshold t is at least as large for the max ensemble as for either Grad-CAM or DeepLIFT individually, simply because the max of two values exceeds t whenever either component exceeds t. Therefore, Figure 6's observation that 'Ensemble Max' consistently activates more regions above thresholds is mathematically guaranteed and does not independently establish that the ensemble captures more meaningful or discriminative features. The authors should either replace this metric with a more substantive evaluation (e.g., agreement with expert-annotated song regions, or a controlled comparison of downstream tasks) or explicitly reframe the claim as a qualitative property rather than empirical evidence of superiority.","section":"IV-B, Figure 6"},{"comment":"The central qualitative claims about the XAI outputs are based on a single randomly selected test sample per class. Statements such as 'both Grad-CAM and DeepLIFT consistently highlight low- to mid-frequency elements' and 'DeepLIFT produced the most interpretable explanations for bird song experts' are made from exactly one Eastern and one Mexican example. This is not sufficient to establish consistency or to justify general conclusions about which XAI method is more interpretable. The authors should report aggregated statistics over a larger set of samples (e.g., multiple examples per class, or quantitative overlap of highlighted regions with the spectrogram signal), or at minimum present several representative samples to show that the patterns are reproducible.","section":"IV-A, Figures 3-4"},{"comment":"The sub-population claim rests on visual t-SNE clusters without any quantitative validation. t-SNE often produces apparent clusters even on random or uniformly distributed data, and the manuscript provides no stability analysis (e.g., repeated runs with different perplexities), no cluster quality metric (e.g., silhouette score), and no control for recording site, individual identity, or ambient noise. The conclusion that the clusters represent 'meaningful subgroups' within Eastern and Mexican song variants is therefore not supported. The authors should add quantitative cluster validation and control analyses, and should temper the claims if such analyses are not feasible with the current dataset.","section":"IV-C, Figures 5 and 7"}],"minor_comments":[{"comment":"There is a typo: 'The STF transformation' should read 'The STFT transformation'.","section":"III-A"},{"comment":"The abstract says 'improve trust and interoperability' where 'interpretability' is presumably intended.","section":"Abstract"},{"comment":"References [7] and [32] appear to cite the same paper (Das et al., 'Exploring explainable AI methods for bird sound-based species recognition systems'); please consolidate or differentiate them.","section":"References"},{"comment":"The classification metrics in Table I are reported without confidence intervals or multiple-seed standard deviations; adding these would help assess significance of the small differences across background conditions.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a genuinely interesting application of XAI to bioacoustics, but the current evidence is not yet solid. The most serious issue is the lack of any control for recording provenance; if the data do not allow a group-wise split or provenance-based analysis, the central biological interpretation may have to be substantially weakened. The definitional nature of the max-ensemble performance metric and the small sample sizes for the XAI comparisons are also fixable with additional experiments and re-analysis. I recommend major revision rather than rejection because the core ideas are plausible and the manuscript could be made sound with targeted additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You can treat this as a fair, moderately careful applied XAI case study, not as a methodological advance. What is actually new is modest: the specific combination of Grad-CAM and DeepLIFT heatmaps for Bewick's wren dialect classification, plus the t-SNE observation that each dialect class contains subclusters. The paper does some things well. It compares four XAI methods on a real bioacoustics dataset, reports a background-color robustness check, and the qualitative observation that DeepLIFT gives cleaner explanations for bird-song experts is plausible. The authors also correctly note the LIME instability and do not oversell the PCA result.\n\nThe soft spots are real and I largely agree with the reader. The load-bearing risk is the missing provenance control. Section III-A says all recordings came from Arizona and New Mexico, but the paper never says whether clips from the same session, individual, or location were kept together when splitting into train/test. With a random split, the same bird or the same recording conditions can appear on both sides, which would inflate the 94.8% accuracy and make the saliency maps reflect site or recorder artifacts rather than song content. That concern applies equally to the t-SNE subclusters. The paper should report a group-wise split (e.g., by recording site or individual) or at least a provenance-based cluster analysis.\n\nTwo more moderate issues. First, the quantitative support for the element-wise max ensemble (Figure 6) is partly definitional: Hmax(i,j) ≥ both H_CAM and H_LDF by construction, so a larger fraction of pixels above any threshold follows automatically. The qualitative visual argument is more meaningful, but it rests on one or two samples per class. Second, the paper cites AudioProtoNet with a conference name and arxiv number that I could not verify; that needs checking. No code or data are provided, which limits reproducibility.\n\nThe paper is not a bad use of referee time. The writing is clear, the pipeline is standard, and the biological interpretation (terminal trills as discriminative features) matches what human observers use. But the provenance gap means the central claim is not yet established. A serious referee should ask for a site/individual-controlled split, more explanation samples, and ideally code/data release. That is heavy revision, but the paper is fixable and the topic is relevant. I would send it out; I just would not accept it in its current form.","headline":"A competent applied XAI case study on Bewick's wren dialects whose central biological interpretation is plausible but under-supported because recording-site provenance is not controlled in the train/test split.","tokens_in":10638,"tokens_out":1366,"would_cite":false,"duration_ms":15324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing Grad-CAM and DeepLIFT heatmaps reveals the song regions a CNN uses to separate Bewick's wren variants.","keywords":["Explainable AI","Bioacoustics","Convolutional neural networks","Spectrogram classification","Grad-CAM","DeepLIFT","Ensemble saliency maps","t-SNE clustering"],"falsifier":"Retrain the identical CNN on the same spectrograms with the song pixels masked out; if classification accuracy remains near 94.8% or the ensemble heatmaps still highlight non-song regions, the model is exploiting background cues rather than the vocal features the paper interprets.","tokens_in":9640,"feed_emoji":"🐦","tokens_out":6337,"duration_ms":52815,"temperature":0.7,"pith_summary":"This paper tries to establish that explainable-AI heatmaps, when combined, can show what a convolutional neural network actually listens to when it classifies bird song. Trained on spectrograms of Eastern and Mexican Bewick's wren songs, the CNN reaches 94.8% accuracy, and the paper argues that fusing Grad-CAM and DeepLIFT saliency maps preserves every region either method independently marks, giving a fuller picture of the model's decision than either alone. The authors further claim that t-SNE analysis of the CNN's learned features reveals distinct sub-populations within both song variants, so explanations that look inconsistent for a whole class become consistent within each cluster. If correct, this makes XAI a practical hypothesis-generating tool for bioacoustics rather than just a model-debugging aid.","feed_headline":"Fusing two heatmaps exposes the acoustic cues behind bird-song AI","feed_subtitle":"Combined Grad-CAM and DeepLIFT saliency maps highlight the terminal trills that tell Eastern and Mexican wrens apart.","key_machinery":"The load-bearing object is the ensemble saliency map built from Grad-CAM and DeepLIFT heatmaps. Grad-CAM weights the last convolutional layer's feature maps by pooled gradients, giving a coarse localization of the class-relevant image region; DeepLIFT back-propagates contributions relative to a reference input, here a plain white image, giving fine-grained relevance at the input pixels. The two normalized maps are fused by Eq. (1) weighted average and Eq. (2) element-wise maximum, with the max strategy doing the main work of preserving the strongest activated regions of either method. The second piece of machinery is t-SNE on the CNN's flattened penultimate-layer features, which reveals the sub-clusters that the subsequent per-cluster heatmaps are aligned with.","core_discovery":"The central claim is that no single XAI technique is sufficient for interpreting a CNN trained on spectrogram images, but a deliberately fused saliency map is. Grad-CAM supplies coarse class-discriminative localization from the last convolutional layer, while DeepLIFT supplies fine-grained, direction-sensitive attribution from a reference input; after normalizing both to $[0,1]$, the paper combines them either as a weighted average $H_c = 0.5H_{\\mathrm{CAM}} + 0.5H_{\\mathrm{LDF}}$ or as an element-wise maximum $H_{\\max}(i,j) = \\max(H_{\\mathrm{CAM}}(i,j), H_{\\mathrm{LDF}}(i,j))$. In the paper's reading, the maximum ensemble captures all key activation regions from either method and, by a threshold-based pixel count, activates more regions than either standalone heatmap. Model-agnostic LIME and SHAP explanations were less consistent, and DeepLIFT was the most interpretable to bird-song experts. A separate t-SNE latent-space analysis then shows multiple distinct clusters within each of the two song variants, with heatmaps consistent within each cluster, which the paper takes as evidence of biologically meaningful sub-populations rather than noisy classifications.","pith_inferences":["This is an inference: a natural next test would be to compare t-SNE cluster assignments against recording locality, date, or individual bird identity; the paper does not report such matching, so whether the sub-populations are geographic dialects, individual signatures, or recording-site artifacts remains open.","This is an inference: the threshold-pixel-count metric used to compare ensemble versus standalone heatmaps measures spatial coverage but not whether the added pixels are biologically informative; coverage should be validated against expert-labeled song regions or a removal test.","This is an inference: if the CNN is chiefly keying on terminal trills, the same fusion method could be used to quantify how much each spectro-temporal element contributes to dialect differences in other song-learning birds, generating comparative hypotheses the paper only gestures at."],"forward_implications":["A single XAI method can miss regions the model relies on; the element-wise maximum fusion is a cheap way to get a more complete saliency map without retraining.","DeepLIFT stands out as the most interpretable single method for experts reading spectrograms, so bioacoustic studies may adopt it as a default.","Cluster-specific explanations imply that a model's global explanations for a whole class can be misleading if the class contains acoustic sub-populations.","Terminal trills, highlighted across methods, are the acoustic features the CNN appears to use to distinguish Eastern from Mexican songs, matching human listening.","The same ensemble-XAI pipeline can be applied to other species or acoustic tasks without model changes."],"supporting_citations":[{"why":"Documents song variation and dialects in Bewick's wren, providing the biological motivation for the Eastern/Mexican distinction.","marker":"[9]"},{"why":"Supplies the spectrogram conversion tool used to turn audio recordings into CNN input images.","marker":"[10]"},{"why":"Provides the LIME local surrogate explanation technique that the paper compares against the model-specific methods.","marker":"[11]"},{"why":"Provides the SHAP attribution method used as the other model-agnostic comparison.","marker":"[12]"},{"why":"Supplies the Grad-CAM heatmap method that forms one component of the ensemble saliency maps.","marker":"[13]"},{"why":"Supplies the DeepLIFT attribution method that forms the other ensemble component and is described as the most interpretable.","marker":"[14]"},{"why":"Provides the t-SNE algorithm used to reveal sub-clusters within the two song variants.","marker":"[15]"},{"why":"Provides the PCA method used as a comparison and baseline dimensionality-reduction technique in the sub-population analysis.","marker":"[16]"}],"fun_headline_variants":["Combining two XAI maps reveals bird-song cues CNNs rely on","Fusing saliency maps beats single XAI for explaining bird-song CNN","Two XAI heatmaps combined reveal more bird-song cues than one","CNN bird-song classifier explained by fusing Grad-CAM and DeepLIFT","Combining Grad-CAM and DeepLIFT heatmaps explains wren song CNN"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The CNN separates Eastern from Mexican songs because of song content itself, not because recordings from the two groups carry different environmental backgrounds, so the heatmaps reveal biologically meaningful vocal features.","fun_headline_variants_meta":{"raw":{"variants":["Combining two XAI maps reveals bird-song cues CNNs rely on","Fusing saliency maps beats single XAI for explaining bird-song CNN","Two XAI heatmaps combined reveal more bird-song cues than one","CNN bird-song classifier explained by fusing Grad-CAM and DeepLIFT","Combining Grad-CAM and DeepLIFT heatmaps explains wren song CNN"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000686,"raw_usage":{"total_tokens":3136,"prompt_tokens":997,"completion_tokens":2139,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":2041}},"tokens_in":613,"tokens_out":2139,"duration_ms":14268,"temperature":1.0,"reasoning_tokens":2041,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:00:42.393331+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the identical CNN on the same spectrograms with the song pixels masked out; if classification accuracy remains near 94.8% or the ensemble heatmaps still highlight non-song regions, the model is exploiting background cues rather than the vocal features the paper interprets.","supporting_citations":[{"cited_title":"Song learning, dialects, and dispersal in the bewick’s wren,","cited_arxiv_id":null,"evidence_quote":"Documents song variation and dialects in Bewick's wren, providing the biological motivation for the Eastern/Mexican distinction."},{"cited_title":"Seewave, a free modular tool for sound analysis and synthesis,","cited_arxiv_id":null,"evidence_quote":"Supplies the spectrogram conversion tool used to turn audio recordings into CNN input images."},{"cited_title":"Why should i trust you? explaining the predictions of any classifier,","cited_arxiv_id":null,"evidence_quote":"Provides the LIME local surrogate explanation technique that the paper compares against the model-specific methods."},{"cited_title":"A unified approach to interpreting model predictions,","cited_arxiv_id":null,"evidence_quote":"Provides the SHAP attribution method used as the other model-agnostic comparison."},{"cited_title":"Grad-cam: Visual explanations from deep networks via gradient-based localization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Grad-CAM heatmap method that forms one component of the ensemble saliency maps."}],"review_version":2}