{"id":"b857c8e7-4dc0-4276-aff5-5ff67e8bb7f3","arxiv_id":"2412.05951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A look-aside adapter with 1D time and frequency convolutions lets a frozen vision transformer match or beat audio-pretrained models on audio classification benchmarks.","lead":"This paper introduces LoAA, a small adapter module that lets a frozen ImageNet-pretrained vision transformer handle audio tasks without large-scale audio pretraining. On EPIC-SOUNDS it slightly beats an audio-pretrained model while updating only 2% of parameters.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 'surpass' rests on a 0.77-point single-run EPIC-SOUNDS gap with no error bars; on ESC-50 and SPC-2 LoAA is below audio-pretrained baselines, so the central claim is not robustly established.","rationale":"I read the central claim as an empirical equivalence statement: an ImageNet-only model with LoAA beats an AudioSet+LibriSpeech-pretrained SSAST on EPIC-SOUNDS, motivating the bypass of large-scale audio pretraining. The evidence for that statement is a single run of one configuration, with a small margin, no reported seeds or confidence intervals, and baselines taken from prior work. This makes the headline claim underpowered. I considered the reader's inductive-bias concern about time/frequency kernel orientation; it matters for explaining why LoAA works, but it is not the most load-bearing risk, because even a generic convolutional adapter would, if reproduced, still support the practical claim that audio pretraining can be bypassed. Conversely, if the 0.77-point margin collapses under repeated seeds or validation-based selection, the headline claim fails regardless of mechanism. I also note that Section 3.3's 'less than 1% discrepancy' wording is difficult to reconcile with the 2%-parameter rows on ESC-50 and SPC-2 (85.4 vs 88.8 and 96.5 vs 98.0), further weakening the 'various tasks' phrasing. The conditional verdict remains appropriate.","tokens_in":7589,"tokens_out":10719,"duration_ms":103592,"concrete_test":"Rerun the best LoAA configuration (attention-block, time 1x3 kernel, r=36, 2% parameters) and the two key baselines, SSAST full fine-tuning and the 1x1 parallel adapter, on EPIC-SOUNDS using at least 5 random seeds, with hyperparameters selected on a held-out validation split rather than the test set. Report mean and 95% CI for top-1 accuracy and mAP. If the LoAA mean is not above SSAST's mean by more than the combined CI width, or if the validation-selected configuration is not the 2% Attn(T) variant, the 'surpass pretrained audio models' claim is not established. Also report ESC-50 and SPC-2 with the same seed protocol to substantiate the 'various tasks' claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 2's central comparison is LoAA-Attn(T) at 2% parameters achieving 54.52 top-1 on EPIC-SOUNDS versus 53.75 for SSAST, which was pretrained on AudioSet-2M plus LibriSpeech. The entire 'reach or surpass' claim therefore rests on a 0.77-point margin from a single reported run. The paper reports no error bars or number of seeds, and Table 1 shows comparable-magnitude swings from architectural variation: at 1.035M params the 1x1 versus 1x3 kernel difference is 0.05 (53.38 vs 53.43), while at 2.026M params the same toggling changes accuracy by 1.19 (53.33 vs 54.52), exactly the spread one would expect from run-to-run noise. The paper does not state that the highlighted 2% Attn(T) configuration was chosen on a held-out validation split; absent that, selecting the best cell from Tables 1 and 3 inflates the apparent gap. Moreover, the 'various audio and speech tasks' wording is not supported: on ESC-50 and SPC-2 the 2%-parameter LoAA rows (85.4 and 96.5) are clearly below SSAST (88.8 and 98.0). This is not an internal inconsistency in the mathematics, but the central empirical claim is underpowered as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Look-Aside Adapter (LoAA), a parameter-efficient fine-tuning module for adapting ImageNet-pretrained vision transformers to audio and speech tasks without a separate large-scale audio pretraining stage. LoAA replaces the linear projections of a parallel adapter with 1x3 and 3x1 convolutions over the time and frequency axes of the spectrogram-token grid, thereby introducing inter-token interactions across those heterogeneous dimensions while the backbone remains frozen. The method is evaluated on EPIC-SOUNDS, ESC-50, and Speech Commands V2 and compared with parallel adapters, full fine-tuning of ImageNet-pretrained AST, and audio-pretrained models SSAST and MAE-AST. The central claim is that a vision model with roughly 2% trainable parameters can match or surpass audio-pretrained SSAST, with the strongest result being 54.52% top-1 accuracy on EPIC-SOUNDS versus SSAST's 53.75%.","tokens_in":7886,"tokens_out":5849,"duration_ms":56845,"significance":"If the headline result were robust, the paper would make a useful practical point: for several audio benchmarks, a cheap single-stage PEFT route from an ImageNet model could rival the expensive two-stage paradigm of AudioSet/LibriSpeech pretraining followed by full fine-tuning. The proposed modification is simple, parameter-efficient, and the paper reports consistently positive comparisons against a parallel-adapter baseline within its own experimental setup, which is a genuine strength. The main limitations are empirical rigor: no error bars or seed counts, test-set-based configuration selection, externally sourced baselines, and an abstract that overstates the breadth of the results. With additional experiments and more careful claims, the contribution could be a solid empirical paper; as it stands, the evidence is not yet load-bearing for the stated conclusions.","major_comments":[{"comment":"The abstract's claim that LoAA allows vision models to \"reach or surpass the performance of pretrained audio models in various audio and speech tasks\" is not supported by Table 2. On ESC-50, Ours-Attn(T) at 2% parameters reaches 85.4 versus SSAST's 88.8, a 3.4-point gap, and on SPC-2 it reaches 96.5 versus 98.0, a 1.5-point gap. Even on the flagship EPIC-SOUNDS dataset, the 2% model's mAP is 0.234, below SSAST's 0.237, so the \"surpass\" result depends on choosing top-1 accuracy rather than mAP. The statement in §3.3 that all results are \"nearly equivalent... with a performance discrepancy of less than 1%\" is also incorrect for ESC-50 and SPC-2. The claims should be narrowed and quantified, or supported with appropriate statistical evidence.","section":"Abstract and §3.3, Table 2"},{"comment":"No experiment is reported with more than one seed, and no standard deviations or confidence intervals are given. The headline gap of 54.52 versus 53.75 (0.77 points) is smaller than the variation caused by architectural choices within Table 1: at 2.026M parameters, the 1x1 kernel gives 53.33 while the T(1,3) kernel gives 54.52, a 1.19-point swing; the FFN row shows a 1.15-point swing. This means the claimed advantage over SSAST is within the range of run-to-run or design-choice noise as currently reported. The authors should report mean and standard deviation over at least five seeds, and ideally a paired significance test, before asserting superiority.","section":"Tables 1-3"},{"comment":"The paper does not clarify whether the best configuration, Ours-Attn(T) at 2% parameters, was selected on a held-out validation split. The implementation details in §3.2 state only that the PEFT learning rate was chosen from {5e-5, 1e-4, 5e-4}, and Tables 1 and 3 enumerate many kernel, placement, and combination choices on EPIC-SOUNDS. If the highlighted cell is the best among many evaluated on the test set, the reported 0.77-point margin is inflated by selection bias. The authors must state the validation protocol or, if none was used, treat the current numbers as exploratory and re-evaluate a pre-registered or validation-selected configuration.","section":"§3.2, Tables 1 and 3"},{"comment":"The central novelty claim is that 1x3 (time) and 3x1 (frequency) kernels provide the right inductive bias for audio spectrograms. The ablations in Table 1 do not consistently support the orientation-specific part of this claim. At 1.035M parameters, F(3,1) is better than T(1,3) in both the attention row (53.79 vs 53.43) and the FFN row (53.53 vs 53.42); at 2.026M parameters, T(1,3) is better than F(3,1) by 0.08 in the attention row and by 0.77 in the FFN row. While T/F kernels are generally better than 1x1 and 3x3, the particular time-versus-frequency orientation is unstable across parameter budgets. The paper also does not vary kernel length (e.g., 1x5, 1x7) or patch size, so the results could be explained by a generic convolutional bypass effect rather than by the specific audio-adjacency assumption. Additional ablations and a consistency analysis are needed.","section":"§2.3, Table 1"},{"comment":"The audio-pretrained baselines SSAST and MAE-AST are not re-run under the same protocol; their numbers are taken directly from other papers. This matters because the PEFT experiments use non-overlapping patches while the paper states that full fine-tuning uses patch overlapping, and learning rates, optimizers, and training schedules also differ. A direct comparison therefore conflates the PEFT method with these protocol differences. The authors should either re-run the audio-pretrained baselines under an identical evaluation protocol or clearly show that the protocol differences do not affect the relative ordering.","section":"§3.3, Table 2 and §3.2"}],"minor_comments":[{"comment":"The text says PEFT updates \"less than 10% of total parameters,\" but Table 2 includes a 10% configuration; the wording should be \"up to 10%\" for consistency.","section":"§2.1"},{"comment":"The column headers label ESC-50 and SPC-2 results as \"mAP,\" but the reported values appear to be top-1 accuracy (e.g., 88.8 for SSAST on ESC-50 matches its published accuracy). The columns should be renamed to avoid misleading readers.","section":"Table 2"},{"comment":"The reshaping of flattened tokens into a two-dimensional time-frequency grid is described only in the implementation details. Defining the tensor layout and the exact convolution operation in Section 2.3 would make the architecture easier to reproduce.","section":"Eq. (2) and Figure 2"},{"comment":"The attention-map visualization does not state whether attention is averaged over heads, which query token is used, or how the colormap is normalized. Adding these details would make the qualitative claim about \"smoother\" attention more interpretable.","section":"Figure 3"},{"comment":"In §3.3, MAE-AST is described as \"based on AST,\" but the cited paper [21] does not use that name in its title; the description should be made consistent with the reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a plausible core idea and a consistent improvement over its own parallel-adapter baseline, but the headline comparisons to audio-pretrained models are underpowered and partly selection-biased. I recommend major revision rather than rejection because the issues are fixable with more seeds, a validation protocol, and corrected claims. The abstract and §3.3 currently overstate the results; after revision, the contribution would be a modest but useful empirical study of PEFT for audio with frozen vision backbones."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the useful contribution is LoAA as a small twist on parallel adapters: replace the linear down/up projections with 1D convolutions oriented along time or frequency, using the 2D spectrogram grid. It is close to ConvPass but not identical, and the paper explicitly targets the time/frequency asymmetry of audio. The ablation of adapter placement (Attn vs FFN), kernel orientation, and bottleneck size is the strongest part. The consistent, if modest, improvement over a matched parallel adapter across configurations is believable and useful for low-resource audio classification.\n\nThe soft spots are real and mostly about the headline claim. “Reach or surpass pretrained audio models” rests on a 0.77-point top-1 gap over SSAST on EPIC-SOUNDS, from a single run with no error bars or seeds. Table 1 shows kernel-orientation toggles moving accuracy by over a point at the same parameter count, which is the noise scale you would expect without repeats. The best configuration also appears to be selected on the target test set—Table 3 is EPIC-SOUNDS, not a held-out validation split—so the highlighted number is likely optimistic. On ESC-50 and SPC-2, the 2% LoAA rows are below SSAST, which undercuts the “various audio and speech tasks” phrasing in the abstract, even though the body later says “comparable.\n\nThe inductive-bias story—that time/frequency adjacency is what helps—is plausible but not tested. With only 1x3, 3x1, and 3x3 kernels, and no variation in patch size or kernel length, you cannot tell whether the gain comes from oriented adjacency or simply from adding convolution anywhere. That is a fixable gap, not a fatal one.\n\nWho is this for? People working on audio PEFT or efficient transfer from vision encoders. It is a modest, honest incremental contribution, not a paradigm shift. It deserves a serious referee: the method is simple, the comparisons are mostly fair on the points they control, and the limitations are the standard ones—seeds, validation-based selection, rerunning baselines. I would send it to review with a request for those fixes.\n\nRecommendation: engage with it, but treat the headline claim as unproven until error bars and held-out model selection are added.","headline":"A sensible adapter variant for audio PEFT with a systematic ablation, but the 'surpass pretrained audio models' headline is built on a single-run 0.77-point gap with no error bars.","tokens_in":8438,"tokens_out":1753,"would_cite":true,"duration_ms":17955,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"2% adapters let vision models beat audio pretraining","keywords":["audio classification","parameter-efficient fine-tuning","look-aside adapter","vision transformer","transfer learning","mel-spectrogram","time-frequency convolutions","audio pretraining bypass"],"falsifier":"Train the same LoAA setup on EPIC-SOUNDS with the time kernel applied along the frequency axis and the frequency kernel applied along the time axis, holding parameter count fixed. If the swapped configuration matches the original accuracy (especially if it still beats the 1x1 linear adapter), the claimed directional inductive bias is not what drives the gain; if the swap drops accuracy back to or below the linear-adapter baseline, the orientation is load-bearing.","tokens_in":7371,"feed_emoji":"🎧","tokens_out":5032,"duration_ms":42127,"temperature":0.7,"pith_summary":"This paper proposes Look-Aside Adapters (LoAA), a parameter-efficient fine-tuning method that lets an ImageNet-pretrained vision transformer process audio spectrograms without any large-scale audio pretraining. The adapters replace the linear projections in parallel adapters with one-dimensional convolutions oriented along the time axis or the frequency axis, so that tokens can exchange information across those two dimensions while the backbone attention remains frozen. On EPIC-SOUNDS, the method reaches 54.52% top-1 accuracy with 2% trainable parameters, surpassing SSAST (53.75%) which was pretrained on AudioSet and LibriSpeech. On ESC-50 and Speech Commands V2 it stays within about one percentage point of audio-pretrained models. If correct, the result means the expensive two-stage paradigm of audio pretraining followed by fine-tuning can be bypassed for several audio and speech tasks.","feed_headline":"2% adapters let vision models beat audio pretraining","feed_subtitle":"A time/frequency look-aside adapter matches or beats an AudioSet-pretrained SSAST using only ImageNet weights.","key_machinery":"The central object is the Look-Aside Adapter (LoAA), a parallel adapter whose down- and up-projection matrices are replaced by 1D convolutions with kernel size Nx1 or 1xN, operating on tokens reshaped into a time-frequency grid. The time kernel (1x3) and frequency kernel (3x1) promote interactions between tokens along the respective spectrogram axis; the output is added back to the transformer block output in the same residual style as a Houlsby adapter. The bottleneck dimension r is adjusted to keep parameter counts equal across kernel shapes, which is what makes the comparison between linear (1x1) and directional kernels controlled.","core_discovery":"The central claim is that the gap between vision-pretrained and audio-pretrained models on audio tasks can be closed by adapters whose internal projections are directional convolutions instead of linear layers, without touching the frozen transformer weights. By reshaping the token sequence into a two-dimensional time-frequency grid inside the adapter, a 1x3 convolution only mixes tokens that are adjacent in time, and a 3x1 convolution only mixes tokens adjacent in frequency. The paper shows this simple change outperforms the 1x1 linear parallel adapter at equal parameter counts, and that an isotropic 3x3 kernel does worse, supporting the claim that the directionality itself, not the mere presence of convolution, carries the benefit. On EPIC-SOUNDS the best single placement (attention block, time kernel, 2% parameters) reaches 54.52% top-1 accuracy, while the audio-pretrained SSAST baseline scores 53.75%, and the combined time/frequency placement at 5% reaches a mAP of 0.238, matching SSAST's 0.237.","pith_inferences":["A direct testable extension would be to swap the kernel orientations (time kernel along frequency, frequency kernel along time) and check whether EPIC-SOUNDS accuracy collapses; if it does not, the directional inductive bias is not the active ingredient.","The same blockwise anisotropic adapter idea could transfer to other inputs with heterogeneous axes, such as video (spatial versus temporal) or sequences with a known layout, where frozen pretrained attention may need a similar reorientation.","Because the method leaves the backbone frozen, it is likely to compose with other PEFT techniques, such as low-rank updates or prompt tuning; the paper does not test those combinations.","The results do not yet address whether the adapter would close the gap on harder audio tasks or larger backbones; the paper's smallest model comparison leaves open whether the benefit scales with backbone size."],"forward_implications":["ImageNet-pretrained vision transformers with LoAA can match or beat SSAST, which was pretrained on AudioSet-2M and LibriSpeech, on EPIC-SOUNDS using only 2% trainable parameters.","The one-stage PEFT recipe removes the need for large-scale audio pretraining and its accompanying data collection and compute for EPIC-SOUNDS, ESC-50, and Speech Commands V2.","A time-oriented adapter on the attention block combined with a frequency-oriented adapter on the FFN block is the best 5%-parameter configuration, outperforming all same-size parallel-adapter variants.","The per-axis design outperforms an isotropic 3x3 convolution at equal parameters, indicating that respecting the heterogeneity of time and frequency is what helps.","Performance on speech tasks (SPC-2) and environmental sounds (ESC-50) stays within about 1% of audio-pretrained models, so the approach does not obviously sacrifice generality on the tested benchmarks."],"supporting_citations":[{"why":"Provides the base ImageNet-pretrained vision transformer (AST) and the spectrogram tokenization used in all experiments.","marker":"[8]"},{"why":"The audio-pretrained SSAST baseline that LoAA matches or surpasses on EPIC-SOUNDS.","marker":"[10]"},{"why":"The parallel Houlsby adapter architecture that LoAA modifies by replacing linear projections with directional convolutions.","marker":"[16]"},{"why":"Prior convolutional-bypass adapters for vision transformers, motivating the use of convolutions inside adapters.","marker":"[12]"},{"why":"EPIC-SOUNDS is the main benchmark dataset where LoAA outperforms audio-pretrained SSAST.","marker":"[5]"},{"why":"MAE-AST is a second large-scale audio-pretrained baseline reported in the comparison table.","marker":"[21]"},{"why":"AudioSet is the large-scale audio pretraining data used by baselines, whose absence is the paper's efficiency point.","marker":"[4]"}],"fun_headline_variants":["2% params: directional convolutions give vision models audio edge","Vision models beat audio pretraining with 2% adapters","No audio pretraining needed: directional adapters close gap","Time/frequency adapters let ImageNet models outdo audio pretraining","2% parameters: time/frequency adapters beat audio pretraining"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the premise that a 1x3 convolution over time-neighboring tokens and a 3x1 convolution over frequency-neighboring tokens supplies the right inductive bias for audio spectrograms when the transformer attention is frozen.","fun_headline_variants_meta":{"raw":{"variants":["2% params: directional convolutions give vision models audio edge","Vision models beat audio pretraining with 2% adapters","No audio pretraining needed: directional adapters close gap","Time/frequency adapters let ImageNet models outdo audio pretraining","2% parameters: time/frequency adapters beat audio pretraining"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001397,"raw_usage":{"total_tokens":5632,"prompt_tokens":908,"completion_tokens":4724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":4636}},"tokens_in":524,"tokens_out":4724,"duration_ms":33039,"temperature":1.0,"reasoning_tokens":4636,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:09:42.112339+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same LoAA setup on EPIC-SOUNDS with the time kernel applied along the frequency axis and the frequency kernel applied along the time axis, holding parameter count fixed. If the swapped configuration matches the original accuracy (especially if it still beats the 1x1 linear adapter), the claimed directional inductive bias is not what drives the gain; if the swap drops accuracy back to or below the linear-adapter baseline, the orientation is load-bearing.","supporting_citations":[{"cited_title":"Exploring simple siamese representation learning,","cited_arxiv_id":null,"evidence_quote":"Provides the base ImageNet-pretrained vision transformer (AST) and the spectrogram tokenization used in all experiments."},{"cited_title":"Epic- sounds: A large-scale dataset of actions that sound,","cited_arxiv_id":null,"evidence_quote":"The audio-pretrained SSAST baseline that LoAA matches or surpasses on EPIC-SOUNDS."},{"cited_title":"Masked autoencoders that lis- ten,","cited_arxiv_id":null,"evidence_quote":"The parallel Houlsby adapter architecture that LoAA modifies by replacing linear projections with directional convolutions."},{"cited_title":"However, it is challenging due to the need of substantial audio data and a well-designed learning objective for large-scale audio pretraining","cited_arxiv_id":null,"evidence_quote":"EPIC-SOUNDS is the main benchmark dataset where LoAA outperforms audio-pretrained SSAST."},{"cited_title":"Parameter-efficient transfer learning for NLP,","cited_arxiv_id":null,"evidence_quote":"MAE-AST is a second large-scale audio-pretrained baseline reported in the comparison table."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AudioSet is the large-scale audio pretraining data used by baselines, whose absence is the paper's efficiency point."}],"review_version":1}