{"id":"8464c12b-3f73-4b69-a8e0-b2e28160293b","arxiv_id":"2509.03292","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A BEATs-plus-LSTM system with triplet loss predicts four perceptual aesthetic scores for synthetic audio while trained only on natural audio.","lead":"Researchers built an automatic system that judges four qualities of AI-generated audio: production quality, complexity, enjoyment, and usefulness. It uses a pre-trained audio model, a memory of recent examples, and a triplet loss that groups similar-sounding clips, and tests it on synthetic speech, music, and sound effects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Triplet-loss benefit is untested (no ablation) and the reported MSE values are internally inconsistent between Tables I and II, so the paper's central causal claim lacks reliable evidence.","rationale":"I agree with the reader's REJECT. The central claim is causal and requires an ablation; its absence is a fatal gap in a scientific argument. The numerical inconsistencies make it impossible to trust the reported results, which further weakens the claim. The reader's weakest_assumption about cross-domain transfer is related but secondary: even if transfer works, the paper does not demonstrate that the triplet loss is the reason. Thus I do not need to manufacture a new concern; the paper itself provides the disconfirming evidence. The verdict should remain REJECT.","tokens_in":5745,"tokens_out":4139,"duration_ms":38476,"concrete_test":"Run the identical training pipeline with the triplet loss disabled (α=0 in Eq. 3), all other hyperparameters/random seeds fixed, and recompute system-level SRCC/KTAU on the AudioMOS 2025 evaluation set; if rank metrics do not consistently improve with triplet loss, the central claim fails. Also recompute aggregated CE/CU MSE from Table I domain rows (or rerun the official scoring script on saved predictions) to determine which table is correct.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The conclusion in Section V states that \"the inclusion of triplet loss enhances both rank-based and correlation-based performance metrics.\" No experiment supports this causal attribution. The only comparison in Table II is to an official baseline [22] that differs in architecture, feature extractor, training objective, and evaluation protocol; removing or adding triplet loss was never tested. Without an α=0 ablation of Eq. (3), any performance difference could be due to any other design choice. Moreover, the empirical support is self-contradictory: Table I reports domain-level MSE for CE of 1.4739 (speech), 2.3145 (music), and 1.8942 (audio), but Table II reports \"Ours\" CE MSE = 3.991. For CU, Table I reports 0.9539, 1.1404, and 1.9407, while Table II reports 0.533. These cannot both describe the same evaluation set. The prose also claims \"higher rank-based metrics (SRCC and KTAU) across all four axes,\" but Table II shows PC SRCC 0.928 vs baseline 0.934 and PC KTAU 0.781 vs 0.800—the opposite. Since the numbers that are supposed to demonstrate the triplet loss benefit are internally inconsistent, the central claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AESA-Net, a multi-axis perceptual audio aesthetic assessment system for AudioMOS Challenge 2025 Track 2. The system uses BEATs self-supervised features, a multi-branch BLSTM/attention architecture, and a per-axis triplet loss with buffer-based sampling. It is trained only on AES-natural audio and evaluated on the official synthetic evaluation set. The authors report system-level MSE, LCC, SRCC, and KTAU across PQ, PC, CE, and CU, and compare against the official AudioMOS baseline. The central claim, stated in the conclusion, is that including the triplet loss improves both rank-based and correlation-based performance metrics.","tokens_in":6091,"tokens_out":3526,"duration_ms":37407,"significance":"If the central claim were established, the work would be a useful contribution to cross-domain audio quality assessment: it targets a practical domain-shift problem (natural training data, synthetic evaluation data) and tackles four aesthetic axes jointly. The use of an external challenge evaluation set is a positive feature, and the architecture is clearly described. However, the paper does not provide a controlled experiment isolating the triplet loss, and the reported results are internally inconsistent. As it stands, the significance is limited because the headline contribution is not evidenced by the experiments presented.","major_comments":[{"comment":"The conclusion states that 'the inclusion of triplet loss enhances both rank-based and correlation-based performance metrics,' but no experiment supports this causal attribution. The only comparison is against the official baseline [22], which differs in architecture, feature extractor, training objective, and possibly evaluation protocol. An ablation with α=0 in Eq. (3) (or at least a sweep over α) is necessary to separate the triplet-loss contribution from all other design choices. Without it, the central claim is unsupported.","section":"Section V, Eq. (3), Table II"},{"comment":"The reported MSE values are internally inconsistent. Table I reports domain-level MSE for CE of 1.4739 (speech), 2.3145 (music), and 1.8942 (audio), while Table II reports 'Ours' CE MSE = 3.991. For CU, Table I reports 0.9539, 1.1404, and 1.9407, while Table II reports 0.533. If Table I is per-domain and Table II is an aggregate, the aggregation formula is missing and the resulting values appear incompatible with any simple average. If both are supposed to be system-level on the same evaluation set, at least one of the tables is wrong. This needs to be resolved before any reliable comparison can be made.","section":"Tables I and II"},{"comment":"The prose after Table II states that 'our model achieves higher rank-based metrics (SRCC and KTAU) across all four perceptual axes.' This is directly contradicted by the PC row: SRCC is 0.928 vs baseline 0.934, and KTAU is 0.781 vs baseline 0.800. Either the prose or the table must be corrected. This is not a cosmetic issue; it affects the claimed superiority of the proposed approach.","section":"Section IV.C, Table II"},{"comment":"The paper assumes that score-similarity structure learned on natural-audio embeddings transfers to synthetic audio. The triplet loss is trained on AES-natural labels, yet the evaluation is entirely on synthetic TTS/TTA/TTM clips. No synthetic validation set is used during development, and no analysis is provided showing that the triplet-structured embedding separates synthetic clips by perceived quality. A concrete test would be to evaluate embedding nearest-neighbor behavior or triplet accuracy on a held-out synthetic set. Without such evidence, the mechanism claimed for domain robustness remains unverified.","section":"Sections II.A and III"}],"minor_comments":[{"comment":"The caption says 'system-level results' but the table is organized by domain. Please clarify whether 'system-level' refers to individual generative systems, domain-level averages, or the entire evaluation set, and define the aggregation in Section IV.B.","section":"Table I caption"},{"comment":"The evaluation metrics list uses both 'PCC' and later 'LCC' for the same metric. Please standardize the terminology.","section":"Section IV.B"},{"comment":"The buffer-based sampling is attributed to references [20] and [21], but [20] is the FaceNet triplet-loss paper and [21] is about non-parametric instance discrimination. The citation-to-method mapping should be checked and corrected.","section":"Section III"},{"comment":"The abstract claims that the method 'improves embedding discriminability,' but no experiment or visualization directly measures embedding discriminability. This claim should either be supported or rephrased as an interpretation.","section":"Abstract / Section IV.C"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads like a challenge system description, but the central scientific claim is currently not supported by the evidence. The missing triplet-loss ablation and the Table I/II MSE inconsistency are both fixable, which is why I recommend major revision rather than reject. However, if the authors cannot produce the α=0 ablation or reconcile the numbers, the paper would not be acceptable even after revision. The PC-row contradiction in the prose should also be caught during revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a standard AudioMOS 2025 challenge write-up: BEATs features, a multi-branch LSTM predictor, and a buffer-based triplet loss, evaluated on the official synthetic test set. What's genuinely new is narrow—measured numbers for one specific system on that track, including a comparison to the official baseline. The architecture is reasonable and the idea of structuring the embedding space by score similarity is a sensible thing to try. The results for PQ and CU actually look pretty good. So it's not fluff; it's a legitimate system description.\n\nBut the paper has two problems that are big enough to sink the central claim.\n\nFirst, there is no ablation. The conclusion says the triplet loss 'enhances both rank-based and correlation-based performance metrics,' but the only comparison is to an official baseline that differs in architecture, feature extractor, training objective, and evaluation protocol. Nothing in the experiments isolates the triplet loss. Removing or changing α in Eq. (3) was never tried. As written, the triplet loss could be doing nothing at all.\n\nSecond, the numbers are internally inconsistent. Table I reports domain-level MSE for CE as roughly 1.5 to 2.3 across speech, music, and audio, and for CU as roughly 0.95 to 1.94. Table II lists 'Ours' CE MSE as 3.991 and CU MSE as 0.533. Those can't both describe the same system on the same evaluation set. The prose after Table II also says the model achieves higher SRCC and KTAU across all four axes, but the table shows PC SRCC and KTAU are lower than baseline. These aren't typos you can ignore; they're contradictions in the evidence that supposedly demonstrates the triplet-loss benefit.\n\nThe stress-test note about all this is accurate. The reader's 'weakest assumption' about L2-distance transfer is plausible but secondary; the immediate problem is that the reported results are unreliable.\n\nTo be fair, the correlations themselves might be real, and a per-domain breakdown explanation would help. But the paper doesn't provide one. Hyperparameters like the triplet margin, epsilon, buffer capacity, and α are also incompletely specified, so reproduction is hard.\n\nWho benefits from reading it? Challenge participants who want to see one competitive entry, maybe. But as a scientific claim about triplet-loss generalization, it fails. I would not cite it, and I would not bring it to a reading group. A serious editor should desk reject this version; if the authors fix the tables, add an ablation with α=0, and reconcile the prose, it could become a decent short paper. As it stands, it's not ready for referee time.","headline":"A plausible challenge system whose central triplet-loss claim is untested and whose reported MSEs are internally inconsistent; the empirical case doesn't hold together.","tokens_in":6533,"tokens_out":2686,"would_cite":false,"duration_ms":28098,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AESA-Net predicts four aesthetic axes of synthetic audio from natural-audio training alone, using triplet-structured BEATs embeddings.","keywords":["audio quality assessment","aesthetic score prediction","BEATs","triplet loss","self-supervised audio representation","domain shift","text-to-speech evaluation","AudioMOS Challenge"],"falsifier":"Running the same training pipeline with triplet loss removed (and with all else fixed) and comparing SRCC/KTAU on the official synthetic evaluation set would settle the central attribution; a second check is to measure whether L2 distances between synthetic clips and natural anchors ordered by score separate synthetic clips by human-rated quality.","tokens_in":5687,"feed_emoji":"🎧","tokens_out":5619,"duration_ms":58845,"temperature":0.7,"pith_summary":"This paper tries to establish that automatic prediction of four perceived-quality axes for synthetic audio—production quality, production complexity, content enjoyment, and content usefulness—can succeed without ever training on synthetic audio. The proposed AESA-Net combines BEATs self-supervised audio representations with a multi-branch LSTM predictor, and adds a triplet loss that organizes the embedding space so clips with similar human scores sit close together. On the official AudioMOS 2025 synthetic evaluation set, the system improves rank correlation (SRCC, KTAU) over the official baseline on all four axes, and linear correlation on most axes, while MSE worsens only for content enjoyment. If the claim holds, aesthetic evaluation of TTS, TTA, and TTM audio becomes feasible as a no-reference, natural-only-trained metric, which matters because collecting human aesthetic ratings for every new generative system is expensive.","feed_headline":"Triplet loss lifts synthetic-audio quality ranking from natural data only","feed_subtitle":"AESA-Net scores TTS, TTA, and TTM audio on four perceptual axes and beats the official baseline on most rank metrics.","key_machinery":"The load-bearing mechanism is the triplet loss with buffer-based sampling, applied to intermediate embeddings. Given an anchor embedding, a positive embedding with similar normalized aesthetic score (|ya − yp| < epsilon) is pulled closer while a negative embedding with dissimilar score (|ya − yn| > epsilon) is pushed apart, subject to a margin; the total loss is LMSE + alpha times LTriplet. The FIFO buffer supplies recent embedding-score pairs so positives and negatives exist within a batch of size one. This structuring is what the paper credits for making a BEATs/LSTM predictor trained only on natural audio rank synthetic audio correctly.","core_discovery":"The central claim is that structuring a self-supervised audio embedding with triplet loss makes multi-axis aesthetic scoring transfer from natural to synthetic audio. AESA-Net extracts BEATs hidden states, combines them through a learnable softmax-weighted sum, encodes with a two-layer BLSTM, and branches into four attention-based scoring heads for Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness. Training minimizes MSE plus a triplet loss whose positives and negatives are drawn from a FIFO buffer of recent embedding-score pairs: a positive lies within epsilon of the anchor's normalized score, a negative lies farther than epsilon. Trained on 2,950 natural","pith_inferences":["Editorial inference: because no ablation removes the triplet loss, the metric gains could partly come from the BEATs backbone or the multi-task heads; a triplet-loss-off run is needed to isolate the mechanism the paper credits.","Editorial inference: the same score-thresholded buffer sampling could be transferred to other subjective ratings (e.g., listener MOS, intelligibility) or other modalities, since the only requirement is a paired embedding-score buffer.","Editorial inference: a small synthetic labeled set, even a few dozen clips per domain, would make the natural-to-synthetic transfer testable in development and could reveal whether larger margins or domain-weighted sampling close the CE MSE gap."],"forward_implications":["If the central claim holds, subjective aesthetic evaluation of TTS, TTA, and TTM audio can be automated without collecting synthetic-audio labels, reducing reliance on expensive listening tests.","Across all four axes, rank-based agreement (SRCC, KTAU) with human raters improves, so the model can order generated audio by perceived quality even where absolute-score calibration is imperfect.","The joint multi-axis design means one BEATs/LSTM backbone supplies four perceptual scores, which can be used as a training signal or filter in generative-audio systems.","Because BEATs is pretrained on general audio events, the same pipeline is reusable for speech, music, and general-audio quality monitoring without per-domain feature engineering."],"supporting_citations":[{"why":"Supplies BEATs, the pretrained transformer feature extractor whose layer-wise hidden states are weighted and fed to the predictor.","marker":"[16]"},{"why":"Inspires the multi-branch architecture (adapter, BLSTM, shared linear layer, task-specific heads) used as AESA-Net's backbone.","marker":"[19]"},{"why":"Supplies the triplet-loss formulation and buffer-based sampling that pulls together score-similar embeddings and pushes apart score-dissimilar ones.","marker":"[20]"},{"why":"Supplies the non-parametric instance-discrimination sampling strategy adapted for choosing positives and negatives from the FIFO buffer.","marker":"[21]"},{"why":"Provides the official challenge baseline whose system-level metric scores AESA-Net is compared against.","marker":"[22]"}],"fun_headline_variants":["Triplet loss lets natural-trained audio scoring tackle synthetic TTS, TTA, TTM","No synthetic audio needed: triplet loss aligns scores for generated clips","AESA-Net: BEATs + triplet loss pushes audio aesthetics across domains","Triplet loss makes robust audio quality scores without synthetic training","Domain-shift fix: triplet-structured BEATs embeddings score synthetic audio"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The system is trained only on natural audio, yet its embedding structure is assumed to organize synthetic audio by perceived quality; if score-similarity in BEATs embeddings does not carry across the natural-to-synthetic domain gap, the reported gains would not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Triplet loss lets natural-trained audio scoring tackle synthetic TTS, TTA, TTM","No synthetic audio needed: triplet loss aligns scores for generated clips","AESA-Net: BEATs + triplet loss pushes audio aesthetics across domains","Triplet loss makes robust audio quality scores without synthetic training","Domain-shift fix: triplet-structured BEATs embeddings score synthetic audio"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000789,"raw_usage":{"total_tokens":3283,"prompt_tokens":681,"completion_tokens":2602,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":425,"completion_tokens_details":{"reasoning_tokens":2515}},"tokens_in":425,"tokens_out":2602,"duration_ms":22075,"temperature":1.0,"reasoning_tokens":2515,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T11:00:23.603747+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Running the same training pipeline with triplet loss removed (and with all else fixed) and comparing SRCC/KTAU on the official synthetic evaluation set would settle the central attribution; a second check is to measure whether L2 distances between synthetic clips and natural anchors ordered by score separate synthetic clips by human-rated quality.","supporting_citations":[{"cited_title":"BEATs: Audio pre-training with acoustic tokenizers,","cited_arxiv_id":null,"evidence_quote":"Supplies BEATs, the pretrained transformer feature extractor whose layer-wise hidden states are weighted and fed to the predictor."},{"cited_title":"HAAQI- Net: A non-intrusive neural music audio quality assessment model for hearing aids,","cited_arxiv_id":null,"evidence_quote":"Inspires the multi-branch architecture (adapter, BLSTM, shared linear layer, task-specific heads) used as AESA-Net's backbone."},{"cited_title":"FaceNet: A unified embed- ding for face recognition and clustering,","cited_arxiv_id":null,"evidence_quote":"Supplies the triplet-loss formulation and buffer-based sampling that pulls together score-similar embeddings and pushes apart score-dissimilar ones."},{"cited_title":"Unsupervised feature learning via non-parametric instance discrimination,","cited_arxiv_id":null,"evidence_quote":"Supplies the non-parametric instance-discrimination sampling strategy adapted for choosing positives and negatives from the FIFO buffer."}],"review_version":1}