{"id":"d9dac318-5284-41b3-bd16-11a854796f72","arxiv_id":"2509.04378","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ASE-MLLM fuses EAT-derived aesthetic saliency into MLLMs via cross-attention, improving aesthetic captioning over fine-tuning alone, but the SOTA claim rests on comparing to non-fine-tuned baselines.","lead":"This paper adds aesthetic saliency maps to a multimodal language model to generate captions about what makes images beautiful. It reports strong results on three aesthetic captioning datasets, but the main comparison is against models that were not fine-tuned.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed benefit of aesthetic saliency is not causally established: the IASC ablation lacks a saliency-content control, and Mc (Eq. 5) is never validated against caption-relevant regions.","rationale":"The reader's weakest assumption is exactly that Mc reliably marks caption-relevant aesthetic regions, and the paper does not test this directly. I agree that this is the most load-bearing issue. Aesthetic saliency enhancement is the paper's stated novelty; if the map's content is irrelevant, the contribution reduces to a fine-tuned MLLM with extra parameters. The ablation evidence across two base models is a genuine strength and argues for a conditional rather than a rejection verdict, but the missing controls prevent the mechanism claim from being accepted as established. The SOTA comparison in Table I is also confounded by unequal fine-tuning, an issue the paper itself acknowledges in §IV-D. Adding the proposed random-map control and, ideally, fine-tuned MLLM baselines would make the acceptance criteria concrete. Therefore the existing CONDITIONAL verdict should stand unchanged.","tokens_in":14936,"tokens_out":7832,"duration_ms":74928,"concrete_test":"Run the Table II DPC2022 condition FT+IASC, but replace Mc with a random map of the same spatial shape and value distribution (or with a standard visual-saliency map, e.g., deep saliency), keeping all hyperparameters, cross-attention structure, and training pipeline identical. If the random/saliency control retains most of the CIDEr/METEOR gain over the FT row, the aesthetic content of Mc is not load-bearing and the central mechanism is unsupported; if the control clearly underperforms the real Mc and matches FT, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that integrating aesthetic saliency—not merely fine-tuning or extra parameters—improves AIC. In §III-B/C, IASM produces Mc from EAT's top class score via LayerCAM (Eqs. 1–5), and IAS-ViT injects Mc through cross-attention. The ablation in Tables II/III shows FT+IASC > FT on most metrics and is consistent across InternVL and Qwen, which is genuine supporting evidence. However, it does not establish that Mc's aesthetic content causes the gain. There is no control using a random or generic visual-saliency map, no comparison with a cross-attention path fed only original features, and no direct test that Mc localizes the aesthetic elements named in target captions. EAT is an aesthetic-scoring model, not a caption-grounding model, so its LayerCAM map may be noisy, object-saliency-like, or dataset-dependent; if so, the improvement could come from added parameters, regularization, or generic attention rather than aesthetic saliency. The SOTA claim is separately weakened because §IV-D states the MLLM baselines in Table I were not fine-tuned, so Table I mostly measures fine-tuning, not the saliency contribution. The decisive issue is causal attribution of the IASC gain.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ASE-MLLM, a framework for Aesthetic Image Captioning (AIC) that injects an externally computed aesthetic-saliency map into an MLLM's vision encoder. Specifically, an Image Aesthetic Saliency Module (IASM) computes a saliency map Mc from the EAT aesthetic-scoring model using LayerCAM (Eqs. 1–5), and an IAS-ViT encoder fuses Mc with the original image features via cross-attention. The framework is instantiated on InternVL2-8B and Qwen-VL-Chat. Experiments on DPC2022, PCCD, and RPCD report eleven captioning metrics. The core quantitative evidence is the ablation in Tables II and III, where fine-tuning plus IASC (IASM+IAS-ViT) outperforms fine-tuning alone on nearly all metrics across two base models, e.g., DPC2022 CIDEr 0.068→0.072 and METEOR 0.142→0.150. The paper also claims state-of-the-art performance over generic (non-fine-tuned) MLLMs and conventional AIC models in Table I.","tokens_in":15289,"tokens_out":3157,"duration_ms":30946,"significance":"The idea of making an MLLM explicitly attend to aesthetic saliency rather than generic visual saliency is timely and plausible, and the paper is, to my knowledge, the first to propose this specific integration for AIC. The internal ablation is a genuine strength: the IASC gain is replicated across two different base MLLMs and is visible across most of the eleven metrics, which suggests the effect is not a single lucky metric. The method itself is simple and easy to build on. However, the evidence is not yet sufficient to establish that the aesthetic content of Mc, rather than added parameters, architectural changes, or regularization, causes the improvement; no control saliency or cross-attention variant is tested, and no significance or variability analysis is provided. The SOTA claim is also overstated because the MLLM baselines in Table I were not fine-tuned. If the missing controls are supplied, the paper could make a solid contribution; in its current form the central causal claim remains under-supported.","major_comments":[{"comment":"The central claim—that integrating aesthetic saliency, not merely fine-tuning or extra parameters, improves AIC—is not causally established. The IASC ablation compares FT+IASC against FT, but IASC bundles IASM (Mc from EAT+LayerCAM, Eq. 5) with the new cross-attention block in IAS-ViT. There is no control using (i) a random or generic visual-saliency map in place of Mc, (ii) a cross-attention path fed only duplicated original features, or (iii) an otherwise identical architecture with the saliency branch removed but parameter count matched. Without such controls, the gains in Tables II/III could come from additional capacity, regularization, or attention dynamics rather than from the aesthetic content of Mc. Additionally, there is no direct validation that Mc localizes the regions named in the target captions; EAT is an aesthetic-scoring model, not a caption-grounding model. I recommend","section":"§III-B/C, Table II, Table III"},{"comment":"The abstract and Section I claim state-of-the-art performance, but Table I compares the fine-tuned ASE-MLLM against MLLM baselines that, as stated in §IV-D, 'had not been fine-tuned.' Since this paper's own ablation shows that fine-tuning alone produces most of the gain (e.g., DPC2022 BLEU-4 0.011→0.144), Table I largely demonstrates the effect of fine-tuning, not the contribution of aesthetic saliency. To support a SOTA claim, the comparison set should include fine-tuned AIC-specific MLLM baselines (e.g., the same InternVL2-8B and Qwen-VL-Chat fine-tuned without IASC, and recent AIC methods such as AesExpert/UNIAA evaluated on shared metrics), rather than only zero-shot general-purpose MLLMs.","section":"§IV-D, Table I"},{"comment":"No error bars, multiple seeds, or significance tests are reported. The IASC improvements are small on several metrics (e.g., DPC2022 CIDEr 0.068→0.072, METEOR 0.142→0.150), and although the pattern across metrics and base models is suggestive, single-run differences of this size are not convincing without variance estimates or paired significance tests. The authors should report results over at least 3 seeds with means and standard deviations, or use bootstrap resampling for the test-set metrics, to establish that the reported margins are not within noise.","section":"Table II, Table III"}],"minor_comments":[{"comment":"The text refers to 'our AES-MLLM framework' when discussing Qwen; the acronym should be ASE-MLLM. This typo appears in a methodological claim and should be corrected.","section":"§IV-D"},{"comment":"Equation (5) is garbled in the manuscript (')kAk ijRELUMc'). Please rewrite it as a clear linear combination of ReLU-activated saliency feature maps across channels. Also, the text says 'linear combination' but does not specify the weights used in that combination.","section":"Eq. (5)"},{"comment":"The evaluation section omits the fact that the ARIC comparison in Table I reports only metrics copied from the original paper, and that some metrics are missing (BLEU, ROUGE, etc.). This should be stated directly next to the table, not only in prose, to avoid misleading comparisons.","section":"§IV-C"},{"comment":"There are several reference formatting issues: [17] contains 'on.1' instead of 'no.1'; [3] lists an incorrect journal volume/issue; [5] uses 'INT. J. PATTERN RECOGN.' with an inconsistent format; and [13] is given as an ACM MM paper but the publisher appears as 'Proc. ACM. Int. Conf. Multimed.' The reference list should be cleaned up.","section":"References"},{"comment":"The implementation details do not state the hardware budget, training epochs, or total number of trainable parameters for the proposed modules. This makes it difficult to judge the 'efficient' claim for IASM and the computational overhead of the 24-block IAS-ViT.","section":"§IV-B"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper makes a plausible, incremental contribution—putting aesthetic saliency maps into an MLLM via a cross-attention encoder—but it does not actually show that the aesthetic content of those maps is what drives the gains. The headline 'SOTA' claim is also overstated, as the paper admits its MLLM baselines were not fine-tuned. Still, the internal ablation is consistent across two base models, and the limitation is fixable in revision.\n\nThe new thing here is the specific combination: IASM uses EAT's classifier gradient via LayerCAM to produce a saliency map, and IAS-ViT injects that map via cross-attention after self-attention. That appears to be first for AIC, and the architecture is clear. The paper also deserves credit for running the same ablation on InternVL and Qwen, showing the effect is not base-model-specific.\n\nWhere it gets soft: the causal story. The ablation in Tables II and III compares fine-tuning with and without IASC, and the latter usually wins. But there is no control that replaces the aesthetic saliency map with something inert—random features, a conventional visual saliency map, or the same cross-attention block fed with the original features twice. Without that, the improvement could come from added parameters, regularization, or generic attention. The stress-test note is right: EAT is an aesthetic scorer, not a caption-grounding model, so the LayerCAM map may be noisy or object-saliency-like. The paper doesn't validate that the map localizes the aesthetic elements named in the captions. Also, no error bars or significance tests on the deltas—many are small.\n\nThe SOTA claim in the abstract and conclusion is not supported by Table I because the comparison MLLMs were not fine-tuned. The paper does state this in Section IV-D, but the abstract overreaches. The comparison would be fair if it included fine-tuned versions of those MLLMs, or at least a fine-tuned InternVL without IASC as the primary baseline.\n\nMinor: the target layer for LayerCAM and the number of IAS-ViT blocks (24) are chosen by 'experimental tests' with no sensitivity analysis, so the reader can't know how robust the design choices are.\n\nWho is this for? Researchers in aesthetic captioning and aesthetics-aware MLLMs. It's a serious, competent integration paper, not a paradigm shift. It deserves peer review—the idea is worth testing properly—but the revision needs the controls and the fine-tuned baselines.","headline":"A plausible incremental integration of aesthetic saliency into MLLMs for AIC, but the causal benefit of the saliency map is not established and the SOTA claim is overstated; worth a serious referee with revisions.","tokens_in":15721,"tokens_out":1991,"would_cite":false,"duration_ms":17504,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that feeding a gradient-derived aesthetic saliency map into a vision encoder through cross-attention lets multimodal LLMs produce state-of-the-art aesthetic image captions.","keywords":["aesthetic image captioning","aesthetic saliency","multimodal large language models","cross-attention","LayerCAM","vision transformer","image aesthetics","benchmark evaluation"],"falsifier":"Take the trained ASE-MLLM, replace M_c with a randomly generated or fixed-center saliency map for a held-out split of DPC2022, and re-run captioning: if CIDEr stays near 0.072, the mechanism is not what carries the gain. Or compare M_c against human-annotated aesthetic regions on test images: if the map overlaps no better than a plain visual-saliency or object-detector map while the caption gains persist, the saliency explanation is not the source of the improvement.","tokens_in":14892,"feed_emoji":"🎨","tokens_out":4612,"duration_ms":39011,"temperature":0.7,"pith_summary":"Generating a caption that says why a photo is beautiful, not just what it shows, requires a model to look where the aesthetic content is. This paper argues that multimodal LLMs, even after fine-tuning on captioning data, do not know where to look, and that explicitly feeding them a map of aesthetically salient regions fixes this. The proposed framework, ASE-MLLM, computes that map from an aesthetics classifier's gradients, then fuses it into the vision encoder with a cross-attention layer before the language model generates the caption. On three standard aesthetic-captioning benchmarks, the saliency-guided model beats fine-tuning alone on nearly every metric and reports state-of-the-art results. If the mechanism holds, it offers a cheap, model-agnostic way to steer LLM captions toward the photographic qualities humans actually critique.","feed_headline":"Saliency map steers MLLMs to better aesthetic captions","feed_subtitle":"First framework to inject aesthetic saliency into multimodal LLMs tops three captioning benchmarks","key_machinery":"The aesthetic saliency map M_c: a per-pixel heatmap produced by the IASM from EAT's aesthetic classifier via LayerCAM (positive-gradient-weighted activation at one chosen layer). It is the load-bearing signal, because IAS-ViT uses it as the query in a cross-attention layer whose keys and values come from the original image features, forcing the encoder's output tokens to be re-weighted toward regions of high aesthetic value before the LLM generates captions.","core_discovery":"ASE-MLLM is claimed to be the first framework to integrate image aesthetic saliency into a multimodal large language model specifically for aesthetic image captioning. Its Image Aesthetic Saliency Module runs an input photo through the EAT aesthetic transformer, takes the classifier's score for the most salient aesthetic class, and uses LayerCAM to turn the positive gradients of that score into a spatial saliency map M_c. A modified vision encoder, IAS-ViT, then applies cross-attention with the saliency features as query and the original image features as key and value, so the model's visual tokens carry aesthetic-focus information into the language model. The paper reports that this combina","pith_inferences":["The reported absolute gains are small (e.g., DPC2022 CIDEr rises from 0.068 to 0.072), so the practical significance depends on whether the added metric points reflect genuinely more aesthetic words rather than style shifts; a human-preference or attribute-level evaluation would settle this—the paper does not run one.","Because M_c is built from the argmax aesthetic class of a single classifier, images with diffuse or multi-focus aesthetics may get mislocalized saliency; a multi-class or distribution-weighted saliency aggregation is a direct, testable variant.","The same query-from-saliency, key/value-from-image cross-attention design transfers naturally to other attribute-focused captioning tasks (e.g., describing materials, defects, or emotions) whenever a cheap attribute saliency map exists.","The admitted weakness on RPCD's CIDEr and SPICE suggests the saliency benefit interacts with the style gap between the base model's language distribution and the dataset's colloquial comments—worth probing with style-aware prompts."],"forward_implications":["Retrofitting is cheap: the saliency branch is added to the existing ViT encoder, so any MLLM with a ViT-style image encoder could adopt the pattern without altering the language model.","The margin over fine-tuning is consistent across all three datasets and both base models, which suggests aesthetic saliency is complementary to, not a replacement for, task-specific fine-tuning.","Because results improved for two different base MLLMs (InternVL2-8B and Qwen-VL-Chat), the framework is claimed to generalize beyond a single architecture.","If the approach holds, aesthetic-relevant outputs beyond captioning—photo critiques, editing suggestions, aesthetic scoring rationales—could use the same saliency-steering mechanism."],"supporting_citations":[{"why":"Supplies the EAT aesthetic transformer whose classifier and attention define which regions count as aesthetically salient.","marker":"[13]"},{"why":"LayerCAM is the algorithm that converts the classifier's positive gradients into the spatial saliency map used in the fusion.","marker":"[44]"},{"why":"InternVL2-8B is the base MLLM providing the ViT encoder, projector, and LLM that are fine-tuned.","marker":"[41]"},{"why":"Provides the DPC2022 benchmark and the ARIC baseline that the method must beat.","marker":"[21]"},{"why":"Introduced the PCCD dataset and several evaluation metrics (SPICE-L, precision, recall) used in the experiments.","marker":"[15]"},{"why":"Supplies the RPCD dataset, testing generalization to Reddit-style photo critiques.","marker":"[20]"},{"why":"Qwen-VL-Chat serves as the second base model to demonstrate the framework's generality.","marker":"[51]"},{"why":"Show-and-Tell is the classical image-captioning baseline included in the comparison tables.","marker":"[56]"}],"fun_headline_variants":["First MLLM to use aesthetic saliency for image captioning","Saliency-guided MLLM writes better aesthetic captions","Aesthetic saliency boosts MLLM captioning to SOTA","ASE-MLLM: injecting saliency into MLLMs for aesthetic captions","Cross-attention saliency makes MLLMs caption aesthetics better"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The aesthetic saliency map taken from an aesthetics classifier's gradients reliably marks the regions that the target captions actually describe; if that map is noisy or misaligned with human aesthetic focus, the cross-attention fusion cannot deliver the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["First MLLM to use aesthetic saliency for image captioning","Saliency-guided MLLM writes better aesthetic captions","Aesthetic saliency boosts MLLM captioning to SOTA","ASE-MLLM: injecting saliency into MLLMs for aesthetic captions","Cross-attention saliency makes MLLMs caption aesthetics better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2341,"prompt_tokens":780,"completion_tokens":1561,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":524,"completion_tokens_details":{"reasoning_tokens":1470}},"tokens_in":524,"tokens_out":1561,"duration_ms":9425,"temperature":1.0,"reasoning_tokens":1470,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:12:33.868894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained ASE-MLLM, replace M_c with a randomly generated or fixed-center saliency map for a held-out split of DPC2022, and re-run captioning: if CIDEr stays near 0.072, the mechanism is not what carries the gain. Or compare M_c against human-annotated aesthetic regions on test images: if the map overlaps no better than a plain visual-saliency or object-detector map while the caption gains persist, the saliency explanation is not the source of the improvement.","supporting_citations":[{"cited_title":"Eat: An enhancer for aesthetics-oriented transformers,","cited_arxiv_id":null,"evidence_quote":"Supplies the EAT aesthetic transformer whose classifier and attention define which regions count as aesthetically salient."},{"cited_title":"Layercam: Exploring hierarchical class activation maps for localization,","cited_arxiv_id":null,"evidence_quote":"LayerCAM is the algorithm that converts the classifier's positive gradients into the spatial saliency map used in the fusion."},{"cited_title":"Understanding aesthetics with language: A photo critique dataset for aesthetic assessment,","cited_arxiv_id":null,"evidence_quote":"Supplies the RPCD dataset, testing generalization to Reddit-style photo critiques."},{"cited_title":"Show and tell: A neural image caption generator,","cited_arxiv_id":null,"evidence_quote":"Show-and-Tell is the classical image-captioning baseline included in the comparison tables."}],"review_version":1}