{"id":"0cc672fb-65b7-42e2-9fe7-f0f3117585c1","arxiv_id":"2412.04090","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An LLM agent reads past loss weights and quality scores, then writes new loss weights, letting image processing models be trained toward non-differentiable objectives like IQA scores and text feedback.","lead":"This paper introduces LossAgent, a large language model that repeatedly proposes new weights for a small set of known loss functions while an image super-resolution or restoration model trains. It is meant to let image processing networks be optimized toward goals that are hard to turn into differentiable losses, such as perceptual quality scores or text descriptions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All-in-one restoration results do not establish that LossAgent converts weak feedback into effective weight updates: Table 9 shows no consistent advantage over Fixed (CLIPIQA avg 0.721 vs 0.722), and Table 5's textual-objective experiment yields identical averages (3.94) across Fixed, Q-Align, and…","rationale":"The reader correctly identified the feedback-discriminability assumption as the weakest link, and Section 7's admission supports that concern. My stress-test sharpens this into a concrete quantitative failure: on the all-in-one restoration task, which the paper itself flags as non-robust, LossAgent shows no consistent advantage over Fixed, and on the textual-objective experiment the averages are identical to two decimal places. The paper's broad 'any objective' claim therefore rests on the classical SR experiments, where the feedback set overlaps with the evaluation set (Set14), creating a selection-bias risk that the ablations in Section 8.4 do not fully rule out. No error bars, repeated seeds, or significance tests are reported anywhere. I do not see fraud or an internally inconsistent method; the framework is plausible and the prompt ablations show the LLM is responding to prompt content. But the evidence as presented supports at most a conditional claim: LossAgent can improve metrics when the feedback is discriminative and the feedback set is informative. The requested experiments are cheap and would settle the question directly. Hence I recommend keeping the CONDITIONAL verdict rather than rejecting or accepting outright.","tokens_in":19672,"tokens_out":1945,"duration_ms":21940,"concrete_test":"Re-run the all-in-one restoration experiment (PromptIR, Table 9) and the textual-objective experiment (Co-Instruct, Table 5) with 3 random seeds each for Fixed, Random, and LossAgent, reporting mean +/- std on every benchmark row. Compute the average Q-Align and CLIPIQA gap between LossAgent and Fixed with a paired test across the five evaluation subsets. If the gap falls within one standard deviation or fails significance, the central claim that LossAgent converts weak or textual feedback into effective loss-weight updates is unsupported on these tasks.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that LossAgent optimizes image processing networks toward 'any customized objectives,' including non-differentiable text and human feedback, by adaptively weighting a compositional loss repository. The load-bearing condition is that the LLM's weight updates are driven by meaningful external feedback rather than by random drift or the generic tendency of continued fine-tuning with fixed losses. The authors concede in Section 7 that for all-in-one restoration, 'the minimal differences between images generated in consecutive stages...limit the instructional information available to the agent from feedback.' Table 9 shows the consequence: LossAgent's CLIPIQA average (0.721) is essentially equal to Fixed (0.722), NIQE worsens on two of five conditions, and the Q-Align gain over Fixed is only ~0.02. For the decisive textual-objective experiment (Table 5), the Q-Align averages for Fixed, Q-Align-optimized, and Co-Instruct-optimized models are 3.92, 3.94, and 3.94 — all within 0.02 with no error bars, no repeated seeds, and no significance test. The only consistently positive evidence comes from the classical SR task, where the feedback images (sampled from Set14) overlap with the evaluation set, producing a potential feedback-selection artifact. Across the board, the claimed advantage over the Fixed baseline is often on the order of 0.01–0.03 metric units, which is within plausible run-to-run noise for these IQA metrics. The paper therefore does not yet establish that the LLM's reasoning, rather than the fixed-loss fine-tuning itself, is responsible for the reported improvements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LossAgent, an LLM-based agent that dynamically adjusts the weights of a fixed repository of differentiable losses (L1, perceptual, GAN, or LPIPS) during training of image processing networks. The agent receives external feedback from non-differentiable or textual objectives—e.g., NIQE, MANIQA, CLIPIQA, Q-Align, or Co-Instruct text—and uses a three-part prompt (system, historical, customized needs) to reason about new loss weights at each training stage. Experiments cover classical image super-resolution, real-world super-resolution, and all-in-one restoration, with three baselines: pre-trained, random weights, and fixed weights. The core claim is that LossAgent enables end-to-end optimization toward any customized objective by converting external feedback into appropriate compositional loss weights.","tokens_in":20017,"tokens_out":4127,"duration_ms":45021,"significance":"If the results are accepted, LossAgent provides a practical and novel way to optimize low-level vision models toward non-differentiable or subjective objectives without designing surrogate losses. The prompt-engineering design is thoughtful, and the paper includes useful ablations (system/historical/format prompts), multiple tasks, and multiple backbones. However, the current evidence does not yet establish the central claim. Key problems are the overlap between the feedback image set and the evaluation set for classical SR, the absence of multiple seeds and error bars, and the fact that several key comparisons are within 0.01–0.03 metric units, which is likely within run-to-run noise. The all-in-one restoration results even show no advantage over the fixed baseline on CLIPIQA, and the authors concede that consecutive-stage outputs are too similar for the feedback to be informative. These gaps are fixable with additional experiments, but they are load-bearing for the paper's claims.","major_comments":[{"comment":"The feedback image set for classical image SR is sampled from Set14, the same benchmark that appears in the evaluation tables (Table 1: Set14 column). The LLM receives scores on these sampled Set14 images during training and adjusts loss weights to improve those scores, so the reported improvement on Set14 (and, to a smaller extent, on other benchmarks through correlated effects) is partly a fitted outcome rather than an independent verification. The discussion in §8.4 compares Set14 vs. DIV2K as feedback sets but does not address this overlap. Please evaluate on a held-out feedback set (e.g., sample feedback images from a separate set or explicitly exclude the feedback images when computing Set14 averages) to establish that the gains generalize to unseen data.","section":"§4.1, Table 1, §8.4"},{"comment":"No error bars, multiple seeds, or significance tests are reported anywhere. Several headline comparisons are extremely small: Q-Align average 3.63 vs. 3.60 for Fixed in Table 1; Table 5 Q-Align averages 3.92, 3.94, 3.94 for Fixed, Q-Align, and Co-Instruct; Table 9 CLIPIQA average 0.721 vs. 0.722 for LossAgent and Fixed. These differences are within plausible run-to-run variation for IQA metrics and for LLM sampling. At minimum, run each method with at least three random seeds and report mean ± std, and state the LLM decoding temperature and whether the same LLM responses were reused.","section":"Tables 1, 3, 5, 9"},{"comment":"The textual-objective experiment does not demonstrate that LossAgent successfully transfers textual feedback into better optimization. The Co-Instruct-optimized model achieves an average Q-Align score of 3.94, identical to the Q-Align-optimized model and only 0.02 above the Fixed baseline (3.92). Since Q-Align is also the evaluation metric, this is at best a null result. The paper argues that Co-Instruct and Q-Align share similar training data, but this does not make Q-Align a sensitive or unbiased measure of text-guided optimization. Please provide additional evidence that the textual feedback actually altered the training trajectory (e.g., show that loss weights changed differently than in the Q-Align condition) and evaluate with a metric that can distinguish the models, such as a human preference study or a pairwise MLLM comparison.","section":"§4.2.3, Table 5"},{"comment":"The all-in-one restoration results contradict the paper's general claim of effectiveness. The authors concede in §7 that \"the minimal differences between images generated in consecutive stages...limit the instructional information available to the agent from feedback,\" and Table 9 confirms the consequence: LossAgent's CLIPIQA average (0.721) is slightly worse than Fixed (0.722), NIQE is worse on Dehaze and Derain, and Q-Align gains only 0.02 over Fixed. The text calls this a \"commendable performance improvement,\" but the table does not support that wording. Either the feedback mechanism must be improved (e.g., by comparing multi-stage images or using accumulated score differences) or the robustness claim must be limited to tasks where consecutive outputs are sufficiently distinguishable.","section":"§7, Table 9"},{"comment":"The \"Fixed\" baseline is a single hand-picked weight vector that is identical to the initial weights used by LossAgent. It is plausible that the fixed vector is suboptimal for some objectives, making the adaptive method look better by comparison. Similarly, the \"Random\" baseline is not described with enough detail: it is unclear what distribution is used, how many random trials are averaged, and whether the same seed is used across objectives. A stronger baseline would be a small grid search or a set of random fixed weight vectors, reported with variance, to show that the adaptive updates, rather than the particular initial point, are what drive the gains.","section":"§4.1, Table 2"}],"minor_comments":[{"comment":"In the Introduction, \"AgentLoss\" appears where \"LossAgent\" is intended; please fix this typo.","section":"§1"},{"comment":"The random sampling of the feedback image set I is not seeded; please provide the seed or state that results are averaged over multiple sampling runs.","section":"§4.1"},{"comment":"The prompt example in Figure 2 is informative but the text is small; a zoomed-in or typeset version would help readability.","section":"§3.4, Figure 2"},{"comment":"The comparison between Set14 and DIV2K as feedback sets is useful, but it does not address the potential feedback-selection artifact; please also report results when the feedback images are excluded from the evaluation set.","section":"§8.4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and interesting problem, and the LLM-as-loss-agent idea is a reasonable contribution if the evidence is strengthened. However, the current experiments do not distinguish LossAgent from a well-chosen fixed loss-weight schedule, primarily due to the Set14 feedback/evaluation overlap, missing variance estimates, and the near-null textual-objective result. I would be willing to re-review a revision that adds held-out feedback evaluation, multi-seed statistics, and a more sensitive evaluation for the text-guided experiment. The 'any customized objective' claim in the title and abstract should be tempered to reflect the tested settings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a genuinely different use of an LLM in low-level vision—a 'loss agent' that converts non-differentiable evaluation feedback into weights for a fixed loss repository. It's not a paradigm shift, but it is a new and sensible tool. The paper is worth reading for that framing alone. The second thing: the evidence is softer than the 'any objectives' claim. The strongest results are on classical super-resolution, where the gains over a fixed-weight baseline are consistent but small (Q-Align 3.63 vs 3.60, NIQE 4.08 vs 4.23), and the feedback images are sampled from Set14, which is also in the evaluation table. That overlap doesn't invalidate the method, since improvements also appear on Set5 and the other benchmarks, but it's a real confound and should be fixed with a disjoint feedback set.\n\nWhat's actually good. The architecture is simple and well explained: a loss repository of standard differentiable losses, an LLM that sees the full history of weights and scores, and an external evaluator that provides feedback. The prompt engineering is careful—system prompt with metric direction, historical trajectory, and format constraints—and the case study shows the LLM's reasoning is plausible. The ablations justify the design choices: removing the system prompt hurts NIQE, including full history helps MANIQA, and format constraints push output parse success from 21% to 99.9%. They also test two other backbones and a five-loss repository. That's solid, reproducible engineering.\n\nWhere it falls short. First, no error bars and a single seed. Gains of 0.01–0.02 on Q-Align are within run-to-run noise for these IQA metrics. Second, the baselines are too weak. Random and Fixed don't tell you whether the LLM's reasoning matters; a simple hill-climbing over the three weight coefficients using the same external feedback would be the right control. Third, the all-in-one restoration task is the paper's own weak point. The authors concede in Section 7 that consecutive stage images are too similar for feedback to be informative, and Table 9 shows CLIPIQA 0.721 vs 0.722 for Fixed with NIQE worsening on several conditions. That limitation is honestly reported, but it directly contradicts the 'any objective' headline. Fourth, the textual-objective experiment (Table 5) ends with identical Q-Align averages for Fixed, Q-Align, and Co-Instruct models (3.92, 3.94, 3.94)—with no variance, that's not evidence of transfer.\n\nWho it's for: image-processing researchers interested in LLM-driven training loops, and anyone working on non-differentiable metrics as training objectives. It deserves a serious referee rather than desk rejection, but the referee should push for a disjoint feedback set, repeated seeds, and a non-LLM adaptive baseline. If those come back positive, the 'loss agent' idea would be a useful tool.","headline":"A genuinely new use of an LLM as a loss agent for image processing, but the experiments don't yet prove the LLM's weight adjustments beat simple adaptive baselines; worth referee time, but the evaluation needs a cleaner feedback/evaluation split and variance reporting.","tokens_in":20547,"tokens_out":3704,"would_cite":false,"duration_ms":36227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LossAgent makes a large language model the loss designer for image super-resolution and restoration, converting non-differentiable quality scores and text feedback into training-weight updates.","keywords":["LossAgent","LLM agents","loss function design","image super-resolution","image restoration","no-reference image quality assessment","non-differentiable objectives","prompt engineering"],"falsifier":"A direct test is to take one of the reported tasks (e.g., classical SR with Q-Align) and run LossAgent against an identical training pipeline in which the LLM's weight updates are replaced by random perturbations drawn from the same range, repeated over many seeds; if the best or average random run matches the agent's Q-Align score, the claimed advantage over random weight updating is not established. A second, cheaper check is to re-run the feedback loop with a held-out set of 10 images different from the fixed feedback set; scores that do not transfer would show the agent is fitting the feedback images rather than the objective.","tokens_in":19444,"feed_emoji":"🤖","tokens_out":11128,"duration_ms":101968,"temperature":0.7,"pith_summary":"LossAgent proposes using a large language model as the loss designer for end-to-end training of image super-resolution and restoration networks. Rather than forcing every optimization objective to be a differentiable loss, the method keeps a small repository of standard trainable losses—L1, perceptual, GAN, and LPIPS—and has the LLM reweight them at each training stage in response to feedback from whatever objective the user cares about, including no-reference quality scores, text descriptions, and comparison-style feedback. The paper reports that this agent-driven weighting beats fixed weights and random weight updates on classical super-resolution, real-world super-resolution, and all-in-one restoration across four quality metrics. If the claim holds, a practitioner can aim an image-processing model at a new metric or a text-specified goal without designing a surrogate loss function.","feed_headline":"LLM agent turns any image-quality metric into training loss weights","feed_subtitle":"An LLM reads non-differentiable metric feedback and reweights standard losses to push scores higher.","key_machinery":"The weighted compositional loss repository is the load-bearing mechanism: a fixed set of differentiable losses $L_1,\\dots,L_M$ combined as $L = w_1 L_1 + \\cdots + w_M L_M$, with the weights as the only thing the agent controls. The LLM agent's three-part prompt engineering—system prompt (role and which score direction means better), historical prompt (the full weight-and-score trajectory), and customized needs prompt (output format and goal constraints)—turns non-differentiable feedback from an external evaluator into parseable weight updates for the next stage. The external evaluator is simply whatever function represents the optimization objective: an IQA metric for score objectives, or an MLLM (a multimodal large language model) for textual objectives.","core_discovery":"LossAgent's central claim is that the optimization trajectory of a low-level image processing network can be steered by an LLM that receives the history of loss weights, the current external feedback, and a prompt stating the goal, and then outputs the next stage's compositional weights. The training loss is always of the form $L = w_1 L_1 + \\cdots + w_M L_M$; only the weights change between stages. The authors argue that representing the objective as an external evaluator rather than as a loss formula is what makes non-differentiable or textual objectives reachable, and they demonstrate the pipeline for single no-reference metrics, simultaneous Q-Align and PSNR, and textual feedback from a multimodal evaluator (Co-Instruct).","pith_inferences":["The 'any objective' claim is bounded by the evaluator's discriminative power: on tasks where consecutive outputs look almost identical, as the authors note for all-in-one restoration, the feedback carries little information and the LLM has little to reason from.","A strict test of the mechanism would replace the LLM with a simple black-box optimizer over the same weight simplex; if that baseline matches LossAgent, the contribution is prompting-guided search rather than LLM-specific reasoning.","Because feedback is computed on only ten fixed images, the reported gains could partly reflect overfitting to that feedback set; evaluating on a separate held-out feedback set would clarify generalization.","The architecture already accepts text feedback, so a human-in-the-loop study—letting a person compare outputs instead of a metric—is a direct next experiment."],"forward_implications":["Image-processing models can be trained toward no-reference quality metrics such as NIQE, MANIQA, CLIPIQA, and Q-Align without a differentiable surrogate.","Because the objective is represented by an external evaluator, the same reweighting loop extends to text descriptions and comparison feedback without changing the training code.","Adding the objective as feedback alongside a standard metric, e.g., Q-Align plus PSNR, improves both scores on the tested super-resolution benchmarks.","The full trajectory of historical loss weights and scores is necessary for good behavior: dropping to the last two stages degrades results.","The extra training cost is roughly 11–18% over the backbone training time, with no added cost at inference."],"supporting_citations":[{"why":"supplies the instruction-tuned LLM that plays the role of the loss agent.","marker":"[29]"},{"why":"supplies implementations of the four no-reference IQA metrics used to generate score feedback.","marker":"[5]"},{"why":"is the super-resolution backbone whose pretrained weights are re-optimized by LossAgent in the SR experiments.","marker":"[22]"},{"why":"is the all-in-one restoration backbone used for the image-restoration experiments.","marker":"[34]"},{"why":"provides the GAN-based super-resolution training recipe whose loss set (L1, perceptual, GAN) forms the initial loss repository.","marker":"[48]"},{"why":"is CLIPIQA, used as a single optimization objective and as feedback for the real-world SR task.","marker":"[44]"},{"why":"is Q-Align, used as a single objective, as one of the double objectives alongside PSNR, and to evaluate the text-optimized model.","marker":"[50]"},{"why":"is Co-Instruct, the multimodal evaluator that generates textual feedback for the textual optimization objective.","marker":"[51]"},{"why":"is the NIQE no-reference metric used as one single optimization objective; its lower-is-better direction must be stated in the prompt.","marker":"[31]"},{"why":"is LPIPS, the perceptual metric used as one of the differentiable losses in the all-in-one restoration repository.","marker":"[63]"}],"fun_headline_variants":["LLM picks loss weights to hit any image metric","Agent reweights losses via LLM for custom image goals","Language model steers image training with custom metrics","LossAgent: LLM adapts losses to any image objective","LLM agent sets loss weights from any image feedback"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole loop depends on the external evaluator's scores on ten fixed test images changing noticeably from one training stage to the next; the authors themselves say that when consecutive outputs are nearly identical, the feedback gives the agent too little information and the method becomes less robust.","fun_headline_variants_meta":{"raw":{"variants":["LLM picks loss weights to hit any image metric","Agent reweights losses via LLM for custom image goals","Language model steers image training with custom metrics","LossAgent: LLM adapts losses to any image objective","LLM agent sets loss weights from any image feedback"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1449,"prompt_tokens":924,"completion_tokens":525,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":540,"tokens_out":525,"duration_ms":5842,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:45:38.725644+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to take one of the reported tasks (e.g., classical SR with Q-Align) and run LossAgent against an identical training pipeline in which the LLM's weight updates are replaced by random perturbations drawn from the same range, repeated over many seeds; if the best or average random run matches the agent's Q-Align score, the claimed advantage over random weight updating is not established. A second, cheaper check is to re-run the feedback loop with a held-out set of 10 images different from the fixed feedback set; scores that do not transfer would show the agent is fitting the feedback images rather than the objective.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the instruction-tuned LLM that plays the role of the loss agent."},{"cited_title":"IQA-PyTorch: Pytorch toolbox for image quality assessment","cited_arxiv_id":null,"evidence_quote":"supplies implementations of the four no-reference IQA metrics used to generate score feedback."},{"cited_title":"Swinir: Image restoration using swin transformer","cited_arxiv_id":null,"evidence_quote":"is the super-resolution backbone whose pretrained weights are re-optimized by LossAgent in the SR experiments."},{"cited_title":"Real-esrgan: Training real-world blind super-resolution with pure synthetic data","cited_arxiv_id":null,"evidence_quote":"provides the GAN-based super-resolution training recipe whose loss set (L1, perceptual, GAN) forms the initial loss repository."},{"cited_title":"Ex- ploring clip for assessing the look and feel of images","cited_arxiv_id":null,"evidence_quote":"is CLIPIQA, used as a single optimization objective and as feedback for the real-world SR task."},{"cited_title":"Towards open-ended visual quality comparison, 2024","cited_arxiv_id":null,"evidence_quote":"is Co-Instruct, the multimodal evaluator that generates textual feedback for the textual optimization objective."},{"cited_title":"completely blind","cited_arxiv_id":null,"evidence_quote":"is the NIQE no-reference metric used as one single optimization objective; its lower-is-better direction must be stated in the prompt."},{"cited_title":"The unreasonable effectiveness of deep features as a perceptual metric","cited_arxiv_id":null,"evidence_quote":"is LPIPS, the perceptual metric used as one of the differentiable losses in the all-in-one restoration repository."}],"review_version":1}