Pith. sign in

REVIEW 5 major objections 4 minor 73 references

LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents

T0 review · 5 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LossAgent makes a large language model the loss designer for image super-resolution and restoration, converting non-differentiable quality scores and text feedback into training-weight updates.

desk verdict A genuinely new use of an LLM as a loss agent for image processing, but the experiments don't yet prove the LLM's weight adjustments beat simple adaptive baselines; worth referee time, but the evaluation needs a cleaner feedback/evaluation split and variance reporting. read the letter →

arxiv 2412.04090 v2 pith:3CR5XTW4 submitted 2024-12-05 cs.CV

Bingchen Li , Xin Li , Yiting Lu , Zhibo Chen This is my paper · ORCID
classification cs.CV
keywords LossAgentLLMagentsfunctiondesignimagesuper-resolutionrestorationno-referencequalityassessmentnon-differentiableobjectivespromptengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LossAgent proposes using a large language model as the loss designer for end-to-end training of image super-resolution and restoration networks. Rather than forcing every optimization objective to be a differentiable loss, the method keeps a small repository of standard trainable losses—L1, perceptual, GAN, and LPIPS—and has the LLM reweight them at each training stage in response to feedback from whatever objective the user cares about, including no-reference quality scores, text descriptions, and comparison-style feedback. The paper reports that this agent-driven weighting beats fixed weights and random weight updates on classical super-resolution, real-world super-resolution, and all-in-one restoration across four quality metrics. If the claim holds, a practitioner can aim an image-processing model at a new metric or a text-specified goal without designing a surrogate loss function.

What carries the argument

The weighted compositional loss repository is the load-bearing mechanism: a fixed set of differentiable losses $L_1,\dots,L_M$ combined as $L = w_1 L_1 + \cdots + w_M L_M$, with the weights as the only thing the agent controls. The LLM agent's three-part prompt engineering—system prompt (role and which score direction means better), historical prompt (the full weight-and-score trajectory), and customized needs prompt (output format and goal constraints)—turns non-differentiable feedback from an external evaluator into parseable weight updates for the next stage. The external evaluator is simply whatever function represents the optimization objective: an IQA metric for score objectives, or an MLLM (a multimodal large language model) for textual objectives.

What would settle it

A direct test is to take one of the reported tasks (e.g., classical SR with Q-Align) and run LossAgent against an identical training pipeline in which the LLM's weight updates are replaced by random perturbations drawn from the same range, repeated over many seeds; if the best or average random run matches the agent's Q-Align score, the claimed advantage over random weight updating is not established. A second, cheaper check is to re-run the feedback loop with a held-out set of 10 images different from the fixed feedback set; scores that do not transfer would show the agent is fitting the feedback images rather than the objective.

Watch

Extended reading notes

Core claim

LossAgent's central claim is that the optimization trajectory of a low-level image processing network can be steered by an LLM that receives the history of loss weights, the current external feedback, and a prompt stating the goal, and then outputs the next stage's compositional weights. The training loss is always of the form $L = w_1 L_1 + \cdots + w_M L_M$; only the weights change between stages. The authors argue that representing the objective as an external evaluator rather than as a loss formula is what makes non-differentiable or textual objectives reachable, and they demonstrate the pipeline for single no-reference metrics, simultaneous Q-Align and PSNR, and textual feedback from a multimodal evaluator (Co-Instruct).

Load-bearing premise

The whole loop depends on the external evaluator's scores on ten fixed test images changing noticeably from one training stage to the next; the authors themselves say that when consecutive outputs are nearly identical, the feedback gives the agent too little information and the method becomes less robust.

Editorial extensions

If this is right

  • Image-processing models can be trained toward no-reference quality metrics such as NIQE, MANIQA, CLIPIQA, and Q-Align without a differentiable surrogate.
  • Because the objective is represented by an external evaluator, the same reweighting loop extends to text descriptions and comparison feedback without changing the training code.
  • Adding the objective as feedback alongside a standard metric, e.g., Q-Align plus PSNR, improves both scores on the tested super-resolution benchmarks.
  • The full trajectory of historical loss weights and scores is necessary for good behavior: dropping to the last two stages degrades results.
  • The extra training cost is roughly 11–18% over the backbone training time, with no added cost at inference.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'any objective' claim is bounded by the evaluator's discriminative power: on tasks where consecutive outputs look almost identical, as the authors note for all-in-one restoration, the feedback carries little information and the LLM has little to reason from.
  • A strict test of the mechanism would replace the LLM with a simple black-box optimizer over the same weight simplex; if that baseline matches LossAgent, the contribution is prompting-guided search rather than LLM-specific reasoning.
  • Because feedback is computed on only ten fixed images, the reported gains could partly reflect overfitting to that feedback set; evaluating on a separate held-out feedback set would clarify generalization.
  • The architecture already accepts text feedback, so a human-in-the-loop study—letting a person compare outputs instead of a metric—is a direct next experiment.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces LossAgent, an LLM-based agent that dynamically adjusts the weights of a fixed repository of differentiable losses (L1, perceptual, GAN, or LPIPS) during training of image processing networks. The agent receives external feedback from non-differentiable or textual objectives—e.g., NIQE, MANIQA, CLIPIQA, Q-Align, or Co-Instruct text—and uses a three-part prompt (system, historical, customized needs) to reason about new loss weights at each training stage. Experiments cover classical image super-resolution, real-world super-resolution, and all-in-one restoration, with three baselines: pre-trained, random weights, and fixed weights. The core claim is that LossAgent enables end-to-end optimization toward any customized objective by converting external feedback into appropriate compositional loss weights.

Significance. If the results are accepted, LossAgent provides a practical and novel way to optimize low-level vision models toward non-differentiable or subjective objectives without designing surrogate losses. The prompt-engineering design is thoughtful, and the paper includes useful ablations (system/historical/format prompts), multiple tasks, and multiple backbones. However, the current evidence does not yet establish the central claim. Key problems are the overlap between the feedback image set and the evaluation set for classical SR, the absence of multiple seeds and error bars, and the fact that several key comparisons are within 0.01–0.03 metric units, which is likely within run-to-run noise. The all-in-one restoration results even show no advantage over the fixed baseline on CLIPIQA, and the authors concede that consecutive-stage outputs are too similar for the feedback to be informative. These gaps are fixable with additional experiments, but they are load-bearing for the paper's claims.

major comments (5)
  1. [§4.1, Table 1, §8.4] The feedback image set for classical image SR is sampled from Set14, the same benchmark that appears in the evaluation tables (Table 1: Set14 column). The LLM receives scores on these sampled Set14 images during training and adjusts loss weights to improve those scores, so the reported improvement on Set14 (and, to a smaller extent, on other benchmarks through correlated effects) is partly a fitted outcome rather than an independent verification. The discussion in §8.4 compares Set14 vs. DIV2K as feedback sets but does not address this overlap. Please evaluate on a held-out feedback set (e.g., sample feedback images from a separate set or explicitly exclude the feedback images when computing Set14 averages) to establish that the gains generalize to unseen data.
  2. [Tables 1, 3, 5, 9] No error bars, multiple seeds, or significance tests are reported anywhere. Several headline comparisons are extremely small: Q-Align average 3.63 vs. 3.60 for Fixed in Table 1; Table 5 Q-Align averages 3.92, 3.94, 3.94 for Fixed, Q-Align, and Co-Instruct; Table 9 CLIPIQA average 0.721 vs. 0.722 for LossAgent and Fixed. These differences are within plausible run-to-run variation for IQA metrics and for LLM sampling. At minimum, run each method with at least three random seeds and report mean ± std, and state the LLM decoding temperature and whether the same LLM responses were reused.
  3. [§4.2.3, Table 5] The textual-objective experiment does not demonstrate that LossAgent successfully transfers textual feedback into better optimization. The Co-Instruct-optimized model achieves an average Q-Align score of 3.94, identical to the Q-Align-optimized model and only 0.02 above the Fixed baseline (3.92). Since Q-Align is also the evaluation metric, this is at best a null result. The paper argues that Co-Instruct and Q-Align share similar training data, but this does not make Q-Align a sensitive or unbiased measure of text-guided optimization. Please provide additional evidence that the textual feedback actually altered the training trajectory (e.g., show that loss weights changed differently than in the Q-Align condition) and evaluate with a metric that can distinguish the models, such as a human preference study or a pairwise MLLM comparison.
  4. [§7, Table 9] The all-in-one restoration results contradict the paper's general claim of effectiveness. The authors concede in §7 that "the minimal differences between images generated in consecutive stages...limit the instructional information available to the agent from feedback," and Table 9 confirms the consequence: LossAgent's CLIPIQA average (0.721) is slightly worse than Fixed (0.722), NIQE is worse on Dehaze and Derain, and Q-Align gains only 0.02 over Fixed. The text calls this a "commendable performance improvement," but the table does not support that wording. Either the feedback mechanism must be improved (e.g., by comparing multi-stage images or using accumulated score differences) or the robustness claim must be limited to tasks where consecutive outputs are sufficiently distinguishable.
  5. [§4.1, Table 2] The "Fixed" baseline is a single hand-picked weight vector that is identical to the initial weights used by LossAgent. It is plausible that the fixed vector is suboptimal for some objectives, making the adaptive method look better by comparison. Similarly, the "Random" baseline is not described with enough detail: it is unclear what distribution is used, how many random trials are averaged, and whether the same seed is used across objectives. A stronger baseline would be a small grid search or a set of random fixed weight vectors, reported with variance, to show that the adaptive updates, rather than the particular initial point, are what drive the gains.
minor comments (4)
  1. [§1] In the Introduction, "AgentLoss" appears where "LossAgent" is intended; please fix this typo.
  2. [§4.1] The random sampling of the feedback image set I is not seeded; please provide the seed or state that results are averaged over multiple sampling runs.
  3. [§3.4, Figure 2] The prompt example in Figure 2 is informative but the text is small; a zoomed-in or typeset version would help readability.
  4. [§8.4] The comparison between Set14 and DIV2K as feedback sets is useful, but it does not address the potential feedback-selection artifact; please also report results when the feedback images are excluded from the evaluation set.

Circularity Check

1 steps flagged · score 5.0 of 10

Feedback images are sampled from the evaluation sets, making part of the reported metric gains in-sample; independent benchmarks limit the circularity.

  1. fitted input called prediction [Section 4.1 (Datasets, Eq. 4) and Section 8.4 (Testing Image Set)]
    "For testing images I mentioned in Equation 4, we randomly sample 10 images from Set14 [61] for classical image SR; randomly sample 10 images from RealSRSet [62] for real-world image SR; randomly sample 10 images from evaluation sets of PromptIR for all-in-one IR. ... We observe that using the sampled Set14 [61] as the testing image set achieves a better CLIPIQA score compared to using the sampled DIV2K [1]. ... Consequently, we choose Set14 as the testing image set for the classical image SR task."

    The loss weights are updated by the LLM using scores F = O(I_Si) computed on the 10 feedback images (Eqs. 4-5). Those same images are drawn from Set14 (CISR), RealSRSet (RISR), and the PromptIR evaluation sets (AIR), and Tables 1, 3, and 9 then report the same metric on the same datasets. The reported gains on Set14, RealSRSet, SOTS, Rain100L, and BSD68 are therefore partly in-sample: the LLM's weight choices were selected to raise the feedback score on those very images, so those columns cannot independently confirm generalization. Section 8.4 makes the fitting explicit by selecting Set14 over DIV2K because it yields a higher CLIPIQA score on Set14.

full rationale

The paper does not derive LossAgent from a formal theorem; it presents an empirical feedback loop: the LLM receives scores F=O(I_Si) from an external IQA/textual expert on a 10-image set I and proposes new weights for the compositional loss L=sum(w_i L_i) (Eqs. 1-6). There is no self-citation chain, no imported uniqueness theorem, and no ansatz relabeled as a discovery. The formal mechanism is therefore not circular. The material circularity is in the evidence: I is sampled from Set14 for CISR, RealSRSet for RISR, and from the PromptIR evaluation sets for AIR, and the same datasets are then tabulated to demonstrate improvement on the very metric being fed back. Consequently, the improvements on those columns are partly fitted to the evaluation set; the LLM's weight trajectories were selected to increase the score on those images. Section 8.4 confirms the selection effect by choosing Set14 because it yields a higher CLIPIQA score than DIV2K. The paper also concedes in Section 7 that for all-in-one restoration the feedback signal is too weak for reliable updates, which aligns with the small differences in Tables 5 and 9. Because several benchmarks and OST300 are not used for feedback, the method retains independent evidence, so this is partial, not total, circularity.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are postulated; LossAgent and the loss repository are software components, and the external evaluation expert is a role assigned to existing IQA models. The central claim instead rests on six groups of hand-chosen hyperparameters and five domain assumptions about LLM reasoning, the span of the loss repository, initialization, feedback informativeness, and instability of direct NR losses.

free parameters (6)
  • Initial compositional loss weights = L1:Perceptual:GAN = 1.0:0.1:0.01 for super-resolution; L1:Perceptual:LPIPS = 1.0:0.1:1.0 for all-in-one restoration
    Hand-chosen starting weights (Table 2), taken from GAN-based training recipes; no justification or sensitivity analysis; the fixed baseline uses these same values, so the comparison is limited.
  • Number of losses in repository (M) = M=3 in main experiments; M=5 in ablation
    Chosen 'to simplify the reasoning process of the LLM' (Section 4.1); ablation Table 12 shows small performance changes, so the results depend on this choice.
  • Training stage length = 5000 iterations for CISR/RISR; 2500 iterations for AIR
    Selected via ablation on the evaluation benchmarks (Table 10), so tuned on test data; affects how often feedback is gathered and the historical prompt length.
  • Feedback image set = 10 images sampled from Set14 for CISR; RealSRSet for RISR; PromptIR eval sets for AIR
    Chosen by comparing Set14 vs DIV2K in Table 13; for CISR the chosen set overlaps with the evaluation benchmark Set14, inflating reported gains.
  • LLM decoding parameters = not reported (likely defaults)
    The LLM's weight outputs are stochastic; temperature, top-p, and seed are not given, so exact weight trajectories are not reproducible.
  • Prompt templates = hand-written system/historical/customized prompts
    Ablations show prompt design strongly affects performance (Tables 6, 7, 8), making the hand-crafted prompts a key tunable component of the method.
assumptions (5)
  • domain assumption The LLM's zero-shot reasoning maps loss-weight history and score feedback to improved loss weights.
    The method has no convergence or optimality guarantee; it relies entirely on LLaMA-3-8B-Instruct following the prompt (Section 3.4, Figure 2). The paper ablates prompts but does not test other controllers (e.g., simple hill climbing).
  • ad hoc to paper The fixed repository of differentiable losses spans any customized optimization objective.
    The abstract claims 'any' objectives, but only three to five standard losses are used; no argument or analysis shows that arbitrary perceptual/text/human objectives lie in the span of L1, perceptual, GAN, and LPIPS losses.
  • ad hoc to paper Starting from pre-trained PSNR-oriented weights is necessary to avoid misleading early-stage fluctuations.
    Section 6 states this rationale; the paper does not ablate random or from-scratch initialization, so the method's behavior without this assumption is unknown.
  • domain assumption IQA scores on 10 sampled images are a stable and informative training signal.
    The feedback loop (Eqs. 4-5) uses a small fixed set; Section 7 admits that when score differences are too small the signal is uninformative, so the assumption is load-bearing and sometimes fails.
  • domain assumption Directly using non-differentiable NR metrics as losses causes training instability, motivating the proxy.
    Supplementary 8.2 tests CLIPIQA/NIQE as losses and reports collapse or suboptimal results; this assumption supports the necessity of the agent, though the test is only on one task.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents." pith.science (2026). https://pith.science/paper/3CR5XTW4

@misc{pith2026241204090,
  author       = {Pith},
  title        = {Pith review of: LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3CR5XTW4}},
  note         = {Machine review of arXiv:2412.04090}
}
read the original abstract

We present the first loss agent, dubbed LossAgent, for low-level image processing tasks, e.g., image super-resolution and restoration, intending to achieve any customized optimization objectives of low-level image processing in different practical applications. Notably, not all optimization objectives, such as complex hand-crafted perceptual metrics, text description, and intricate human feedback, can be instantiated with existing low-level losses, e.g., MSE loss, which presents a crucial challenge in optimizing image processing networks in an end-to-end manner. To eliminate this, our LossAgent introduces the powerful large language model (LLM) as the loss agent, where the rich textual understanding of prior knowledge empowers the loss agent with the potential to understand complex optimization objectives, trajectory, and state feedback from external environments in the optimization process of the low-level image processing networks. In particular, we establish the loss repository by incorporating existing loss functions that support the end-to-end optimization for low-level image processing. Then, we design the optimization-oriented prompt engineering for the loss agent to actively and intelligently decide the compositional weights for each loss in the repository at each optimization interaction, thereby achieving the required optimization trajectory for any customized optimization objectives. Extensive experiments on three typical low-level image processing tasks and multiple optimization objectives have shown the effectiveness and applicability of our proposed LossAgent.

Figures

Figures reproduced from arXiv: 2412.04090 by the authors.

Figure 1
Figure 1. During the training of image processing models (Part I), the loss agent (Part II) gathers feedback from various optimization [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overview of LossAgent. LossAgent bridges image processing models with any optimization objectives through the following [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative comparisons between other methods and LossAgent on CISR. Zoom in for better views. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Qualitative comparisons between baseline and LossAgent [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 5
Figure 5. Figure 5: Illustration of loss weight curves on classical image super-resolution task across four optimization objectives. Zoom in for better [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]
Figure 6
Figure 6. Figure 6: A case study of LossAgent on classical image super-resolution task at 25000 iterations. The optimization objective is NIQE. [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

73 extracted references · 50 canonical work pages

  1. [1]

    Ntire 2017 challenge on single image super-resolution: Dataset and study

    Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In Proceed- ings of the IEEE conference on computer vision and pattern recognition workshops, pages 126–135, 2017. 5, 13

    work page 2017

  2. [2]

    Contour detection and hierarchical image segmen- tation

    Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jiten- dra Malik. Contour detection and hierarchical image segmen- tation. IEEE transactions on pattern analysis and machine intelligence, 33(5):898–916, 2010. 5

    work page 2010

  3. [3]

    Low-complexity single-image super-resolution based on nonnegative neighbor embedding

    Marco Bevilacqua, Aline Roumy, Christine Guillemot, and Marie Line Alberi-Morel. Low-complexity single-image super-resolution based on nonnegative neighbor embedding

  4. [4]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. 1, 3

    work page 1901

  5. [5]

    IQA-PyTorch: Pytorch toolbox for image quality assessment

    Chaofeng Chen and Jiadi Mo. IQA-PyTorch: Pytorch toolbox for image quality assessment. [Online]. Available: https: //github.com/chaofengc/IQA-PyTorch, 2022. 5

    work page 2022

  6. [6]

    Activating more pixels in image super-resolution transformer

    Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Chao Dong. Activating more pixels in image super-resolution transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22367–22377,

    work page

  7. [7]

    Dual aggregation transformer for image super-resolution

    Zheng Chen, Yulun Zhang, Jinjin Gu, Linghe Kong, Xiaokang Yang, and Fisher Yu. Dual aggregation transformer for image super-resolution. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 12312–12321,

  8. [8]

    High- quality image restoration following human instructions.arXiv preprint arXiv:2401.16468, 2024

    Marcos V Conde, Gregor Geigle, and Radu Timofte. High- quality image restoration following human instructions.arXiv preprint arXiv:2401.16468, 2024. 1

    arXiv 2024

Show all 73 references
  1. [9]

    Image super-resolution using deep convolutional net- works

    Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional net- works. IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2015. 2, 3

    2015

  2. [10]

    Large language model for lossless image compression with visual prompts

    Junhao Du, Chuqin Zhou, Ning Cao, Gang Chen, Yunuo Chen, Zhengxue Cheng, Li Song, Guo Lu, and Wenjun Zhang. Large language model for lossless image compression with visual prompts. arXiv preprint arXiv:2502.16163, 2025. 3

    2025 arXiv

  3. [11]

    Generative diffusion prior for unified image restoration and enhancement

    Ben Fei, Zhaoyang Lyu, Liang Pan, Junzhe Zhang, Weidong Yang, Tianyue Luo, Bo Zhang, and Bo Dai. Generative diffusion prior for unified image restoration and enhancement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9935–9946, 202...

    2023

  4. [12]

    Openagi: When llm meets domain experts

    Yingqiang Ge, Wenyue Hua, Kai Mei, Juntao Tan, Shuyuan Xu, Zelong Li, Yongfeng Zhang, et al. Openagi: When llm meets domain experts. Advances in Neural Information Processing Systems, 36, 2024. 1, 3

    2024

  5. [13]

    Mambairv2: Attentive state space restoration

    Hang Guo, Yong Guo, Yaohua Zha, Yulun Zhang, Wenbo Li, Tao Dai, Shu-Tao Xia, and Yawei Li. Mambairv2: Attentive state space restoration. arXiv preprint arXiv:2411.15269 ,

    arXiv

  6. [14]

    Mambair: A simple baseline for image restoration with state-space model

    Hang Guo, Jinmin Li, Tao Dai, Zhihao Ouyang, Xudong Ren, and Shu-Tao Xia. Mambair: A simple baseline for image restoration with state-space model. In European conference on computer vision, pages 222–241. Springer, 2024. 15

    2024

  7. [15]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 1, 3

    2023

  8. [16]

    Single image super-resolution from transformed self-exemplars

    Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5197–5206, 2015. 5

    2015

  9. [17]

    Photo- realistic single image super-resolution using a generative ad- versarial network

    Christian Ledig, Lucas Theis, Ferenc Husz´ar, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. Photo- realistic single image super-resolution using a generative ad- versarial network. In Proceedings of the IE...

  10. [18]

    Benchmarking single- image dehazing and beyond

    Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng, Wenjun Zeng, and Zhangyang Wang. Benchmarking single- image dehazing and beyond. IEEE Transactions on Image Processing, 28(1):492–505, 2018. 5

    2018

  11. [19]

    All-in-one image restoration for unknown corruption

    Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17452– 17462, 2022. 5

    2022

  12. [20]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

  13. [21]

    Efficient and explicit modelling of image hierarchies for image restora- tion

    Yawei Li, Yuchen Fan, Xiaoyu Xiang, Denis Demandolx, Rakesh Ranjan, Radu Timofte, and Luc Van Gool. Efficient and explicit modelling of image hierarchies for image restora- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 18278–18289,

  14. [22]

    Swinir: Image restoration using swin transformer

    Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1833– 1844, 2021. 1, 2, 5, 12

    2021

  15. [23]

    Enhanced deep residual networks for single image super-resolution

    Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In Proceedings of the IEEE confer- ence on computer vision and pattern recognition workshops, pages 136–144, 2017. 2, 3

    2017

  16. [24]

    Chameleon: Plug-and-play compositional reasoning with large language models

    Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jianfeng Gao. Chameleon: Plug-and-play compositional reasoning with large language models. Advances in Neural Information Processing Systems, 36, 2024. 1

    2024

  17. [25]

    Prores: Exploring degradation- 9 aware visual prompt for universal image restoration

    Jiaqi Ma, Tianheng Cheng, Guoli Wang, Qian Zhang, Xing- gang Wang, and Lefei Zhang. Prores: Exploring degradation- 9 aware visual prompt for universal image restoration. arXiv preprint arXiv:2306.13653, 2023. 2

    2023 arXiv

  18. [26]

    Waterloo exploration database: New challenges for image quality as- sessment models

    Kede Ma, Zhengfang Duanmu, Qingbo Wu, Zhou Wang, Hongwei Yong, Hongliang Li, and Lei Zhang. Waterloo exploration database: New challenges for image quality as- sessment models. IEEE Transactions on Image Processing, 26(2):1004–1016, 2016. 5

    2016

  19. [27]

    A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics

    David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings Eighth IEEE International Conference on Computer Vision. IC...

    2001

  20. [28]

    Sketch-based manga retrieval using manga109 dataset

    Yusuke Matsui, Kota Ito, Yuji Aramaki, Azuma Fujimoto, Toru Ogawa, Toshihiko Yamasaki, and Kiyoharu Aizawa. Sketch-based manga retrieval using manga109 dataset. Mul- timedia Tools and Applications, 76(20):21811–21838, 2017. 5

    2017

  21. [29]

    MetaAI. Llama3. https : / / llama . meta . com / llama3,, 2024. 1

    2024

  22. [30]

    Augmented language models: a survey

    Gr´egoire Mialon, Roberto Dess `ı, Maria Lomeli, Christo- foros Nalmpantis, Ram Pasunuru, Roberta Raileanu, Baptiste Rozi`ere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, et al. Augmented language models: a survey. arXiv preprint arXiv:2302.07842, 2023. 3

    2023 arXiv

  23. [31]

    completely blind

    Anish Mittal, Rajiv Soundararajan, and Alan C Bovik. Mak- ing a “completely blind” image quality analyzer. IEEE Signal processing letters, 20(3):209–212, 2012. 3, 5, 6, 8, 12

    2012

  24. [32]

    Embodiedgpt: Vision-language pre-training via embodied chain of thought

    Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 5

    2024

  25. [33]

    Gpt-4 technical report, 2023

    OpenAI. Gpt-4 technical report, 2023. 1, 5

    2023

  26. [34]

    Promptir: Prompting for all-in-one blind image restoration

    Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one blind image restoration. arXiv preprint arXiv:2306.13090,

    arXiv

  27. [35]

    Code llama: Open founda- tion models for code

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J´er´emy Rapin, et al. Code llama: Open founda- tion models for code. arXiv preprint arXiv:2308.12950, 2023. 1

    2023 arXiv

  28. [36]

    Toolformer: Lan- guage models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dess `ı, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024. 1, 3

    2024

  29. [37]

    Velma: Verbalization embodiment of llm agents for vision and language navigation in street view

    Raphael Schumann, Wanrong Zhu, Weixi Feng, Tsu-Jui Fu, Stefan Riezler, and William Yang Wang. Velma: Verbalization embodiment of llm agents for vision and language navigation in street view. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 18924–18933, 2024. 1

    2024

  30. [38]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weim- ing Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024. 1, 3, 5

    2024

  31. [39]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural In- formation Processing Systems, 36, 2024. 1, 3

    2024

  32. [40]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 2

    2014 arXiv

  33. [41]

    Vipergpt: Vi- sual inference via python execution for reasoning

    D´ıdac Sur´ıs, Sachit Menon, and Carl V ondrick. Vipergpt: Vi- sual inference via python execution for reasoning. InProceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 1, 3, 5

    2023

  34. [42]

    Ntire 2017 challenge on single image super-resolution: Methods and results

    Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pages 114–125, 2017. 5

    2017

  35. [43]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. 1, 3, 5

    2023 arXiv

  36. [44]

    Ex- ploring clip for assessing the look and feel of images

    Jianyi Wang, Kelvin CK Chan, and Chen Change Loy. Ex- ploring clip for assessing the look and feel of images. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2555–2563, 2023. 3, 4, 6, 12

    2023

  37. [45]

    Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method

    Tao Wang, Kaihao Zhang, Tianrun Shen, Wenhan Luo, Bjorn Stenger, and Tong Lu. Ultra-high-definition low-light image enhancement: A benchmark and transformer-based method. In Proceedings of the AAAI Conference on Artificial Intelli- gence, pages 2654–2662, 2023. 2

    2023

  38. [46]

    Re- covering realistic texture in image super-resolution by deep spatial feature transform

    Xintao Wang, Ke Yu, Chao Dong, and Chen Change Loy. Re- covering realistic texture in image super-resolution by deep spatial feature transform. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 606–615, 2018. 1, 5

    2018

  39. [47]

    Esrgan: En- hanced super-resolution generative adversarial networks

    Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong, Yu Qiao, and Chen Change Loy. Esrgan: En- hanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pages 0–0, 2018. 2

    2018

  40. [48]

    Real-esrgan: Training real-world blind super-resolution with pure synthetic data

    Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-esrgan: Training real-world blind super-resolution with pure synthetic data. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pages 1905–1914,

    1905

  41. [49]

    Images speak in images: A generalist painter for in-context visual learning

    Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6830–6839, 2023. 2

    2023

  42. [50]

    Q-align: Teaching lmms for vi- sual scoring via discrete text-defined levels

    Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching lmms for vi- sual scoring via discrete text-defined levels. arXiv preprint arXiv:2312.17090, 2023. 2, 3, 6

    2023 arXiv

  43. [51]

    Towards open-ended visual quality comparison, 2024

    Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, 10 Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guangtao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison, 2024. 2, 3, 6

    2024

  44. [52]

    Diffir: Efficient diffusion model for image restoration

    Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang, and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pages 13095–13105, 2023. 1, 2, 3

    2023

  45. [53]

    Learning texture transformer network for image super-resolution

    Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Bain- ing Guo. Learning texture transformer network for image super-resolution. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 5791– 5800, 2020. 5

    2020

  46. [54]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Chencheng Jiang, Haoran Tan, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. arXiv preprint arXiv:2310.08588, 2023. 1, 3, 5

    arXiv

  47. [55]

    Maniqa: Multi-dimension attention network for no-reference image quality assessment

    Sidi Yang, Tianhe Wu, Shuwei Shi, Shanshan Lao, Yuan Gong, Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pa...

    2022

  48. [56]

    Mm-react: Prompting chatgpt for multimodal reasoning and action

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint arXiv:2303.11381, 2023. 1, 3

    2023 arXiv

  49. [57]

    Depicting beyond scores: Advanc- ing image quality assessment through multi-modal language models

    Zhiyuan You, Zheyuan Li, Jinjin Gu, Zhenfei Yin, Tianfan Xue, and Chao Dong. Depicting beyond scores: Advanc- ing image quality assessment through multi-modal language models. In European Conference on Computer Vision, pages 259–276. Springer, 2024. 2, 3, 6

    2024

  50. [58]

    Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild

    Fanghua Yu, Jinjin Gu, Zheyuan Li, Jinfan Hu, Xiang- tao Kong, Xintao Wang, Jingwen He, Yu Qiao, and Chao Dong. Scaling up to excellence: Practicing model scaling for photo-realistic image restoration in the wild. arXiv preprint arXiv:2401.13627, 2024. 1, 2, 3

    2024 arXiv

  51. [59]

    Resshift: Efficient diffusion model for image super-resolution by resid- ual shifting

    Zongsheng Yue, Jianyi Wang, and Chen Change Loy. Resshift: Efficient diffusion model for image super-resolution by resid- ual shifting. Advances in Neural Information Processing Systems, 36, 2024. 1, 2, 3

    2024

  52. [60]

    Restormer: Efficient transformer for high-resolution image restoration

    Syed Waqas Zamir, Aditya Arora, Salman Khan, Mu- nawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728–5739,

  53. [61]

    On single image scale-up using sparse-representations

    Roman Zeyde, Michael Elad, and Matan Protter. On single image scale-up using sparse-representations. In International conference on curves and surfaces, pages 711–730. Springer,

  54. [62]

    Designing a practical degradation model for deep blind image super-resolution

    Kai Zhang, Jingyun Liang, Luc Van Gool, and Radu Timofte. Designing a practical degradation model for deep blind image super-resolution. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision , pages 4791–4800,

  55. [63]

    The unreasonable effectiveness of deep features as a perceptual metric

    Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018. 3

    2018

  56. [64]

    Residual dense network for image super-resolution

    Yulun Zhang, Yapeng Tian, Yu Kong, Bineng Zhong, and Yun Fu. Residual dense network for image super-resolution. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2472–2481, 2018. 2, 3

    2018

  57. [65]

    Lm4lv: A frozen large language model for low-level vision tasks

    Boyang Zheng, Jinjin Gu, Shijun Li, and Chao Dong. Lm4lv: A frozen large language model for low-level vision tasks. arXiv preprint arXiv:2405.15734, 2024. 3 11 LossAgent: Towards Any Optimization Objectives for Image Processing with LLM Agents Supplementary Material

    2024 arXiv

  58. [66]

    We list the details of training iterations for each stage, the total number of training iterations, and the initial weights of loss functions in Table 2

    Training Details As demonstrated in Section 3.2, we divide the whole train- ing process of image processing models into several stages to enable the dynamic adjustment of loss weights through LossAgent. We list the details of training iterations for each stage, the total numbe...

  59. [67]

    As demonstrated in the Table, in the all-in-one IR task, LossAgent does not perform as robustly as in the other two tasks

    Additional Results for All-in-One Image Restoration We provide the quantitative results of LossAgent compared to other methods on all-in-one image restoration tasks across four optimization objectives in Table 9. As demonstrated in the Table, in the all-in-one IR task, LossAge...

  60. [68]

    More Ablation Studies In this section, we provide more ablation studies to verify the reliability of our design for LossAgent. 8.1. Iterations for Each Stage In this part, we conduct ablation studies about training it- erations for each stage. As demonstrated in Table 10, a mo...

  61. [70]

    L1:Perceptual:GAN=0.7:0.3:0.05

    Case Study In this section, we provide a case study on classical image super-resolution in Figure 6 to help readers better understand the process of LossAgent. As demonstrated, LossAgent is capable of analyzing the relationships between loss weights and score feedback from the...

  62. [71]

    The output should in format 'Weights: L1:Perceptual:GAN=?:?:? (for example: L1:Perceptual:GAN=1.0:0.1:0.01)' 'Thoughts: (The reason you give this weights)'

  63. [72]

    Your ultimate goal is to help SR model achieve a lower score feedback (which indicates a better image quality)

  64. [73]

    You can randomly increase or decrease the weight of each loss by a small amount if historical weights are not enough for making a decision. LLM’s Feedback Based on the historical data and the goal of improving image quality (lower score), here's a proposed set of updated loss ...

  65. [5000]

    Model Training

    We apply the same principle to the other two tasks, as listed in Table 2. 8.2. Objective as Loss Function vs. LossAgent In the loss repository of LossAgent, we adopt commonly used reference-based loss functions for different tasks, such as L1 loss, LPIPS loss, and perceptual l...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.