{"id":"3b4225b0-2e3a-47bf-8e35-3d1964ae81c7","arxiv_id":"2505.06176","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Training a multimodal language model on visual puzzles about image operations lets it produce interpretable procedural retouching plans that preserve identity and rival commercial auto-enhance.","lead":"This paper trains a multimodal AI on custom visual puzzles to teach it how photo editing operations change an image, then uses that model to suggest concrete retouching steps. It matters because it offers an explainable, identity-preserving alternative to generative image editing and shows puzzles can transfer operation knowledge to large language models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Quantitative claim relies on best-of-five-expert scoring and lacks error bars; the reported margins may not survive a fixed single-expert or average-expert evaluation.","rationale":"I focused on the evaluation protocol rather than the optimality assumption because it is the most direct threat to the paper's headline empirical claim as stated in Section 5 and the abstract. The best-of-five-expert scoring, combined with the absence of error bars and significance tests, makes the reported margins over baselines and the 'comparable to Google Photos' claim difficult to assess. This is a concrete, falsifiable issue that can be settled by re-analysis of the existing outputs. The reader's weakest_assumption on the optimality of expert-edited images is also important, but it is more about the internal mechanism; even if the optimality assumption is imperfect, the model could still learn a useful mapping from source images to expert-style edits. The evaluation protocol, however, directly determines whether the paper's central claim of superiority is true. I agree with the reader's overall CONDITIONAL verdict: the method is plausible and the qualitative results and user study support it, but the quantitative evidence needs to be strengthened. My concern does not change the verdict, so I recommend UNCHANGED.","tokens_in":16827,"tokens_out":7351,"duration_ms":75064,"concrete_test":"Recompute Table 1 under three protocols: (a) score every method against expert A only, the expert whose edits trained MonetGPT and the regression baseline; (b) score every method against the average of all five Adobe5k experts; (c) report per-sample standard errors and paired bootstrap 95% confidence intervals for the differences between MonetGPT and the best open-source baseline (RSFNet) and between MonetGPT and Google Photos. If MonetGPT no longer leads on at least three of four metrics under (a) or (b), or if the differences are not statistically significant, the paper's central quantitative claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim (Section 5, Table 1) is that MonetGPT 'outperforms all open-source baselines on three of the four metrics and achieves performance comparable to the closed-source Google Photos.' However, the evaluation protocol states: 'For each sample and metric, we take the highest score achieved against any of these five experts.' This best-of-five oracle scoring gives every method five chances to match an expert edit, which can inflate scores for methods producing generic, style-averaged outputs. Because MonetGPT and the regression baseline are trained on a single expert (expert A) from PPR10K, the fair comparison is against expert A's edits, or at least against a pre-registered fixed expert. The current protocol leaves open the possibility that MonetGPT's advantage over RSFNet (SSIM 0.90 vs 0.88, LPIPS 0.07 vs 0.08, PSNR 23.75 vs 22.94) is within sampling variability; no error bars, confidence intervals, or significance tests are reported. Additionally, the 'comparable to Google Photos' claim is strained: MonetGPT is 2.11 dB lower in PSNR and 6.97 lower in histogram intersection, and whether this counts as comparable is not statistically tested. If the ranking changes under a single-expert or average-expert protocol, the headline quantitative contribution is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MonetGPT, a multimodal large language model (MLLM) fine-tuned on three synthetic visual puzzles to learn procedural image retouching. Puzzle A teaches operation identification, Puzzle B teaches aesthetic ordering and optimal-level identification, and Puzzle C teaches multi-stage retouching planning. Reasoning text is generated by Gemini 2.0 Flash, grounded on ground-truth operations and adjustment values. At inference, the model produces a staged plan that is executed with a custom Python library of 33 procedural operations. The authors evaluate on 400 held-out Adobe5k images against Exposure, Unpaired, RSFNet, InstructP2P, MGIE, Gemini CoT, Google Photos, and a direct-regression MLLM baseline, and report a user study with 15 novices and 10 experts. They claim that MonetGPT outperforms all open-source baselines on three of four metrics and is comparable to Google Photos.","tokens_in":17062,"tokens_out":7356,"duration_ms":75843,"significance":"If the quantitative claims hold, MonetGPT supplies a novel and practical recipe: a small open MLLM can be made operation-aware for procedural retouching using unpaired synthetic puzzles and limited expert data, while preserving identity, resolution, interpretability, and user override. This is a good fit for TOG, and the paper has concrete strengths: held-out Adobe5k generalization testing, a wide baseline set including a commercial auto-enhance tool, a user study with novice/expert splits, and a commitment to release code, data, and models. The central novelty—puzzle-based operation awareness plus staged procedural planning—is clearly articulated and worth publishing if the evaluation concerns are resolved. The main weaknesses are in the quantitative evaluation protocol (best-of-five expert scoring, no uncertainty measures) and the lack of an ablation that isolates the puzzle-solving contribution; both are fixable with additional experiments.","major_comments":[{"comment":"The headline quantitative claim is not supported by the reporting protocol. The paper states, 'For each sample and metric, we take the highest score achieved against any of these five experts.' Because MonetGPT and the MLLM Regression baseline are trained on expert A from PPR10K, the fair comparison is against expert A's edits or a fixed pre-registered expert, not the maximum over five experts. The best-of-five protocol can inflate scores for methods that produce generic edits by giving each method five chances to match some expert. The reported margins (SSIM 0.90 vs 0.88, LPIPS 0.07 vs 0.08, PSNR 23.75 vs 22.94 over RSFNet) may not persist under a single-expert or average-expert protocol. Please report per-expert scores, a fixed-expert comparison, and the exact aggregation rule for the histogram metric.","section":"Section 5, Quantitative Comparison"},{"comment":"No measure of uncertainty is reported for any quantitative result. With 400 test images, the differences between Ours and RSFNet on LPIPS (0.07 vs 0.08) and SSIM (0.90 vs 0.88) may be within sampling variability; no confidence intervals, standard deviations, or significance tests are provided. Similarly, the claim of being 'comparable to Google Photos' is not statistically tested, and the gaps of 2.11 dB in PSNR and 6.97 in histogram intersection could be substantial. Please report bootstrap confidence intervals or per-metric paired tests for all methods.","section":"Section 5, Table 1"},{"comment":"The comparison with the 'MLLM Regression' baseline does not isolate the effect of puzzle-solving. The regression baseline differs from MonetGPT in two major ways: it is trained on paired expert edits (source and target) rather than the synthetic unpaired puzzles, and it directly regresses parameters rather than being trained on the generated reasoning and staged plans. The observed gain could therefore come from the synthetic perturbation data, the reasoning supervision, or the staged planning, rather than from the puzzles themselves. A proper ablation should train an MLLM on the same synthetic unpaired data and reasoning while ablating the puzzle tasks, or train the regression baseline on the puzzle data, to support the paper's central claim that solving puzzles enhances retouching.","section":"Section 5, ablation ('MLLM Regression')"},{"comment":"The design of Puzzles B and C assumes that any perturbation of an expert-edited image IX is worse, and that operations are invertible; the paper states this explicitly. This assumption is load-bearing because, combined with the portrait-heavy PPR10K training set and single expert A, the model may learn to invert the synthetic perturbation process rather than acquire a general aesthetic prior. The Adobe5k evaluation tests transfer to other expert edits but does not separate these two hypotheses. I request a concrete diagnostic: evaluate the trained model on inputs whose edits were not generated by the same perturbation procedure, or compare the model's performance on images perturbed by unseen magnitudes or distributions, to demonstrate that the learned prior is not merely an inverse of the puzzle generator.","section":"Section 4.2 and 4.3, optimality assumption"}],"minor_comments":[{"comment":"The number of test images is inconsistent: the qualitative comparison paragraph says '15 source images' and 'a total of 200 questions', while the perceptual user study paragraph says the study was 'on 50 images from Adobe5k and Reddit'. Please clarify the actual number of images and how images map to questions.","section":"Section 5, Perceptual User Study"},{"comment":"The sentence 'except for histogram intersections, which consider all experts' is ambiguous: does 'consider all experts' mean averaging the histogram metric over five experts or taking the maximum? Specify the exact aggregation.","section":"Section 5, Quantitative Comparison"},{"comment":"The phrase 'regress precise adjustment values regress from high-level reasoning' contains a duplicated verb; please revise the sentence.","section":"Section 4.4"},{"comment":"The caption reads 'orange saturation + 2 5', which should presumably be '+25'; please also check for similar spacing or typographical issues in other figure text.","section":"Figure 1 caption"},{"comment":"The table lists no sample size or measures of spread; at minimum add n and standard deviations or confidence intervals to each metric.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely publishable after major revision. The evaluation-protocol and ablation issues are substantive but addressable with additional experiments; I do not see grounds for rejection, as the method itself is well motivated and clearly presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a solid, well-written paper with a genuinely new idea—teaching an MLLM to be operation-aware via three visual puzzles—and it shows a real effect over direct regression. But the main quantitative claim is under-built: best-of-five-expert scoring with no error bars, and the 'comparable to Google Photos' line is stronger than the numbers support.\n\nThe three-puzzle curriculum is the real contribution. Having the model identify an operation from before/after pairs, order perturbations of an expert edit, and plan a full staged retouch goes beyond simple regression or prompting. The comparison against the MLLM regression baseline on the same backbone is the right kind of control, and it suggests the curriculum is doing something. The qualitative results, style tagging, and staged interactive editing are thoughtful extras. I also appreciate the candor about assumptions in Section 4 and the limitations section at the end.\n\nWhere it gets shaky: the Table 1 protocol takes the best score against any of five Adobe5k experts. That's an oracle that flatters methods producing generic edits, and it makes the headline 'beats all open baselines' hard to interpret. No error bars or significance tests are reported, so the ~0.8 dB PSNR lead over MGIE could easily be sampling noise. Calling the result 'comparable to Google Photos' when you're 2.1 dB lower in PSNR and 7 points lower on histogram intersection is a stretch unless you can show equivalence statistically. More important than the scoring detail: there's no ablation isolating the puzzle structure from the distilled reasoning text. The regression baseline has neither, so you can't tell whether the gain comes from being fine-tuned on reasoning or from the puzzles themselves. That's the missing experiment that would nail the paper's central claim.\n\nThe optimality assumption—perturbing an expert edit always degrades it—is a reasonable training prior for a proof of concept, and the authors mention it. It does mean the aesthetic prior is tied to one expert's taste on portrait-heavy PPR10K, so the generalization to arbitrary photos is plausible but unproven.\n\nWho this is for: people building MLLM agents for image editing, and anyone looking for a recipe to make models operation-aware via synthetic puzzles. It deserves peer review, but the authors should redo the quantitative evaluation with a fixed or averaged expert, report variance, and add the ablation. If those changes land, the core claim will be on solid ground.","headline":"A genuinely new puzzle-based fine-tuning recipe for MLLMs, with promising results that would be much more convincing with fixed-expert scoring, error bars, and a proper ablation.","tokens_in":17624,"tokens_out":3448,"would_cite":true,"duration_ms":36019,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T45","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"A puzzle-solving training recipe lets a multimodal LLM retouch photos with explainable, identity-preserving edits, beating open-source baselines on three of four metrics.","keywords":["image retouching","multimodal large language model","procedural editing","visual puzzles","fine-tuning","interpretability","explainable AI","portrait photography"],"falsifier":"Take a set of photographs in one domain (e.g., landscapes) plus expert edits from multiple professional retouchers who disagree stylistically; if MonetGPT's retouching plans rated by independent expert judges are not preferred over the original images or over a simple histogram-matching baseline at least as often as the paper reports on portraits, the assumption that perturbation-of-expert-edit approximates aesthetic optimality fails for that domain. More directly, train the same model on Puzzles A and C but omit Puzzle B's ranking objective; if the resulting plans are not measurably worse on SSIM and LPIPS, then the aesthetic-ordering task is not the component carrying the benefit.","tokens_in":16607,"feed_emoji":"🎨","tokens_out":1842,"duration_ms":20240,"temperature":0.7,"pith_summary":"MonetGPT claims that a multimodal large language model can learn to retouch raw photographs procedurally if it is first trained to solve visual puzzles about what image-editing operations do, how much to apply, and how to plan multi-operation fixes. The paper argues that pretrained MLLMs lack an internal model of image operations, so direct fine-tuning on paired edits regresses toward trivial or overfit parameters. By generating synthetic puzzle datasets from expert-edited portraits and grounding a pretrained MLLM's reasoning on actual visual changes, MonetGPT becomes operation-aware and can critique, plan, and produce executable edit sequences with explanations. The authors evaluate on Adobe5k and report that their method outperforms all open-source baselines on SSIM, LPIPS, and Histogram intersection, and matches Google Photos' closed-source auto-enhance on SSIM. The practical payoff is a retouching assistant that preserves object identity and resolution, works on 16-bit high-resolution images, and lets users override any proposed step.","feed_headline":"Puzzle-trained MLLM beats open-source baselines at photo retouching","feed_subtitle":"MonetGPT turns visual puzzles into explainable, identity-preserving edit sequences that rival Google Photos on generalization metrics.","key_machinery":"The key machinery is the three-puzzle curriculum, each puzzle acting as a proxy loss function for the MLLM's image-operation awareness. Puzzle A teaches operation-to-effect mapping by supervised regression on (source, edited) pairs with reasoning text; Puzzle B teaches aesthetic optimality by ranking perturbed expert edits, exploiting the assumption that any sufficiently large perturbation of an expert edit yields a worse image; Puzzle C teaches sequential planning by inverting synthetic degradations of expert edits within three operation categories (lighting, color/temperature, color-specific). The training data is synthesized without using the expert-pair labels directly: a pretrained LLM (Gemini 2.0 Flash) generates grounded reasoning for each puzzle, and the fine-tuned MLLM (Qwen-VL-7B) learns to reproduce that reasoning, which then mediates parameter regression at inference. The procedural library of 33 operations, each with a single master parameter on a perceptually linear scale from -100 to +100, keeps the parameter space small enough for the MLLM to regress values the library can execute directly as JSON.","core_discovery":"The paper's central claim is that solving carefully designed visual puzzles teaches an MLLM the 'strategic knowledge' of image retouching: identifying visual issues, deciding which procedural operations to apply, and regressing precise parameter values. Three puzzles build this knowledge sequentially: Puzzle A teaches what each operation does by asking the model to identify an operation and its adjustment value given a before-and-after pair; Puzzle B teaches aesthetic judgment by asking the model to order perturbed variants of an expert-edited image and identify the optimal version; Puzzle C teaches planning by asking the model to reconstruct an expert edit from a degraded source, stage by stage, producing an <Adjustment, Issue, Solution> reasoning triplet per operation. The authors show that the resulting fine-tuned model, built on Qwen-VL-7B with DoRA adapters, generalizes to unpaired Adobe5k test images and responds to subtle input variation, while a direct regression baseline overfits and predicts nearly identical values. Reasoning acts as a pathway: the model first writes a plan, then regresses numerical parameters conditioned on that plan, with the staged pipeline feeding each stage's edited image back into the next stage.","pith_inferences":["A testable extension would be to evaluate whether the puzzle curriculum transfers to non-portrait domains, since the paper trains on PPR10K (mostly portraits) and the aesthetic prior may not generalize to landscapes or macro photography.","The paper's implicit claim that 'expert edit equals optimum' could be probed directly by measuring how model performance degrades when the expert edit is itself stylistically unusual or when different experts disagree.","A more surgical falsification would compare MonetGPT against a variant trained on the same synthetic perturbations but without the reasoning-text grounding, to isolate whether the benefit comes from the puzzles or from the reasoning supervision.","The staged pipeline's feedback loop (feeding the edited image into the next stage) resembles a model-predictive or closed-loop control scheme; it might be worth testing whether a single forward pass with all parameters predicted at once performs comparably, which would indicate the benefit of stage-wise visual feedback."],"forward_implications":["If MonetGPT's recipe is correct, MLLM-based photo editing can be non-destructive, resolution-preserving, and explainable by design, providing an alternative to generative editing that can alter identity.","The puzzle-based fine-tuning strategy may transfer to other procedural graphics tasks where an MLLM must learn to reason about a constrained tool library, such as video color grading or print prepress.","Operation-aware MLLMs could serve as interactive copilots where users edit a plan at any stage and the model re-derives downstream parameters, an interaction demonstrated by the paper's autoregressive editing.","Because the system trains only on synthetically perturbed expert edits, it can be adapted to new operation libraries or new expert styles without requiring paired human annotations at scale.","The quantitative comparison suggests that closed-source auto-enhance systems are not necessarily superior to an open, fine-tuned MLLM pipeline, making it plausible that such an assistant can be deployed fully locally on consumer hardware such as an RTX 4090."],"supporting_citations":[{"why":"Supplies the PPR10K portrait retouching dataset of expert-edited images used to synthesize all puzzle training data.","marker":"Liang et al. 2021"},{"why":"Supplies the Adobe5k input/output image pairs used for generalization testing against five experts' edits.","marker":"Bychkovsky et al. 2011"},{"why":"Defines the Exposure reinforcement-learning baseline for procedural retouching and the histogram-intersection evaluation metric.","marker":"Hu et al. 2018"},{"why":"Provides Qwen-VL-7B-Instruct, the base MLLM that is fine-tuned with DoRA adapters for MonetGPT.","marker":"Wang et al. 2024a"},{"why":"Supplies the DoRA weight-decomposed low-rank adaptation method used for parameter-efficient fine-tuning.","marker":"Liu et al. 2024"},{"why":"Supplies Gemini 2.0 Flash Experimental, the pretrained MLLM used to generate the grounded reasoning texts for all three puzzles and as the Gemini+library chain-of-thought baseline.","marker":"Team 2024"},{"why":"Supplies InstructPix2Pix as a generative-editing baseline representing identity-loss-prone text-to-image editing.","marker":"Brooks et al. 2023"},{"why":"Supplies MGIE as a baseline where an MLLM guides a diffusion-based editor, representing the generative alternative to procedural editing.","marker":"Fu et al. 2024"}],"fun_headline_variants":["Puzzle-trained MLLM masters explainable photo retouching","MonetGPT: solving visual puzzles sharpens MLLM retouching","Visual puzzles teach MLLMs to plan photo edits step-by-step","Puzzle training beats direct regression for MLLM retouching","MonetGPT: puzzle-solving MLLM preserves identity in retouching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that any sufficiently large perturbation of an expert-edited image produces a worse image, so the expert edit is the unique 'optimal' image, and the synthesized puzzles therefore teach genuine aesthetic judgment rather than just inversion of synthetic distortions.","fun_headline_variants_meta":{"raw":{"variants":["Puzzle-trained MLLM masters explainable photo retouching","MonetGPT: solving visual puzzles sharpens MLLM retouching","Visual puzzles teach MLLMs to plan photo edits step-by-step","Puzzle training beats direct regression for MLLM retouching","MonetGPT: puzzle-solving MLLM preserves identity in retouching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1535,"prompt_tokens":1087,"completion_tokens":448,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":703,"completion_tokens_details":{"reasoning_tokens":353}},"tokens_in":703,"tokens_out":448,"duration_ms":4803,"temperature":1.0,"reasoning_tokens":353,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:40.576081+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of photographs in one domain (e.g., landscapes) plus expert edits from multiple professional retouchers who disagree stylistically; if MonetGPT's retouching plans rated by independent expert judges are not preferred over the original images or over a simple histogram-matching baseline at least as often as the paper reports on portraits, the assumption that perturbation-of-expert-edit approximates aesthetic optimality fails for that domain. More directly, train the same model on Puzzles A and C but omit Puzzle B's ranking objective; if the resulting plans are not measurably worse on SSIM and LPIPS, then the aesthetic-ordering task is not the component carrying the benefit.","supporting_citations":[],"review_version":1}