{"id":"cb90e771-d434-4775-a179-8540341a7e78","arxiv_id":"2412.18525","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Explanatory instructions, detailed text descriptions of image-to-image transformations, are introduced with a 12M-pair dataset and show qualitative evidence of zero-shot generalization on unseen vision tasks.","lead":"This paper proposes replacing task names like \"image segmentation\" with detailed textual instructions that describe how an image should change. The authors build a 12-million-pair dataset of such instructions, fine-tune an autoregressive vision-language model, and report zero-shot ability on tasks not seen in training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control with terminological instructions is run: the claimed causal role of explanatory instructions for zero-shot generalization is untested.","rationale":"The reader's weakest_assumption captures the central epistemological gap. The paper's explanatory hypothesis is supported only by Sec. 4.2's qualitative demonstrations; no experiment varies the instruction format while holding data and initialization fixed. I agree this is the most load-bearing concern. The pretrained Lumina-mGPT initializer is itself a multimodal generalist, and Table 1 shows it can already produce nonzero task outputs from fixed terminological instructions, so initialization effects cannot be dismissed. Additionally, the paper's admitted limitations (Sec. 5, Appendix D)—that task-level zero-shot fails for inverse tasks such as Image-to-Depth and is 'unstable'—narrow the claim but do not rescue the missing control. The dataset itself (12M triplets) and the observed quantitative gains over the base model are real contributions, so the appropriate verdict remains CONDITIONAL: the central claim should be accepted only after the terminological-instruction control and the base-model-with-explanatory-instructions baseline are reported. This is purely an experimental-design issue; no stronger criticism is warranted.","tokens_in":36673,"tokens_out":5521,"duration_ms":50127,"concrete_test":"Run the missing control: initialize from Lumina-mGPT-7B-768 and fine-tune on the identical 1.5M-pair DECVT subset used in Sec. 4.2, but with terminological instructions ('HED-to-Image', 'Depth-to-Image', 'Image-to-Depth', etc.) replacing the explanatory texts for the seen tasks. Evaluate on the same held-out tasks (HED-to-Image, Canny-to-Image, Depth-to-Image) using both the provided explanatory instructions and fixed terminological prompts, with the same qualitative settings and, where possible, the Table 1 metrics (F1, FID, CLIP-S, SSIM). Also evaluate the untuned base model with the explanatory instructions from Figs. 6-8. If the terminological control matches the explanatory model's outputs, the causal claim fails; if the base model already produces comparable outputs, the fine-tuning contribution is near zero.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim, that explanatory instructions (not the underlying image-to-image fine-tuning) enable task-level zero-shot generalization, rests on the Sec. 4.2 experiment, which omits HED-to-Image, Depth-to-Image, and related tasks from training and shows qualitative successes (Figs. 6-8). The design never includes a control trained on the same 1.5M-pair subset with terminological instructions (e.g., 'depth map to image'). Without that control, the observed generalization could be attributed to (a) the pretrained Lumina-mGPT-7B-768 initialization, which already performs these tasks weakly—Table 1 shows Lumina-mGPT, without any DECVT fine-tuning, achieves F1=10.09 on Canny-to-Image and FID=69.49 on HED-to-Image with fixed terminological instructions—or (b) generic supervised fine-tuning on 1.5M diverse image-to-image pairs, which may teach the model to 'naturalize' edge/depth inputs regardless of how the task is verbalized. The paper's own limitations (Sec. 5, Appendix D) admit that task-level zero-shot is unstable and fails for Image-to-Depth and Image-to-Canny, so the boundary of the claimed phenomenon is unclear. Because the mechanism (explanatory language conveying task objectives) is the central intellectual contribution, the absence of an instruction-format control leaves the claim unverified. The quantitative evaluations in Tables 1-2 do not cover the exact Sec. 4.2 held-out-task setting, so the qualitative figures are the sole evidence for the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes \"Explanatory Instructions\": free-form linguistic descriptions of image-to-image transformations, as an alternative to short terminological task labels such as \"semantic segmentation\" or \"depth estimation.\" The authors construct DECVT, a dataset of roughly 12 million \"input image → explanatory instruction → output image\" triplets, and fine-tune the 7B autoregressive multimodal model Lumina-mGPT-7B on subsets of this data (Sections 3, 4.1). The central claim is that after fine-tuning, the model acquires (a) instruction-level zero-shot generalization to unseen paraphrases of instructions and (b) task-level zero-shot generalization to entire vision tasks omitted from training, such as HED-to-Image, Canny-to-Image, and Depth-to-Image (Section 4.2, Figures 6-8). The quantitative sections (Tables 1, 2, and Appendix B) report results on Canny-to-Image, HED-to-Image, inpainting/outpainting, dense prediction, and low-level restoration tasks, showing that the fine-tuned model is competitive with the vanilla Lumina-mGPT initialization but generally well below task-specific and vision-generalist baselines. The paper also includes explicit limitation statements in Section 5 and Appendix D, acknowledging that task-level zero-shot is unstable and fails for Image-to-Canny and Image-to-Depth.","tokens_in":36956,"tokens_out":5267,"duration_ms":48858,"significance":"If the causal claim were established—that a textual explanation of a transformation is what enables a vision-language model to execute a task it never trained on—this would be a meaningful step toward task-level zero-shot generalization in computer vision, analogous to instruction following in NLP. The paper has several strengths: the DECVT dataset is large and integrates many tasks and inverse directions; the method is simple and reproducible in principle; the authors do not claim state-of-the-art performance; and the limitation sections are unusually candid about instabilities and poor cases. The main weakness is evidential: the paper's headline phenomenon is supported almost entirely by three selected qualitative examples in Section 4.2, and the experimental design lacks the control condition needed to attribute the behavior to explanatory instructions rather than to the pretrained initialization or to generic image-to-image fine-tuning. The same GPT-4o pipeline is used to generate both training and evaluation instructions, which further weakens the \"zero-shot instruction\" interpretation.","major_comments":[{"comment":"The central causal claim—that explanatory instructions, rather than the underlying image-to-image fine-tuning, enable task-level zero-shot generalization—is not tested because the design lacks a control condition. The model is initialized from Lumina-mGPT-7B-768 (Section 3), and Table 1 shows that this initialization already performs the held-out tasks weakly: Canny-to-Image F1=10.09 and HED-to-Image FID=69.49 under fixed terminological instructions. A control model fine-tuned on the same 1.5M-pair subset with only terminological instructions (e.g., \"canny edge map to image\" or \"depth map to image\") is necessary to rule out that the observed generalization comes from (a) the pretrained model's latent ability to naturalize edge/depth inputs or (b) generic supervised fine-tuning on diverse image-to-image pairs, independent of instruction wording. Appendix D item 7 states that additional validation experiments could not be run due to resource constraints; this is exactly the missing experiment. Without it, the paper's headline claim remains unverified.","section":"Section 4.2 and Figures 6-8"},{"comment":"The claimed \"strong zero-shot generalization for unseen CV tasks\" rests on three qualitative examples (Figures 6-8), and the exact held-out setting of Section 4.2 is never evaluated quantitatively. Tables 1 and 2 use the full-DECVT model from Section 4.1, not the Section 4.2 model from which HED-to-Image, Canny-to-Image, and Depth-to-Image were excluded. Meanwhile, Section 5 states that task-level zero-shot fails for Image-to-Canny and Image-to-Depth, and Appendix D item 7 and the caption of Figure 48 describe the behavior as \"unstable\" and of \"relatively low\" success rate. The authors should provide quantitative metrics (e.g., F1, FID, SSIM, or a success-rate estimate) on a random sample of held-out inputs for the Section 4.2 model, and should report failure cases alongside successes; otherwise the strength of the generalization claim cannot be assessed.","section":"Section 4.2, Section 5, Tables 1-2"},{"comment":"Instruction-level zero-shot is evaluated with instructions generated by the same GPT-4o prompt pipeline used to produce the training instructions. The test instructions are therefore drawn from the same distribution as the training instructions, so \"unseen\" means token-level novelty within one generator distribution rather than distributional novelty. This circularity also affects the task-level instructions in Section 4.2, which are constructed with the same template. The authors should either test with out-of-distribution instruction sources (human-written instructions, a different generation model, or expert paraphrases) or soften the claim from \"zero-shot instruction generalization\" to \"robustness to paraphrases within the GPT-4o instruction distribution.\"","section":"Section 4.1, Appendix A.1, Figures 33-43"}],"minor_comments":[{"comment":"The heading contains a typo: \"Computer Vison Tasks\" should be \"Computer Vision Tasks.\"","section":"Section 2 heading"},{"comment":"The abstract and Section 1 state that code and dataset will be or have been released, but no repository link is included in the manuscript; please add the URL or note that it will be added in the camera-ready version.","section":"Abstract and Introduction"},{"comment":"The labels \"T. Z.-s.\" and \"I. Z.-s.\" are easy to misread: for example, HED-to-Image is marked \"I. Z.-s.\" while Canny-to-Image is marked \"T. Z.-s.,\" even though both are evaluated with unseen instructions. Clarify that Table 1 uses the full-DECVT model, and state explicitly which tasks are seen or unseen for that model.","section":"Tables 1 and 2"},{"comment":"The latent-space visualization claims that explanatory instructions form a continuum while terminological instructions cluster discretely, but it uses only one text encoder and no quantitative separation measure; please add a metric or a second encoder to support the claim.","section":"Appendix D item 2 and Figure 57"},{"comment":"The top-k value of 2048 for image generation is a free hyperparameter, and Appendix D notes that its effect is task-dependent; a brief ablation or a discussion of sensitivity would improve reproducibility.","section":"Section 3, Inference"},{"comment":"The hypothesis of an optimal instruction-complexity threshold is not tested; consider labeling it explicitly as a conjecture, since the current phrasing reads as a finding.","section":"Appendix D item 5"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is honest about its limitations, which is commendable, but the missing terminological-instruction control is a standard experimental expectation for a paper whose entire contribution is the causal role of explanatory wording. The claim is falsifiable and the proposed fixes are feasible: run a control fine-tuned on the same subset with fixed task names, add quantitative evaluation of the Section 4.2 held-out setting, and address the GPT-4o circularity. I recommend major revision rather than rejection because the dataset and framework have value, and the current text already contains the admission that task-level zero-shot is unstable, making the overclaim in the abstract correctable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a real contribution on the data side and a thought-provoking hypothesis. The central causal story — explanatory instructions, not the pretrained backbone or generic image-to-image supervised fine-tuning, produce task-level zero-shot generalization — is not currently supported by the experiments. A single control would fix it: fine-tune the same Lumina-mGPT on the same 1.5M-pair subset with terminological instructions and compare on the held-out tasks.\n\nWhat is genuinely new: DECVT, 12M bidirectional triplets with detailed GPT-4o transformation descriptions, is a substantial asset. The task-level zero-shot setup — omitting Canny/HED/Depth-to-Image from training and testing with textual explanations — is a fair probe, and the qualitative results in Figures 6–8 are non-trivial. The paper is also unusually honest: Section 5 and Appendix D admit instability, failures on Image-to-Canny and Image-to-Depth, and the large performance gap. I appreciate that.\n\nSoft spots, in order. (1) The missing control is the big one. Lumina-mGPT, with no DECVT fine-tuning, already produces Canny-to-Image outputs (F1=10.09) under a fixed instruction. After SFT on 1.5M image-to-image pairs, the model improves — but nothing isolates whether the improvement comes from the explanatory wording or simply from learning to naturalize edge and depth inputs. The abstract and Sec. 4.2 overstate what the design can show. (2) The quantitative tables undercut the proof-of-concept: on every listed metric the model lands below every strong baseline, and the Sec. 4.2 held-out-task setting has no quantitative evaluation at all. The headline claim rests entirely on qualitative examples. (3) The \"unseen instructions\" are generated by the same GPT-4o pipeline used for training, so instruction-level zero-shot is measured inside the training distribution of instruction style. The held-out tasks give partial independent grounding, but the generalization claim is weaker than stated. (4) Minor: the top-k sampling value and dataset mixing ratios are reasonable but not swept; a short analysis would help.\n\nWho this is for: people working on instruction-conditioned image generation, unified vision models, and dataset construction. The dataset and paradigm are worth engaging, and the weak spots are fixable with an explicit control and a quantitative evaluation of the task-level setting. I would send this to peer review, and ask for the control experiment as a condition of acceptance.","headline":"A useful new dataset and a plausible but unproven central claim; the missing terminological-instruction control is load-bearing.","tokens_in":37528,"tokens_out":2379,"would_cite":true,"duration_ms":21135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A vision-language model fine-tuned on plain-language descriptions of image-to-image transformations can carry out vision tasks it never trained on, including turning edge maps, depth maps, and outlines into photographs.","keywords":["explanatory instructions","zero-shot task generalization","vision-language model","autoregressive image generation","instruction following","image-to-image translation","unified vision tasks","controllable image generation"],"falsifier":"Fine-tune the same pretrained model on the same data with only terminological labels instead of explanatory instructions and test on the same held-out tasks; if that control also produces faithful edge-to-image and depth-to-image outputs, the explanatory format is not the cause.","tokens_in":36454,"feed_emoji":"🖼️","tokens_out":7577,"duration_ms":60387,"temperature":0.7,"pith_summary":"The paper argues that computer vision has not reached the zero-shot task generalization of NLP because vision tasks are defined by discrete labels such as 'semantic segmentation' rather than by what the task actually does to an image. It introduces Explanatory Instructions, detailed linguistic descriptions of the transformation from an input image to an output image, and builds a 12-million-triplet dataset of image, instruction, and output pairs. Fine-tuning a 7B autoregressive vision-language model on these triplets is claimed to produce instruction-level zero-shot ability on seen tasks and task-level zero-shot generalization on unseen tasks such as edge-to-image, canny-to-image, and depth-to-image. If this holds, a written description of a vision objective is enough to make a model execute a task it has never seen.","feed_headline":"Plain-language instructions unlock vision tasks a model never trained on","feed_subtitle":"Trained on 12M image-text-image triplets, a 7B model turns depth maps, edge maps, and outlines into photos on the spot.","key_machinery":"The load-bearing object is the Explanatory Instruction, defined as a detailed linguistic description of the transformation from an input image to an output image, written without task jargon and usually provided in both directions. Around it sits DECVT, a dataset of about 12 million 'image input to explanatory instruction to output' triplets assembled from terminological tasks such as restoration, detection, depth estimation, pose estimation, segmentation, and controllable generation, plus open-ended image-pair editing data, with instructions generated by a large multimodal model and by manual templates. The model is a vanilla token-based autoregressive vision-language model: a VQ-VAE image tokenizer converts images to discrete tokens, the tokens are concatenated with instruction text, and a decoder-only transformer trained with next-token prediction generates the output image, initialized from a pretrained multimodal generative model rather than trained from scratch. A supporting observation is that in a PCA-reduced feature space, explanatory instructions form continuous, overlapping distributions across tasks, whereas terminological instructions cluster discretely, which the paper identifies as the driver of zero-shot generalization.","core_discovery":"The central discovery claimed is that representing a vision task as an explanatory instruction, a natural-language description of the transformations between an input image and an output image, lets an autoregressive vision-language model generalize beyond the terminological task categories it was trained on. After supervised fine-tuning on the DECVT dataset, the model performs previously seen tasks from instructions it has not encountered, and, when entire tasks such as HED-to-Image, Canny-to-Image, and Depth-to-Image are held out of training, it can still produce plausible outputs when given a textual explanation of the desired transformation. The paper reports that generalization is asymmetric: the model succeeds on generation tasks that turn edges, maps, or masks into images, and on low-level tasks such as low-light enhancement and deblurring, but fails on the inverse tasks such as Image-to-Canny and Image-to-Depth, a failure the authors attribute to missing alignment between the image tokenizer and text modality in the pretrained initialization. On standard benchmarks the fine-tuned model still lags behind task-specific and specialist generalist models, so the claimed contribution is the new form of task specification and the zero-shot capability it enables, not state-of-the-art generation quality.","pith_inferences":["If explanatory instructions rather than pretrained abilities cause the reported behavior, the same dataset format could be applied to video, 3D, or audio-visual transformations, where task vocabularies are even less standardized.","The paper's own explanation of failures suggests a concrete test: aligning the image tokenizer with text-image data before fine-tuning should extend task-level zero-shot to inverse tasks such as image-to-depth and image-to-canny; this is an inference, not a paper claim.","A control experiment that fine-tunes the same initialization on the same data with terminological labels would isolate whether the explanatory format, rather than data scale and diversity, drives zero-shot generalization.","The latent-space continuity argument implies that generalization strength should track the overlap of instruction embeddings between training and test tasks, a prediction the paper does not test."],"forward_implications":["If the claim holds, a vision model's ability to do a new task depends on being told what the task does, not on having seen that task category in training.","The same model can be steered by arbitrary user-written descriptions of image transformations, making instruction following a general interface for image generation and editing.","Task-level zero-shot generalization is expected mainly for tasks whose output distribution the pretrained image tokenizer can already represent, such as photographs, and fails for outputs like depth maps and Canny edges that the tokenizer was not aligned to produce.","Combining multiple tasks in one fine-tuning run with explanatory instructions supports mixed and combined task objectives, rather than one instruction format per task.","Explanatory instructions also help on seen tasks for unseen categories or descriptions: a model can follow 'the creature on the leaf' even when the category name was never trained."],"supporting_citations":[{"why":"Supplies the pretrained Lumina-mGPT-7B initialization and the resolution-flexible image representation used in the architecture.","marker":"Liu et al., 2024"},{"why":"GPT-4o generates most of the explanatory instructions in DECVT and recaptions the MultiGen-20M evaluation images.","marker":"OpenAI, 2024"},{"why":"Chameleon provides the mixed-modal autoregressive token-sequence design that the fine-tuning framework follows.","marker":"Team, 2024"},{"why":"Motivates the goal of zero-shot task generalization through instruction-following in NLP.","marker":"Brown et al., 2020"},{"why":"Provides the MultiGen-20M validation split on which Canny-to-Image and HED-to-Image zero-shot results are measured.","marker":"Qin et al., 2023"},{"why":"Supplies ADE20K data for segmentation and dense prediction tasks in DECVT and in the evaluation set.","marker":"Zhou et al., 2017b"},{"why":"MagicBrush contributes instruction-based image editing pairs used as explanatory-based training data.","marker":"Zhang et al., 2023a"},{"why":"Depth Anything V2 generates the depth-image pairs that DECVT uses for depth estimation and depth-to-image tasks.","marker":"Yang et al., 2024"},{"why":"PixWizard is a comparison baseline representing open-instruction vision generalist models.","marker":"Lin et al., 2024"}],"fun_headline_variants":["Text-based task specs let vision models generalize to unseen tasks","Explanatory instructions let vision models handle never-seen tasks","Zero-shot vision tasks via natural language explanations","Explain a vision task in words, and the model generalizes on the spot"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the zero-shot behavior shown for unseen tasks comes from the explanatory-instruction fine-tuning and not from abilities already present in the pretrained model used to initialize training; the paper does not compare against a control fine-tuned on the same data with terminological-only instructions.","fun_headline_variants_meta":{"raw":{"variants":["Text-based task specs let vision models generalize to unseen tasks","Explanatory instructions let vision models handle never-seen tasks","Zero-shot vision tasks via natural language explanations","Explain a vision task in words, and the model generalizes on the spot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001547,"raw_usage":{"total_tokens":6222,"prompt_tokens":1016,"completion_tokens":5206,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":5137}},"tokens_in":632,"tokens_out":5206,"duration_ms":36735,"temperature":1.0,"reasoning_tokens":5137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:42:25.025826+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same pretrained model on the same data with only terminological labels instead of explanatory instructions and test on the same held-out tasks; if that control also produces faithful edge-to-image and depth-to-image outputs, the explanatory format is not the cause.","supporting_citations":[],"review_version":1}