{"id":"4ea43f0c-b99e-4bcf-8467-9d5f73383eda","arxiv_id":"2502.08667","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A survey and benchmark that groups unpaired image-to-image translation tasks into fully, partially, and non-content preserving categories, and evaluates six models on a vehicle-focused Sim2Real benchmark.","lead":"This paper reviews unpaired image-to-image translation methods, organizing them by neural architecture and by how much of the source image content is preserved. It also presents a simulation-to-real benchmark as a way to judge content-preserving translation models in a practical setting.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 6 ranking claims rest on FID/KID values copied from different papers under different training protocols, with no shared split or metric implementation; this undermines the main empirical support for the taxonomy's usefulness.","rationale":"The reader identifies the use of FID/KID values copied from original papers as the weakest assumption. I agree: this is the single most load-bearing concern because the paper's proposed taxonomy is useful only if it helps choose among models, and the tables of copied metric values are the only large-scale quantitative basis for that choice. The concern is concrete rather than stylistic: Section 6 states that results are taken from the models' own articles, and the tables contain empty cells and no error bars, so there is no reason to believe the numbers were produced under comparable conditions. A lower copied FID can easily be an artifact of a different resolution, split, evaluation code, or training compute. The paper's own Sim2Real benchmark is the natural place to run a fair comparison, but the benchmark section does not specify a held-out test split, does not release code or data, and reports no variance, so it does not currently fix the comparability problem. These issues justify the reader's CONDITIONAL verdict: the taxonomy and benchmark are plausible and potentially reusable, but the evaluation supporting them needs reworking. I do not see a reason to reject the paper outright, because the taxonomy itself is a conceptual contribution that does not strictly require the copied tables, and the benchmark could become usable after a fair, reproducible evaluation is added. The verdict should remain CONDITIONAL, with the condition being a controlled, released evaluation that either reproduces the rankings or replaces them with corrected ones.","tokens_in":39197,"tokens_out":4501,"duration_ms":43203,"concrete_test":"Run a controlled re-evaluation on one Fully Content preserving task (GTA→Cityscape) and one Partially Content preserving task (Horse→Zebra) under a single protocol: identical train/validation split, image size, FID/KID implementation (for example pytorch-fid with the same InceptionV3 features and the same number of generated samples), and comparable training budget, for at least CycleGAN, CUT, DRIT++, UNSB, and DCLGAN. Compare the resulting rankings with the values in Tables 4–7. If the ranking changes, Section 6 cannot support model-selection conclusions from copied metrics; if the ranking is stable, the comparability concern is resolved for those tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that organizing tasks by content-preservation degree helps a user select a suitable I2I model. The empirical evidence for that claim is Section 6, where ranking statements such as 'DRIT++ achieve the lowest FID' on GTA→Cityscape and 'CUT performs better than other methods' on GTA→Cityscape by KID are made from tables of FID and KID values. Section 6 explicitly says: 'These results are taken from the models' own articles.' For such cross-model rankings to be meaningful, the numbers must be commensurable: the same dataset split, same image resolution and preprocessing, same FID/KID implementation, same number of evaluated samples, and comparable training budgets. None of these conditions is established or corrected in the paper. The tables contain sparse cells, different tasks per model, and no error bars or evaluation-set sizes. A lower copied FID may therefore reflect a favorable experimental setup rather than a genuinely better model. Because these rankings are the principal quantitative evidence that the FCP/PCP/NCP taxonomy supports model selection, the central empirical claim is not yet demonstrated. The paper's own Sim2Real benchmark is a controlled alternative, but it also lacks a stated train/test split, a released data/code artifact, and error bars, so it does not currently repair this weakness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a review of unpaired image-to-image translation organized around the degree to which the source image content must be preserved. It introduces a three-way task taxonomy (Fully Content Preserving, Partially Content Preserving, Non-Content Preserving), surveys around 70 models grouped by architecture, summarizes datasets and evaluation metrics, compiles FID and KID results from the original papers, and presents a new Sim2Real benchmark for content-preserving simulation-to-real translation. The paper concludes that the degree of content preservation should be considered when selecting an I2I model for a given application.","tokens_in":39402,"tokens_out":3435,"duration_ms":33583,"significance":"The proposed FCP/PCP/NCP taxonomy addresses a real and practically useful distinction that is often implicit in I2I papers, and the broad survey of models, datasets, and metrics provides a useful entry point for practitioners. The Sim2Real benchmark, combining VisDrone, MIO, and VeRi data with a simulated vehicle-image domain, is a potentially reusable evaluation resource, and the paper is explicit about the hyperparameters and metrics used in its own experiments. However, the central empirical claim that the taxonomy supports model selection rests on Section 6, where FID/KID values are copied from heterogeneous original papers without a shared evaluation protocol, and on the Sim2Real section, which has internal inconsistencies (the stated eight models are not the reported six) and no released data, code, or error bars. Because these issues affect the main practical conclusion, the paper needs revision before the claim can be accepted. I see no circular reasoning or fabricated entities; the weaknesses are methodological and presentation-related rather than conceptual.","major_comments":[{"comment":"The cross-model rankings in Section 6 are not commensurable because the FID and KID values are taken from the models' own articles under different training protocols, data splits, resolutions, and metric implementations, as the text itself states: 'These results are taken from the models' own articles.' Consequently, statements such as 'DRIT++ achieve the lowest FID' (Table 4, GTA-to-Cityscape) and 'CUT performs better than other methods' (Table 5, GTA-to-Cityscape) are not supported as general claims about model quality. The authors should either re-run all compared methods on a common split with a common metric implementation, or explicitly rephrase these statements as reports of the original papers' numbers rather than as findings of this review.","section":"§6, Tables 4–5"},{"comment":"The evaluation section states that eight models were selected (CycleGAN, DRIT, GcGAN, StyleFlow, SRUNIT, VSAIT, UNSB), but the text lists only seven model names and Table 12 reports results for only six models, with no DRIT row and no explanation for its absence. This discrepancy makes the benchmark evaluation incomplete and raises the question of whether the missing model's results were omitted selectively. The authors should specify the exact set of models, justify any exclusions, and either add the missing results or remove the claim of eight models.","section":"§7.2, Table 12"},{"comment":"The Sim2Real benchmark lacks a stated train/test split, repeated runs with seeds, error bars, and a released data/code artifact. For example, in §7.2.1 the authors say CycleGAN was 'trained and tested on Sim2Real dataset' for 20 epochs, but no held-out evaluation set is described, so the reported FID, IS, NDB, JSD, and LPIPS numbers cannot be interpreted as generalization measures. The conclusion that 'changing hyperparameter λ from 10 to 5 do not increase the performance' is therefore not verifiable. The authors should describe the exact split, report multiple seeds with variance, and release the dataset construction code and evaluation scripts.","section":"§7.1, §7.2"},{"comment":"The FCP/PCP/NCP assignment is not operationalized: the paper gives qualitative descriptions of the three categories but no quantitative or procedural rule by which a reader can assign a new task to a category. Some assignments in Table 3 also appear inconsistent with the definitions, for instance Label2Cityscape is listed as PCP while Keypoint2Photo is listed as NCP, although both involve structured source information being transformed into a target image. Without an explicit annotation protocol or inter-annotator agreement, the taxonomy cannot be applied reproducibly by future users, which weakens the paper's central contribution.","section":"§4, Table 3"}],"minor_comments":[{"comment":"The sentence listing selected models ends with 'an UNSB [38]'; this should read 'and UNSB [38]'.","section":"§7.2"},{"comment":"The column header 'V an' contains a stray space and should be 'Van'.","section":"Table 11"},{"comment":"The text contains several typos and grammatical errors, for example 'benckmark' in §7.2.1, 'metircs' in the Table 12 caption, and 'do not increase the performance results' in §7.2.1; a careful proofread is needed.","section":"Throughout"},{"comment":"The caption says the first row is simulation images and the second row is real images, but the text in §7.1 says 'images of 5 classes in the VeRi data set have been randomly sampled and demonstrated'; the figure and text should be reconciled to avoid confusion about which domain is shown.","section":"Figure 10 caption and text"},{"comment":"Several KID values in Table 9 are on the order of 50–100, which is far outside the typical range for KID on standard benchmarks (usually below 0.1); the authors should state whether these values are scaled (e.g., multiplied by 100) or whether a different implementation was used.","section":"Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful organizing idea and a substantial survey, but the current empirical support is not yet at the level needed for the central claim. The Section 6 ranking tables and the Section 7 benchmark both need methodological fixes: a common evaluation protocol or explicit recasting of claims, a consistent and complete model set, a described train/test split, multiple seeds, and ideally released code/data. These are fixable within the manuscript's scope, so major revision rather than rejection seems appropriate. I also note that the paper's fit with Expert Systems with Applications depends on the authors strengthening the benchmark's reproducibility; without that, the contribution is more of a survey than an evaluated system."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe new thing here is a three-way split of unpaired I2I tasks by how much content must survive translation – Fully/Partially/Non-content preserving – plus a new Sim2Real benchmark that concatenates Visdrone, MIO, VeRi and a synthetic vehicle set and evaluates six or seven models on it. The taxonomy is a reasonable organizing device, and the benchmark, limited as it is, is a genuine attempt to compare content-preserving methods under one controlled setup. The paper also compiles a lot of useful material: roughly seventy models, ten metrics, and thirty tasks, with FID/KID tables per category. That alone makes it a possible reference for practitioners who want a map of the area.\n\nThe soft spots are real but not disqualifying. Section 6's rankings rest on FID and KID numbers copied from the models' own papers, with no shared splits, resolution, metric implementation, or training budget. The paper says this explicitly, but then draws comparative conclusions ('DRIT++ achieve the lowest FID', 'CUT performs better') as if the numbers were commensurable. That is the main empirical support for the claim that the taxonomy helps model selection, and it does not currently hold up. The benchmark section is better because it runs models under their own protocol, but Table 12 reports single runs with no error bars or seeds, and the paper says 8 models were selected, lists 7, and tabulates 6 – DRIT is missing. The benchmark data and code are not released, which limits immediate usefulness. Also, the text in Section 3.1.3 says 'In this paper, NAIN is proposed', which is wrong for a review; that should be attributed to the cited SCS-UIT work.\n\nThe taxonomy labels are applied post hoc, but that's a labeling choice, not circular reasoning. I don't see a load-bearing flaw in the central idea – it is a plausible descriptive framework. What it needs is honest calibration: treat Section 6 as a literature survey, not a benchmark, and rebuild the Sim2Real benchmark as the controlled comparison, with released artifacts and multiple seeds.\n\nWho is this for? Practitioners and newcomers to unpaired I2I who want a structured overview and a quick testbed for Sim2Real vehicle translation. It deserves a serious referee, but only if the referees push for the fixes above. I'd recommend sending it to review with major-revision expectations.","headline":"A useful review taxonomy and a modest Sim2Real benchmark, but the cross-paper FID/KID rankings in Section 6 are not commensurable and need to be reframed.","tokens_in":39981,"tokens_out":2056,"would_cite":false,"duration_ms":18499,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that unpaired image-to-image translation tasks should be chosen by how much source content must survive, and that this choice is best made before picking a model.","keywords":["Image-to-image translation","Unpaired translation","Content preservation","Sim-to-real benchmark","Generative adversarial networks","Diffusion models","FID KID evaluation"],"falsifier":"Re-run a handful of the compared models end-to-end on the same train and validation split of the Sim2Real benchmark and of one Fully Content preserving task such as GTA to Cityscape, computing FID and KID with a single script. If the best model by the copied numbers is not the best model in the unified run, the paper's comparative conclusions collapse.","tokens_in":38984,"feed_emoji":"🖼️","tokens_out":3070,"duration_ms":28866,"temperature":0.7,"pith_summary":"The paper argues that unpaired image-to-image translation tasks differ in how much of the source image must survive, and that this difference should drive model choice. It divides tasks into Fully, Partially, and Non-Content preserving, sorts roughly seventy models by architecture, and collects FID and KID numbers across thirty tasks. It also builds a new Sim2Real benchmark combining simulated vehicle images with real images from three datasets and evaluates eight models on it. The practical payoff would be a checklist: decide how much content must be preserved, then pick a model family known to respect that constraint.","feed_headline":"Content preservation splits image-to-image tasks into three buckets","feed_subtitle":"A review of ~70 models and a new vehicle Sim2Real benchmark helps pick translators when source detail must survive.","key_machinery":"The carrying object is the three-way content-preservation categorization: Fully Content preserving, Partially Content preserving, and Non-Content preserving, defined by how much of the source image must be reproduced in the output. Around this axis the paper organizes an architectural taxonomy of GAN-based, VAE-based, diffusion, flow-based, and transformer models, and a new Sim2Real benchmark built from simulated vehicle images and real images from MIO, Visdrone, and VeRi, with FID, KID, IS, NDB, JSD, and LPIPS as evaluation tools. The taxonomy supplies the structure of the review, and the benchmark supplies a reusable testbed for the content-preserving label.","core_discovery":"On its own terms, the central claim is that content preservation is not a side effect but a design axis. The paper shows that tasks and datasets can be sorted into Fully Content preserving, Partially Content preserving, and Non-Content preserving, and that different models perform best in different categories. For example, DRIT++ achieves the lowest FID for GTA to Cityscape, CUT achieves the best KID on the same task, UNSB achieves the lowest FID for Horse to Zebra, and DCLGAN performs best on Label to Cityscapes. The paper also introduces a Sim2Real benchmark and reports evaluations of CycleGAN, StyleFlow, GcGAN, VSAIT, SRUNIT, and UNSB on it, showing that hyperparameter choices such as batch size and loss weights visibly change content preservation and image quality.","pith_inferences":["A testable extension would measure content preservation directly, for instance by comparing segmentation or keypoint consistency between source and translated images, rather than relying only on FID and KID.","The taxonomy could plausibly extend to newer diffusion and transformer models that were not fully benchmarked here, since those architectures also face the same content-versus-style tradeoff.","The benchmark could be reused for partially content-preserving tasks by adding tasks that deliberately change object class or layout, which would stress-test where the FCP/PCP boundary actually lies."],"forward_implications":["If the categorization is right, practitioners can narrow a model search by first deciding whether a task is fully, partially, or non-content preserving.","The Sim2Real benchmark gives later researchers a shared vehicle-focused dataset on which content-preserving methods can be compared directly.","The paper's tables show that no single model dominates all three categories, so reporting results per content-preservation category could become a useful convention.","The hyperparameter studies on CycleGAN, GcGAN, and StyleFlow suggest that content preservation in a given model is tunable through batch size and loss weights, not fixed by architecture alone."],"supporting_citations":[{"why":"CycleGAN is the baseline unpaired translation method used throughout the review and one of the models evaluated on the new Sim2Real benchmark.","marker":"[1]"},{"why":"CUT supplies the contrastive learning baseline and produces the best KID value the paper cites for the GTA to Cityscape task.","marker":"[58]"},{"why":"Art2Real defines several Fully Content preserving painting-to-real tasks and supplies results the paper tabulates for those tasks.","marker":"[3]"},{"why":"DRIT and DRIT++ provide the disentangled representation approach and the lowest FID values the paper reports for GTA to Cityscape and GTA to KITTI.","marker":"[12]"},{"why":"StyleFlow is the flow-based content-preserving model the paper evaluates on the Sim2Real benchmark.","marker":"[18]"},{"why":"GcGAN contributes the geometry-consistency constraint and is one of the models benchmarked on Sim2Real.","marker":"[19]"},{"why":"VSAIT represents the vector symbolic architecture approach and is benchmarked on the Sim2Real data.","marker":"[17]"},{"why":"UNSB supplies the Schrödinger bridge method, reported as the best FID on Horse to Zebra and benchmarked on Sim2Real.","marker":"[38]"}],"fun_headline_variants":["Review sorts image translation by content preservation","Three content-preservation buckets cover 70 image models","Content preservation drives image-translation model choice","Sim2Real benchmark pairs with content-preservation review","Image translation reviewed via content-preservation axis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation tables compare FID and KID scores copied from each model's original paper, so the comparison assumes those numbers were produced under fairly comparable training conditions; if they were not, the rankings and the conclusions drawn from them are not reliable.","fun_headline_variants_meta":{"raw":{"variants":["Review sorts image translation by content preservation","Three content-preservation buckets cover 70 image models","Content preservation drives image-translation model choice","Sim2Real benchmark pairs with content-preservation review","Image translation reviewed via content-preservation axis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000217,"raw_usage":{"total_tokens":1460,"prompt_tokens":996,"completion_tokens":464,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":612,"tokens_out":464,"duration_ms":5381,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:23:28.017699+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run a handful of the compared models end-to-end on the same train and validation split of the Sim2Real benchmark and of one Fully Content preserving task such as GTA to Cityscape, computing FID and KID with a single script. If the best model by the copied numbers is not the best model in the unified run, the paper's comparative conclusions collapse.","supporting_citations":[{"cited_title":"StyleFlow For Content-Fixed Image to Image Translation","cited_arxiv_id":"2207.01909","evidence_quote":"StyleFlow is the flow-based content-preserving model the paper evaluates on the Sim2Real benchmark."}],"review_version":1}