{"id":"162ea8a5-d85f-4b48-b5c6-02169d21a012","arxiv_id":"2412.12683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new bronze ware dating benchmark with seven out-of-distribution image types shows vision-language models detect unusual inputs best, while near-distribution domain shifts remain the hardest case.","lead":"The authors built ShiftedBronzes, a new image benchmark for dating ancient Chinese bronze vessels and for testing out-of-distribution detection, adding Gui vessels plus sketches, rubbings, generated images, and modern containers as unusual inputs. They compared six classification and eighteen detection methods and found that vision-language models handle the unusual inputs best, while domain-specific images that look close to real bronzes remain hard.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transferred OOD categories are generated from the same 2,861 bronze test images used as ID evaluation, confounding the reported small-shift difficulty ranking.","rationale":"The reader's weakest assumption concerned realism of the ZeST-generated images and reliability of expert annotations. My concern is related but sharper: even if the generated images were visually realistic, the construction procedure uses the same bronze test images as both the ID evaluation set and the source material for the transferred OOD categories. This creates a direct statistical dependence between ID and OOD evaluation sets, violating the standard independence expected for an OOD benchmark. The paper's headline finding about small distribution shifts being harder is based on these transferred categories, so this design choice is load-bearing. The proposed control experiment would settle whether the finding survives when OOD generation uses held-out bronze images. This does not change the overall verdict from the reader's CONDITIONAL, but it adds a specific, testable condition that should be satisfied before the benchmark is relied upon. The paper's other issues (no dataset/code release, placeholder references, undisclosed few-shot configuration) further support keeping the verdict conditional.","tokens_in":18565,"tokens_out":6630,"duration_ms":61395,"concrete_test":"Hold out a new set of bronze images (e.g., 2,861) that appear in neither the ID training nor ID test splits. Generate fresh transferred-container and transferred-bronze sets from these held-out images using the same ZeST procedure and container pool. Re-run the 18 OOD detectors under identical settings, with the original ID test set. If the average FPR@95/AUROC on the two regenerated OOD categories changes by more than a few points, or if the ordering between hard and easy OOD data shifts, the original small-shift finding is confounded by using the ID test set as source. Also disclose the few-shot configuration selected for VLM methods in Section 4.4.1 to ensure the comparison is reproducible.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"In Section 3.3, the two \"transferred\" OOD categories are created by applying ZeST to the 2,861 bronze images from the test set: 51,023 transferred-container images (bronze material onto containers) and 51,023 transferred-bronze images (container material onto bronze test images). Section 4.1 defines the test set as the ID evaluation split for OOD detection. Thus every transferred OOD sample shares exact material/texture content with at least one ID evaluation image, creating a dependency absent from the other OOD categories. The hard-OOD finding in Section 4.4.2 — that current methods struggle more with small distribution shifts — may therefore reflect this artificial ID-OOD overlap rather than a genuine property of real bronze-ware dating scenarios. The paper provides no control using OOD data generated from bronze images held out from the ID evaluation split, and no human or algorithmic validation that the ZeST outputs are realistic enough to represent real distribution shifts. Without such a control, the benchmark's central claim to model shifts \"commonly encountered in bronze ware dating scenarios\" is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ShiftedBronzes, a benchmark for fine-grained bronze-ware dating under open-world distribution shifts. The benchmark consists of two in-distribution (ID) bronze-ware categories (Ding and Gui) with expert era annotations, and seven OOD data types: sketches, rubbings, two types of ZeST-generated material-transferred images, and container images from ImageNet-21K. The authors benchmark six FGVC methods for the dating task and eighteen OOD detection methods (post-hoc, VLM-based, and generation-based) on ShiftedBronzes and five general OOD datasets. The main empirical findings are that VLM-based methods, especially ID-like, outperform other categories; that detection is harder on domain-specific OOD data with small distribution shifts (the 'hard OOD' split) than on general OOD data; and that diffusion-based methods are comparatively strong on sketch/rubbing data. The paper also analyzes the effect of few-shot training size on VLM methods.","tokens_in":18799,"tokens_out":4359,"duration_ms":38967,"significance":"If the benchmark is valid, it fills a real gap: there is no existing OOD benchmark for archaeological bronze-ware dating, and the dataset includes expert-annotated fine-grained era labels and a variety of OOD types that plausibly occur in archaeological practice. The paper benchmarks a broad set of eighteen OOD methods, and the finding that VLM-based methods are more robust on this specialized domain, as well as the sensitivity analysis of few-shot prompt learning, are useful to the OOD detection community. The dataset and code are promised, and the scale (over 50k images per transferred/container category) is substantial. At the same time, the benchmark's validity depends on the realism of the ZeST-generated images and on the absence of leakage between the generated OOD data and the ID evaluation set; these issues are not adequately addressed in the current manuscript.","major_comments":[{"comment":"The transferred container and transferred bronze OOD categories are generated from the 2,861 bronze images that constitute the ID test set defined in Section 4.1. Consequently, every transferred OOD image shares exact material and texture content with at least one ID evaluation image. The small-shift difficulty finding in Section 4.4.2(6) may therefore be an artifact of this artificial content overlap rather than a genuine property of real bronze-ware distribution shifts. This is a load-bearing issue for the benchmark's central claim to model shifts 'commonly encountered in bronze ware dating scenarios.' Please add a control experiment in which transferred OOD images are generated from bronze images held out from the ID evaluation split (e.g., from the training/validation portions), and re-run the key comparisons; alternatively, report a sensitivity analysis that quantifies the effect of this overlap.","section":"Section 3.3 and Section 4.1"},{"comment":"For the VLM-based methods ID-like and LoCoOp, the text states that the authors selected the best-performing configuration across varying training sample sizes, but the chosen shot count is not reported in Table 3, Figure 7, or the text. Since Figure 7 shows substantial performance variation with the number of shots (e.g., best OOD performance at 1-shot, with declines as shots increase), the reported rankings are not reproducible and the comparison may be cherry-picked. Please report the exact few-shot configuration used for each method and dataset in Table 3, and provide results for all shot counts in an appendix. In addition, the paper reports no error bars or repeated runs for any of the Table 3 results, which makes it difficult to assess whether the reported differences are significant.","section":"Section 4.4.1"},{"comment":"The split of OOD data into 'hard' (the seven ShiftedBronzes OOD types) and 'easy' (the five general OOD datasets) is based on expert judgment and a t-SNE visualization of 200 random images per type (Figure 2c). This split is load-bearing for the paper's central finding (Section 4.4.2(6)) that current methods struggle more with small distribution shifts. The t-SNE plot is only illustrative and does not provide a quantitative measure of distribution shift. Please add quantitative shift estimates, such as FID, MMD, or class-conditional feature distances computed with the same backbone(s), and show that the hard/easy ordering is robust to the choice of feature extractor.","section":"Section 4.1 and Section 3.4"},{"comment":"The ZeST-generated OOD images are never validated for realism or for their correspondence to distribution shifts actually encountered in bronze-ware dating. If the generated images contain material-transfer artifacts (e.g., unrealistic texture bleeding, lighting inconsistencies, or structural distortions), the benchmark conclusions may not transfer to real archaeological workflows. At minimum, the paper should include a human evaluation (e.g., expert raters judging whether the generated images look like real bronze artifacts or crafts) or a quantitative artifact analysis (e.g., comparison against real sketches/rubbings, or detection of obvious generation artifacts). Without this validation, the benchmark's key assumption that the generated OOD data are realistic proxies remains unsupported.","section":"Section 3.3"}],"minor_comments":[{"comment":"The abstract contains a duplicated phrase: 'for bronze ware dating and the and the development of OOD detection methods.'","section":"Abstract"},{"comment":"The table header says 'AUPOC' in multiple places; this should be 'AUROC.'","section":"Table 3"},{"comment":"The text refers to 'Table 1' and 'Figure 4' when presenting top-2 results; these should be 'Table 3' and likely 'Figure 5' or a different figure number.","section":"Section 4.4.2"},{"comment":"References [1]-[4] and [49]-[50] appear to be leftover placeholder entries (e.g., 'Alpher' and 'Frobnication') and should be removed or replaced with actual citations before any public release.","section":"References"},{"comment":"The statement that DiffGuard performs 'just below ID-like' on sketch/rubbing data is vague; please report the exact FPR@95 and AUROC values (37.4/87.37 vs. 24.55/94.94 according to Table 3).","section":"Section 4.4.2(1)"},{"comment":"HRN achieves an overall accuracy of 39.57%, far lower than the other methods; the paper does not comment on this, and such a large gap may indicate a bug in the evaluation setting. Please verify and briefly explain.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early version, as evidenced by placeholder references and typos. The empirical contribution is potentially valuable, but the benchmark's validity hinges on the content overlap between the ID test set and the transferred OOD data, and on the realism of the generated images. The editor may also wish to confirm that the promised dataset and code release will actually happen before publication, since the benchmark's utility depends on it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate new domain-specific OOD benchmark — the first for bronze ware dating — and the method comparison is broad enough to be useful. But the two 'transferred' OOD categories are generated from the exact 2,861 bronze test images used as the ID evaluation split, which puts a real dent in the paper's central 'small distribution shifts are harder' claim.\n\nWhat's genuinely new: the ShiftedBronzes dataset expands the authors' earlier bronze Ding data with Gui images, sketch and rubbing images, and expert era annotations. Adding seven OOD types and benchmarking 18 detection methods (post-hoc, VLM-based, generation-based) plus six FGVC methods is solid work. The empirical findings — VLM methods clearly ahead, DiffGuard strong on sketch/rubbing, post-hoc methods like MDS and ViM making good use of multiple feature sources — are plausible and match the broader OOD literature. The expert annotation effort and the archaeology grounding are real assets.\n\nThe main soft spot is the confound the stress-test note identifies. Using the test-set bronze images as the source for material transfer means every transferred container and transferred bronze image shares exact material/texture content with an ID evaluation image. For transferred bronze, each OOD sample is literally the ID photo with a different material texture overlaid. That creates a near-duplicate relationship that can inflate or distort difficulty measures, and the paper has no control using held-out bronze images. The sketch and rubbing categories are not affected, so the 'small shift is hard' story doesn't collapse entirely, but the transferred data are a large part of the hard-OOD aggregate, and the conclusion needs to be re-examined with a cleaner construction.\n\nOther issues are more minor but real: no error bars or repeated runs; the best few-shot configuration for VLM methods is selected without being reported; the hard/easy split is author-defined with only a t-SNE plot as justification; and the reference list contains placeholder entries (Alpher, LastName) and the abstract has typos, which points to an unfinished manuscript. The dataset and code are promised but not released, which is a significant gap for a benchmark paper.\n\nWho this is for: researchers building or evaluating OOD detection methods on fine-grained, domain-specific data, and the AI-for-archaeology crowd. It won't reshape a major field, but it's a legitimate testbed.\n\nMy recommendation: send it to peer review. The confound is fixable — regenerate transferred data from held-out bronze images, add error bars, and release the data — and the underlying benchmark effort deserves referee time. It should not get a pass without those fixes.","headline":"A useful niche OOD benchmark for bronze dating, but the transferred OOD categories are generated from the same test-set images as the ID evaluation, which muddies the paper's main difficulty-ranking claim.","tokens_in":19303,"tokens_out":3182,"would_cite":false,"duration_ms":27579,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ShiftedBronzes tests bronze ware dating against seven real-world distribution shifts, and finds VLM-based OOD detection consistently ahead of post-hoc and generation-based methods.","keywords":["out-of-distribution detection","fine-grained classification","bronze ware dating","benchmark dataset","vision-language models","distribution shift","material transfer","archaeological dating"],"falsifier":"Ask a panel of bronze experts to inspect a random sample of transferred container and transferred bronze images and classify them as realistic or artifact-laden; if a substantial fraction show material-transfer artifacts, or if a detector's ranking on ShiftedBronzes changes when evaluated on a held-out set of real photos with analogous shifts, the benchmark's empirical claims would be undermined.","tokens_in":18387,"feed_emoji":"🏺","tokens_out":6168,"duration_ms":49724,"temperature":0.7,"pith_summary":"This paper builds a benchmark, ShiftedBronzes, for fine-grained bronze ware dating in open-world settings. It pairs two in-distribution classes of bronze vessels (Ding and Gui) with seven out-of-distribution types—sketches, rubbings, material-transferred containers and bronzes, and modern container photos—that reflect shifts archaeologists actually face. On this benchmark and five general OOD datasets, the paper compares six fine-grained classifiers and eighteen OOD detection methods, and its central empirical claim is that VLM-based detectors consistently beat post-hoc and generation-based ones, while all methods struggle more with domain-specific OOD samples whose distribution shift is small. If the benchmark is accepted, it gives the archaeology and computer vision communities a shared testbed and a set of rankings that characterize the current state of OOD detection in a specialized domain.","feed_headline":"VLM-based OOD detection leads on bronze ware dating benchmark","feed_subtitle":"New benchmark with seven shift types shows small-domain shifts trip up most detectors.","key_machinery":"The central object is the ShiftedBronzes dataset itself, constructed by expanding the bronze Ding dataset to include Gui vessels, sketches, rubbings, and 51,023 container images, and by using a zero-shot material transfer technique to create transferred container and transferred bronze images. The mechanism that carries the argument is the pairing of ID data (bronze ware color photographs dated by experts into four dynasties) with OOD data arranged on a spectrum of distribution shift, from close (sketch and rubbing) to far (general ImageNet-derived datasets). The benchmark's analytical leverage comes from comparing eighteen OOD detection methods in three families—post-hoc, VLM-based, and generation-based—on both this spectrum and five general OOD datasets, which lets the paper separate domain-specific behavior from conclusions already established on general OOD data.","core_discovery":"The central claim is that ShiftedBronzes is a valid and useful benchmark for bronze ware dating under distribution shift, and that on it, VLM-based OOD detection methods consistently outperform post-hoc and generation-based methods. The paper also reports that current OOD detectors find domain-specific OOD samples with small distribution shifts—sketches, rubbings, transferred images—harder than general OOD images, and that among post-hoc methods those combining multiple feature sources do best. A further finding is that pre-trained diffusion-model-based methods are especially sensitive to color distribution shifts and perform well on sketch and rubbing data, indicating a possible route for improving generation-based detection.","pith_inferences":["A natural next step the paper does not take is to validate the generated transferred images with archaeologist raters; if experts flag material-transfer artifacts, the benchmark's difficulty ranking could change, so such a human study would strengthen or revise the conclusions.","The benchmark's design could transfer to other fine-grained heritage and materials domains—ceramics, coins, manuscripts—where sketches, rubbings, and material-transfer negatives are equally natural OOD types.","The observed 1-shot advantage for VLM prompt learning on specialized data suggests that few-shot regimes may be the default operating point for domain OOD detection, a pattern worth testing on other specialized benchmarks.","Because the paper finds the best post-hoc methods combine multiple feature sources, a testable extension is to combine the two top post-hoc scores into a single detector and measure whether it closes part of the gap to VLM methods."],"forward_implications":["Bronze ware dating systems deployed in the field can be evaluated against realistic OOD inputs, including sketches, rubbings, and container-like objects, rather than only clean color photographs.","VLM-based OOD detection, especially prompt-learning methods with trainable ID prompts, becomes the recommended starting point for open-world archaeology applications.","The finding that small distribution shifts are hardest implies that specialized domains need OOD benchmarks with near-distribution negatives, not only easy general OOD sets.","Diffusion-model-based detectors that are sensitive to color shifts could be improved by training or selecting them on domain-specific color distributions.","The benchmark provides a reusable testbed for future OOD detection methods, so progress can be measured against a fixed, expert-annotated reference."],"supporting_citations":[{"why":"Supplies the bronze Ding dataset that ShiftedBronzes expands and the AKG knowledge-guided dating model used as the pre-model for post-hoc OOD methods.","marker":"[111]"},{"why":"Supplies the zero-shot material transfer technique used to generate the transferred container and transferred bronze OOD images.","marker":"[16]"},{"why":"The prompt-learning VLM method that achieves the best average OOD performance in the experiments.","marker":"[5]"},{"why":"The virtual-logit matching method, one of the two best post-hoc detectors.","marker":"[94]"},{"why":"The Mahalanobis-distance detector, the other top post-hoc method.","marker":"[51]"},{"why":"The diffusion-based detector whose strong results on sketch and rubbing data support the color-shift sensitivity finding.","marker":"[27]"},{"why":"The maximum-softmax-probability baseline that anchors the post-hoc family.","marker":"[37]"},{"why":"Supplies the evaluation code and protocol used to run the post-hoc baselines and general OOD comparisons.","marker":"[100]"}],"fun_headline_variants":["Bronze dating benchmark shows VLM OOD detectors lead","New bronze benchmark: small shifts stump OOD detectors","VLM OOD detection tops bronze dating benchmark","Bronze ware dating benchmark: VLM OOD methods excel"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's conclusions depend on the assumption that the material-transferred images (51,023 per type) are realistic proxies for the distribution shifts faced in real bronze ware dating, and that the expert era labels are reliable enough to serve as ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Bronze dating benchmark shows VLM OOD detectors lead","New bronze benchmark: small shifts stump OOD detectors","VLM OOD detection tops bronze dating benchmark","Bronze ware dating benchmark: VLM OOD methods excel"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2900,"prompt_tokens":889,"completion_tokens":2011,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":1946}},"tokens_in":505,"tokens_out":2011,"duration_ms":13993,"temperature":1.0,"reasoning_tokens":1946,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:49:56.540509+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask a panel of bronze experts to inspect a random sample of transferred container and transferred bronze images and classify them as realistic or artifact-laden; if a substantial fraction show material-transfer artifacts, or if a detector's ranking on ShiftedBronzes changes when evaluated on a held-out set of real photos with analogous shifts, the benchmark's empirical claims would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the bronze Ding dataset that ShiftedBronzes expands and the AKG knowledge-guided dating model used as the pre-model for post-hoc OOD methods."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The virtual-logit matching method, one of the two best post-hoc detectors."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Mahalanobis-distance detector, the other top post-hoc method."},{"cited_title":"and Gimpel, K","cited_arxiv_id":null,"evidence_quote":"The maximum-softmax-probability baseline that anchors the post-hoc family."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the evaluation code and protocol used to run the post-hoc baselines and general OOD comparisons."}],"review_version":1}