{"id":"306a182f-cc5c-41d0-a548-5b0f6b6cf36c","arxiv_id":"2508.20243","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid CLIP/FLAVA similarity score with positive/negative text and image references classifies Ni-WC metal matrix composite micrographs against six expert criteria, but the in-sample evaluation does not demonstrate zero-shot generalization.","lead":"This paper proposes a framework that uses CLIP and FLAVA embeddings to compare microstructural images with expert-written criteria and reference images, scoring samples as acceptable or defective. It aims to speed up industrial qualification of additively manufactured materials without retraining models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Evaluation of zero-shot claim is in-sample: reference images and z-score statistics are derived from the same 40-image dataset that is scored, so reported accuracies cannot demonstrate generalization to unseen microstructures.","rationale":"I agree with the reader's weakest assumption. The paper presents a plausible framework, but the evidence for the zero-shot claim is structurally compromised: reference images and z-score statistics are drawn from the same pool that is scored. This is not a question of model choice or prompt quality; it is an evaluation-design flaw that directly invalidates the generalization claim. The proposed check (leave-one-out) would settle it. I see no reason to alter the reader's REJECT verdict: the paper cannot be accepted as evidence of zero-shot performance until a proper held-out evaluation is performed. I also note the paper honestly lists limitations, but the in-sample evaluation is not among them. No ad hominem intended.","tokens_in":21984,"tokens_out":3294,"duration_ms":34416,"concrete_test":"Leave-one-out evaluation: for each sample in Table A1, construct positive/negative reference embeddings and z-score μ/σ using only the other 39 samples, then score the held-out sample. Repeat for distribution, dilution, and reinforcement-area criteria; report held-out accuracy and confusion matrices. If accuracy drops materially or boundary decisions flip relative to Figs. 26-28, the reported results are in-sample and the zero-shot claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is zero-shot classification of previously unseen microstructures (Abstract, Section 4E, Fig. 7). In Section 5, the authors 'selected representative positive and negative image samples,' computed average embeddings per Eq. (7), and then 'used cosine similarity to score all images in the dataset with respect to both positive and negative references.' Because those reference images are part of the scored set, the reported confusion matrices (Figs. 26-28) measure retrieval of the reference images, not generalization to new microstructures. Additionally, z-score normalization (Eqs. 9-10) computes μ and σ over the entire dataset, so each query contributes to its own normalization statistics—another form of leakage. The paper explicitly claims this permits 'zero-shot classification of previously unseen microstructures,' but the experimental protocol has no held-out set, no cross-validation, and no ablation where references are restricted to samples excluded from scoring. Without such a split, the headline accuracy values are in-sample and cannot support the central claim. This is the load-bearing flaw: the method might still work, but this paper's evidence does not demonstrate it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a vision-language framework for industrial qualification of additively manufactured Ni-WC metal matrix composites. It uses semantic segmentation (MicroSegQ+) to extract microstructure information, then encodes this information and expert characterization texts with CLIP and FLAVA. To improve over generic embeddings, the authors introduce customized representations based on positive/negative reference images and text prompts, compute per-modality similarity deltas, z-score normalize them, and fuse the deltas into a hybrid score that is thresholded at zero. The authors claim this enables zero-shot classification of previously unseen microstructures and validate it on a 40-image metallography dataset across several expert-assessment criteria, reporting confusion matrices and retrieval accuracies. The paper also proposes an industrial knowledge-base use case and a decision-tree workflow.","tokens_in":22265,"tokens_out":4309,"duration_ms":51854,"significance":"If the central zero-shot claim were properly supported, the paper would make a useful contribution: it addresses a real bottleneck in materials qualification, encodes domain expertise in text prompts, and combines complementary strengths of CLIP and FLAVA. The expert-assessment prompt design and the idea of customizing similarity with positive/negative references are interesting, and the paper includes interpretability analyses of embedding dimensions. However, the current evidence does not support the headline claim. The evaluation is in-sample: references and z-score statistics are derived from the same 40-image dataset that is then scored. No held-out split, cross-validation, or independent test set is described. The load-bearing assertion of zero-shot generalization is therefore unsubstantiated, and the reported accuracies cannot be interpreted as predictive performance.","major_comments":[{"comment":"The experimental protocol is in-sample. The text states that 'representative positive and negative image samples were selected' and then 'cosine similarity was used to score all images in the dataset with respect to both positive and negative references.' Since the reference images are drawn from the same 40-image dataset that is subsequently scored, the confusion matrices in Figs. 26-28 measure nearest-class-mean performance on the training set, not zero-shot classification of unseen microstructures. The z-score normalization in Eqs. (9)-(10) compounds the leakage: the mean and standard deviation of the deltas are computed over the entire dataset, so each query contributes to its own normalization statistics. No held-out split, leave-one-out scheme, or restriction of references to samples excluded from scoring is reported. This directly contradicts the abstract and Section 4E claim of z","section":"Section 5"},{"comment":"The retrieval accuracies in Tables 2 and 3 are computed by ranking all images in the dataset against text prompts and comparing the rankings with expert labels. This is an in-library retrieval evaluation, not a test of generalization. In addition, no baseline comparisons are provided (e.g., segmentation-statistics classifiers, simple image features, or supervised models), and no error bars, confidence intervals, or statistical tests support the reported differences between FLAVA, CLIP, and CLIP-with-color. The paper's claims that hybrid scoring 'significantly improved classification performance' are therefore not quantitatively established.","section":"Section 5"},{"comment":"There is a substantive inconsistency in the expert-assessment numbering. Section 4C defines EA-1 Dilution, EA-2 HAZ, EA-3 Reinforcement Area, EA-4 Porosity, EA-5 Dissolution, and EA-6 Distribution. However, Tables 2 and 3 use rows labeled '#1', '#3', '#5', and the accompanying text refers to these as Dilution, Porosity, and Distribution, respectively. This mapping does not match the definitions in Section 4C and makes the retrieval results difficult to interpret. The manuscript also does not specify which sample IDs were used as positive/negative references in each of the three hybrid classification experiments, which is essential for reproducibility and for assessing the degree of overlap between reference and scored images.","section":"Section 5"}],"minor_comments":[{"comment":"The token 'FLA V A' appears with artificial spaces throughout the manuscript; it should be 'FLAVA' consistently.","section":"Throughout"},{"comment":"The limitations paragraph lists manual prompts, lack of fine-tuning, and sensitivity to selected references, but does not acknowledge the in-sample evaluation protocol or the absence of a held-out set. This should be stated explicitly.","section":"Section 7"},{"comment":"The data availability statement says data 'will be released upon publication' but provides no repository or access mechanism. Given that the experimental claims depend on the reference-selection choices, a concrete data/code release would materially aid reproducibility.","section":"Data Availability"},{"comment":"The row labels '#1', '#3', '#5' should be expanded to the full EA names, and the numbering should be aligned with Section 4C. Also, the text reports 'top-5 accuracies of 60% in EA1... 40% in EA3... 80% in EA5' but the table rows do not identify which EA each row represents.","section":"Tables 2-3"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the in-sample evaluation. If the authors can provide a strict held-out evaluation (e.g., references and z-score statistics computed only on a training subset, with scoring on the remaining images) or substantially reframe the contribution as in-library retrieval/transductive classification rather than zero-shot generalization, the paper could be viable. Given the small dataset (40 images), the authors should also address statistical significance and provide baselines. If no such revision is possible, rejection would be warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper proposes a sensible engineering combination: CLIP for image-text similarity and FLAVA for image-image similarity, scored against hand-built positive/negative references and fused with z-score normalization. That hybrid is a modest but legitimate extension of prototype-based few-shot methods. The expert assessment structure for Ni-WC MMC is clearly described, and the discussion of which samples get confused with which is honest and informative.\n\nThe problem is the evaluation. Section 5 explicitly scores all images in the dataset against reference embeddings built from that same dataset. The z-score normalization uses full-dataset means and standard deviations. So the confusion matrices measure retrieval of the reference images that are already in the scored set; they do not measure generalization to unseen microstructures. The abstract and Section 4 claim \"zero-shot classification of previously unseen microstructures,\" but the protocol has no held-out set, no cross-validation, no baselines, and no error bars. This is a load-bearing flaw: the central claim rises or falls on independent evaluation, and that evaluation is missing.\n\nThere is also no code or data released yet, which another reason the numbers are hard to check.\n\nWhat the paper does well: the authors are transparent about what they did, including the exact samples used as references, and they discuss failure cases in detail. They also acknowledge some limitations, such as manually crafted prompts and no fine-tuning, though they do not flag the in-sample issue.\n\nThe fix is straightforward in principle: restrict the reference images to a training subset and score a held-out test subset, report baselines (e.g., nearest class mean on raw embeddings, or a simple classifier), and ideally release the dataset and code. With that, the framework could be publishable.\n\nWho is this for? People working on VLM-based inspection pipelines in manufacturing, and anyone who needs a reminder that in-sample evaluation does not support zero-shot claims. It deserves a serious referee, but right now the verdict should be major revision or reject, not accept.","headline":"Plausible VLM-based microstructure qualification framework, but the reported evaluation is in-sample, so the zero-shot generalization claim is unsupported.","tokens_in":22725,"tokens_out":1557,"would_cite":false,"duration_ms":16639,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that CLIP and FLAVA embeddings, customized with expert-annotated positive and negative references, can zero-shot classify microstructure micrographs as acceptable or defective against industrial qualification criteria with","keywords":["microstructure informatics","vision-language models","zero-shot classification","CLIP","FLAVA","additive manufacturing qualification","metal matrix composites","semantic segmentation"],"falsifier":"Score metallographs from a build or batch that contributed no reference images and no z-score calibration statistics: build the positive and negative references from one set of samples and classify a completely separate set, then compare labels against expert annotations. If accuracy on the held-out set drops toward chance or degrades markedly relative to the in-sample numbers, the zero-shot generalization claim fails. A quicker audit is to check whether any image used to compute the average reference embeddings (Eq. 7) is also in the scored set; if so, the reported accuracies are in-sample by","tokens_in":21910,"feed_emoji":"🔬","tokens_out":8543,"duration_ms":86496,"temperature":0.7,"pith_summary":"Industrial qualification of additively manufactured materials currently stalls at the manual step where experts interpret segmented micrographs against written criteria. This paper tries to automate that step without training a task-specific classifier: it takes two pre-trained vision-language models, CLIP and FLAVA, and customizes their generic embeddings with positive and negative references, averaged embeddings of expert-annotated images and of textual descriptions of ideal versus defective microstructures. Each new micrograph is scored by its net similarity to the positive minus the negative reference, the two models' scores are z-score normalized and summed, and the sign of the hybrid score decides acceptance for each of six expert assessments (dilution, heat-affected zone, bead reinforcement, porosity, carbide dissolution, carbide distribution). On 40 optical metallographs of laser-deposited nickel-tungsten carbide composites, this zero-shot scheme separates acceptable from defective samples, with residual errors attributed to reference sets that undersample defect variants. If the approach holds, materials qualification could move from manual expert inspection toward reusable, interpretable, human-in-the-loop pipelines.","feed_headline":"Zero-shot vision-language scoring screens metal-composite defects","feed_subtitle":"CLIP and FLAVA embeddings, fused with expert text, separate acceptable from defective Ni-WC samples.","key_machinery":"The net similarity delta with customized positive and negative references. For each expert assessment, Equations 5-8 define mean positive and negative embeddings: average CLIP text embeddings of ideal and non-ideal textual descriptions, and average FLAVA image embeddings of pre-qualified and defective example micrographs. A query image's signed score is cos(image, positive reference) minus cos(image, negative reference), computed separately in the CLIP vision-language space and the FLAVA vision-vision space. The two deltas are standardized by z-scores over the dataset and summed into the hybrid score whose sign yields the zero-shot label. This object carries the argument because it converts","core_discovery":"The central claim is that general-purpose embeddings from pre-trained vision-language models, too generic on their own for microstructure data, can be turned into domain-grounded qualification criteria by comparing every query image against averaged positive and negative reference embeddings built from expert-annotated images and their textual descriptions. For each expert assessment the score is a similarity delta: cosine similarity to the positive reference minus cosine similarity to the negative reference. The framework then hybridizes two complementary models: CLIP supplies the image-to-text delta, which stays stable across phrasings of the same criterion and improves when prompts name t","pith_inferences":["A direct test of the zero-shot claim would score metallographs from an entirely different build or batch, with references drawn only from the current one; the paper scores all images in a single dataset against references selected from that same dataset, so its accuracies are likely an optimistic bound on cross-build transfer.","The hybrid sum of z-scored deltas is one fusion rule; weighted or learned fusion, or voting across criteria, are untested alternatives, and the paper's own tool proposal lists them only as interface options.","The color-aware prompting result suggests a transferable recipe: aligning prompt vocabulary with the segmentation color map may boost cross-modal similarity in other segmentation-based inspection pipelines, and could be tested cheaply on grayscale or SEM imagery.","Reference-set ablation, adding one positive or negative example at a time and measuring accuracy gain, would reveal how many expert-annotated examples are actually needed, which the paper does not quantify."],"forward_implications":["Zero-shot qualification of microstructures against expert criteria is feasible without task-specific retraining, provided the positive and negative references cover the range of defect variants for each criterion.","FLAVA and CLIP contribute complementary strengths; a hybrid score exploits FLAVA's sharper visual discrimination and CLIP's stable, phrasing-robust textual alignment.","Z-score normalization is decision-relevant, not cosmetic: it flips borderline classifications by rebalancing the influence of each model near the zero threshold.","Color-aware textual prompts that mirror the segmentation color code materially improve CLIP image-to-text alignment, a cheap prompt-engineering lever for this domain.","The same reference-and-threshold machinery can be assembled into a modular detection tree and multimodal knowledge base that supports retrieval, semantic search, and human-in-the-loop threshold adjustment."],"supporting_citations":[{"why":"Supplies CLIP, the pre-trained contrastive vision-language model whose image-text similarity delta contributes the language side of the hybrid score.","marker":"[42]"},{"why":"Supplies FLAVA, the pre-trained model whose vision-vision similarity delta contributes sharper visual discrimination to the hybrid score.","marker":"[43]"},{"why":"Supplies MicroSegQ+, the deep semantic segmentation tool that converts raw metallographs into the information-level feature representations the framework then qualifies.","marker":"[3]"},{"why":"Grounds the dataset in the directed energy deposition additive manufacturing process and its characteristic defects, which the expert assessments target.","marker":"[41]"},{"why":"Prior convolutional segmentation work on the same nickel-chromium-boron-silicon tungsten-carbide material system that the dataset and defect vocabulary build on.","marker":"[4]"},{"why":"Early computer-vision treatment of microstructural image data that motivates treating images as searchable, classifiable informatics rather than raw pixels.","marker":"[12]"}],"fun_headline_variants":["Zero-shot hybrid VL scoring screens metal-composite defects","Expert-guided zero-shot vision-language scoring flags defects","Positive-minus-negative embedding delta screens AM defects","Hybrid CLIP-FLAVA zero-shot microstructural defect scoring","Zero-shot VL scoring with expert references separates AM defects"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The argument rests on the assumption that a handful of hand-picked reference images and text prompts, selected from the same dataset that is then scored, represent the full range of acceptable and defective microstructures; the paper does not report excluding reference images from the scored set, so the demonstrated accuracies may be in-sample rather than evidence of generalization to unseen microstructures.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot hybrid VL scoring screens metal-composite defects","Expert-guided zero-shot vision-language scoring flags defects","Positive-minus-negative embedding delta screens AM defects","Hybrid CLIP-FLAVA zero-shot microstructural defect scoring","Zero-shot VL scoring with expert references separates AM defects"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001152,"raw_usage":{"total_tokens":4622,"prompt_tokens":767,"completion_tokens":3855,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":3779}},"tokens_in":511,"tokens_out":3855,"duration_ms":32761,"temperature":1.0,"reasoning_tokens":3779,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:12:58.541459+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Score metallographs from a build or batch that contributed no reference images and no z-score calibration statistics: build the positive and negative references from one set of samples and classify a completely separate set, then compare labels against expert annotations. If accuracy on the held-out set drops toward chance or degrades markedly relative to the in-sample numbers, the zero-shot generalization claim fails. A quicker audit is to check whether any image used to compute the average reference embeddings (Eq. 7) is also in the scored set; if so, the reported accuracies are in-sample by","supporting_citations":[{"cited_title":"W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., and Clark, J","cited_arxiv_id":null,"evidence_quote":"Supplies CLIP, the pre-trained contrastive vision-language model whose image-text similarity delta contributes the language side of the hybrid score."},{"cited_title":"Flava: A foundational language and vision alignment model","cited_arxiv_id":null,"evidence_quote":"Supplies FLAVA, the pre-trained model whose vision-vision similarity delta contributes sharper visual discrimination to the hybrid score."},{"cited_title":"F., El Haddad, R., Zimmermann, M., Wood, G., Lamouche, G., Wanjara, P., and Zhao, Y","cited_arxiv_id":null,"evidence_quote":"Supplies MicroSegQ+, the deep semantic segmentation tool that converts raw metallographs into the information-level feature representations the framework then qualifies."},{"cited_title":"L., Bose, S., Bandyopadhyay, A., Schoenung, J","cited_arxiv_id":null,"evidence_quote":"Grounds the dataset in the directed energy deposition additive manufacturing process and its characteristic defects, which the expert assessments target."},{"cited_title":"J., Automated semantic segmentation of NiCrBSi-WC optical microscopy images using convolutional neural networks","cited_arxiv_id":null,"evidence_quote":"Prior convolutional segmentation work on the same nickel-chromium-boron-silicon tungsten-carbide material system that the dataset and defect vocabulary build on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Early computer-vision treatment of microstructural image data that motivates treating images as searchable, classifiable informatics rather than raw pixels."}],"review_version":1}