{"id":"d7b639c7-0c34-41f9-ab98-c494da118c90","arxiv_id":"2505.06030","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A method uses word-level sampling and a genetic algorithm to generate counterfactual utterances that flip a language-grounding model's wrong prediction to the correct one.","lead":"This paper generates counterfactual explanations for why a 3D object identification model misclassifies an object, by swapping words in the user's description until the model predicts correctly. The work is a new explanation tool for language-guided robotics and design systems, and it reveals model biases such as treating 'small' differently from 'short'.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own LLM evaluation shows only roughly a third to a half of counterfactuals rated semantically similar, which contradicts the abstract's unqualified 'semantically similar' claim; a second issue is that the reported per-sample counterfactual is selected by different rules in Sections III…","rationale":"The paper is a genuinely constructive step: the method is specified completely, code is released, and the success-rate results are consistently high across three backbones, which supports the claim that the GA can usually find an utterance that flips the model's decision. I am not raising the reader's weakest assumption about single-word edits as the main issue; the empirical success rates suggest that, for this task, the decision boundary is often reachable within the sampled word changes. The load-bearing failure is in the second half of the central claim: semantic similarity. The authors' own LLM-based evaluation places most outputs in the \"Dissimilar\" category, and the cosine-similarity evidence is circular because it is the quantity being optimized. The selection-rule ambiguity compounds this: success rates and similarity scores may refer to different counterfactuals, so the paper never conclusively shows one output that is simultaneously valid and similar. This is an internal inconsistency rather than a disagreement with external consensus, and it can be settled by re-running the provided code with a single explicit selection rule and reporting paired validity and similarity per sample. If the paired numbers still show the same trade-off, the abstract and conclusion need to be revised to claim valid counterfactuals that are sometimes semantically similar, or the optimization should be reweighted toward semantic preservation. Thus I agree with the reader's CONDITIONAL verdict: the underlying method and empirical findings about model bias are valuable, but the headline claim is currently overbroad.","tokens_in":11042,"tokens_out":6935,"duration_ms":71679,"concrete_test":"Re-run the released pipeline for ViT+Qwen on the 1,000 samples with the selection rule fixed and stated once: the final explanation is the valid counterfactual in the final population with the highest USE cosine, and separately the one with the best fitness. For each selected utterance, record validity, USE cosine, Levenshtein distance, and the GPT-4o-mini similarity rating. Then recompute Table I's success rate and the \"Equivalent+Very similar+Similar\" fraction on this same per-sample utterance. If the dissimilar fraction remains at or above about 50% under either rule, or if the success rate drops materially below the reported 97-100%, the combined validity-plus-similarity claim fails. If both metrics hold simultaneously for the same utterance, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the central claim—counterfactuals are \"semantically similar and meaningful\"—is contradicted by the paper's own evaluation. In Table I and Fig. 3, the best context-aware samplers produce large fractions of \"Dissimilar\" and \"Very dissimilar\" ratings: for ViT+Qwen, 53.2% dissimilar plus 6.2% very dissimilar, with only 3.7% equivalent and 29.4% very similar; the \"similar or better\" total is about 39.3%. Across models and strategies, \"Dissimilar\" is usually the modal category. This is not an external dispute about what \"similar\" means; it is the paper's own GPT-4o-mini judge failing to confirm the abstract. The USE cosine column cannot repair this because Eq. (2) is one of the two objectives optimized in Eq. (3); the GA is selecting for high USE cosine, so using the same metric for evaluation is circular and cannot independently establish semantic preservation. The conclusion's own limitation statement admits that this metric \"under-penalize[s] the insertion of contextually unrelated words,\" which is exactly the gap the LLM evaluation exposes. A compounding issue is that the reported utterance is not uniquely defined: Section III.C selects the counterfactual with highest cosine similarity, while Section IV.B says the evaluated candidate achieves the best fitness value. These two rules can name different utterances, so the high success rate—which counts existence of at least one valid counterfactual in the population—and the semantic-similarity scores may be computed on different outputs. The paper therefore does not currently demonstrate that a single output is both valid and semantically similar, which is the core promise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for generating counterfactual utterances for object referent identification with 3D objects. Given a misclassified sample consisting of two objects and a text description, the method first samples alternative utterances by replacing a single content word using one of four strategies (unaware, word-type-aware, word-aware, or context-aware via small LLMs), then applies a genetic algorithm whose fitness combines a class-flip objective and a USE-cosine similarity objective. Experiments on 1,000 ShapeTalk misclassified samples with three backbone models report high success rates (94.5-100%), normalized Levenshtein distances around 0.2-0.3, and use GPT-4o-mini to rate grammaticality and semantic similarity. The paper claims that the resulting counterfactuals are valid, semantically similar, and meaningful, and that context-aware LLM sampling is superior.","tokens_in":11283,"tokens_out":5924,"duration_ms":59669,"significance":"The problem is timely and the paper is the first to address counterfactual explanations for 3D object language grounding. The method is fully described in Algorithm 1, uses reproducible components, and the code is provided; the success-rate results are strong and consistently above the random-search baseline. However, the central semantic-similarity claim is not supported by the paper's own independent evaluation, and the headline similarity metric is also the optimized objective. As presented, the contribution is better described as a method that produces valid counterfactuals with mixed semantic fidelity, not as a method whose counterfactuals are 'semantically similar and meaningful' as stated in the abstract.","major_comments":[{"comment":"The central claim that the generated counterfactuals are 'semantically similar and meaningful' is contradicted by the paper's own GPT-4o-mini evaluation. For the best-performing context-aware configuration (ViT+Qwen), only 3.7% of counterfactuals are rated 'equivalent' and 29.4% 'very similar', while 53.2% are 'dissimilar' and 6.2% 'very dissimilar'; across nearly all model/strategy rows in Table I, 'dissimilar' is the modal category, and 'similar or better' totals are around 39% or less. Since this is the paper's own evaluation in Section IV.B, not an external disagreement about the meaning of 'similar', the abstract's unqualified claim is not supported by the presented evidence.","section":"Table I and Fig. 3; Abstract; Section V"},{"comment":"The USE cosine similarity in Eq. (2) is one of the two objectives optimized in Eq. (3), yet the same quantity is reported as the primary 'Cosine Similarity' metric in Table I and plotted in Fig. 2. A high cosine score is therefore partly a property of the optimizer, not independent evidence of semantic preservation. The paper's own limitation statement in Section V explicitly concedes that this metric 'under-penalize[s] the insertion of contextually unrelated words,' which is exactly the gap exposed by the GPT-4o-mini evaluation. The evaluation should either separate the optimization objective from the reported metric or justify why the reported cosine similarity is not circular.","section":"Section III.C (Eq. (2), Eq. (3)) and Section IV.B (Table I)"},{"comment":"The paper does not consistently define which counterfactual utterance is analyzed. Section III.C states that 'we will always refer to a single counterfactual per sample, defined as the one with the highest cosine similarity,' while Section IV.B's GPT-4o-mini prompt evaluates 'the counterfactual utterance that achieves the best fitness value during the optimization process.' These selection rules can name different utterances, and Section IV.B defines success rate as the fraction of samples with at least one valid counterfactual in the population. As a result, the high success rates and the semantic-similarity distributions in Table I may characterize different utterances than the one a user would receive as the explanation. Please align the selection rule and report both the existence-based and the reported-utterance-based statistics.","section":"Section III.C vs. Section IV.B"}],"minor_comments":[{"comment":"The last sentence contains a typo: 'Theses insights' should be 'These insights'.","section":"Abstract"},{"comment":"The evaluation prompt lists the similarity scale as 'equivalent, very similar, similar, neutral, dissimilar, very dissimilar, or unrelated,' but Table I reports only 'Very dissimilar' and not 'unrelated.' Please clarify whether 'unrelated' is a separate category and how it was mapped in Table I.","section":"Section IV.B"},{"comment":"The word clouds lack a numeric scale or counts, making the statement that 'the larger a word appears, the more frequently it is replaced or inserted' difficult to verify; adding counts or a size legend would improve reproducibility.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The reader's conditional assessment matches my reading of the manuscript. The discrepancies between the abstract, the reported similarity metric, and the GPT-4o-mini results are substantial and must be resolved before publication. I recommend major revision rather than rejection because the method and experiments are reproducible and the validity results are strong; the authors may be able to reframe the contribution, add a stricter selection criterion, or adopt a non-circular evaluation metric. However, if the revision leaves the abstract's 'semantically similar' claim unchanged, I would not support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is the first to apply counterfactual explanations to object referent identification in 3D, and the comparison of four sampling strategies—including three LLM-based samplers—is a legitimate contribution. The method is fully specified, code is available, and the valid-counterfactual success rates (97–100%) are consistently high across three backbones. The empirical findings on model bias, such as the small/short distinction and the engines/propellers error, are genuinely useful for practitioners debugging grounding models. The GA and USE cosine similarity are standard machinery, but the application is new and the paper is honest about its setup.\n\nThe soft spot is real and load-bearing. The abstract claims the counterfactuals are 'semantically similar and meaningful,' but the paper's own Table I shows that for the best context-aware samplers, around half of the generated counterfactuals are rated 'Dissimilar' or 'Very dissimilar' by GPT-4o-mini. That is not an external critique; it is the authors' own evaluation contradicting their claim. The USE cosine score cannot fix this because it is one of the two objectives the GA optimizes, so reporting it as evidence is circular. The conclusion also admits the metric 'under-penalize[s] the insertion of contextually unrelated words.' On top of that, the paper never pins down which counterfactual is being analyzed: Section III.C says the one with highest cosine similarity, Section IV.B says the one with best fitness. Those can be different utterances, so the high success rate (existence of at least one valid in the population) and the similarity ratings may be computed on different outputs. The core promise—a single output that is both valid and semantically similar—is not demonstrated.\n\nNone of this kills the contribution. The method works at finding valid counterfactuals, and the model-bias analysis is worth having. But the semantic-similarity claim has to be reconciled, either by softening the claim, reweighting the objectives, reporting a consistent selection rule, or adding variance estimates. This is a conditional accept, not a reject.\n\nWho is the paper for? Practitioners in robotics and language-assisted design who want a debugging tool, and researchers working on XAI for grounded language. I would bring it to a reading group and cite it if the authors fix the similarity reporting. A serious editor should send it to review.\n\n— your name","headline":"A genuinely new XAI application with solid valid-counterfactual success rates, but the semantic-similarity claim is contradicted by the paper's own LLM evaluation and needs revision.","tokens_in":11947,"tokens_out":2755,"would_cite":true,"duration_ms":25857,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"When a model misreads a 3D object description, a one-word change can flip its answer and show why it erred.","keywords":["counterfactual explanations","object referent identification","language grounding","3D objects","genetic algorithm","explainable AI","ShapeTalk dataset","semantic similarity"],"falsifier":"A concrete check: take the 1,000 misclassified ShapeTalk samples used in the paper, and for each test whether any single-word substitution (from the full vocabulary or from LLM suggestions) can flip the model's prediction while keeping the counterfactual's embedding-based cosine similarity above a chosen threshold. If a substantial fraction of samples have no such word and the genetic algorithm's success rate is carried only by easier samples, the central claim would be limited. Alternatively, run the method on a harder dataset with longer, multi-clause descriptions; if the success rate drops sharply, the single-word assumption is the bottleneck.","tokens_in":10765,"feed_emoji":"🤖","tokens_out":5707,"duration_ms":48789,"temperature":0.7,"pith_summary":"The paper sets out to answer, for 3D object referent identification, the question 'Why did the model pick the wrong object?' Its answer: generate a counterfactual utterance—a minimally altered version of the original text description—that makes the model choose the correct object. The paper claims that a sampling-and-genetic-algorithm search over content-word substitutions can find such counterfactuals while keeping the utterance semantically close to the original. Across three different shape-backbone models and the ShapeTalk dataset, the method achieves high success rates in flipping mispredictions, and the LLM-based context-aware samplers produce counterfactuals that are both more grammatically sound and more semantically similar to the original. If the method works as claimed, it gives practitioners a concrete, human-readable explanation of model failure and a recipe for rewording instructions to robots or design tools.","feed_headline":"One word swap can flip a 3D-object model's wrong pick","feed_subtitle":"Counterfactual rewrites of object descriptions expose why language-grounding models misidentify targets.","key_machinery":"The load-bearing mechanism is the counterfactual utterance generator: a population of candidate utterances is initialized by sampling a single mutable content word (NVAA) and replacing it using one of four strategies, then a genetic algorithm with tournament selection, single-point crossover, and word-level mutation optimizes a fitness score that rewards flipping the model's class decision (with a penalty for invalid counterfactuals) plus the cosine similarity between Universal Sentence Encoder embeddings of original and candidate. The key design choice is restricting mutations to content words so sentence structure is preserved, and using a context-aware LLM sampler that proposes replacement words aware of the full utterance.","core_discovery":"The paper's central claim is that misclassifications in object referent identification can be explained by finding a nearby utterance, differing by one content word (noun, verb, adjective, adverb), that flips the model's prediction from distractor to target. The authors show that their combined sampler-plus-genetic-algorithm procedure produces such valid counterfactuals for the large majority of misclassified samples across three model architectures, and that the resulting counterfactuals keep high embedding-based semantic similarity to the original. The method also exposes systematic biases—for instance, that models link 'small' to horizontal thinness while 'short' is tied to height—and pinpoints weak description words. The paper further claims that replacing words with LLM-suggested context-appropriate alternatives yields counterfactuals that are more grammatically acceptable and semantically faithful than synonym-based or type-preserving random replacements.","pith_inferences":["The success rate depends on the assumed single-word edit budget; on harder, multi-word or syntactically ambiguous misclassifications the method may fail, and a valid counterfactual would require editing several coordinated words.","The embedding-similarity objective could be replaced by a human-aligned or task-level semantic metric to avoid under-penalizing contextually unrelated insertions, a limitation the paper itself notes.","The counterfactual utterances could be used as data augmentation for training; the observed biases suggest that training on such counterfactuals might make models more robust to rephrasings of object descriptions.","Because the method is model-agnostic and only queries class probabilities, it could transfer to other black-box language-vision models without retraining."],"forward_implications":["Practitioners can reword a failed instruction by a single word and the robot or design tool will pick the right object.","The generated counterfactuals expose which words the model relies on and which it ignores, allowing engineers to debug and retrain the identifier.","The same method could be applied to other grounding tasks beyond pairs of objects, such as selecting among many objects or interpreting spatial-relation descriptions.","Using LLM-suggested replacements during search yields counterfactuals that are more natural to human readers, increasing trust in the explanation."],"supporting_citations":[{"why":"Supplies the ShapeTalk dataset and the transformer-based listener used as the to-be-explained model.","marker":"[3]"},{"why":"Provides the Universal Sentence Encoder whose cosine similarity defines the semantic-similarity objective and evaluation metric.","marker":"[6]"},{"why":"Introduced the counterfactual-as-optimization formulation (class flip plus minimal change) that the fitness function adapts.","marker":"[30]"},{"why":"Provides the multi-objective genetic-algorithm baseline and the random-search comparison used in the experiments.","marker":"[9]"},{"why":"Defines the validity and minimality criteria for counterfactual explanations that motivate the success-rate and similarity metrics.","marker":"[29]"},{"why":"Supports using an LLM as a human-surrogate evaluator for grammatical correctness and semantic similarity of generated utterances.","marker":"[7]"}],"fun_headline_variants":["Counterfactual rewrites expose why 3D-object models misclassify","Swap a word, change a model's 3D object pick","One-word tweak reveals biases in 3D language grounding","Explaining mispicks: counterfactual rewrites of object descriptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that replacing a single content word in the original utterance is enough to traverse the model's decision boundary from the wrong object to the right one while keeping the sentence meaningful; if the correct decision requires several coordinated word changes or a syntactic restructuring, the search cannot find a valid, faithful counterfactual.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual rewrites expose why 3D-object models misclassify","Swap a word, change a model's 3D object pick","One-word tweak reveals biases in 3D language grounding","Explaining mispicks: counterfactual rewrites of object descriptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3989,"prompt_tokens":946,"completion_tokens":3043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2966}},"tokens_in":562,"tokens_out":3043,"duration_ms":23332,"temperature":1.0,"reasoning_tokens":2966,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:49:33.254736+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: take the 1,000 misclassified ShapeTalk samples used in the paper, and for each test whether any single-word substitution (from the full vocabulary or from LLM suggestions) can flip the model's prediction while keeping the counterfactual's embedding-based cosine similarity above a chosen threshold. If a substantial fraction of samples have no such word and the genetic algorithm's success rate is carried only by easier samples, the central claim would be limited. Alternatively, run the method on a harder dataset with longer, multi-clause descriptions; if the success rate drops sharply, the single-word assumption is the bottleneck.","supporting_citations":[{"cited_title":"ShapeTalk: A language dataset and framework for 3d shape edits and deformations","cited_arxiv_id":null,"evidence_quote":"Supplies the ShapeTalk dataset and the transformer-based listener used as the to-be-explained model."},{"cited_title":"Universal sentence encoder for En- glish","cited_arxiv_id":null,"evidence_quote":"Provides the Universal Sentence Encoder whose cosine similarity defines the semantic-similarity objective and evaluation metric."},{"cited_title":"Counterfactual explanations without opening the black box: Automated decisions and the GDPR","cited_arxiv_id":null,"evidence_quote":"Introduced the counterfactual-as-optimization formulation (class flip plus minimal change) that the fitness function adapts."},{"cited_title":"Multi-objective counterfactual explanations","cited_arxiv_id":null,"evidence_quote":"Provides the multi-objective genetic-algorithm baseline and the random-search comparison used in the experiments."},{"cited_title":"Can Large Lan- guage Models Be an Alternative to Human Evalua- tions?","cited_arxiv_id":null,"evidence_quote":"Supports using an LLM as a human-surrogate evaluator for grammatical correctness and semantic similarity of generated utterances."}],"review_version":1}