{"id":"76f14335-9d45-465f-ac47-905a6ec202e6","arxiv_id":"2509.07308","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A simple trained basis-vector metric achieves 66.14% average accuracy on MIT-States noun-adjective classification, 0.91 points above Naive Bayes, but fails to beat logistic regression on adjective disambiguation.","lead":"The paper proposes a 'basis vector' scoring method that trains per-state reference vectors on CLIP image embeddings and tests it on the MIT-States dataset. It reports a tiny average accuracy gain over Naive Bayes for noun-adjective classification, but no statistically significant advantage for adjective disambiguation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 0.91-point BVM advantage over Naive Bayes rests on an unrandomized split and post-hoc noun filtering; one randomized re-run would settle whether the win is real.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing weakness: the evaluation protocol in Section 5. The reported BVM advantage is small, and the unrandomized, filtered split leaves the central claim vulnerable to dataset-order artifacts and selection bias. My stress-test pass found no additional concern that would change the verdict; if anything, the garbled method equations and absence of code reinforce the need for a principled re-evaluation. Since the reader already rejected the paper on this basis, my recommendation is unchanged. The proposed randomized re-run is the minimal concrete check that would determine whether the claimed win is real or an artifact.","tokens_in":18177,"tokens_out":2272,"duration_ms":32247,"concrete_test":"Re-run the noun-adjective experiment with 20 random 80/20 train/test splits stratified by noun-adjective pair, using both the original filtered noun set and the full unfiltered set, for BVM and Naive Bayes. Report per-seed accuracy, the mean and standard deviation of the BVM−Naive Bayes difference, and a bootstrap 95% CI for that difference. If the difference is not consistently positive across seeds or its CI includes zero, the Table 1 claim should be treated as an artifact of the split/filtering rather than a property of BVM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BVM classifies noun-adjective states best (Table 1: 66.14% vs. Naive Bayes 65.23%). Section 5's protocol is the load-bearing support for this claim, and it has two uncontrolled aspects. First, training uses 'the first 20 embeddings for each adjective class' with no shuffle or randomization, so the split follows the dataset's native ordering. If that ordering correlates with any nuisance factor—acquisition time, background, camera, object pose—the comparison is not an unbiased estimate of method quality; it is a comparison on one arbitrary partition. Second, all nouns with 20 or fewer images per adjective are discarded. That post-hoc filtering changes the target population and can shift results if the excluded nouns are not uniform across classes. The reported margin is only 0.91 percentage points, and no error bars, per-seed variance, or significance test is given. The appendix table shows BVM loses to Naive Bayes on many individual nouns (e.g., bush, cord, water, wax) and wins by large margins on others; the average is therefore sensitive to which nouns survive filtering and how test-set sizes are weighted. Given that the method is also not fully reproducible from the garbled equations (e.g., loss = D·B^T − T has incompatible dimensions), the empirical win is the only evidence for the abstract's 'performs the best' claim. If the win disappears under randomized splits, the central claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BVM (Basis Vector Method), a supervised method that trains a set of basis vectors on CLIP image embeddings to classify the state (adjective) of a noun. The method is evaluated on MIT-States in two settings: (1) noun-adjective pair classification, where BVM is compared against cosine similarity, dot product, binary index, product quantization, Naive Bayes, and a custom neural network; and (2) adjective discrimination, where BVM is compared against logistic regression. The paper reports that BVM achieves the highest average accuracy in the first setting (66.14% vs. 65.23% for Naive Bayes) and that its performance in the second setting is inconclusive, though it outperforms logistic regression when using VGG19 embeddings. The authors conclude that BVM is a promising, computationally lightweight method for state-change detection.","tokens_in":18530,"tokens_out":3963,"duration_ms":46142,"significance":"If the reported result were robust, BVM would be a simple and inexpensive supervised baseline for open-ended state classification, requiring only access to image embeddings and a small training set. The manuscript is honest about the null result in the adjective-discrimination experiment, which is a positive feature, and it evaluates a broad set of baselines. However, the main claim—that BVM performs best—is supported only by a single non-randomized split with no confidence intervals, significance tests, or repeated runs. The method is also not sufficiently specified to be reproduced from the text, because the core loss equation has incompatible matrix dimensions. The contribution is therefore not yet established to the standard of a journal publication, but the approach and empirical question are worth revisiting with proper evaluation.","major_comments":[{"comment":"The central claim that BVM performs best (66.14% vs. 65.23% for Naive Bayes) rests on a single train/test split with no randomization. The text states that 'the first 20 embeddings for each adjective class would be used for training, and the rest would be used for testing,' with no shuffling. If the dataset's native ordering correlates with any nuisance factor, the comparison is not an unbiased estimate of method quality. The 0.91 percentage-point gap could vanish or reverse under a different split. The paper must report results over multiple random splits with error bars (e.g., standard deviation or confidence intervals) and a significance test for the BVM-vs-Naive-Bayes comparison.","section":"Section 5 (Noun-Adjective Pairs Testing), Table 1"},{"comment":"The protocol filters out 'all folders containing 20 images or less for our noun' before evaluation. This post-hoc filtering changes the target population and can materially affect the average. The appendix table shows that BVM loses to Naive Bayes on many individual nouns (e.g., bush, cord, water, wax) and wins by large margins on others; the aggregate is therefore sensitive to which nouns survive filtering and how test-set sizes are weighted. The authors should either justify the filtering on a priori grounds or report results on the full dataset, and should also report the number of nouns/classes retained.","section":"Section 5 (Noun-Adjective Pairs Testing), filtering step"},{"comment":"The loss equation, loss = 1/N Σ D·Bᵀ − T, is dimensionally inconsistent: D is (k·N, d), B is (N, d), so Bᵀ is (d, N) and D·Bᵀ is (k·N, N), whereas T is defined as (k·N, k). Similarly, the match-score equation M = Q·Bᵀ·Tᵀ is not dimensionally coherent as written. This makes the method impossible to reproduce from the text and prevents verification of the reported accuracies. The authors must give a correct, self-contained specification of the loss, the update rule, and the inference procedure (including any norms, reductions, and normalization steps).","section":"Section 3 (BVM training), loss equation"},{"comment":"The noun-adjective experiment does not specify the number of training epochs, learning rate, optimizer settings, or initialization for BVM; the text only says 'x amount of epochs.' The adjective experiment later mentions 5000 epochs. Since BVM is a trained model, these hyperparameters are load-bearing for replicability and for a fair comparison against baselines. The authors should report the exact training configuration for each experimental section.","section":"Section 3 and Section 5, training hyperparameters"}],"minor_comments":[{"comment":"The description of cosine similarity states that '0 meaning the vectors are the same and 1 meaning they are different.' Standard cosine similarity is maximized at 1 for identical vectors and decreases toward 0 for orthogonal vectors; the text appears to have the semantics reversed.","section":"Section 3, baseline definitions"},{"comment":"The adjective-experiment results are introduced with 'Here are the results we obtained:' but no table number or caption is provided. Later, the text says 'Based on table 5,' yet no Table 5 exists. The noun-adjective table is numbered Table 1, so the other results should be given numbered tables as well.","section":"Section 5, tables"},{"comment":"Minor typographical errors: 'Product Quantizor' should be 'Product Quantizer,' and the table does not include per-noun sample sizes or any measure of variance.","section":"Section 5, Table 1"},{"comment":"The stated limitation—'we didn’t have a large amount of time due to the deadlines'—is not a substantive scientific limitation. If time constraints prevented proper evaluation, this should be addressed by additional experiments rather than used as a caveat.","section":"Section 6, Discussion"},{"comment":"The custom neural network is described as having input layer (20, 768), hidden (768, 5), and output (5, # of adjectives). It is unclear why the input dimension is 20 and how the network processes a single test embedding; a precise description of the architecture and training would improve reproducibility.","section":"Section 4, Custom Neural Network architecture"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is more of a workshop-style technical report than a fully developed journal article. The main empirical claim is not supported by the current evaluation protocol, and the method specification is too incomplete to verify. However, the negative result in the adjective experiment is honestly reported, and the proposed method is simple enough that a rerun with proper randomization and corrected equations could provide a real contribution. I recommend major revision rather than rejection, but the revision must be substantial: randomized evaluation with uncertainty quantification, corrected and complete method specification, and reproducible training details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a small, honestly written empirical note, but the main claim—that BVM is the best of the compared metrics on MIT-States noun-adjective pairs—is not supported by the evidence as presented. The 0.91 point gap over Naive Bayes rests on a single split that is not randomized (first 20 images per adjective), a post-hoc filter that drops nouns with few examples, and no error bars or significance test. One randomized re-run could easily erase the gap. The second experiment, where BVM loses to logistic regression on CLIP embeddings, is reported honestly, which I credit.\n\nWhat the paper does well: it is plainly written, the appendix contains full per-noun results, and the authors acknowledge the negative result and limitations. The idea itself is not new—they attribute the core concept to Equiangular Basis Vectors [6]—and applying it to MIT-States with CLIP embeddings is routine. There is no code and the equations are garbled: loss = D·B^T − T has incompatible dimensions (D·B^T is (k*N,N) while T is (k*N,k)), and the match score Q·B^T·T^T produces a vector of length k*N, not k. As written, the method cannot be implemented, which is a load-bearing problem because the empirical result is the only evidence for the method.\n\nThe soft spots are exactly where the reader's report puts them. The split protocol and filtering are uncontrolled; the comparison is not unbiased. Also, the per-noun appendix shows BVM loses to Naive Bayes on several nouns (bush, cord, water, wax) and wins by large margins on others, so the average is sensitive to which nouns survive the filter and how test sizes are weighted. Without randomized splits and variance reporting, the 66.14% number is just a point estimate on one arbitrary partition.\n\nWho is this for? A reader who wants an early look at a simple prototype-classifier idea on MIT-States might skim the appendix, but I wouldn't rely on the headline result. It deserves a desk reject in its current form, not a referee. If the authors fix the equations, add code, and rerun with multiple random splits, it could become a modest workshop note. I would not cite it as-is.\n\nRecommendation: desk reject; the central claim is too weakly supported and the method is not reproducible.","headline":"The 0.91-point win over Naive Bayes is not robust evidence; the method is also not reproducible from the text, though the honest negative result and full per-noun appendix are points in its favor.","tokens_in":19048,"tokens_out":3534,"would_cite":false,"duration_ms":38282,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BVM, a basis-vector metric trained on CLIP embeddings, classifies noun states in MIT-States images better than seven comparison metrics.","keywords":["basis vector metric","state change detection","image embeddings","MIT-States","CLIP","noun-adjective classification","similarity metrics","dynamic image classification"],"falsifier":"Rerun the noun-adjective experiment with the same dataset but randomly shuffle image order within each adjective class before assigning the first 20 to training, repeated over many seeds; if BVM's average accuracy does not consistently exceed Naive Bayes's, the claim that BVM is the best performer is refuted.","tokens_in":18043,"feed_emoji":"🖼️","tokens_out":5951,"duration_ms":63697,"temperature":0.7,"pith_summary":"The paper sets out to make image state detection—identifying what has changed about an object, such as 'peeled' versus 'pureed' apple—cheap and open-ended. It introduces BVM, which trains one basis vector per state (adjective) so that the dimensions of an image embedding that separate states are amplified and irrelevant ones are suppressed. On the MIT-States dataset, BVM reaches 66.14% average accuracy at picking the correct adjective for a noun, ahead of Naive Bayes (65.23%) and all other tested metrics. A second experiment, distinguishing each adjective from all others, finds BVM does not beat logistic regression on CLIP embeddings, though it does edge out logistic regression on VGG19 embeddings. The paper reads this as evidence that with the right embedding model BVM could become a useful general tool for state-change classification.","feed_headline":"Basis-vector metric tops six comparison metrics on object states","feed_subtitle":"On MIT-States images, BVM hits 66.1% accuracy, edging Naive Bayes—no video or heavy compute required.","key_machinery":"The carrying mechanism is the trained basis-vector matrix B. Before training, each row of B is set to the average embedding of images carrying that adjective; a target matrix T (0/1) encodes which adjective each training image should match. Training minimizes loss = (1/N) Σ (D·Bᵀ − T), with Adam stepping the basis vectors so that embedding dimensions that separate states grow in weight and irrelevant dimensions shrink. At test time the query embedding Q is scored by M = Q·Bᵀ·Tᵀ, and the adjective corresponding to the largest column value is the prediction. The paper supports the intuition with 2-D and t-SNE visualizations showing trained basis vectors moving toward their class centers and aw","core_discovery":"The central claim is that a learned set of basis vectors—one per adjective state—can outperform standard similarity and classification baselines when the task is to name the state of a noun from a single image embedding. Concretely, BVM scores 66.14% average accuracy over noun-adjective pair classification on MIT-States, besting Naive Bayes (65.23%), cosine similarity (55.99%), binary index (52.17%), dot product (45.43%), product quantization (41.95%), and a custom neural network (22.99%). The same method is less successful at the separate task of telling one adjective apart from all others: against logistic regression, BVM averages 40.46% versus 45.13% with CLIP embeddings, but 4.71% versus","pith_inferences":["The evaluation uses the first 20 images per adjective class for training and discards nouns with 20 or fewer images; if image order in the dataset correlates with difficulty, the reported 66.14% could overstate BVM's edge. A randomized multi-split replication would tell whether the win is robust.","BVM's failure on the one-vs-rest adjective task suggests its strength is in discriminating among a small set of states for a known noun, not in judging an adjective's presence across many nouns. That distinction is worth testing before applying BVM to open-set state detection.","The same basis-vector scoring could be applied to embeddings from text, audio, or sensor streams, making it a candidate general-purpose change detector, not just an image-state classifier.","The VGG19 result, though low in absolute accuracy, hints that a disentangled or more state-focused embedding space would give BVM a larger advantage; one testable extension is to fine-tune embeddings with a state-discrimination loss before applying BVM."],"forward_implications":["State detection can be run on static image embeddings alone: no video, no fine-tuned image classifier, and no heavy computation beyond embedding extraction.","BVM gives a stronger per-noun state-classification baseline on MIT-States than Naive Bayes, cosine, dot product, binary index, product quantization, and a small neural network.","Embedding choice is not neutral: the same BVM loses to logistic regression on CLIP embeddings but beats it on VGG19 embeddings, so improving BVM may be largely a matter of picking or training the right encoder.","The reported gap between BVM and Naive Bayes could widen with more training epochs, since BVM was trained for only 1000 epochs.","Because BVM trains a basis vector per adjective, it extends naturally to new states or new nouns without retraining the embedding model."],"supporting_citations":[{"why":"Supplies the MIT-States dataset of 53,000 images with noun-adjective state labels, and the logistic-regression comparison method proposed by the MIT-States paper for the adjective experiment.","marker":"[5]"},{"why":"CLIP-ViT-Large-Patch14 produces the 768-dimensional image embeddings on which all metrics are evaluated in the main noun-adjective experiment.","marker":"[7]"},{"why":"Named as the core concept underlying BVM: training basis vectors so they exemplify differences between images.","marker":"[6]"},{"why":"The Faiss library implements the baseline metrics (binary index, product quantization, and the similarity searches) used for comparison.","marker":"[8]"},{"why":"PyTorch provides the Adam optimizer and backpropagation used to train the BVM basis vectors.","marker":"[14]"},{"why":"Previous study of cosine similarity effectiveness on text classification motivates including cosine as a baseline and comparing embedding-scoring combinations.","marker":"[1]"}],"fun_headline_variants":["BVM beats six baselines on image state classification","State detection: basis-vector method tops six comparison metrics","BVM: one embedding, no heavy compute, bests six state detectors","66.1% on MIT-States: BVM beats cosine, dot product, Naive Bayes","Basis-vector metric outdoes six rivals on image state tasks"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The reported 66.14% assumes that taking the first 20 images per adjective class for training, and discarding nouns with 20 or fewer images per adjective, gives an unbiased estimate of BVM's accuracy.","fun_headline_variants_meta":{"raw":{"variants":["BVM beats six baselines on image state classification","State detection: basis-vector method tops six comparison metrics","BVM: one embedding, no heavy compute, bests six state detectors","66.1% on MIT-States: BVM beats cosine, dot product, Naive Bayes","Basis-vector metric outdoes six rivals on image state tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000756,"raw_usage":{"total_tokens":3229,"prompt_tokens":809,"completion_tokens":2420,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":553,"completion_tokens_details":{"reasoning_tokens":2325}},"tokens_in":553,"tokens_out":2420,"duration_ms":23756,"temperature":1.0,"reasoning_tokens":2325,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T22:26:11.610645+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the noun-adjective experiment with the same dataset but randomly shuffle image order within each adjective class before assigning the first 20 to training, repeated over many seeds; if BVM's average accuracy does not consistently exceed Naive Bayes's, the claim that BVM is the best performer is refuted.","supporting_citations":[{"cited_title":"Isola, J","cited_arxiv_id":null,"evidence_quote":"Supplies the MIT-States dataset of 53,000 images with noun-adjective state labels, and the logistic-regression comparison method proposed by the MIT-States paper for the adjective experiment."},{"cited_title":"Equiangular Basis Vectors","cited_arxiv_id":"2303.11637","evidence_quote":"Named as the core concept underlying BVM: training basis vectors so they exemplify differences between images."},{"cited_title":"Johnson, M","cited_arxiv_id":null,"evidence_quote":"The Faiss library implements the baseline metrics (binary index, product quantization, and the similarity searches) used for comparison."},{"cited_title":"A Comparison of Semantic Similarity Methods for Maximum Human Interpretability","cited_arxiv_id":"1910.09129","evidence_quote":"Previous study of cosine similarity effectiveness on text classification motivates including cosine as a baseline and comparing embedding-scoring combinations."}],"review_version":1}