Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Basis Vector Metric: A Method for Robust Open-Ended State Change Detection

T0 review · 4 major / 5 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BVM, a basis-vector metric trained on CLIP embeddings, classifies noun states in MIT-States images better than seven comparison metrics.

desk verdict The 0.91-point win over Naive Bayes is not robust evidence; the method is also not reproducible from the text, though the honest negative result and full per-noun appendix are points in its favor. read the letter →

arxiv 2509.07308 v1 pith:BM2A6AOY submitted 2025-09-09 cs.CL cs.AI

David Oprea , Sam Powers This is my paper · ORCID
classification cs.CLcs.AI
keywords basisvectormetricstatechangedetectionimageembeddingsMIT-StatesCLIPnoun-adjectiveclassificationsimilaritymetricsdynamic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to make image state detection—identifying what has changed about an object, such as 'peeled' versus 'pureed' apple—cheap and open-ended. It introduces BVM, which trains one basis vector per state (adjective) so that the dimensions of an image embedding that separate states are amplified and irrelevant ones are suppressed. On the MIT-States dataset, BVM reaches 66.14% average accuracy at picking the correct adjective for a noun, ahead of Naive Bayes (65.23%) and all other tested metrics. A second experiment, distinguishing each adjective from all others, finds BVM does not beat logistic regression on CLIP embeddings, though it does edge out logistic regression on VGG19 embeddings. The paper reads this as evidence that with the right embedding model BVM could become a useful general tool for state-change classification.

What carries the argument

The carrying mechanism is the trained basis-vector matrix B. Before training, each row of B is set to the average embedding of images carrying that adjective; a target matrix T (0/1) encodes which adjective each training image should match. Training minimizes loss = (1/N) Σ (D·Bᵀ − T), with Adam stepping the basis vectors so that embedding dimensions that separate states grow in weight and irrelevant dimensions shrink. At test time the query embedding Q is scored by M = Q·Bᵀ·Tᵀ, and the adjective corresponding to the largest column value is the prediction. The paper supports the intuition with 2-D and t-SNE visualizations showing trained basis vectors moving toward their class centers and aw

What would settle it

Rerun the noun-adjective experiment with the same dataset but randomly shuffle image order within each adjective class before assigning the first 20 to training, repeated over many seeds; if BVM's average accuracy does not consistently exceed Naive Bayes's, the claim that BVM is the best performer is refuted.

Watch

Extended reading notes

Core claim

The central claim is that a learned set of basis vectors—one per adjective state—can outperform standard similarity and classification baselines when the task is to name the state of a noun from a single image embedding. Concretely, BVM scores 66.14% average accuracy over noun-adjective pair classification on MIT-States, besting Naive Bayes (65.23%), cosine similarity (55.99%), binary index (52.17%), dot product (45.43%), product quantization (41.95%), and a custom neural network (22.99%). The same method is less successful at the separate task of telling one adjective apart from all others: against logistic regression, BVM averages 40.46% versus 45.13% with CLIP embeddings, but 4.71% versus

Load-bearing premise

The reported 66.14% assumes that taking the first 20 images per adjective class for training, and discarding nouns with 20 or fewer images per adjective, gives an unbiased estimate of BVM's accuracy.

Editorial extensions

If this is right

  • State detection can be run on static image embeddings alone: no video, no fine-tuned image classifier, and no heavy computation beyond embedding extraction.
  • BVM gives a stronger per-noun state-classification baseline on MIT-States than Naive Bayes, cosine, dot product, binary index, product quantization, and a small neural network.
  • Embedding choice is not neutral: the same BVM loses to logistic regression on CLIP embeddings but beats it on VGG19 embeddings, so improving BVM may be largely a matter of picking or training the right encoder.
  • The reported gap between BVM and Naive Bayes could widen with more training epochs, since BVM was trained for only 1000 epochs.
  • Because BVM trains a basis vector per adjective, it extends naturally to new states or new nouns without retraining the embedding model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The evaluation uses the first 20 images per adjective class for training and discards nouns with 20 or fewer images; if image order in the dataset correlates with difficulty, the reported 66.14% could overstate BVM's edge. A randomized multi-split replication would tell whether the win is robust.
  • BVM's failure on the one-vs-rest adjective task suggests its strength is in discriminating among a small set of states for a known noun, not in judging an adjective's presence across many nouns. That distinction is worth testing before applying BVM to open-set state detection.
  • The same basis-vector scoring could be applied to embeddings from text, audio, or sensor streams, making it a candidate general-purpose change detector, not just an image-state classifier.
  • The VGG19 result, though low in absolute accuracy, hints that a disentangled or more state-focused embedding space would give BVM a larger advantage; one testable extension is to fine-tune embeddings with a state-discrimination loss before applying BVM.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes BVM (Basis Vector Method), a supervised method that trains a set of basis vectors on CLIP image embeddings to classify the state (adjective) of a noun. The method is evaluated on MIT-States in two settings: (1) noun-adjective pair classification, where BVM is compared against cosine similarity, dot product, binary index, product quantization, Naive Bayes, and a custom neural network; and (2) adjective discrimination, where BVM is compared against logistic regression. The paper reports that BVM achieves the highest average accuracy in the first setting (66.14% vs. 65.23% for Naive Bayes) and that its performance in the second setting is inconclusive, though it outperforms logistic regression when using VGG19 embeddings. The authors conclude that BVM is a promising, computationally lightweight method for state-change detection.

Significance. If the reported result were robust, BVM would be a simple and inexpensive supervised baseline for open-ended state classification, requiring only access to image embeddings and a small training set. The manuscript is honest about the null result in the adjective-discrimination experiment, which is a positive feature, and it evaluates a broad set of baselines. However, the main claim—that BVM performs best—is supported only by a single non-randomized split with no confidence intervals, significance tests, or repeated runs. The method is also not sufficiently specified to be reproduced from the text, because the core loss equation has incompatible matrix dimensions. The contribution is therefore not yet established to the standard of a journal publication, but the approach and empirical question are worth revisiting with proper evaluation.

major comments (4)
  1. [Section 5 (Noun-Adjective Pairs Testing), Table 1] The central claim that BVM performs best (66.14% vs. 65.23% for Naive Bayes) rests on a single train/test split with no randomization. The text states that 'the first 20 embeddings for each adjective class would be used for training, and the rest would be used for testing,' with no shuffling. If the dataset's native ordering correlates with any nuisance factor, the comparison is not an unbiased estimate of method quality. The 0.91 percentage-point gap could vanish or reverse under a different split. The paper must report results over multiple random splits with error bars (e.g., standard deviation or confidence intervals) and a significance test for the BVM-vs-Naive-Bayes comparison.
  2. [Section 5 (Noun-Adjective Pairs Testing), filtering step] The protocol filters out 'all folders containing 20 images or less for our noun' before evaluation. This post-hoc filtering changes the target population and can materially affect the average. The appendix table shows that BVM loses to Naive Bayes on many individual nouns (e.g., bush, cord, water, wax) and wins by large margins on others; the aggregate is therefore sensitive to which nouns survive filtering and how test-set sizes are weighted. The authors should either justify the filtering on a priori grounds or report results on the full dataset, and should also report the number of nouns/classes retained.
  3. [Section 3 (BVM training), loss equation] The loss equation, loss = 1/N Σ D·Bᵀ − T, is dimensionally inconsistent: D is (k·N, d), B is (N, d), so Bᵀ is (d, N) and D·Bᵀ is (k·N, N), whereas T is defined as (k·N, k). Similarly, the match-score equation M = Q·Bᵀ·Tᵀ is not dimensionally coherent as written. This makes the method impossible to reproduce from the text and prevents verification of the reported accuracies. The authors must give a correct, self-contained specification of the loss, the update rule, and the inference procedure (including any norms, reductions, and normalization steps).
  4. [Section 3 and Section 5, training hyperparameters] The noun-adjective experiment does not specify the number of training epochs, learning rate, optimizer settings, or initialization for BVM; the text only says 'x amount of epochs.' The adjective experiment later mentions 5000 epochs. Since BVM is a trained model, these hyperparameters are load-bearing for replicability and for a fair comparison against baselines. The authors should report the exact training configuration for each experimental section.
minor comments (5)
  1. [Section 3, baseline definitions] The description of cosine similarity states that '0 meaning the vectors are the same and 1 meaning they are different.' Standard cosine similarity is maximized at 1 for identical vectors and decreases toward 0 for orthogonal vectors; the text appears to have the semantics reversed.
  2. [Section 5, tables] The adjective-experiment results are introduced with 'Here are the results we obtained:' but no table number or caption is provided. Later, the text says 'Based on table 5,' yet no Table 5 exists. The noun-adjective table is numbered Table 1, so the other results should be given numbered tables as well.
  3. [Section 5, Table 1] Minor typographical errors: 'Product Quantizor' should be 'Product Quantizer,' and the table does not include per-noun sample sizes or any measure of variance.
  4. [Section 6, Discussion] The stated limitation—'we didn’t have a large amount of time due to the deadlines'—is not a substantive scientific limitation. If time constraints prevented proper evaluation, this should be addressed by additional experiments rather than used as a caveat.
  5. [Section 4, Custom Neural Network architecture] The custom neural network is described as having input layer (20, 768), hidden (768, 5), and output (5, # of adjectives). It is unclear why the input dimension is 20 and how the network processes a single test embedding; a precise description of the architecture and training would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BVM is a supervised classifier evaluated on held-out test embeddings.

full rationale

The paper's central claim is empirical: BVM scores 66.14% average accuracy in Table 1 versus 65.23% for Naive Bayes on MIT-States noun-adjective pairs. The BVM procedure is a training-time fit: basis vectors B are optimized so that D·B^T approximates the target matrix T of training labels, and match scores for a query are computed from Q·B^T·T^T. This is a supervised linear classifier, not a first-principles derivation, and the reported accuracy is measured on test images ('The first 20 embeddings for each adjective class would be used for training, and the rest would be used for testing'). Because the test labels are not used to fit B, the result is not equivalent to its input by construction. No parameter fitted to the test set is renamed as a prediction. The only citation for the core concept, [6], is to Equiangular Basis Vectors by different authors and is not used to forbid alternatives or as a uniqueness theorem. The paper contains no self-citations from the present authors. The evaluation-split and filtering choices may threaten validity or reproducibility, and the loss equation has incompatible dimensions, but those are correctness concerns, not circularity. Therefore the circularity score is 0.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper's central result depends on a small set of empirical choices rather than on new physics or mathematics: the use of CLIP embeddings, the assumption of linear separability, and a training protocol with arbitrary hyperparameters. The BVM is a trained linear model; no new entities (particles, forces, dimensions) are introduced. The main load-bearing assumptions are domain assumptions about the embedding space and the dataset, plus a set of unstated training details that make reproduction difficult.

free parameters (6)
  • BVM training epochs (noun-adjective experiment) = 1000 (implied)
    The Discussion mentions 'if we trained BVM for more than 1000 epochs', implying 1000 epochs was used for the noun-adjective results. The choice is not justified via validation.
  • BVM training epochs (adjective experiment) = 5000
    The Results state logistic regression 'performed better than our BVM metric, even when it was trained on 5000 epochs', implying BVM was trained for 5000 epochs here. No validation-based selection is reported.
  • Minimum images per noun filter = 20
    Nouns with 20 or fewer images per adjective class are excluded from the noun-adjective experiment. This post-hoc threshold changes the set of nouns evaluated and can bias the average accuracy.
  • Train/test split (noun-adjective experiment) = first 20 per adjective class for training; remainder for testing
    The split is by dataset order, not randomized; no seed is given. This can introduce systematic differences if images are ordered by collection time or source.
  • Train/test split (adjective experiment) = 80:20 random shuffle
    The paper says images are randomly shuffled before the 80/20 split, but no seed or repeated runs are reported, so the reported single numbers have no variance estimate.
  • Custom neural network architecture = input (20,768), hidden (768,5), output (5,k)
    The density and dimensions of the baseline neural network are arbitrary and severely underperform (22.99%), indicating an underspecified model rather than a competitive baseline.
assumptions (4)
  • domain assumption CLIP embeddings linearly encode adjective state differences of images
    BVM computes match scores as Q·B^T·T^T, a purely linear operation on the query embedding. The entire method presumes that state information (peeled, melted, etc.) is recoverable by a linear map from the CLIP feature space. If this is false, the method's performance and the comparison are not meaningful (Section 3, match score equation).
  • domain assumption MIT-States ground-truth labels are correct and the dataset is representative
    All accuracy numbers are computed against MIT-States adjective/noun labels. The paper itself notes 'some inconsistent images' (Section 6, image 8b), which can throw off predictions. Label noise directly corrupts the reported accuracies.
  • domain assumption The BVM loss and training procedure are well-defined and converge to a useful solution
    The loss equation is typeset as 'loss= 1 N X D·B ⊺ −T' and is not mathematically clear. The paper assumes that minimizing this quantity with Adam produces basis vectors that separate the states, but the objective is never precisely stated.
  • domain assumption The first 20 images per adjective class form an unbiased training set
    The noun-adjective experiment uses the first 20 embeddings in dataset order for training and the rest for testing, without randomization or a seed. If the dataset ordering correlates with image content, the split is biased and the reported 66.14% is not a reliable estimate (Section 5, noun-adjective pairs testing).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Basis Vector Metric: A Method for Robust Open-Ended State Change Detection." pith.science (2026). https://pith.science/paper/BM2A6AOY

@misc{pith2026250907308,
  author       = {Pith},
  title        = {Pith review of: Basis Vector Metric: A Method for Robust Open-Ended State Change Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BM2A6AOY}},
  note         = {Machine review of arXiv:2509.07308}
}
read the original abstract

We test a new method, which we will abbreviate using the acronym BVM (Basis Vectors Method), in its ability to judge the state changes in images through using language embeddings. We used the MIT-States dataset, containing about 53,000 images, to gather all of our data, which has 225 nouns and 115 adjectives, with each noun having about 9 different adjectives, forming approximately 1000 noun-adjective pairs. For our first experiment, we test our method's ability to determine the state of each noun class separately against other metrics for comparison. These metrics are cosine similarity, dot product, product quantization, binary index, Naive Bayes, and a custom neural network. Among these metrics, we found that our proposed BVM performs the best in classifying the states for each noun. We then perform a second experiment where we try using BVM to determine if it can differentiate adjectives from one another for each adjective separately. We compared the abilities of BVM to differentiate adjectives against the proposed method the MIT-States paper suggests: using a logistic regression model. In the end, we did not find conclusive evidence that our BVM metric could perform better than the logistic regression model at discerning adjectives. Yet, we were able to find evidence for possible improvements to our method; this leads to the chance of increasing our method's accuracy through certain changes in our methodologies.

Figures

Figures reproduced from arXiv: 2509.07308 by the authors.

Figure 3
Figure 3. Basis Vectors operating on 100 dimensions [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Example images from the dataset. (a) and (b) show different adjectives [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. The basic neural network we created as a metric for comparison [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figures from the paper (3 more)
Figure 6
Figure 6. Figure 6: A diagram of how our logistic regression model would work for [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Some results from noun adjective testing with both training and [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Some results from adjective testing with both training and testing [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages

  1. [6]

    Y. Shen, X. Sun, and X.-S. Wei.Equiangular Basis Vectors. arXiv preprint arXiv:2303.11637, 2023

    work page Pith review arXiv 2023

  2. [1]

    A Comparison of Semantic Similarity Methods for Maximum Human Interpretability

    P. Sitikhu, K. Pahi, P. Thapa, and S. Shakya.A Comparison of Semantic Similarity Methods for Maximum Human Interpretability. arXiv preprint arXiv:1910.09129, 2019

    work page Pith review arXiv 1910

  3. [2]

    C. Zhou, F. Zhong, and C. Oztireli.CLIP-PAE: Projection- Augmentation Embedding to Extract Relevant Features for a Disentangled, Interpretable, and Controllable Text-Guided Face Manipulation. arXiv preprint arXiv:2210.03919, 2023

    work page Pith review arXiv 2023

  4. [3]

    Russakovsky, J

    O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L. Fei-Fei.ImageNet Large Scale Visual Recognition Challenge. arXiv preprint arXiv:1409.0575, 2015

    arXiv 2015

  5. [4]

    Stani¯ ut˙ e and D

    R. Stani¯ ut˙ e and D. ˇSeˇ sok.A Systematic Literature Review on Image Captioning. Applied Sciences, 9(10):2024, 2019

    work page 2024

  6. [5]

    Isola, J

    P. Isola, J. J. Lim, and E. H. Adelson.Discovering States and Transformations in Image Collections. In CVPR, 2015

    work page 2015

  7. [7]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2103.00020, 2021

    arXiv 2021

  8. [8]

    Johnson, M

    J. Johnson, M. Douze, and H. J´ egou.Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019

    work page 2019

Show all 16 references
  1. [9]

    M. Z. Alom, T. M. Taha, C. Yakopcic, S. Westberg, P. Sidike, M. S. Nasrin, B. C. Van Esesn, A. A. S. Awwal, and V. K. Asari.The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches. arXiv preprint arXiv:1803.01164, 2018

    2018 arXiv

  2. [10]

    K. He, X. Zhang, S. Ren, and J. Sun.Deep Residual Learning for Image Recognition. arXiv preprint arXiv:1512.03385, 2015

    2015 arXiv

  3. [11]

    Simonyan and A

    K. Simonyan and A. Zisserman.Very Deep Convolutional Networks for Large-Scale Image Recognition. arXiv preprint arXiv:1409.1556, 2015

    2015 arXiv

  4. [12]

    Szegedy, W

    C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.Going Deeper with Convolutions. arXiv preprint arXiv:1409.4842, 2014

    2014 arXiv

  5. [13]

    T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll´ ar.Microsoft COCO: Common Objects in Context. arXiv preprint arXiv:1405.0312, 2015. 12

    2015 arXiv

  6. [14]

    Paszke, S

    A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K¨ opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.PyTorch: An Imperative Style, High-Pe...

    1912 arXiv

  7. [15]

    C. R. Harris, K. J. Millman, S. J. van der Walt, R. Gommers, P. Virtanen, D. Cournapeau, E. Wieser, J. Taylor, S. Berg, N. J. Smith, R. Kern, M. Picus, S. Hoyer, M. H. van Kerkwijk, M. Brett, A. Haldane, J. F. del R ´ ıo, M. Wiebe, P. Peterson, P. G´ erard-Marchant, K. Sheppar...

    2020

  8. [16]

    M. S. Hossain and M. Arefin.Development of an Intelligent Job Recommender System for Freelancers using Client’s Feedback Classification and Association Rule Mining Techniques. Journal of Software, 14(7):312– 331, 2019. 13 (a) Training Image - True Value: Thin Cord Cosine Simil...

    2019

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.