{"id":"775e5166-75c0-4f73-9b88-19e9ed1def09","arxiv_id":"2509.01198","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A relationship-matrix preservation loss for dimensionality reduction, with perturbation-theoretic bounds showing orthogonality and rank are kept when the Gram matrix is matched.","lead":"The paper proposes a loss function that trains a neural network to keep pairwise relationships between data points unchanged when the data is compressed to fewer dimensions. Initial experiments on image embeddings show retrieval quality is roughly maintained when the dimension is cut from 1024 to 256.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4.4's rank-preservation guarantee is false as stated: the eigenvalue index is wrong and k≥r does not force rank(Y)=r; a 2-point example with ε<σ_r^4 has rank(Y)>r.","rationale":"The reader's weakest assumption flagged k≥r as unverified; I go further: even when k≥r holds, the rank theorem is internally inconsistent. The proof uses Weyl on an eigenvalue index that is zero for G_X under the paper's ordering, and the conclusion exact rank does not follow from perturbation theory alone because k>r permits extra positive eigenvalues. The 2×2 counterexample satisfies all theorem hypotheses and violates the conclusion. This is a load-bearing flaw: Theorem 4.4 and 4.5 are the paper's main theoretical support for preserving linear independence and subspace structure. The orthogonality corollary (4.3) is unaffected, so the paper has salvageable parts, but the central claim as stated is false. I therefore recommend REJECT as written; a corrected theorem restricting to k=r and using the correct eigenvalue index would be needed before the rank and subspace guarantees could be accepted.","tokens_in":5942,"tokens_out":12767,"duration_ms":153395,"concrete_test":"Analytical check: set X=[[1],[0]], Y=[[1,0],[0,0.1]], compute G_X, G_Y, ε=||G_X-G_Y||_F^2=10^-4, σ_1(X)=1, and rank(Y)=2. This directly contradicts Theorem 4.4's conclusion under its stated assumptions. Then verify whether the repaired version (k=r, eigenvalue index λ_r) restores the theorem; if the repaired version requires k=r, the paper's compression claims to k0, i.e. rank(Y)≥r; since the assumption is k≥r, rank(Y) may be >r. Explicit counterexample: X=[1;0] (n=2,d=1,r=1), Y=[[1,0],[0,δ]] with δ<1. Then G_X=diag(1,0), G_Y=diag(1,δ^2), ε=δ^4<1=σ_1^4, yet rank(Y)=2≠1. The theorem's conclusion is not merely unverified; it is false. Theorem 4.5 inherits the problem because it assumes range(G_Y) has dimension r. A valid exact-rank statement requires k=r (or an explicit rank(Y)≤r), which is not the claim made and is not satisfied by the §5.1 compression experiments unless r≤target dimension.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Relationship Preserving Loss (RPL) for dimensionality reduction. RPL trains a neural network f by minimizing a discrepancy D between relationship matrices R(X) and R(f(X)), where R is a user-specified kernel such as Gram, cosine, or RBF. The theoretical section derives perturbation-theoretic bounds: Lemma 4.1 transfers a mini-batch error to a global Frobenius error via Serfling's inequality; Theorem 4.2 and Corollary 4.3 give entry-wise preservation of zero relationships; Theorem 4.4 claims exact rank preservation when k≥r and ε<σ_r^4(X); Theorem 4.5 bounds the subspace angle. Section 5 reports preliminary retrieval experiments on MS COCO and qualitative manifold-preservation experiments on synthetic data.","tokens_in":6282,"tokens_out":5726,"duration_ms":71753,"significance":"The core idea—matching relationship matrices as a differentiable training objective—is simple and potentially useful, and the orthogonality bound in Corollary 4.3 is an immediate but transparent consequence of the loss. If the rank and subspace theorems were correct, the framework would offer quantitative guarantees beyond what is typically provided for neural dimensionality reduction. However, the central rank-preservation theorem is false as stated, and the experimental section does not check the assumptions of the theory. The significance of the paper is therefore currently not established; a corrected and empirically grounded version could be of interest.","major_comments":[{"comment":"The theorem is false as stated. For an n×n PSD Gram matrix G_X of rank r, the positive eigenvalues are λ_1,...,λ_r; the smallest positive one is λ_r, not λ_{n−r+1}. The latter is zero whenever r0, i.e., rank(G_Y)≥r, and rank(Y) may exceed r. Explicit counterexample: X=(1,0)^T so G_X=diag(1,0), r=1; take Y=[[1,0],[0,δ]] with 0<δ<1, k=2. Then ε=δ^4<σ_1^4=1, but rank(Y)=2≠1. Theorem 4.5 inherits this defect because it assumes range(G_Y) has dimension r. Exact rank preservation would require k=r (or an additional bound forcing rank(Y)≤r), not the stated k≥r.","section":"§4.3, Theorem 4.4"},{"comment":"The kernel extension is oversold. Section 4.5 states that the guarantees extend to any symmetric relationship function for which R(X) is PSD, including RBF and polynomial kernels. But cosine similarity, listed as a relationship function in §3.2 and discussed in the Lipschitz table in §4.5, does not generally produce PSD matrices. Cosine matrices are indefinite (e.g., for three vectors on a circle). Thus the rank/subspace and entry-wise results cannot be applied to cosine similarity without separate proof. The paper should either restrict the extension to PSD kernels or provide a dedicated analysis for cosine.","section":"§4.5 and §3.2"},{"comment":"The experiments do not verify the assumptions of the theoretical guarantees. Theorem 4.4 requires ε<σ_r^4(X) and k≥r. The paper compresses 1024-D ViT embeddings to 768-D and 256-D without reporting r or ε. For real image embeddings, r may well exceed 256, in which case the rank-preservation guarantee cannot hold. The reported performance differences (R@1 0.466 vs 0.464) are also within the range one would expect from noise, and no error bars or standard baselines (PCA, MDS, kPCA) are provided. This section does not substantiate the claim that RPL preserves vector-space structure in practice.","section":"§5.1, Tables 2–3"},{"comment":"The claim that 'the regime m=Θ(n^2)' is reached after a handful of epochs is not justified. Serfling's inequality as applied here governs a single fixed sample S drawn uniformly without replacement. In Algorithm 1, mini-batches are drawn per iteration, and across epochs the same pair can appear in multiple mini-batches. The cumulative number of distinct observed pairs is not m, and the probability statement in Lemma 4.1 does not transfer to the union of mini-batches without additional arguments. This weakens the scalability justification for the loss.","section":"§4.1, Lemma 4.1 and 'Practical corollary'"}],"minor_comments":[{"comment":"The paper says in the introduction that 'full experimental analysis will follow in future work' but Section 5 is titled 'Experimental Validation'. This inconsistency should be resolved, perhaps by explicitly labeling the experiments as preliminary sanity checks.","section":"General"},{"comment":"The term 'operator-Lipschitz' is used for functions g:R^{n×n}->R, but the standard notion applies to matrix-valued functions. The coordinate functions g_ij(M)=M_ij are in fact 1-Lipschitz in the spectral norm, so the proof is correct, but the terminology and notation should be clarified.","section":"§4.2"},{"comment":"The entry for PCA under 'Orthogonality' says 'Partial' while 'Linear Independence' says 'Yes'. Since PCA is an orthogonal projection, it can make non-orthogonal vectors orthogonal after projection; the meaning of 'partial' should be defined to avoid confusion.","section":"Table 1"},{"comment":"The qualitative claims about 'angular coherence' and 'foldovers' are not quantified. A simple measure (e.g., correlation of pairwise angles or nearest-neighbor agreement) would make the comparison informative.","section":"§5.2, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The central theoretical claim (Theorem 4.4) is false as stated, and the error propagates to Theorem 4.5. This is a load-bearing defect, not a presentation issue. A correction is possible in principle—e.g., replacing the assumption k≥r with k=r and fixing the eigenvalue index—but that would narrow the applicability and would not cover the reported 1024→256 compression unless the data rank is verified. I recommend major revision rather than outright rejection because the loss formulation and orthogonality bound may still be salvageable, but the authors must either fix the rank theorem and its assumptions or explicitly retract the rank/subspace claims. The experiments need error bars, baselines, and verification of ε and r."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the stress-test note is right. Theorem 4.4 has an eigenvalue-index error. For a rank-r PSD matrix with r < n, the smallest positive eigenvalue is λ_r, not λ_{n−r+1}; λ_{n−r+1} is zero when r < n. The Weyl step therefore proves nothing about rank(Y) = r. Concrete counterexample: X = [1; 0] (n=2, d=1, r=1), Y = [[1, 0], [0, δ]] with δ < 1. Here ε = δ^4 < 1 = σ_1^4(X), yet rank(Y) = 2, not 1. The theorem can be fixed by assuming k = r (or explicitly constraining rank(Y) ≤ r), but that is not the claim the paper makes, and it doesn't cover the 1024→256 compression experiments unless the true rank of the ViT embeddings is ≤ 256—which the paper never reports. Theorem 4.5 inherits the problem because it assumes both ranges have dimension r.\n\nWhat is genuinely worth keeping: the RPL framework is clear and flexible—arbitrary relationship functions, discrepancy choices, masking, and minibatch neural training. Corollary 4.3, the entry-wise orthogonality bound from operator Lipschitzness, is correct and useful. The Serfling transfer argument is a reasonable way to connect minibatch loss to full Gram error, even if the constants are loose. The paper is honest that the experiments are preliminary, and the citation pattern looks standard and appropriate. I don't agree with the reader report's complaint about cosine: a cosine matrix of nonzero vectors is a Gram matrix of normalized vectors, hence PSD.\n\nSoft spots beyond the theorem: the experiments have no error bars, no comparison against PCA/MDS/kPCA or simple linear baselines, and the masking variants labeled “Linear” and “Gaussian” in Table 3 are never defined in Section 3. The rank and subspace guarantees are also conditional on ε being small, which is the training objective itself, so they read more as sensitivity analysis than independent prediction. That is not fatal for a framework paper, but it should be stated more modestly.\n\nWho this is for: people working on embedding compression, retrieval, or Gram-matching losses who want a clear design-space overview and a cautionary example of why eigenvalue bookkeeping matters. As a validated method, it isn't there yet.\n\nRecommendation: this deserves a serious referee, not a desk reject. The error is identifiable and fixable, and the framework has practical value. Acceptance should be conditional on correcting Theorems 4.4 and 4.5, adding proper baselines and error bars, and checking whether the theoretical conditions actually hold in the reported experiments.","headline":"The RPL framework is a readable, useful extension of Gram-matrix matching, and the orthogonality bound is fine, but the headline rank-preservation theorem is false as stated and the experiments don't validate the assumptions.","tokens_in":820,"tokens_out":1014,"would_cite":false,"duration_ms":65092,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["15A42","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The Relationship Preserving Loss (RPL) trains neural networks to compress high-dimensional data while provably preserving orthogonality, rank, and subspace structure, with all distortions bounded by the training error ε.","keywords":["dimensionality reduction","relationship preserving loss","Gram matrix","orthogonality preservation","rank preservation","subspace structure","matrix perturbation theory","embedding compression"],"falsifier":"Take a data matrix whose Gram matrix has known rank r, choose k = r, train RPL until ε is far below σ_r^4, then check whether (i) rank(f(X)) = r, (ii) |Y_i·Y_j| ≤ √ε for every originally orthogonal pair, and (iii) the measured subspace angle respects sin Θ ≤ √ε/σ_r^2. A violation of any of these on a held-out sample would falsify the claimed guarantees.","tokens_in":5850,"feed_emoji":"📐","tokens_out":10565,"duration_ms":114343,"temperature":0.7,"pith_summary":"This paper proposes the Relationship Preserving Loss (RPL), a training objective that asks a neural network to make the Gram (or other pairwise relationship) matrix of its low-dimensional output match that of the original high-dimensional data. The central claim is that once this loss ε is small, vector space structure is provably retained: originally orthogonal vectors stay nearly orthogonal, the output matrix keeps the full rank of the input when the target dimension is large enough, and the linear span of the embedded points stays close to the original span. These guarantees come from matrix perturbation inequalities, not from heuristics, and hold for any positive semidefinite relationship function. The paper reports initial retrieval experiments where 1024-dimensional ViT embeddings compressed to 256 dimensions retain cross-modal recall, suggesting the preserved structure is practically useful. The authors themselves frame the experimental part as preliminary; the contribution is the mathematical framework.","feed_headline":"New loss provably preserves orthogonality and rank","feed_subtitle":"Matrix perturbation theory ties the loss value to geometric fidelity, so compressed embeddings stay faithful.","key_machinery":"The central object is the relationship matrix R(X), whose (i,j) entry is φ(x_i, x_j); for dot-product φ this is the Gram matrix of pairwise dot products. The scalar ε = ||R(X) − R(f(X))||_F^2 is the bridge: matrix perturbation theory converts a small ε into geometric consequences, using an operator-Lipschitz argument for entrywise bounds, an eigenvalue perturbation inequality for rank, and the sin-theta theorem for subspace angles. Because training observes only a subset of pairs, a sampling inequality for sums without replacement is used to connect the observed mini-batch error to the global ε.","core_discovery":"For dot-product φ, the relationship matrix is the Gram matrix—the matrix of all pairwise dot products. On the paper's terms, minimizing ε = ||R(X) − R(f(X))||_F^2 over the Gram relationship R yields: entrywise preservation (if X_i·X_j = 0 then |Y_i·Y_j| ≤ √ε), rank preservation (if k ≥ r and ε < σ_r^4(X) then rank(Y) = r), and subspace preservation (the largest principal angle between the ranges of the two Gram matrices satisfies sin Θ ≤ √ε/σ_r^2(X)). The proofs are short applications of the eigenvalue perturbation inequality and the sin-theta bound for eigenspaces, together with a sampling inequality that transfers a mini-batch estimate of ε to the full data. The same framework extends to a","pith_inferences":["Because the guarantees require k ≥ rank(X), any use of RPL below the intrinsic dimension must be understood as preserving only the top-k subspace; checking the spectrum of G_X before compressing would make the claim testable.","The loss's rotational and reflection invariance means RPL outputs may be mirrored; systems consuming these embeddings should be trained in an orientation-invariant way, or a canonical alignment step must be added.","The mini-batch analysis implies the effective number of observed pairs matters; reporting the final ε, not just downstream metrics, would let users verify whether the small-error regime was actually reached.","A direct stress test—on synthetic data with known rank, train until ε < σ_r^4 and check rank and sin Θ—would quantify how tight the bounds are end-to-end and how quickly training reaches the guaranteed regime."],"forward_implications":["If the loss is driven small, any two orthogonal input vectors remain nearly orthogonal in the embedding, with deviation at most √ε.","Whenever the target dimension is at least the data's rank and ε is below the fourth power of the smallest positive singular value, the embedding preserves the rank, so no independent directions are collapsed.","The subspace angle bound means the linear span of the embedded points is close to the original span, so linear classifiers and nearest-neighbor retrieval directions should transfer to the compressed space.","The guarantees extend to any positive semidefinite relationship function (e.g., RBF kernels), so the effective dimension of the kernel feature space is preserved, not just the ambient Gram structure.","In the reported experiments, compressing ViT-H image embeddings from 1024 to 256 dimensions with RPL keeps cross-modal recall close to the uncompressed baseline, and top-k masking slightly improves it."],"supporting_citations":[{"why":"Supplies the sampling inequality that transfers the observed mini-batch error to the global error ε.","marker":"[11]"},{"why":"Provides the sin-theta bound on eigenspace perturbation that yields the subspace angle guarantee.","marker":"[6]"},{"why":"Supplies the pre-trained cross-modal embeddings (ViT-H/ViT-L) used in the compression experiments.","marker":"[9]"},{"why":"Defines the kernel relationship-matrix approach that RPL extends to arbitrary PSD relationship functions.","marker":"[10]"},{"why":"The classical multidimensional scaling formulation that RPL generalizes by replacing eigendecomposition with a differentiable loss.","marker":"[3]"}],"fun_headline_variants":["Loss provably preserves orthogonality and rank","Keep vector space properties in low-dim embeddings","Gram-based loss ensures geometry stays faithful","Dimension reduction without geometric distortion","Theoretical support for relationship-preserving loss"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The guarantees only hold when the target dimension is at least the rank of the data and when training actually achieves a sufficiently small loss; neither condition is verified for the reported real-image experiments.","fun_headline_variants_meta":{"raw":{"variants":["Loss provably preserves orthogonality and rank","Keep vector space properties in low-dim embeddings","Gram-based loss ensures geometry stays faithful","Dimension reduction without geometric distortion","Theoretical support for relationship-preserving loss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1362,"prompt_tokens":714,"completion_tokens":648,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":585}},"tokens_in":458,"tokens_out":648,"duration_ms":7555,"temperature":1.0,"reasoning_tokens":585,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:47:09.079871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a data matrix whose Gram matrix has known rank r, choose k = r, train RPL until ε is far below σ_r^4, then check whether (i) rank(f(X)) = r, (ii) |Y_i·Y_j| ≤ √ε for every originally orthogonal pair, and (iii) the measured subspace angle respects sin Θ ≤ √ε/σ_r^2. A violation of any of these on a held-out sample would falsify the claimed guarantees.","supporting_citations":[{"cited_title":"Serfling","cited_arxiv_id":null,"evidence_quote":"Supplies the sampling inequality that transfers the observed mini-batch error to the global error ε."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sin-theta bound on eigenspace perturbation that yields the subspace angle guarantee."},{"cited_title":"Smola, and Klaus-Robert M¨ uller","cited_arxiv_id":null,"evidence_quote":"Defines the kernel relationship-matrix approach that RPL extends to arbitrary PSD relationship functions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The classical multidimensional scaling formulation that RPL generalizes by replacing eigendecomposition with a differentiable loss."}],"review_version":1}