{"id":"39092ef7-8e4d-4b8a-92b4-ffebcbec718e","arxiv_id":"2607.07047","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":7,"one_line_summary":"Aggregating per-token pullback metrics via the Fréchet mean on the SPD manifold outperforms Euclidean mean pooling for sentence classification, with most of the gain attributable to geometric aggregation rather than learned encoder structure.","lead":"This paper replaces standard averaging of language model token embeddings with a geometric aggregation on the SPD manifold, showing improved sentence classification on three benchmarks. A smart generalist might read it to understand whether the internal geometry of AI representations carries information that flat averaging discards.","discovery_kind":"unclear","skeptic_critique":{"model":"glm-5.2","headline":"RMP's gain over Euclidean pooling may stem from dimensionality expansion (768-dim vectors → 2080-dim SPD features) rather than Riemannian geometry; no dimensionality-controlled baseline is reported.","rationale":"The reader identified the right general area—the random-encoder result is the most fragile part of the argument—but framed it as a philosophical question about whether the metric encodes 'genuine geometric structure.' The more precise and load-bearing concern is a specific confound: RMP operates on 2080-dimensional representations while the baselines operate on 768 or 64 dimensions. This is not an abstract worry; it is a standard machine-learning confound that the paper does not address. The paper's ablations are thoughtful (trained vs random encoder, MLP vs linear projection), but none of them controls for the dimensionality of the final classification input. The FEVER-Symmetric negative control does not resolve this: if the dimensionality expansion provides no signal on a dataset with no signal, that is expected regardless of whether the mechanism is geometric or dimensional. The paper is honest about limitations (§6.2) and the experimental design is otherwise careful, which is why the verdict should be CONDITIONAL rather than REJECT—the concern is testable and the authors may already have the machinery to run it. But until a dimensionality-matched baseline is reported, the central claim that 'geometric aggregation extracts signal that flat-Euclidean pooling discards' is not fully supported, because the comparison is not apples-to-apples in feature count.","tokens_in":17523,"tokens_out":2184,"duration_ms":109889,"concrete_test":"Add a baseline: random-project BERT tokens to 64-dim (same as IGL encoder), compute per-token outer products v_i v_i^T ∈ R^{64×64}, take the arithmetic (Euclidean) mean across tokens, vectorize the upper triangle (2080 features), and classify with the same logistic regression used for the Linear Probe. Run on all four datasets with the same CV protocol. If this Euclidean-outer-product baseline matches or exceeds RMP on CoLA, CREAK, and RTE, the Fréchet mean and SPD-manifold operations are not the source of the gain—dimensionality and second-order statistics are. If RMP significantly outperforms this baseline, the Riemannian structure genuinely contributes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Riemannian Mean Pooling extracts geometric signal that Euclidean mean pooling discards. But the comparison is confounded by dimensionality. The Linear Probe baseline classifies on 768-dimensional mean-pooled vectors. RMP classifies on tangent-space projections of 64×64 SPD matrices, yielding d(d+1)/2 = 2080 unique features per sentence. The Encoder-Only ablation (64-dim Euclidean) is even lower-dimensional at 64 features. So RMP has ~2.7× more features than the Linear Probe and ~32× more than Encoder-Only. With logistic regression and no regularization (as stated for the Linear Probe in §3.2), higher dimensionality alone can improve classification, especially on small datasets (CoLA: 3000, CREAK: 3000, RTE: 2490 samples). The random-encoder ablation is consistent with this confound: any random nonlinear projection to 64-dim, converted to 2080 SPD features, would be expected to outperform 768-dim Euclidean averaging purely from the feature expansion. The paper frames the random-encoder result as evidence that 'geometric aggregation' carries the signal (§5.3, §6.1), but it does not distinguish between (a) Riemannian/Fréchet structure being essential and (b) second-order statistics in a higher-dimensional space being sufficient. A Euclidean baseline that computes the same 64×64 outer-product matrices (v_i v_i^T after random projection), averages them arithmetically (not Fréchet), vectorizes the upper triangle to 2080 features, and classifies with the same logistic regression would isolate whether the Fréchet mean and SPD-manifold operations contribute beyond the dimensionality increase. The reader's concern about 'genuine geometric structure vs arbitrary nonlinear feature' is related but less precise; the specific, testable confound is dimensionality.","agreement_with_reader":"partial"},"referee_report":{"model":"glm-5.2","summary":"The paper proposes Riemannian Mean Pooling (RMP), which aggregates per-token pullback metrics extracted from a learned encoder's Jacobian via the Fréchet mean on the SPD manifold. The method is evaluated on four binary classification datasets (CoLA, CREAK, RTE, and FEVER-Symmetric as a negative control) using BERT-base layer 9 embeddings. The authors find that RMP outperforms Euclidean mean pooling on the three signal-bearing datasets while remaining at chance on the negative control. Ablations using randomly initialized encoders suggest that the geometric aggregation, rather than learned manifold structure, accounts for most of the gain. The experimental design is methodologically careful, featuring controlled comparisons on identical embeddings, multi-seed cross-validation, permutation testing, and a negative control.","tokens_in":17736,"tokens_out":1029,"duration_ms":142119,"significance":"The paper addresses a well-motivated question about the geometric structure of PLM embeddings. Its primary strength is the rigorous experimental design: the controlled comparison isolates the aggregation step, the FEVER-Symmetric negative control tests for artifact exploitation, and the random-encoder ablation is a genuinely informative test of whether learned manifold structure is necessary. The finding that random encoders suffice for the geometric gain on two of three datasets is a valuable and honest negative result that refines the claim from 'embeddings lie on a learned manifold' to 'embeddings exhibit locally manifold-like structure.' However, the central claim is currently confounded by a dimensionality mismatch between the RMP pipeline and the baselines, which must be addressed before the contribution can be considered fully sound.","major_comments":[{"comment":"§3.2, §3.5, Table 1: The comparison between RMP and the Linear Probe baseline is confounded by a large difference in feature dimensionality. The Linear Probe classifies on 768-dimensional mean-pooled vectors, while RMP classifies on tangent-space projections of 64×64 SPD matrices, yielding d(d+1)/2 = 2080 features. With logistic regression and no regularization (as stated for the Linear Probe in §3.2), higher dimensionality alone can improve classification on these small datasets (CoLA: 3000, CREAK: 3000, RTE: 2490). The random-encoder ablation results (§5.3) are also consistent with this confound: any random nonlinear projection to 64-dim, converted to 2080 SPD features, could outperform 768-dim Euclidean averaging purely from feature expansion. The paper frames the random-encoder result as evidence that 'geometric aggregation' carries the signal, but it does not distinguish between (a)","section":null},{"comment":"§5.3, §6.1: To support the claim that the gain comes from Riemannian/Fréchet structure rather than second-order statistics in a higher-dimensional space, a dimensionality-controlled baseline is needed. Specifically, a Euclidean baseline that computes the same 64×64 outer-product matrices (e.g., v_i v_i^T after random projection), averages them arithmetically (not Fréchet), vectorizes the upper triangle to 2080 features, and classifies with the same logistic regression would isolate the contribution of the Riemannian operations from the contribution of dimensionality expansion. Without this control, the central claim that 'geometric aggregation extracts signal that flat-Euclidean pooling discards' is not fully established.","section":null}],"minor_comments":[{"comment":"§3.2: The Linear Probe is stated to use 'no regularisation' with the LBFGS solver. If the dimensionality-controlled baseline suggested above is added, regularization should be matched carefully across all compared methods to ensure a fair comparison.","section":null},{"comment":"Figure 1: The schematic is helpful, but the distinction between the 'Product Green's kernel' and 'Variable projection' boxes in the training path could be clearer for readers unfamiliar with IGL.","section":null},{"comment":"§3.4, Eq. (6): The two-stage regularization (epsilon=1e-6 and lambda=1e-2) is mentioned, but the sensitivity to lambda is noted as unverified on signal-bearing datasets (§6.2). A brief note on the stability of the results to this choice would strengthen the manuscript.","section":null}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern regarding dimensionality confounding is well-founded and is the primary reason for the major revision recommendation. The paper's framing of the random-encoder result as evidence for 'geometric signal' is currently too strong given that the feature expansion alone could explain the gain. If the authors add the dimensionality-controlled baseline and the Riemannian gain persists, the paper would be a solid contribution. If the gain disappears, the paper would need to be reframed substantially. The reliance on IGL (Quemy 2026), co-authored by one of the present authors, is transparently disclosed and the ablations appropriately test its necessity, so I do not see a circularity issue there."},"author_rebuttal":null,"desk_editor":{"model":"glm-5.2","letter":"The paper applies SPD-manifold techniques from the BCI literature to BERT token embeddings: extract a per-token pullback metric from an encoder's Jacobian, aggregate via the Fréchet mean, classify in the tangent space. On CoLA, CREAK, and RTE, this beats Euclidean mean pooling by 2-6 AUC points. On FEVER-Symmetric (a debiased negative control), it stays at chance. The experimental hygiene is good: identical embeddings across conditions, multi-seed CV, permutation testing, and ablations that isolate the encoder from the aggregation step. The negative control working as intended is a real plus. The random-encoder ablation is the most interesting result: a randomly initialized MLP with Fréchet aggregation already beats Euclidean pooling on two of three datasets. The paper frames this as evidence that geometric aggregation carries the signal. But there's a confound the paper doesn't address. The Linear Probe baseline classifies on 768-dimensional mean-pooled vectors. RMP classifies on vectorized upper triangles of 64×64 SPD matrices, which is 2080 features. That's roughly 2.7× more features than the baseline, with logistic regression and no regularization. Higher dimensionality alone can improve classification on datasets this small (2490-3000 samples). The random-encoder result is fully consistent with this confound: any random nonlinear projection to 64 dimensions, converted to 2080 second-order features, would be expected to outperform 768-dim arithmetic averaging purely from feature expansion. The missing experiment is straightforward: take the same 64-dim projected tokens, compute outer products, average them arithmetically (not Fréchet), vectorize to 2080 features, classify with the same logistic regression. If that Euclidean second-order baseline matches RMP, the Fréchet mean and SPD-manifold operations aren't contributing beyond the dimensionality increase. If RMP still wins, the geometric claim holds. The paper is honest about limitations and doesn't overclaim about global manifold structure. But the central claim—that Riemannian aggregation extracts geometric signal—rests on a comparison that doesn't control for feature dimensionality. That's the experiment a referee should request. The reader's concern about 'genuine geometric structure vs. arbitrary nonlinear feature' is related but less precise; the specific testable confound is dimensionality. This deserves a serious referee. The experimental design is careful enough that the missing baseline is likely an oversight rather than evasion, and adding it would either strengthen or appropriately weaken the claim.","headline":"Riemannian SPD aggregation of BERT token embeddings beats Euclidean pooling, but the gain may be a dimensionality artifact rather than a geometric one.","tokens_in":18634,"tokens_out":600,"would_cite":false,"duration_ms":157955,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"glm-5.2","headline":"Curved pooling beats flat averaging on BERT embeddings","keywords":[],"falsifier":"If a non-learned, fixed nonlinear transformation (e.g., a fixed random matrix followed by a fixed nonlinearity) combined with Fréchet aggregation produces the same gains as the trained-encoder RMP on CoLA, CREAK, and RTE, then the contribution of learned manifold structure is zero and the method reduces to a deterministic feature-engineering step with no dependence on the data manifold's actual geometry. Conversely, if RMP fails to outperform Euclidean pooling on datasets beyond these three, the geometric advantage may be an artifact of specific dataset properties rather than a general signal.","tokens_in":17714,"feed_emoji":"🌀","tokens_out":1362,"duration_ms":137755,"temperature":0.7,"pith_summary":"The paper asks whether sentence-level classification signal in pre-trained language model embeddings lives in their local Riemannian geometry rather than in their flat Euclidean coordinates. To test this, the authors extract a per-token pullback metric—a symmetric positive definite (SPD) matrix capturing how a learned encoder locally stretches and compresses the embedding space at each token—by computing the analytical Jacobian of the encoder at that token's embedding. They aggregate these per-token SPD matrices into a single sentence-level descriptor using the Fréchet mean on the SPD manifold, a procedure they call Riemannian Mean Pooling (RMP), then classify in the tangent space at the population mean. On three signal-bearing datasets (CoLA, CREAK, RTE), RMP outperforms standard Euclidean mean pooling under a controlled comparison on identical BERT embeddings, while on FEVER-Symmetric—a dataset constructed to remove annotation artifacts—RMP correctly stays at chance. The central discovery comes from the ablations: a randomly initialised, untrained encoder combined with Fréchet aggregation already beats Euclidean pooling on two of three signal-bearing datasets. This localises most of the gain to the geometric aggregation step itself—the Fréchet mean on the SPD manifold—rather than to any learned manifold structure in the encoder. The trained encoder contributes additional signal only on CREAK, the most knowledge-heavy dataset, suggesting that learned geometric structure matters specifically where classification depends on world knowledge rather than local linguistic form.","feed_headline":"Curved pooling beats flat averaging on BERT embeddings","feed_subtitle":"Aggregating per-token metrics on the SPD manifold extracts signal that Euclidean mean pooling discards—and the gain comes from the geometry,","key_machinery":"The pipeline has two stages. In the training stage, a small MLP encoder maps 768-dimensional BERT token embeddings to a 64-dimensional latent code, trained via Intrinsic Green's Learning (IGL)—an inverse-PDE formulation where a closed-form Green's-function kernel readout replaces the decoder, forcing the encoder to find coordinates aligned with the data's smooth structure. In the inference stage, the trained encoder's analytical Jacobian J(v) = ∂Ψ/∂v at each token embedding v yields a pullback metric g(v) = (JJ^T + εI)^{-1}, an SPD matrix capturing local stretching/compression of the embedding space. Per-token SPD matrices are aggregated via the Fréchet mean—the point on the SPD manifold最小化总","core_discovery":"The Fréchet mean of per-token pullback metrics on the SPD manifold extracts classification signal from BERT embeddings that Euclidean mean pooling discards, and this gain is attributable primarily to the geometric aggregation operation itself rather than to learned manifold structure in the encoder. A randomly initialised encoder's Jacobian already produces metrics that, when aggregated via the Fréchet mean, outperform Euclidean pooling on two of three signal-bearing datasets, because the pullback metric captures local anisotropy of the token's position under any smooth nonlinear transformation—even a random one—and the SPD aggregation preserves this position-dependent structure that Euclide","pith_inferences":["The random-encoder result raises the possibility that RMP is not probing 'the Riemannian geometry of the data manifold' but rather exploiting the fact that any smooth nonlinear transformation of a high-dimensional vector produces a position-dependent SPD matrix whose Fréchet mean is a more informative summary than the arithmetic mean. The 'geometric signal' may be a deterministic feature-engineeri","If the pullback metric of a random encoder already encodes token position through local Jacobian anisotropy, then the method may be closely related to random-feature methods (random kitchen sinks, extreme learning machines) where a nonlinear projection into a richer representation space improves separability—reinterpreted through the lens of SPD-manifold aggregation.","The FEVER-Symmetric result where the trained encoder + Euclidean aggregation detects residual signal (0.553 AUC) but the full geometric pipeline stays at chance suggests the pullback metric acts as a low-pass filter on content features, which could be either a bug (information loss) or a feature (built-in debiasing) depending on the application."],"forward_implications":["If geometric aggregation rather than learned manifold structure drives the gain, then any differentiable encoder—not just IGL—could serve as a metric source, and the specific training objective may be largely irrelevant for most tasks.","The finding that random encoders suffice suggests the signal lies in the nonlinear transformation of token position itself, opening the question of whether simpler deterministic nonlinear maps (fixed random features, kernel functions) could replace the encoder entirely.","The trained encoder's contribution specifically on CREAK (knowledge-heavy) versus CoLA/RTE (linguistic structure) suggests a division of labor: geometric aggregation captures local linguistic form, while learned coordinates capture world-knowledge content.","If the pullback metric filters content features by construction (as argued for FEVER-Symmetric), then RMP could serve as a debiasing tool: a representation that preserves linguistic structure while discarding specific knowledge content.","Extending to other architectures (decoder-only LLMs, different layers) would test whether the geometric-aggregation gain is specific to BERT layer 9 or reflects a general property of transformer token spaces."],"fun_headline_variants":["Riemannian pooling extracts signal Euclidean averaging discards from BERT","Fréchet mean on SPD manifold beats Euclidean pooling for BERT embeddings","Geometric aggregation—not learned structure—drives BERT pooling gains","Random encoder + Fréchet pooling already beats Euclidean mean on BERT","SPD manifold aggregation captures BERT signal Euclidean pooling misses"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The pullback metric g(v) = (JJ^T + εI)^{-1} is assumed to encode meaningful local geometric structure of the embedding space that carries classification signal. But the random-encoder ablation shows that even a random MLP's Jacobian produces metrics that beat Euclidean pooling, which means the 'geometric signal' may simply be a deterministic nonlinear transformation of token position whose SPD aggregation happens to be a useful feature—raising the question of whether this isR","fun_headline_variants_meta":{"raw":{"variants":["Riemannian pooling extracts signal Euclidean averaging discards from BERT","Fréchet mean on SPD manifold beats Euclidean pooling for BERT embeddings","Geometric aggregation—not learned structure—drives BERT pooling gains","Random encoder + Fréchet pooling already beats Euclidean mean on BERT","SPD manifold aggregation captures BERT signal Euclidean pooling misses","Pullback metrics with Fréchet mean outperform Euclidean pooling on BERT","Geometry of pooling extracts BERT signal, not learned manifold structure","Fréchet aggregation on random Jacobians beats Euclidean pooling","Riemannian Mean Pooling recovers BERT signal Euclidean mean discards","Curved pooling on SPD manifold beats flat averaging for BERT embeddings"]},"model":"glm-5.2","effort":"low","cost_usd":0.0,"raw_usage":{"total_tokens":1938,"prompt_tokens":495,"completion_tokens":1443,"prompt_tokens_details":null},"tokens_in":495,"tokens_out":1443,"duration_ms":54597,"temperature":1.0,"reasoning_tokens":1261,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-09T21:00:07.914158+00:00","model_set":{"reader":"glm-5.2"},"falsifier":"If a non-learned, fixed nonlinear transformation (e.g., a fixed random matrix followed by a fixed nonlinearity) combined with Fréchet aggregation produces the same gains as the trained-encoder RMP on CoLA, CREAK, and RTE, then the contribution of learned manifold structure is zero and the method reduces to a deterministic feature-engineering step with no dependence on the data manifold's actual geometry. Conversely, if RMP fails to outperform Euclidean pooling on datasets beyond these three, the geometric advantage may be an artifact of specific dataset properties rather than a general signal.","supporting_citations":[],"review_version":1}