{"id":"5be35088-6fdd-4e91-9df9-f1b1bb2d1882","arxiv_id":"2508.19683","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Applying Topological Uncertainty to hidden-layer activations of a trained FNN detects failed neutron-star EoS inferences with over 90% success in the best-tested configuration.","lead":"This paper tests whether a topological measure of a trained neural network's internal structure can flag when the network's neutron-star equation-of-state inference fails. It reports anomaly detection success rates above 90% for some network designs, using only the network's own hidden-layer activations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The >90% best-case success rate is selected over architectures and δ on the same test set without repeated seeds; it may not be a stable estimate.","rationale":"The reader's weakest assumption focuses on separability and generalization of the Fréchet-mean/TU classifier. My stress-test concern is closely related but more specific: the claimed >90% success rate is a selected maximum over architectures and tolerance parameters, evaluated on the same test set used for reporting, without repeated training seeds or a validation-based model-selection protocol. This makes the quantitative headline insufficiently supported, even though the underlying method is coherent and the figures suggest a real signal. The paper's own results show strong architecture dependence (F(2,2,2,2) failing on R_{k=1}), so the 'best case' should not be treated as representative without a selection protocol. A multi-seed, held-out validation experiment would settle whether the >90% figure is reproducible or an artifact of selection/luck. This does not require rejecting the paper; it requires conditioning the main claim on such robustness checks, hence the same CONDITIONAL verdict as the reader.","tokens_in":14709,"tokens_out":8195,"duration_ms":101444,"concrete_test":"Retrain F(4,4,2,2) and F(2,2,2,2) with 10 random seeds. Split data into train/validation/test; select δ and architecture only on validation using Eq. (25); then report mean ± std of R_tot, R_{k=0}, R_{k=1} on the untouched test set, along with the natural-base-rate R_tot computed with the true k=1 fraction. If the selected best-case mean drops below 90%, or seed-to-seed std exceeds about 5 percentage points, the headline claim should be softened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central numerical claim rests on R_tot(δ) for the best configuration F(4,4,2,2) at a favorable δ (Fig. 6, bottom), but every point in Fig. 6 comes from a single training run, and the δ/architecture choice is not made on a separate validation set. The paper itself shows that F(2,2,2,2), with comparable train/val loss, has drastically worse R_{k=1} (Fig. 7, bottom-left), so performance is highly sensitive to hyperparameters in an unexplained way. Without error bars or a validation-based selection protocol, the 'exceeds 90%' claim could be an artifact of test-set peeking or seed luck. Moreover, the reported R_tot uses a balanced 500/500 test set; under the natural base rate the same per-class rates would yield a lower overall success rate, since k=1 is only 10–20% of data for many δ. These issues directly affect the quantitative headline claim, even though the qualitative observation that TU carries information about inference failure is plausible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes using Topological Uncertainty (TU), a persistent-homology-based summary of a trained feedforward network's edge-weight distributions, as a post-hoc anomaly detector for neural-network inference of the neutron-star equation of state. The authors label training inputs as successful (k=0) or unsuccessful (k=1) according to whether the L_infinity logarithmic-pressure deviation between predicted and true EoS is below a tolerance delta (Eqs. (19)-(20)), construct per-class Frechet means of sorted MST edge weights (Eq. (10)), and define cross-TU distances (Eqs. (11),(23)). An input is flagged anomalous if it is closer to the k=1 Frechet mean. On a held-out mock dataset balanced at 500 per class, they report detection rates exceeding 90% for the best architecture F(4,4,2,2) and favorable delta, and they study dependence on delta and architecture.","tokens_in":14996,"tokens_out":8162,"duration_ms":93189,"significance":"If the result holds, the paper offers a cheap, architecture-agnostic way to flag unreliable neural predictions, which is of genuine value for EoS inference and for physics surrogate models more broadly. The exposition of the TU construction is clear, the numerical setup is coherent, and the use of Wasserstein distances between activation-graph persistence distributions is a sensible extension of the TU idea. However, the quantitative evidence for the headline claim is currently thin: there are no repeated training runs or error bars, the best delta and architecture are selected on the same test set, and no comparison is made with output-based uncertainty baselines. The paper's claimed robustness is not supported by its own results, and the novelty claim needs to be calibrated against Ref. [50]. These issues are load-bearing for the central 'exceeds 90%' statement, though the underlying method appears plausible.","major_comments":[{"comment":"The headline claim that the anomaly-detection success rate exceeds 90% rests on a single training run per architecture and on selecting delta and F(.) after looking at the same test set. No repeated seeds, confidence intervals, or validation-based model-selection protocol are provided. The strong sensitivity of the result is visible in the paper itself: F(2,2,2,2) and F(4,4,2,2) have nearly equal training/validation losses (Table II), but their R_{k=1}(delta) curves in Fig. 6 are dramatically different, and the R_{k=1} histogram for F(2,2,2,2) in Fig. 7 shows almost no separation. The reported 'best case' 90% could therefore be a seed or delta-selection artifact. Please provide repeated training runs and either a pre-registered validation selection or per-seed error bars, or temper the quantitative claim.","section":"Sec. III C, Figs. 6 and 7"},{"comment":"R_tot is computed on a test set with 500 examples per class, which is not the natural base rate. For delta > 0.2, Fig. 4 shows RLQ(delta) >= 0.8, meaning the k=1 class is only 10-20% of a naturally sampled dataset. Weighting the same per-class detection rates by this base rate can lower the overall success rate considerably, so the balanced R_tot overstates what a user would experience in practice. Please report R_tot under the empirical class prior, or report ROC/AUC and per-class rates separately, and discuss how delta is chosen in an unsupervised setting.","section":"Sec. III B, Eqs. (22)-(25)"},{"comment":"The statement that this is the 'first successful application of the TU to Anomaly Detection' is overstated. Ref. [50] introduced Topological Uncertainty precisely for monitoring trained neural networks through persistence of activation graphs, and that framework already includes detecting inputs for which the network is unreliable. The present paper even cites Ref. [50] as the original TU proposal. The novelty should be framed as a first physics application or a first demonstration in EoS inference, not as the first anomaly-detection use of TU.","section":"Sec. I, Sec. IV, Ref. [50]"},{"comment":"The conclusion says the method is 'robust against moderate variations of hyperparameters', but this is contradicted by Fig. 6. For example, F(2,2,2,2), with train/val losses close to those of the best F(4,4,2,2), has R_{k=1} below F(2,2) at many delta values. The paper gives no mechanism for this non-monotonic architecture sensitivity. Either provide an explanation or soften the robustness claim to 'performance varies with architecture and delta, and the best configuration can exceed 90%'.","section":"Sec. IV vs Sec. III C"}],"minor_comments":[{"comment":"Typo: 'The number of the edges is |V| = d_l + d_{l+1}' should be 'number of vertices'. The edge count is |E| = d_l * d_{l+1}, as correctly stated immediately after.","section":"Sec. II B"},{"comment":"The Wasserstein-1 formula Dist(mu,nu) = (1/N) sum |w_i - w_i'| is only valid after sorting the edge weights within each MST. This is implicitly done later in Sec. III C ('in our convention of the ordering'), but Eq. (9) should state the sorting explicitly to avoid confusion.","section":"Sec. II B, Eq. (9)"},{"comment":"The notation 'Ntr = 3' is inconsistent with the previously defined truncation parameter Ntrun in Eq. (19). Please use Ntrun consistently.","section":"Sec. III B"},{"comment":"Minor typos: 'F(2,2,)' should be 'F(2,2)', and 'RLR should increase' should be 'RLQ'.","section":"Sec. III C"},{"comment":"The normalization of the plotted distribution (division by N_l = |V_l| - 1) is mentioned only in the text; the figure caption should state it explicitly for readability.","section":"Fig. 5"},{"comment":"Typo: 'classifiction' should be 'classification'.","section":"Sec. IV"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not provide code, data, or seed information. For a numerical claim at the 90% level in a deep-learning setup, this is a reproducibility concern that should be addressed as part of the revision. The abstract's novelty claim should also be checked carefully against Ref. [50] before publication; as written it is likely to draw criticism from the TDA/ML community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Kenji and Syo apply Topological Uncertainty (an existing TDA-based method from Lacombe et al.) to flag failed EoS inferences from their FNNEoS. The genuinely new pieces are the cross-TU construction, the labeled-subdataset evaluation protocol, and the concrete neutron-star test. They explain the TU machinery clearly and show that, for some architectures and tolerances, the TU separates successful from failed inferences reasonably well. The histograms in Fig. 7 are instructive. That is worth something.\n\nThe soft spots are mainly statistical. Every point in Fig. 6 comes from a single training run; there are no error bars and no repeated seeds. The '>90%' headline is selected over architectures and delta on the same test set, so it is not a stable estimate. The paper itself shows F(2,2,2,2), with comparable loss, does much worse on R_{k=1}, so the claim is sensitive to hyperparameters in an unexplained way. Also, R_tot is computed on a balanced 500/500 test set; under the natural base rate (k=1 is 10-20% for many delta) the overall success would be lower. The self-referential element--labels and evaluation share the same delta and the Frechet means come from the same trained network--is a mild concern but not a fatal one; the evaluation against ground-truth labels on held-out data prevents definitional circularity.\n\nThe novelty claim overstates prior work: Ref [50] already used TU for anomaly detection in classification. What is new here is the physics application and the evaluation design, not the concept.\n\nIs the paper worth a serious referee? Yes. The method is interesting, the physics setup is sensible, and the issues are addressable. But the authors need to run multiple seeds, report variances, and choose delta and architecture on a validation set, not the test set. They should also report results under the natural base rate. As is, I would not trust the '>90%' number, but I would trust the qualitative conclusion that TU carries information about inference failure. Conditional accept for me.","headline":"A plausible but not yet robust demonstration of TU for anomaly detection in EoS inference; the qualitative signal is real, the 90% number is not.","tokens_in":15444,"tokens_out":2271,"would_cite":false,"duration_ms":24606,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hidden-layer connection patterns in a trained network can flag failed neutron-star EoS inferences, reaching above 90% accuracy in the best case.","keywords":["Topological Uncertainty","Anomaly Detection","Feedforward Neural Network","Equation of State","Neutron Star","Persistent Homology","Maximum Spanning Tree","Wasserstein Distance"],"falsifier":"Run the same experiment many times: train F(4,4,2,2) from, say, 20 different random initializations and evaluate R_tot at δ=0.5 on an independent Dnew for each seed. If the reported >90% accuracy reproduces in every seed, the claim is supported; if any seed gives chance-level R_k=1, or if R_k=1 collapses when Dnew is sampled from a slightly shifted mass-radius distribution, the claimed generalization fails.","tokens_in":14625,"feed_emoji":"🌟","tokens_out":8962,"duration_ms":90239,"temperature":0.7,"pith_summary":"The paper argues that a trained feedforward network carries a readable trace of whether its own output is trustworthy, and that this trace can be extracted without seeing the correct answer. The testbed is the inverse problem of recovering the dense-matter equation of state from neutron-star mass-radius data, where the same network inference sometimes matches the true EoS and sometimes misses it badly. The authors label training samples by the actual error—k=0 for a close match, k=1 for a failure—and build a Topological Uncertainty for each class by converting each hidden layer's weighted connections into a maximum spanning tree, then comparing the sorted edge-weight distributions with a Wasserstein distance. Their central claim is that the class giving the smaller Topological Uncertainty is the correct one, so k=1 (anomalous) data are identified by having a smaller distance to the k=1 class average than to the k=0 one. On held-out data the rule achieves above 90% success in the best tested architecture, with performance depending on the network's width, depth, and tolerance parameter.","feed_headline":"Network geometry catches bad equation-of-state fits >90% of the time","feed_subtitle":"Topological patterns in hidden layers separate good from bad equation-of-state inference without needing the true answer.","key_machinery":"The maximum-spanning-tree edge-weight distribution is the TU's core object. For each layer, the trained weights and the input activation give an edge weight w(e)=|Wℓ(i,j)xℓ(i)|; edges are added in decreasing weight order to form the maximum spanning tree, and the sorted weights define a probability distribution μ(G). The Topological Uncertainty TU_k(X) is the layer-averaged p=1 Wasserstein distance between this distribution and the Fréchet mean distribution for class k, built from the labeled training subsets. The machinery turns the entire trained network into a per-class distance, so the lowest TU value acts as a predicted class label.","core_discovery":"On the paper's own terms, the discovery is that the cross-Topological-Uncertainty rule works: for held-out neutron-star inputs, an input is assigned to the class (normal or anomalous) whose Fréchet-mean MST weight distribution is closer in Wasserstein distance, and the assignment matches the true error label often enough to exceed 90% accuracy in the best architecture and tolerance range. The TU is a post-hoc quantity: it is computed from the already-trained weights by taking, for each layer, the edge weight |Wℓ(i,j)xℓ(i)|, forming the maximum spanning tree of the layer's bipartite graph, reading off the sorted edge weights as a probability distribution, and averaging the Wasserstein distanc","pith_inferences":["Editorial inference: the same construction should transfer to other regression FNNs—gravitational-wave parameter estimation or heavy-ion surrogate models—provided a calibration set with known errors can be labeled, and the paper's own conclusions gesture at this.","Editorial inference: since the paper trains each architecture only once, the above-90% figure is a point estimate; repeated-seed and distribution-shift tests would tell whether the separation of k=0/k=1 Fréchet means is stable.","Editorial inference: the tolerance parameter δ is doing double duty—it defines the labels that train the TU classes and sets the yardstick for success—so a fully unsupervised deployment would need a principled way to choose δ without peeking at answers.","Editorial inference: replacing the argmin decision rule with a calibrated threshold on TU_1 − TU_0 could yield anomaly scores with controllable false-alarm rates, which the paper does not explore."],"forward_implications":["Because TU is computed from the already-trained network, any existing FNNEoS can be turned into an anomaly detector post hoc: no retraining or architectural change is required.","The correct-inference rate R_k=0 and failure-detection rate R_k=1 behave differently with the tolerance δ; the largest tested network gives the best overall score, so architecture choice matters for anomaly detection.","Hidden-layer topology supplies signal beyond the final output: the paper's histograms show TU separates classes even when output-based confidence would not.","The success criterion is concrete: the anomaly is detected when the cross-TU for (j,k)=(1,1) is smaller than for (j,k)=(0,1), and this rule can be applied to any new input without its true EoS.","The method generalizes, in principle, to any FNN inference problem where a label function k can be defined by comparing the prediction with a known answer during calibration."],"supporting_citations":[{"why":"Defines Topological Uncertainty as a persistence-based measure on activation graphs; supplies the method this paper adapts.","marker":"[50]"},{"why":"Introduces the idea of characterizing the shape of activation space in deep networks; an earlier step the TU builds on.","marker":"[54]"},{"why":"Provides stability of persistence diagrams under small perturbations, which justifies robustness of the TU construction.","marker":"[55]"},{"why":"Previous FNNEoS study that provides the neutron-star EoS inference framework and the dataset generation used here.","marker":"[28]"},{"why":"Establishes the one-to-one correspondence between the EoS and the mass-radius relation via the TOV equations, making the inverse problem well-posed.","marker":"[46]"},{"why":"Supplies the piecewise-polytrope parametrization of the EoS used to generate the training data.","marker":"[62]"},{"why":"Conventional output-based out-of-distribution indicators that TU is positioned as complementary to.","marker":"[56–58]"}],"fun_headline_variants":["Topological uncertainty spots bad EoS fits over 90% of time","Neural net geometry detects failed EoS inference >90% accuracy","Cross-topological uncertainty flags bad neutron-star EoS fits","Hidden layer topology catches anomaly in EoS inference 90%+","Neural network hidden shapes reveal bad EoS fits >90%"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the hidden-layer activation patterns of the trained network are reliably different for successful versus failed inferences, and that the difference seen in the paper's training runs carries over to new inputs; the paper tests this only on a small number of runs per architecture, without repeated-seed error bars or distribution-shift experiments.","fun_headline_variants_meta":{"raw":{"variants":["Topological uncertainty spots bad EoS fits over 90% of time","Neural net geometry detects failed EoS inference >90% accuracy","Cross-topological uncertainty flags bad neutron-star EoS fits","Hidden layer topology catches anomaly in EoS inference 90%+","Neural network hidden shapes reveal bad EoS fits >90%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000222,"raw_usage":{"total_tokens":1350,"prompt_tokens":866,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":610,"tokens_out":484,"duration_ms":4829,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:33:18.726607+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same experiment many times: train F(4,4,2,2) from, say, 20 different random initializations and evaluate R_tot at δ=0.5 on an independent Dnew for each seed. If the reported >90% accuracy reproduces in every seed, the claim is supported; if any seed gives chance-level R_k=1, or if R_k=1 collapses when Dnew is sampled from a slightly shifted mass-radius distribution, the claimed generalization fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Topological Uncertainty as a persistence-based measure on activation graphs; supplies the method this paper adapts."},{"cited_title":"Topological data analysis and machine learning","cited_arxiv_id":"2206.15075","evidence_quote":"Provides stability of persistence diagrams under small perturbations, which justifies robustness of the TU construction."},{"cited_title":"From NS observations to nuclear matter properties: a machine learning approach","cited_arxiv_id":"2401.05770","evidence_quote":"Establishes the one-to-one correspondence between the EoS and the mass-radius relation via the TOV equations, making the inverse problem well-posed."},{"cited_title":"Br¨ uel Gabrielsson, B","cited_arxiv_id":null,"evidence_quote":"Supplies the piecewise-polytrope parametrization of the EoS used to generate the training data."}],"review_version":1}