{"id":"43ba6b2e-ebb0-45cb-9b7b-556e5f14bbac","arxiv_id":"2412.14330","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"A proposed planarity metric for satellite galaxy systems is shown to be orientation-dependent and reactive to non-planar anisotropies, invalidating the claimed consistency of the Milky Way satellite plane with LCDM.","lead":"This paper stress-tests a newly proposed statistical measure of how 'planar' a galaxy's satellite system is, and finds it fails basic tests: it changes when the system is rotated and it responds to distortions unrelated to planes. The authors conclude the measure cannot support the claim that the Milky Way's satellite plane is consistent with standard cosmology.","discovery_kind":"replication","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the rotation-invariance failure is decisive, and the equal-area scaling check covers the main implementation uncertainty.","rationale":"The reader's verdict of ACCEPT with high confidence is justified. The paper's strongest claim is not a theoretical derivation but a direct numerical demonstration that the metric is not rotation-invariant and is sensitive to several types of anisotropy that are present in Lambda-CDM satellite systems but are not satellite planes. The rotation test is particularly compelling because it uses the same systems before and after a 90-degree rotation, so the only change is the orientation relative to the metric's spherical coordinate grid. A correlation coefficient near 0.35 instead of 1.0 is conclusive evidence that the metric's output depends strongly on orientation. The other tests reinforce this conclusion and show that the metric would over-report planarity in realistic Lambda-CDM-like systems, undermining the claim that high quantiles for Milky Way and simulated satellite systems indicate consistency with Lambda-CDM. The weakest assumption, noted by the reader and by the authors themselves, is that the publicly released code faithfully implements the metric used in the original publication. The paper addresses this concern by testing both the code as provided and a version with the equal-area bin scaling described in the text; the main conclusions are unchanged. This is adequate mitigation because the orientation dependence is inherent to binning a sphere into fixed angular bins, not a quirk of one code implementation. I considered whether the toy models are too artificial to support the conclusion, but the purpose of the paper is to test specificity, and toy models are the appropriate tool for that. The paper does not claim to prove that the Milky Way plane is inconsistent with Lambda-CDM; it only claims that the specific metric cannot be used to infer consistency. That claim is well supported. Therefore no change to the reader's verdict is needed.","tokens_in":13763,"tokens_out":6046,"duration_ms":63004,"concrete_test":"Independently implement the planarity metric from the description in Uzeirbegovic et al. (2024), including both the raw code version and the equal-area scaling used in Appendix B, and rerun the 90-degree rotation test on 1000 isotropic systems with Nsat = 40. If the quantile correlation remains far below 1, the orientation-dependence conclusion is robust to implementation details. As an additional check, run the released code on the exact Milky Way satellite sample used by Uzeirbegovic et al. and compare the returned quantile to their published value to confirm that the code matches the original analysis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the Uzeirbegovic et al. (2024) planarity metric is unsuitable for measuring satellite plane flattening because it responds to orientation and to anisotropies unrelated to planes. The strongest evidence is the rotation test in Section 2.1: for the same isotropic mock systems rotated by 90 degrees, the quantiles show only weak correlation (Pearson r = 0.352, Spearman rho = 0.351). A rotation-invariant metric would give identical quantiles, so this is a direct and unambiguous failure. The additional tests on halo shape, satellite number, clustering, radial concentration, and lopsidedness all support the conclusion that the metric lacks specificity. The only plausible weak point is that the public code may not match the exact implementation used in the original paper. The authors explicitly acknowledge this in Section 2 and mitigate it by repeating the full set of tests with an equal-area bin scaling in Appendix B; the conclusions hold in both cases. Because the orientation dependence follows from the basic spherical-coordinate histogram binning, it is unlikely to disappear under a different but similar implementation. I find no internal inconsistency, unsupported inference, or statistically fragile result that would change the verdict.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper tests the \"planarity\" metric proposed by Uzeirbegovic et al. (2024), which measures the degree of satellite-galaxy planarity by constructing the cross-products of all satellite position vectors, binning the resulting normal-vector directions in spherical coordinates, and computing the Gini coefficient of the bin counts, reported as a quantile relative to 1000 isotropic mock systems. The authors show that this metric is not invariant under rotation: for 1000 isotropic mock systems rotated by 90 degrees, the resulting quantiles show almost no correlation (Pearson r = 0.352, Spearman rho = 0.351), a direct violation of the rotational invariance required of any planarity measure. They further demonstrate that oblateness or prolateness of the overall satellite distribution, the number of satellites, satellite clustering, and lopsidedness all bias the metric toward higher inferred planarity, even in the complete absence of planar substructure. The paper also critiques the error-sampling procedure used in the original study (Appendix A). The authors conclude that the metric is unsuitable for measuring planarity and that consequently the claimed consistency of the Milky Way satellite plane with Lambda-CDM cannot be inferred from it.","tokens_in":13979,"tokens_out":5051,"duration_ms":46569,"significance":"If correct, this paper invalidates the central claim of Uzeirbegovic et al. (2024) and provides a clear methodological caution for the community. The strength of the paper lies in its direct, falsifiable tests with explicit controls: 1000 realizations per configuration, tests of both the publicly released code and an equal-area bin-scaled variant (Appendix B), and a decisive orientation test that alone establishes the metric's failure of rotational invariance. The paper is appropriately cautious in its conclusion: it does not claim that the Milky Way satellite plane is inconsistent with Lambda-CDM, but only that this particular metric cannot be used to demonstrate consistency. The use of publicly available code and transparent toy models enhances reproducibility. The paper's scope is limited, but the conclusion is well supported.","major_comments":[],"minor_comments":[{"comment":"The text reads \"the metric infers an decreased degree\" and should be corrected to \"a decreased degree\".","section":"Section 2.2"},{"comment":"The phrase \"can thus let one to falsely infer\" should be revised to \"can thus cause one to falsely infer\".","section":"Section 2.5"},{"comment":"The correlation coefficients (r = 0.352, rho = 0.351) are reported without uncertainties or p-values; since these are computed over 1000 independent realizations, the standard error is roughly 0.03, so the conclusion is robust, but a p-value or a statement of significance would strengthen the presentation.","section":"Section 2.1"},{"comment":"The description \"the metric constructs the cross-products of all possible combinations of satellite galaxy position vectors\" could be misread as including ordered pairs; it would be clearer to say \"all unique pairs of satellite position vectors\".","section":"Section 2"},{"comment":"The two panels of Figure A.1 are described in the caption but the text does not fully explain why Leo V shows a wider spread than Crater II; a short sentence attributing this to the larger proper-motion uncertainties would help the reader interpret the figure.","section":"Appendix A"}],"recommendation":"accept","confidential_remarks":"This is a well-executed, targeted falsification of a recently proposed metric. The orientation test is decisive, and the additional tests on shape, number, clustering, and lopsidedness provide a comprehensive case. The only residual uncertainty is whether the public code exactly matches the implementation in Uzeirbegovic et al. (2024), but the authors explicitly acknowledge this and show that the conclusions hold for a scaled equal-area version as well. The paper is suitable for a brief Letter format and should be acceptable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a solid, narrowly focused methods paper that convincingly shows the Uzeirbegovic et al. planarity metric should not be used. The orientation test is the killer: 1000 isotropic systems rotated by 90 degrees give quantiles with Pearson r=0.35, nearly uncorrelated. Any sensible metric should be invariant under rotation. That alone is enough to disqualify it. The authors then pile on with tests showing that halo shape, satellite number, clustering, and lopsidedness all shift the quantile distribution toward higher values, meaning the metric is also not specific to planes. All of these are real features of LCDM satellite systems, so the claimed consistency of the MW plane with LCDM doesn't follow from this metric.\n\nThe paper does the job carefully. They use 1000 realizations per configuration, vary parameters systematically, and—importantly—test both the public code as-is and a version with equal-area bin scaling. The conclusions hold either way, which handles the biggest uncertainty about whether the original analysis used a different implementation. They also flag the 6D Cartesian sampling issue in the original paper's error treatment, with a concrete demonstration using Crater II and Leo V. That's a useful side note.\n\nSoft spots are minor. The toy models are simplified, but they don't need to be realistic to show non-invariance; that's a mathematical property. The paper is a negative result and doesn't offer a replacement metric, but that's not a flaw in a critique. If I have any hesitation, it's that the original code's implementation remains uncertain, and the authors themselves say 'we cannot be sure what procedure has been applied.' They mitigate this convincingly, but a pedant could ask whether the rotated-system test should be done using the actual published quantiles from the original paper. The authors do not reproduce the original MW measurement—they only test the metric's behavior. That's fine for the argument they make.\n\nWho is this for? Anyone working on satellite galaxy planes and anyone who uses summary statistics in extragalactic work. It's a cautionary template for validating a metric before adopting it. I'd send it to review; it deserves refereeing, and it will likely be accepted after minor revisions. I'd cite it if I ever reference the planarity metric or discuss satellite plane statistics. Definitely bring it to reading group.","headline":"A decisive and careful falsification of a new planarity metric; the orientation test alone kills it, and the equal-area check closes the main loophole.","tokens_in":14504,"tokens_out":2060,"would_cite":true,"duration_ms":23215,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The proposed 'planarity' metric for satellite systems cannot measure planarity on its own terms, because it responds to any anisotropy and depends on the chosen coordinate orientation, so the claimed consistency of the Milky Way satellite…","keywords":["satellite galaxy planes","planarity metric","Milky Way satellites","Lambda CDM cosmology","anisotropy tests","Gini coefficient","cosmological simulations","metric validation"],"falsifier":"Apply the metric to an isotropically drawn 40-satellite system, rotate it by 90 degrees, and compare the reported quantiles: a suitable metric must return the same value, and the paper's Fig. 1 predicts almost no correlation.","tokens_in":13547,"feed_emoji":"🛰️","tokens_out":6198,"duration_ms":50349,"temperature":0.7,"pith_summary":"This paper tests a recently proposed 'planarity' metric for satellite galaxy systems, a single-number tool that was used to argue that the Milky Way's satellite plane is consistent with Lambda CDM cosmology. The authors build mock satellite systems that are isotropic except for one specific anisotropy at a time—overall oblate or prolate shape, satellite clustering, radial concentration, or lopsidedness—and run the metric on each. All of these features, none of which involve an actual satellite plane, shift the metric's reported planarity toward high values. The metric's output also changes sharply when the same system is rotated by 90 degrees, showing it depends on the chosen coordinate pole. The paper concludes that the metric lacks specificity and cannot be used to infer consistency between the Milky Way satellite plane and Lambda CDM.","feed_headline":"Satellite-plane metric fails rotation and anisotropy tests","feed_subtitle":"Oblate shapes, clustering, and lopsidedness inflate it; so does a 90-degree rotation.","key_machinery":"The central object is the proposed metric itself: for each pair of satellites it takes the cross-product of their position vectors (relative to the host), bins the resulting normal directions in a 2D histogram of azimuth and inclination in a chosen spherical coordinate system, computes the Gini coefficient of the bin counts, and reports the quantile of that Gini value among 1000 isotropic mock systems. That machinery is what carries the argument, because its dependence on the coordinate pole and its response to any anisotropy are the properties the paper tests. The authors also check a variant with equal-area bins, since the public code does not appear to implement the equal-area scaling described in the original paper; the orientation sensitivity and the inflation from anisotropy persist in both versions.","core_discovery":"On the paper's own terms, the central discovery is that the Uzeirbegovic et al. (2024) planarity metric measures general deviation from isotropy, not planarity in particular. Because the metric bins the spherical coordinates of all pairwise cross-products of satellite position vectors in a fixed coordinate frame and then converts the Gini coefficient of bin counts into a quantile relative to isotropic mocks, any anisotropy that concentrates those normals—flattening, elongation, clustered pairs, radially concentrated or offset distributions—raises the reported quantile. The orientation test is the cleanest: 1000 isotropic mock systems rotated by 90 degrees give quantiles with almost no correlation (Pearson r = 0.352, Spearman rho = 0.351), so the metric is not invariant to the coordinate system. Therefore the high quantiles previously reported for the Milky Way and for Lambda CDM simulated systems cannot be read as evidence of consistency; the paper argues the consistency claim does not follow.","pith_inferences":["A general lesson implicit in these results: any summary statistic that uses a fixed coordinate pole and bins directions will inherit orientation artifacts; validation for such tools should include rotation tests and null controls built from anisotropies unrelated to the target feature.","The same battery of tests could be applied to other proposed planarity measures for satellite systems, and the failure modes documented here suggest some may share the sensitivity to lopsidedness or radial concentration.","The equal-area binning variant reduces but does not remove the orientation dependence, which hints that the problem is not the binning scheme but the use of a global pole for pairwise normal vectors; a rotation-invariant statistic would need to avoid that choice entirely."],"forward_implications":["The reported consistency of the Milky Way satellite plane with Lambda CDM, which rested on this metric, is not supported by the metric's output.","Studies using this metric will overestimate the fraction of simulated satellite systems that look planar, because common Lambda CDM features such as triaxial halos, clustering, and lopsidedness all push the quantile upward.","The metric cannot be used to compare the Milky Way sample with simulated hosts of different satellite numbers, since the quantile depends on the number of satellites even when the underlying anisotropy is identical.","Any future use of the metric would need to demonstrate rotation invariance and specificity to planar sub-structure before drawing cosmological conclusions, which the paper shows the current version lacks."],"supporting_citations":[{"why":"The proposed metric under test; supplies the code and the original consistency claim between the Milky Way satellite plane and Lambda CDM.","marker":"Uzeirbegovic et al. (2024)"},{"why":"Establishes that Lambda CDM dark matter halos are triaxial, motivating the oblate and prolate shape tests.","marker":"Allgood et al. 2006"},{"why":"Documents the triaxial shapes of dark matter halos, supporting the expectation that satellite systems are not spherical.","marker":"Bailin & Steinmetz 2005"},{"why":"Sets the hierarchical clustering picture that motivates the satellite clustering test.","marker":"White & Rees 1978"},{"why":"Provides the hierarchical structure formation context used to argue that clustered satellites occur independently of planes.","marker":"White & Frenk 1991"},{"why":"Supplies the observed Milky Way satellite radial distribution used in the lopsidedness and radial concentration comparisons.","marker":"Li et al. 2021"},{"why":"Provides the Milky Way satellite data used in Appendix A to demonstrate that the original error-sampling procedure produces nonphysical phase-space positions.","marker":"Battaglia et al. 2022"},{"why":"Describes the NewHorizon cosmological simulation whose satellite systems were used in the original consistency claim.","marker":"Dubois et al. 2021"}],"fun_headline_variants":["Rotation test exposes planarity metric's flaws","Planarity metric misled by shape, clustering, and orientation","Satellite-plane metric fails rotation invariance test","Proposed planarity metric just measures anisotropy, not planes","Planarity metric's quantile flips with orientation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The tests assume that the public code released by Uzeirbegovic et al. (2024) accurately implements the metric used in their published analysis, since the provided code lacks the equal-area bin scaling described in the paper and the authors can only run the versions they have.","fun_headline_variants_meta":{"raw":{"variants":["Rotation test exposes planarity metric's flaws","Planarity metric misled by shape, clustering, and orientation","Satellite-plane metric fails rotation invariance test","Proposed planarity metric just measures anisotropy, not planes","Planarity metric's quantile flips with orientation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001044,"raw_usage":{"total_tokens":4431,"prompt_tokens":1030,"completion_tokens":3401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":3326}},"tokens_in":646,"tokens_out":3401,"duration_ms":20052,"temperature":1.0,"reasoning_tokens":3326,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:20:02.182424+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the metric to an isotropically drawn 40-satellite system, rotate it by 90 degrees, and compare the reported quantiles: a suitable metric must return the same value, and the paper's Fig. 1 predicts almost no correlation.","supporting_citations":[{"cited_title":"New tools for studying planarity in galaxy satellite systems: Milky Way satellite planes are consistent with {\\Lambda}CDM","cited_arxiv_id":"2411.17813","evidence_quote":"The proposed metric under test; supplies the code and the original consistency claim between the Milky Way satellite plane and Lambda CDM."},{"cited_title":"& Steinmetz, M","cited_arxiv_id":null,"evidence_quote":"Documents the triaxial shapes of dark matter halos, supporting the expectation that satellite systems are not spherical."}],"review_version":1}