{"id":"62ad667a-ea47-42b6-905d-8ae483e2e8a7","arxiv_id":"2411.17024","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A threshold-free outlier rule that labels a survey as an outlier if it participates in all of the k−1 least-overlapping pairs of Beta posterior densities.","lead":"This paper proposes a rule for flagging outlier samples in multiple surveys of the same proportion without a user-set threshold. The rule is relevant to meta-analysis and survey aggregation, but it is neither derived nor tested against existing methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The rule in Eq. (19) has no demonstrated link to selection effects, and the paper's own admission that the best samples can be outliers leaves the central claim unsupported.","rationale":"The reader's weakest assumption identifies the same load-bearing gap: no argument connects the k-1-smallest-overlap condition to selection effects, and the paper itself allows that the best samples may be flagged. My read adds a concrete failure mechanism: exchangeable no-selection samples can satisfy Eq. (19); even identical densities are resolved only by tie-breaking in the code. The paper does include code and synthetic figures, which is useful, but the figures are illustrations, not a validation against known ground truth. Because the central promise is that removing the flagged samples avoids biasing the final result, and because no formal proof, simulation, or real-data check supports that promise, the correctness risk is high. I therefore keep the reader's REJECT verdict rather than adjusting it.","tokens_in":4262,"tokens_out":12987,"duration_ms":118883,"concrete_test":"Simulate no-selection data: fix theta=0.5 and n=(10,20,50,100), draw N_i ~ Binomial(n_i, theta) for 1,000 trials, apply the supplied main(N,n) code, and record the fraction of trials in which at least one observation is declared an outlier. If that fraction is materially above zero, the combinatorial rule in Eq. (19) responds to ordinary binomial sampling noise, not to a selection effect, and the promised bias-free cleaning is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The only characterization of an outlier is combinatorial: Eq. (19) declares i an outlier if it is incident to every one of the k-1 smallest pairwise overlaps in Checklist(Slist(Obs_k)). The paper supplies no theorem, derivation, simulation, or real-data example connecting that graph condition to a strong selection effect. It is a statement about the geometry of Beta densities, not about bias. This gap is load-bearing because the abstract promises that excluding such samples avoids biasing the final result. Section 1 concedes the opposing case: 'sometimes the majority of samples make the same error, and the best sampling results may end up being outliers.' In that case the rule flags the few accurate samples, exactly contrary to the stated goal. The supplied code also illustrates an elementary artifact: for k=4 identical Beta densities all six overlaps are equal, and np.argsort returns the first three pairs (0,1),(0,2),(0,3), so sample 0 is called an outlier purely by tie-breaking; the paper's no-repetition caveat does not remove the instability for near-identical densities. Thus Eq. (19) is neither derived from nor validated against the concept of selection effect, and the central claim fails as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a threshold-free rule for flagging outlier sampling results when several Bayesian posterior densities for the same binomial proportion are available. Each posterior is a Beta density, and pairwise similarities are defined by the overlap coefficient S_j^i = ∫ min(p_i,p_j)dθ (Eq. 9). The rule takes the k−1 smallest pairwise overlaps in the complete graph on k densities, and declares observation i an outlier if i is incident to all of those k−1 edges (Eq. 19). The procedure is iterated after removing flagged observations, and a \"fragmented\" set is declared when the iterative removal reaches a degenerate condition (Eqs. 20–22). The paper also provides Python code for the rule and illustrates it with four synthetic examples in Figure 1.","tokens_in":4497,"tokens_out":2731,"duration_ms":27837,"significance":"If the proposed definition were reliably connected to selection effects, a threshold-free outlier rule would be practically valuable in fields such as astronomy and social surveys, where selection biases are difficult to model and results should not depend on arbitrary cutoffs. The manuscript is commendably explicit: it gives a precise combinatorial definition, provides reproducible code, and makes the falsifiable claim that the rule identifies samples with strong selection effects and that removing them avoids biasing the final result. However, that central claim is not established. The definition is purely geometric/combinatorial, with no derivation, simulation, or real-data demonstration linking the k−1 smallest pairwise overlaps to selection effects. The tie-breaking instability for identical or near-identical densities is a concrete artifact that can cause the rule to flag an arbitrary sample as an outlier in a clearly innocuous setting. As submitted, the paper does not support its advertised conclusion.","major_comments":[{"comment":"The central claim—that Eq. (19) identifies sampling results with strong selection effects—is unsupported. The definition of an outlier as a vertex incident to all k−1 smallest pairwise overlaps is a combinatorial condition on the geometry of Beta densities; the paper supplies no theorem, simulation, or real-data example connecting that condition to a biased sampling process. Section 1 even concedes that \"sometimes the majority of samples make the same error, and the best sampling results may end up being outliers,\" in which case the rule would flag the accurate samples, contrary to the stated purpose of avoiding bias. This gap is load-bearing because the abstract promises that excluding such outliers avoids compromising the final result.","section":"Section 2, Eqs. (14)–(19)"},{"comment":"The rule is unstable under ties, and the tie-breaking is implementation-defined rather than threshold-free. For k identical Beta densities, all pairwise overlaps are equal, so the k−1 smallest pairs are not uniquely defined. The supplied Python code invokes np.argsort on the overlap list and takes the first k−1 pairs; for k=4 with identical densities, this selects (0,1), (0,2), (0,3), flagging sample 0 as an outlier purely by array order. The paper's caveat that Obs_k contains no repeated elements does not help for near-identical densities, which can arise in practice. A rule whose output changes under arbitrary tie-breaking is not a stable, threshold-free definition of outliers.","section":"Section 2, Eq. (10) and Supplementary code"},{"comment":"The iterative outlier removal and the notion of a \"fragmented\" set are not well-defined. Eq. (20) returns None when no observation satisfies Eq. (19), but the recursive definitions in Eqs. (21)–(22) do not specify a stopping criterion, and the sentence \"If Out_{k−3}(Obs_k) is defined, Out_k(Obs_k) is defined\" is unclear: for k=4, Out_1 being defined would make Out_4 defined, yet the code's while loop removes outliers until only one sample remains and then prints \"Fragmented!\" for any set that reaches that point. This inconsistency between the mathematical definition and the implemented algorithm needs to be resolved before the method can be evaluated.","section":"Section 2, Eqs. (20)–(22)"},{"comment":"The paper provides no empirical validation. Figure 1 shows four synthetic examples, but the code in the supplement includes only one example (N=[15,11,7,29,100], n=[30,20,15,60,200]) and does not reproduce all four panels. There are no experiments with known ground-truth selection effects, no comparisons with existing outlier methods, and no sensitivity analysis. Without such evidence, the claim that the rule detects strong selection effects remains an assertion rather than a demonstrated result.","section":"Figure 1 and Supplementary code"}],"minor_comments":[{"comment":"The uniform prior is written as p(θ)=1 on 0<θ<1, which is correct, but the surrounding text says \"the prior distribution cannot be estimated, so the prior distribution is assumed to be an uniform distribution\" without justifying why a uniform prior is an appropriate default in this setting.","section":"Section 2, Eq. (3)"},{"comment":"Reference [1] contains a typo: \"Journal of the American statistical association\" should be \"Journal of the American Statistical Association.\" Reference [2] would benefit from a modern citation format for Bernoulli's Ars Conjectandi.","section":"References"},{"comment":"In the main() function, the variables outliersN and outliersn are used before assignment (they should be outN and outn); the code as printed would raise an UnboundLocalError when an outlier is found.","section":"Supplementary code"}],"recommendation":"reject","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a short, readable paper with a new rule, but the rule is a definition, not a discovery. The only evidence is four illustrative figures; there is no simulation, no real-data test, and no argument tying the combinatorial condition to selection bias. So the central claim is unsupported.\n\nWhat is new and good: the specific criterion—declare sample i an outlier iff it is incident to every one of the k−1 smallest pairwise Beta-overlaps—is new as far as I know. The recursive removal procedure that can declare a set \"fragmented\" is also defined, and code is included, so the procedure is reproducible. The writing is clear, and the author is honest that thresholds are arbitrary and that majority-error cases can flip the rule.\n\nThe soft spots are load-bearing. Eq. (19) is a graph-theoretic condition on Beta densities; nothing connects it to selection effects. The paper itself concedes that the best sample can be an outlier when the majority errs, which is exactly the case where the rule fails its purpose. The code exposes a tie-breaking artifact: for k identical Beta densities all overlaps are equal, np.argsort returns the first three pairs, and sample 0 is flagged as an outlier. The manuscript says to ensure no repeated elements in Obs_k, but near-identical densities will be unstable, so the rule can manufacture outliers where none exist. That is not a minor issue. There is also no comparison to existing robust methods (Z-score, robust location estimation, mixture models), no theoretical guarantee, and no simulation. For a paper proposing a new outlier rule, the abstract's promise—excluding such samples avoids biasing the final result—is simply not demonstrated.\n\nWho is this for? Someone looking for a threshold-free heuristic to try as a quick filter. But as a research claim it needs much more work: simulations with known selection effects, real-data examples, tie-breaking analysis, and comparison to existing methods. My recommendation: I would not send this out for peer review in its current form. The central claim fails as stated. The author should either provide evidence or substantially weaken the claim to a heuristic proposal rather than a method with a guaranteed effect.","headline":"A cleanly written, genuinely new combinatorial rule for outlier flagging, but the central claim connecting it to selection effects is unsupported and the method can flag arbitrary samples in near-identical data.","tokens_in":4958,"tokens_out":1651,"would_cite":false,"duration_ms":15727,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a sampling result is an outlier exactly when it appears in every one of the k-1 smallest pairwise overlaps among the posterior densities, so no threshold is needed.","keywords":["outlier detection","threshold-free","Bayesian posterior","Beta distribution","selection effects","pairwise overlap","data cleaning"],"falsifier":"Construct a known truth θ, take k-1 sampling results from a heavily biased distribution and one accurate result sampled from θ, then run the algorithm; if the accurate result is flagged as an outlier and the cleaned estimate moves far from θ, the claim that the rule identifies results with strong selection effects is falsified.","tokens_in":4036,"feed_emoji":"📊","tokens_out":6458,"duration_ms":57954,"temperature":0.7,"pith_summary":"This paper tries to establish a definition of outliers that needs no threshold when several sampling results estimate the same event. Each result is turned into a Beta posterior density; the method measures pairwise overlap by the integral of the minimum of two densities, then looks at the k-1 smallest overlaps among k results. A result that takes part in every one of those smallest overlaps is declared an outlier, and the rule is applied again after each removal. The point of a threshold-free rule is to make the cleaned estimate reproducible and defensible when no prior distribution can be justified, as in astronomical surveys or social investigations. A direct consequence is that a set in which all remaining results are outliers is called 'fragmented' and is considered too unstable to yield a reliable estimate.","feed_headline":"Outliers found without thresholds in repeated sampling","feed_subtitle":"Each sample becomes a Beta curve; any curve that sits in all k-1 smallest overlaps is removed.","key_machinery":"The central object is the pairwise overlap S_j^i = ∫$_0^{1}$ min(p_i(θ), p_j(θ)) dθ between two Beta posterior densities, where p_i(θ)=Beta(θ | N_i+1, n_i-N_i+1). From the list of all such overlaps, the machinery selects the k-1 smallest values, forming the checklist; the operator Unsi(i | Obs_k) counts how many checklist entries contain i. The claim is that Unsi = k-1, i.e., participation in every minimal-overlap pair, is the defining signature of an outlier, and the iterative operator Out_n applies this test to successively reduced sets. This construction replaces a threshold on a score with a purely combinatorial condition on the list of pairwise overlaps.","core_discovery":"The paper's central claim is equation (19): observation i is an outlier precisely when Unsi(i | Obs_k) = k-1, meaning i belongs to all k-1 entries of the checklist of smallest pairwise overlaps. The author argues this combinatorial condition captures the effect of strong selection biases, because a heavily biased sample will be dissimilar from every other sample and therefore appear in all the smallest overlap pairs. The rule first removes the single density satisfying that condition, then re-applies the same test to the remaining set, continuing until no density satisfies it. If the set ever reaches a state where all remaining densities are flagged, the set is 'fragmented' and the author says no reliable result should be given. This definition depends only on the shape of the posterior densities, so sample size enters through the Beta distribution rather than through a user-chosen weight.","pith_inferences":["An implication the author leaves implicit is that the rule is essentially a one-class dissimilarity test, so a single accurate sample surrounded by many similarly biased samples will be flagged as an outlier, and the paper offers no safeguard for that case.","A testable extension is to weight each pairwise overlap by sample sizes or information content and compare the resulting outlier sets on synthetic data with a known true θ.","The choice of k-1 smallest overlaps is itself a structural parameter; generalizing the rule to the m smallest overlaps for m < k-1 would reveal how sensitive the outlier set is to that choice.","The method's performance could be compared against standard robust pooling schemes, such as majority voting or a weighted median over the same density representation, which the paper does not provide."],"forward_implications":["If the rule holds, outlier removal in repeated sampling becomes reproducible because no threshold must be chosen and no weighting scheme must be assumed.","The approach automatically accounts for sample size since each result is represented by its full Beta posterior rather than by a single point estimate.","Iterative application of the rule can identify multiple outliers one at a time, and a set where all remaining results are flagged signals that the sampling process is too unstable to trust.","A non-uniform prior that leads to a different parametric form of p_i(θ) does not affect the definition, since the similarity measure uses the densities directly."],"supporting_citations":[{"why":"Introduces Bayesian linear statistics, the approach this paper contrasts by arguing that heavily biased sampling results should be identified as outliers first.","marker":"[1]"},{"why":"Supplies the law of large numbers background that justifies the point estimate θ=N/n and motivates the use of a density rather than a single value.","marker":"[2–4]"},{"why":"Information entropy is invoked to justify that larger samples give narrower posterior densities, which underlies the use of the full density in the similarity measure.","marker":"[5]"}],"fun_headline_variants":["Threshold-free outlier flag for repeated sampling","No thresholds: outlier sits in every smallest overlap","Beta curves pick outliers by overlap, no user cutoff","Flag sample in all k-1 minimal overlaps as outlier"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that any sampling result distorted by a strong selection effect will be involved in all k-1 smallest pairwise overlaps, so the rule removes exactly the biased samples and leaves the accurate ones; the paper gives no argument linking the combinatorial condition to the selection mechanism.","fun_headline_variants_meta":{"raw":{"variants":["Threshold-free outlier flag for repeated sampling","No thresholds: outlier sits in every smallest overlap","Beta curves pick outliers by overlap, no user cutoff","Flag sample in all k-1 minimal overlaps as outlier"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1271,"prompt_tokens":784,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":400,"completion_tokens_details":{"reasoning_tokens":427}},"tokens_in":400,"tokens_out":487,"duration_ms":5269,"temperature":1.0,"reasoning_tokens":427,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:36:06.453083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a known truth θ, take k-1 sampling results from a heavily biased distribution and one accurate result sampled from θ, then run the algorithm; if the accurate result is flagged as an outlier and the cleaned estimate moves far from θ, the claim that the rule identifies results with strong selection effects is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Bayesian linear statistics, the approach this paper contrasts by arguing that heavily biased sampling results should be identified as outliers first."}],"review_version":1}