{"id":"64906716-b0b1-4d7a-a5c0-3b5bbefbe9fa","arxiv_id":"2412.03744","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A small-sample co-occurrence network method with bootstrap filtering is applied to chicken cecal 16S data, identifying an 8-species persistent microbiome across Eimeria infection stages.","lead":"This paper proposes a bootstrap-filtered co-occurrence network workflow for 16S microbiome datasets with very few samples, applied to chicken gut samples from an Eimeria tenella infection trial. If the method worked as claimed, it would let researchers extract network features from small datasets where standard tools like SPIEC-EASI fail to converge.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The bootstrap filter tests edges against the data's own empirical distribution, not against a null of no association; the 'without replacement' resampling is internally inconsistent.","rationale":"The reader's weakest assumption is the same load-bearing concern I identify: the bootstrap filter's null distribution is the observed data itself, not a model of no association, and the 'without replacement' wording is a fatal internal inconsistency. The central claim of the paper -- that the method yields a 'statistically significant' and 'biologically meaningful' network -- rests entirely on the validity of this filter. If the filter is invalid, every downstream result (persistent microbiome, K-S tests, biological interpretations) is uninterpretable. I checked the full text for independent support: the Nextflow pipeline is reproducible and the data are real, but the paper provides no external validation of the statistical filtering, no simulation study where ground truth is known, and no comparison against a proper null. The contradiction between Table II (0 nodes filtered at 2-sigma) and the text's claim that species are pruned, plus the implausible 95% edge removal in Table III, further undermines the method's internal consistency. The proper fix would be to construct a null model that preserves marginal abundances but breaks associations, and to calibrate the filter's false-positive rate under that null. Without such a demonstration, the paper's conclusions cannot be accepted. My verdict therefore agrees with the reader's rejection, and I would not change it to a conditional acceptance because the flaw is in the core statistical argument, not a minor parameter choice.","tokens_in":16677,"tokens_out":2905,"duration_ms":32067,"concrete_test":"Run a permutation null: for each experimental condition, independently shuffle the abundance values of each ASV across the 10 samples to destroy all inter-species co-occurrence structure while preserving marginal distributions. Apply the proposed bootstrap filter (with replacement, as the text likely intended) to the permuted tables and record the fraction of edges retained. If the retention rate is similar to the rate reported in Table III for real data (e.g., 795/15715 for the unchallenged condition), the filter has no ability to distinguish spurious from real associations and the central claim fails. Additionally, recompute edge-level filtering using a genuine independence null, such as a permutation test of co-occurrence counts, and check whether the persistent microbiome of 8 species or the differential expression results survive that test.","verdict_should_be":"REJECT","load_bearing_attack":"Section III.C generates synthetic ASV tables by 'randomly selecting 10 samples without replacement from the 10 observed samples in each experimental condition.' With only 10 observed samples, this selection is deterministic: it reproduces the original table every time, so all n=1000 bootstrapped graphs are identical and the confidence intervals have zero width. If the intended text is 'with replacement,' the procedure still does not provide a valid null distribution: bootstrapping the observed sample estimates the sampling distribution of the observed statistic, not the distribution under a model of no association. The hypothesis tested, H0: mu_w,bootstrapped = observed w, is circular because the bootstrap mean is that observed value. Edges are called 'spurious' if their weights fall outside a 2-sigma interval, but this flags only edges whose weights are atypical relative to the observed data's own empirical distribution, not edges created by experimental or preprocessing artifacts. The reported results are also internally inconsistent: Table II shows zero species filtered at 2-sigma, while Figure 1 and the text claim species beyond two standard deviations are pruned; Table III shows over 95% of edges removed at the '5% significance level,' which cannot be a 5% false-positive rate under any standard null. Because the entire pipeline -- the persistent microbiome, the K-S tests, the biological interpretations -- depends on edges being statistically meaningful, the central claim that the method 'statistically removes spurious associations' is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a graph-based co-occurrence network inference and filtering workflow for small-sample 16S microbiome data. Edges represent co-occurrence of ASVs across samples, edge weights are derived from ratios of relative abundances, and node attributes summarize local network topology. The authors generate bootstrap resamples of the observed samples, construct confidence intervals for node- and edge-level weights, and prune observations that fall outside these intervals. The method is applied to chicken caecal samples across three Eimeria infection stages, followed by downstream analyses including a persistent-microbiome subnetwork, K-S tests between conditions, and hierarchical clustering. The central claim is that this approach yields networks that are feature-rich, biologically meaningful, and statistically significant from small-sampled data.","tokens_in":16962,"tokens_out":4035,"duration_ms":41063,"significance":"The application domain is relevant, and the engineering contributions are real: the Nextflow/Docker pipeline, containerization, use of public sequencing data, and the explicit discussion of small-sample challenges are strengths. If the statistical filter were valid, the paper could offer a useful addition to the microbiome network inference toolkit. However, the central filtering procedure is not a valid test for spurious associations, and the reported results are internally inconsistent. Because the persistent-microbiome result, the K-S based differential-expression claims, and the biological interpretations all depend on the filtered networks being statistically meaningful, these conclusions are not supported by the evidence as presented.","major_comments":[{"comment":"The description that synthetic ASV tables are generated by 'randomly selecting 10 samples without replacement from the 10 observed samples' is deterministic when the sample size equals the number of observed samples: every bootstrap graph is identical, the bootstrap distribution is degenerate, and no confidence interval or empirical p-value can be produced. If the intended text is 'with replacement', the bootstrap still only characterizes the sampling variability of the observed statistic; it does not provide a null distribution under a model of no association or under an artifact-free model. This invalidates the claimed identification of spurious edges.","section":"Section III.C"},{"comment":"The null hypothesis is stated as H0: µ_w,bootstrapped = observed w. Since the bootstrap mean is approximately the observed value by construction, this null is circular; the test can only flag weights that are atypical relative to the data's own empirical distribution, not edges that arise from experimental or preprocessing artifacts. Additionally, the alternative hypothesis is printed identically to H0 and should be µ_w,bootstrapped ≠ observed w. This is not a statistical test for spurious associations.","section":"Section III.C, null hypothesis"},{"comment":"Table II reports zero species filtered at the 2σ level in all three conditions, yet Figure 1 and the accompanying text state that a number of species lie beyond two standard deviations and are pruned. The 1σ filter removes 106/142/26 species, which, under a normal approximation and a true null, would correspond to roughly a 32% false-positive rate rather than a controlled error rate. The table, figure, and narrative are mutually inconsistent.","section":"Section VI.A.1, Table II and Figure 1"},{"comment":"Edge-level filtering at the 5% significance level leaves 795/132/42 edges out of 15715/17536/6064, i.e., only about 5% of edges remain. A two-tailed 5% test would be expected to remove about 5% of edges under the null; removing 95% indicates that the procedure is not behaving as a significance filter. No multiple-testing correction is applied across the thousands of simultaneous edge tests. The column heading 'Filtered number of edges' is also ambiguous, since the text says edges outside the confidence interval are pruned but the table appears to report retained counts.","section":"Section VI.A.2, Table III"},{"comment":"The K-S tests are performed on the filtered distributions, using the same data that determined which nodes and edges were filtered. This circular reuse of the data invalidates the reported p-values, and no correction is made for testing multiple pairs of distributions. The claim that the post-filtering distributions are 'statistically significant' differential expression is therefore not supported.","section":"Section VI.C.1"}],"minor_comments":[{"comment":"The formula for w_mn as printed appears malformed; please rewrite with explicit parentheses and define x_mn in a displayed equation before introducing w_mn.","section":"Section III.A, Eq. (2)"},{"comment":"The alternative hypothesis H1 should read µ_w,bootstrapped ≠ observed w, not µ_w,bootstrapped = observed w.","section":"Section III.C"},{"comment":"The figure lacks axis labels and the caption does not state whether the plotted quantity is a node-level or edge-level weight; please clarify.","section":"Figure 1"},{"comment":"There are repeated typos, including 'co-occurences' for 'co-occurrences', 'wadvocating' in Section II, 'ecostystem' in Section III.C, and 'microbome' in the Table IV heading.","section":"Throughout"},{"comment":"Reference [45] is malformed: the year '2019,' appears before the article title; please fix the citation format.","section":"Reference [45]"},{"comment":"Please clarify whether the 'persistent microbiome' of 8 species refers to the three-condition quasi-time series or to all experimental conditions; Section III.D defines it across 'all conditions', while the text says the filtered quasi-time-series data contains 8 species.","section":"Section VI.C.2"},{"comment":"The SPIEC-EASI comparison is reported only for 'All samples'; please state explicitly that per-condition SPIEC-EASI did not converge and explain why the all-sample comparison is a valid benchmark for the proposed method.","section":"Section VI.B.2, Table III"}],"recommendation":"reject","confidential_remarks":"I agree with the reader's assessment. The statistical core of the manuscript is invalid as written: the 'without replacement' bootstrap is deterministic, the null hypothesis is circular, and the reported filtering results in Tables II and III are internally inconsistent. Correcting these issues would require redesigning the null model, re-running all analyses, and re-deriving the biological conclusions, which is beyond the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: the statistical filtering that carries the paper's central claim doesn't work as described, and the text contradicts itself on the results. The paper is not publishable in its current form, but the underlying workflow and dataset are real and worth a look.\n\nWhat's actually new: the specific combination of abundance-ratio edge weights, a bootstrap-based node/edge filter, and a persistent-microbiome intersection analysis on 10-sample 16S cohorts is a new workflow, even though each ingredient is known. The Nextflow pipeline is containerized and reproducible, and the Eimeria chicken dataset is a genuine applied test case. The comparison to SPIEC-EASI (which fails to converge on this data) is a useful practical data point.\n\nThe soft spots are load-bearing. Section III.C says synthetic ASV tables are made by 'randomly selecting 10 samples without replacement from the 10 observed samples.' That's a deterministic procedure—every bootstrap graph is exactly the original graph. So the confidence intervals have zero width, and no edge should ever be filtered at any significance level. Table II shows exactly that (0 species filtered at 2σ), but Figure 1 and the text claim many species lie beyond two standard deviations. Then Table III reports that edge-level filtering removes over 95% of edges at the '5% significance level,' which cannot come from the described bootstrap. Even if 'with replacement' was intended, the null distribution is the data's own empirical distribution, not a model of no association; the test is circular. Multiple testing across thousands of edges is never corrected, and the post-filtering K-S tests reuse the same data that set the filter.\n\nThe biological conclusions—mutualism vs parasitism, the persistent microbiome—are asserted from edge weights without independent validation. That's a secondary issue, but it compounds the statistical one.\n\nMy take: this is a working paper that needs a major statistical rework before the results can be trusted. The problem is important and the authors have a real pipeline, so I'd send it to a serious referee—but with a request to focus on the resampling scheme and the null model. For a reading group, it's a good case study in why bootstrap filters need a proper null. I wouldn't cite it in its current form.\n\nRecommendation: desk reject with an invitation to resubmit after fixing the statistical core.","headline":"The statistical filter at the paper's core is internally inconsistent and tests against the data's own distribution; the workflow and dataset are real, but the results don't stand.","tokens_in":17481,"tokens_out":3500,"would_cite":false,"duration_ms":33958,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a bootstrap-based statistical filter to remove spurious associations from co-occurrence networks built from small-sampled 16S microbiome data, and demonstrates it on chicken Eimeria data.","keywords":["microbiome","16S rRNA","co-occurrence network","bootstrapping","small-sample data","network filtering","Eimeria tenella","persistent microbiome"],"falsifier":"Run the bootstrap exactly as described, sampling ten of ten observed samples without replacement, and verify that every synthetic ASV table equals the original, so the 2-sigma interval is degenerate and no edge can be pruned; Table III reports extensive pruning, so this observation would show the stated mechanism cannot be what produced the reported networks.","tokens_in":16477,"feed_emoji":"🦠","tokens_out":7118,"duration_ms":66711,"temperature":0.7,"pith_summary":"The paper aims to establish that co-occurrence networks can be inferred from 16S microbiome data even when each experimental condition has only ten samples, by pairing a bespoke edge-weighting scheme with a bootstrap-based statistical filter. The filter is meant to identify and discard spurious nodes and edges that arise from sequencing noise and sampling artifacts, so that the remaining network is statistically significant and biologically interpretable. The authors apply the method to chicken gut microbiota across three stages of Eimeria tenella infection and claim that the filtered networks show disease-progression shifts in node-weight distributions, reveal clusters of mutualistic and parasitic interactions, and expose an eight-species persistent microbiome. A sympathetic reader would care because small-sample microbiome studies are common, and existing tools such as SparCC and SPIEC-EASI either need large samples or fail to converge on ten samples.","feed_headline":"Bootstrap filter digs real microbe links out of tiny gut datasets","feed_subtitle":"A statistical pruning step turns ten-sample chicken microbiome counts into meaningful networks and exposes an 8-species core.","key_machinery":"The central object is the weighted co-occurrence graph plus a bootstrap filtering loop. An edge between ASV $m$ and $n$ exists if they co-occur in at least one sample, with weight $w_{mn} = \\left(\\frac{x_{mn} + 1/x_{mn}}{2}\\right)^{-1}$ where $x_{mn}=r_m/r_n$ and $r_m,r_n$ are relative abundances, so weights near 1 correspond to similar abundances. Node attributes include average relative abundance, degree, weighted degree, betweenness centrality, and co-occurrence counts. Filtering works by generating $n$ bootstrapped graphs from synthetic ASV tables created by resampling observed samples, computing the distribution of mean node/edge weights, and discarding nodes or edges whose observed weight lies outside the $2\\sigma$ confidence interval, justified by the central limit theorem. An intersection of nodes present in all filtered condition networks defines the persistent microbiome.","core_discovery":"On its own terms, the paper's central discovery is that a Monte-Carlo bootstrapping procedure can serve as a principled replacement for prevalence-based or correlation-based filtering in small-sample microbiome networks. For each experimental condition, the authors build an aggregated co-occurrence graph whose edges record co-presence of two ASVs across samples and are weighted by a function of the ratio of their relative abundances. They then generate a large population of synthetic graphs by resampling from the observed samples, compute the distribution of each node's average edge weight and each edge's weight under that population, and prune any node or edge whose observed value falls outside the two-standard-deviation confidence interval. On the chicken dataset this pruning leaves networks that cluster into distinct communities and a core of eight taxa present across all conditions, which the paper interprets as a persistent microbiome. The claimed result is that the statistical properties of the simulated population identify spurious artifacts that prevalence filtering would miss while preserving rare but potentially important species.","pith_inferences":["If the resampling step is corrected to sampling with replacement, the method still only resamples the observed ten-sample empirical distribution; the null it constructs is a null of no edge only if spurious edges are equally likely to appear in any resample, which is not argued. A proper test would need a null that destroys the co-occurrence structure while preserving marginals.","The edge weight formula is a monotone transform of the absolute log-ratio of abundances, $w_{mn}=2/(e^{|\\ln x_{mn}|}+e^{-|\\ln x_{mn}|})$, so it rewards equal abundance. This conflates mutualism with equal abundance under uniform sampling depth; disentangling them would require a compositionally aware null.","The same bootstrap-filtering logic could be ported to any sparse compositional data type such as metabolomics or transcriptomics where sample sizes are small and prevalence thresholds are arbitrary.","The claim that the persistent microbiome is biologically meaningful is a post-hoc interpretation; a natural extension is to test whether the eight core ASVs are functionally linked in metagenomic or metabolomic data."],"forward_implications":["At the 5% significance level, filtered node-weight distributions differ significantly between unchallenged and challenged conditions and between 6 and 10 days post infection, supporting the claim that network expression tracks disease progression.","After edge-level filtering, the algorithm reveals four distinct clusters where clustering before filtering had failed, indicating that spurious edges were masking community structure.","The persistent microbiome of eight taxa, including Burkholderia, Escherichia-Shigella, Prevotella, Bacteroides, and Clostridia, is present across all conditions and could serve as candidate targets for vaccine or probiotic development.","Because the pipeline is implemented in Nextflow with containerised tools, the proposed inference and filtering method is reusable and extensible to multi-omics data."],"supporting_citations":[{"why":"Supplies the experimental chicken vaccination and Eimeria challenge dataset used to demonstrate the method.","marker":"[7]"},{"why":"Provides the initial co-occurrence network construction approach that this method extends to small samples.","marker":"[5]"},{"why":"Supplies the central limit theorem used to justify treating bootstrapped mean weights as normal for confidence intervals.","marker":"[31]"},{"why":"Provides the SparCC baseline; the paper argues it needs larger sample sizes and may produce non-positive-definite matrices for small data.","marker":"[4]"},{"why":"Provides the SPIEC-EASI baseline; the paper shows it fails to converge on separate 10-sample conditions.","marker":"[29]"},{"why":"DADA2 is used to infer ASVs from the raw amplicon reads in the preprocessing workflow.","marker":"[38]"}],"fun_headline_variants":["Bootstrap pruning reveals core microbes in tiny gut samples","Statistical resampling filters noise from small microbiome nets","Bootstrap filter uncovers stable microbe core in chicken gut","Similarity pruning finds persistent microbes in sparse data","Resampling-based network pruning reveals gut microbiome core"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that resampling the ten observed samples yields a valid null distribution for distinguishing real from spurious edges, despite the text saying the samples are drawn without replacement from the same ten samples.","fun_headline_variants_meta":{"raw":{"variants":["Bootstrap pruning reveals core microbes in tiny gut samples","Statistical resampling filters noise from small microbiome nets","Bootstrap filter uncovers stable microbe core in chicken gut","Similarity pruning finds persistent microbes in sparse data","Resampling-based network pruning reveals gut microbiome core"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000695,"raw_usage":{"total_tokens":3188,"prompt_tokens":1036,"completion_tokens":2152,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":2077}},"tokens_in":652,"tokens_out":2152,"duration_ms":15276,"temperature":1.0,"reasoning_tokens":2077,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:07:47.069110+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the bootstrap exactly as described, sampling ten of ten observed samples without replacement, and verify that every synthetic ASV table equals the original, so the 2-sigma interval is degenerate and no edge can be pruned; Table III reports extensive pruning, so this observation would show the stated mechanism cannot be what produced the reported networks.","supporting_citations":[{"cited_title":"Novel whole yeast-based subunit oral vaccine against eimeria tenella in chickens,","cited_arxiv_id":null,"evidence_quote":"Supplies the experimental chicken vaccination and Eimeria challenge dataset used to demonstrate the method."},{"cited_title":"Central limit theorem: the cornerstone of modern statistics,","cited_arxiv_id":null,"evidence_quote":"Supplies the central limit theorem used to justify treating bootstrapped mean weights as normal for confidence intervals."},{"cited_title":"Sparse and compositionally robust inference of microbial ecological networks,","cited_arxiv_id":null,"evidence_quote":"Provides the SPIEC-EASI baseline; the paper shows it fails to converge on separate 10-sample conditions."},{"cited_title":"Dada2: High resolution sample inference from amplicon data,","cited_arxiv_id":null,"evidence_quote":"DADA2 is used to infer ASVs from the raw amplicon reads in the preprocessing workflow."}],"review_version":1}