{"id":"38c41a6b-bc8a-4984-bbfc-148a6f555bd6","arxiv_id":"2411.17901","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper introduces q2-fmt, a QIIME 2 plugin that computes several engraftment metrics including a new permutation test for donor feature transfer after fecal microbiota transplant.","lead":"q2-fmt is a new free software plugin for the QIIME 2 platform that measures how much of a donor's gut microbiome takes hold in a patient after fecal microbiota transplant. It packages three engraftment criteria into one suite, adding a new metric and a permutation test to help standardize FMT outcome assessment.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Permutation test may be invalid: actual donor-recipient pairs are not described as rarefied to the same depth as mismatched pairs, confounding PEDF with sequencing depth.","rationale":"The reader's weakest assumption correctly identified the permutation test as the key unsupported premise, but the concern here is more specific and more damaging: as described, the test is not merely unvalidated; it appears internally inconsistent regarding rarefaction depth. This directly undermines the paper's novel statistical claim. The rest of the plugin (CC pipeline, PEDF/PRDF/PPRF metrics, ANCOM-BC integration) may be useful and well tested, and the availability of the tool is a reasonable contribution. The appropriate response is a conditional acceptance requiring either proof that actual pairs are rarefied identically in the implementation or a corrected algorithm with simulation-based validation. I do not recommend outright rejection because the flaw may be a documentation omission rather than a code error, and the other actions are not affected.","tokens_in":4190,"tokens_out":4873,"duration_ms":43237,"concrete_test":"Inspect the q2-fmt source code for pedf-permutation-test and verify whether the actual donor-recipient pair is rarefied to the same user-specified depth before computing PEDF. If it is not, run a null-model simulation (e.g., random feature tables with no donor-specific transfer but varying recipient sequencing depths) and check whether p-values are uniform; a correlation between p-value and depth confirms the bias. If it is rarefied, the manuscript must explicitly state this and the procedure should be re-evaluated with a simulation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central novel contribution is the pedf-permutation-test action, which the paper presents as a valid significance check for donor feature transfer. The described procedure rarefies mismatched donor-recipient pairs ('for each pair the samples are rarified to a user-specified even sampling depth'), but for the actual pairs it only states 'PEDF is computed for each actual donor-recipient pair' with no mention of rarefaction. If actual pairs are not rarefied to the same depth, an actual recipient sample with higher sequencing depth will spuriously yield a higher PEDF, inflating the test statistic and biasing the permutation p-value toward significance. The paper's claim that the test distinguishes true transfer from shared features therefore rests on a procedural detail that is either missing or incorrect as written. Even if the implementation does rarefy actual pairs, the manuscript's description is incomplete, and the validity of the test is not established by any simulation or gold-standard comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents q2-fmt, a QIIME 2 plugin for quantifying microbiome engraftment after fecal microbiota transplant (FMT). It implements three conceptual criteria from a companion review: Chimeric Asymmetric Community Coalescence (the cc pipeline), Donated Microbiome Indicator Features (PEDF, PRDF, and detect-donor-indicators), and Temporal Stability (through longitudinal analysis). The plugin introduces a new permutation test (pedf-permutation-test) intended to distinguish true donor-to-recipient feature transfer from features shared by chance. The paper describes the design, actions, unit tests, and availability of the plugin, and illustrates the cc pipeline on an autologous FMT dataset from Taur et al. (2018).","tokens_in":4345,"tokens_out":4435,"duration_ms":38847,"significance":"If the plugin works as described, it would provide the microbiome field with a standardized, reproducible toolkit for FMT engraftment assessment, which is currently lacking. The paper's strengths include the use of the QIIME 2 framework with provenance tracking, unit tests run on every commit, open-source BSD-3 licensing, and worked documentation on real data. The arithmetic for PEDF, PRDF, and PPRF is straightforward and internally consistent, and these metrics are clearly tied to prior work by Aggarwala et al. However, the central novel contribution—the permutation test—is presented without any statistical validation, and the manuscript text contains an asymmetry in how actual versus mismatched donor-recipient pairs are treated with respect to rarefaction. These issues need to be addressed before the claimed significance check can be considered reliable.","major_comments":[{"comment":"The permutation test as described rarefies mismatched donor-recipient pairs to a user-specified even sampling depth, but the sentence 'PEDF is computed for each actual donor-recipient pair' does not state that the actual pairs are rarefied to the same depth. Since PEDF is the proportion of donor features observed in a recipient sample, an unrarefied recipient sample with higher sequencing depth can spuriously elevate PEDF, biasing the permutation p-value toward significance. This is a load-bearing procedural detail for the central statistical claim. Please clarify whether actual pairs are also rarefied to the same depth; if they are, state this explicitly, and if they are not, rarefy them or otherwise correct for depth, and validate the procedure with simulations.","section":"Proportional Engraftment of Donor Features (PEDF), pedf-permutation-test"},{"comment":"The novel permutation test is presented as a significance check for genuine donor feature transfer, but the manuscript provides no statistical validation: no simulations to verify Type I error control, no power analysis, no comparison against a gold-standard or biological expectation, and no real-data example showing the null distribution. The unit tests confirm that the code runs, but they do not establish that the test answers the scientific question correctly. Given that this is the paper's main new algorithmic contribution, the authors should either add a benchmark demonstrating the test's validity or revise the claims to avoid asserting statistical validity without support.","section":"Proportional Engraftment of Donor Features (PEDF), pedf-permutation-test"},{"comment":"The p-value calculation is described ambiguously: 'the number of mismatched donor pairs that achieved a PEDS score greater than or equal to the actual pair plus one is divided by num_resamples plus one.' It is unclear whether '+1' applies to the score threshold or to both numerator and denominator as a Monte Carlo correction, and 'PEDS' is an undefined acronym at this point in the text. For a statistical procedure, the exact definition of the p-value must be stated unambiguously and without typographical variants; otherwise the procedure is not reproducible from the manuscript.","section":"Proportional Engraftment of Donor Features (PEDF), pedf-permutation-test"}],"minor_comments":[{"comment":"The displayed PRDF equation is garbled in the manuscript text, appearing as a fraction whose denominator is itself a fraction; the prose description is clear, but the equation should be properly typeset as number of recipients in which the feature engrafted divided by number of recipients whose donor contained the feature.","section":"Proportion of Recipients with Donor Feature (PRDF)"},{"comment":"The caption says 'Jaccard Distance to Donor following FMT intervention' while the main text describes raincloud plots and associated statistics; please clarify whether the figure is a raincloud plot or another visualization, and add uncertainty intervals or point to the interactive version in the documentation.","section":"Figure 1"},{"comment":"In the pedf-permutation-test description, 'PEDS' is used instead of 'PEDF' in the sentence about the permutation p-value; this is confusing because PEDS is a strain-level metric from Aggarwala et al. and the plugin's action is called PEDF.","section":"Design and Implementation, PEDF"},{"comment":"The sentence 'qiimefmtpedfis similarly able to be visualized' in the PPRF paragraph is missing a space and should read 'qiime fmt pedf'; this appears to be a typesetting artifact but should be corrected.","section":"Availability and Future Directions"},{"comment":"The paper states that q2-fmt is a 'comprehensive plugin providing 6 actions and 2 pipelines' but does not provide a complete list of these actions and pipelines in one place; a summary table of actions, inputs, outputs, and corresponding manuscript sections would greatly improve usability and verifiability.","section":"Results"},{"comment":"The claim that 'all actions' in q2-fmt investigate temporal stability through longitudinal data analysis is a stretch, because PEDF, PRDF, and PPRF can be run on a single post-FMT time point; the text should clarify which actions require longitudinal data and which merely support it.","section":"Temporal Stability"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a software description with one genuinely novel statistical component, the pedf-permutation-test. The core issue is that this component is both unvalidated and described asymmetrically with respect to rarefaction. These are fixable within the manuscript's scope: the authors can either add simulations/benchmarks and correct the rarefaction description, or explicitly downgrade the permutation test to an exploratory tool. I do not see circularity in the use of external metrics, and the plugin infrastructure appears solid. The paper is within the scope of q-bio.OT as a methods/software contribution, provided the statistical claims are brought in line with the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a workmanlike software paper that packages existing FMT engraftment metrics into a QIIME 2 plugin, plus two small novelties (PRDF and a permutation test for PEDF). The plugin itself looks real and well-tested, with unit tests, docs, and worked examples on a public dataset. That's worth something. But the permutation test—the one statistical contribution—is not validated anywhere, and the manuscript's own description leaves open a confounding detail: mismatched pairs are rarefied to even depth, while actual pairs are just said to have PEDF computed. If actual recipient samples aren't rarefied to the same depth, the test statistic is biased by sequencing depth and the p-values are suspect. That's a fixable but real gap. I also think 'comprehensive' and 'fully assessing' oversell what a few derived metrics can establish.\n\nWhat's actually new: the generalization from PEDS/PPRS to PEDF/PPRF is trivial but useful; PRDF is a simple per-feature engraftment fraction that will help researchers ask which taxa transfer. The permutation test is a sensible idea—comparing real donor-recipient pairs to mismatched pairs—but the paper gives no simulation, no power analysis, no real-data check that the null distribution behaves. The unit tests confirm the code, not the statistical validity. There's also a typo where the permutation test refers to 'PEDS' instead of 'PEDF'.\n\nThe integration with QIIME 2 is the main selling point: provenance tracking, installability in amplicon and metagenome distributions, and a readthedocs site with examples. For FMT researchers without heavy bioinformatics support, this level of packaging genuinely helps standardization.\n\nMy recommendation: this deserves a serious referee, but the revision should require a validation section (simulated datasets with known engraftment, or a real dataset compared to a gold standard) and an explicit statement about rarefaction of actual pairs. I'd be happy to use the plugin after that; I wouldn't cite the permutation test as evidence until then.","headline":"Useful QIIME 2 plugin for FMT engraftment metrics, but the new permutation test needs validation and the rarefaction ambiguity fixed.","tokens_in":4882,"tokens_out":2490,"would_cite":true,"duration_ms":20702,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"q2-fmt is an open-source plugin that measures how much of a donor gut microbiome persists in a transplant recipient.","keywords":["fecal microbiota transplant","microbiome engraftment","community coalescence","donor feature transfer","permutation test","temporal stability","bioinformatics plugin","PEDF"],"falsifier":"Run the permutation test on synthetic data in which no true donor-recipient transfer occurs; if the resulting p-values are not roughly uniform, or cluster near zero, the test overstates engraftment.","tokens_in":28,"feed_emoji":"🦠","tokens_out":6340,"duration_ms":119348,"temperature":0.7,"pith_summary":"This paper introduces q2-fmt, an open-source software suite for measuring how thoroughly a donor's gut microbiome takes hold in a recipient after fecal microbiota transplant. It implements the three engraftment criteria from a companion review: community coalescence, donor indicator features, and temporal stability. It also generalizes earlier strain-level metrics to feature-level metrics, adds a new feature-level engraftment metric, and proposes a permutation test to check that donor features are transferred rather than shared by chance. If the suite works as described, researchers gain a standardized, reproducible way to compare engraftment across FMT studies, which could clarify whether treatment failures come from the therapy itself or from poor microbial transfer.","feed_headline":"Measure how much of a fecal transplant actually takes","feed_subtitle":"An open-source plugin quantifies donor microbe transfer, persistence and stability in FMT recipients.","key_machinery":"The central object is the ratio of donor features observed in a recipient sample to the total number of donor features, generalized from an earlier strain-level metric. Around this ratio, the plugin builds a permutation test that resamples mismatched donor-recipient pairs with replacement, rarifies each pair to an even depth, recomputes the ratio, and forms a p-value from the fraction of mismatched pairs whose ratio reaches or exceeds the true pair's value. A second ratio tracks the baseline recipient features still present after transplant, and the newly defined recipient-level metric counts, for each donor feature, the fraction of recipients whose donor carried it and in whom it later appeared. Supporting machinery includes a differential-abundance step for flagging donor indicator features and a plotting pipeline for following these quantities over time.","core_discovery":"On its own terms, the paper claims that a single plugin can cover all three published criteria for FMT engraftment: chimeric asymmetric community coalescence, donated microbiome indicator features, and temporal stability. To that end it contributes six actions and two pipelines: community-coalescence visualizations, calculation of the proportional engraftment of donor features with a mismatched-donor permutation test, persistent-recipient-feature calculation, a new recipient-level donor-feature metric, donor-indicator detection through differential abundance, and heatmap and raincloud visualization. The load-bearing new piece is the permutation test: actual donor-recipient feature-sharing values are compared against values computed from donor-recipient pairs that are not actually matched, yielding a p-value for whether observed sharing exceeds chance. The paper also defines a metric equal to the fraction of recipients in which a donor feature appears when the donor carries it, as a way to rank which microbes are most transplantable.","pith_inferences":["The same ratio-and-permutation machinery could be adapted to quantify engraftment in other microbial transfer settings, such as vaginal or skin transplants, where a donor community is introduced into a recipient.","If the recipient-level feature metric proves stable across studies, it could be used to screen donor candidates by preferentially selecting donors whose features have high historical engraftment rates.","The permutation test's validity could be checked retrospectively on existing FMT datasets by comparing its p-values against strain-level transmission calls from a gold-standard method; that comparison is not made in the paper."],"forward_implications":["Engraftment results from different FMT studies can be placed on the same quantitative footing, because the suite fixes the definitions of donor-feature sharing, persistence, and temporal stability.","A significant p-value from the mismatched-donor permutation test gives a study-specific check that an observed donor-feature proportion is higher than would be expected from coincidental sharing alone.","The new recipient-level metric yields a per-feature engraftment rate across recipients, so researchers can identify which donor microbes are most likely to take hold and track whether that changes over time.","Longitudinal visualizations make it possible to watch engraftment, detect whether an individualized microbiome re-emerges years later, and separate treatment failure from failure to engraft."],"supporting_citations":[{"why":"Defines the three engraftment criteria that the plugin implements.","marker":"[1]"},{"why":"Supplies the real-world autologous FMT cancer dataset used for worked examples and figures.","marker":"[2]"},{"why":"Defines the earlier strain-level metric that the paper generalizes to feature-level engraftment.","marker":"[3]"},{"why":"Provides evidence that recipient microbiomes can drift back toward an individualized state, motivating the temporal-stability criterion.","marker":"[4]"},{"why":"Supplies the provenance-replay mechanism that makes the plugin's analyses reproducible.","marker":"[5]"}],"fun_headline_variants":["q2-fmt: measure donor microbe takeover after FMT","FMT engraftment quantified via permutation test","New plugin assesses fecal transplant engraftment","How much donor microbiome persists? q2-fmt tells","Quantify FMT engraftment with q2-fmt's permutation test"],"cache_read_input_tokens":7168,"weakest_assumption_plain":"The weakest load-bearing premise is that the new permutation test's null distribution, built from mismatched donor-recipient pairs, truly represents chance-level feature sharing; the preprint reports no simulation or gold-standard check of this assumption.","fun_headline_variants_meta":{"raw":{"variants":["q2-fmt: measure donor microbe takeover after FMT","FMT engraftment quantified via permutation test","New plugin assesses fecal transplant engraftment","How much donor microbiome persists? q2-fmt tells","Quantify FMT engraftment with q2-fmt's permutation test"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2786,"prompt_tokens":814,"completion_tokens":1972,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":430,"completion_tokens_details":{"reasoning_tokens":1890}},"tokens_in":430,"tokens_out":1972,"duration_ms":14322,"temperature":1.0,"reasoning_tokens":1890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:42:28.890244+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the permutation test on synthetic data in which no true donor-recipient transfer occurs; if the resulting p-values are not roughly uniform, or cluster near zero, the test overstates engraftment.","supporting_citations":[{"cited_title":"raincloudplots","cited_arxiv_id":null,"evidence_quote":"Defines the three engraftment criteria that the plugin implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the real-world autologous FMT cancer dataset used for worked examples and figures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the earlier strain-level metric that the paper generalizes to feature-level engraftment."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides evidence that recipient microbiomes can drift back toward an individualized state, motivating the temporal-stability criterion."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the provenance-replay mechanism that makes the plugin's analyses reproducible."}],"review_version":1}