Pith. sign in

REVIEW 3 major objections 5 minor 29 references

inMOTIFin: a lightweight end-to-end simulation software for regulatory sequences

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read inMOTIFin is a Python tool that generates regulatory DNA sequences with user-defined motifs, co-occurrence grammars, positions, and orientations, and also edits real sequences, giving downstream tools reproducible ground-truth labels.

desk verdict Useful integrated simulator with honest demonstrations; the unaddressed overlap-handling edge case in ground-truth labeling is the one real gap. read the letter →

arxiv 2506.20769 v1 pith:THHLSMFO submitted 2025-06-25 q-bio.GN

Katalin Ferenc , Lorenzo Martini , Ieva Rauluseviciute , Geir Kjetil Sandve , Anthony Mathelier This is my paper
classification q-bio.GN
keywords generegulationsequencesimulationtranscriptionfactormotifsregulatorygrammarmotifdiscoverybenchmarkingdeeplearningexplainabilityPythonsoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

inMOTIFin is a Python package and command-line tool for generating and editing DNA regulatory sequences with fully known ground truth. It lets users define motif grammars, such as which motifs tend to appear together, in what orientation, at what distances, and at which positions, and then produces sequences along with BED, CSV, and MEME files recording exactly where every motif instance lies. The authors demonstrate the tool in three settings: probing a de novo motif discovery tool's sensitivity to motif length and information content, generating dimeric motif sets for studying cooperative transcription factor binding, and inserting or masking motifs to interpret deep learning sequence models. If synthetic sequences generated this way are a fair proxy for real regulatory DNA, the tool provides a reproducible ground truth that real genomic data cannot offer.

What carries the argument

At the core is a directed acyclic graph of sampling steps, implemented with the general-purpose DagSim simulation framework cited in the paper. Each node is a random process or user input, covering background sequence, motif group, motif, instance, orientation, position, and number of instances, and sampled values flow downstream to assemble one final sequence per round. The user controls the grammar through group-level and motif-level multinomial probabilities plus conditional group-group co-occurrence probabilities; a Dirichlet prior on motif columns controls information content per position, and negative spacings allow multimer or noised motifs. The output files record the sampled ground truth so downstream tools can be scored against known positions.

What would settle it

Take a fixed set of real cis-regulatory regions with known binding sites, run a de novo motif discovery tool on them, then run the same tool on an inMOTIFin-generated set matched for motif content, GC content, length, and instance count; if the tool's sensitivity or ranking changes materially between the real and simulated sets, the adequacy of simulated sequences as a benchmarking proxy is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that inMOTIFin is a complete, lightweight simulation environment for regulatory sequences: it generates background DNA, generates position weight matrices with user-controlled information content via a Dirichlet prior, organizes motifs into groups with co-occurrence probabilities, and inserts motif instances at user-specified or distribution-sampled positions and orientations. It also modifies real sequences directly, including masking known motif sites with arbitrary nucleotides. The authors demonstrate in three use cases that this is enough to evaluate de novo motif discovery, simulate primary-secondary motif dimers for cooperativity analysis, and create perturbed inputs plus attribution maps for deep learning explainability. They present these as illustrative applications rather than a comprehensive benchmark against downstream tools, and they emphasize that the modular design lets users replace components such as PWM simulation with other motif representations.

Load-bearing premise

The load-bearing premise is that sequences built from user-specified background distributions plus PWM-sampled motif instances reproduce the difficulty structure of real regulatory DNA; the authors explicitly describe the three use cases as illustrative potential uses, not a comprehensive evaluation against real data.

Editorial extensions

If this is right

  • Users can generate tens of thousands of labeled regulatory sequences in linear time, making it practical to sweep motif length, information content, and insertion density when testing a discovery tool.
  • Dimer and group simulations let cooperativity detectors be tested over controlled mixture compositions; the paper finds that SpaMo's site-level accuracy improves when more dimer combinations are present in the dataset.
  • The masking and exact-position editing functions convert any real sequence into a perturbation testbed, with BED coordinates tying model attribution scores to known motif instances.
  • Because every simulation component can be subclassed, the tool can be extended beyond PWMs to sequence-based motif models, seqlets, or extended alphabets without rewriting the pipeline.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A fair extension of the paper's argument is that inMOTIFin is most valuable for mapping a tool's failure boundary, such as missing short or low-information motifs, rather than for ranking tools absolutely, since simulated difficulty may not match real genomic difficulty.
  • The three use cases suggest a testable hypothesis: replacing the default independent background model with Markov chains, %GC, or chromatin-state-aware backgrounds could change downstream tool rankings; this is not tested in the paper.
  • The finding that dimer-site recovery depends on mixture diversity rather than motif identity is specific to SpaMo and the JASPAR motifs tested, so a cautious reader should not generalize it to all cooperativity tools.
  • An implicit corollary is that the generated BED and CSV ground truth can serve as a negative-control generator for attribution methods, enabling calibration of saliency scores against known motif positions.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. inMOTIFin is an open-source Python package and command-line tool for simulating and modifying regulatory DNA sequences. It can generate background sequences and position weight matrices, group motifs with configurable co-occurrence probabilities, and insert sampled motif instances at user-specified or stochastic positions and orientations through a DagSim-based directed acyclic graph. Outputs include FASTA, BED, CSV, JSON, MEME, and TSV files that record the inserted instances and simulation parameters. Three use cases illustrate de novo motif discovery with RSAT, recovery of co-occurring dimer motifs with SpaMo, and perturbation/attribution analyses of a deep learning model; the authors explicitly frame these as demonstrations of potential use rather than comprehensive benchmarks.

Significance. The software fills a concrete niche: it combines background simulation, motif simulation, grammar-based insertion, and real-sequence modification in one modular package with a documented Python API, command-line interface, PyPI and Docker distributions, and reproducible evaluation code. Transparent support for negative spacing and exact-position insertion are useful for benchmarking motif discovery, cooperativity, and interpretability tools. The paper is appropriately modest about the demonstrations, labeling them as illustrative rather than as validations of downstream tools; for a simulator, the relevant requirement is that the emitted sequences and ground-truth labels match the user's specification exactly. That requirement is currently at risk in exactly the edge cases the tool advertises, namely overlapping insertions and multi-group grammar settings.

major comments (3)
  1. [Features and implementation; Additional features; Supplementary Figure 1] The manuscript advertises precise control over positions and spacing, and explicitly allows negative spacing in multimerization as well as exact per-instance positions through the Python interface. What it does not state is how the assembler resolves motif instances whose intervals overlap. Supplementary Figure 1 describes the final sequence only as being assembled from the selected background sequence, motif instances, positions, and orientations, with no conflict rule. If overlapping insertions silently overwrite, truncate, or shift one instance, then the BED and CSV files will not describe the actual sequence, breaking the ground-truth guarantee that is central to the tool's value. Please specify the semantics for overlapping instances (reject such inputs, define a precedence or merge rule, or resample positions to avoid conflicts), document the behavior, and add tests covering negative-spacing multimers and exact-position overlaps.
  2. [Use cases; Supplementary Section 3] The flagship feature is simulation of regulatory grammars with group-level and group-group co-occurrence, including conditional group-group probabilities when multiple groups occur in one sequence. The only grammar simulation actually presented, however, states in Supplementary Section 3 that 'each sequence contains only one group, making this file ignored.' The pairwise intersection analysis in Supplementary Figure 2 therefore verifies within-group motif co-occurrence but does not exercise the group-group conditional machinery. Since this is a central claimed capability, please add a simulation with multiple groups per sequence and check the observed group-group frequencies against the configured ones, or state explicitly that multi-group co-occurrence is not yet validated.
  3. [Supplementary Section 4.1] The evaluation definitions in the de novo motif discovery use case are internally inconsistent. The protocol inserts all ten motifs in every round (each is observed on average 4,750 times among 47,500 insertions), yet true negatives are defined as motifs 'not found and indeed not inserted,' which cannot occur, and false negatives are defined as motifs matching a pool motif 'not inserted in the specific round,' which is not a false negative in the standard per-round sense. The pooling of motifs from ten rounds into 100 motifs is also not tied to which motifs were inserted in which round. Because the reported MCC values and the stated detection thresholds depend on these definitions, please correct the protocol and recompute the results under a per-round ground-truth comparison.
minor comments (5)
  1. [Use cases, first paragraph] The group assignment in the main text ('motifs 0–2 in group 0') disagrees with Supplementary Table 4, where group 0 contains motifs 0 and 1 and motif 2 is in group 1; please align the text with the configuration actually used.
  2. [Supplementary Table 3] In the group-pair probability table, the third row is labeled 'group 1' twice; it should presumably be labeled 'group 2'. Please correct this typo.
  3. [Implementation] The O(n) complexity claim is supported only by a few timing points in Supplementary Figures 7-9, without fitted curves or a statement of what n denotes for each module; please add a short scaling argument or curve fits.
  4. [Outputs] The BED file documentation does not state whether coordinates are 0-based half-open or 1-based, and the exact schemas of the CSV and JSON outputs are not given in the paper; a one-sentence coordinate convention and schema summary would help reproducibility.
  5. [Supplementary Section 4.3] The deep learning model used in the perturbation and DeepLift analyses is described only as 'simple,' with no architecture or training details; please provide the model description or a pointer to the code used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper's demonstrations are self-consistency checks of a simulator, not predictions that reduce to their own inputs.

full rationale

inMOTIFin is a simulation tool; its central product is synthetic sequences with user-specified inserted motif instances. The three use cases do not derive external predictions from fitted parameters. They show that downstream tools (RSAT, SpaMo, a simple deep learning model) can recover or respond to signals that the user deliberately planted. For example, varying motif length and information content and then measuring RSAT's discovery rate is an evaluation of RSAT's sensitivity, not a circular prediction: the simulation provides controlled inputs, and the observed detection boundaries are an independent outcome. The pairwise intersection analysis in Supplementary Figure 2 confirms that the simulator reproduces the user-provided grammar; this is an implementation check, not a scientific derivation that equates input with output. The paper explicitly disclaims a benchmarking role: 'These examples highlight some of the potential uses of inMOTIFin rather than providing a comprehensive evaluation or benchmarking against downstream tools.' Self-citations (JASPAR, DagSim, Intervene) involve co-authors but are used as external software components and do not carry a load-bearing uniqueness claim or fitted parameter. No equation or construction in the paper defines a claimed result in terms of itself, and no 'prediction' is statistically forced by a prior fit. The unspecified overlap-resolution behavior is a possible correctness issue, but it is not circularity. Therefore the paper is self-contained with respect to circularity concerns.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

inMOTIFin introduces no new free parameters, physical entities, or mathematical axioms. The simulation has user-facing parameters (motif counts, group probabilities, position distributions), but these are inputs to the tool, not constants fitted to make a claim. The central claim does depend on the correctness of the DagSim sampling engine, on the PWM representation of TF binding, and on the realism of randomly generated background sequences.

assumptions (3)
  • domain assumption DagSim draws each node in the directed acyclic graph independently according to the specified distributions without implementation bias.
    inMOTIFin delegates all sampling to DagSim (cited as [11], Sandve is a co-author). If DagSim's sampling has systematic bias, the grammar controls would not hold. The paper does not audit DagSim internally.
  • domain assumption Position weight matrices are a sufficient model of transcription factor binding for the target benchmarking tasks.
    All inserted motif instances are samples from PWMs. This ignores dependencies within binding sites and chromatin context, which may matter for the deep learning use case.
  • domain assumption Random sequences generated with independent per-base frequencies are a valid null background for motif discovery benchmarking.
    Use case 1 uses independent equal base frequencies; no comparison to real background sequence properties or higher-order sequence features is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of inMOTIFin: a lightweight end-to-end simulation software for regulatory sequences." pith.science (2026). https://pith.science/paper/THHLSMFO

@misc{pith2026250620769,
  author       = {Pith},
  title        = {Pith review of: inMOTIFin: a lightweight end-to-end simulation software for regulatory sequences},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/THHLSMFO}},
  note         = {Machine review of arXiv:2506.20769}
}
read the original abstract

The accurate development, assessment, interpretation, and benchmarking of bioinformatics frameworks for analyzing transcriptional regulatory grammars rely on controlled simulations to validate the underlying methods. However, existing simulators often lack end-to-end flexibility or ease of integration, which limits their practical use. We present inMOTIFin, a lightweight, modular, and user-friendly Python-based software that addresses these gaps by providing versatile and efficient simulation and modification of DNA regulatory sequences. inMOTIFin enables users to simulate or modify regulatory sequences efficiently for the customizable generation of motifs and insertion of motif instances with precise control over their positions, co-occurrences, and spacing, as well as direct modification of real sequences, facilitating a comprehensive evaluation of motif-based methods and interpretation tools. We demonstrate inMOTIFin applications for the assessment of de novo motif discovery prediction, the analysis of transcription factor cooperativity, and the support of explainability analyses for deep learning models. inMOTIFin ensures robust and reproducible analyses for studying transcriptional regulatory grammars. inMOTIFin is available at PyPI https://pypi.org/project/inMOTIFin/ and Docker Hub https://hub.docker.com/r/cbgr/inmotifin. Detailed documentation is available at https://inmotifin.readthedocs.io/en/latest/. The code for use case analyses is available at https://bitbucket.org/CBGR/inmotifin_evaluation/src/main/.

Figures

Figures reproduced from arXiv: 2506.20769 by the authors.

Figure 1
Figure 1. inMOTIFin simulation framework: In preparation, background and motif pools are created, and motifs are assigned to groups. Distributions are set up for sampling of various entities. In the sam￾pling phase, backgrounds, motif groups, and subse￾quently motifs are sampled. Two motifs are more likely to co-occur in a sequence if assigned to the same group. When motifs from multiple groups can be inserted into the same s… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 26 canonical work pages

  1. [1]

    Access to ground truth at unconstrained size makes simulated data as indispensable as experimental data for bioinformatics methods development and benchmarking

    Geir Kjetil Sandve and Victor Greiff. Access to ground truth at unconstrained size makes simulated data as indispensable as experimental data for bioinformatics methods development and benchmarking. Bioinformatics , 38(21):4994--4996, October 2022

    work page 2022

  2. [2]

    Perspective on recent developments and challenges in regulatory and systems genomics

    Julia Zeitlinger, Sushmita Roy, Ferhat Ay, Anthony Mathelier, Alejandra Medina-Rivera, Shaun Mahony, Saurabh Sinha, and Jason Ernst. Perspective on recent developments and challenges in regulatory and systems genomics. Bioinformatics Advances , page vbaf106, May 2025

    work page 2025

  3. [3]

    Van Helden, B

    J. Van Helden, B. André, and J. Collado-Vides. Extracting regulatory sites from the upstream region of yeast genes by computational analysis of oligonucleotide frequencies 1 1Edited by G . von Heijne . Journal of Molecular Biology , 281(5):827--842, September 1998

    work page 1998

  4. [4]

    Identification of transcription factor co-binding patterns with non-negative matrix factorization

    Ieva Rauluseviciute, Timothée Launay, Guido Barzaghi, Sarvesh Nikumbh, Boris Lenhard, Arnaud Regis Krebs, Jaime A Castro-Mondragon, and Anthony Mathelier. Identification of transcription factor co-binding patterns with non-negative matrix factorization. Nucleic Acids Research , 52(18):e85--e85, October 2024

    work page 2024

  5. [5]

    Unlocking gene regulation with sequence-to-function models

    Alexander Sasse, Maria Chikina, and Sara Mostafavi. Unlocking gene regulation with sequence-to-function models. Nature Methods , 21(8):1374--1377, August 2024

    work page 2024

  6. [6]

    Assessing computational tools for the discovery of transcription factor binding sites

    Martin Tompa, Nan Li, Timothy L Bailey, George M Church, Bart De Moor, Eleazar Eskin, Alexander V Favorov, Martin C Frith, Yutao Fu, W James Kent, Vsevolod J Makeev, Andrei A Mironov, William Stafford Noble, Giulio Pavesi, Graziano Pesole, Mireille Régnier, Nicolas Simonis, Saurabh Sinha, Gert Thijs, Jacques Van Helden, Mathias Vandenbogaert, Zhiping Weng...

    work page 2005

  7. [7]

    Quantifying similarity between motifs

    Shobhit Gupta, John A Stamatoyannopoulos, Timothy L Bailey, and William Stafford Noble. Quantifying similarity between motifs. Genome Biology , 8(2):R24, February 2007

    work page 2007

  8. [8]

    Price, and Donald Geman

    David Simcha, Nathan D. Price, and Donald Geman. The Limits of De Novo DNA Motif Discovery . PLoS ONE , 7(11):e47836, November 2012

    work page 2012

Show all 29 references
  1. [9]

    Learning Important Features Through Propagating Activation Differences , October 2019

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning Important Features Through Propagating Activation Differences , October 2019. arXiv:1704.02685 [cs]

    2019 arXiv

  2. [10]

    Prakash, Avanti Shrikumar, and Anshul Kundaje

    Eva I. Prakash, Avanti Shrikumar, and Anshul Kundaje. Towards more realistic simulated datasets for benchmarking deep learning models in regulatory genomics. In David A. Knowles, Sara Mostafavi, and Su-In Lee, editors, Proceedings of the 16th Machine Learning in Computational ...

    2022

  3. [11]

    Al Hajj, Johan Pensar, and Geir K

    Ghadi S. Al Hajj, Johan Pensar, and Geir K. Sandve. Dagsim: Combining dag-based model structure with unconstrained data types and relations for flexible, transparent, and modularized data simulation. PLOS ONE , 18(4):1--9, 04 2023

    2023

  4. [12]

    RSAT 2022: regulatory sequence analysis tools

    Walter Santana-Garcia, Jaime A Castro-Mondragon, Mónica Padilla-Gálvez, Nga Thi Thuy Nguyen, Ana Elizondo-Salas, Najla Ksouri, François Gerbes, Denis Thieffry, Pierre Vincens, Bruno Contreras-Moreira, Jacques van Helden, Morgane Thomas-Chollier, and Alejandra Medina-Rivera. RS...

    2022

  5. [13]

    Absence of a simple code: how transcription factors read the genome

    Matthew Slattery, Tianyin Zhou, Lin Yang, Ana Carolina Dantas Machado, Raluca Gord \^a n, and Remo Rohs. Absence of a simple code: how transcription factors read the genome. Trends in biochemical sciences , 39(9):381--399, 2014

    2014

  6. [14]

    Frith, James Johnson, and Timothy L

    Tom Whitington, Martin C. Frith, James Johnson, and Timothy L. Bailey. Inferring transcription factor complexes from ChIP -seq data. Nucleic Acids Research , 39(15):e98--e98, August 2011. Publisher: Oxford University Press (OUP)

    2011

  7. [15]

    A review of deep learning applications in human genomics using next-generation sequencing data

    Wardah S Alharbi and Mamoon Rashid. A review of deep learning applications in human genomics using next-generation sequencing data. Hum. Genomics , 16(1):26, 2022

    2022

  8. [16]

    Predicting gene expression from DNA sequence using deep learning models

    Luc \' a Barbadilla-Mart \' nez, Noud Klaassen, Bas van Steensel, and Jeroen de Ridder. Predicting gene expression from DNA sequence using deep learning models. Nat. Rev. Genet. , 2025

    2025

  9. [17]

    Exploring explainability and transparency in deep neural networks: A comparative approach

    J Bhagya, Jeena Thomas, and Ebin Deni Raj. Exploring explainability and transparency in deep neural networks: A comparative approach. In 2023 7th International Conference on Intelligent Computing and Control Systems (ICICCS) , pages 664--669, 2023

    2023

  10. [18]

    Perturbation-based methods for explaining deep neural networks: A survey

    Maksims Ivanovs, Roberts Kadikis, and Kaspars Ozols. Perturbation-based methods for explaining deep neural networks: A survey. Pattern Recognition Letters , 150:228--234, 2021

    2021

  11. [19]

    tangermeme: Tutorial a4: Seqlets

    Jacob Schreiber. tangermeme: Tutorial a4: Seqlets

  12. [20]

    Ishak, James Johnson, Nicolas J

    Coby Viner, Charles A. Ishak, James Johnson, Nicolas J. Walker, Hui Shi, Marcela K. Sjöberg-Herrera, Shu Yi Shen, Santana M. Lardo, David J. Adams, Anne C. Ferguson-Smith, Daniel D. De Carvalho, Sarah J. Hainer, Timothy L. Bailey, and Michael M. Hoffman. Modeling methyl-sensit...

    2024

  13. [21]

    Beyond authorship: attribution, contribution, collaboration, and credit

    Amy Brand, Liz Allen, Micah Altman, Marjorie Hlava, and Jo Scott. Beyond authorship: attribution, contribution, collaboration, and credit. Learned Publishing , 28(2):151--155, April 2015

    2015

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION string.to.integer 't := t text.length 'k := #1 'char.num := t char.num #1 substring 's := s is.num s "." = or char.num k = not and char.num #1 + 'char.num := while char.num #1 - 'char.num := t #1 char.num substring FUNCTI...

  15. [23]

    JASPAR 2024: 20th anniversary of the open-access database of transcription factor binding profiles

    Ieva Rauluseviciute, Rafael Riudavets-Puig, Romain Blanc-Mathieu, Jaime.A Castro-Mondragon, Katalin Ferenc, Vipin Kumar, Roza Berhanu Lemma, Jérémy Lucas, Jeanne Chèneby, Damir Baranasic, Aziz Khan, Oriol Fornes, Sveinung Gundersen, Morten Johansen, Eivind Hovig, Boris Lenhard...

    2024

  16. [24]

    pyJASPAR : a Pythonic interface to JASPAR transcription factor motifs, February 2021

    Aziz Khan. pyJASPAR : a Pythonic interface to JASPAR transcription factor motifs, February 2021

    2021

  17. [25]

    universalmotif: An R package for biological motifanalysis

    Benjamin Jean-Marie Tremblay. universalmotif: An R package for biological motifanalysis. Journal of Open Source Software , 9(100):7012, August 2024

    2024

  18. [26]

    Base-resolution models of transcription-factor binding reveal soft motif syntax

    Ziga Avsec, Melanie Weilert, Avanti Shrikumar, Sabrina Krueger, Amr Alexandari, Khyati Dalal, Robin Fropf, Charles McAnany, Julien Gagneur, Anshul Kundaje, and Julia Zeitlinger. Base-resolution models of transcription-factor binding reveal soft motif syntax. Nature Genetics , ...

    2021

  19. [27]

    rMotifGen : random motif generator for DNA and protein sequences

    Eric C Rouchka and C Timothy Hardin. rMotifGen : random motif generator for DNA and protein sequences. BMC Bioinformatics , 8(1):292, December 2007

    2007

  20. [28]

    Intervene: a tool for intersection and visualization of multiple gene or genomic region sets

    Aziz Khan and Anthony Mathelier. Intervene: a tool for intersection and visualization of multiple gene or genomic region sets. BMC Bioinformatics , 18(1):287, December 2017

    2017

  21. [29]

    A unified approach to interpreting model predictions

    Scott Lundberg. A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874 , 2017

    2017 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.