BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

Alex Wang , Kyunghyun Cho

Authors on Pith no claims yet

classification 💻 cs.CL cs.LG

keywords bertlanguagemodelfieldgenerationsmarkovrandomsentences

read the original abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

This paper has not been read by Pith yet.

discussion (0)

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation
q-bio.GN 2026-04 unverdicted novelty 7.0

CellxPert uses inference-time MCMC steering on a multi-omics single-cell foundation model to predict genome-wide transcriptomic responses to gene perturbations and outperforms baselines on cell-type annotation, pertur...
Interpolating Discrete Diffusion Models with Controllable Resampling
cs.LG 2026-04 unverdicted novelty 6.0

IDDM interpolates diffusion transitions with a resampling mechanism to lessen dependence on intermediate latents and improve sample quality over masked and uniform discrete diffusion models.