BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
read the original abstract
We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.
This paper has not been read by Pith yet.
Forward citations
Cited by 4 Pith papers
-
CellxPert: Inference-Time MCMC Steering of a Multi-Omics Single-Cell Foundation Model for In-Silico Perturbation
CellxPert uses inference-time MCMC steering on a multi-omics single-cell foundation model to predict genome-wide transcriptomic responses to gene perturbations and outperforms baselines on cell-type annotation, pertur...
-
Discrete Stochastic Localization for Non-autoregressive Generation
Discrete Stochastic Localization lets a single trained network support an entire family of per-token SNR paths for discrete sequence generation, with masked diffusion as a special case, and improves MAUVE scores when ...
-
Interpolating Discrete Diffusion Models with Controllable Resampling
IDDM interpolates diffusion transitions with a resampling mechanism to lessen dependence on intermediate latents and improve sample quality over masked and uniform discrete diffusion models.
-
Patent Claim Generation by Fine-Tuning OpenAI GPT-2
Fine-tunes GPT-2 on patent claims, probes training steps, analyzes conditional and unconditional sampling outputs, proposes a new sampling method, and releases an email bot for exploration.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.