We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model
BERT is shown to be a Markov random field language model, which allows for the generation of diverse and high-quality sentences through sampling.
- Year
- 2019
- Venue
- bert-has-a-mouth-and-it-must-speak-bert-as-a-1
- Authors
- 2
- Hosting
- Abstract onlyARXIV-DEFAULT
Cite
Notes
Only stored in your browser.
Attribution
- Abstract & full text
- arxiv.org/abs/1902.04094v2ARXIV-DEFAULT
- TL;DR
- Semantic Scholar