We provide new estimates of an asymptotic upper bound on the entropy of English using the large language model LLaMA-7B as a predictor for the next token given a window of past tokens. This estimate is significantly smaller than currently available estimates in \cite{cover1978convergent}, \cite{lutati2023focus}. A natural byproduct is an algorithm for lossless compression of English text which combines the prediction from the large language model with a lossless compression scheme. Preliminary results from limited experiments suggest that our scheme outperforms state-of-the-art text compression schemes such as BSC, ZPAQ, and paq8h.
LLMZip: Lossless Text Compression using Large Language Models
A new estimate of English entropy using LLaMA-7B for next-token prediction leads to a compression algorithm that outperforms current text compression methods.
- Year
- 2023
- Venue
- arXiv 2023
- Authors
- 5
- Hosting
- Abstract onlyARXIV-DEFAULT
Cite
Notes
Only stored in your browser.
Attribution
- Abstract & full text
- arxiv.org/abs/2306.04050v2ARXIV-DEFAULT
- TL;DR
- Semantic Scholar