Module 11 · Lesson 3 of 5
Quantization: smaller numbers, more room
Fewer bits, more requests
Storing each parameter in fewer bits shrinks the model in GPU memory, leaving more room for saved work and so more requests at once.
Scene 1 of 5 · 16 bits each
At 16 bits, the 7 billion parameters fill 14 GB of the 24 GB GPU.
- Waiting requests — healthy
- GPU, 24 GB — healthy — 16-bit parameters
- 16 bits each. At 16 bits, the 7 billion parameters fill 14 GB of the 24 GB GPU.
- Room for saved work. The 10 GB left holds 19,073 tokens of saved work: 9 requests of 2,000 tokens.
- Down to 8 bits. In 8 bits the parameters need 7 GB, so 16 requests fit at once.
- Down to 4 bits. In 4 bits they need 3.5 GB, and 19 requests fit.
- Check the answers. Rounding every number can change some answers, so the smaller model must pass your test set first.
- Now you try. Pick how many bits each parameter gets, and see how many requests fit. For example: Parameters: 7 GB. Left: 17 GB, enough for 16 requests at once. Smaller numbers free memory for saved work, as long as answers stay good enough.
Scene 1 of 5: 16 bits each. At 16 bits, the 7 billion parameters fill 14 GB of the 24 GB GPU.
Key points
- Parameters stored in 16 bits take 2 bytes each.
- Halving the bits halves the memory the parameters need.
- Freed memory holds more saved work, so more requests fit.
- Fewer bits can change answers, so test before switching.
A model's (One of the numbers inside a model that training adjusts. Large language models have billions of them.) are numbers, often stored in 16 bits, which is 2 bytes each. A model with 7 billion parameters then needs 14 GB of (A graphics processing unit: a chip that does many simple sums at the same time. Language models run on GPUs because they need huge numbers of sums.) memory before it serves anyone.
(Storing a model's numbers in fewer bits, such as 8 or 4 instead of 16, so it needs less memory.) stores each number in fewer bits, such as 8 or 4. The parameters then take a half or a quarter of the memory, and the freed memory holds more saved work.
Each (The second stage of an answer, where the model writes one new token per step, using the work it saved.) step reads the parameters from memory, so smaller numbers can make steps faster too. How much depends on the hardware and the software.
Rounding the numbers loses some detail, and answers can get worse. How much depends on the model and the method, so test the smaller model on your test set before switching.
In system design: Quantization is a capacity choice: more requests per GPU, traded against answer quality that you measure.