Qwen3 4B: more context allocated more GPU memory
With the same Q4_K_M model file and runtime commit, increasing the configured context from 2,048 to 8,192 tokens raised the allocated FP16 KV cache from 288 to 1,152 MiB. Both our T4 and GTX 1660 Ti runs reported those cache sizes.
The measured change
On the T4, sampled request peaks rose from 2,845–2,847 MiB at 2K context to 3,811 MiB at 8K. On the GTX 1660 Ti, the corresponding peaks were 2,803 and 3,767 MiB. Each context setting was repeated three times on each GPU.
| Configured context | Runtime KV allocation | T4 request peak | GTX 1660 Ti request peak | Reproduce estimate |
|---|---|---|---|---|
| 2,048 tokens | 288 | 2,845–2,847 | 2,803 | Open T4 inputs |
| 4,096 tokens | 576 | 3,169 | 3,125 | Open T4 inputs |
| 8,192 tokens | 1,152 | 3,811 | 3,767 | Open T4 inputs |
The 2K-to-8K change added 864 MiB of KV cache. The sampled total-device peak increased by roughly 964–966 MiB on T4 and 964 MiB on GTX 1660 Ti. Runtime and device allocation explain why the total change is not identical to the cache change alone.
What ran
- Qwen3 4B Q4_K_M, 2,497,280,256 bytes. Both machines used the same verified file.
- llama.cpp b11429, commit
d81235049384534c167caea52b85a694f6103d14, all layers on the GPU, automatic fit disabled. - FP16 K and V cache, Flash Attention off, batch 512, microbatch 128, one request at a time.
- 40 prompt tokens and 128 generated tokens, sampled at a target interval of 200 ms.
- T4: Colab Linux, 15,360 MiB exposed device capacity, driver 580.82.07, CUDA build 12.8; measured October 16, 2026.
- GTX 1660 Ti: Windows WDDM, 6,144 MiB, driver 617.42, CUDA build 12.4; measured October 14, 2026.
Download T4 logs and reproduction script · Download GTX 1660 Ti logs and reproduction script · Exact model file
Interpretation and test limits
A short request can still reserve a larger cache when the configured context grows. Sizing GPU memory from model weights alone misses that allocation.
These runs did not fill the entire configured context. Long-prompt prefill, concurrent requests and other backends can allocate more memory. A 200 ms sample can miss a shorter peak, so the results are sampled observations rather than exact maximum allocations.
The machines used different operating systems, drivers and CUDA packages. Their small difference in device memory is not evidence that one GPU is faster or more memory-efficient across workloads.
The calculator uses the same cache geometry, plus a separate adjustable runtime allowance and reserve. Its output remains an estimate; the measured values have not been substituted for the general formula.
Calculate your model and context · Full measurement records and method