KV Cache Memory: How Many Llama 3 Tokens Fit on One H200
KV cache memory on one H200: Llama 3 70B at 16-bit weights leaves 1.0 GB, about 3,052 tokens. Byte formula, unit trap, and a worked example.
KV cache
KV cache memory on one H200: Llama 3 70B at 16-bit weights leaves 1.0 GB, about 3,052 tokens. Byte formula, unit trap, and a worked example.