The part of this I keep rereading is the throughput table for the KV cache. At one concurrent request, BF16 beats FP8 — 137 tokens per second against 125. FP8 is slower. It only wins because BF16 runs out of memory at 32 concurrent requests and FP8 keeps going to 64, ending up about 41% higher at peak for roughly 30% less cost per token. The optimization looks like a loss on the benchmark that most people would run first.
That’s the shape of every memory-constrained decision I make on a boat. You don’t tune for the single-user case, you tune for the case where you run out. Cloudflare’s answer is to stop treating it as one knob: because they run prefill and decode as separate pools, they keep BF16 for prefill, which is compute-bound, and FP8 for decode, which isn’t. Same with GLM 5.2’s INT4 weights — faster decode, slower prefill at 8,660 tokens per second versus 10,160, so they run each where it wins.
What I’d steal: measure at the concurrency you actually hit, not at one. And notice they published the accuracy tables to prove nothing moved.
The story — Cloudflare described three techniques layered onto Workers AI to serve Moonshot’s Kimi and Z.ai’s GLM: FP8 KV cache quantization, which doubles Kimi K2.6’s in-memory context from roughly 686,000 to about 1.37 million tokens; INT4 weight compression, shrinking GLM 5.2’s checkpoint from 705 GB to 421 GB; and KV cache integrity checking costing under 1% throughput. All benchmarked on SGLang. (Source)