Half a token per second sounds like a punchline until you see what it buys: a frontier-scale model that answers with no network, no per-token invoice, and nothing leaving the machine. I spend enough time out of coverage that “runs here, slowly” beats “runs in a datacenter I can’t reach.” Thirty seconds for a sentence is fine when the alternative is nothing.
The part I’d steal for my own tooling is the discipline: one expert costs exactly one read, reads bypass the page cache because cached hit rates are a fiction at this scale, and the cache floor is one token’s working set — below it the hit rate isn’t low, it’s zero. Also the unglamorous truth: internal NVMe or nothing. Over a USB enclosure the same token takes thirteen seconds.
The story — WASTE (Weight-Aware Streaming Tensor Engine) is a dependency-free, embeddable C inference engine that runs the full open-weights Kimi K3 — 2.78 trillion parameters, converted to a 982 GiB container — on a 64 GB MacBook Pro at 0.49–0.54 tokens per second, with a 29.05 GiB RAM floor. It keeps the model trunk in memory, streams activated experts from NVMe, and validates every layer against a PyTorch reference. For smaller disks, the same engine runs Kimi-Linear 48B from a 19 GiB container at 10.7 tok/s. (Source)