Running Local AI on Your GPUThe MXFP4 and MoE Breakdown

Why 117B models run on a single 80 GB GPU

A one hundred and seventeen billion parameter model can now run on a single eighty gigabyte GPU, and it generates tokens at the speed of a tiny five billion parameter model. That is not a benchmark trick, and it is not aggressive pruning. It is the result of two architectural shifts that broke the old rules of local AI: Mixture of Experts and native microscaling FP4 quantization.

For years, the formula for running local models on your GPU was simple: multiply parameter count by bytes per weight. In twenty twenty six, that formula is completely dead. Here is the math behind why active parameters dictate your speed while total parameters dictate your VRAM, how gpt-oss-120b and gpt-oss-20b fit on real hardware, and how to calculate your true memory footprint including the KV cache.

The old dense formula: Gemma 4 31B vs 26B

To see why things changed, look at how we used to calculate local model memory. On a traditional dense model, every single parameter activates on every single token. If you run a dense model like Gemma 4 with thirty one billion parameters, your GPU has to stream all thirty one billion weights through its memory bus for every token you produce. At sixteen-bit precision, that takes sixty two gigabytes.

Even quantized down to four-bit weights, Gemma 4 twenty six B needs sixteen gigabytes of VRAM just to load, and twenty six gigabytes at eight-bit. If you do not have the memory bandwidth, your tokens per second crawl to a halt. On edge devices, Google designed Gemma 4 E4B to stay under four gigabytes so it runs on basic laptops, but dense architectures always force a direct tradeoff between reasoning capability and hardware cost.

Mixture of Experts: active speed vs total VRAM

Mixture of Experts breaks that tradeoff by separating parameter capacity from compute cost. Take OpenAI open-weight gpt-oss-120b. The model contains one hundred and seventeen billion total parameters, but it only activates five point one billion parameters per token. Instead of passing your prompt through every layer uniformly, an internal router sends the token to a small subset of specialized expert networks.

Because only five point one billion weights compute each token, the generation speed feels like running a lightweight model. But here is the critical rule that catches developers buying GPUs: all one hundred and seventeen billion weights still have to reside in VRAM. The router cannot predict which expert will be needed on the next token, so every expert must stay warm in GPU memory.

Compute speed scales with active parameters, but memory capacity scales with total parameters.

Native MXFP4 quantization: gpt-oss-120b and 20b

That brings us to the second shift: microscaling FP4 quantization, or MXFP4. Historically, quantizing a model down to four bits caused noticeable accuracy loss, especially in coding and complex mathematical reasoning. Both gpt-oss-120b and gpt-oss-20b were post-trained with MXFP4 quantization directly on their Mixture of Experts weights.

Microscaling groups weights into small blocks with shared scale factors, preserving dynamic range and keeping benchmark precision intact. Because of MXFP4, all one hundred and seventeen billion parameters of gpt-oss-120b compress into eighty gigabytes of memory, fitting entirely on a single professional accelerator like an NVIDIA H100 or AMD MI300X.

On the consumer side, gpt-oss-20b packs twenty one billion total parameters with three point six billion active parameters into sixteen gigabytes of VRAM, running comfortably on a standard RTX 4080.

The KV cache trap: why 16 GB GPUs crash

Now here is the hidden trap that causes out of memory crashes after twenty minutes of work: the KV cache. When you read a model spec, the listed VRAM requirement only covers the model weights at idle. The moment you start a long context session, your GPU begins allocating memory for key and value tensors across every layer and attention head.

At sixteen-bit precision, a one hundred and twenty eight thousand token context window consumes between four and eight gigabytes of additional VRAM depending on the attention head count. If your weights take fifteen gigabytes on a sixteen gigabyte card, that KV cache will push you over the edge and trigger an out of memory error. The solution is KV cache quantization.

Compressing your cache to eight-bit or four-bit cuts your context memory overhead by fifty to seventy five percent with virtually zero degradation in retrieval accuracy.

Hardware sweet spots: RTX 4080 to H100

So how do you match your hardware to local models today? If you have a sixteen gigabyte consumer GPU like an RTX 4080, your sweet spot is gpt-oss-20b or Gemma 4 twenty six B in four-bit. You get twenty one billion parameters of reasoning with three point six billion active compute speed, fitting model weights and a solid context window within your VRAM budget.

0 license, running on a single accelerator without requiring multi-GPU pipeline parallelism. Here is the recap.

Three rules for running local AI

First, never look at total parameter count alone. In an MoE architecture, active parameters decide your token speed, while total parameters decide your VRAM wall. Second, native MXFP4 quantization lets twenty one billion parameters fit into sixteen gigabytes, and one hundred and seventeen billion fit into eighty gigabytes.

Third, always budget four to eight gigabytes of headroom for your KV cache, or enable cache quantization when running long context. If this helped, subscribe. Build with AI. For real.