As I discussed in my last post, the price of consumer GPUs likely isn’t changing anytime soon. Following various threads, most are either accepting the price point or looking at dedicated hardware like DGX Spark which is around the same $5,000 price point (and many joke of “get 2 or 3!”).

My homelab rack is still rocking the GTX 1080 I found in a storage closet and got running inside an LXC container a while back. I hadn’t planned to dig into it anymore (it’s setup and working), but then I went looking for a bigger local model and found that while not much had changed on my end, a lot had changed inside the models.

Today, the same card runs models that would have been flatly out of reach twelve months back at a speed that doesn’t make me regret asking a question, because smarter quantization, sparse model architectures that only wake up part of themselves per token, and a decoding trick that predicts more than one token at a time all matured at close to the same time.

Let’s dig in and see what we can run.

Squeezing without losing the shape

Quantization is the oldest trick in this list. A large language model is billions of numbers, weights, arranged into multi-dimensional matrices called tensors. During inference, your prompt flows through attention layers, which calculate how much each word relates to every other word, and feed-forward layers, multiplying your input against those weights billions of times.

When a lab trains a model or hosts it on OpenRouter, those weights are typically stored in 16-bit floating point, FP16 or BF16, where each weight has more than 65,000 possible values. That fidelity is great, but storing billions of 16-bit numbers costs a lot of VRAM. Quantization squeezes those dials into smaller ranges: 8-bit integers (256 values), 4-bit integers (16 values), or even 2-bit integers (4 values). Smaller numbers means proportionally less memory.

The problem was uniform quantization. Cutting every layer to the same bit depth treated the most sensitive attention weights the same as the most forgiving feed-forward layers. Push the bit depth low enough and the model starts forgetting how to finish a sentence.

What changed is how selectively that cutting happens. Unsloth, one of the more active teams working on quantization for local models, now calibrates which layers tolerate aggressive compression and which don’t, using a dataset built for chat and agentic performance rather than generic web text. Their Dynamic 3.0 release claims more than 10% better top-1% accuracy at the same file size than other providers, measured with Divergence-300 @32 and KL Divergence.

Kullback-Leibler (KL) Divergence is a statistical measure of how much one probability distribution differs from a reference baseline. When an LLM generates text, it doesn’t just pick a word; it assigns a probability percentage to every word in its entire 32,000+ token vocabulary. KL Divergence compares the entire probability curve generated by a compressed local model against the uncompressed original model.

Standard benchmarks like MMLU only check if the model picked the single right multiple-choice letter (e.g., “B”), meaning lucky guesses can mask real cognitive decline. KL Divergence acts like an MRI for the model’s confidence: if the uncompressed model was 99% confident in an answer, but the quantized version dropped to 51% (even if it still guessed the right letter), KL Divergence catches that subtle confidence erosion immediately.

To see how bit precision affects token selection in practice, think of how a model predicts the next word in:

“The quick brown ___”:

• In FP16 (Full Precision): The model calculates distinct probabilities: fox: 88.4%, dog: 7.2%, squirrel: 0.4%. fox is the decisive winner.
• In well-calibrated 4-bit (UD-Q4_K_XL): Minor rounding shifts the math slightly: fox: 86.1%, dog: 8.5%, squirrel: 0.6%. The gap remains huge, and the model picks fox every time.
• In aggressive 2-bit or uncalibrated quants: Thousands of tiny rounding errors compound across dozens of layers. The probability curve flattens: fox: 38.2%, dog: 34.1%, squirrel: 27.7%.

Apply standard generation temperature, and suddenly your sentence swerves off like squirrels or forgets grammar rules entirely.

This trade also explains why a model like Qwen-3.8-27B on OpenRouter can feel subtly smarter than the same model on an 8GB card at home. In the cloud the provider serves unquantized FP16 or high-precision FP8 across H100s. When you compress that to a 4-bit or 3-bit local file, rounding errors creep in across billions of multiplications.

For 95% of everyday tasks, summarizing text, drafting code, or answering conversational questions, those rounding errors are imperceptible. A well-calibrated Q4_K_M or UD-Q4_K_XL is practically indistinguishable from raw FP16 to a human, which makes avoiding quants for fear of “dumbed-down” models mostly unwarranted. The gap shows up in high-entropy edge cases: strict JSON schemas, long logic chains where errors compound, and autonomous tool calling.

KL Divergence catches degradation that raw accuracy can miss, though not as cleanly as it’s usually pitched. Raw accuracy on a benchmark like MMLU can stay flat while behavior degrades, because wrong answers flip to right about as often as right answers flip to wrong. A 2024 paper on pruning and quantization, Accuracy is Not All You Need, showed this directly: shrinking a model can leave its benchmark score nearly untouched while its outputs diverge from the original.

The comparison gets messier in the range most people choose between. A June 2026 study evaluating 28 GGUF quantizations of Qwen3.6-35B-A3B and 41 of Devstral-Small-2-24B found KL Divergence correlates strongly with quality across the full range, badly degraded 2-bit quants included, but that correlation collapses to non-significance when limited to near-baseline candidates, the well-calibrated 4-bit-and-up quants people are picking between. KL Divergence reliably flags a quant that’s clearly broken. It is much weaker at ranking two quants that both look fine, which is why it belongs alongside accuracy rather than in place of it.

Push a quantized model below about 2 bits per weight and Unsloth’s own docs are blunt about what breaks. Tool calling and reliable responses degrade, and general knowledge is the only thing that survives that low. The practical range for real work sits well above that floor, but the floor has dropped: where 4-bit used to be the safe minimum for a model you could trust, well-calibrated 2-bit now holds up for a wider set of tasks than it did a year ago.

Only waking up part of the model

Quantization shrinks every weight a little, while mixture-of-experts takes a different approach by making the model bigger overall but only using a fraction per token.

A dense model activates every parameter for every token. A 31-billion-parameter dense model does 31 billion parameters worth of math on each word it generates. A mixture-of-experts model splits its weights into many small “experts,” and a router picks a handful, often 8 out of 128, to run for a given token, plus in many designs one or more shared experts that fire on every token. The rest sits idle. It still takes disk and system memory, but it doesn’t burn compute.

Google’s Gemma 4 family ships both versions side by side, which makes the trade easy to see. The 31B dense model and the 26B-A4B mixture-of-experts model land close on quality, at 85.2% versus 82.6% on MMLU Pro and 80.0% versus 77.1% on LiveCodeBench v6, according to Google’s own model card. The MoE version has 25.2 billion total parameters, but only 3.8 billion activate per token, about 15% of the total. Google puts it directly: the MoE model “runs almost as fast as a 4B-parameter model” despite carrying about 6.6 times as many total parameters as it uses on any token.

The tradeoff, about 2.6 points of MMLU Pro for a model that acts like a fraction of its size, is what makes an old GPU relevant again. Full weights still have to live somewhere, but they don’t all have to live in VRAM. llama.cpp added a flag that keeps the shared, always-active layers on GPU while routed experts stay on CPU and system RAM, so an 8GB card only needs the always-resident slice plus whatever context you asked for. Ollama, which wraps llama.cpp and is what I run day to day, exposes the same offloading through its own layer settings rather than the raw flag name. The “3.8B active” figure counts everything touched per token, including whichever experts the router selects, not only what’s pinned to VRAM.

Predicting more than one word ahead

The third technique helps with speed. A standard transformer generates one token, feeds it back in, generates the next, and repeats, so each token costs a full forward pass, which is most of why local inference feels slower than a hosted API even on comparable hardware.

Multi-token prediction changes that. Instead of committing to one token and waiting, the model predicts several tokens ahead in one pass using a small extra module, then checks which predictions the full model would have produced. DeepSeek’s technical report describes the MTP objective as a training improvement first, then notes the same modules can be repurposed for speculative decoding to cut latency. The drafter proposes future tokens, the main model verifies them, and accepted predictions let generation skip ahead instead of computing every token one at a time.

Unsloth takes the same idea in a different direction by shipping Gemma 4 with a separate MTP drafter file, under a gigabyte, next to the main quantized weights on Hugging Face. Unsloth’s version is a standalone draft checkpoint. DeepSeek’s version uses heads baked into the main model. For local use the effect is the same: the drafter proposes several tokens, the full model checks them in one pass, and accepted predictions skip the per-token work. Smaller quants sometimes ship without the drafter to save disk, losing the speed benefit with it.

What this buys on old hardware

Back in March, I settled on phi-4-mini, a 3.8B model with 128K context that ran quite well for what I needed, but it wasn’t a powerhouse of reasoning and lacked vision services.

Today, stack the three concepts we just discussed and the GTX 1080 still has life in it.

Layer-aware quantization keeps more judgment at a smaller file size, mixture-of-experts means most of the file never has to touch the GPU since llama.cpp can pin the always-active layers to VRAM and leave rarely used experts in system memory, and multi-token prediction cuts the number of full forward passes needed for the same text where it is supported.

For an 8GB card specifically, not holding the whole model in VRAM leaves room for something else: context. That means longer conversations or more room for the model to work through a problem before answering. It also leaves room for the KV cache, the memory that stores past tokens, which grows quickly in long chats. With quantized KV caching (q8_0 or q4_0 in llama.cpp and Ollama), an 8GB buffer that used to choke on 4k context can comfortably handle 16k or 32k tokens. That trade, less model in VRAM for more room to think, is close to the opposite of what hardware marketing sells.

This looks like it should contradict the dense 13B result I reported on this same GTX 1080 back in March, where CPU offloading dropped throughput to 1-3 tokens per second. It doesn’t, because the difference is what has to cross PCIe. A dense model needs every parameter for every token, so offloading any of it makes the GPU wait on system RAM every forward pass. A mixture-of-experts model only needs the router’s chosen experts for that token, and consecutive tokens often reuse the same few experts, so CPU-resident weights get touched less often per token.

So what’s this get us today?

• gemma-4-12B is solid at 4-bit with MTP turned on coming in with a solid 50-60tps and offering reasoning, multimodal (vision), and agentic capabilities.
• qwen3.8-27B is a heavy duty coding model with tooling for research, coding, and vision (including video). It’s a tight fit and requires some tuning, but a 27B model works at about 25-35tps.

That’s 12B and 27B models with reasoning and vision running on the exact same hardware where we were pushing limits with a 3.8B model less than a year ago.

Where this doesn’t help

None of these three techniques are hard to find. Most serious releases ship quantized GGUFs within days, mixture-of-experts has become common enough that dense versus MoE is now a routine choice on the download page, and multi-token prediction is spreading model by model instead of staying one lab’s trick. All three matured into well-documented options within about the same year, close enough to stack instead of compete.

I’m still watching where the accuracy lands for the lower-bit models I’m working with. Aggressive quantization below roughly 2 bits still breaks tool calling more than I’d trust for anything beyond casual use, but at 4-bit or so, things have been solid. If your work needs a model to behave identically every time, or you need frontier judgment for something that matters, none of this replaces paying for the real thing.

For testing ideas on a card that was already paid for, the ceiling moved, and that’s worth knowing before anyone spends $5,000 on a GPU to solve a problem the model itself already solved.