A single 24GB card is the sensible flooring for severe native inference. It’s sufficient for genuinely succesful fashions, and sufficiently small to sit down on one GPU. An RTX 3090 or RTX 4090 each land on this tier. The cardboard you personal issues lower than the fashions you choose for it.
The outdated hobbyist transfer was to squeeze the largest 70B quant onto the cardboard. That recommendation is now outdated. The stronger 2026 technique makes use of trendy 20B–35B-class fashions that match cleanly. These go away room for context, and nonetheless reply quick sufficient for coding, chat, and brokers. This information covers the fashions that really match, why every is value working, and the way 24GB will get spent.
How 24GB of VRAM really will get spent
Three issues devour reminiscence throughout inference. Getting the cut up proper decides whether or not a mannequin matches.
The first is mannequin weights. Their dimension is dependent upon parameter rely and quantization. At Q4_K_M, a typical home-inference default, every parameter prices roughly 0.58 bytes. A 32B mannequin due to this fact wants about 18–20GB in weights alone. Mixtral-style Combination-of-Consultants (MoE) fashions are the widespread lure right here. Each skilled stays resident in VRAM even when only some route per token. So that you dimension MoE reminiscence by complete parameters, by no means lively parameters.
The second is the KV cache, which grows with context size. Longer prompts and longer periods eat extra VRAM. The third is runtime overhead from the serving stack. A secure rule of thumb provides roughly 1–2GB for the KV cache and runtime at brief context.
Quantization is the lever that makes 24GB workable. Q4_K_M is the usual steadiness of high quality and footprint. Q5_K_M and Q6_K elevate high quality at a reminiscence value. Q8_0 and BF16 are normally too massive for 30B-class fashions on a single 24GB card. The interactive software under helps you to swap quantization and see every mannequin’s match change stay.
The six finest native LLMs for a 24GB GPU
Each mannequin under carries a permissive license, both Apache 2.0 or MIT. All of them match a single 24GB card at Q4_K_M with room for context. They’re grouped by the job each does finest.
Qwen3.6-27B — finest all-around and agentic coding
Alibaba’s Qwen3.6-27B is the strongest single default for the cardboard. It’s a dense 27B mannequin launched in April 2026 beneath Apache 2.0. The discharge focuses on agentic coding, repository-level reasoning, and frontend workflows. At Q4_K_M it wants roughly 16GB, leaving comfy headroom for context. Full mannequin playing cards and the changelog sit within the Qwen3.6 GitHub repository.
Qwen3.6-35B-A3B — quickest general-purpose choice
The Qwen3.6-35B-A3B is a Combination-of-Consultants mannequin with 35B complete parameters and about 3B lively per token. It decodes far sooner than a dense 35B mannequin as a result of solely a fraction of weights hearth every step. The reminiscence footprint nonetheless tracks complete parameters, so plan for roughly 20GB at Q4_K_M. That makes it a good however legitimate match, finest while you need velocity for common chat and power use.
Gemma 4 26B — multimodal and multilingual
Google DeepMind launched Gemma 4 on April 2, 2026, beneath Apache 2.0. It was the primary Gemma technology to ship with a totally open license, per the DeepMind Gemma web page. The household spans edge fashions, a 12B unified multimodal mannequin, a 26B MoE with 3.8B lively, and a bigger dense flagship. The 26B MoE is the pure 24GB choose while you want imaginative and prescient enter and 140+ language protection.
Mistral Small 3.2 24B — polished each day assistant
Mistral’s Small line targets the on a regular basis assistant workload with low latency. Mistral Small 3.1 added multimodal enter and an extended context window, and Small 3.2 refined instruction following. It’s a 24B dense mannequin beneath Apache 2.0, and the lightest footprint on this record at roughly 14GB at Q4_K_M. That headroom helps you to push context additional than the heavier 32B choices enable. In 2026 Mistral additionally shipped bigger successors, however these sit above the single-card tier.
gpt-oss-20b — the simple reasoning fallback
OpenAI’s gpt-oss-20b is an open-weight reasoning mannequin beneath Apache 2.0. It’s a Combination-of-Consultants design with 21B complete parameters and three.6B lively per token. It ships in a local MXFP4 4-bit format, loading in roughly 14GB with beneficiant headroom. It’s sturdy at structured reasoning and power use, and weaker on broad world information.
DeepSeek-R1-Distill-Qwen-32B — deepest reasoning, tightest match
DeepSeek distilled its R1 reasoning traces into smaller dense fashions. The DeepSeek-R1-Distill-Qwen-32B is the 32B variant, constructed on a Qwen2.5 base and launched beneath an MIT license. At Q4_K_M it makes use of about 18–20GB, which is the tightest match on this information. It exposes its chain of thought by seen reasoning tokens, which is beneficial for sluggish, deliberate issues. Prepared-made GGUF builds can be found from bartowski on Hugging Face.
What doesn’t match on 24GB
The frontier open fashions of 2026 are massive sparse MoE methods. They’re fairly good in efficiency and reasoning, however they don’t run on one client card. GLM-5.2 from Z.ai is a roughly 753B-total MoE. Moonshot’s Kimi K2.7 is round 1T complete. DeepSeek shipped V4 as a public preview in April 2026, with a V4-Professional checkpoint close to 1.6T parameters. Alibaba’s Qwen3.5-397B and Mistral Massive 3 sit in the identical server-class vary.
As a result of MoE reminiscence tracks complete parameters, all of those want multi-GPU rigs or high-memory unified methods. They’re value figuring out as API choices. They don’t change what runs on a single 24GB card.
Tips on how to run these fashions regionally
Three runtimes cowl virtually each setup. Ollama is the only path, with automated quantization choice and an OpenAI-compatible API. llama.cpp provides effective management over GGUF quantization and offload. vLLM is the throughput-focused server for heavier concurrent workloads.
Begin with one mannequin that matches your major job, not the largest file you possibly can load. Maintain context beneath management, and let the cardboard do what it does finest. Run severe native AI with out sending a single token to a cloud endpoint.
Key Takeaways
- 24GB is the sensible flooring: run right-sized 20B–35B fashions, not the largest 70B quant you possibly can squeeze in.
- Qwen3.6-27B is the strongest single default; DeepSeek-R1-Distill-Qwen-32B is the tightest match at ~18–20GB.
- MoE reminiscence tracks complete parameters, so each skilled stays resident even when only some route per token.
- Quantization is the match lever: Q4_K_M is the house default; Q8_0 and BF16 not often match a 30B mannequin on one card.
- GLM-5.2, Kimi K2.7, DeepSeek V4, and Mistral Massive 3 are server-class and won’t run on a single 24GB GPU.
Michal Sutter is a knowledge science skilled with a Grasp of Science in Knowledge Science from the College of Padova. With a strong basis in statistical evaluation, machine studying, and information engineering, Michal excels at remodeling advanced datasets into actionable insights.
