Local LLMs: Models, VRAM and Quantization

Local AI starts with the task

A language model running on your own machine can revise texts, explain code or analyze internal documents. You don't necessarily need a server room for that. The key question is: what quality, response time and context length does the specific workflow need to achieve?

If you only compare parameter counts, you can easily miss the real bottleneck. A large model that constantly has to shuffle data between system memory and the graphics card can be less useful day to day than a smaller model that fits entirely into video memory.

This guide groups selected models that can run locally by use case. The selection is not a ranking of the latest releases. The linked documentation reflects the state of September 2026. The memory figures are planning values for inference, meaning running a finished model. They are not our own hardware measurements, and they don't apply to training or fine-tuning.

Which models fit which use case?

For a chat assistant, the specific variant should be trained for instructions or dialogue. A base model from the same family is not automatically a good substitute. The right runtime and the chat format intended by the model provider matter just as much.

The model types and capabilities are described in the providers' documentation: Qwen3-8B, Qwen2.5-Coder-7B-Instruct, Qwen3-Coder on Ollama, Gemma 3 12B IT and Qwen3-Embedding-0.6B. The mapping to workflows is our own editorial assessment.

Writing and structured routines

For summaries, text variants or assigning a ticket to a category, a model in the 7B to 8B class is a sensible first test. B stands for billions of parameters. For very narrowly defined tasks, a smaller model may be enough.

Qwen3-8B can run with or without thinking. For short rewrites, the extra reasoning step is often unnecessary; for more demanding questions, a comparison is worthwhile. More generated thinking tokens mean additional waiting time, but no guarantee of a correct answer. The switches provided are described in the Qwen3 model card.

Coding: start with small tasks

A local coding assistant is best evaluated on limited tasks: explaining a function, adding a test case or narrowing down a bug based on a stack trace. Qwen2.5-Coder-7B-Instruct is a compact candidate for this from a code-specialized family. Model description from Qwen.

For larger tasks, Qwen3-Coder-30B-A3B-Instruct is an option. One distinction is crucial here: roughly 30 billion parameters exist in total, while about 3.3 billion are activated per token. This mixture-of-experts architecture reduces the active compute. It does not, however, turn the model into a 3B model in terms of memory requirements. The local 30B version is listed on Ollama with a file size of around 19 GB. Qwen3-Coder on Ollama.

For agentic workflows, you also need to check whether the model, chat format and application support the same tool calls. A good code suggestion on its own does not prove that the model can reliably work on an entire repository.

Document knowledge needs search

An internal knowledge base should not be copied into the prompt in full with every request. With retrieval-augmented generation, or RAG for short, the system first searches for relevant passages and passes them to the answer model.

Qwen3-Embedding-0.6B generates numerical representations of texts for this purpose. It is an embedding model, not a chat model, and does not write answers itself. Provider documentation.

In practice, a lot of groundwork determines quality here: can PDFs be extracted cleanly? Are headings and table relationships preserved? Does the search find the right passage? The answer should point to specific sources and make missing evidence visible. More parameters won't fix poorly prepared documents.

Images and screenshots

Gemma 3 12B IT accepts text and images and produces text. This lets you test, for example, whether an assistant can capture information from a screenshot or a product image. Gemma model card.

With invoices and scanned tables, however, what counts is exactly transcribed characters and numbers. A plausible-sounding result isn't enough. Image processing also requires additional memory, so the text-based estimates below cannot be applied unchanged to image batches.

How much VRAM does a local LLM need at minimum?

There is no universal VRAM minimum. CPU inference can work without dedicated video memory, as long as there is enough RAM. For interactive use, though, it often comes down to whether the model runs entirely on the GPU. CPU and GPU can be combined with llama.cpp, for example; speed and memory requirements then depend heavily on the hardware. llama.cpp documentation.

For the weights alone, a rough rule applies: parameter count × bits per parameter ÷ 8 = bytes. Eight billion parameters work out to 16 GB at 16 bits, 8 GB at 8 bits or 4 GB at 4 bits. These are decimal GB; 4 GB corresponds to roughly 3.73 GiB. This calculation leaves out quantization metadata, tensors stored at different precisions and the entire runtime overhead.

Weights, starting point and headroom

The table refers to typical dense text models with Q4_K-style weight quantization. It assumes a single request, a total context of roughly 4,096 tokens including output, a standard FP16 KV cache and full GPU execution. Model architecture and runtime can shift the values. The lower bound is a tight planning estimate, not a guaranteed minimum requirement.

These ranges are derived from weight memory plus an estimated runtime reserve. For Qwen3-8B in Q4, an 8 GB GPU is therefore a plausible starting point for short single requests. A 6 GB card can already get tight. For the roughly 19 GB Qwen3-Coder-30B version, 24 GB of VRAM is a tight setup with limited context, not a promise for the maximum advertised context window.

Before buying hardware, test the exact combination of model file, quantization, runtime, context and parallelism on comparable hardware. Your desktop display and other programs can also take up VRAM.

Why the model file isn't enough

On top of the weights come the KV cache for tokens already processed and working buffers for computation. Long contexts in particular can make the cache a major memory item. For models with full attention throughout, it grows roughly in proportion to the token count under the same configuration; sliding-window and hybrid architectures behave differently. Hugging Face: caching strategies.

Multiple concurrent requests also require additional memory. An advertised context window of 128K or 256K tokens therefore doesn't mean it's usable on your own GPU. In Ollama, ollama ps shows whether a loaded model sits entirely on the GPU or runs partly on the CPU. Ollama FAQ.

RAM and Apple unified memory

With a dedicated graphics card, system RAM is not an equivalent substitute for VRAM. Offloading can make larger models possible, but it can significantly worsen response times. For a workstation running a 7B to 8B model, 16 GB of system RAM is a tight starting point; 32 GB leaves more room for the application, documents and other programs. This is a planning recommendation, not a hard model minimum.

On Apple silicon systems, the CPU and GPU share unified memory. Of, say, 32 GB of total memory, not all of it is available to the model. The operating system, applications and runtime need space too. Supported GPU memory budgets and memory bandwidth also matter; capacity alone says little about tokens per second.

Quantization: what Q4, Q5 and Q8 mean

Quantization stores model weights at lower numerical precision. This reduces memory requirements. How much answer quality suffers and whether execution gets faster depends on the method, the model and the available hardware kernels. Methods are therefore not interchangeable based on their bit count alone. Hugging Face: quantization methods.

The following guide is a practical heuristic for common GGUF variants, not a measured quality ranking:

GGUF is the file format, not the quantization. A GGUF file can contain different precisions. With labels like Q4_K_M, not all weights are necessarily stored at exactly four bits. Block information and mixed tensor precisions explain why the file can be larger than the simple four-bit calculation suggests. The tools and options are described in llama.cpp on quantization.

For the decision, a direct comparison pays off: run the same model in Q4 and Q5 on the same tasks and measure errors, response time and memory. If memory is tight, also test a smaller model at higher precision. Which trade-off is better can't be derived from the parameter count across the board.

Weights and KV cache are separate levers

A Q4 model can still use an FP16 KV cache. Quantizing the weights does not automatically compress this cache. Under the documented conditions, Ollama also supports q8_0 and q4_0 for the KV cache. This can save memory but requires its own quality test, especially with long contexts. Ollama on KV cache quantization.

Where local models hit their limits

Well phrased doesn't mean correct

Locally run models also invent sources, mix up terms or produce faulty code. Quantization doesn't eliminate these problems. For structured output, schema and required fields should be validated technically; for coding, executable tests and review are what count.

For the selection, a fixed set of your own tasks is recommended: normal cases, incomplete inputs, long documents and examples where the correct answer is explicitly "cannot be derived from the sources." This exposes weaknesses that general benchmarks don't capture.

Local has to cover the whole workflow

A local model doesn't automatically mean that document search, embeddings and connected tools also run locally. If you want to keep data on your own hardware, you have to check the entire path. Ollama distinguishes between local and cloud-based models and offers a mode for disabling cloud features. Ollama FAQ on local operation.

In a team application, document permissions also belong in the search itself. An assistant must not output internal information just because it technically sits in the shared index.

Open weights don't mean uniform licenses

Downloadable models come with different terms of use. Qwen3-8B is labeled Apache 2.0, while Gemma 3 uses its own Gemma terms. When selecting a model, you should therefore document the specific model version and its license. Qwen3 license information, Gemma model card and terms of use.

A fast single chat is not yet a team service

As soon as several people work at the same time, queues, memory requirements and response times change. Purchase costs, power consumption, updates and maintenance therefore belong in the assessment. A free model download doesn't make operation free.

How to arrive at a reliable selection

For a digital team, a manageable pilot with a clear goal is worthwhile: classifying support requests, searching internal guides or drafting tests, for example. We also describe how such routines fit into everyday work in AI in the daily work of a digital team.

1. Define the task: Collect real examples and business success criteria, including difficult edge cases.
2. Start small: Choose a suitable instruction model with moderate quantization and limited context.
3. Measure operation: Document the exact model file, runtime version, quantization, context length, peak usage and response time.
4. Compare deliberately: Change one lever at a time, such as Q4 versus Q5 or a small versus a larger model.
5. Test the real workflow: Test with document search, multiple concurrent requests and the planned quality checks.

Which configuration is a good starting point?

For writing and smaller coding tasks, a 7B to 8B model in Q4 on a GPU with 8–12 GB of VRAM is a sensible first attempt. With 16 GB, there is room for models in the 12B to 14B class. At 24 GB, larger models become interesting, though still with deliberately limited context.

What matters is whether the configuration handles your own tasks reliably and fast enough. If you look at weight memory, context and parallelism separately and compare quantizations on real examples, you can use local AI far more effectively.