GPU Memory Required Calculator

"Can I run this model on my GPU?" is the question. The answer is more complicated than just comparing the model size to VRAM. Parameter count is the start. Precision, KV cache, batch size, context length, and overhead all add real memory requirements. A "70 billion parameter" model needs 140 GB at full precision but only 35 GB at 4-bit quantization. Knowing the math saves you from buying the wrong hardware or hitting out-of-memory errors at the worst moment.

The basic VRAM formula

Base memory needed equals (parameters × bytes per parameter). At FP32 (full precision), each parameter is 4 bytes. At FP16 (the most common modern default for inference), each parameter is 2 bytes. With quantization to INT8 or INT4, each parameter takes less.

That table gives you the base. Real inference needs 20-30% more for activation memory and KV cache. So a 70B model at INT4 needs about 45 GB VRAM in practice, not the 35 GB the simple math suggests.

KV cache: the hidden memory cost

Beyond storing the model itself, transformer inference needs memory for the KV (key-value) cache. This cache stores intermediate activations for each token in the context, and it grows linearly with context length.

The KV cache size formula: 2 × layers × heads × head_dim × bytes_per_value × context_length. For a 70B model with typical architecture (80 layers, 8 KV heads, 128 head_dim) running at FP16:

At long context, the KV cache often exceeds the model itself in size. This is why running long-context models is much harder than running standard-context versions. Techniques like grouped-query attention and KV cache compression are how modern long-context models stay practical.

Inference vs training memory

Training a model requires 4-8× more memory than inference at the same precision because of optimizer state and gradient buffers. This is why training large models is so much harder than running them.

Common GPUs and what they can run

What each GPU realistically runs

For frontier model sizes (200B+), you need multi-GPU setups with tensor parallelism. The infrastructure for these costs hundreds of thousands of dollars upfront. This is why most production systems use API access to providers rather than self-hosting.

Quantization: the great equalizer

Quantization is the technique that lets you run frontier models on more modest hardware. The principle: most of the parameters in a trained model can be represented with fewer bits than the original 16 or 32 without significant accuracy loss. INT8 quantization (8-bit integers instead of 16-bit floats) halves memory with minimal degradation. INT4 quantization quarters memory with small noticeable degradation.

Modern quantization techniques (GPTQ, AWQ, GGUF) preserve quality remarkably well. Llama 3.1 70B quantized to 4-bit (about 45 GB VRAM total) performs within 2-5% of the full-precision version on most benchmarks, while running on a single RTX 4090 or H100.

Measuring actual VRAM usage

Theoretical VRAM math is a starting point. Real measurements often surprise. Modern inference frameworks add overhead, and the same model can use 10-30% more VRAM than the formula suggests depending on the framework, batch size, and runtime configuration.

For accurate profiling on Linux, use nvidia-smi while the model is loaded and serving requests. Watch the "Memory-Usage" column. The first request after model load typically shows peak memory — sustained usage is often a bit lower.

For PyTorch-based inference, the torch.cuda.max_memory_allocated() function returns the peak memory used during a session. Useful for profiling before deployment.

Key insight: VRAM usage isn't constant. It spikes during model loading, fluctuates with batch sizes, and grows with longer contexts. Always test with realistic production workloads, not just a single short request.

Multi-GPU sharding strategies

When a model is too large for a single GPU, you split it across multiple GPUs. Three common strategies, each with different tradeoffs:

The catch with multi-GPU: inter-GPU communication adds latency. NVLink-connected GPUs (300+ GB/s bandwidth) work well together. GPUs connected only via PCIe (32 GB/s) become bottlenecked. For multi-GPU inference, the interconnect is often as important as the GPUs themselves.

CPU offloading: the bandwidth tradeoff

Modern inference frameworks (vLLM, llama.cpp, exllamav2) support CPU offloading — keeping parts of the model in system RAM and moving them to GPU as needed. This lets you run larger models than would fit in pure VRAM, at the cost of significantly slower inference.

The penalty is severe. Inference with CPU offloading typically runs at 5-20% the speed of full-VRAM inference. A model that generates 30 tokens/sec entirely in VRAM might generate 2-5 tokens/sec with significant offloading. Acceptable for personal use or batch processing; rarely viable for production user-facing applications.

Apple Silicon: unified memory changes the math

Apple's M-series chips (M1/M2/M3/M4 Max, Ultra) use unified memory architecture — the same memory pool serves both CPU and GPU. This means a Mac Studio with 192 GB RAM can effectively use up to ~140 GB for model inference, larger than most discrete GPUs.

The catch: memory bandwidth on Apple Silicon (400-800 GB/s peak) is lower than datacenter GPUs (3,000+ GB/s for H100). Result: Apple Silicon runs larger models than a single consumer NVIDIA GPU can, but slower than a similar-sized model on a smaller-VRAM datacenter GPU. For local development or moderate-scale inference, this tradeoff often makes sense.

Future-proofing hardware purchases

Model sizes have grown roughly 2-3× per year since 2020. Hardware bought for "today's largest model" gets outpaced within 18 months. Planning rules of thumb for purchases meant to last 3+ years:

• Buy 2-3× the VRAM you currently need for inference
• Prioritize memory bandwidth alongside VRAM capacity
• Ensure NVLink or equivalent fast interconnect if multi-GPU
• Plan for quantization advances — what's INT4 today may be INT2 next year

Common mistakes

• Forgetting overhead. The base model size isn't the total VRAM you need. Always budget 20-30% extra for activations, KV cache, and CUDA overhead.
• Ignoring context length in memory math. A model that fits at 2K context may not fit at 32K context. Long-context applications need significantly more VRAM.
• Buying based on inference numbers for training. If you plan to fine-tune, multiply inference VRAM requirements by 4-6×.
• Assuming quantization is free. 4-bit quantization works well for most tasks but degrades quality measurably for some (complex reasoning, code generation). Always test on your specific use case.
• Building for current model sizes. Models grow 2-3× per year. Hardware bought today for "today's models" gets outpaced quickly.

Questions and answers

Can I split a model across multiple GPUs?

Yes — this is called tensor parallelism or pipeline parallelism. Modern inference frameworks (vLLM, TensorRT-LLM) handle this automatically. The catch: inter-GPU communication adds latency, and multi-GPU setups need NVLink or fast PCIe interconnects to perform well.

Why does quantization sometimes break models?

Certain model components (especially attention weights and embeddings) are more sensitive to quantization than others. Mixed-precision quantization keeps these sensitive parts at higher precision while quantizing the bulk of the model heavily — this is what modern techniques like GGUF Q4_K_M do.

Should I rent or buy GPUs?

Rent (cloud GPUs from Lambda, RunPod, vast.ai) below ~$10K/month of usage. The break-even for buying is roughly 12-18 months of constant utilization. Most workloads aren't constant — they have peaks and idle periods that favor renting.

Sources

• NVIDIA technical documentation for A100, H100, H200, B200 specifications
• Hugging Face Transformers: model loading and quantization documentation
• vLLM project: inference optimization techniques
• Dettmers et al.: "LLM.int8()" and AWQ quantization research papers

Related: AI Inference Cost · Vector Database Cost · RAG System Cost · Data Storage Converter