Methodology

Model Size Estimation

The memory footprint of a model depends on its parameter count and quantization level (bits per weight). For each model, we store pre-computed VRAM values derived from actual GGUF file measurements, which account for format overhead, embedding tables, and architecture-specific tensor layouts. When measured values are unavailable, we fall back to:

Example: A 7B model at Q4_K_M (4.5 bpw) quantization: 7 × 4.5 / 8 × 1024 = 4,032 MB ≈ 3.9 GB. The actual GGUF file is typically 5–15% larger due to format and tensor overhead.

KV Cache

The KV (Key-Value) cache stores attention states for every token in the context window and grows linearly with context length. The theoretical formula for KV cache per token is:

The factor of 2 accounts for both Key and Value tensors. For models using Grouped-Query Attention (GQA), n_kv_heads is much smaller than the number of attention heads (typically 8 regardless of model size), which keeps the KV cache sub-linear with parameter count.

Since our dataset does not include per-model architecture details (layer count, head counts, head dimension), we use an empirical power-law approximation calibrated against measured KV cache sizes:

The base multiplier of 128 MB per 1K tokens corresponds to a 7B GQA model at FP16 (verified: 32 layers × 8 KV heads × 128 head_dim × 2 bytes × 2 tensors = 128 KB/token = 128 MB/1K). The 0.4 exponent reflects that layer count grows slower than parameter count: larger models widen their hidden dimensions rather than just stacking more layers, while GQA keeps KV heads fixed. This fits measured values to within 5% for 7B–70B models.

Mixture-of-Experts (MoE) Models

MoE models contain multiple “expert” sub-networks but only activate a fraction of them per token. This creates a fundamental distinction between VRAM requirements and inference speed:

Every MoE model in our database includes an explicit active_params_b value sourced from the model's official documentation or architecture paper. This ensures accurate speed estimates:

Example: DeepSeek R1 (671B total, 37B active) at Q4 on an H100 (3,350 GB/s): VRAM needed = 671 × 4.5 / 8 ≈ 377 GB (requires multi-GPU), but decode speed = 3350 / (37 × 4.5 / 8) ≈ 161 tok/s per shard (before multi-GPU overhead).

Inference Modes

We determine the best way to run each model based on available memory:

Token Generation Speed

Token generation (decode) is memory-bandwidth bound. Each decode step reads the active model weights plus the accumulated KV cache from GPU memory. The full theoretical formula is:

For short contexts (≤4K tokens), the KV cache is small relative to the model weights, so the simplified form tok/s ≈ bandwidth / active_model_size is a close approximation. For MoE models, only the active expert weights are read per token, not the full model.

Example: RTX 4090 (1,008 GB/s bandwidth) running a 70B Q4 dense model (~39 GB): 1008 / 39 ≈ ~26 tokens/second (at short context). At 32K context, the KV cache adds ~10 GB, reducing speed to 1008 / 49 ≈ ~21 tok/s.

GPU + RAM Offload Speed

When a model doesn't fully fit in VRAM, some layers run on GPU and the rest on CPU/RAM. These layers process sequentially, not in parallel: the total time per token is the sum of both parts, not an average:

CPU layers read weights directly from system RAM, so the bottleneck is RAM bandwidth (not PCIe). PCIe only carries the small activation vectors (~8–32 KB) between GPU and CPU layer groups, adding a small latency overhead (~0.1–0.2 ms per token). Faster PCIe generations (4.0, 5.0) reduce this latency.

RAM Bandwidth

For CPU inference and GPU+RAM offload modes, system RAM bandwidth is a critical factor. We estimate it from the DDR specification and apply a real-world utilization factor:

The 75% utilization factor accounts for memory controller overhead, refresh cycles, cache line alignment, NUMA effects, and contention with the OS and other processes. For example, DDR5-6400 in dual-channel configuration: 6400 × 8 × 2 / 1000 × 0.75 ≈ 77 GB/s effective.

Uncertainty Ranges

All performance estimates are shown as ranges rather than single numbers. Real-world performance varies due to inference runtime (Ollama, llama.cpp, vLLM), driver version, thermal throttling, background system load, and model-specific optimizations.

These bands reflect the inherent variance in real-world setups. For precise numbers, we recommend running your own benchmarks and submitting them to our community benchmarks page.

Prefill Speed & Time to First Token (TTFT)

Processing the input prompt (prefill) is compute-bound, not bandwidth-bound. The forward pass requires approximately 2 FLOPs per active parameter per token:

Utilization is typically 30% without tensor cores, 60% with tensor cores. For MoE models, the active parameter count is used since only the selected experts participate in each forward pass.

Time to First Token (TTFT) is derived from prefill speed:

The full theoretical TTFT also includes a memory-loading component (model_size / bandwidth), but since prefill is compute-bound for typical prompt lengths (100+ tokens), the memory component is overlapped with computation by GPU pipelining and does not add to total latency. For very short prompts (<30 tokens), memory loading may dominate; our estimates are optimistic in that regime.

Multi-GPU Scaling

Multiple GPUs can combine their VRAM and bandwidth via tensor parallelism, but with overhead from inter-GPU communication:

The bandwidth formula for PCIe multi-GPU is: effective_bw = base_bw × (1 + (gpu_count − 1) × 0.3). This conservative estimate reflects the PCIe synchronization overhead during tensor-parallel inference. Apple Silicon devices do not support multi-GPU configurations (Ultra variants achieve high memory capacity through a single unified memory pool).

CPU Inference

CPU inference speed is primarily limited by system RAM bandwidth. The base speed is calculated the same way as GPU inference (bandwidth divided by active model size), then adjusted by CPU-specific multipliers:

Intel AMX (Advanced Matrix Extensions) provides significant acceleration for matrix operations, making newer Intel CPUs notably faster for LLM inference. CPU inference speed is capped at 60 tok/s to reflect practical limits.

Hardware Recommendations

When recommending hardware for a specific model (“Find Hardware”), we evaluate every GPU in our database across single and multi-GPU configurations (1, 2, 4, and 8 GPUs). For each viable option:

• 1.VRAM check: The model's pre-computed VRAM value (from measured GGUF data) must fit within the GPU's total VRAM (with multi-GPU efficiency scaling applied).
• 2.Speed estimate: Decode tokens/sec is calculated from the GPU's memory bandwidth and the model's active parameter count (accounting for MoE architecture).
• 3.Tier classification: Options are categorized as Budget (<15 tok/s), Recommended (15–40 tok/s), or Premium (40+ tok/s) and filtered by user preferences (minimum speed, maximum budget).

The final recommendations include a minimum viable option (cheapest that can physically run the model), a best value option (highest tok/s per dollar), and a premium option (fastest within a reasonable price range). Cloud alternatives (RunPod, Vast.ai) are shown alongside local hardware for comparison. All GPU vendors are supported: NVIDIA, AMD, Intel Arc, and Apple Silicon.

Final Score Calculation

Each model's final score combines three factors with weights that vary by use case:

Benchmark scores are renormalized when some benchmarks are missing for a model, so models with partial coverage are not unfairly penalized. The best available quantization is selected per model: we prefer GPU Full inference over offload or CPU, and within the same inference mode, higher-quality quantization is preferred.

Weights by Use Case

Data Sources

• •Model benchmarks: Open LLM Leaderboard on HuggingFace
• •GPU specifications: Official NVIDIA, AMD, Apple, Intel spec sheets (206+ GPUs)
• •Model VRAM: Pre-computed from actual GGUF file measurements, with formula fallback based on llama.cpp memory estimation
• •MoE active parameters: Sourced from official model cards and architecture papers for 137 of 156 MoE models; remainder uses a 20% heuristic estimate
• •CPU specifications: Intel, AMD, Apple Silicon spec sheets (78 CPUs)

Limitations