How to Run Mistral Locally

Yes, you can run Mistral locally — and thanks to the wide size range, from the 7B dense model to Mixtral 8x22B, there is a realistic option for almost any hardware budget.

Mistral AI is the French AI lab that pioneered the MoE-for-everyone approach. Mixtral 8x7B proved you could get near-frontier quality on a single high-end GPU via Mixture-of-Experts, and the family has only grown since. Most community models ship under the Apache 2.0 license, which gives businesses clear commercial self-hosting rights.

This guide covers choosing the right Mistral or Mixtral size for your hardware, quantization options, which serving tool fits (Ollama, vLLM, SGLang), a step-by-step setup path, and when to self-host versus use the API.

Can You Run Mistral Locally?

Yes — Mistral is one of the best-supported model families for local hosting, because the range of sizes means you can match the model to your actual hardware rather than renting a cluster.

Mistral 7B runs comfortably on a single consumer GPU with 8 GB or more of VRAM, and can even run on a modern laptop CPU for light use. Mixtral 8x7B (46.7B total, roughly 12.9B active per token) fits a single 24 GB GPU at 4-bit quantization. Mixtral 8x22B (176B total) needs a multi-GPU setup.

The practical question is not whether you can run Mistral, but which size fits your hardware and your task. Start with the smallest model that could plausibly handle your workload and size up only if it falls short.

• Mistral 7B — runs on a single 8 GB consumer GPU or a laptop CPU.
• Mixtral 8x7B — fits one 24 GB GPU at 4-bit quantization, near-frontier quality.
• Mixtral 8x22B — needs a multi-GPU server with 80+ GB combined VRAM.
• Mistral NeMo 12B — a strong mid-range option for one workstation GPU.

Picking the Right Mistral Size for Your Hardware

Match the model to the GPU memory you already have, then size up only if your task demands it. Running a model that barely fits in memory means painfully slow inference — a smaller model that loads comfortably almost always feels faster and works better in practice.

For 8 GB of VRAM (a typical gaming GPU), Mistral 7B at 4-bit quantization is the sweet spot. For 16 GB, Mistral NeMo 12B or a higher-precision Mistral 7B. For 24 GB (an RTX 4090 or similar), Mixtral 8x7B at 4-bit is the standout — it activates only about 12.9B parameters per token, so inference is fast despite the 46.7B total weight.

Beyond 24 GB you enter multi-GPU territory: Mixtral 8x22B at 4-bit needs roughly 80 to 100 GB of combined VRAM, spread across two or more datacenter GPUs. For most teams, Mixtral 8x7B is the practical ceiling before the cost curve gets steep.

• 8 GB VRAM — Mistral 7B (4-bit quantization).
• 16 GB VRAM — Mistral NeMo 12B or Mistral 7B at higher precision.
• 24 GB VRAM — Mixtral 8x7B (4-bit), the best quality-per-dollar sweet spot.
• 80–100+ GB combined — Mixtral 8x22B across multiple datacenter GPUs.

Quantization: Running Mistral on Less Hardware

Quantization stores model weights at lower numeric precision, shrinking the memory footprint so a given Mistral model fits a smaller GPU. This is the main lever for running Mixtral 8x7B on one card instead of two, or fitting Mistral 7B on an 8 GB laptop GPU.

Mistral models have excellent community quantization support. GGUF builds (for llama.cpp and Ollama) and AWQ/GPTQ builds (for vLLM) appear on Hugging Face within hours of every release. For Mixtral 8x7B, 4-bit GGUF is the most popular format and delivers strong quality with a roughly 4x memory reduction.

The trade-off is always quality on hard tasks. Test your own prompts at each quantization level and keep the lowest precision that still passes your bar. For most business text tasks, 4-bit Mixtral 8x7B is nearly indistinguishable from full precision.

• 4-bit quantization cuts memory use by roughly 4x, the most common choice.
• GGUF (Ollama/llama.cpp) and AWQ/GPTQ (vLLM) are the standard formats for Mistral.
• Community quantized builds appear on Hugging Face within hours of each release.
• Test your own prompts — 4-bit is usually fine for business text, but verify on your hardest tasks.

Ollama, vLLM, and SGLang: Which Tool?

For single-machine use, Ollama is the fastest way to run Mistral locally, because Mistral 7B and Mixtral 8x7B are first-class models in Ollama's library — one command pulls and serves them. LM Studio is an equally simple GUI-based alternative.

For production serving with concurrent users or multi-GPU setups, vLLM and SGLang are the right tools. They handle batched requests, multi-GPU memory management, and OpenAI-compatible API endpoints out of the box. Mixtral 8x22B specifically needs this class of tool.

A common pattern: develop against Ollama on your workstation, then point the same OpenAI-compatible client code at a vLLM endpoint when you deploy to a server. The client code stays identical, only the backend changes.

• Single machine, quick start — Ollama or LM Studio with built-in Mistral/Mixtral support.
• Power users — llama.cpp for fine-grained control over GGUF quantization.
• Production / multi-GPU — vLLM or SGLang, serving an OpenAI-compatible API.
• All tools serve OpenAI-compatible endpoints, so client code stays unchanged across them.

A Practical Setup Outline

Getting Mistral running locally is one of the fastest local-model setups available, especially for the smaller sizes. The whole path from zero to serving can take under 15 minutes for Mistral 7B on Ollama.

Start small, prove the workflow works, then scale. Most teams discover that Mixtral 8x7B at 4-bit is enough for their production needs and never need the 8x22B variant.

Keep a set of your own real prompts as an evaluation harness from day one. It tells you instantly whether a size upgrade, a quantization change, or a config tweak helped or hurt.

• Step 1 — Install Ollama (or your chosen tool) and pull Mistral 7B to validate your setup.
• Step 2 — Test against your own prompts. If 7B is sufficient, stop here.
• Step 3 — If you need more capability, pull Mixtral 8x7B (requires 24 GB VRAM at 4-bit).
• Step 4 — For multi-GPU or production use, switch to vLLM or SGLang and download weights from Mistral AI's Hugging Face org.
• Step 5 — Put an OpenAI-compatible gateway in front so your app code stays unchanged if you later swap models.
• Step 6 — Monitor latency, throughput, and quality with your evaluation harness.

Licensing and When to Self-Host vs Use the API

Mistral's community models — Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, and Mistral NeMo — ship under the Apache 2.0 license, one of the most permissive open-source licenses available. You can self-host, modify, fine-tune, and use them commercially with no restrictions.

Mistral's commercial models (Mistral Small, Medium, Large, and newer proprietary releases) are available only through Mistral's API. These are stronger on some benchmarks but are not open-weight, so they cannot be self-hosted. If you need the latest flagship quality and your data is not sensitive, the API is the right choice.

Self-host a community model when you need data residency, offline access, or your usage volume is high enough that the hardware cost beats ongoing API fees. The Apache 2.0 license makes the legal side straightforward.

• Apache 2.0 (community models) — permissive commercial rights, no restrictions on self-hosting or modification.
• Commercial models (Small/Medium/Large) — API-only, not self-hostable.
• Self-host — data residency, offline access, or high-volume cost savings.
• API — fastest start, access to the latest proprietary models, no hardware.

What you need to run Mistral yourself

How much hardware you need depends entirely on the model’s size. A small model runs on a laptop; a frontier model needs a server or the cloud. Here is the map by size class — pick the row that matches the model you have in mind.

Whichever size you land on, a one-click runner like Ollama or LM Studio gets small and mid models going in minutes; for a hosted endpoint, point Cursor at the model through OpenRouter. For specific hardware picks, see Best mini PCs for local AI and Local AI hardware calculator.

NVIDIA GeForce RTX 4090

Phi / Gemma / small Qwen-class

View on Amazon →

Apple Mac Studio (M4 Max, 128GB)

Llama-70B / Mixtral-class

View on Amazon →