How to Run Nemotron Locally

Yes, you can run Nemotron locally, and if you already own NVIDIA GPUs, it is one of the most hardware-optimized choices available. Nemotron models are trained and tuned specifically for CUDA and TensorRT-LLM, which means they extract more performance per GPU dollar on NVIDIA hardware than a generic open-weight model would.

Nemotron ships in two main tiers: Nemotron-2 (smaller variants in the 15B range) and Nemotron-3 (larger variants in the 70B range). Both are built on the Llama architecture with NVIDIA's additional training. The NVIDIA Open Model License permits commercial use.

This guide covers choosing between Nemotron-2 and Nemotron-3, the quantization options, how NVIDIA's own deployment tools compare to generic serving engines, and a step-by-step path to a working local deployment.

Can You Run Nemotron Locally?

Yes — Nemotron-2 runs on a single workstation-class GPU, and Nemotron-3 runs on a multi-GPU workstation or a small server. Both are realistic self-hosting targets for teams that already have NVIDIA hardware.

The key advantage is first-party tooling. NVIDIA provides TensorRT-LLM for optimized inference, NIM containers for one-command deployment, and integration with its own hardware profiling tools. No other model family gets this level of vendor-side deployment support on NVIDIA GPUs.

If you have NVIDIA GPUs and want the most optimized local inference, Nemotron is the model family to evaluate first. If you do not have NVIDIA hardware, other open-weight models with broader hardware support may be a better fit.

• Nemotron-2 (~15B) — runs on a single workstation GPU with 24+ GB VRAM.
• Nemotron-3 (~70B) — needs multi-GPU (2 to 4 GPUs) or a quantized single-GPU setup.
• First-party NVIDIA tooling — TensorRT-LLM, NIM containers, hardware profiling.

Picking the Right Nemotron Size for Your Hardware

Nemotron-2 and Nemotron-3 serve different hardware tiers. Pick the one that matches what you already own, then size up only if your task needs the larger model's capability.

Nemotron-2 (roughly 15B parameters) fits on a single NVIDIA GPU with 24 GB or more of VRAM — an RTX 4090, an A5000, or an L4. This is the right starting point for most teams, because it runs on hardware that a single workstation provides and still delivers strong performance on coding, reasoning, and instruction-following tasks.

Nemotron-3 (roughly 70B parameters) needs 2 to 4 GPUs at full precision, or a single high-memory GPU (A100 80 GB or H100) with quantization. This tier is for teams that need the strongest capability and already have server-class NVIDIA hardware or a cloud budget for it.

• Nemotron-2 (~15B) — single GPU, 24+ GB VRAM (RTX 4090, A5000, L4).
• Nemotron-3 (~70B) — 2 to 4 GPUs at full precision, or 1 high-memory GPU with quantization.
• Start with Nemotron-2 unless your task clearly needs the larger model.

Quantization: Running Nemotron on Less Hardware

Quantization shrinks Nemotron's memory footprint by storing weights at lower numeric precision. For Nemotron-3, this is often the difference between needing four GPUs and needing one or two.

NVIDIA's own TensorRT-LLM supports INT8 and FP8 quantization natively, with minimal quality loss on most tasks. This is a significant advantage over generic quantization tools: TensorRT-LLM calibrates quantization against NVIDIA hardware specifically, producing better speed-to-quality ratios than a generic GGUF conversion.

Community GGUF quantized builds also appear on Hugging Face for Nemotron models, which work with Ollama and llama.cpp. These are a good fit for quick prototyping, but for production serving on NVIDIA GPUs, TensorRT-LLM quantization is the better path.

• TensorRT-LLM — native INT8 and FP8 quantization, optimized for NVIDIA GPUs.
• GGUF (community) — works with Ollama and llama.cpp for quick prototyping.
• 4-bit quantized Nemotron-3 can fit on a single 48 GB GPU (A6000, L40).
• TensorRT-LLM calibration produces better quality-speed ratios than generic tools.

Ollama, vLLM, and SGLang: Which Tool?

For Nemotron, NVIDIA's own tools are the strongest option, but the generic ecosystem works too. The choice depends on whether you want maximum performance or maximum simplicity.

TensorRT-LLM with NIM containers is the first-party path. NVIDIA's build portal provides pre-built NIM container images for Nemotron that handle model loading, quantization, and serving in a single Docker container. This is the fastest path to production-grade inference on NVIDIA hardware.

Ollama supports Nemotron models and is the easiest way to try them locally. vLLM and SGLang also serve Nemotron well for production workloads, especially if your infrastructure already uses those tools for other models and you want a single serving stack.

• Maximum NVIDIA performance — TensorRT-LLM + NIM containers from build.nvidia.com.
• Quick local start — Ollama, one command to pull and run Nemotron.
• Existing infrastructure — vLLM or SGLang, if you already run other models on them.

A Practical Setup Outline

Getting Nemotron running locally follows two paths: the NVIDIA-native path (TensorRT-LLM and NIM) for production, or the Ollama path for quick prototyping. Both work, but the NVIDIA path extracts more performance from the same hardware.

Start with Nemotron-2 on Ollama to validate that the model fits your task. If it does, move to TensorRT-LLM for production serving. If Nemotron-2 falls short, evaluate Nemotron-3 before jumping to a different model family.

Keep a test harness of your own real prompts from day one. This tells you whether the model, the quantization level, and the serving config meet your quality and latency bar.

• Step 1 — Confirm you have NVIDIA GPUs with sufficient VRAM (24+ GB for Nemotron-2, 80+ GB across GPUs for Nemotron-3).
• Step 2 — Quick test: ollama pull nemotron and run it to validate the model fits your task.
• Step 3 — For production: pull the NIM container from build.nvidia.com and launch with Docker.
• Step 4 — Choose quantization (FP8 or INT8 via TensorRT-LLM) if you need to fit on fewer GPUs.
• Step 5 — The NIM container exposes an OpenAI-compatible endpoint. Point your app at it.
• Step 6 — Run your eval harness and monitor latency, throughput, and quality.

Licensing and When to Self-Host vs Use the API

Nemotron ships under the NVIDIA Open Model License, which permits commercial use and modification. The license is broadly permissive but is not MIT or Apache 2.0 — read the specific terms if you plan to redistribute a modified version.

Self-host Nemotron when you have NVIDIA GPUs, need data to stay on your hardware, or want the TensorRT-LLM performance advantage. The combination of first-party tooling and hardware optimization makes Nemotron one of the smoothest self-hosting experiences for teams already invested in NVIDIA infrastructure.

Use a hosted NVIDIA API (via build.nvidia.com or cloud partners) when you want to start fast without provisioning hardware, or when your usage is too low to justify dedicated GPUs. The hosted option uses the same NIM serving stack, so migration to self-hosted later is straightforward.

• NVIDIA Open Model License — commercial use and modification allowed; review redistribution terms.
• Self-host — data residency, TensorRT-LLM optimization, existing NVIDIA hardware.
• Hosted API (build.nvidia.com) — no hardware, same NIM stack, easy migration to self-hosted later.

What you need to run Nemotron yourself

Nemotron needs real memory, but it is within reach of a high-end workstation or a couple of professional GPUs — and many teams simply rent instead of buying. Match the path below to whether you want to own the hardware or pay by the hour.

To put Nemotron to work once it is live, connect a coding client like Cursor (via OpenRouter) or a local runner such as Ollama.

Apple Mac Studio (M4 Max, 128GB)

A single quiet on-prem box

View on Amazon →

NVIDIA RTX 6000 Ada (48GB)

Power users who want hardware they own

View on Amazon →