Guided Build · Local AI Infrastructure

Set Up Ollama

Run private, free, open-weight LLMs directly on your GPU with zero per-token cost, calibrated for 100% GPU execution.

Time: 15 min
Cost: $0 free & local
Stack: Ollama · Qwen3 · PowerShell
Hardware: NVIDIA/AMD GPU

1. Check Setup
2. Check VRAM
3. Calibrate Context
4. Test Model
5. Connect Agent

Step 1 of 5

Coach Qwen

First, verify whether Ollama is already installed and see what models you have on disk. Never delete existing base models.

Step 1 of 5 · Inventory

Inspect Existing Ollama Installation

Ollama runs as a background service on Windows and exposes both a command-line tool and an HTTP server on port 11434.

Open PowerShell and run these commands to check your installed version and current models:

ollama --version; ollama list

✓ Observable Checkpoint

Ollama prints its version (e.g. ollama version is 0.5.x or newer) and lists any models already on disk. If not installed, download from https://ollama.com/download/windows.

Troubleshooting: Command not found or service stopped

If ollama is not recognized, check if the installer added Ollama to your user PATH. If Ollama is installed but not responding, look in your system tray or run Start-Process "ollama app.exe".

Step 2 of 5 · Hardware Sizing

Inspect GPU VRAM and System RAM

Local LLM speed depends critically on fitting the active model entirely inside your GPU's dedicated VRAM. If even 20% spills into CPU RAM, token generation slows to a crawl.

Run nvidia-smi to check available VRAM and systeminfo for system RAM:

nvidia-smi; systeminfo | findstr /C:"Total Physical Memory"

✓ Observable Checkpoint

Note your total VRAM (e.g. 8192 MiB / 8 GB). On an 8 GB card, Windows DWM uses ~1.5 GB, leaving ~6.5 GB free for models and KV cache.

Troubleshooting: No NVIDIA GPU?

If you have an AMD or Intel GPU, run Get-CimInstance Win32_VideoController | Select-Object Name, AdapterRAM to view your graphics hardware. Ollama supports ROCm for modern AMD cards.

Step 3 of 5 · Tuning

Calibrate Context Size for 100% GPU Fit

Default models often allocate a huge context window (32k+ tokens) that swells KV cache memory by several gigabytes, spilling onto the CPU. Creating a custom Modelfile with num_ctx 8192 ensures 100% GPU residency.

Create a calibrated 8k context model from qwen3:8b:

"FROM qwen3:8b`nPARAMETER num_ctx 8192" | Set-Content Modelfile; ollama create qwen3:8b-8k -f Modelfile

✓ Observable Checkpoint

Ollama reports using existing layer... creating new layer... writing manifest... success. The new calibrated model is ready!

Troubleshooting: Base model not pulled yet

If qwen3:8b is not on your machine yet, run ollama pull qwen3:8b first. It downloads the ~5.2 GB base weights. Never delete existing models!

Step 4 of 5 · Verification

Test Inference and Verify 100% GPU Fit

Send a prompt to your calibrated model and inspect its real-time GPU/CPU distribution with ollama ps.

Run a test prompt and verify processor status:

ollama run qwen3:8b-8k "Explain GPU KV cache in two sentences."; ollama ps

✓ Observable Checkpoint

Check the PROCESSOR column in ollama ps. It must say 100% GPU. Total size should be ~6.2 GB with zero CPU offloading.

Troubleshooting: If processor shows CPU offload (e.g. 22%/78%)

Close other GPU-heavy apps (browsers with hardware acceleration, games) or lower num_ctx to 4096 in your Modelfile to fit in tighter VRAM budgets.

Step 5 of 5 · Integration

Connect Ollama to AI Coding Agents

Ollama provides an OpenAI-compatible endpoint at http://127.0.0.1:11434/v1. You can plug it directly into Cursor, Claude Code, Python scripts, or custom bots.

Test the OpenAI-compatible REST API using PowerShell:

Invoke-RestMethod -Uri "http://127.0.0.1:11434/v1/chat/completions" -Method Post -Headers @{"Content-Type"="application/json"} -Body '{"model":"qwen3:8b-8k","messages":[{"role":"user","content":"Say ready!"}]}' | Select-Object -ExpandProperty choices

✓ Observable Finish Line

You receive a structured response with role: assistant and content: ready!. Your local LLM pipeline is verified and ready for autonomous agent tasks!

PowerShell · Localhost:11434

PS C:\Users\learner> ollama --version
ollama version is 0.5.12

PS C:\Users\learner> ollama list
NAME           ID              SIZE      MODIFIED
qwen3:8b-8k    2c8b2db94ab4    6.2 GB    Just now
qwen3:8b       def456789abc    5.2 GB    1 day ago
qwen2.5:14b    7cdf5a0187d5    9.0 GB    5 weeks ago
qwen2.5:7b     845dbda0ea48    4.7 GB    5 weeks ago

Ready for Step 1.