Local inference setup
Run open models locally or on a GPU endpoint before committing to paid frontier APIs or reserved infrastructure.
Audience
Developers
Metric
Open models
Sources
3 official links
Start with a local baseline
Use Ollama, llama.cpp, or vLLM locally before renting GPUs. Measure quality, tokens per second, memory use, and prompt-cache behavior with your real prompts.
Use the benchmark to find the cheapest model that clears your quality bar before paying frontier API rates for steady traffic.
ollama pull qwen3:32b
ollama run qwen3:32bMove to a GPU server only after the model passes
For NVIDIA GPUs, serve an OpenAI-compatible endpoint with vLLM so application code can route routine calls without provider-specific rewrites.
- Use local hardware for the first quality benchmark when possible.
- Move to Modal, RunPod, Lambda, Vast.ai, Baseten, or another GPU host only after the model passes.
- Measure utilization before reserving or committing to a dedicated GPU.
python -m venv .venv
source .venv/bin/activate
pip install vllm
vllm serve Qwen/Qwen3-32B --host 0.0.0.0 --port 8000Price the full system
Track GPU hours, idle time, storage, egress, monitoring, retries, and the frontier-model fallback budget. Cheap local tokens still lose if utilization is low or operations work grows.
curl http://localhost:8000/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"Qwen/Qwen3-32B","messages":[{"role":"user","content":"Summarize this workload."}]}'Routing rule
Keep local or GPU-hosted open models on extraction, classification, summaries, test generation, and first-pass drafts. Escalate to Codex, Claude, OpenAI, Gemini, or another frontier lane only for planning, architecture, difficult bug fixing, research synthesis, reviews, and high-risk reasoning.