Ollama: Servicio e Inferencia Local de LLMs

Inferencia 100% Local

Todos los datos se quedan en tu máquina. Sin API keys, sin dependencia cloud, sin logging de datos. Crítico para salud, legal, finanzas y compliance (HIPAA, GDPR, SOC 2).

Cero Costo por Token

Después de la inversión en hardware, la inferencia es gratis. Una RTX 4090 corriendo Qwen 3 8B procesa ~80 tok/s indefinidamente. A tarifas cloud, eso costaría $2,000-5,000/mes.

API Compatible con OpenAI y Anthropic

Reemplazo directo para endpoints OpenAI. Cambia base_url a http://localhost:11434/v1 y tu código existente funciona. Desde enero de 2026, Ollama también es compatible con la Anthropic Messages API, por lo que herramientas como Claude Code pueden usar modelos abiertos.

Amplio Soporte de Modelos

Acceso a 200+ modelos: Llama 4, Qwen 3, Gemma 3, DeepSeek V3, Phi-4, Mistral, CodeLlama y más. Modelos nuevos disponibles en días desde su lanzamiento.

Linux (Recomendado)

Instalación de una línea con detección automática de GPU. Crea servicio systemd. Soporta NVIDIA (CUDA 11.7+) y AMD (ROCm 6.0+).

# Install Ollama (auto-detects GPU)
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Start the server (auto-starts via systemd)
ollama serve

# Pull and run a model
ollama pull llama3.2
ollama run llama3.2 "Explain GGUF quantization"

macOS (Apple Silicon)

Descarga la app o usa Homebrew. Metal GPU funciona automáticamente en M1/M2/M3/M4. Memoria unificada = toda la RAM disponible como VRAM.

# Install via Homebrew
brew install ollama

# Or download from https://ollama.com/download

# Start the server
ollama serve

# Apple Silicon: Metal GPU is auto-detected
# 32GB M2 Max can run 30B models comfortably
ollama run qwen3:32b

Windows

Descarga el instalador de ollama.com. CUDA funciona con drivers NVIDIA 452.39+. También corre en WSL2 con GPU passthrough.

# Download installer from https://ollama.com/download
# Or install via winget:
winget install Ollama.Ollama

# WSL2 alternative (recommended for dev):
# Install in WSL2 Ubuntu with GPU passthrough
curl -fsSL https://ollama.com/install.sh | sh

# Verify GPU detection
ollama run llama3.2 "Hello from Windows"

Docker (Producción)

Imagen Docker oficial con passthrough de GPU NVIDIA. Requiere NVIDIA Container Toolkit. Ideal para servidores y Kubernetes.

# CPU-only
docker run -d -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama \
  ollama/ollama

# NVIDIA GPU (requires nvidia-container-toolkit)
docker run -d --gpus=all \
  -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama \
  ollama/ollama

# AMD GPU (ROCm)
docker run -d --device /dev/kfd --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 --name ollama \
  ollama/ollama:rocm

# Pull a model inside the container
docker exec ollama ollama pull llama3.2

Requisitos de GPU

La VRAM determina el tamaño máximo de modelo. ~0.5-1GB VRAM por billón de parámetros en Q4. CPU funciona pero es 10-50x más lento.

# VRAM requirements (Q4_K_M quantization):
#   7B model  ~  4.5 GB VRAM
#  13B model  ~  8.0 GB VRAM
#  30B model  ~ 18.0 GB VRAM
#  70B model  ~ 40.0 GB VRAM
# 110B model  ~ 64.0 GB VRAM

# Recommended GPUs:
# Budget:  RTX 3060 12GB (7-13B models)
# Mid:     RTX 4070 Ti Super 16GB (13-30B)
# Pro:     RTX 4090 24GB (30B models)
# Server:  A100 80GB / H100 (70B+ models)
# Mac:     M2/M3/M4 Pro/Max (unified memory)

Comandos Core

Comandos CLI esenciales: pull descarga, list muestra instalados, show muestra metadata, rm elimina modelos.

# Pull a model (downloads GGUF weights)
ollama pull llama3.2
ollama pull qwen3:8b
ollama pull gemma3:12b-it-q4_K_M

# List installed models
ollama list
# NAME              ID         SIZE    MODIFIED
# llama3.2:latest   a80c4f17   2.0 GB  2 hours ago
# qwen3:8b          ...        4.9 GB  1 day ago

# Show model details
ollama show llama3.2
# architecture: llama
# parameters:   3.2B
# quantization: Q4_K_M
# context:      131072

# Copy/rename a model
ollama cp llama3.2 my-assistant

# Remove a model
ollama rm llama3.2

# List running models
ollama ps
# NAME         ID        SIZE     PROCESSOR  UNTIL
# qwen3:8b     ...       5.4 GB   100% GPU   4 minutes

Modelfiles Personalizados

Definen configuraciones con modelos base, system prompts y parámetros. Como Dockerfiles para LLMs — reproducibles, versionados, compartibles.

# Modelfile for a code review assistant
FROM qwen3:8b

SYSTEM """You are an expert code reviewer.
Focus on: security vulnerabilities, performance
issues, error handling gaps, and maintainability.
Be concise. Use bullet points. Cite line numbers."""

PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER num_ctx 8192
PARAMETER stop "<|endoftext|>"

# Build the custom model
# ollama create code-reviewer -f Modelfile

# Run it
# ollama run code-reviewer "Review this function..."

Gestión de Almacenamiento

Los modelos consumen espacio significativo. Monitorea uso, configura directorios personalizados y limpia modelos sin usar.

# Check total model storage
du -sh ~/.ollama/models
# 28G   /home/user/.ollama/models

# Change model storage location
export OLLAMA_MODELS=/data/ollama/models

# Or in systemd override:
# sudo systemctl edit ollama
# [Service]
# Environment="OLLAMA_MODELS=/data/models"

# List models sorted by size
ollama list | sort -k3 -h

# Pull specific quantization
ollama pull llama3.2:3b-instruct-q8_0
ollama pull llama3.2:3b-instruct-q4_K_M

Importando Modelos GGUF

Importa cualquier modelo GGUF desde Hugging Face o archivos locales. Acceso a miles de modelos cuantizados por la comunidad.

# Import a GGUF from Hugging Face
# 1. Download the GGUF file
wget https://huggingface.co/bartowski/\
Qwen2.5-7B-Instruct-GGUF/resolve/main/\
Qwen2.5-7B-Instruct-Q5_K_M.gguf

# 2. Create a Modelfile pointing to the file
cat > Modelfile <<EOF
FROM ./Qwen2.5-7B-Instruct-Q5_K_M.gguf

TEMPLATE """{{ if .System }}<|im_start|>system
{{ .System }}<|im_end|>{{ end }}
<|im_start|>user
{{ .Prompt }}<|im_end|>
<|im_start|>assistant
"""

PARAMETER stop "<|im_end|>"
PARAMETER num_ctx 4096
EOF

# 3. Create and run the model
ollama create my-qwen -f Modelfile
ollama run my-qwen

Endpoints API Nativos

Generate, chat, embeddings y gestión de modelos. Streaming habilitado por defecto.

# Generate (single-turn)
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Explain Docker networking",
  "stream": false
}'

# Chat (multi-turn with history)
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [
    {"role": "system", "content": "You are a DevOps expert."},
    {"role": "user", "content": "How do I set up Traefik?"}
  ],
  "stream": false
}'

# Streaming (default, returns NDJSON)
curl http://localhost:11434/api/chat -d '{
  "model": "qwen3:8b",
  "messages": [
    {"role": "user", "content": "Write a haiku"}
  ]
}'

Endpoint Compatible con OpenAI

Reemplazo directo para la API de OpenAI. Cambia la URL base y tu código funciona.

# OpenAI-compatible chat completions
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" -d '{
  "model": "qwen3:8b",
  "messages": [
    {"role": "user", "content": "Hello!"}
  ]
}'

# With streaming (SSE format, like OpenAI)
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" -d '{
  "model": "qwen3:8b",
  "messages": [
    {"role": "user", "content": "Explain RAFT"}
  ],
  "stream": true
}'

# List available models
curl http://localhost:11434/v1/models

# Embeddings
curl http://localhost:11434/v1/embeddings -d '{
  "model": "nomic-embed-text",
  "input": "Ollama is great for local inference"
}'

Cliente Python (OpenAI SDK)

Usa el SDK de OpenAI con Ollama. Cero cambios de código excepto la URL base.

from openai import OpenAI

# Point to local Ollama server
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # required but unused
)

# Chat completion (identical to OpenAI API)
response = client.chat.completions.create(
    model="qwen3:8b",
    messages=[
        {"role": "system", "content": "You are helpful."},
        {"role": "user", "content": "Explain GGUF format"}
    ],
    temperature=0.7,
    max_tokens=1024
)
print(response.choices[0].message.content)

# Streaming
stream = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Hello"}],
    stream=True
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

SDK Oficial Python de Ollama

Interfaz más rica con gestión de modelos, salidas estructuradas y soporte multimodal integrado.

import ollama

# Chat with streaming
stream = ollama.chat(
    model="qwen3:8b",
    messages=[
        {"role": "user", "content": "Explain RAG"}
    ],
    stream=True
)
for chunk in stream:
    print(chunk["message"]["content"], end="")

# List local models
models = ollama.list()
for m in models["models"]:
    print(f"{m['name']}: {m['size'] / 1e9:.1f}GB")

# Pull a model programmatically
ollama.pull("gemma3:4b")

# Generate embeddings
emb = ollama.embed(
    model="nomic-embed-text",
    input="Local LLM inference with Ollama"
)
print(f"Dim: {len(emb['embeddings'][0])}")

# Structured output (JSON mode)
resp = ollama.chat(
    model="qwen3:8b",
    messages=[{"role": "user",
      "content": "List 3 Python frameworks as JSON"}],
    format="json"
)

Llama 4 (Meta)

Scout (17B activos / 109B total, MoE) y Maverick (17B activos / 400B total, MoE). Contexto de 10M+ tokens. Mejor modelo abierto para multilingual.

ollama pull llama4:scout     # 109B MoE, ~26GB
ollama pull llama4:maverick  # 400B MoE, ~95GB
ollama run llama4:scout "Summarize this codebase"

Qwen 3 (Alibaba)

0.6B a 32B más MoE 30B-A3B. Modo pensamiento integrado. Excelente coding, matemáticas y multilingual. Qwen3.6 (27B y 35B, contexto 256K) es la generación más reciente, con un gran salto en coding agéntico.

ollama pull qwen3:8b     # Best value (4.9GB)
ollama pull qwen3:32b    # Near-frontier quality
ollama pull qwen3.6:27b  # Newest gen, 256K ctx (17GB)
ollama run qwen3:8b "Write a FastAPI endpoint"

Gemma 4 (Google)

Abril 2026, Apache 2.0. Variantes compactas E2B y E4B más 12B, 26B MoE y 31B densa, multimodal y contexto 256K. Sobresale en seguimiento de instrucciones y eficiencia. Gemma 3 (1B/4B/12B/27B) sigue en la librería si la necesitas.

ollama pull gemma4:e4b   # Great for constrained envs
ollama pull gemma4:12b   # Strong general-purpose
ollama pull gemma4:31b   # Highest quality
ollama run gemma4:12b "Explain microservices"

DeepSeek V3 / R1

V3 (685B MoE, 37B activos) para tareas generales. R1 para razonamiento avanzado. Variantes destiladas (1.5B-70B).

ollama pull deepseek-r1:8b     # Reasoning, 4.9GB
ollama pull deepseek-r1:32b    # Strong reasoning
ollama pull deepseek-v3:latest # Full 685B MoE
ollama run deepseek-r1:8b "Prove sqrt(2) is irrational"

Phi-4 (Microsoft)

Phi-4 Mini (3.8B) rivaliza con modelos mucho más grandes. Phi-4 (14B) compite con modelos 3-5x su tamaño.

ollama pull phi4-mini       # 3.8B, 2.2GB
ollama pull phi4:14b        # 14B, 8.5GB
ollama run phi4-mini "Optimize this SQL query"

Modelos de Código

CodeLlama, Qwen 3 Coder, DeepSeek Coder V3, StarCoder2 para generación, completado y revisión de código.

ollama pull codellama:13b          # Meta, code-focused
ollama pull deepseek-coder-v2:16b  # Strong code + math
ollama pull starcoder2:7b          # Fast code completion
ollama run codellama:13b "Write a Redis cache layer"

Mistral / Mixtral

Mistral Small 3.1 (24B) con visión y 128K contexto. Mixtral 8x7B y 8x22B MoE para alto throughput.

ollama pull mistral-small3.1:24b  # 24B, vision
ollama pull mixtral:8x7b          # MoE, fast
ollama pull mistral:7b             # Classic, efficient
ollama run mistral-small3.1:24b "Analyze this diagram"

Formato GGUF

Estándar para inferencia CPU/GPU con llama.cpp. Almacena pesos, tokenizer y metadata en un archivo. Reemplazó GGML.

# GGUF files contain everything:
# - Model architecture metadata
# - Quantized weight tensors
# - Tokenizer vocabulary and merges
# - Chat template
# - Recommended parameters

# Pull specific quantization
ollama pull qwen3:8b-q4_K_M    # 4-bit, medium
ollama pull qwen3:8b-q5_K_M    # 5-bit, higher quality
ollama pull qwen3:8b-q8_0      # 8-bit, near-lossless

# Import your own GGUF
cat > Modelfile <<EOF
FROM ./my-model.Q4_K_M.gguf
EOF
ollama create my-model -f Modelfile

Niveles de Cuantización

Cada nivel intercambia calidad por tamaño/velocidad. K-quants usan precisión mixta inteligente.

# Quantization comparison for 8B model:
#
# Level     Size   Quality  Speed   Use Case
# -------   -----  -------  ------  --------
# FP16      16 GB  100%     1.0x    Reference
# Q8_0       8 GB   99%     1.5x    Quality-first
# Q6_K       6 GB   98%     1.7x    High quality
# Q5_K_M     5 GB   97%     1.8x    Recommended
# Q4_K_M     5 GB   95%     2.0x    Best balance
# Q4_K_S     4 GB   93%     2.1x    Smaller Q4
# Q3_K_M     4 GB   88%     2.2x    Low VRAM
# Q2_K       3 GB   75%     2.4x    Extreme savings
#
# Rule of thumb:
# - Q4_K_M: default for most users
# - Q5_K_M: when quality matters more
# - Q8_0: when VRAM is not a concern

Cuantizando Tus Modelos

Convierte safetensors de HuggingFace a GGUF con herramientas de llama.cpp.

# Install llama.cpp quantization tools
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp && make -j$(nproc)

# Convert safetensors to GGUF (FP16)
python convert_hf_to_gguf.py \
  /path/to/hf-model/ \
  --outfile model-fp16.gguf \
  --outtype f16

# Quantize to Q4_K_M
./llama-quantize \
  model-fp16.gguf \
  model-Q4_K_M.gguf \
  Q4_K_M

# Import into Ollama
echo "FROM ./model-Q4_K_M.gguf" > Modelfile
ollama create my-model -f Modelfile

Cuantización con Matriz de Importancia

Usa datos de calibración para preservar calidad en capas críticas. Mejores resultados en Q3/Q2.

# Generate importance matrix
./llama-imatrix \
  -m model-fp16.gguf \
  -f calibration-data.txt \
  -o imatrix.dat \
  --chunks 100

# Quantize with importance matrix
./llama-quantize \
  --imatrix imatrix.dat \
  model-fp16.gguf \
  model-IQ4_XS.gguf \
  IQ4_XS

# IQ quantization types (importance-based):
# IQ4_XS: smaller than Q4_K_M, similar quality
# IQ3_XXS: very small, usable quality
# IQ2_XXS: extremely small, for testing

CUDA (NVIDIA)

Backend con mejor soporte. Driver 450+ y CUDA 11.7+. GeForce RTX 3000+, Quadro, Tesla, A100, H100, L40S.

# Verify NVIDIA GPU detection
nvidia-smi
ollama run llama3.2 "Test GPU"

# Check GPU utilization during inference
watch -n 0.5 nvidia-smi

# Control GPU usage
export CUDA_VISIBLE_DEVICES=0      # GPU 0 only
export CUDA_VISIBLE_DEVICES=0,1    # GPU 0 and 1

# Force number of GPU layers (0 = CPU only)
OLLAMA_NUM_GPU=99 ollama serve

# Docker with NVIDIA GPU
docker run --gpus all \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama

ROCm (AMD)

Solo Linux. ROCm 6.0+. RX 7900 XTX (24GB), MI250X, MI300X. Rendimiento competitivo con CUDA.

# Install ROCm (Ubuntu 22.04+)
# Follow: https://rocm.docs.amd.com/

# Verify AMD GPU detection
rocm-smi
ollama run llama3.2 "Test AMD GPU"

# Docker with AMD GPU
docker run -d \
  --device /dev/kfd \
  --device /dev/dri \
  -v ollama:/root/.ollama \
  -p 11434:11434 \
  ollama/ollama:rocm

# Override GPU target (unsupported GPUs)
HSA_OVERRIDE_GFX_VERSION=11.0.0 ollama serve

# Check VRAM usage
rocm-smi --showmeminfo vram

Metal (Apple Silicon)

Cero configuración en M1/M2/M3/M4. Memoria unificada = toda la RAM disponible. Excelente rendimiento por watt.

# Metal is auto-detected on Apple Silicon
ollama run qwen3:32b  # Uses Metal automatically

# Apple Silicon memory guide:
# M2/M3/M4 (8GB):    7B Q4 models
# M2/M3/M4 (16GB):   13B Q4 or 7B Q8
# M2/M3/M4 Pro (36GB): 30B Q4 models
# M2/M3/M4 Max (64GB): 70B Q4 models
# M2/M3/M4 Max (128GB): 70B Q8 or 110B Q4

# Monitor memory pressure
# Activity Monitor > Memory tab

# Limit concurrent loaded models
launchctl setenv OLLAMA_MAX_LOADED_MODELS 1

Multi-GPU y Gestión de VRAM

Auto-distribuye capas en múltiples GPUs. Configura límites de VRAM, concurrencia y distribución de capas.

# Multi-GPU: auto-distributes layers
# 2x RTX 4090 (48GB total) = 70B Q4 models

# Control which GPUs to use
export CUDA_VISIBLE_DEVICES=0,1

# Limit concurrent models in memory
export OLLAMA_MAX_LOADED_MODELS=2

# Set model idle timeout (default: 5m)
export OLLAMA_KEEP_ALIVE=30m

# Force CPU-only mode (for testing)
export OLLAMA_NUM_GPU=0
ollama serve

# Monitor multi-GPU utilization
watch -n 1 nvidia-smi

# Per-request GPU control via API
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Hello",
  "options": {"num_gpu": 99}
}'

Referencia de Modelfile

Sintaxis completa con todas las instrucciones soportadas.

# Complete Modelfile reference
FROM qwen3:8b                # Base model or GGUF

SYSTEM """You are a senior backend engineer.
You specialize in Python, Go, and Kubernetes.
Always provide production-ready code with error
handling, logging, and type hints."""

PARAMETER temperature 0.4
PARAMETER top_p 0.9
PARAMETER top_k 40
PARAMETER repeat_penalty 1.1
PARAMETER num_ctx 16384
PARAMETER num_predict 2048
PARAMETER stop "<|eot_id|>"
PARAMETER seed 42

TEMPLATE """{{ if .System }}<|start|>system
{{ .System }}<|end|>
{{ end }}{{ if .Prompt }}<|start|>user
{{ .Prompt }}<|end|>
{{ end }}<|start|>assistant
{{ .Response }}<|end|>"""

LICENSE "Apache 2.0"

Adaptadores LoRA

Aplica adaptadores LoRA fine-tuneados. Fine-tuning específico de dominio sin modificar pesos completos. Adapter: 50-200MB.

# Modelfile with LoRA adapter
FROM llama3.2

# Apply a GGUF-format LoRA adapter
ADAPTER ./my-lora-adapter.gguf

SYSTEM """You are a medical coding assistant
specializing in ICD-10 and CPT codes."""

PARAMETER temperature 0.2
PARAMETER num_ctx 4096

# Build the model with adapter
# ollama create medical-coder -f Modelfile

# The adapter modifies base model behavior
# without full fine-tuning overhead:
# - Base model: 4.5GB
# - LoRA adapter: 50-200MB
# - Combined: domain-specific knowledge

Modelos Prácticos

Ejemplos reales: asistente SQL, escritor de commits, auditor de seguridad.

# SQL Assistant
cat > Modelfile.sql <<'EOF'
FROM qwen3:8b
SYSTEM """You are a SQL expert. Given a natural
language question and a database schema, generate
the optimal SQL query. Use CTEs for readability.
Always include column aliases."""
PARAMETER temperature 0.1
PARAMETER num_ctx 8192
EOF
ollama create sql-assistant -f Modelfile.sql

# Git Commit Message Writer
cat > Modelfile.commit <<'EOF'
FROM phi4-mini
SYSTEM """Write concise git commit messages.
Format: type(scope): description
Types: feat, fix, refactor, docs, test, chore
Keep under 72 characters. No period at end."""
PARAMETER temperature 0.3
PARAMETER num_predict 100
EOF
ollama create commit-writer -f Modelfile.commit

# Usage
git diff --staged | ollama run commit-writer

Compartir y Registro

Sube modelos al registro Ollama o exporta como archivos para distribución offline.

# Push to Ollama registry
ollama push username/my-model

# Pull someone else's custom model
ollama pull username/their-model

# Copy models between machines (offline)
# 1. Find model blob location
ollama show --modelfile qwen3:8b

# 2. Copy the model directory
# Source: ~/.ollama/models/
scp -r ~/.ollama/models/ user@target:~/.ollama/

# Export as Modelfile for versioning
ollama show --modelfile my-model > Modelfile
# Commit Modelfile to Git for team sharing

Modelos de Embedding

Embeddings vectoriales para RAG, búsqueda semántica y clustering. Todo el procesamiento se queda local.

# Popular embedding models
ollama pull nomic-embed-text    # 137M, 768-dim
ollama pull mxbai-embed-large   # 335M, 1024-dim
ollama pull all-minilm           # 23M, 384-dim (fast)
ollama pull snowflake-arctic-embed2  # 568M

# Generate embeddings via API
curl http://localhost:11434/api/embed -d '{
  "model": "nomic-embed-text",
  "input": "Local LLM inference is the future"
}'

# Python: batch embeddings
import ollama

texts = [
    "Kubernetes pod scheduling",
    "Docker container networking",
    "Helm chart templating"
]
result = ollama.embed(
    model="nomic-embed-text",
    input=texts
)
print(f"Dims: {len(result['embeddings'][0])}")
# Dims: 768

Modelos de Visión

Analiza imágenes localmente. Diagramas, screenshots, documentos, fotos. Sin upload a la nube.

# Vision-capable models
ollama pull llava:13b              # LLaVA 1.6
ollama pull gemma3:12b             # Gemma 3 (vision)
ollama pull mistral-small3.1:24b   # Mistral (vision)
ollama pull llama4:scout           # Llama 4 (vision)

# API: send image as base64
import ollama, base64

with open("screenshot.png", "rb") as f:
    img_b64 = base64.b64encode(f.read()).decode()

response = ollama.chat(
    model="gemma3:12b",
    messages=[{
        "role": "user",
        "content": "What errors are in this log?",
        "images": [img_b64]
    }]
)
print(response["message"]["content"])

Pipeline RAG Local

Sistema RAG completo con Ollama: indexa documentos, almacena vectores, recupera contexto, genera respuestas.

import ollama
import numpy as np

# 1. Embed your documents
docs = [
    "Ollama runs LLMs locally using GGUF",
    "CUDA acceleration requires NVIDIA GPU",
    "Quantization reduces model size and VRAM",
    "LoRA adapters enable domain fine-tuning",
]
doc_embs = ollama.embed(
    model="nomic-embed-text", input=docs
)["embeddings"]

# 2. Embed the query
query = "How do I reduce VRAM usage?"
q_emb = ollama.embed(
    model="nomic-embed-text", input=query
)["embeddings"][0]

# 3. Find most similar documents
scores = [np.dot(q_emb, d) for d in doc_embs]
context = docs[np.argmax(scores)]

# 4. Generate answer with context
response = ollama.chat(model="qwen3:8b", messages=[
    {"role": "system",
     "content": f"Answer based on: {context}"},
    {"role": "user", "content": query}
])
print(response["message"]["content"])

RAG con ChromaDB

RAG de producción con ChromaDB y Ollama. Almacenamiento persistente y búsqueda eficiente.

import chromadb, ollama

client = chromadb.PersistentClient(path="./chroma")

class OllamaEmbed:
    def __call__(self, input):
        return ollama.embed(
            model="nomic-embed-text", input=input
        )["embeddings"]

collection = client.get_or_create_collection(
    "docs", embedding_function=OllamaEmbed()
)

# Add documents
collection.add(
    documents=["Doc content here..."],
    ids=["doc1"],
    metadatas=[{"source": "manual.pdf"}]
)

# Query with automatic embedding
results = collection.query(
    query_texts=["How to configure GPU?"],
    n_results=3
)

# Generate answer
context = "\n".join(results["documents"][0])
resp = ollama.chat(model="qwen3:8b", messages=[
    {"role": "system",
     "content": f"Context:\n{context}"},
    {"role": "user",
     "content": "How to configure GPU?"}
])

LangChain

Integración nativa para chains, agentes y pipelines RAG. ChatOllama para chat, OllamaEmbeddings para vectores.

from langchain_ollama import (
    ChatOllama, OllamaEmbeddings
)
from langchain_core.messages import (
    HumanMessage, SystemMessage
)

llm = ChatOllama(
    model="qwen3:8b",
    temperature=0.3,
    num_ctx=8192,
    base_url="http://localhost:11434"
)

messages = [
    SystemMessage(content="You are a DevOps expert."),
    HumanMessage(content="Explain K8s Ingress")
]
response = llm.invoke(messages)
print(response.content)

# Embeddings for RAG
embeddings = OllamaEmbeddings(
    model="nomic-embed-text"
)
vectors = embeddings.embed_documents([
    "Kubernetes pods run containers",
    "Helm manages Kubernetes packages"
])

# Streaming
for chunk in llm.stream(messages):
    print(chunk.content, end="")

LlamaIndex

Apps RAG con LlamaIndex usando Ollama como backend de LLM y embedding. Todo local.

from llama_index.llms.ollama import Ollama
from llama_index.embeddings.ollama import (
    OllamaEmbedding
)
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader,
    Settings
)

Settings.llm = Ollama(
    model="qwen3:8b",
    request_timeout=120,
    temperature=0.3
)
Settings.embed_model = OllamaEmbedding(
    model_name="nomic-embed-text"
)

# Load and index documents
documents = SimpleDirectoryReader(
    "./docs"
).load_data()
index = VectorStoreIndex.from_documents(documents)

# Query the index
query_engine = index.as_query_engine()
response = query_engine.query(
    "How do I configure GPU acceleration?"
)
print(response)

Open WebUI

Interfaz tipo ChatGPT auto-hosteada. Multi-usuario, historial, cambio de modelo, RAG, plugins. Docker deploy.

# Deploy Open WebUI with Docker
docker run -d --name open-webui \
  -p 3000:8080 \
  --add-host=host.docker.internal:host-gateway \
  -v open-webui:/app/backend/data \
  -e OLLAMA_BASE_URL=\
http://host.docker.internal:11434 \
  ghcr.io/open-webui/open-webui:main

# Features:
# - Multi-user with role-based access
# - Conversation history and search
# - Model switching mid-conversation
# - File upload and RAG
# - Web search integration
# - Custom system prompts per model
# - Plugin/function support
# - Mobile-responsive UI

# Access at http://localhost:3000

Continue.dev (Entorno de Desarrollo)

Asistente de código IA para VS Code y JetBrains con modelos locales. Tab completion, chat inline, refactoring privado.

// ~/.continue/config.json
{
  "models": [
    {
      "title": "Qwen 3 8B (Local)",
      "provider": "ollama",
      "model": "qwen3:8b",
      "apiBase": "http://localhost:11434"
    }
  ],
  "tabAutocompleteModel": {
    "title": "Qwen 3 4B (Fast)",
    "provider": "ollama",
    "model": "qwen3:4b"
  },
  "embeddingsProvider": {
    "provider": "ollama",
    "model": "nomic-embed-text"
  }
}

// Shortcuts:
// Ctrl+L: Chat with codebase context
// Tab: AI autocomplete
// Ctrl+I: Inline edit
// Highlight + Ctrl+L: Explain/refactor

Más Integraciones

n8n, Dify, CrewAI, Aider, AnythingLLM y más del ecosistema LLM open-source.

# n8n: AI workflow automation
# Add Ollama node in n8n workflows
# Base URL: http://ollama:11434

# Dify: AI application platform
# Settings > Model Providers > Ollama

# CrewAI: multi-agent orchestration
from crewai import Agent, Crew
from langchain_ollama import ChatOllama

llm = ChatOllama(model="qwen3:8b")
researcher = Agent(
    role="Researcher",
    goal="Find technical information",
    llm=llm
)

# Aider: AI pair programming
# pip install aider-chat
# aider --model ollama/qwen3:8b

# AnythingLLM: document chat
# Configure Ollama as LLM provider

Stack Docker Compose

Docker Compose con Ollama, Open WebUI, GPU passthrough, volúmenes persistentes y health checks.

# docker-compose.yml
services:
  ollama:
    image: ollama/ollama:latest
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - ollama_data:/root/.ollama
    environment:
      - OLLAMA_HOST=0.0.0.0
      - OLLAMA_KEEP_ALIVE=30m
      - OLLAMA_MAX_LOADED_MODELS=2
      - OLLAMA_NUM_PARALLEL=4
    healthcheck:
      test: ["CMD", "curl", "-f",
             "http://localhost:11434/"]
      interval: 30s
      timeout: 10s
      retries: 3
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - webui_data:/app/backend/data
    depends_on:
      ollama:
        condition: service_healthy
    restart: unless-stopped

volumes:
  ollama_data:
  webui_data:

Reverse Proxy y TLS

Nginx con TLS, rate limiting y soporte de streaming. Esencial para exposición segura.

# /etc/nginx/sites-available/ollama
upstream ollama {
    server 127.0.0.1:11434;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name llm.example.com;

    ssl_certificate /etc/letsencrypt/live/
        llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/
        llm.example.com/privkey.pem;

    limit_req_zone $binary_remote_addr
        zone=ollama:10m rate=10r/m;

    location / {
        limit_req zone=ollama burst=5;
        proxy_pass http://ollama;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;

        # SSE streaming support
        proxy_buffering off;
        proxy_cache off;
        chunked_transfer_encoding on;
    }
}

Monitoreo y Métricas

Prometheus, Grafana y health checks. GPU, tiempos de carga, latencia y throughput.

#!/usr/bin/env python3
"""Ollama health monitor with Prometheus."""
import time, requests, subprocess
from prometheus_client import (
    start_http_server, Gauge, Histogram
)

GPU_UTIL = Gauge("ollama_gpu_util", "GPU %")
GPU_MEM = Gauge("ollama_gpu_mem_gb", "VRAM GB")
MODELS = Gauge("ollama_loaded_models", "Count")

def collect():
    out = subprocess.check_output([
        "nvidia-smi",
        "--query-gpu=utilization.gpu,memory.used",
        "--format=csv,noheader,nounits"
    ]).decode()
    util, mem = out.strip().split(", ")
    GPU_UTIL.set(float(util))
    GPU_MEM.set(float(mem) / 1024)

    resp = requests.get(
        "http://localhost:11434/api/ps"
    ).json()
    MODELS.set(len(resp.get("models", [])))

start_http_server(9090)
while True:
    collect()
    time.sleep(15)

Autenticación y Control de Acceso

Sin auth integrada. Implementa API keys en el reverse proxy. Nunca expones Ollama directo a internet.

# Nginx API key authentication
location /v1/ {
    set $api_key "";
    if ($http_authorization ~* "Bearer (.+)") {
        set $api_key $1;
    }
    if ($api_key != "your-secret-key") {
        return 401 '{"error":"unauthorized"}';
    }
    proxy_pass http://ollama;
    proxy_buffering off;
}

# Bind to localhost only
OLLAMA_HOST=127.0.0.1:11434

# Firewall rules (UFW)
sudo ufw deny 11434
sudo ufw allow from 10.0.0.0/8 to any port 11434

# Docker: internal networking only
services:
  ollama:
    expose:          # NOT ports:
      - "11434"

Escalamiento y Balanceo

Múltiples instancias con load balancer. Cada instancia en GPU separada. Least-connections routing.

# Multiple instances on different GPUs
# Instance 1: GPU 0
CUDA_VISIBLE_DEVICES=0 \
OLLAMA_HOST=0.0.0.0:11434 ollama serve

# Instance 2: GPU 1
CUDA_VISIBLE_DEVICES=1 \
OLLAMA_HOST=0.0.0.0:11435 ollama serve

# Nginx load balancer
upstream ollama_cluster {
    least_conn;
    server 127.0.0.1:11434;
    server 127.0.0.1:11435;
    keepalive 32;
}

server {
    location / {
        proxy_pass http://ollama_cluster;
        proxy_buffering off;
    }
}

Contexto y KV Cache

Contexto impacta VRAM y velocidad directamente. num_ctx mínimo = inferencia más rápida.

# Override context per request:
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Hello",
  "options": { "num_ctx": 4096 }
}'

# KV cache VRAM usage (8B model, Q4_K_M):
#   2K ctx:  ~4.5 GB total
#   4K ctx:  ~5.0 GB total
#   8K ctx:  ~5.5 GB total
#  16K ctx:  ~6.5 GB total
#  32K ctx:  ~8.5 GB total
# 128K ctx: ~20.0 GB total

# Set default in Modelfile
# PARAMETER num_ctx 4096

# Or via environment variable
OLLAMA_NUM_CTX=4096 ollama serve

Requests Paralelos

Procesamiento concurrente con KV cache compartido. Slots paralelos para multi-usuario.

# Enable parallel request processing
export OLLAMA_NUM_PARALLEL=4  # 4 concurrent

# Each slot uses additional KV cache VRAM
# 4 parallel x 8K ctx = 4x KV cache VRAM

# Start server with parallel config
OLLAMA_NUM_PARALLEL=4 \
OLLAMA_MAX_LOADED_MODELS=2 \
  ollama serve

# Test concurrent requests
for i in $(seq 1 4); do
  curl -s http://localhost:11434/api/generate \
    -d '{"model":"qwen3:8b",
         "prompt":"Count to 10",
         "stream":false}' &
done
wait

# Monitor parallel utilization
ollama ps

Carga y Keep-Alive

Carga toma 2-10s. Keep-alive para evitar recarga. Pre-carga al inicio para respuestas instantáneas.

# Keep models in memory longer (default: 5m)
export OLLAMA_KEEP_ALIVE=30m

# Keep model loaded indefinitely
export OLLAMA_KEEP_ALIVE=-1

# Per-request keep_alive override
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "prompt": "Hello",
  "keep_alive": "1h"
}'

# Pre-load a model at startup
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "keep_alive": "24h"
}'

# Unload a model immediately
curl http://localhost:11434/api/generate -d '{
  "model": "qwen3:8b",
  "keep_alive": 0
}'

# Limit concurrent loaded models
export OLLAMA_MAX_LOADED_MODELS=2

Benchmarking y Profiling

Mide throughput, TTFT y latencia total con metadata de respuesta de la API.

#!/usr/bin/env python3
"""Benchmark Ollama model performance."""
import time, requests, statistics

def benchmark(model, prompt, n=5):
    results = []
    for i in range(n):
        start = time.time()
        resp = requests.post(
            "http://localhost:11434/api/generate",
            json={"model": model, "prompt": prompt,
                  "stream": False,
                  "options": {"num_ctx": 4096}}
        ).json()
        elapsed = time.time() - start

        eval_count = resp.get("eval_count", 0)
        eval_dur = resp.get("eval_duration", 1)
        tok_s = eval_count / (eval_dur / 1e9)
        ttft = resp.get(
            "prompt_eval_duration", 0) / 1e9

        results.append(tok_s)
        print(f"Run {i+1}: {tok_s:.1f} tok/s, "
              f"TTFT: {ttft:.2f}s")

    print(f"\nMedian: {statistics.median(results):.1f}")
    print(f"Mean:   {statistics.mean(results):.1f}")

benchmark("qwen3:8b", "Explain Docker networking")

Variables de Entorno

Referencia completa de variables para rendimiento, almacenamiento, networking y límites.

# Server configuration
OLLAMA_HOST=0.0.0.0:11434     # Bind address
OLLAMA_ORIGINS=*               # CORS origins
OLLAMA_MODELS=/data/models     # Model storage

# Performance tuning
OLLAMA_NUM_PARALLEL=4          # Concurrent requests
OLLAMA_MAX_LOADED_MODELS=2     # Models in VRAM
OLLAMA_KEEP_ALIVE=30m          # Model idle timeout
OLLAMA_NUM_GPU=99              # GPU layers (99=all)
OLLAMA_MAX_QUEUE=512           # Request queue size

# GPU control
CUDA_VISIBLE_DEVICES=0,1       # NVIDIA GPU select
HSA_OVERRIDE_GFX_VERSION=11.0.0  # AMD override
OLLAMA_FLASH_ATTENTION=1       # Flash attention
OLLAMA_KV_CACHE_TYPE=q8_0     # Quantized KV cache

# Debug
OLLAMA_DEBUG=1                 # Verbose logging
OLLAMA_LLM_LIBRARY=cpu        # Force CPU backend

# Apply via systemd override:
sudo systemctl edit ollama
# [Service]
# Environment="OLLAMA_NUM_PARALLEL=4"
# Environment="OLLAMA_KEEP_ALIVE=30m"
sudo systemctl restart ollama

Flash Attention y KV Cache Cuantizado

Flash attention para inferencia más rápida. KV cache cuantizado para más contexto en el mismo VRAM.

# Enable flash attention (experimental)
export OLLAMA_FLASH_ATTENTION=1

# KV cache quantization
export OLLAMA_KV_CACHE_TYPE=q8_0

# Impact on 8B model, 32K context:
# FP16 KV cache: ~8.5 GB VRAM
# Q8_0 KV cache: ~6.5 GB VRAM (-24%)
# Q4_0 KV cache: ~5.5 GB VRAM (-35%)

# Combined optimizations:
OLLAMA_FLASH_ATTENTION=1 \
OLLAMA_KV_CACHE_TYPE=q8_0 \
OLLAMA_NUM_PARALLEL=4 \
OLLAMA_KEEP_ALIVE=1h \
  ollama serve

# Verify settings in debug mode
OLLAMA_DEBUG=1 ollama serve 2>&1 | grep -i flash