Ollama Provider
The Ollama provider enables local LLM inference using the Ollama platform. Run models like Llama 3, Mistral, and Gemma locally without sending data to external APIs, perfect for privacy-sensitive applications and development. The same provider reaches Ollama Cloud — the large open models Ollama serves from ollama.com, metered per token — and lists and describes the models any Ollama host serves.
Configuration
Basic Setup
Configure Ollama in your agent:
class OllamaAgent < ApplicationAgent
generate_with :ollama, model: "deepseek-r1:latest"
# @return [ActiveAgent::Generation]
def ask
prompt(message: params[:message])
end
endBasic Usage Example
response = OllamaAgent.with(
message: "What is a design pattern?"
).ask.generate_nowResponse Example
Configuration File
Set up Ollama in config/active_agent.yml:
ollama: &ollama
service: "Ollama"
model: "gpt-oss:20b"Environment Variables
A local server needs no API key. The provider reads the same variables the ollama command does:
OLLAMA_HOST=http://localhost:11434 # the server; a bare origin is completed to its /v1 endpoint
OLLAMA_API_KEY=... # Ollama Cloud, or a remote server behind an authenticating proxyOLLAMA_HOST is read the way ollama reads it: a scheme-less host[:port] is http on port 11434, https://ollama.com is the cloud, and an explicit path (https://ai.example.com/ollama/v1, a reverse proxy) is kept. A host: given to generate_with or in config/active_agent.yml wins over the variable. OLLAMA_ACCESS_TOKEN is read as OLLAMA_API_KEY is.
Ollama Cloud
Ollama Cloud serves open models too large for most machines — gpt-oss:120b, kimi-k3, glm-5.3, deepseek-v4-flash, qwen3.5:397b and more — behind the same OpenAI-compatible API as a local server, metered per token. There are two ways to reach it, and the provider handles both.
Directly, with cloud: true and an API key (created at ollama.com/settings/keys), the model named as the cloud lists it:
class OllamaCloudAgent < ApplicationAgent
generate_with :ollama, cloud: true, model: "gpt-oss:120b"
# @return [ActiveAgent::Generation]
def ask
prompt(message: params[:message])
end
endThe key comes from OLLAMA_API_KEY or api_key:; a request to the cloud without one fails before it is sent, with ActiveAgent::Providers::Errors::AuthenticationFailed naming what to set. cloud: true sends requests to https://ollama.com/v1 unless a host: is given, so a proxy in front of the cloud keeps its address.
response = OllamaCloudAgent.with(
message: "What is a design pattern?"
).ask.generate_nowIn config/active_agent.yml:
ollama_cloud: &ollama_cloud
service: "Ollama"
cloud: true
model: "gpt-oss:120b"
api_key: <%= ENV["OLLAMA_API_KEY"] %>Through a local server signed in to the cloud (ollama signin), nothing changes: the host is the local one, no key is needed, and the model carries the local server's cloud marker — generate_with :ollama, model: "gpt-oss:120b-cloud". The provider knows that gpt-oss:120b-cloud and glm-4.6:cloud name the cloud's gpt-oss:120b and glm-4.6 (ActiveAgent::Providers::Ollama::Cloud.canonical_model), which is how the dashboard prices them alike.
Looking Up Models
ActiveAgent::Providers::Ollama::Catalog reads the models a host serves from the host's own API — GET /api/tags to list them, POST /api/show to describe one — for a local server and for Ollama Cloud alike. A listing says which models the cloud serves (listed by the cloud, or relayed to it by a signed-in local server); a description adds the capabilities and the context length.
catalog = ActiveAgent::Providers::Ollama::Catalog.new(
host: "https://ollama.com", api_key: ENV["OLLAMA_API_KEY"]
)
names = catalog.models.map(&:name) # => ["gpt-oss:120b", "kimi-k3"]
model = catalog.model_info("gpt-oss:120b")
model.capabilities # => ["completion", "tools", "thinking"]
model.context_length # => 131072
model.cloud? # => trueAn agent's provider offers the same two reads for its own host: provider.models and provider.model_info(name). A host with only the OpenAI-compatible API (a proxy in front of Ollama) is listed through GET /v1/models, which names models and nothing else. Failures are typed — Catalog::ConnectionError, Catalog::AuthenticationError for a refused key, Catalog::NotFoundError for a model the host does not serve — and the dashboard's Settings → Provider API Keys runs the same lookup as its connection test.
Pricing on Traces and Evaluations
Ollama Cloud is metered per token, and the dashboard prices a model run under Ollama from the cloud's listing by the name the cloud uses: gpt-oss:120b-cloud through a local server and gpt-oss:120b through the cloud cost the same, and deepseek-v4-flash under Ollama is Ollama's price, not DeepSeek's. The rates come from RubyLLM's model registry (its ollama_cloud provider) when the host app loads RubyLLM, and from the same rows bundled in the dashboard otherwise, so a cloud model shows a catalog rate on traces, metrics and evaluation runs rather than a fallback guess. A model the cloud does not list — a local qwen3:8b — is priced as before. The connection test and the model pickers' GET /api/provider_models?provider=ollama carry each listed model's rate.
Installing Ollama
macOS/Linux
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh
# Start Ollama service
ollama serve
# Pull a model
ollama pull llama3Docker
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull llama3Supported Models
Ollama supports a wide range of open-source models that run locally on your machine. For the complete list of available models, see Ollama's Model Library.
Popular Models
| Model | Sizes | Context Window | Best For |
|---|---|---|---|
| llama3 | 8B, 70B | 8K tokens | General purpose reasoning |
| mistral | 7B | 32K tokens | Balanced performance |
| gemma | 2B, 7B | 8K tokens | Lightweight, efficient |
| codellama | 7B, 13B, 34B | 16K tokens | Code generation and analysis |
| mixtral | 8x7B | 32K tokens | High quality, mixture of experts |
| phi | 2.7B | 2K tokens | Fast, small footprint |
| qwen | 0.5B to 72B | 32K tokens | Multilingual support |
| deepseek-r1 | 1.5B to 70B | 64K tokens | Advanced reasoning |
Recommended model identifiers:
- llama3 - Best for general use and reasoning
- codellama - Best for code-related tasks
- mistral - Best for long context understanding
Quantized Models
Ollama offers quantized versions that reduce memory usage and increase speed with minimal quality loss. For example: ollama pull qwen3:0.6b
List Installed Models
# List all locally available models
ollama list
# Pull a new model
ollama pull llama3
# Remove a model
ollama rm llama3Provider-Specific Parameters
Required Parameters
model- Model name (e.g., "llama3", "mistral")
Sampling Parameters
temperature- Controls randomness (0.0 to 1.0)top_p- Nucleus sampling parameter (0.0 to 1.0)top_k- Top-k sampling parameter (integer ≥ 0)num_predict- Maximum tokens to generateseed- For reproducible outputs (integer)stop- Array of stop sequences
System Configuration
host- Ollama server URL (default:http://localhost:11434, orOLLAMA_HOST); a bare origin is completed to its/v1endpointcloud-truesends requests to Ollama Cloud and expects an API keyapi_key- Bearer token for Ollama Cloud or a remote server (default:OLLAMA_API_KEY)keep_alive- Keep model loaded in memory (e.g., "5m", "1h")timeout- Request timeout in seconds
Advanced Options
class AdvancedOllamaAgent < ApplicationAgent
generate_with :ollama,
model: "llama3",
temperature: 0.7,
options: {
num_ctx: 4096, # Context window size
num_gpu: 1, # Number of GPUs to use
num_thread: 8, # Number of threads
repeat_penalty: 1.1, # Penalize repetition
mirostat: 2, # Mirostat sampling
mirostat_tau: 5.0, # Mirostat tau parameter
mirostat_eta: 0.1 # Mirostat learning rate
}
def ask
prompt(message: params[:message])
end
endEmbeddings
embedding_model- Embedding model name (e.g., "nomic-embed-text")host- Ollama server URL for embeddings
Streaming
stream- Enable streaming responses (boolean, default: false)
Local Inference
Run models completely offline without external API calls. All inference happens on your machine without requiring an internet connection.
Privacy Benefits:
- All data stays on your machine
- No external API calls
- No internet connection required after model download
- Full control over your data
Performance Optimization
Model Loading
Keep models in memory for faster responses:
class FastOllamaAgent < ApplicationAgent
generate_with :ollama,
model: "llama3",
keep_alive: "5m" # Keep model loaded for 5 minutes
def quick_response
prompt(message: params[:query])
end
endHardware Acceleration
Configure GPU usage for better performance:
class GPUAgent < ApplicationAgent
generate_with :ollama,
model: "llama3",
options: {
num_gpu: -1, # Use all available GPUs
main_gpu: 0 # Primary GPU index
}
def ask
prompt(message: params[:message])
end
endQuantization
Use quantized models for faster inference with less memory:
# Pull quantized versions
ollama pull llama3:8b-q4_0 # 4-bit quantization
ollama pull llama3:8b-q5_1 # 5-bit quantizationclass EfficientAgent < ApplicationAgent
# Use quantized model for faster inference
generate_with :ollama, model: "qwen3:0.6b"
def ask
prompt(message: params[:message])
end
endStructured Output
Ollama does not have native structured output support. However, many models can generate JSON through careful prompting. For comprehensive structured output patterns, see the Structured Output Documentation.
Limitations
- No guaranteed JSON output - Depends on model following instructions
- No schema enforcement - Cannot guarantee specific field requirements
- Quality varies by model - Llama 3, Mixtral, and Mistral work best
- Requires validation - Always parse and validate responses
TIP
For applications requiring guaranteed schema conformance, use OpenAI with strict mode or Anthropic. For local processing, implement robust validation and error handling.
Embeddings
Generate embeddings locally using Ollama's embedding models. For comprehensive embedding usage patterns, see the Embeddings Documentation.
Available Embedding Models
| Model | Dimensions | Best For |
|---|---|---|
| nomic-embed-text | 768 | High-quality text embeddings |
| mxbai-embed-large | 1024 | Large embedding model |
| all-minilm | 384 | Lightweight embeddings |
Error Handling
Ollama-specific error handling for connection failures and missing models. For comprehensive error handling strategies, see the Error Handling Documentation.
Common Ollama Errors
Errno::ECONNREFUSED- Ollama service not running (start withollama serve)Net::OpenTimeout- Connection timeoutActiveAgent::GenerationError- Model not found or generation failure
Example
class RobustOllamaAgent < ApplicationAgent
generate_with :ollama, model: "llama3"
rescue_from ::OpenAI::Errors::APIConnectionError do |error|
Rails.logger.error "Ollama not running: #{error.message}"
"Ollama is not running. Start it with: ollama serve"
end
rescue_from StandardError do |error|
if error.message.include?("model not found")
# Pull the model if it's not found
# system("ollama pull #{generation_provider.model}")
raise error # Re-raise for this example
else
raise
end
end
def ask
prompt(message: params[:message])
end
endBest Practices
- Pre-pull models - Download models before first use:
ollama pull llama3 - Monitor memory usage - Large models require significant RAM (8GB+ recommended)
- Use appropriate models - Balance size, speed, and capability for your use case
- Keep models loaded - Use
keep_aliveparameter for frequently used models - Implement fallbacks - Handle connection failures and missing models gracefully
- Use quantization - Reduce memory usage and increase speed with quantized models
- Test locally - Ensure models work in development before deployment
- Consider GPU - Use GPU acceleration for better performance with larger models
Related Documentation
- Dashboard - Traces, metrics and evaluation costs, priced per model and provider
- Streaming - Real-time response streaming patterns
- Embeddings Framework - Complete guide to embeddings
- Configuration - Global provider setup
- Structured Output - Structured output patterns
- Providers Overview - Provider comparison
- Configuration Guide - Setup and configuration
- Error Handling - Error handling strategies
- Ollama Documentation - Official Ollama docs
- Ollama Model Library - Available models
- Ollama Cloud - The cloud's models and per-token pricing