Skip to content

Ollama Provider ​

The Ollama provider enables local LLM inference using the Ollama platform. Run models like Llama 3, Mistral, and Gemma locally without sending data to external APIs, perfect for privacy-sensitive applications and development. The same provider reaches Ollama Cloud — the large open models Ollama serves from ollama.com, metered per token — and lists and describes the models any Ollama host serves.

Configuration ​

Basic Setup ​

Configure Ollama in your agent:

ruby
class OllamaAgent < ApplicationAgent
  generate_with :ollama, model: "deepseek-r1:latest"

  # @return [ActiveAgent::Generation]
  def ask
    prompt(message: params[:message])
  end
end

Basic Usage Example ​

ruby
response = OllamaAgent.with(
  message: "What is a design pattern?"
).ask.generate_now
Response Example

Configuration File ​

Set up Ollama in config/active_agent.yml:

yaml
ollama: &ollama
  service: "Ollama"
  model: "gpt-oss:20b"

Environment Variables ​

A local server needs no API key. The provider reads the same variables the ollama command does:

bash
OLLAMA_HOST=http://localhost:11434   # the server; a bare origin is completed to its /v1 endpoint
OLLAMA_API_KEY=...                   # Ollama Cloud, or a remote server behind an authenticating proxy

OLLAMA_HOST is read the way ollama reads it: a scheme-less host[:port] is http on port 11434, https://ollama.com is the cloud, and an explicit path (https://ai.example.com/ollama/v1, a reverse proxy) is kept. A host: given to generate_with or in config/active_agent.yml wins over the variable. OLLAMA_ACCESS_TOKEN is read as OLLAMA_API_KEY is.

Ollama Cloud ​

Ollama Cloud serves open models too large for most machines — gpt-oss:120b, kimi-k3, glm-5.3, deepseek-v4-flash, qwen3.5:397b and more — behind the same OpenAI-compatible API as a local server, metered per token. There are two ways to reach it, and the provider handles both.

Directly, with cloud: true and an API key (created at ollama.com/settings/keys), the model named as the cloud lists it:

ruby
class OllamaCloudAgent < ApplicationAgent
  generate_with :ollama, cloud: true, model: "gpt-oss:120b"

  # @return [ActiveAgent::Generation]
  def ask
    prompt(message: params[:message])
  end
end

The key comes from OLLAMA_API_KEY or api_key:; a request to the cloud without one fails before it is sent, with ActiveAgent::Providers::Errors::AuthenticationFailed naming what to set. cloud: true sends requests to https://ollama.com/v1 unless a host: is given, so a proxy in front of the cloud keeps its address.

ruby
response = OllamaCloudAgent.with(
  message: "What is a design pattern?"
).ask.generate_now

In config/active_agent.yml:

yaml
ollama_cloud: &ollama_cloud
  service: "Ollama"
  cloud: true
  model: "gpt-oss:120b"
  api_key: <%= ENV["OLLAMA_API_KEY"] %>

Through a local server signed in to the cloud (ollama signin), nothing changes: the host is the local one, no key is needed, and the model carries the local server's cloud marker — generate_with :ollama, model: "gpt-oss:120b-cloud". The provider knows that gpt-oss:120b-cloud and glm-4.6:cloud name the cloud's gpt-oss:120b and glm-4.6 (ActiveAgent::Providers::Ollama::Cloud.canonical_model), which is how the dashboard prices them alike.

Looking Up Models ​

ActiveAgent::Providers::Ollama::Catalog reads the models a host serves from the host's own API — GET /api/tags to list them, POST /api/show to describe one — for a local server and for Ollama Cloud alike. A listing says which models the cloud serves (listed by the cloud, or relayed to it by a signed-in local server); a description adds the capabilities and the context length.

ruby
catalog = ActiveAgent::Providers::Ollama::Catalog.new(
  host: "https://ollama.com", api_key: ENV["OLLAMA_API_KEY"]
)
names = catalog.models.map(&:name)                 # => ["gpt-oss:120b", "kimi-k3"]
model = catalog.model_info("gpt-oss:120b")
model.capabilities                                  # => ["completion", "tools", "thinking"]
model.context_length                                # => 131072
model.cloud?                                        # => true

An agent's provider offers the same two reads for its own host: provider.models and provider.model_info(name). A host with only the OpenAI-compatible API (a proxy in front of Ollama) is listed through GET /v1/models, which names models and nothing else. Failures are typed — Catalog::ConnectionError, Catalog::AuthenticationError for a refused key, Catalog::NotFoundError for a model the host does not serve — and the dashboard's Settings → Provider API Keys runs the same lookup as its connection test.

Pricing on Traces and Evaluations ​

Ollama Cloud is metered per token, and the dashboard prices a model run under Ollama from the cloud's listing by the name the cloud uses: gpt-oss:120b-cloud through a local server and gpt-oss:120b through the cloud cost the same, and deepseek-v4-flash under Ollama is Ollama's price, not DeepSeek's. The rates come from RubyLLM's model registry (its ollama_cloud provider) when the host app loads RubyLLM, and from the same rows bundled in the dashboard otherwise, so a cloud model shows a catalog rate on traces, metrics and evaluation runs rather than a fallback guess. A model the cloud does not list — a local qwen3:8b — is priced as before. The connection test and the model pickers' GET /api/provider_models?provider=ollama carry each listed model's rate.

Installing Ollama ​

macOS/Linux ​

bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh

# Start Ollama service
ollama serve

# Pull a model
ollama pull llama3

Docker ​

bash
docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama
docker exec -it ollama ollama pull llama3

Supported Models ​

Ollama supports a wide range of open-source models that run locally on your machine. For the complete list of available models, see Ollama's Model Library.

ModelSizesContext WindowBest For
llama38B, 70B8K tokensGeneral purpose reasoning
mistral7B32K tokensBalanced performance
gemma2B, 7B8K tokensLightweight, efficient
codellama7B, 13B, 34B16K tokensCode generation and analysis
mixtral8x7B32K tokensHigh quality, mixture of experts
phi2.7B2K tokensFast, small footprint
qwen0.5B to 72B32K tokensMultilingual support
deepseek-r11.5B to 70B64K tokensAdvanced reasoning

Recommended model identifiers:

  • llama3 - Best for general use and reasoning
  • codellama - Best for code-related tasks
  • mistral - Best for long context understanding

Quantized Models

Ollama offers quantized versions that reduce memory usage and increase speed with minimal quality loss. For example: ollama pull qwen3:0.6b

List Installed Models ​

bash
# List all locally available models
ollama list

# Pull a new model
ollama pull llama3

# Remove a model
ollama rm llama3

Provider-Specific Parameters ​

Required Parameters ​

  • model - Model name (e.g., "llama3", "mistral")

Sampling Parameters ​

  • temperature - Controls randomness (0.0 to 1.0)
  • top_p - Nucleus sampling parameter (0.0 to 1.0)
  • top_k - Top-k sampling parameter (integer ≥ 0)
  • num_predict - Maximum tokens to generate
  • seed - For reproducible outputs (integer)
  • stop - Array of stop sequences

System Configuration ​

  • host - Ollama server URL (default: http://localhost:11434, or OLLAMA_HOST); a bare origin is completed to its /v1 endpoint
  • cloud - true sends requests to Ollama Cloud and expects an API key
  • api_key - Bearer token for Ollama Cloud or a remote server (default: OLLAMA_API_KEY)
  • keep_alive - Keep model loaded in memory (e.g., "5m", "1h")
  • timeout - Request timeout in seconds

Advanced Options ​

ruby
class AdvancedOllamaAgent < ApplicationAgent
  generate_with :ollama,
    model: "llama3",
    temperature: 0.7,
    options: {
      num_ctx: 4096,         # Context window size
      num_gpu: 1,            # Number of GPUs to use
      num_thread: 8,         # Number of threads
      repeat_penalty: 1.1,   # Penalize repetition
      mirostat: 2,           # Mirostat sampling
      mirostat_tau: 5.0,     # Mirostat tau parameter
      mirostat_eta: 0.1      # Mirostat learning rate
    }

  def ask
    prompt(message: params[:message])
  end
end

Embeddings ​

  • embedding_model - Embedding model name (e.g., "nomic-embed-text")
  • host - Ollama server URL for embeddings

Streaming ​

  • stream - Enable streaming responses (boolean, default: false)

Local Inference ​

Run models completely offline without external API calls. All inference happens on your machine without requiring an internet connection.

Privacy Benefits:

  • All data stays on your machine
  • No external API calls
  • No internet connection required after model download
  • Full control over your data

Performance Optimization ​

Model Loading ​

Keep models in memory for faster responses:

ruby
class FastOllamaAgent < ApplicationAgent
  generate_with :ollama,
    model: "llama3",
    keep_alive: "5m"  # Keep model loaded for 5 minutes

  def quick_response
    prompt(message: params[:query])
  end
end

Hardware Acceleration ​

Configure GPU usage for better performance:

ruby
class GPUAgent < ApplicationAgent
  generate_with :ollama,
    model: "llama3",
    options: {
      num_gpu: -1,  # Use all available GPUs
      main_gpu: 0   # Primary GPU index
    }

  def ask
    prompt(message: params[:message])
  end
end

Quantization ​

Use quantized models for faster inference with less memory:

bash
# Pull quantized versions
ollama pull llama3:8b-q4_0  # 4-bit quantization
ollama pull llama3:8b-q5_1  # 5-bit quantization
ruby
class EfficientAgent < ApplicationAgent
  # Use quantized model for faster inference
  generate_with :ollama, model: "qwen3:0.6b"

  def ask
    prompt(message: params[:message])
  end
end

Structured Output ​

Ollama does not have native structured output support. However, many models can generate JSON through careful prompting. For comprehensive structured output patterns, see the Structured Output Documentation.

Limitations ​

  • No guaranteed JSON output - Depends on model following instructions
  • No schema enforcement - Cannot guarantee specific field requirements
  • Quality varies by model - Llama 3, Mixtral, and Mistral work best
  • Requires validation - Always parse and validate responses

TIP

For applications requiring guaranteed schema conformance, use OpenAI with strict mode or Anthropic. For local processing, implement robust validation and error handling.

Embeddings ​

Generate embeddings locally using Ollama's embedding models. For comprehensive embedding usage patterns, see the Embeddings Documentation.

Available Embedding Models ​

ModelDimensionsBest For
nomic-embed-text768High-quality text embeddings
mxbai-embed-large1024Large embedding model
all-minilm384Lightweight embeddings

Error Handling ​

Ollama-specific error handling for connection failures and missing models. For comprehensive error handling strategies, see the Error Handling Documentation.

Common Ollama Errors ​

  • Errno::ECONNREFUSED - Ollama service not running (start with ollama serve)
  • Net::OpenTimeout - Connection timeout
  • ActiveAgent::GenerationError - Model not found or generation failure

Example ​

ruby
class RobustOllamaAgent < ApplicationAgent
  generate_with :ollama, model: "llama3"

  rescue_from ::OpenAI::Errors::APIConnectionError do |error|
    Rails.logger.error "Ollama not running: #{error.message}"
    "Ollama is not running. Start it with: ollama serve"
  end

  rescue_from StandardError do |error|
    if error.message.include?("model not found")
      # Pull the model if it's not found
      # system("ollama pull #{generation_provider.model}")
      raise error  # Re-raise for this example
    else
      raise
    end
  end

  def ask
    prompt(message: params[:message])
  end
end

Best Practices ​

  1. Pre-pull models - Download models before first use: ollama pull llama3
  2. Monitor memory usage - Large models require significant RAM (8GB+ recommended)
  3. Use appropriate models - Balance size, speed, and capability for your use case
  4. Keep models loaded - Use keep_alive parameter for frequently used models
  5. Implement fallbacks - Handle connection failures and missing models gracefully
  6. Use quantization - Reduce memory usage and increase speed with quantized models
  7. Test locally - Ensure models work in development before deployment
  8. Consider GPU - Use GPU acceleration for better performance with larger models