Testing ActiveAgent Applications

This guide covers testing strategies and utilities for ActiveAgent applications, including credential management, VCR integration, and test patterns.

Credential Management

ActiveAgent provides helper methods for checking provider credentials in tests. These helpers check both Rails credentials and environment variables.

Available Helper Methods

All test classes that inherit from ActiveSupport::TestCase have access to these credential helpers:

ruby

# Check if any provider has credentials
has_provider_credentials?(provider)  # :openai, :anthropic, :open_router, :ollama

# Provider-specific helpers
has_openai_credentials?       # Checks Rails credentials and ENV vars
has_anthropic_credentials?    # Checks Rails credentials and ENV vars
has_openrouter_credentials?   # Checks Rails credentials and ENV vars
has_ollama_credentials?       # Checks for Ollama server configuration

Using Credential Helpers in Tests

Skip tests when credentials aren't available:

ruby

class MyAgentTest < ActiveSupport::TestCase
  test "generates response with OpenAI" do
    skip "Requires API credentials" unless has_openai_credentials?
    
    # Test implementation
  end
  
  test "uses Anthropic for complex reasoning" do
    skip "Requires API credentials" unless has_anthropic_credentials?
    
    # Test implementation
  end
end

Credential Configuration

Credentials are checked in this order:

Rails Credentials (Recommended)

bash

rails credentials:edit

yaml

openai:
  access_token: your-api-key
anthropic:
  access_token: your-api-key
open_router:
  access_token: your-api-key  # or api_key

Environment Variables (Fallback)

bash

export OPENAI_ACCESS_TOKEN=your-api-key
export OPENAI_API_KEY=your-api-key  # Alternative
export ANTHROPIC_ACCESS_TOKEN=your-api-key
export ANTHROPIC_API_KEY=your-api-key  # Alternative
export OPENROUTER_API_KEY=your-api-key

Testing Agent Actions

The Correct Pattern

ActiveAgent uses a specific pattern for calling agent actions with parameters:

ruby

# CORRECT: Use the class method 'with' to pass parameters
generation = MyAgent.with(param1: "value1", param2: "value2").action_name
result = generation.generate_now

# INCORRECT: Don't call actions directly with arguments
# agent = MyAgent.new
# result = agent.action_name(param1: "value1")  # This won't work!

Complete Test Example

ruby

test "web search with responses API example" do
  skip "Requires API credentials" unless has_openai_credentials?

  VCR.use_cassette("doc_web_search_responses") do
    generation = WebSearchAgent.with(
      query: "Latest Ruby on Rails 8 features",
      context_size: "high"
    ).search_with_tools

    result = generation.generate_now

    # The response includes web search results
    assert result.message.content.present?
    assert result.message.content.include?("Rails")

    doc_example_output(result)
  end
end

Testing with VCR

Use VCR to record and replay API responses:

ruby

class MyAgentTest < ActiveSupport::TestCase
  test "performs complex task" do
    skip "Requires API credentials" unless has_openai_credentials?
    
    VCR.use_cassette("my_agent_complex_task") do
      generation = MyAgent.with(
        input: "test data",
        mode: "analysis"
      ).analyze
      
      result = generation.generate_now
      
      assert result.message.content.present?
      assert result.message.content.include?("expected text")
      
      # Generate documentation examples
      doc_example_output(result)
    end
  end
end

Testing Concerns

ActiveAgent supports using concerns to share functionality across agents:

Creating a Test for a Concern

ruby

require "test_helper"
require_relative "../dummy/app/agents/research_agent"
require_relative "../dummy/app/agents/concerns/research_tools"

class ConcernToolsTest < ActiveSupport::TestCase
  setup do
    @agent = ResearchAgent.new
  end

  test "research agent includes concern actions as available tools" do
    # The concern adds these actions which should be available as tools
    expected_actions = [
      "search_academic_papers",
      "analyze_research_data",
      "generate_research_visualization",
      "search_with_mcp_sources"
    ]

    agent_actions = @agent.action_methods
    expected_actions.each do |action|
      assert_includes agent_actions, action, "Expected #{action} to be available from concern"
    end
  end

  test "concern can add built-in tools for responses API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_web_search_responses_api") do
      # When using responses API with multimodal content
      # Use the search_academic_papers action from the concern
      generation = ResearchAgent.with(
        query: "latest research on large language models",
        year_from: 2024,
        year_to: 2025,
        field: "AI"
      ).search_academic_papers

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern can configure web search for chat completions API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_web_search_chat_api") do
      # When using chat API with web search model
      # Use the comprehensive_research action which builds tools dynamically
      generation = ResearchAgent.with(
        topic: "latest research on large language models",
        depth: "detailed"
      ).comprehensive_research

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern supports MCP tools only in responses API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_mcp_tools") do
      # MCP is only supported in Responses API
      # Use the search_with_mcp_sources action from the concern
      generation = ResearchAgent.with(
        query: "Ruby on Rails best practices",
        sources: [ "github" ]
      ).search_with_mcp_sources

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern actions work with both chat and responses API" do
    # Test that the same action can work with different APIs

    # Test with Chat Completions API (function calling)
    chat_prompt = ActiveAgent::ActionPrompt::Prompt.new
    chat_prompt.options = { model: "gpt-4o" }
    chat_prompt.actions = @agent.action_schemas  # Function schemas

    assert chat_prompt.actions.any? { |a| a["function"]["name"] == "search_academic_papers" }

    # Test with Responses API (can use built-in tools)
    responses_prompt = ActiveAgent::ActionPrompt::Prompt.new
    responses_prompt.options = {
      model: "gpt-5",
      use_responses_api: true,
      tools: [
        { type: "web_search_preview" },
        { type: "image_generation" }
      ]
    }

    # Should have both function tools and built-in tools
    assert responses_prompt.options[:tools].any? { |t| t[:type] == "web_search_preview" }
    assert responses_prompt.options[:tools].any? { |t| t[:type] == "image_generation" }
  end

  test "concern can dynamically configure tools based on context" do
    # The concern can decide which tools to include based on parameters

    # The ResearchAgent.comprehensive_research method should configure tools dynamically
    # based on the depth parameter. We verify this by checking that the agent
    # has the method and that it accepts the expected parameters.

    agent = ResearchAgent.new
    assert agent.respond_to?(:comprehensive_research)

    # Verify the agent has access to the tool configuration methods
    assert ResearchAgent.research_tools_config[:enable_web_search]
    assert ResearchAgent.research_tools_config[:mcp_servers].present?
  end

  test "concern configuration is inherited at class level" do
    # ResearchAgent configured with specific settings
    assert ResearchAgent.research_tools_config[:enable_web_search]
    assert_equal [ "arxiv", "github" ], ResearchAgent.research_tools_config[:mcp_servers]
    assert_equal "high", ResearchAgent.research_tools_config[:default_search_context]
  end

  test "multiple concerns can add different tool types" do
    # Create an agent with multiple concerns
    class MultiToolAgent < ApplicationAgent
      include ResearchTools
      # Could include other tool concerns like ImageTools, DataTools, etc.

      generate_with :openai, model: "gpt-4o"
    end

    agent = MultiToolAgent.new

    # Should have all actions from all concerns
    assert agent.respond_to?(:search_academic_papers)
    assert agent.respond_to?(:analyze_research_data)
    assert agent.respond_to?(:generate_research_visualization)
  end

  test "concern tools respect API limitations" do
    # Test that we don't try to use unsupported features

    # MCP should not be available in Chat API
    chat_prompt = ActiveAgent::ActionPrompt::Prompt.new
    chat_prompt.options = {
      model: "gpt-4o",  # Regular chat model
      tools: [
        { type: "mcp", server_url: "https://example.com" }  # This won't work
      ]
    }

    # Provider should filter out MCP for chat API
    provider = ActiveAgent::GenerationProvider::OpenAIProvider.new({ "model" => "gpt-4o" })
    provider.instance_variable_set(:@prompt, chat_prompt)

    # When using chat API, MCP tools should not be included
    # This test verifies that the configuration is set up correctly
    assert chat_prompt.options[:model] == "gpt-4o"
    assert chat_prompt.options[:tools].any? { |t| t[:type] == "mcp" }
  end
end

Testing Concern Configuration

ruby

require "test_helper"
require_relative "../dummy/app/agents/research_agent"
require_relative "../dummy/app/agents/concerns/research_tools"

class ConcernToolsTest < ActiveSupport::TestCase
  setup do
    @agent = ResearchAgent.new
  end

  test "research agent includes concern actions as available tools" do
    # The concern adds these actions which should be available as tools
    expected_actions = [
      "search_academic_papers",
      "analyze_research_data",
      "generate_research_visualization",
      "search_with_mcp_sources"
    ]

    agent_actions = @agent.action_methods
    expected_actions.each do |action|
      assert_includes agent_actions, action, "Expected #{action} to be available from concern"
    end
  end

  test "concern can add built-in tools for responses API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_web_search_responses_api") do
      # When using responses API with multimodal content
      # Use the search_academic_papers action from the concern
      generation = ResearchAgent.with(
        query: "latest research on large language models",
        year_from: 2024,
        year_to: 2025,
        field: "AI"
      ).search_academic_papers

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern can configure web search for chat completions API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_web_search_chat_api") do
      # When using chat API with web search model
      # Use the comprehensive_research action which builds tools dynamically
      generation = ResearchAgent.with(
        topic: "latest research on large language models",
        depth: "detailed"
      ).comprehensive_research

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern supports MCP tools only in responses API" do
    skip "Requires API credentials" unless has_openai_credentials?

    VCR.use_cassette("concern_mcp_tools") do
      # MCP is only supported in Responses API
      # Use the search_with_mcp_sources action from the concern
      generation = ResearchAgent.with(
        query: "Ruby on Rails best practices",
        sources: [ "github" ]
      ).search_with_mcp_sources

      response = generation.generate_now

      assert response.message.content.present?
    end
  end

  test "concern actions work with both chat and responses API" do
    # Test that the same action can work with different APIs

    # Test with Chat Completions API (function calling)
    chat_prompt = ActiveAgent::ActionPrompt::Prompt.new
    chat_prompt.options = { model: "gpt-4o" }
    chat_prompt.actions = @agent.action_schemas  # Function schemas

    assert chat_prompt.actions.any? { |a| a["function"]["name"] == "search_academic_papers" }

    # Test with Responses API (can use built-in tools)
    responses_prompt = ActiveAgent::ActionPrompt::Prompt.new
    responses_prompt.options = {
      model: "gpt-5",
      use_responses_api: true,
      tools: [
        { type: "web_search_preview" },
        { type: "image_generation" }
      ]
    }

    # Should have both function tools and built-in tools
    assert responses_prompt.options[:tools].any? { |t| t[:type] == "web_search_preview" }
    assert responses_prompt.options[:tools].any? { |t| t[:type] == "image_generation" }
  end

  test "concern can dynamically configure tools based on context" do
    # The concern can decide which tools to include based on parameters

    # The ResearchAgent.comprehensive_research method should configure tools dynamically
    # based on the depth parameter. We verify this by checking that the agent
    # has the method and that it accepts the expected parameters.

    agent = ResearchAgent.new
    assert agent.respond_to?(:comprehensive_research)

    # Verify the agent has access to the tool configuration methods
    assert ResearchAgent.research_tools_config[:enable_web_search]
    assert ResearchAgent.research_tools_config[:mcp_servers].present?
  end

  test "concern configuration is inherited at class level" do
    # ResearchAgent configured with specific settings
    assert ResearchAgent.research_tools_config[:enable_web_search]
    assert_equal [ "arxiv", "github" ], ResearchAgent.research_tools_config[:mcp_servers]
    assert_equal "high", ResearchAgent.research_tools_config[:default_search_context]
  end

  test "multiple concerns can add different tool types" do
    # Create an agent with multiple concerns
    class MultiToolAgent < ApplicationAgent
      include ResearchTools
      # Could include other tool concerns like ImageTools, DataTools, etc.

      generate_with :openai, model: "gpt-4o"
    end

    agent = MultiToolAgent.new

    # Should have all actions from all concerns
    assert agent.respond_to?(:search_academic_papers)
    assert agent.respond_to?(:analyze_research_data)
    assert agent.respond_to?(:generate_research_visualization)
  end

  test "concern tools respect API limitations" do
    # Test that we don't try to use unsupported features

    # MCP should not be available in Chat API
    chat_prompt = ActiveAgent::ActionPrompt::Prompt.new
    chat_prompt.options = {
      model: "gpt-4o",  # Regular chat model
      tools: [
        { type: "mcp", server_url: "https://example.com" }  # This won't work
      ]
    }

    # Provider should filter out MCP for chat API
    provider = ActiveAgent::GenerationProvider::OpenAIProvider.new({ "model" => "gpt-4o" })
    provider.instance_variable_set(:@prompt, chat_prompt)

    # When using chat API, MCP tools should not be included
    # This test verifies that the configuration is set up correctly
    assert chat_prompt.options[:model] == "gpt-4o"
    assert chat_prompt.options[:tools].any? { |t| t[:type] == "mcp" }
  end
end

Generating Documentation Examples

Use the doc_example_output method to generate documentation examples from test results:

ruby

test "example for documentation" do
  VCR.use_cassette("doc_example") do
    generation = MyAgent.with(
      query: "How do I use ActiveAgent?"
    ).help
    
    result = generation.generate_now
    
    # Generate example file in docs/parts/examples/
    doc_example_output(result)
  end
end

The generated examples can be included in documentation:

markdown

::: details Response Example
<!-- @include: @/parts/examples/my-agent-test-example-for-documentation.md -->
:::

Test Helpers

ActiveAgentTestCase

Use ActiveAgentTestCase for tests that need to manage ActiveAgent configuration:

ruby

class ConfigurationTest < ActiveAgentTestCase
  def test_with_custom_config
    with_active_agent_config(custom_config) do
      # Test with custom configuration
    end
  end
end

Testing Multiple Providers

Test agent behavior across different providers:

ruby

class MultiProviderTest < ActiveSupport::TestCase
  test "works with OpenAI" do
    skip unless has_openai_credentials?
    
    agent_class = Class.new(ApplicationAgent) do
      generate_with :openai, model: "gpt-4o"
      
      def test_action
        prompt message: "test"
      end
    end
    
    generation = agent_class.with.test_action
    result = generation.generate_now
    assert result.message.content.present?
  end
  
  test "works with Anthropic" do
    skip unless has_anthropic_credentials?
    
    agent_class = Class.new(ApplicationAgent) do
      generate_with :anthropic, model: "claude-3-5-sonnet-latest"
      
      def test_action
        prompt message: "test"
      end
    end
    
    generation = agent_class.with.test_action
    result = generation.generate_now
    assert result.message.content.present?
  end
end

Testing Built-in Tools

When testing agents that use OpenAI's built-in tools (web search, image generation, MCP):

Web Search Testing

ruby

test "searches the web for information" do
  skip unless has_openai_credentials?
  
  VCR.use_cassette("web_search_test") do
    generation = WebSearchAgent.with(
      query: "Latest Ruby on Rails features",
      context_size: "high"
    ).search_with_tools
    
    result = generation.generate_now
    assert result.message.content.present?
  end
end

Image Generation Testing

ruby

test "generates images" do
  skip unless has_openai_credentials?
  
  VCR.use_cassette("image_generation_test") do
    generation = MultimodalAgent.with(
      description: "A peaceful mountain landscape",
      size: "1024x1024",
      quality: "high"
    ).create_image
    
    result = generation.generate_now
    assert result.message.content.present?
  end
end

MCP Integration Testing

ruby

test "connects to MCP servers" do
  skip unless has_openai_credentials?
  
  VCR.use_cassette("mcp_integration_test") do
    generation = ResearchAgent.with(
      query: "Ruby performance optimization",
      sources: ["github"]
    ).search_with_mcp_sources
    
    result = generation.generate_now
    assert result.message.content.present?
  end
end

Best Practices

Always use credential helpers - Skip tests gracefully when credentials aren't available
Use VCR for API calls - Record responses for consistent, fast tests
Follow the ActiveAgent pattern - Use Agent.with(params).action.generate_now
Generate documentation examples - Use doc_example_output for real examples
Test concerns separately - Ensure shared functionality works correctly
Clean up VCR cassettes - Remove old cassettes when updating test implementations
Test error handling - Ensure agents handle API errors gracefully

Common Issues

Wrong Pattern Errors

If you see wrong number of arguments errors, you're likely calling actions incorrectly:

ruby

# WRONG - Actions don't accept arguments directly
agent = MyAgent.new
agent.my_action(param: "value")  # Error!

# RIGHT - Use the with pattern
MyAgent.with(param: "value").my_action.generate_now

Missing Credentials

If tests are skipped unexpectedly, check:

Rails credentials are properly set: rails credentials:edit
Environment variables are exported in your shell
The credential helper is checking the right keys

VCR Cassette Issues

If tests fail with API errors when cassettes exist:

Delete old cassettes: rm test/fixtures/vcr_cassettes/your_test.yml
Re-run tests to record new cassettes
Ensure API credentials are valid

Testing ActiveAgent Applications ​

Credential Management ​

Available Helper Methods ​

Using Credential Helpers in Tests ​

Credential Configuration ​

Testing Agent Actions ​

The Correct Pattern ​

Complete Test Example ​

Testing with VCR ​

Testing Concerns ​

Creating a Test for a Concern ​

Testing Concern Configuration ​

Generating Documentation Examples ​

Test Helpers ​

ActiveAgentTestCase ​

Testing Multiple Providers ​

Testing Built-in Tools ​

Web Search Testing ​

Image Generation Testing ​

MCP Integration Testing ​

Best Practices ​

Common Issues ​

Wrong Pattern Errors ​

Missing Credentials ​

VCR Cassette Issues ​

Related Documentation ​

Testing ActiveAgent Applications

Credential Management

Available Helper Methods

Using Credential Helpers in Tests

Credential Configuration

Testing Agent Actions

The Correct Pattern

Complete Test Example

Testing with VCR

Testing Concerns

Creating a Test for a Concern

Testing Concern Configuration

Generating Documentation Examples

Test Helpers

ActiveAgentTestCase

Testing Multiple Providers

Testing Built-in Tools

Web Search Testing

Image Generation Testing

MCP Integration Testing

Best Practices

Common Issues

Wrong Pattern Errors

Missing Credentials

VCR Cassette Issues

Related Documentation