Telemetry & Observability

ActiveAgent includes built-in telemetry for collecting and reporting agent traces. Monitor your AI operations, track token usage and costs, and debug generation flows with comprehensive observability.

Overview

The telemetry system captures:

Generation Traces: Full lifecycle of agent generations
Token Usage: Input, output, and thinking tokens per request
Tool Calls: Invocations with timing and results
Errors: Exceptions with backtraces for debugging
Performance Metrics: Response times and latencies

Quick Start

Hosted Service (ActiveAgents.ai)

The fastest way to get started is with the hosted observability service:

yaml

# config/active_agent.yml
development:
  openai:
    service: "OpenAI"
    access_token: <%= Rails.application.credentials.dig(:openai, :access_token) %>

  telemetry:
    enabled: true
    endpoint: https://api.activeagents.ai/v1/traces
    api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>

Self-Hosted

For complete control, add the actionagent gem to an app of your own, mount its engine, and point telemetry at the ingest path — <mount>/api/traces, so /activeagents/api/traces at the default mount (see Self-Hosted Dashboard). Only the app that hosts the dashboard needs that gem; the apps reporting to it need activeagent alone:

yaml

# config/active_agent.yml
production:
  telemetry:
    enabled: true
    endpoint: https://observability.mycompany.com/activeagents/api/traces
    api_key: <%= ENV["TELEMETRY_API_KEY"] %>
    service_name: my-rails-app

The app that mounts the dashboard itself stores its own traces with local_storage: true instead, writing through the engine's trace model with no HTTP involved. That switch needs actionagent installed and migrated — without it, telemetry logs that it has nowhere to write rather than falling back to HTTP.

Configuration

YAML Configuration

Configure telemetry in your config/active_agent.yml:

yaml

telemetry:
  enabled: true
  endpoint: https://api.activeagents.ai/v1/traces
  api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>
  sample_rate: 1.0        # 1.0 = 100%, 0.5 = 50%
  batch_size: 100         # Traces per batch
  flush_interval: 5       # Seconds between flushes
  service_name: my-app    # Override app name
  capture_bodies: false   # Include request/response bodies
  resource_attributes:    # Custom attributes for all traces
    deployment: production
    team: ai-platform

Programmatic Configuration

Configure in an initializer for dynamic settings:

ruby

# config/initializers/active_agent.rb
ActiveAgent::Telemetry.configure do |config|
  config.enabled = Rails.env.production?
  config.endpoint = ENV.fetch("TELEMETRY_ENDPOINT", "https://api.activeagents.ai/v1/traces")
  config.api_key = Rails.application.credentials.dig(:activeagents, :api_key)
  config.sample_rate = 1.0
  config.service_name = Rails.application.class.module_parent_name.underscore
end

Rails Configuration

You can also configure via Rails config:

ruby

# config/application.rb
config.active_agent.telemetry = {
  enabled: true,
  endpoint: "https://api.activeagents.ai/v1/traces",
  api_key: Rails.application.credentials.dig(:activeagents, :api_key)
}

Configuration Options

Option	Type	Default	Description
`enabled`	Boolean	`false`	Enable telemetry collection
`endpoint`	String	`https://api.activeagents.ai/v1/traces`	Telemetry receiver URL
`local_storage`	Boolean	`false`	Store traces through the mounted dashboard's trace model instead of posting them (requires the `actionagent` gem)
`api_key`	String	`nil`	Authentication token
`sample_rate`	Float	`1.0`	Sampling rate (0.0 - 1.0)
`batch_size`	Integer	`100`	Traces per batch before flush
`flush_interval`	Integer	`5`	Seconds between auto-flushes
`timeout`	Integer	`10`	HTTP request timeout
`capture_bodies`	Boolean	`false`	Include message bodies
`service_name`	String	App name	Service identifier
`environment`	String	`Rails.env`	Environment name
`resource_attributes`	Hash	`{}`	Custom trace attributes
`redact_attributes`	Array	`["password", "secret", ...]`	Keys to redact

Trace Structure

Each trace captures the complete generation lifecycle:

Trace: WeatherAgent.forecast
├── Span: agent.prompt (prompt preparation)
├── Span: llm.generate (API call)
│   ├── tokens: { input: 150, output: 75, total: 225 }
│   └── model: "gpt-4o"
└── Span: tool.get_weather (tool invocation)
    └── duration: 234ms

Span Types

Type	Description
`root`	Root span for the entire generation
`prompt`	Prompt preparation and rendering
`llm`	LLM API call
`tool`	Tool/function invocation
`thinking`	Extended thinking (Anthropic)
`embedding`	Embedding generation
`error`	Error handling

Manual Tracing

Add custom spans to your traces:

ruby

class WeatherAgent < ApplicationAgent
  def forecast(location:)
    @location = location

    # Add custom span
    if ActiveAgent::Telemetry.enabled?
      span = ActiveAgent::Telemetry.span("geocode.lookup")
      span.set_attribute("location", location)
      coordinates = geocode_location(location)
      span.finish
    end

    prompt
  end
end

Trace Block

Use the trace block for automatic timing and error handling:

ruby

ActiveAgent::Telemetry.trace("custom.operation") do |span|
  span.set_attribute("user_id", current_user.id)
  span.set_attribute("operation", "data_enrichment")

  result = perform_operation

  span.set_tokens(input: 100, output: 50)
  result
end

Sampling

Control trace volume with sampling:

ruby

ActiveAgent::Telemetry.configure do |config|
  # Sample 10% of production traffic
  config.sample_rate = Rails.env.production? ? 0.1 : 1.0
end

Sampling is deterministic per-trace, so all spans within a trace are included or excluded together.

Flushing & Shutdown

Traces are batched and sent asynchronously. Force flush when needed:

ruby

# Flush buffered traces immediately
ActiveAgent::Telemetry.flush

# Graceful shutdown (flush and wait)
ActiveAgent::Telemetry.shutdown

Rails Integration

Telemetry automatically flushes on Rails shutdown:

ruby

# config/initializers/active_agent.rb
at_exit { ActiveAgent::Telemetry.shutdown }

Self-Hosting

The actionagent dashboard engine already implements this endpoint spec: add the gem, mount the engine, and its ingest API (<mount>/api/traces) receives traces from any app in your fleet — see Self-Hosted Observability. The requirements below are for building your own receiver instead.

Endpoint Requirements

Your telemetry endpoint must accept POST requests with:

Headers:

Content-Type: application/json
Authorization: Bearer <api_key>
X-Service-Name: <service_name>
X-Environment: <environment>

Payload:

json

{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "environment": "production",
      "timestamp": "2024-01-15T10:30:00.123456Z",
      "resource_attributes": {},
      "spans": [
        {
          "span_id": "def456",
          "trace_id": "abc123...",
          "parent_span_id": null,
          "name": "WeatherAgent.forecast",
          "type": "root",
          "start_time": "2024-01-15T10:30:00.123456Z",
          "end_time": "2024-01-15T10:30:01.234567Z",
          "duration_ms": 1111.11,
          "status": "OK",
          "attributes": {
            "agent.class": "WeatherAgent",
            "agent.action": "forecast"
          },
          "tokens": {
            "input": 150,
            "output": 75,
            "total": 225
          },
          "events": []
        }
      ]
    }
  ],
  "sdk": {
    "name": "activeagent",
    "version": "0.5.0",
    "language": "ruby",
    "runtime_version": "3.3.0"
  }
}

Example Rails Endpoint

ruby

# app/controllers/api/traces_controller.rb
class Api::TracesController < ApplicationController
  skip_before_action :verify_authenticity_token
  before_action :authenticate_api_key!

  def create
    traces = params[:traces]

    traces.each do |trace|
      Trace.create!(
        trace_id: trace[:trace_id],
        service_name: trace[:service_name],
        environment: trace[:environment],
        spans: trace[:spans],
        timestamp: trace[:timestamp]
      )
    end

    head :accepted
  end

  private

  def authenticate_api_key!
    api_key = request.headers["Authorization"]&.gsub(/^Bearer /, "")
    head :unauthorized unless ApiKey.exists?(key: api_key)
  end
end

Comparison with Instrumentation

ActiveAgent provides two complementary observability systems:

Feature	Instrumentation	Telemetry
Purpose	Local logging & metrics	Distributed tracing
Transport	ActiveSupport::Notifications	HTTP POST
Destination	Rails logs, local metrics	Remote endpoint
Use Case	Debugging, local monitoring	Production observability
Overhead	Minimal	Async, batched

Use Instrumentation for local development and debugging. Use Telemetry for production observability and analytics.

Security Considerations

Sensitive Data

By default, telemetry redacts common sensitive attributes:

password, secret, token, key, credential, api_key

Add custom redactions:

ruby

ActiveAgent::Telemetry.configure do |config|
  config.redact_attributes += ["ssn", "credit_card"]
end

Message Bodies

Message bodies — the rendered system prompt (prompt.input.instructions), the outbound messages (prompt.input.messages), the completion (llm.output.message) and tool arguments and results (tool.input.args, tool.output.result) — are not captured by default when traces leave the process. Counts, tool names, token usage and finish reasons are always recorded. Enable bodies with caution:

ruby

ActiveAgent::Telemetry.configure do |config|
  config.capture_bodies = true  # Only in controlled environments
end

The one exception is local_storage: true: traces then persist in your own database and never leave the process, and the dashboard's Interactions and Evaluations views are built from the bodies, so capture_bodies defaults to true there. Set capture_bodies: false explicitly to keep bodies out of locally stored traces too.

Troubleshooting

Traces Not Appearing

Check enabled status:

ruby

puts ActiveAgent::Telemetry.enabled?  # Should be true

Verify configuration:

ruby

puts ActiveAgent::Telemetry.configuration.to_h

Check logs for errors:

ruby

ActiveAgent::Telemetry.configure do |config|
  config.logger = Rails.logger
end

High Memory Usage

Reduce batch size or increase flush frequency:

ruby

ActiveAgent::Telemetry.configure do |config|
  config.batch_size = 25
  config.flush_interval = 2
end

Instrumentation - Local logging with ActiveSupport::Notifications
Usage Statistics - Token usage and cost tracking
Configuration - General framework configuration

Telemetry & Observability ​

Overview ​

Quick Start ​

Hosted Service (ActiveAgents.ai) ​

Self-Hosted ​

Configuration ​

YAML Configuration ​

Programmatic Configuration ​

Rails Configuration ​

Configuration Options ​

Trace Structure ​

Span Types ​

Manual Tracing ​

Trace Block ​

Sampling ​

Flushing & Shutdown ​

Rails Integration ​

Self-Hosting ​

Endpoint Requirements ​

Example Rails Endpoint ​

Comparison with Instrumentation ​

Security Considerations ​

Sensitive Data ​

Message Bodies ​

Troubleshooting ​

Traces Not Appearing ​

High Memory Usage ​

Related Documentation ​

Telemetry & Observability

Overview

Quick Start

Hosted Service (ActiveAgents.ai)

Self-Hosted

Configuration

YAML Configuration

Programmatic Configuration

Rails Configuration

Configuration Options

Trace Structure

Span Types

Manual Tracing

Trace Block

Sampling

Flushing & Shutdown

Rails Integration

Self-Hosting

Endpoint Requirements

Example Rails Endpoint

Comparison with Instrumentation

Security Considerations

Sensitive Data

Message Bodies

Troubleshooting

Traces Not Appearing

High Memory Usage

Related Documentation