Skip to content

Telemetry & Observability ​

ActiveAgent includes built-in telemetry for collecting and reporting agent traces. Monitor your AI operations, track token usage and costs, and debug generation flows with comprehensive observability.

Overview ​

The telemetry system captures:

  • Generation Traces: Full lifecycle of agent generations
  • Token Usage: Input, output, and thinking tokens per request
  • Tool Calls: Invocations with timing and results
  • Errors: Exceptions with backtraces for debugging
  • Performance Metrics: Response times and latencies

Quick Start ​

Hosted Service (ActiveAgents.ai) ​

The fastest way to get started is with the hosted observability service:

yaml
# config/active_agent.yml
development:
  openai:
    service: "OpenAI"
    access_token: <%= Rails.application.credentials.dig(:openai, :access_token) %>

  telemetry:
    enabled: true
    endpoint: https://api.activeagents.ai/v1/traces
    api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>

Self-Hosted ​

For complete control, add the actionagent gem to an app of your own, mount its engine, and point telemetry at the ingest path — <mount>/api/traces, so /activeagents/api/traces at the default mount (see Self-Hosted Dashboard). Only the app that hosts the dashboard needs that gem; the apps reporting to it need activeagent alone:

yaml
# config/active_agent.yml
production:
  telemetry:
    enabled: true
    endpoint: https://observability.mycompany.com/activeagents/api/traces
    api_key: <%= ENV["TELEMETRY_API_KEY"] %>
    service_name: my-rails-app

The app that mounts the dashboard itself stores its own traces with local_storage: true instead, writing through the engine's trace model with no HTTP involved. That switch needs actionagent installed and migrated — without it, telemetry logs that it has nowhere to write rather than falling back to HTTP.

Configuration ​

YAML Configuration ​

Configure telemetry in your config/active_agent.yml:

yaml
telemetry:
  enabled: true
  endpoint: https://api.activeagents.ai/v1/traces
  api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>
  sample_rate: 1.0        # 1.0 = 100%, 0.5 = 50%
  batch_size: 100         # Traces per batch
  flush_interval: 5       # Seconds between flushes
  service_name: my-app    # Override app name
  capture_bodies: false   # Include request/response bodies
  resource_attributes:    # Custom attributes for all traces
    deployment: production
    team: ai-platform

Programmatic Configuration ​

Configure in an initializer for dynamic settings:

ruby
# config/initializers/active_agent.rb
ActiveAgent::Telemetry.configure do |config|
  config.enabled = Rails.env.production?
  config.endpoint = ENV.fetch("TELEMETRY_ENDPOINT", "https://api.activeagents.ai/v1/traces")
  config.api_key = Rails.application.credentials.dig(:activeagents, :api_key)
  config.sample_rate = 1.0
  config.service_name = Rails.application.class.module_parent_name.underscore
end

Rails Configuration ​

You can also configure via Rails config:

ruby
# config/application.rb
config.active_agent.telemetry = {
  enabled: true,
  endpoint: "https://api.activeagents.ai/v1/traces",
  api_key: Rails.application.credentials.dig(:activeagents, :api_key)
}

Configuration Options ​

OptionTypeDefaultDescription
enabledBooleanfalseEnable telemetry collection
endpointStringhttps://api.activeagents.ai/v1/tracesTelemetry receiver URL
local_storageBooleanfalseStore traces through the mounted dashboard's trace model instead of posting them (requires the actionagent gem)
api_keyStringnilAuthentication token
sample_rateFloat1.0Sampling rate (0.0 - 1.0)
batch_sizeInteger100Traces per batch before flush
flush_intervalInteger5Seconds between auto-flushes
timeoutInteger10HTTP request timeout
capture_bodiesBooleanfalseInclude message bodies
service_nameStringApp nameService identifier
environmentStringRails.envEnvironment name
resource_attributesHash{}Custom trace attributes
redact_attributesArray["password", "secret", ...]Keys to redact

Trace Structure ​

Each trace captures the complete generation lifecycle:

Trace: WeatherAgent.forecast
├── Span: agent.prompt (prompt preparation)
├── Span: llm.generate (API call)
│   ├── tokens: { input: 150, output: 75, total: 225 }
│   └── model: "gpt-4o"
└── Span: tool.get_weather (tool invocation)
    └── duration: 234ms

Span Types ​

TypeDescription
rootRoot span for the entire generation
promptPrompt preparation and rendering
llmLLM API call
toolTool/function invocation
thinkingExtended thinking (Anthropic)
embeddingEmbedding generation
errorError handling

Manual Tracing ​

Add custom spans to your traces:

ruby
class WeatherAgent < ApplicationAgent
  def forecast(location:)
    @location = location

    # Add custom span
    if ActiveAgent::Telemetry.enabled?
      span = ActiveAgent::Telemetry.span("geocode.lookup")
      span.set_attribute("location", location)
      coordinates = geocode_location(location)
      span.finish
    end

    prompt
  end
end

Trace Block ​

Use the trace block for automatic timing and error handling:

ruby
ActiveAgent::Telemetry.trace("custom.operation") do |span|
  span.set_attribute("user_id", current_user.id)
  span.set_attribute("operation", "data_enrichment")

  result = perform_operation

  span.set_tokens(input: 100, output: 50)
  result
end

Sampling ​

Control trace volume with sampling:

ruby
ActiveAgent::Telemetry.configure do |config|
  # Sample 10% of production traffic
  config.sample_rate = Rails.env.production? ? 0.1 : 1.0
end

Sampling is deterministic per-trace, so all spans within a trace are included or excluded together.

Flushing & Shutdown ​

Traces are batched and sent asynchronously. Force flush when needed:

ruby
# Flush buffered traces immediately
ActiveAgent::Telemetry.flush

# Graceful shutdown (flush and wait)
ActiveAgent::Telemetry.shutdown

Rails Integration ​

Telemetry automatically flushes on Rails shutdown:

ruby
# config/initializers/active_agent.rb
at_exit { ActiveAgent::Telemetry.shutdown }

Self-Hosting ​

The actionagent dashboard engine already implements this endpoint spec: add the gem, mount the engine, and its ingest API (<mount>/api/traces) receives traces from any app in your fleet — see Self-Hosted Observability. The activeagent command line reads a stored trace back out of that dashboard as the same payload and posts it to another (traces export, traces import; see Command Line). The requirements below are for building your own receiver instead.

Endpoint Requirements ​

Your telemetry endpoint must accept POST requests with:

Headers:

  • Content-Type: application/json
  • Authorization: Bearer <api_key>
  • X-Service-Name: <service_name>
  • X-Environment: <environment>

Payload:

json
{
  "traces": [
    {
      "trace_id": "abc123...",
      "service_name": "my-app",
      "environment": "production",
      "timestamp": "2024-01-15T10:30:00.123456Z",
      "resource_attributes": {},
      "spans": [
        {
          "span_id": "def456",
          "trace_id": "abc123...",
          "parent_span_id": null,
          "name": "WeatherAgent.forecast",
          "type": "root",
          "start_time": "2024-01-15T10:30:00.123456Z",
          "end_time": "2024-01-15T10:30:01.234567Z",
          "duration_ms": 1111.11,
          "status": "OK",
          "attributes": {
            "agent.class": "WeatherAgent",
            "agent.action": "forecast"
          },
          "tokens": {
            "input": 150,
            "output": 75,
            "total": 225
          },
          "events": []
        }
      ]
    }
  ],
  "sdk": {
    "name": "activeagent",
    "version": "0.5.0",
    "language": "ruby",
    "runtime_version": "3.3.0"
  }
}

Example Rails Endpoint ​

ruby
# app/controllers/api/traces_controller.rb
class Api::TracesController < ApplicationController
  skip_before_action :verify_authenticity_token
  before_action :authenticate_api_key!

  def create
    traces = params[:traces]

    traces.each do |trace|
      Trace.create!(
        trace_id: trace[:trace_id],
        service_name: trace[:service_name],
        environment: trace[:environment],
        spans: trace[:spans],
        timestamp: trace[:timestamp]
      )
    end

    head :accepted
  end

  private

  def authenticate_api_key!
    api_key = request.headers["Authorization"]&.gsub(/^Bearer /, "")
    head :unauthorized unless ApiKey.exists?(key: api_key)
  end
end

Comparison with Instrumentation ​

ActiveAgent provides two complementary observability systems:

FeatureInstrumentationTelemetry
PurposeLocal logging & metricsDistributed tracing
TransportActiveSupport::NotificationsHTTP POST
DestinationRails logs, local metricsRemote endpoint
Use CaseDebugging, local monitoringProduction observability
OverheadMinimalAsync, batched

Use Instrumentation for local development and debugging. Use Telemetry for production observability and analytics.

Security Considerations ​

Sensitive Data ​

By default, telemetry redacts common sensitive attributes:

  • password, secret, token, key, credential, api_key

Add custom redactions:

ruby
ActiveAgent::Telemetry.configure do |config|
  config.redact_attributes += ["ssn", "credit_card"]
end

Message Bodies ​

Message bodies — the rendered system prompt (prompt.input.instructions), the outbound messages (prompt.input.messages), the completion (llm.output.message) and tool arguments and results (tool.input.args, tool.output.result) — are not captured by default when traces leave the process. Counts, tool names, token usage and finish reasons are always recorded. Enable bodies with caution:

ruby
ActiveAgent::Telemetry.configure do |config|
  config.capture_bodies = true  # Only in controlled environments
end

The one exception is local_storage: true: traces then persist in your own database and never leave the process, and the dashboard's Interactions and Evaluations views are built from the bodies, so capture_bodies defaults to true there. Set capture_bodies: false explicitly to keep bodies out of locally stored traces too.

Troubleshooting ​

Traces Not Appearing ​

  1. Check enabled status:

    ruby
    puts ActiveAgent::Telemetry.enabled?  # Should be true
  2. Verify configuration:

    ruby
    puts ActiveAgent::Telemetry.configuration.to_h
  3. Check logs for errors:

    ruby
    ActiveAgent::Telemetry.configure do |config|
      config.logger = Rails.logger
    end

High Memory Usage ​

Reduce batch size or increase flush frequency:

ruby
ActiveAgent::Telemetry.configure do |config|
  config.batch_size = 25
  config.flush_interval = 2
end

User feedback ​

Collect thumbs up/down with optional comments and review votes in the agent’s Feedback tab. See User feedback for submission, authentication, upgrade instructions and dashboard filters.

Trace window totals and capped details ​

The dashboard's Traces view computes totals and chart buckets over the full selected time window, scoped to the caller and the active server filters. The detail list returns the newest 500 traces by default (at most 1000). When that list is truncated, the view says how many traces it loaded; its lists, agent/action breakdowns and span details describe those loaded rows. Clicking a chart bucket filters the loaded rows, not a second complete page.

GET <mount>/api/traces includes total_count, returned_count, truncated, summary (requests, errors, average recorded latency and requests per minute), and zero-filled buckets. The bucket_seconds parameter selects the chart interval, bounded to at most 180 buckets. window_start_ms and window_end_ms pin the window to the server clock; its start is inclusive and its end is exclusive. A missing duration does not count as zero when averaging latency.