Telemetry & Observability
ActiveAgent includes built-in telemetry for collecting and reporting agent traces. Monitor your AI operations, track token usage and costs, and debug generation flows with comprehensive observability.
Overview
The telemetry system captures:
- Generation Traces: Full lifecycle of agent generations
- Token Usage: Input, output, and thinking tokens per request
- Tool Calls: Invocations with timing and results
- Errors: Exceptions with backtraces for debugging
- Performance Metrics: Response times and latencies
Quick Start
Hosted Service (ActiveAgents.ai)
The fastest way to get started is with the hosted observability service:
# config/active_agent.yml
development:
openai:
service: "OpenAI"
access_token: <%= Rails.application.credentials.dig(:openai, :access_token) %>
telemetry:
enabled: true
endpoint: https://api.activeagents.ai/v1/traces
api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>Self-Hosted
For complete control, add the actionagent gem to an app of your own, mount its engine, and point telemetry at the ingest path — <mount>/api/traces, so /activeagents/api/traces at the default mount (see Self-Hosted Dashboard). Only the app that hosts the dashboard needs that gem; the apps reporting to it need activeagent alone:
# config/active_agent.yml
production:
telemetry:
enabled: true
endpoint: https://observability.mycompany.com/activeagents/api/traces
api_key: <%= ENV["TELEMETRY_API_KEY"] %>
service_name: my-rails-appThe app that mounts the dashboard itself stores its own traces with local_storage: true instead, writing through the engine's trace model with no HTTP involved. That switch needs actionagent installed and migrated — without it, telemetry logs that it has nowhere to write rather than falling back to HTTP.
Configuration
YAML Configuration
Configure telemetry in your config/active_agent.yml:
telemetry:
enabled: true
endpoint: https://api.activeagents.ai/v1/traces
api_key: <%= Rails.application.credentials.dig(:activeagents, :api_key) %>
sample_rate: 1.0 # 1.0 = 100%, 0.5 = 50%
batch_size: 100 # Traces per batch
flush_interval: 5 # Seconds between flushes
service_name: my-app # Override app name
capture_bodies: false # Include request/response bodies
resource_attributes: # Custom attributes for all traces
deployment: production
team: ai-platformProgrammatic Configuration
Configure in an initializer for dynamic settings:
# config/initializers/active_agent.rb
ActiveAgent::Telemetry.configure do |config|
config.enabled = Rails.env.production?
config.endpoint = ENV.fetch("TELEMETRY_ENDPOINT", "https://api.activeagents.ai/v1/traces")
config.api_key = Rails.application.credentials.dig(:activeagents, :api_key)
config.sample_rate = 1.0
config.service_name = Rails.application.class.module_parent_name.underscore
endRails Configuration
You can also configure via Rails config:
# config/application.rb
config.active_agent.telemetry = {
enabled: true,
endpoint: "https://api.activeagents.ai/v1/traces",
api_key: Rails.application.credentials.dig(:activeagents, :api_key)
}Configuration Options
| Option | Type | Default | Description |
|---|---|---|---|
enabled | Boolean | false | Enable telemetry collection |
endpoint | String | https://api.activeagents.ai/v1/traces | Telemetry receiver URL |
local_storage | Boolean | false | Store traces through the mounted dashboard's trace model instead of posting them (requires the actionagent gem) |
api_key | String | nil | Authentication token |
sample_rate | Float | 1.0 | Sampling rate (0.0 - 1.0) |
batch_size | Integer | 100 | Traces per batch before flush |
flush_interval | Integer | 5 | Seconds between auto-flushes |
timeout | Integer | 10 | HTTP request timeout |
capture_bodies | Boolean | false | Include message bodies |
service_name | String | App name | Service identifier |
environment | String | Rails.env | Environment name |
resource_attributes | Hash | {} | Custom trace attributes |
redact_attributes | Array | ["password", "secret", ...] | Keys to redact |
Trace Structure
Each trace captures the complete generation lifecycle:
Trace: WeatherAgent.forecast
├── Span: agent.prompt (prompt preparation)
├── Span: llm.generate (API call)
│ ├── tokens: { input: 150, output: 75, total: 225 }
│ └── model: "gpt-4o"
└── Span: tool.get_weather (tool invocation)
└── duration: 234msSpan Types
| Type | Description |
|---|---|
root | Root span for the entire generation |
prompt | Prompt preparation and rendering |
llm | LLM API call |
tool | Tool/function invocation |
thinking | Extended thinking (Anthropic) |
embedding | Embedding generation |
error | Error handling |
Manual Tracing
Add custom spans to your traces:
class WeatherAgent < ApplicationAgent
def forecast(location:)
@location = location
# Add custom span
if ActiveAgent::Telemetry.enabled?
span = ActiveAgent::Telemetry.span("geocode.lookup")
span.set_attribute("location", location)
coordinates = geocode_location(location)
span.finish
end
prompt
end
endTrace Block
Use the trace block for automatic timing and error handling:
ActiveAgent::Telemetry.trace("custom.operation") do |span|
span.set_attribute("user_id", current_user.id)
span.set_attribute("operation", "data_enrichment")
result = perform_operation
span.set_tokens(input: 100, output: 50)
result
endSampling
Control trace volume with sampling:
ActiveAgent::Telemetry.configure do |config|
# Sample 10% of production traffic
config.sample_rate = Rails.env.production? ? 0.1 : 1.0
endSampling is deterministic per-trace, so all spans within a trace are included or excluded together.
Flushing & Shutdown
Traces are batched and sent asynchronously. Force flush when needed:
# Flush buffered traces immediately
ActiveAgent::Telemetry.flush
# Graceful shutdown (flush and wait)
ActiveAgent::Telemetry.shutdownRails Integration
Telemetry automatically flushes on Rails shutdown:
# config/initializers/active_agent.rb
at_exit { ActiveAgent::Telemetry.shutdown }Self-Hosting
The actionagent dashboard engine already implements this endpoint spec: add the gem, mount the engine, and its ingest API (<mount>/api/traces) receives traces from any app in your fleet — see Self-Hosted Observability. The activeagent command line reads a stored trace back out of that dashboard as the same payload and posts it to another (traces export, traces import; see Command Line). The requirements below are for building your own receiver instead.
Endpoint Requirements
Your telemetry endpoint must accept POST requests with:
Headers:
Content-Type: application/jsonAuthorization: Bearer <api_key>X-Service-Name: <service_name>X-Environment: <environment>
Payload:
{
"traces": [
{
"trace_id": "abc123...",
"service_name": "my-app",
"environment": "production",
"timestamp": "2024-01-15T10:30:00.123456Z",
"resource_attributes": {},
"spans": [
{
"span_id": "def456",
"trace_id": "abc123...",
"parent_span_id": null,
"name": "WeatherAgent.forecast",
"type": "root",
"start_time": "2024-01-15T10:30:00.123456Z",
"end_time": "2024-01-15T10:30:01.234567Z",
"duration_ms": 1111.11,
"status": "OK",
"attributes": {
"agent.class": "WeatherAgent",
"agent.action": "forecast"
},
"tokens": {
"input": 150,
"output": 75,
"total": 225
},
"events": []
}
]
}
],
"sdk": {
"name": "activeagent",
"version": "0.5.0",
"language": "ruby",
"runtime_version": "3.3.0"
}
}Example Rails Endpoint
# app/controllers/api/traces_controller.rb
class Api::TracesController < ApplicationController
skip_before_action :verify_authenticity_token
before_action :authenticate_api_key!
def create
traces = params[:traces]
traces.each do |trace|
Trace.create!(
trace_id: trace[:trace_id],
service_name: trace[:service_name],
environment: trace[:environment],
spans: trace[:spans],
timestamp: trace[:timestamp]
)
end
head :accepted
end
private
def authenticate_api_key!
api_key = request.headers["Authorization"]&.gsub(/^Bearer /, "")
head :unauthorized unless ApiKey.exists?(key: api_key)
end
endComparison with Instrumentation
ActiveAgent provides two complementary observability systems:
| Feature | Instrumentation | Telemetry |
|---|---|---|
| Purpose | Local logging & metrics | Distributed tracing |
| Transport | ActiveSupport::Notifications | HTTP POST |
| Destination | Rails logs, local metrics | Remote endpoint |
| Use Case | Debugging, local monitoring | Production observability |
| Overhead | Minimal | Async, batched |
Use Instrumentation for local development and debugging. Use Telemetry for production observability and analytics.
Security Considerations
Sensitive Data
By default, telemetry redacts common sensitive attributes:
password,secret,token,key,credential,api_key
Add custom redactions:
ActiveAgent::Telemetry.configure do |config|
config.redact_attributes += ["ssn", "credit_card"]
endMessage Bodies
Message bodies — the rendered system prompt (prompt.input.instructions), the outbound messages (prompt.input.messages), the completion (llm.output.message) and tool arguments and results (tool.input.args, tool.output.result) — are not captured by default when traces leave the process. Counts, tool names, token usage and finish reasons are always recorded. Enable bodies with caution:
ActiveAgent::Telemetry.configure do |config|
config.capture_bodies = true # Only in controlled environments
endThe one exception is local_storage: true: traces then persist in your own database and never leave the process, and the dashboard's Interactions and Evaluations views are built from the bodies, so capture_bodies defaults to true there. Set capture_bodies: false explicitly to keep bodies out of locally stored traces too.
Troubleshooting
Traces Not Appearing
Check enabled status:
rubyputs ActiveAgent::Telemetry.enabled? # Should be trueVerify configuration:
rubyputs ActiveAgent::Telemetry.configuration.to_hCheck logs for errors:
rubyActiveAgent::Telemetry.configure do |config| config.logger = Rails.logger end
High Memory Usage
Reduce batch size or increase flush frequency:
ActiveAgent::Telemetry.configure do |config|
config.batch_size = 25
config.flush_interval = 2
endRelated Documentation
- Instrumentation - Local logging with ActiveSupport::Notifications
- Usage Statistics - Token usage and cost tracking
- Configuration - General framework configuration
User feedback
Collect thumbs up/down with optional comments and review votes in the agent’s Feedback tab. See User feedback for submission, authentication, upgrade instructions and dashboard filters.
Trace window totals and capped details
The dashboard's Traces view computes totals and chart buckets over the full selected time window, scoped to the caller and the active server filters. The detail list returns the newest 500 traces by default (at most 1000). When that list is truncated, the view says how many traces it loaded; its lists, agent/action breakdowns and span details describe those loaded rows. Clicking a chart bucket filters the loaded rows, not a second complete page.
GET <mount>/api/traces includes total_count, returned_count, truncated, summary (requests, errors, average recorded latency and requests per minute), and zero-filled buckets. The bucket_seconds parameter selects the chart interval, bounded to at most 180 buckets. window_start_ms and window_end_ms pin the window to the server clock; its start is inclusive and its end is exclusive. A missing duration does not count as zero when averaging latency.