Publishing an externally executed evaluation

ActiveAgent::Evals::Publisher sends a completed report to a compatible ActiveAgents collector. It does not execute an agent. Keep report publication opt-in: the payload contains the report's prompts, answers and tool results.

require "active_agent/evals"

publisher = ActiveAgent::Evals::Publisher.new(
  endpoint: ENV.fetch("ACTIVEAGENTS_EVALUATIONS_ENDPOINT"),
  api_key: ENV.fetch("ACTIVEAGENTS_API_KEY")
)
receipt = publisher.call(report: report, source: "support-app", agent_name: "SupportBot", suite: "orders")

The run is stored under the report's run_id, which Correlation#with_run records in report.metadata["run_id"] and the publisher reads from there, from a Report or its saved JSON hash alike. Pass run_id: to publish under another one; the keyword wins over the report:

publisher.call(report: report, run_id: "ci-2026-10-21",
  source: "support-app", agent_name: "SupportBot", suite: "orders")

The run_id is read under a String or Symbol key and must be a nonempty string. A report whose metadata carries none, published without the keyword, raises ArgumentError before anything is sent.

Collectors

A collector is either the hosted platform or a Rails app that mounts the actionagent engine. Both take the same envelope and return the same receipt:

Collector	Endpoint	Key
The hosted platform (setup)	https://api.activeagents.ai/v1/evaluations (the publisher's default)	A workspace API key (Settings → API Keys), or the workspace telemetry key
A full install of the actionagent engine (not one generated with --traces-only)	<mount>/api/evaluation_reports, e.g. https://ops.example.com/activeagents/api/evaluation_reports	The install's ActionAgent.ingest_api_key, or the tenant's key on a multi-tenant install

A self-hosted collector authenticates exactly as the engine's trace ingest endpoint does, so an application that already reports traces to a mount publishes its reports there with the same key:

ACTIVEAGENTS_EVALUATIONS_ENDPOINT=https://ops.example.com/activeagents/api/evaluation_reports
ACTIVEAGENTS_API_KEY=<the mount's ingest_api_key>

The engine stores the report as its own evaluation rows, so the dashboard's Evaluations page shows it the way it shows a run the dashboard executed. See Self-hosted collector below.

Publisher.from_env builds the publisher from those variables: ACTIVEAGENTS_EVALUATIONS_ENDPOINT (the hosted platform's endpoint when it is unset or blank), ACTIVEAGENTS_API_KEY (required; unset or blank raises ArgumentError), and ACTIVEAGENTS_DASHBOARD_URL, the address people open the dashboard at when it is not the collector's, which the receipt's url path is resolved against (see The envelope and the receipt). It reads ENV, or the hash it is given.

ActiveAgent::Evals::Publisher.from_env.call(report: report, source: "support-app", agent_name: "SupportBot", suite: "orders")

The envelope and the receipt

The version-1 JSON envelope contains version, run_id, source, agent_name, suite, and report (the existing Report#to_h shape). Authorization is a Bearer API key. Result correlation belongs in each result's metadata: result_id, trace_id, and judge_trace_ids. Run-level judge traces can be retained in report metadata. Applications assign these IDs and propagate the same IDs into telemetry.

Three optional parts of the report say more about the run:

• report.release — { "digest", "revision", "label" }, the release of the agent the run scored (Report.new(release:)). The collector pins the run to that version of the agent, recording the version when it has not seen the digest, so the run's pass rate reads as current or stale against the agent's latest release. A report naming no release stays unrecorded.
• result.judge_usage — what the judge spent on that result, as Result#to_h writes it from the replay's metadata (calls, input_tokens, output_tokens, cost, model, by_kind, source).
• report.judge_usage.run — the judge's calls no result owns (the verdict).

Where a result reports no cost, the collector estimates one from its tokens, else from the trace its trace_id names, and shows it as an estimate; where no judge_usage is reported, the judge traces named by judge_trace_ids are priced instead.

Delivery is synchronous. A valid receipt has HTTP 201 (new) or 200 (identical retry), the same run_id, status: "complete", id, and evaluation_id. Anything else raises Publisher::Error (see Errors and retries). Save the report before calling the publisher so a rejected upload does not discard an expensive evaluation. The publisher does not follow redirects.

A receipt's url is the run's page on the dashboard. A collector sends it as a path beginning with / (the engine's is <mount>/evaluations/:evaluation_id/runs/:id), and call and progress both return that path joined onto the publisher's dashboard_url, by default the endpoint's origin, so https://ops.example.com/activeagents/api/evaluation_reports resolves /activeagents/evaluations/7/runs/42 to https://ops.example.com/activeagents/evaluations/7/runs/42. Give Publisher.new(dashboard_url:) (or ACTIVEAGENTS_DASHBOARD_URL) an HTTP(S) URL when people reach the dashboard at another address than the collector's. Publisher#dashboard_url returns the URL given, and the receipt's path, which already carries the engine's mount, replaces any path on it; a URL carrying credentials, a query or a fragment is refused. A url that is already absolute, or that is neither a path nor absolute (a relative reference without a leading slash, a //host/path reference, or a path that is not a valid URI), is returned as the collector sent it, a receipt without one gets none, and nothing else in the receipt changes.

To retry delivery without repeating model calls, load the saved report JSON and pass that hash as report, with the same run ID (the saved report's metadata carries it) and envelope identities. A compatible collector treats the report as immutable within its authenticated account: identical delivery is idempotent; different content with the same run ID is a conflict. Never change the report's IDs between retries.

Checking the collector before a run

Publisher#verify! asks the collector whether it is up and accepts the key, without storing anything. Call it before an expensive evaluation, so a stopped collector or a refused key fails before the first model call rather than after the last one:

publisher = ActiveAgent::Evals::Publisher.new(
  endpoint: ENV.fetch("ACTIVEAGENTS_EVALUATIONS_ENDPOINT"),
  api_key: ENV.fetch("ACTIVEAGENTS_API_KEY")
)
publisher.verify! # true, or raises Publisher::Error

report = run_the_evaluation
publisher.call(report: report, source: "support-app", agent_name: "SupportBot", suite: "orders")

The check posts an empty JSON object. A compatible collector authenticates the key, parses the body, and refuses it with a 422 whose error names version, as not a version-1 report; that refusal, and only that one, is the ready answer, and verify! returns true. Anything else raises Publisher::Error with the status, detail and guidance a delivery would carry (see the table below): 401 for a refused key, 404 when nothing at the endpoint takes reports, or a retryable delivery failure when the collector cannot be reached. A 422 that says anything else, or a collector that stores the empty object, means the endpoint is not a compatible collector, and verify! raises rather than returning true. The probe is part of the collector contract; see Self-hosted collector.

On a single-tenant install whose mount requires no ingest key, there is no key to prove: a 422 there confirms the collector is reachable and its evaluation store is migrated, and nothing more.

Publisher#endpoint returns the collector URL as a string, for a message that names where a report goes, and Publisher#dashboard_url the address a receipt's url path is resolved against: the dashboard_url given, else the endpoint's origin, except that the hosted collector at api.activeagents.ai resolves to https://activeagents.ai, where the dashboard is signed in to.

Showing a run while it is in progress

Publisher#progress sends the results a run has so far. The collector stores them as a running run under the run_id, and the dashboard follows it on the Evaluations page as it does a run it executes itself. Each later progress call replaces the results the previous one sent, and call with the finished report under the same run_id completes the run:

identities = { run_id: SecureRandom.uuid, source: "support-app", agent_name: "SupportBot", suite: "orders" }
live = publisher.supports_progress?
landed = []

report = ActiveAgent::Evals::Runner.new(
  scenarios: scenarios, models: models, replay: replay, judge: judge,
  on_result: lambda do |result|
    landed << result
    next unless live

    partial = ActiveAgent::Evals::Report.new(results: landed.dup, models: models, metadata: { "run_id" => identities[:run_id] })
    publisher.progress(report: partial, **identities)
  end
).call
publisher.call(report: report, **identities)

progress sends the envelope with "status": "running"; a complete report's envelope carries no status. Its receipt's status is running, or complete when a progress delivery lands after the run finished, which changes nothing. Only the run's first delivery counts against the account's quota and rate limit, so progress is never refused for either once the run is stored. Throttle it anyway: each call uploads every result so far.

A collector that predates progress would store a running envelope as a complete run and then refuse the finished report with 409. So progress sends a report only to a collector whose readiness answer lists "progress" in its capabilities (see Checking the collector), and otherwise raises Publisher::Error without sending anything. supports_progress? asks once, with the probe verify! sends, unless verify! already has.

A run abandoned before its final call stays running on the dashboard.

Errors and retries

Every failure to deliver a report raises Publisher::Error. Arguments the publisher cannot use raise ArgumentError before anything is sent:

• an endpoint that is not an HTTP(S) URL, uses plain HTTP off loopback, or carries credentials, a query or a fragment
• a missing or blank API key, or one with characters other than visible ASCII once surrounding whitespace is stripped
• a dashboard_url that is not an HTTP(S) URL, or carries credentials, a query or a fragment
• a timeout that is not a positive, finite number
• a run_id, source, agent_name or suite that is not a nonblank string, or no run_id at all: none given and none in the report's metadata as a nonblank string
• a report that is nil, an array, or anything else whose to_h does not return a hash

Besides its message, Publisher::Error carries:

Attribute	Value
status	The collector's HTTP status (an Integer) when it rejected the delivery, otherwise nil
detail	The collector's explanation of a rejection, or nil when it gave none
retryable?	true when delivering the same report under the same run_id again may succeed

A rejection's message names the status, the collector's explanation, and what to do next:

Evaluation delivery rejected (HTTP 422: results[0].scenario_key is required); correct what the collector refused before retrying

The explanation is the error string of a JSON object body, the shape a compatible collector returns. The rest of the body is ignored, and a body that is not a JSON object with a string error gives no explanation. Control characters and runs of whitespace become one space, the API key becomes [FILTERED] if the collector echoes it, and the text is cut to 200 characters. The publisher adds nothing from the request, the report or its headers; any report content in the message is what the collector's own text quotes. A collector should name the field it refused and keep report content, such as answers and prompts, out of its error.

A rejection's guidance follows its status:

Status	retryable?	What to do
409	false	The collector already holds a different report under this run_id, so this report can never be delivered under it. Publish the originally saved report, or give a separate run its own run_id.
413	false	The report exceeds the collector's size limit. Publish a smaller selection.
422	false	The collector refused something in the report or envelope, named in detail: a missing or invalid field, or a suite naming an evaluation this report does not belong to. Correct it and deliver again.
429	true	The account is over its quota or rate limit. Retry later with the saved report and the same run_id.
408, 5xx	true	Retry later with the saved report and the same run_id.
401	false	The collector refused the API key. Check the key against the collector's account.
403	false	The account may not store this report until an operator acts, for example on a cap on observed agents, evaluations or scenarios. Resolve that before retrying.
404	false	Nothing at the endpoint takes evaluation reports. Check that it is a collector's /v1/evaluations or <mount>/api/evaluation_reports URL.
415	false	The collector did not receive application/json. Check anything between the publisher and the collector that rewrites the Content-Type.
501	false	The collector has no evaluation store: an install generated with --traces-only, or one not yet migrated.
Other	false	Resolve the cause before retrying.

Failures without a collector rejection have a nil status and detail:

Failure	retryable?
Network failure or timeout	true
Malformed response: a bad status line or header, or corrupt compression	true
Invalid receipt: a 200 or 201 body that is not JSON, or that does not report this run_id as complete	true
Report over the 2 MiB limit	false
Report that cannot be encoded as JSON: invalid UTF-8, NaN or Infinity, or nesting deeper than JSON allows	false

The first three are retryable because the collector may already have stored the report, and an identical delivery with the same run_id does not store it twice. The last two are refused before the report leaves the process.

Only a network failure or timeout keeps the underlying error as the exception's cause. No other Publisher::Error has a cause, so neither the response nor the report reaches a log through the exception chain.

Bound retries: a retryable failure can persist, for example while a collector is down, and a job that re-enqueues without a limit never stops.

begin
  publisher.call(report: saved_report, run_id: run_id, source: "support-app",
    agent_name: "SupportBot", suite: "orders")
rescue ActiveAgent::Evals::Publisher::Error => error
  raise unless error.retryable? && attempt < 5
  PublishEvaluationJob.set(wait: 10.minutes * attempt).perform_later(run_id, attempt: attempt + 1)
end

Limits and endpoints

Reports are limited to 2 MiB per request. Select smaller scenario/model cohorts when needed. HTTPS is required except for loopback development endpoints. Applications should allow an explicitly configured endpoint and enable publishing only after the collector deployment supports this contract. Installing activeagent alone does not add ingestion to a host application; mounting actionagent does.

Self-hosted collector

POST <mount>/api/evaluation_reports is served by ActionAgent::Api::EvaluationReportsController and stored by ActionAgent::EvaluationReportImport. It never executes the reporting application's agent.

Response	When
201	The report was stored. The receipt carries id, evaluation_id, run_id (exactly as sent), status: "complete", duplicate: false and url, the run's dashboard page, sent as the path <mount>/evaluations/:evaluation_id/runs/:id and returned by the publisher as an absolute URL on the dashboard (see The envelope and the receipt).
200	The same report was already stored under this run_id, or the report is this install's own export of the run the run_id names (Exporting a stored run); the receipt names the stored run, with duplicate: true. A retry is answered before the quota and the rate limit are consulted, so a report that used the last unit of either still gets its receipt.
409	A different report is already stored under this run_id.
422	The report is not a valid version-1 report (the error names the field), or the agent already has an evaluation of that name that no report with this source, suite and scope created: publish under another suite or scope.
403	Storing the report needs an operator first: the owner holds as many observed agents as it can (ActionAgent::AgentRegistrar::MAX_OBSERVED_PER_OWNER), the agent holds 100 evaluations, the evaluation would hold more than 2,000 scenarios, or trace_owner_resolver placed the tenant's agent nowhere.
429	A new report the host's quota_checker denied as :evaluation_report, or one past 30 new reports a minute from the key.
413	The body is over 2 MiB, chunked or not; the collector reads no further than that.
415	The body is not declared Content-Type: application/json.
400	The body is not JSON.
401	No key, or the wrong one, when the mount requires one.
501	The install has no evaluation tables (it was generated with --traces-only), or has not run the migrations this collector needs.
503	Another report for the same agent was being stored for too long (MySQL); retry shortly.

Publisher#verify! relies on the 422 row: it posts {}, and a compatible collector answers 422 with an error naming version — after authenticating the key, and without storing, counting or rate-limiting anything. The engine's collector adds "capabilities": ["progress"] to that answer, which is what Publisher#progress checks for. A collector that stores the empty object, or refuses it without naming version, does not support the publisher's readiness check.

A valid report has:

• run_id, source and suite of 1-200 characters, and an agent_name of 2-100, none with a control character (NUL included). A run_id is compared exactly, case and accents included, on every database.
• Scope values (scope, environment, role in the report metadata) of 1-100 letters, digits, spaces or . : / @ _ -, which together with the suite name an evaluation of at most 255 characters.
• One result per scenario, model label and trial (an integer from 1 to 1000; absent or null reads as 1), each label naming one provider/model and no two labels the same one.
• Token counts and durations that fit a 32-bit integer, and a cost below 1,000,000 — in each result and in each judge_usage, whose by_kind counts are non-negative integers.
• A release.digest of 1-64 letters, digits or . _ -, and a revision and label of at most 200.
• Tool calls that are objects with a name, and a verdict and diagnosis in the shapes ActiveAgent::Evals writes, with each result's fault and recommendation equal to its diagnosis's (both absent is fine), as Result#to_h writes them.

NUL characters are removed from every string inside report. An answer is stored up to its first 20,000 bytes, and a prompt, error or recommendation up to its first 65,535, what a MySQL TEXT column holds; the result's scenario snapshot and diagnosis keep the whole text. The run's per-model summary, criterion scores and recommendations are computed from its stored results; the judge's verdict and label are kept as the report gives them, and a report with no judge is recorded as scored on rules.

Where a report lands

Record	Identity
Agent	The observed agent for the envelope's source and agent_name. It is read-only, like the agents trace ingest observes.
Evaluation	That agent's evaluation named for the suite, qualified by the report metadata's scope, environment and role, in that order: orders (eu, support).
Scenarios	One per reported scenario key, updated to the prompt and group the report ran. Scenarios the report did not run are left alone.
Run	One complete run per tenant and run_id, with one scenario result per scenario, model and trial, pinned to the agent version of report.release when the report names one. Each result keeps its metadata (result_id, trace_id, judge_trace_ids) and its judge_usage; report.judge_usage.run is kept on the run.

On a single-tenant install the agent has no owner, as a traced agent has none, and a run_id is unique across the install. On a multi-tenant install the tenant is the account the key names: the agent is owned by whatever ActionAgent.trace_owner_resolver returns for a trace of that account (the account itself when it is unset), so it appears in that tenant's dashboard beside its traced agents, and a run_id is unique within the tenant.

The run cannot be started again from the dashboard, because its agent runs elsewhere. Evaluate again from the application and publish the new run.

Exporting a stored run

GET <mount>/api/evaluation_reports/:run_id, authenticated and scoped to the tenant exactly as the POST is, returns a complete run as the version-1 envelope ActiveAgent::Evals::Publisher sends — built by the same Publisher.envelope — so it can be POSTed to another install's collector, and POSTed back to this one as an identical retry (200 with duplicate: true). :run_id is:

• for a run an application published, its run_id exactly as published;
• for a run the dashboard executed, which has none, the stable dashboard-run-<id>-<created_at as YYYYMMDDhhmmss UTC>. The receipt of a posted-back export names it, and another install stores the run under it.

An export names when the run finished in its report's metadata.finished, and an import dates the run it stores by that time (its creation and completion), so a run moved to another install keeps its place in the run history. A report that names no finish time is dated when it is stored, and a finish time later than that is read as that time. A finished that is not an ISO 8601 time is refused with 422.

The export's identities are the run's: an imported run's source and suite, a dashboard run's agent service (else the host application's name) and evaluation name, and the agent's class name (else its name). The report is rebuilt from what the run recorded, costs as reported with nothing estimated, so the same run always exports the same envelope; its metadata leaves out the run's database id. Responses: 200 with the envelope, 404 for a run_id the key's tenant holds no run under (another tenant's run is a 404), 422 for a run that is not complete or an unknown format, 401 and 501 as for the POST. On a single-tenant install the export also needs ActionAgent.ingest_api_key to be set: a mount with no key still takes reports, but answers the GET with 401, since every stored run would otherwise be readable under its run_id by anyone who reaches the mount. Instead of the envelope, ?format=html returns the run's self-contained report page (text/html) — the page the dashboard's run report serves, following the viewer's light or dark preference, its fix actions linking into the dashboard by absolute path — and ?format=md its Markdown (text/markdown). ?download=1 sends any format as an attachment named <evaluation>-run-<n>.json|html|md.

curl -H "Authorization: Bearer $KEY" \
  https://dashboard.example.com/activeagents/api/evaluation_reports/ci-2026-10-21 > run.json
curl -OJ -H "Authorization: Bearer $KEY" \
  "https://dashboard.example.com/activeagents/api/evaluation_reports/ci-2026-10-21?format=html&download=1"
curl -X POST -H "Authorization: Bearer $OTHER_KEY" -H "Content-Type: application/json" \
  --data @run.json https://other.example.com/activeagents/api/evaluation_reports

The dashboard's run report serves the same envelope to a signed-in session at <mount>/api/evaluations/:id/runs/:run_id/report.json, and ?download=1 saves it (or the .html page, or the .md report) as a file named <evaluation>-run-<n>.<ext>.

From the command line (on a multi-tenant install name the tenant with ACCOUNT_ID=<id>):

# A run by its run_id, or by evaluation and run id; FORMAT=json (default), html or md
bin/rails action_agent:evaluations:export RUN=ci-2026-10-21 FILE=tmp/run.json
bin/rails action_agent:evaluations:export EVALUATION=3 RUN=12 FORMAT=md > tmp/run.md

# Stored through the collector's own import; importing it again is an identical retry
bin/rails action_agent:evaluations:import FILE=tmp/run.json
# => Stored: run ci-2026-10-21 as run #41 of evaluation #7 (orders (eu)), complete

From any machine with the gem, the activeagent CLI does the same over HTTP with the ingest key, and evals runs shows each run's run_id:

activeagent evals report ci-2026-10-21 --format json -o run.json
activeagent evals import run.json --profile other-install
activeagent evals report ci-2026-10-21 --format md | gh pr comment 123 --body-file -

Metering

A host that meters its install answers the new :evaluation_report kind from ActionAgent.quota_checker, which receives the tenant (nil on a single-tenant install). It is asked only for a report that would be stored, never for an identical retry, and a denial is a 429 whose body merges the checker's message or Hash. ActionAgent.usage_recorder is called with :evaluation_report once for each stored report, and not again for a retry. A report post does not call the tenant's increment_telemetry_usage!, which counts trace ingest requests.

ActionAgent.configure do |config|
  config.quota_checker = ->(account, kind) {
    "Evaluation report allowance used up" if kind == :evaluation_report && account&.reports_exhausted?
  }
end