Skip to main content
The Observability service records a trace of every inference request — how long it took, what it cost, which model and provider served it, and which pipeline stages ran — so you can see exactly what your application is doing and where time and credits go. You will find Observability under Services > Observability.

​
Turning Observability on

Use the Enabled switch in the top right of the page. It takes effect the moment you flip it — there is no separate save step. While it is off, no traces are recorded and nothing is metered. Turning it on starts recording from the next request; it never changes how your requests are served.

​
How recording works

When the service is on, each request to the inference API produces one trace with a span for every stage of the pipeline that ran: cache, Recall, guard, the PII firewall, routing and inference, and PII restore. Each span is timed individually, so a trace is a true breakdown of where the request spent its time. Recording happens inside the request, not on a background queue. For a normal response, the trace is written just before the response is returned, so it costs a small amount of the time your request takes, and it exists the instant you have your answer. For a streamed response, it is written just after the last chunk instead, so it never delays the first byte. Either way, a recording failure can never cost you the completion: it is logged and swallowed, and your response is returned regardless. Failed requests are traced too: a guard block, a PII block, an out-of-credits error, or a provider error each produce a trace stamped with the matching status, so you can see failures as clearly as successes.

​
The Traces tab

The Traces tab lists every recorded request, newest first. Each row shows the model, provider, any tags you sent, status, latency, token count, cost in credits, and when it ran. You can filter by model, provider, status, end-user, conversation thread, and time range, and sort by any column. Click a row to open the trace beside the table. It shows what you labelled the request, what we measured about it, and a span waterfall. The waterfall lays every pipeline stage on one shared axis, so each bar starts where the previous one ended and its length is that stage’s share of the whole request. Failed stages are red. That makes it obvious not just which stage was slowest, but where the time went: if the bars stop short of the right edge, the remaining time was spent outside every measured stage. The panel has a few controls in its header:
  • Up and down, or the K and J keys, step through the traces on the current page without closing the panel.
  • Expand widens the panel to the full width of the page, for a longer time axis.
  • Open in a new tab takes you to the trace’s own page.

​
What each stage was given

The trace’s own page also shows the payloads: what each pipeline stage was handed and what it produced. That is the difference between knowing a request took 900 ms and knowing what it actually asked. What is stored depends on whether the PII firewall ran: You can turn capture off per workspace, under Observability. Timings, cost and status are unaffected, and traces already captured are left alone: clearing those is what the retention window and a memory purge are for. With the firewall on, what it replaced is stored replaced. The firewall runs third of five stages, so retrieval and the guard scan see your content before it has been pseudonymized. They are captured all the same, and pseudonymized before they are written, so what is stored reads Invoice for <PERSON_1> at <EMAIL_ADDRESS_1> rather than the names. The restore stage, whose whole job is putting the real values back, stores what it was given and never what it produced. The token map that would reverse those placeholders is held in memory for the length of the request and is never written down, so nothing in your trace history can be turned back into personal data. Three limits on that, stated rather than left to be discovered:
  • A model’s answer is its own. The firewall pseudonymizes what you send and restores placeholders in what comes back; it does not scan completions for personal data the model introduced. A completion is stored as the model wrote it, which is the same text the API returned to you.
  • Tool content is only covered if you asked for it to be. When firewall_tool_content is off for a workspace, the firewall skips tool results and tool-call arguments, so traces omit them rather than store them unprotected. You will see [not captured: tool content is not firewalled for this workspace] in their place.
  • The route stage stores the pseudonymized prompt, even on an EU-native route where the real values are sent in the clear. Storing what was literally on the wire would mean storing those values, which is the one thing this design will not do.
Above the payloads, the firewall panel shows what was replaced: how many of each entity type, and the placeholders standing where each value was. It shows the shape of the request without showing one protected value. Payloads are capped at 64 KB each. Anything longer is stored cut, and says so: the trace shows “Showing 64 KB of 340 KB” rather than presenting a fragment as the whole thing. Payloads are kept for as long as the trace they belong to and are removed with it. There is no extra charge for capturing them.

​
A trace on its own page

Every trace has a permanent URL of the form /services/observability/traces/{id}. It shows everything the panel shows plus the captured payloads, so it is the link to paste into a ticket or send to a colleague. The panel beside the table links here rather than carrying the payloads itself, so listing traces never loads content it does not display. Anyone you send it to needs access to the same workspace: a trace from another workspace or another organization returns “not found” rather than telling them it exists.

​
Labelling a request

A trace can carry your own identifiers, so the explorer can answer questions about your application rather than only about ours. All four are optional, none of them change how the request is served, and none of them are charged:
  • user — your own end-user id, the same field OpenAI uses. Filter the explorer by it to see everything one of your users did. (It also scopes long-term memory; see Recall.)
  • thread — the conversation. Clicking a thread in the drawer filters the table to that whole conversation. A thread is what Langfuse calls a session and what OpenTelemetry calls a conversation.
  • name — a human label for this request, up to 120 characters.
  • tags — up to 10 labels of up to 64 characters, shown as chips on the trace.

​
Correlating with your own tracing

If your application sends a W3C traceparent header (every OpenTelemetry SDK does), the trace id and the incoming span id are recorded on the trace and shown in the drawer, so you can line an Akumi trace up with the same request in your own APM. Nothing is required of you here, and a traceparent we cannot parse is ignored rather than rejected: a malformed tracing header never affects your inference call. Akumi does not send a traceparent on to model providers. Successful completions also carry an X-Request-Id header naming the platform’s own request id, the same id you use to score that request afterward.

​
The Metrics tab

The Metrics tab aggregates your traces, and your total workspace spend, over a window you choose (last 24 hours, last 7 days, or last 30 days). It shows:
  • Headline figures: total requests, error rate, total workspace spend for the window (every billed usage record, not only the requests shown here, so it also carries charges with no trace at all, such as a Recall ingest), and p50 / p95 latency.
  • Request volume over time: how traffic rose and fell across the window.
  • Latency over time: p50 and p95 total_latency_ms per bucket, computed over successful requests only, so a guard block or an egress refusal never distorts it, and you can spot slow periods.
  • By model and by provider: which models and providers carried the most traffic.
  • Score averages: each score name’s plain average and how many scores it is drawn from, empty until your application sends its first score.
The window switches the whole tab; reads are free, so you can slice the data as often as you like.

​
Scores

A score is a named judgment you attach to a request after the fact: a thumbs up or down, a 1-5 accuracy rating, a category label, whatever your application, or a person reviewing its output, decides. It answers a question a trace alone cannot: not just what a request cost and how long it took, but whether the answer was any good. Send it to POST /v1/scores:
  • request_id says which request you are judging. You get it from the X-Request-Id header on the original completion response (see Correlating with your own tracing above), or it is whatever you already sent as your own Idempotency-Key request header, if you set one yourself. If you set your own, make it unique per request: it is also the key billing dedupes on, so reusing one is treated as a replay of the first request rather than a new one, and every later request carrying it is matched to the existing usage record instead of being debited.
  • name identifies the question you are answering, for example accuracy or user-feedback: lowercase letters, numbers and hyphens.
  • Send exactly one of value (a number, so a thumbs up or down travels as 1 or 0) or value_string (a short label, for a category).
  • comment is optional free text.
Sending the same name again for the same request replaces the score outright, comment included. Re-sending a name replaces, so retries are safe and changed minds win: post accuracy: 1 today and accuracy: 0 tomorrow and the stored row reads 0. That also means a resend which omits comment clears whatever comment was there before. A resend is always a full replacement of the score, never a partial patch, so include the comment again on every call if you want it kept. Scores are free: recording one never debits your credit balance and never appears as a usage line. They live and die with the trace they judge, pruned together on the same retention window, with no separate schedule of their own. A sampled-out request cannot be scored. Sampling means most requests are never written to a trace at all, so there is nothing for POST /v1/scores to attach to, and the call returns a 404. If you plan to score requests, score-collecting workloads should sample at 100%. Failures and requests routed to an external, non-EU provider are always recorded and metered whatever the sample rate (see Sampling below), so those stay scoreable no matter how low the rate is set. A failed request’s error response does not carry an X-Request-Id header, though: supply your own Idempotency-Key on the original call if you want to score it afterward. A score’s comment is recorded exactly as you send it and is never pseudonymized, so do not put personal data in it.

​
Evaluators

An evaluator is a saved judge or check, configured under the Evaluators tab, that scores your traces automatically so quality scoring does not depend on your own code calling POST /v1/scores after every request. Evaluators belong to one workspace, and come in two kinds:
  • LLM judges ask a model to grade a trace against a prompt: one of five starter templates (hallucination, answer relevance, tone, context relevance, and context faithfulness) to use as written or adapt, or a prompt you write yourself. Judge prompts use at least one of four placeholders: {{input}} and {{output}} are filled from the trace’s captured prompt and answer, and {{query}} and {{context}} are filled from the retrieval stage’s captured query and retrieved items, so retrieval quality is judgeable separately from answer quality.
  • Checks are deterministic: is_json, contains, not_contains, regex, and length_between. A check makes no model call, and reads only the trace’s captured output, never its input. is_json accepts any syntactically valid JSON, including a bare value such as 42 or "text", not only an object or an array.
Every evaluator names the score it writes, and that name is unique across your workspace’s evaluators: you cannot save two that would write the same one. Whichever kind wrote it, the score lands on the same surface Scores writes to, with source recorded as judge or check so you can tell an automatic score from one your own application sent (source: api). A judge’s reasoning is written into that score’s comment, the same field a score you send yourself uses, and shown the same way: hover the score to read it.

​
Running continuously, or on demand

Every enabled evaluator rolls its own sample rate as each successful request is recorded, on top of your trace sample rate rather than instead of it: the two multiply. A workspace recording half its traffic, with an evaluator set to half its own rate, evaluates about one request in four, not one in two. Sample at 100% while you are actively building or trusting an evaluator; otherwise its own rate only thins an already-thinned stream further. The roll and the enqueue happen inline, but the evaluation itself, the judge call or the check, runs in a background job, so a score can land after you already have your response, not before it. Each evaluator also has a Backfill action, for scoring what already happened rather than waiting for new traffic. One run queues every one of the workspace’s successful traces from the last 7 days that this evaluator has not scored yet and can score honestly, capped at 500 traces. You can run it once a minute per evaluator; a click before that just refuses rather than queueing the same traces twice.

​
What gets skipped, and why

An evaluator that cannot judge a trace honestly writes nothing, rather than guess. From the outside, you will see:
  • A truncated or omitted payload is never judged. A score built on a fragment, or on content that was never captured for a policy reason, is worse than no score, so the evaluator skips it instead.
  • A trace without a retrieval stage, or whose retrieval found nothing, is never scored by a retrieval evaluator. Context relevance and context faithfulness read the query and items the retrieval stage captured. A trace that never called retrieval, or whose retrieval came back empty, gives them nothing honest to judge, so they skip it instead of scoring an empty result as good or bad.
  • Payload capture has to be on. With it off, there is nothing for a judge or a check to read, so nothing runs.
  • A sampled-out request has no trace, so there is nothing to evaluate, the same reason it cannot be scored by hand either (see Scores above).
  • A trace this evaluator already scored is never re-judged. One evaluator writes at most one score per trace; a repeated backfill, or a continuous roll that happens to land twice, finds that score already there and skips rather than overwrite it or bill it again.
  • An empty credit balance, or a workspace usage cap already in force, skips LLM judging. Checks make no model call, so they keep running regardless: only the judge’s own spend is gated on your balance.

​
The judge model

Judges run on one EU-resident model the platform chooses, in the Mistral family today, not one you pick: there is no judge model selector. If that configured model is ever unavailable in an EU-resident form, a check made at call time catches it before anything is sent, and the evaluation is skipped rather than quietly sent somewhere else.

​
Pricing

An LLM evaluation that reaches the judge is metered on observability.judge: the judge call’s own prompt and completion tokens, at the same per-token rates a chat completion on that model would pay, plus a small fixed amount per evaluation. Nothing is billed for a skipped evaluation, whichever of the rules above caused the skip. Checks make no model call, so a check evaluator is always free, the same as a score you send yourself through POST /v1/scores. Billing never repeats either: an evaluator charges at most once per trace, permanently, for the same reason it never re-judges one (see above).

​
Seeing what a request cost

You do not need Observability, or the dashboard at all, to see what a single request was charged. Every chat completion returns the breakdown in its usage object:
base is the inference charge. services is everything else billed against that same request, such as retrieval, the PII firewall or a guard check. total is the sum, and it matches what your ledger was debited, so you can reconcile against it directly. The trace shows the same total. Hover its Cost chip and it breaks down meter by meter — inference, retrieval, a guard check, the trace recording itself — so you can see which part of a request was expensive rather than only what it came to. The lines are in credits, since a single credit is a tenth of a cent and rounding each line to cents would print € 0,00 beside charges that were real; the total carries both. Streamed responses carry the same object. It arrives on the finishing chunk, and also in a trailing chunk with an empty choices array if you set stream_options: {"include_usage": true}. Observability adds the dimension usage cannot: where the time went inside a single request, and how cost moves across many of them.

​
Metering and retention

Each recorded trace is a single, metered event (observability.trace) and appears in your usage. Reading traces and dashboards is always free. If a request is not recorded (because the service was off, or because sampling skipped it), it is never charged. Traces, their spans and any captured payloads are kept for a retention window set by your plan, and pruned together automatically after it, the same way audit logs are. Free and Pay as you go keep 7 days, Team keeps 30 days, Business keeps 90 days, and Enterprise keeps 365 days. A workspace can set a shorter window than its plan allows, and can turn payload capture off entirely.

​
Sampling

High-volume tenants can record a share of requests instead of every one. At a 10% rate, one EU-resident successful request in ten is recorded, and most of the other nine leave no record at all. That changes what is recorded, not what happens. Every request is still served, still billed for inference, and still on your invoice. Sampling only reduces trace storage and the per-trace observability.trace charge for the EU-resident successes it skips; every other request category, described below, is always recorded and metered exactly the same as any other trace. Two kinds of request are always recorded, whatever the rate, and both are still metered per trace. A guard block, a PII block, an out-of-credits error and a provider error each produce a trace regardless of the sample rate, since those are exactly the requests you are most likely to come looking for. So does a request routed to an external, non-EU provider: it is what a compliance record has to list, and the residency report and Article 30 ROPA are built by reading those traces, so a real transfer must never be the one row a low sample rate happened to drop. A workspace that routes externally at a low rate will see its observability.trace charges run higher than the sample rate alone predicts, since every external request contributes one regardless of the roll. The cost is real: the one EU-resident successful request someone asks you about probably is not in the explorer. On the Metrics tab, request counts become estimates scaled up from the sample; spend and error counts stay exact, because billing and errors are never sampled out. Set the rate from the menu next to the Enabled switch, in the top right of the page: presets run from 100% down to 1%, and sampling is available on every plan.

​
Good to know

Observability is metered per recorded trace. With the service on, every served request that is recorded produces one billable trace event. See Sampling for what that means below 100%. Reads are free. Recording adds a small amount of latency, but only for non-streamed requests. The trace is written before a normal response is returned. For a streamed response, it is written after the last chunk, so it never delays the first byte. Either way, if recording fails, it is logged and swallowed: your response is returned regardless. Payloads are captured by default, pseudonymized wherever the firewall runs. See What each stage was given below, and turn it off per workspace if you would rather keep only the timings. The identifiers you attach with user, thread, name and tags are recorded exactly as you send them and are never pseudonymized, so do not put personal data in them. Clearing a workspace’s memory unlinks its traces, deletes their payloads, and clears score comments. Clearing Recall’s stored memory for a workspace removes the user and thread identifiers from that workspace’s traces, deletes every payload captured on them, and clears the comment on every score attached to them. The traces and their scores both survive, so your cost, latency and error history stays intact, and so does every rating and label; only the identifiers, the payload content and the score comments are gone. Captured payloads are in your data export. They come out under type: "payload", with the stage that produced each one and whether it was firewalled. Scores are in your data export too. They come out under type: "score", with the trace they judge, the name, the value, and the comment.