Telemetry
observability the system emits and how the export pipeline is shaped. Also relevant to
integrators who want to understand the trace structure their calls appear in.
What Temper emits
1.
configure. Every request to the API or MCP server produces a root span; work within that
request produces child spans.
2.
(the CLI). Logs and traces are independent layers: raising the log level does not widen
what is exported, and silencing logs does not stop trace export.
Prometheus view that a backend's metrics-generator produces from the spans Temper exports.
RED panels and alert rules read that derived view, not a native metrics stream.
The OTLP export model
over the public internet, to whichever OTLP/HTTP-compatible backend you point it at — Grafana
Cloud, Honeycomb, Dash0, Axiom, and others all expose OTLP/HTTP ingest on the public internet.
POSTs directly to the vendor's ingest URL.
URL and auth header — both read from spec-standard OpenTelemetry environment variables. No
vendor name appears in Temper's code.
mishandled by some collectors and surfaces as wrong timestamps rather than an error.
its in-dashboard trace view through a JavaScript hook a Rust process cannot reach. What that
costs is the platform's in-dashboard view of Temper's spans (and the platform paying the
egress). What it does not cost is the trace itself — a span POSTed from a Rust function to your
backend is an ordinary span in your backend.
The trace structure
properties of that structure are architectural decisions, not accidents.
traceparent. A trustedcaller's trace is joined with an OpenTelemetry span link recorded after authentication —
where trusted means the request passed an authentication gate (a verified JWT or an HMAC
signature over the body keyed on a secret only Temper's own services hold). So a linked trace
in your backend is always a caller that authenticated; an anonymous request carries the inbound
trace ids as inert log fields and joins nothing. Every trace worth joining is one Temper sent
itself, so refusing everything else costs nothing.
sampled flag is recorded, never obeyed. Honoring it would let anyone set thesampled bit on flood traffic and bill you for exporting every span of it.
sandbox after a response rather than exiting the process, so a batch-export timer may never
fire — spans queued at freeze are lost silently. Temper exports on an explicit flush at the
response seam instead of trusting a timer. That flush is a real network round trip, bounded at
500ms, run on a blocking thread so it cannot stall an async runtime, and single-flight so
concurrent flushes do not each pay the sum of those ahead. Past the budget the response goes out
and the span rides the next flush or is lost. Losing a span is the right trade against stalling
a request.
flush_ms, a field separate from latency_ms. The flush canonly happen after the request span closes, so it is genuinely not part of the span it flushes.
A caller's observed latency is
latency_ms + flush_ms.bought on credit, and a too-low setting is a silent kill switch — every flush times out,
nothing exports, and the logs still say export is on. The signal to watch is a
warn: "spanflush exceeded its budget; spans may be lost." If that fires routinely in a healthy deployment,
the constant is wrong.
Span attributes and navigation
to
(trace_id, span_id), and something has to have sent those ids from a span that reached thesame backend. Temper injects
traceparent on its own outbound calls, so a link in your backendresolves to a real span rather than dangling.
tracestate is omitted rather than sent empty —W3C makes it optional, and a valueless header on every request is noise.
Logging
info. Used by the API, MCP, and internalfunctions. Their stdout is the log stream.
warn. The CLI's stdout is reserved formachine-readable output so
temper … | jq stays clean.RUST_LOG overrides either default. An unparseable RUST_LOG falls back to the default ratherthan refusing to start.
What the architecture provides vs. what a deployment configures
off" default, and the protocol choice are fixed by the architecture. A deployment configures
the destination:
OTEL_EXPORTER_OTLP_HEADERS.OTEL_TRACES_SAMPLER / OTEL_TRACES_SAMPLER_ARG.OTEL_SDK_DISABLED=true, or simply unsetting the endpoint.localhost:4318. The OTLP spec defaults to a localcollector, which would make every unconfigured process (your laptop, CI, a self-hosted
install) export at something that is not there. Temper treats "unset" as "off."
RUST_LOG does not control export in either direction. Both stacks filter per layer: thefmt layer follows
RUST_LOG, the export layer carries its own fixed filter. RUST_LOG=debugis safe on a live deployment — it widens logs, not what is billed. The surprising half:
RUST_LOG=off still exports spans. The switches that stop export are OTEL_SDK_DISABLED=trueand unsetting the endpoint.
Reading the exported spans
every span. Span metrics are a derived Prometheus view the backend's metrics-generator
produces, and everything Prometheus-backed — RED panels, alert rules — reads only that view.
CLIENT, SERVER, CONSUMER, and PRODUCER span kinds. NoINTERNAL span appears in any RED panel or Prometheus alert — which includes agent tool-callspans and the drain spans. The trace store answers directly over the spans that already exist,
with no generator change: a TraceQL metrics query over tool-call spans with
status=errorreturns per-tool error rates and is accepted as an alert query.
no-export with a warning — an exporter that cannot reach its backend never fails a startup and
never lengthens a request without bound.
Further reading
Send traces to an OTLP backend.
self-hosting Temper.
temperkb.io/operating/observability-and-audit.
temperkb.io/operating/deployment.
Modified at 2026-08-20 02:05:41