Pablo Albaladejo

OBSERVABILITY · · 6 min read

Observability for LLM pipelines: from streams to sinks

A practical guide to LLM observability: how to monitor LLM pipelines in production when classic APM misses streaming and agents. Telemetry as a swappable sink, deferred-flush metrics, and real code.

Observability for an LLM pipeline is the ability to answer, after the fact and in aggregate, what your model did. Which prompt and model version ran, how long it took, how many tokens it used, and whether it failed and why. And all of that without ever capturing the user’s text or the model’s output. That last clause is why you cannot point your existing monitoring at it and stop there.

Classic APM assumes a request comes in, work happens, a response goes out, and you measure the gap. LLM pipelines break that assumption in two ways. They stream, so the “response” is not done when the HTTP response is. The interesting signal only exists after the last chunk. And the payloads you would normally log are the ones you are not allowed to keep. So the discipline has three parts. Capture identifiers, versions and metrics. Wait until the stream completes to emit them. Route them through an interface you can swap without touching the pipeline.

I have built this three times now, at three very different scales, and the same shape keeps coming back. These are the three artifacts that taught it to me.

Artifact one: streaming, and the flush() trick

The first place the classic model breaks is AWS Lambda streaming a structured response from Bedrock. I wrote this up as an open-source reference, streaming-lambda-ai-sdk, with a four-part companion series that walks through it. Here I summarise only the observability idea.

The instinct is to attach logging and metrics to a middleware “after” hook. Middy has one, and most frameworks do. It does not work for a streamed body. The after hook fires when the handler returns the stream, not when the client finishes reading it. So anything you compute there sees an empty or half-filled picture. Token counts, finish reason, total latency: none of them exist yet.

The fix is to move the observation point to where the data ends. You wrap the outgoing stream in a WHATWG TransformStream, accumulate what you need as chunks flow through transform(), and emit telemetry from the flush() callback. The runtime calls flush() exactly once, after the last chunk has passed. The metrics that only exist at the end get emitted at the end.

The lesson goes beyond Lambda: for anything streamed, the emission point is the flush, not the return. Design for that and the rest of your telemetry follows.

Artifact two: agent telemetry as a port

The second artifact is the telemetry layer inside Kaiord’s AI package, which runs a small fleet of agents. Here the streaming detail matters less than the shape of the seam. That seam is boring on purpose: telemetry is a port, one method wide.

export type AiTelemetrySink = {
  emit: (event: AiTelemetryEvent) => void;
};

The runtime holds a sink and calls emit when a run ends. It never knows or cares where the event goes. What it emits is a closed union of exactly two shapes:

export type AiTelemetryEvent =
  | ({ type: "run_finished"; usage?: AiUsage } & RunIdentity)
  | ({ type: "run_failed";
       error: { name: string; retriable: boolean } } & RunIdentity);

Every event carries the same RunIdentity. It holds a trace id, the agent id and version, and the prompt id and version. It also holds the provider string, the model id, the run’s purpose, and the latency in milliseconds. A finished run may add token usage. A failed run adds an error with a name and a retriable flag. That is the whole vocabulary.

Notice what is not in there. There is no field for the prompt, the completion, document bytes or API keys. As the source comment puts it, the two event shapes “make payload capture impossible rather than optional.” You cannot leak what you have nowhere to store. The field names also map onto the OpenTelemetry GenAI semantic conventions: provider lines up with gen_ai.system, for example. That gives you the vocabulary without an OTel dependency you do not need yet.

Behind that one-method port sit three implementations, and the differences between them are the point:

Swapping console for a real exporter is one new file that implements emit. The pipeline that produces the events does not change a line.

Artifact three: what scale taught me

I cannot hand you the source of the third artifact, but it is where these patterns were tested under load. At Aircall, where the platform handles 1M+ calls a day, a few things stopped being style preferences and became rules:

None of that depends on a specific stack. These are the general patterns that survive real traffic.

A practical adoption path

You do not need a platform team or a tracing backend to start. In order:

  1. Define the event set first. Two shapes, finished and failed, that carry only ids, versions and metrics. If a reviewer cannot find a field that could hold prompt or completion text, you have done it right.
  2. Start with a ring buffer plus a console sink. The ring buffer lets your tests assert what was emitted. The console sink shows you the events in development. It needs no infrastructure, and you learn the shape of your own telemetry at once.
  3. Default the port to a no-op. Production without a configured sink should run fine and observe nothing, so nobody has to guard every call site.
  4. Move to a real sink when you need one. An OTel exporter, a metrics backend or your log pipeline: each is a single adapter behind the same emit. The pipeline never notices.
  5. Put the emission at the flush for anything streamed. It is the one place where the numbers are complete.

Observability for LLM pipelines is not a product you buy or a dashboard you set up on day one. It is a seam you build early: a one-method port, a closed event set, and a flush in the right place. Then, on the day traffic arrives, you already know what your models are doing.

FAQ

How is LLM observability different from normal APM?
Classic APM measures the request/response lifecycle: span duration, p99 latency, error rate. Two things break that model for LLMs. First, the call streams, so the response is not finished when the HTTP response returns. The signals you want (total tokens, finish reason, end-to-end latency) only exist after the last chunk. Second, the payloads you would normally log, prompts and completions, are the ones you must not log, because they carry user data. LLM observability captures identifiers, versions and metrics, waits for the stream to complete, and sends them through an interface you can swap.
How do I observe a streaming response if the metrics only exist at the end?
Do not attach observability to the handler's return. For a streamed body it fires before the client has consumed the stream. Wrap the stream in a WHATWG TransformStream and emit your telemetry in the flush() callback, which runs exactly once after the last chunk passes through. You accumulate counts as chunks flow and emit the complete picture at the end.
How do I keep user data and PII out of LLM telemetry?
Make payload capture impossible instead of optional. Define a small, closed event set whose fields are only identifiers, versions and metrics: trace id, agent and prompt versions, provider, model, latency, token counts. Leave no field that can hold prompt or completion text. If there is nowhere to put the bytes, nobody leaks them by accident.
Do I need OpenTelemetry to start?
No. Start with a console sink and an in-memory ring-buffer sink behind a one-method port, which needs no infrastructure. Name your fields to match the OpenTelemetry GenAI semantic conventions. Then a real exporter later is a new adapter, not a rewrite of the pipeline.

SOURCES & PROOF