Pin Your GenAI Telemetry Schema or Your LLM Release Gates Will Lie

OpenTelemetry is the right foundation for transporting and instrumenting LLM telemetry. What it is not, at least not yet, is a stable production contract for LLM observability. That gap matters more every quarter, because release decisions now hinge on token usage, streaming latency, evaluator results, and model-specific failure rates.

Treat the OpenTelemetry GenAI semantic conventions the way you’d treat any upstream dependency: pin them, translate them into a schema you own, and upgrade them on purpose. When a Development-stage convention can shift underneath an SLO query or a release gate, that gate isn’t protecting anything. It’s producing the appearance of control.

The GenAI repository split moved the dependency boundary

On June 12, 2026, OpenTelemetry Semantic Conventions v1.42.0 deprecated the GenAI, OpenAI, and MCP spans, metrics, events, and attributes in the core semantic conventions repository. Their definitions moved into a separate repository, semantic-conventions-genai.

That repository has a wide remit: inference, retrieval, agents, workflows, tool execution, token usage, streaming latency. Organizationally this makes sense. GenAI telemetry is evolving far faster than the mature conventions for HTTP or databases, and splitting it out lets maintainers iterate without dragging the core repository along at the same pace.

The operational consequence is easy to wave off. The GenAI metrics specification is still explicitly marked Development, and OpenTelemetry’s own stability policy is blunt about what that means: Development signals may receive breaking changes, and users are advised not to take long-term dependencies on them. So gen_ai.* is an upstream interface you consume, not a contract your application owns.

Depending on a Development specification is fine. Teams lean on unstable libraries and pre-1.0 APIs all the time. The mistake is letting that dependency bleed straight into every PromQL query, dashboard, cost report, alert, evaluator join, and deployment gate.

“Standardize on OpenTelemetry” sounds like an architectural decision, but it quietly leaves a pile of decisions unresolved. Which semantic-convention release does the application emit, and which one does the Collector configuration expect? How do you normalize provider-specific token categories? What happens when an attribute is renamed or its meaning narrows? Which schema does a historical baseline run against? When telemetry goes missing, does a release gate fail open or closed?

Until those have answers, vendor-neutral observability is mostly an aspiration. OpenTelemetry standardizes the mechanisms and, over time, shared semantics. It does not make every Development namespace safe to consume as permanent infrastructure.

A stale query can approve a worse model

Telemetry failures are most dangerous when they look like valid measurements. An exporter crash is loud and obvious. A renamed attribute produces a clean graph plotted over the wrong population.

Picture a deployment gate comparing a model canary against the current production version. It checks evaluator scores, p95 streaming latency, and estimated cost per successful task. Midway through a convention upgrade, the model attribute changes, or a token metric moves to a different instrument. The canary emits the new schema; the production baseline still speaks the old one.

Now the gate can fail in several quiet ways. The query returns an empty baseline. A dashboard expression coalesces the missing data to zero. Canary and baseline show up as two unrelated time series. An aggregation silently drops records whose dimensions no longer match. The gate approves the release without ever running the comparison it was built to run.

A time-to-first-chunk SLO is unusually sensitive to this kind of drift. “First chunk” might mean the first received streaming frame, the first frame carrying non-empty text, or the first usable application payload. Some providers send role metadata before any content arrives. Retries and proxy buffering blur the clock boundary further. Renaming a metric is survivable. Keeping the name while quietly changing which of those events it marks is much worse, because nothing in the graph tells you the definition moved.

Token cost monitoring carries the same hazard. Input, output, cached, and provider-specific token categories don’t always map cleanly onto each other. If an adapter stops populating one category and the cost query reads the absence as zero, the new model looks cheaper for the worst possible reason: the accounting became incomplete.

None of this is limited to attribute names. A convention upgrade can change metric names, units, or histogram boundaries; the enum values that filters and grouping depend on; whether a field is required, optional, or gone; the meaning of the operation, provider, model, or error dimensions; and which event marks the start or end of streaming latency. Any one of those can split a time series or reset a baseline while the graph still renders and the query still returns a number. That’s what I mean when I say the gate lies. It gives a confident answer to a question no one asked.

Missing telemetry has to count as a gate failure, not a healthy pass. A production gate should demand a sufficient number of comparable observations, matching schema versions, and explicit coverage of every input feeding the decision. An empty vector is not zero latency, zero cost, or zero errors.

The canonical schema belongs at the provider boundary

Normalize provider telemetry into an internal canonical schema before it becomes an operational dependency. That boundary can sit in application instrumentation, in an adapter service, or in an OpenTelemetry Collector transformation. Location matters far less than ownership: the mapping has to be version-controlled, testable, and shipped as production code.

The internal schema should describe the measurements the system actually relies on. Borrow OpenTelemetry terminology where the semantics genuinely line up, but keep the compatibility policy in the hands of the team running the service.

A workable canonical record usually has to distinguish the requested model from the resolved model the provider returns, preserve the operation type, identify the prompt artifact and application revision, and record token categories without folding unknown ones into zero. It also needs precise latency boundaries and explicit versions for both the canonical schema and the upstream convention used to populate it.

Keep high-cardinality identifiers such as prompt hashes, trace IDs, response IDs, and evaluator-run IDs in spans or logs, not in metric labels. Metrics should carry only the bounded dimensions that SLOs and release comparisons need. Owning the schema is not an excuse to turn the metrics backend into an unbounded index.

Cost deserves its own contract. Token counts and monetary cost are different facts. Counts come from request or provider telemetry; cost comes from applying a price catalog with a currency and an effective version. Historical estimated cost should stay reproducible even after a provider changes its pricing, which means a gate using cost per successful task should read a pinned pricing version rather than an unversioned “current prices” table.

The strongest objection to all of this is that an internal schema adds a translation layer and erodes interoperability. Fair. Bespoke schemas can rot into undocumented local dialects, and every adapter costs something to maintain. I’d still take that cost over wiring production controls straight to a Development specification. The internal model should stay thin, documented, and close to the upstream conventions, and it should keep unknown values and raw provider fields where it makes sense instead of pretending every provider is identical. OpenTelemetry stays the wire format, the context mechanism, the SDK ecosystem, the export path. The internal schema supplies the one thing the upstream project is explicit about not promising yet: a stable compatibility boundary.

Vendor neutrality comes from owning that boundary. Piping every provider’s telemetry through OpenTelemetry without normalizing its meaning just relocates the lock-in from a provider API to a moving external schema.

The telemetry version belongs in the release manifest

A model name is not enough to identify a release. Provider aliases move, prompt templates change behavior, evaluator suites evolve, and application code decides how tool calls, retries, and streaming are handled.

Record a release tuple that captures at least the application revision; the prompt version or an immutable content hash; the requested and, where available, resolved model identifier; the evaluator and evaluation-dataset versions; the internal telemetry schema version; the pinned upstream GenAI convention version; and the pricing-catalog version whenever cost feeds the gate. This is the practical core of LLM observability schema versioning. Telemetry is part of the release artifact because it decides whether the release can be measured against its baseline at all.

Pinning has to reach past a dependency declaration. The instrumentation library version, the semantic-convention release or commit, the Collector image and its configuration, the transformation rules, the dashboard definitions, and the release-gate queries all need to be recoverable together. Otherwise a rollback of application code keeps flowing through a newer, incompatible telemetry pipeline, and the rollback isn’t really a rollback.

Evaluator results demand the same discipline. An aggregate score is only comparable when the evaluator logic and dataset version match, or when an explicit migration has established equivalence. Joining evaluations to production traces through an attribute that changed name or cardinality is one more quiet way to lose half your population.

The release system should refuse mixed or unknown schema versions unless the query in question was deliberately designed and tested to handle them. Compatibility should be a decision, not an accident. A broad coalesce across old and new fields can keep a dashboard populated while it hides duplicate records, changed units, and shifted definitions.

A GenAI telemetry migration needs dual-writing and query tests

Handle a convention upgrade the way you’d handle a database schema migration. Bumping instrumentation packages and eyeballing whether the dashboards still look plausible is not a plan.

Start by pinning both the current and target OpenTelemetry GenAI semantic conventions. Build a field-level mapping that classifies every metric, attribute, and event as unchanged, renamed, transformed, newly available, removed, or semantically different. A unit conversion is a transformation. A new definition of “first streaming output” is a semantic change and must not be dressed up as a rename.

Then introduce a new internal schema version. Additive changes are easiest to live with, but any field whose meaning has moved should get a new name or a new metric. Reusing an old name with a new unit or a new clock boundary corrupts every historical comparison that touches it.

Dual-write the old and new representations from the same observations, as close to the normalization boundary as you can manage, so both versions see identical provider responses, retries, stream events, and errors. Writing old telemetry in the application and deriving new telemetry later in the Collector opens up room for sampling and timing differences that will haunt the comparison.

The dual-write window has to cover the longest baseline any alert or release gate uses. A day of plausible-looking graphs proves nothing if the production comparison reaches back over a longer window. Throughout, the schema version should be auditable without becoming an uncontrolled label sprayed across every metric.

Regression tests should exercise the consumers, not just the mapping code. Worth covering at minimum:

• Feed recorded or synthetic provider payloads through both mappings and compare operation counts, errors, token totals, and latency samples.
• Verify token-cost calculations against fixed token fixtures and a pinned price catalog, including missing and unknown token categories.
• Test streaming sequences with metadata-only chunks, empty chunks, retries, and failures that occur before the first usable output.
• Run old and new SLO queries over the same fixtures and compare the populations they select, not only the final percentiles.
• Evaluate alert rules against healthy, degraded, absent, and mixed-schema data.
• Run the real release gates in shadow mode and require an explicit error when baseline data or schema coverage is missing.
• Render or validate dashboard queries so a renamed attribute can’t leave a panel syntactically valid but semantically empty.

Percentile equality is a weak test on its own. Two queries can report the same p95 while selecting different requests. Compare counts, grouping dimensions, missing-value behavior, and, where feasible, the identities of the sampled traces.

Once the new pipeline has survived the full comparison window, switch dashboards, alerts, cost reports, and gates to it in one coordinated release. Keep the old write path alive long enough to support a rollback, and retire it only after every known consumer has migrated and queries against the old schema have gone silent.

This is heavier than adopting the latest gen_ai.* definitions directly, in exactly the way a versioned database migration is heavier than editing a column in place. The weight buys a property that earns its keep under production load: a convention release cannot silently redefine the evidence you use to ship a model. OpenTelemetry remains the right foundation, and its own stability documentation is clear about where that foundation is still curing. Keep the Development conventions behind a pinned, owned, and tested contract, and take the warning at face value.

References

• OpenTelemetry Semantic Conventions v1.42.0 release notes: https://github.com/open-telemetry/semantic-conventions/releases/tag/v1.42.0
• OpenTelemetry GenAI metrics specification: https://github.com/open-telemetry/semantic-conventions-genai/blob/main/docs/gen-ai/gen-ai-metrics.md
• OpenTelemetry versioning and stability policy: https://opentelemetry.io/docs/specs/otel/versioning-and-stability/