Telemetry (OpenTelemetry / OTLP)
strad emits OpenTelemetry traces and metrics over OTLP. It is off by
default: with no OTEL_EXPORTER_OTLP_ENDPOINT set, no SDK starts, nothing is
exported, and the gateway behaves exactly as it did before. Turning it on is one
environment variable.
The instrumentation is biased hard toward one question — is this component failing because it ran out of resources, and if so, which resource? — because that is the question that is nearly impossible to answer from the outside. App Platform restarts a component that gets OOM-killed and tells you almost nothing about why.
Turning it on
Section titled “Turning it on”Seed two secrets in GCP Secret Manager (deploy-staging.yml already loads every
strad-<env>-* secret into the deploy job’s environment, so no workflow change
is needed — seeding the secret is the whole job):
| Secret | Required | Example |
|---|---|---|
strad-staging-OTEL_EXPORTER_OTLP_ENDPOINT | yes | https://otlp.example.com |
strad-staging-OTEL_EXPORTER_OTLP_HEADERS | only if your ingestor needs auth | Authorization=Bearer <token> |
render-spec.ts picks both up: the endpoint becomes a GENERAL env var (a
collector URL is not a credential, and an operator should be able to see what the
deployment is pointed at in the App Platform console), and the headers become a
SECRET env var, encrypted at rest, on the same path as every other credential.
On a store-backed deployment the header is not in the deploy environment at
all — it is read from /strad/{env}/gateway/static/OTEL_EXPORTER_OTLP_HEADERS.
The core and any strad-image bundle hydrate it themselves at boot; a third-party
supplementary-image component cannot, so the renderer reads it from the store
and bakes it onto that component. Rotating it is therefore a deploy, not a
restart, and a deploy where any component would end up without it
fails
rather than shipping a container that exports into a rejection.
The endpoint must speak OTLP over HTTP/protobuf — the OTLP spec’s default for
the HTTP transport, port 4318 on a standard collector. strad appends the signal
paths itself, so give it the base URL (https://host:4318), not
https://host:4318/v1/traces.
Optional knobs, all standard: OTEL_SERVICE_NAME, OTEL_RESOURCE_ATTRIBUTES,
OTEL_EXPORTER_OTLP_PROTOCOL, OTEL_METRIC_EXPORT_INTERVAL (defaults to 15s, not
the usual 60s — a 60s window can miss the entire memory ramp that precedes an OOM
kill), OTEL_TRACES_SAMPLER.
What every component reports
Section titled “What every component reports”Each process carries these resource attributes:
service.name—strad-core,strad-<component>service.version— the image tag actually deployed, not the VERSION fileservice.namespace—stradservice.instance.id— a fresh UUID per process startdeployment.environment(and.name) —staging|prod|devstrad.mode(core|bundle),strad.component
The resource-constraint signals
Section titled “The resource-constraint signals”Memory — the OOM predictor
Section titled “Memory — the OOM predictor”| Metric | Why |
|---|---|
strad.container.memory.limit | The cgroup limit — the only number the OOM killer consults |
strad.container.memory.usage | cgroup usage, including child processes (i.e. Chromium) |
strad.container.memory.utilization | usage / limit, 0..1. This is the OOM predictor. |
strad.process.memory.rss / .heap.used / .heap.total / .external | Where the memory went |
Two of the bundle’s tenants are not in the Node numbers and are in the
cgroup one: the shared Chromium, and the mcp-grafana child the Grafana mount
supervises. strad.process.memory.rss will look calm while
strad.container.memory.utilization climbs, which is the whole reason the
container metric is the one to alert on.
The limit is read from the cgroup (/sys/fs/cgroup/memory.max), not from
os.totalmem(). This matters more than it sounds: inside the bundle’s 2GB App
Platform container, os.totalmem() reports the host’s memory (tens of GB), so a
“percent memory used” computed the obvious way reads a comfortable single digit
while the kernel is about to kill you.
CPU / saturation
Section titled “CPU / saturation”| Metric | Why |
|---|---|
strad.eventloop.delay.p99 / .mean / .max | The single best saturation signal for Node. |
strad.eventloop.utilization | Fraction of the interval the loop was busy |
strad.process.cpu.utilization | CPU-seconds per wall-second (>1 = more than a core) |
strad.gc.duration, strad.gc.collections | GC pause distribution, by kind |
A saturated 1-vCPU box does not announce itself as “CPU: 100%”. It announces itself as an event loop that is 800ms behind — which is why every request suddenly takes a second. Chromium is the expected culprit, and it shows up here, as lag in the Node process waiting on it.
Restarts
Section titled “Restarts”App Platform restarts a component on OOM. To make that loud:
service.instance.idis regenerated per process, socount(distinct service.instance.id)over a window is the restart count.strad.process.uptimeis a sawtooth. A reset to ~0 is a restart.
The OOM fingerprint is the two together: strad.container.memory.utilization
climbing toward 1.0, and then a new service.instance.id with uptime back at zero.
Chromium
Section titled “Chromium”good-eggs, fetchpet and pointsyeah all drive a real headless Chromium — and
in the collapsed bundle they drive the same one, with a context per call. So
these metrics describe a single shared browser inside the strad-bundle process,
not one per server.
| Metric | Why |
|---|---|
strad.browser.launches | By outcome. In steady state this fires once, at boot — a nonzero rate means Chromium keeps dying. |
strad.browser.launch.duration | Baseline ~1.06s. Watch it climb as the box saturates. |
strad.browser.active | Live browser processes |
strad.browser.contexts.active | Live contexts. Should return to ~0 between calls; if it climbs, contexts are leaking. |
strad.browser.crashes | Tagged memory.pressure = high | normal | unknown |
That last attribute is the important one. Playwright reports an OOM-killed
Chromium and a Chromium that crashed on a bad page identically — a bare
disconnected event, no exit code, no reason. The container’s memory pressure at
the instant of the crash is the only thing that separates them, and it is gone a
second later, so we sample it there and bucket it. memory.pressure=high (≥0.85
utilisation) is the OOM fingerprint.
MCP / gateway behaviour
Section titled “MCP / gateway behaviour”Spans, nested into one waterfall:
mcp.request (SERVER; mode, principal kind, requested vs selected servers)└── mcp.tools.call (server slug, tool name, surface, arg count, byte sizes, outcome) └── mcp.upstream.call_tool (CLIENT; server slug, upstream kind, timeout, outcome)strad.surface on the tool-call span and its metrics is mcp for a call that
arrived at the endpoint and playground for one a signed-in human made from the
console playground. Both run the same
code, so without the attribute a latency graph would carry a human’s thinking
time in the same series as an agent’s calls. A playground call has no
mcp.request parent — it never crossed the HTTP endpoint.
W3C trace context (traceparent) is injected into every upstream hop, so the
bundle’s spans join the core’s trace. That is the difference between “the request
took 8s” and “the request took 8s, of which 7.6s was Chromium inside the bundle,
serving fetchpet”.
Metrics: strad.mcp.requests, strad.mcp.request.duration,
strad.mcp.in_flight (concurrency), strad.mcp.auth_failures (401, by reason),
strad.mcp.unavailable (424 fail-closed, by server), strad.mcp.unlistable (a
tools/list that failed for a server the client named, by server),
strad.mcp.tool_calls and strad.mcp.tool_call.duration (by server + tool),
strad.mcp.upstream.duration (per-server latency — this is what shows you
which server is starving the others), strad.mcp.upstream.errors,
strad.mcp.upstream.timeouts, and strad.mcp.tool_surface (is a slug serving
anything at all — see below).
Timeouts get their own counter deliberately. The MCP SDK’s request timeout defaults to 60s and applies at every hop, so a slow tool behind the gateway dies at exactly 60s and presents as a mysterious platform failure. Counting them separately turns that into a one-glance diagnosis.
strad.mcp.tool_surface — is a slug serving anything at all?
Section titled “strad.mcp.tool_surface — is a slug serving anything at all?”A gauge: how many tools each mounted server’s upstream is currently offering,
attributed by mcp.server and mcp.upstream.kind. The background monitor
(architecture) records it on every probe.
It is the signal this fleet was missing. A slug can be configured, entitled,
routed and dead, and the only symptom is a tools/list that comes back empty —
which reads exactly like a server that has no tools. That state ran for a week on
both environments and left nothing behind but a stdout line nothing ingested (the
log exporter is new, and is the other
half of this fix). A gauge pinned at zero for a slug, on the other hand, is a
thing you can alert on:
min_over_time(strad_mcp_tool_surface{mcp_upstream_kind="supplementary-image"}[15m]) == 0Scope it by mcp_upstream_kind. A remote-http is somebody else’s server on the
public internet, and paging you at 3am because deepwiki is having an afternoon is
how an alert gets muted. Scope by component too (service_name) if you want one
page per fault: core and each bundle run their own monitor, so a slug mounted in
both is two series for one problem.
Only immutable attributes are on it, deliberately. The state word — ok /
empty / unreachable — is on /healthz and in the log line and is not here.
OTLP defaults to cumulative temporality, under which every attribute set ever
recorded keeps being re-exported at its last value, with no expiry. A
state="unreachable" series would be minted by the first probe of nearly every
deploy, since core comes up before its bundles do, and would then sit at zero for
the life of the process — latching the alert above the first time anything
recovered. The value already carries the fact.
It measures the upstream’s surface, not what a client sees: the tools:
allow-list is applied on the listing path and not in the probe, so a slug whose
policy filters every tool away reads healthy here. mcp.policy_unmatched_tools is
the signal for that one.
strad.mcp.unlistable is its request-side companion — a tools/list that failed
for a server the client named. It needs its own counter because that failure
answers with a JSON-RPC error inside an HTTP 200, so it moves nothing else.
What strad will never send
Section titled “What strad will never send”strad proxies 1Password vault items, Gmail bodies, Google Docs and bank data. Telemetry leaves the box and lands in an ingestor. A pipeline that exfiltrated 1Password contents to a third party would be far worse than the resource problem the telemetry exists to diagnose.
What is recorded is shape and identity: which server, which tool, how many
arguments, how many bytes, how long, and whether it failed — in one of a fixed,
closed set of error kinds (timeout, upstream_unavailable,
local_machine_offline, server_unconfigured, not_selected, …).
Two things that look like omissions and are not:
recordException()is never called. It writesexception.messageandexception.stacktraceas span-event attributes, and an upstream MCP server’s error text is arbitrary text we did not author — it can echo the request back. Errors are classified instead.- The stdout STREAM is never shipped. It carries upstream error messages — arbitrary text strad did not author — and exporting it wholesale would drag those across this boundary. What is exported is strad’s own events, field by field, against an allowlist. See below.
test/telemetry.redaction.test.ts pins all of this: it drives a real MCP tool
call with a credential-shaped canary planted in both the arguments and the
result, serialises every exported span, metric and log record, and asserts
the canary appears nowhere. It also asserts the useful signals do emit —
otherwise it would pass happily on an implementation that exports nothing at all.
See src/telemetry/redact.ts.
Logs — strad’s own events, and only those
Section titled “Logs — strad’s own events, and only those”strad exports its structured events as OTLP logs. It does not export stdout.
That distinction is the entire design, and it was learned the expensive way. For
a long time there was no log exporter at all, on the reasoning above: stdout
carries upstream error text, so shipping the stream would breach the boundary.
Correct — and it cost a week. The secrets slug served zero tools in production;
mcp.list_tools_failed named the failure on every single request and went to a
stream with no ingestor. On App Platform, a log nobody ships is a log nobody
reads. The boundary held perfectly while the deployment was silently broken.
So src/telemetry/events.ts carries an allowlist: every event that may leave
the process, and for each one the exact fields that may go with it.
| Event | Why it is exported |
|---|---|
mcp.list_tools_failed / mcp.list_resources_failed | a mounted slug is serving nothing |
mcp.local_machine_offline | a local-tunnel host is off or refusing — not a fault |
strad.tool_surface_degraded / _recovered | which slug, and when it changed |
strad.tool_surface_offline | a local-tunnel’s machine is off — not degraded |
strad.tool_surface_unconfigured | a slug waiting on a mayBeUnseeded: credential |
mcp.server_unconfigured | the same, on a listing that named the slug |
strad.tool_surface | the fleet summary, on change |
mcp.unavailable | 424 fail-closed: a slug asked for and not entitled |
mcp.policy_unmatched_tools | a tools: typo silently narrowing a toolset |
strad.readiness_failed / _store_diagnostics_failed / strad.console_render_failed | the event name only — that surface is failing |
strad.listening / strad.secrets_hydrated | what this build mounted, and which keys resolved |
strad.shutdown | a clean SIGTERM. Its absence is an OOM kill |
strad.param_written / _write_failed / _refused | a server persisting a credential that rotated under it |
strad.param_read | a server reading one of its own parameters live — a paramsWritable: boot read, or a paramsRefresh: poll. The deploy record a runtime rotation would otherwise not leave |
strad.connector_begin / _connected / _refused | a human consenting, in the console, for one slug |
Three rules make it safe, and they are worth stating because the boundary is the
reason this exists at all rather than a pino transport:
- Deny by default. A field not on its event’s list is not exported — not truncated, not redacted, absent. Adding a field to a log call does not silently start shipping it; getting it on the list is how it ships.
erroris on no list. Every upstream error message in this codebase lands in a field of that name — the one that mattered here was a full HTML page from someone else’s Express. Where an event carries one, the exporter substituteserror.kind, the classification fromredact.ts, which is one of a fixed set of words strad wrote.- The streams still carry every event in full, upstream text and all. An
operator with a shell sees the upstream’s own words; the ingestor sees
strad’s. One thing did change about them: an event whose name ends
_failedor_degradednow goes to stderr regardless of which call site emitted it, somcp.auth_failedmoved off stdout. The stream now agrees with what the event is rather than with how it was written.
The connector events are the newest and the tightest. slug and variable come
from the config — the flow reads the slug out of sealed state and the variable
out of that server’s oauth: block, never from a request — and reason on a
refusal is a ConnectorErrorCode: a closed set of strad’s own words
(account_mismatch, scope_not_granted, no_refresh_token, …). Google’s own
error bodies are read for nothing at all, not even for a message, so there is no
upstream text on this path to redact. Nothing derived from the credential — no
length, no prefix, no digest — exists anywhere in the flow to be exported. See
Connectors.
The 424’s requested-server list is filtered to slugs that actually exist in
the config, with the rest reduced to a count — ?servers= is validated against
nothing, so it is one of the two places a client’s own bytes reach strad’s
vocabulary; the other is the tool name on a tools/call, which the table below
returns to. The same filtering applies to the mcp.server attribute on
strad.mcp.unavailable, where an unfiltered value would let any token holder
mint metric series at will.
Every exportable event a client can drive — mcp.list_tools_failed,
mcp.list_resources_failed, mcp.local_machine_offline,
mcp.policy_unmatched_tools and mcp.unavailable — is throttled to one exported
record per slug per minute, for the same queue-starvation reason. A repeated
fault is therefore under-reported by up to a minute in the ingestor; the streams
are never throttled, and strad.mcp.tool_surface carries the same fact
continuously. Fault events export at SeverityNumber.ERROR, so
severity >= ERROR finds them.
The second table: what does not leave, and why
Section titled “The second table: what does not leave, and why”An empty slot in a deny-by-default allowlist reads the same whether it is a
verdict or nobody looking, and those are not the same fact. So EXPORTABLE has a
counterpart in the same file: NOT_EXPORTED, keyed the same way, whose values
are the reason. Nine events are on it.
| Event | Why it stays on the stream |
|---|---|
mcp.auth_failed | its fields are safe, but it fires before authentication succeeds, so anyone who can POST /mcp could fill the batch queue and evict the signals that matter. strad.mcp.auth_failures counts the same thing by the same reason, in a channel with no queue to starve |
mcp.request | it fires on every authenticated request, which is mcp.auth_failed’s queue-starvation argument one token further in. Its fields would pass the field test — by that line resolveSelection has matched every requested slug against the entitled set — but the volume is the objection. The mcp.request span carries requested and selected; strad.mcp.requests counts the outcomes |
mcp.tool_denied | tool is denamespaced from the name the caller sent, so a client could write any string into the ingestor by calling a tool that does not exist. server is not that — by that line the slug has been matched against the selection — so an exportable shape — a slug and a count — is possible and unwritten |
ui.playground.tool_call | the tool is a string a human typed into a console text field, not a word from a set this repo wrote down. strad.mcp.tool_calls with strad.surface=playground counts the call, in a channel with no queue to starve |
telemetry.disabled / telemetry.started / telemetry.start_failed | boot lines from startTelemetry’s own log seam, not event() — they run around the lifecycle of the exporter an export would need |
errors.started / errors.start_failed | the same, one layer over: startErrorReporter runs before startTelemetry, so there is no log exporter yet to reach |
Those last five are a blind spot by construction, and that is the reason to
write them down rather than the reason not to: when telemetry goes quiet, the
line saying why is on the container’s stdout and nowhere else. mcp.tool_denied
is the one with real work behind it — a policy refusal reaches no counter either,
because the throw sits ahead of the strad.mcp.tool_calls block, so an operator’s
view of a withheld tool is dark in both channels.
Every evt: literal under src/ appears in exactly one of the two tables.
test/telemetry.event-export-decisions.test.ts reads the call sites as text and
fails the build for an event in both or in neither — so a new event() call is a
verdict somebody wrote, not a record that quietly reaches nobody. It also demands
that evt be written as a string literal, because one built from a variable would
be invisible to that scan.
An offline local-tunnel is not a fault, and does not export as one
Section titled “An offline local-tunnel is not a fault, and does not export as one”A machine you own being switched off is the advertised behaviour of
local-tunnel, not an incident. Two
things follow, and both were learned by paging a production alert channel over a
laptop somebody had closed:
error.kindhas a member for it:local_machine_offline. The other kinds are chosen by matching the error’s message, which is safe (strad reads the text, never exports it) and is all there is for an error thrown byfetchor by the SDK. ButLocalMachineOfflineError’s message is strad’s own — it ends “Nothing is wrong with strad” — so it matched nothing and classified asunknown, the same word an unclassified fault gets. It now declares its kind instead of being guessed at, and a declared kind is honoured only against the closed set, so declaring one can never widen what may be exported. This is the label on the hop’s span and onstrad.mcp.upstream.errors, and the hop asks the offline verdict directly rather than reading the declaration: it is instrumented inside the frame that later builds that error, so it only ever sees the rawfetchor SDK failure. Before this, one closed laptop was scattered acrossupstream_unavailable,timeoutandunknownby how the tunnel happened to fail.- It is reported as
mcp.local_machine_offline, nevermcp.list_tools_failedormcp.list_resources_failed. An event whose name ends_failedexports at ERROR, which is whatseverity >= ERROR— the first filter any operator writes — is for. Naming a closed laptop a strad fault there is a false page, and it cannot be fixed from the alert’s side: excludingerror.kind:unknownwould silently stop alerting on real unclassified faults, and naming slugs in the rule duplicates deployment topology and fails silent for the next tunnel anyone adds. The event name is what carries the distinction in the log channel —mcp.local_machine_offlineis not inCLASSIFY_ERRORand carries noerror.kindat all, because there is nothing left for one to say. The resources path matters as much as the tools path here:resourcesis advertised unconditionally, so an offline machine throws there on every session that lists them, whether or not anyone asked for that slug.
A client that names the slug gets a successful listing carrying one tool,
<slug>__troubleshoot_host — the slug stays connectable and says what to run on
the machine, rather than returning 200 {"tools":[]}, which would read as “this
server has no tools”. explicit on the record says who was asking: true when a
client named the slug, false when it was a fleet-wide listing.
Records are batched, not sent per-record: these are diagnostics, and a synchronous
round trip on the failure path would add latency to exactly the requests already
going badly. As with every other signal here, no OTEL_EXPORTER_OTLP_ENDPOINT
means no exporter and no behaviour change.
Exception reporting — GlitchTip, over the Sentry protocol
Section titled “Exception reporting — GlitchTip, over the Sentry protocol”Everything above stops at the process boundary. If strad crashes — an
uncaught TypeError, an unhandled rejection, a boot that throws — none of it
goes through event(), so none of it is exported. The stack lands on stdout,
which is never shipped, the component restarts, and the ingestor can tell you
service.instance.id changed but not one word about why.
strad closes that with a Sentry-protocol reporter pointed at a self-hosted GlitchTip. Like OTLP, it is off by default and one environment variable turns it on:
| Variable | Type | Example |
|---|---|---|
SENTRY_DSN | SECRET | https://<key>@glitchtip.example.com/1 |
Unset — the default — and nothing happens: the SDK is behind a dynamic
import(), so it is not even loaded. No client, no process handler, no network,
and no log line. Set it and one errors.started line names the ingest host (never
the key) at boot.
It travels the same path as every other credential: seeded as
strad-<env>-SENTRY_DSN in GCP Secret Manager, injected as a SECRET env var by
render-spec, or — on a store-backed deployment — read from
/strad/{env}/gateway/static/SENTRY_DSN at boot like the other
boot keys.
Absent, the store roster reports it missing and the deploy is unaffected: it is
optional, like every other telemetry name.
Unlike OTEL_EXPORTER_OTLP_HEADERS, it does not reach a supplementary image.
Only strad’s own process reports exceptions; a third-party container runs code
strad did not write, and a DSN is a write credential for the issue tracker.
What it reports
Section titled “What it reports”Uncaught exceptions, unhandled rejections, and boot failures — the last
explicitly, because main().catch() exits by a path no process handler ever
sees, and a boot that throws is the likeliest crash strad has. The reporter
therefore starts before the config is loaded, so a config that will not parse
is reported too; a deployment whose DSN is in the parameter store gets a second
start after hydration.
A crash still behaves like a crash. Both handlers preserve what Node does with no listener registered: print, then exit non-zero. That takes two deliberate departures from the SDK’s defaults, because the mere presence of a listener disables Node’s own behaviour:
- The uncaught-exception integration runs with
exitEvenIfOtherHandlersAreRegistered: true. Its default is to decline to exit if any otheruncaughtExceptionlistener exists — so one dependency adding one handler would leave a reported, unprinted, still-running process. - strad registers its own
unhandledRejectionhandler and does not install the SDK’s. That integration’s ignore list is unconditional and includesAbortError— the shape every timed-out store read in this codebase takes — so such a rejection would have been neither reported nor fatal.
Each event carries the identity OTLP already stamps — environment is
deployment.environment (prod, staging, dev), release is
service.version (the deployed image tag), and strad.component / strad.mode
are tags — so a GlitchTip issue and a Grafana log record line up without anyone
maintaining a mapping.
What it deliberately does not report
Section titled “What it deliberately does not report”The event() fault path. mcp.list_tools_failed, strad.tool_surface_degraded
and the rest are not captured, and that is a decision:
- They already export as OTLP log records at ERROR severity, against a per-event field allowlist. Sending them twice buys no signal.
- They are the events an upstream can drive.
mcp.list_tools_failedfires once per failing slug per listing, and itserroris text strad did not author — the reason the allowlist shipserror.kindinstead. Feeding that into an issue tracker would fill it with someone else’s HTML error page, grouped as a strad defect.
A caught, classified error is a condition strad handled. A crash is not. GlitchTip holds the second kind.
Secrets
Section titled “Secrets”This is the one channel in strad that ships free text, so it is also the one with a filter in front of it:
sendDefaultPii: false, and no default integrations at all. The SDK’s defaults would attach request data (headers, cookies, query strings), console output as breadcrumbs — which is every upstream error message strad logs — and source-context lines read off disk. The only integrations installed are the two process handlers anddedupe.beforeSendrebuilds the event from a key allowlist, the same discipline as the log exporter’s.request,user,extra,breadcrumbsandmodulesare dropped; so are stack-framevarsand source context. A field a future SDK version adds is dropped rather than shipped.- Every string that survives goes through
scrubText(): every credential-shaped env var this process holds is struck out by value (including the string leaves of a JSON one likeSTRAD_TOKENS), then PEM blocks, JWTs,Bearerlines, URL userinfo andsecret-ish-key = valuepairs are struck out by shape. Then it is truncated — a message long enough to be a payload is a payload. tracesSampleRate: 0, andskipOpenTelemetrySetup: trueso the SDK never registers a tracer provider. Traces go to Tempo, from the OTel SDK above.
test/telemetry.errors.test.ts pins both ends: that an unset SENTRY_DSN
initialises nothing and installs no handler, and that a secret-bearing exception
does not survive beforeSend intact.
See src/telemetry/errors.ts and src/telemetry/redact.ts.
The supplementary images
Section titled “The supplementary images”The gateway starts its own SDK in-process. A supplementary image can’t — it isn’t strad — so each one boots through a vendored OTEL bootstrap instead:
CMD ["node", "--import", "/app/host/otel/instrument.mjs", "/app/host/dist/index.js"]otel/instrument.mjs is copied byte-identical into every image that has one.
Each image is its own Docker build context and cannot COPY from the repo root —
the same reason the shared/ trees are vendored. telemetry/bundle/instrument.mjs
is the source of truth; it is the telemetry entry in the vendoring manifest
(scripts/sync-vendored.mjs), npm run telemetry:sync copies it to seven targets,
and test/vendored.sync.test.ts fails CI if any copy drifts.
Seven targets, because the per-server images (bundle-api, good-eggs, fetchpet,
pointsyeah, onepassword, bundle-google) still exist and are still built even
though only strad-bundle is deployed. See Known limitations.
The telemetry env vars reach a supplementary component the same way — appspec.ts
forwards exactly the OTEL_* keys into a component that otherwise receives only its
own secrets. test/appspec.telemetry.test.ts pins that.
The bootstrap also publishes the Chromium hooks on globalThis, which the browser
servers call through a small local shim that no-ops when the bootstrap is absent.
That keeps the vendored shared/ trees free of any dependency on strad.
Failure modes (all of them safe)
Section titled “Failure modes (all of them safe)”Telemetry must never be able to take a live deployment down.
- Endpoint unset → no SDK starts, one
telemetry.disabledlog line, normal boot. - Endpoint malformed → caught, logged as
telemetry.start_failed, normal boot. - Collector unreachable at runtime → the exporter retries and drops on a background timer. It never propagates into a request.
- SIGTERM → telemetry is flushed last, with its errors swallowed, so a dead collector cannot wedge the shutdown path. When a container is being killed for OOM, those last few seconds of memory metrics are the diagnosis.