---
name: observability-and-instrumentation
description: Instrument features with structured logs, metrics, traces, and alerts so production behavior is diagnosable from telemetry alone. Use when shipping anything that runs in production, adding logging or alerting, or when incidents took too long to diagnose. Not for diagnosing a live failure: use systematic-debugging. Not for fixing measured slowness: use performance-optimization. Not for launch-day monitoring gates: use shipping-and-launch.
license: MIT
metadata:
  author: TechTide AI (Alex Cinovoj)
  provenance: rewritten from patterns in addyosmani/agent-skills (MIT)
  category: Quality & Security
---

# Observability and Instrumentation

Code you cannot observe is code you cannot operate. Instrumentation is written alongside the feature, like tests. Ship without telemetry and the first bug report becomes archaeology instead of a query.

## Workflow

1. **Write the on-call questions first.** 2-4 questions an on-call engineer will ask about this feature ("what fraction of payments succeed after retry?", "when one fails permanently, why?"). Every signal you add must answer one. No questions means you will log everything and learn nothing.
2. **Pick the signal per question.** Metrics tell you that something is wrong, traces tell you where, logs tell you why.

| Signal | Answers | Cost |
|---|---|---|
| Structured log | What happened in this case | Per event, grows with traffic |
| Metric | How often, how fast, in aggregate | Fixed per series, cheap |
| Trace | Where time went across services | Per request, sampled |

3. **Structured logs.** Log events, not prose. Stable event name plus machine-readable fields:

```typescript
// BAD: unqueryable prose
logger.info(`Payment ${id} failed for ${userId} after ${n} retries`);
// GOOD: stable event + fields
logger.warn({ event: 'payment_failed', paymentId: id, provider, errorCode: err.code, attempt: n }, 'payment failed');
```

Levels: error means invariant broken and someone may act, warn means degraded but handled, info means significant business event, debug is off in production. Correlation IDs are mandatory: generate or accept a request ID at the boundary, attach it to every log line, span, and outbound call. Never log secrets, tokens, passwords, or unredacted PII. Allowlist fields, never dump request bodies.

4. **Metrics.** RED on every endpoint and external dependency: rate, errors, duration as a histogram. USE for resources: utilization, saturation, errors. Cardinality is the failure mode: labels come from small fixed sets (route template, status class like "5xx", provider name). User IDs, raw URLs, and error text are never labels, they belong in logs and traces. Track percentiles, never averages: an average hides the 1% having a terrible time.
5. **Traces.** Use OpenTelemetry, the vendor-neutral standard. Auto-instrumentation covers HTTP, gRPC, and common DB clients with near-zero code, imported before anything else. Add manual spans only around meaningful units of work, with the attributes on-call will filter by. Propagate context across every async boundary (headers, queue metadata) or the trace dies at the gap. Sample low by default, keep 100% of errors if your backend supports tail sampling.
6. **Alerts.** Alert on symptoms users feel (error rate over 1% for 5 min, p99 over 2s, queue age over 10 min), not causes (CPU 85%, one pod restart). Cause alerts fire when nothing is wrong and miss failures you did not predict. Every alert: actionable (if the response is "ignore, it self-heals", delete it), links a runbook (even three lines), threshold justified by SLO or history, and one of exactly two severities: page (act now) or ticket (act this week).

## Verification

Force an error in staging. Then, using telemetry alone, no source reading: find the failure by request ID in the logs with structured fields intact, see the metric move with expected labels, follow the request end to end in the tracing UI with no broken spans. Expect all three. Any gap: the instrumentation is incomplete, fix the missing signal.

Test-fire each new alert once (temporarily lower the threshold). Expect it in the right channel with a working runbook link. If not, fix routing before shipping.

## Good vs Bad

**Bad:** `console.log("payment failed: " + err)` scattered through the handler, plus an alert on CPU > 80%.

**Good:** `payment_failed` event with provider, error code, attempt, and request ID; a failure-rate histogram per provider; a page-severity alert on user-visible error rate with a runbook link.

## Completion checklist

- [ ] On-call questions written, each signal maps to one
- [ ] All logs structured with stable event names and a correlation ID
- [ ] No secrets or PII in log output (spot-check real output, not the code)
- [ ] RED metrics on every new endpoint and external dependency, bounded labels
- [ ] Latency is a histogram, p95/p99 queryable
- [ ] One request traceable end to end without broken spans
- [ ] Every new alert symptom-based, runbook-linked, test-fired once
- [ ] Induced staging failure located via telemetry alone

Any box unchecked: not done. Fix or say so.

## Footguns

- **String-interpolated logs.** Cannot filter, correlate, or alert. Fix: structured logger with event names, five extra minutes once.
- **Cardinality bombs.** A user-ID label creates one time series per user and falls the metrics backend over. Fix: bounded label sets only, high-cardinality data goes in logs and traces.
- **Missing context propagation.** The trace ends where the queue begins. Fix: pass trace context in message metadata, verify one full trace in the UI.
- **Noisy pager.** Alerts that fire daily and get acked without action train the team to ignore the real page. Fix: delete or demote every alert that has not driven action in a month.

## Red Flags

Stop if you hear yourself think any of these:

- "I'll add logging after it works"
- "console.log is fine for now"
- "More logs is more observability"
- "We can just look at the dashboards when something breaks"
- "Alert on everything, we'll tune later"
- "Tracing is overkill for two services"

Each is the rationalization that precedes a blind incident. The rule has no "just this once" clause: a feature PR with retries, queues, or external calls and zero new telemetry is not done.