“The record is the proof” is a claim about content: an object is its digest, a pointer chain is signed and offline-verifiable. None of that helps if the process serving it is a black box. SP1 made the record verifiable; SP13 makes the server operable. The arc has two slices — observability (SP13b) and abuse-resistance (SP13c) — bound by one constraint: the domain object.Service and every seam interface stay byte-for-byte unchanged. Everything here lives in the transport and composition layer, so you can operate the service without renegotiating the contract that makes the record trustworthy.
One line and one ID per request
Before SP13b, a running chumd emitted almost nothing per request. The fix is LogAndMeasure in chum/internal/transport/obs/obs.go: middleware that assigns a request ID, logs one structured line, and records metrics — without buffering the body, so large blob streams still pass straight through.
The request ID is 16 bytes from crypto/rand, hex-encoded, set on the response as X-Chum-Request-Id (the RequestIDHeader constant) before the wrapped handler’s first write — otherwise the header is already flushed and the set is a no-op. Generation is deliberately non-fatal: if rand.Read fails, requestID() returns "" and the request proceeds without an ID. A missing correlation token must never fail a request.
Each request then emits exactly one slog line — log.Info("http request", ...) — carrying request_id, plane, method, path, status, bytes, and duration_ms. Status and byte count come from a small recorder wrapping the http.ResponseWriter, defaulting status to 200 and accumulating bytes on every Write. The recorder forwards Flush so a streaming blob handler still flushes through it, but intentionally does not implement http.Hijacker or io.ReaderFrom: no handler in either plane needs them, and claiming them would silently break the streaming passthrough — an omission documented at the recorder type, not left as a surprise.
A dedicated registry, low-cardinality on purpose
chum/internal/metrics/metrics.go owns the Prometheus side. The first decision is prometheus.NewRegistry() — a dedicated registry, not the global default. Tests get isolation, and there are no accidental global registrations. New() registers the chum vectors plus collectors.NewGoCollector() and collectors.NewProcessCollector(...), so operators get go_* and process_* for free, and Handler() renders it all via promhttp.HandlerFor.
The HTTP metrics are chum_http_requests_total (counter, labelled plane/method/code), chum_http_request_duration_seconds (histogram on default buckets, labelled plane/method), and chum_http_requests_in_flight (gauge, labelled plane). The reaper from earlier sprints feeds chum_reaper_scanned_total, chum_reaper_orphans_reclaimed_total, chum_reaper_kept_total, chum_upload_sweep_sessions_total, and chum_upload_sweep_reclaimed_total via RecordReap / RecordUploadSweep, called after each pass in startReaper.
The load-bearing detail, called out in the design at docs/superpowers/specs/2026-06-19-sp13b-observability-design.md, is cardinality. Labels are plane, method, code — never the raw request path. Every distinct bucket/key would mint a new time series and blow up the TSDB; that is why LogAndMeasure takes a fixed plane string instead of deriving a label from the URL. Full paths still appear in the logs, which are not cardinality-bound. ObserveHTTP even seeds the gauge series (_ = m.inFlight.WithLabelValues(plane)) so the family scrapes at zero rather than vanishing when nothing is in flight.
A limiter that lives inside the logger
SP13c adds per-IP rate limiting in chum/internal/transport/ratelimit/ratelimit.go. Each client IP gets a golang.org/x/time/rate token bucket; a request over the rate is rejected with 429 and Retry-After: 1. The design choice that matters is placement: in buildMux (chum/cmd/chumd/main.go), the limiter’s Middleware wraps inside obs.LogAndMeasure, not outside it. A rejected request is therefore still logged and still counted — it lands in chum_http_requests_total{code="429"} and bumps the dedicated chum_http_ratelimited_total{plane} counter via RecordRateLimited. Abuse you can’t see isn’t abuse-resistance.
The limiter is default-off. ratelimit.New returns nil when rps <= 0, and a nil *Limiter is a valid passthrough — Middleware on a nil receiver returns next unwrapped, zero overhead. It only turns on when CHUM_RATE_LIMIT > 0, matching the posture of CHUM_GC_INTERVAL. A background goroutine (runEviction) drops IP buckets unseen within evictAfter (default 10 minutes, CHUM_RATE_EVICT) so the map stays bounded under churn; losing an idle bucket’s token state is harmless because an idle IP isn’t mid-burst. /health and /metrics are registered directly on the mux, outside both wrappers — liveness and scraping must always work, so they are never rate-limited.
The trusted-proxy subtlety
The hard part of per-IP anything is which IP. clientIP defaults to r.RemoteAddr with the port stripped. It honors the first X-Forwarded-For hop only when trustProxy is set (CHUM_TRUSTED_PROXY=true), because XFF is client-spoofable: trust it unconditionally and any client picks its own bucket key, defeating the limiter. Even when trusted, the hop is accepted only if net.ParseIP parses it — a misconfigured or hostile proxy could otherwise inject arbitrary text, minting an unbounded set of bucket keys and spamming the eviction map. A blank or unparseable trusted hop falls back to RemoteAddr.
This is why chum/docs/deployment.md makes a reverse proxy mandatory. CHUM_TRUSTED_PROXY=true is only safe behind a proxy that overwrites X-Forwarded-For (Caddy’s header_up X-Forwarded-For {remote_host}). The same topology solves the other exposure problem: /metrics is unauthenticated and shares the listener with the API, so chumd publishes no host port and the proxy refuses /metrics at the edge — Prometheus scrapes it from inside the internal network. The launch checklist gates exactly these: rate limiting on, trusted-proxy set correctly, /metrics not public.
What this enables
A chumd operator can now answer the questions a black box can’t: how many requests, at what latency, in which plane, and — via the request ID echoed in every response header and log line — trace a single user-reported failure to its exact log entry. Abuse has a bounded blast radius and a visible signal. And none of it touched the domain: the verifiability story from SP1 is intact, because observability and rate limiting are purely transport-layer concerns.
What’s deferred is named honestly in the design. No distributed tracing yet — the request ID is the lightweight precursor to OpenTelemetry spans, not a replacement. No per-DID or per-bucket metrics (the same cardinality risk; logs carry per-request attribution). No auth on /metrics in-app — that is a firewall/proxy concern enforced by the deployment topology, not a code path. No Grafana dashboards, no alerting rules, no histogram-bucket tuning. The server is observable and abuse-resistant; the dashboards on top of it are the next sprint’s work, not this one’s claim.