Runbooks

On-call runbooks for a degraded or down GoodData.CN deployment. Commands assume the gooddata-cn namespace.

Triage flow

1. Open the Overall Health dashboard. It puts readiness, API 5xx rate, OOM kills, Pulsar backlog, PostgreSQL/Hikari saturation, analytics latency, and exports on one screen. See Grafana Dashboards.

2. Check pod state and events. Look for pods that are not Running or Ready, a high RESTARTS count, and OOMKilling, BackOff, or FailedScheduling events:

   kubectl get pods -n gooddata-cn -o wide
   kubectl get events -n gooddata-cn --sort-by=.lastTimestamp | tail -n 40

3. Follow the logs. Logs are structured JSON. Lines carry traceId and spanId when tracing is enabled for the service — monitoring.tracing.enabled covers the JVM services and the API gateway, and pythonOtelExporterType covers the Python services (see Distributed Tracing). Grep one traceId across services to reconstruct a request:

   kubectl logs -n gooddata-cn deploy/<release>-<service> --since=15m \
     | grep '<traceId>'

   With tracing off, correlate on the timestamp of the first failing line and the pod name instead, and walk the request path service by service:

   kubectl logs -n gooddata-cn <pod> --since=15m --timestamps

   Centralized logging and tracing: Observability.

Before you contact support

Generate a support bundle from the affected cluster, then open a case at GoodData Support, attaching the bundle plus when the problem started and what changed before it did.

Runbook index