Collect a Support Bundle

A support bundle is a single archive of cluster state and component logs that GoodData Support uses to investigate a problem in your GoodData.CN deployment. You generate it yourself with troubleshoot, an open-source tool from Replicated, driven by a YAML specification that tells it what to collect and which checks to run.

The GoodData.CN Helm chart ships that specification with it and publishes it into your cluster, so the namespaces, release name and Secret names it collects always match the deployment you actually installed. In most cases you do not need a specification file at all.

The specification collects no Secret values: it records which Secrets exist and which keys they contain, which is what diagnoses a failed install. The connectivity probes, which are off by default, are the one exception – they read the credentials they need into a temporary pod in order to open a connection, and write none of them to the archive. It does collect container logs, and those logs can carry credentials, tokens, connection strings or customer data that a component wrote to its output. Troubleshoot applies its built-in redactors plus a set of GoodData-specific ones covering license keys, database connection strings, bearer tokens and object-storage credentials.

Prerequisites

• kubectl configured against the cluster running GoodData.CN.
• Read access to the namespaces you collect from and to cluster-scoped resources (nodes, cluster version). A read-only cluster role is enough; cluster-admin also works.

Install the support-bundle plugin

Install the plugin with krew, the kubectl plugin manager:

kubectl krew install support-bundle

If you don’t use krew, the installer script from Replicated does both steps for you:

curl -s https://krew.sh/support-bundle | bash

The script downloads and runs code from the internet. Read it first if your environment requires that. Then add krew’s binary directory to your PATH:

export PATH="${PATH}:${HOME}/.krew/bin"

Collect the bundle

Point the plugin at your cluster and let it discover the specification the chart published:

kubectl support-bundle --load-cluster-specs -n <namespace>

Use the namespace your GoodData.CN release is installed in; the examples below use gooddata-cn.

--load-cluster-specs finds any ConfigMap labelled troubleshoot.sh/kind: support-bundle, which is what the chart creates as <release>-support-bundle. Nothing needs editing, and there is no namespace to get wrong.

In CI or any non-interactive shell, skip the results browser:

kubectl support-bundle --load-cluster-specs -n gooddata-cn --interactive=false

The run writes a compressed archive named support-bundle-<TIMESTAMP>.tar.gz to the current directory. Unpack it to review what was collected:

tar -xzf support-bundle-<TIMESTAMP>.tar.gz

Tune what gets collected

Everything below is a Helm value, so the change survives upgrades and applies to whoever runs the collection next. Apply them the way you apply any other value, then re-run the collection.

Include your ingress controller and other namespaces

The chart collects from its own namespace and from Pulsar’s automatically. It cannot guess which ingress controller you run, so add it:

supportBundle:
  extraNamespaces:
    - ingress-nginx
    - cert-manager
  extraLogs:
    - name: ingress-nginx
      namespace: ingress-nginx
      selector:
        - app.kubernetes.io/name=ingress-nginx
    - name: cert-manager
      namespace: cert-manager
      selector:
        - app.kubernetes.io/instance=cert-manager

extraNamespaces collects Kubernetes objects and events; extraLogs collects container logs. Add both for a component you want fully covered.

Adjust log volume

This is the setting to reach for if collection is slow or times out, or if the archive is too large to attach.

supportBundle:
  logs:
    maxLines: 3000
    maxBytes: 2000000

Logs are collected one namespace at a time, with no label selector, so these caps apply to every container in the namespace. They are enforced by the Kubernetes API rather than after the fact, so a capped log is never transferred.

They are also the only thing bounding the collection: a request covering a whole namespace runs under a hard 60 second timeout, and each container is fetched twice (its current and previous instance). Lower maxBytes first if a collection times out; raise both when support asks for more history.

Report on your TLS certificate and public endpoints

supportBundle:
  certificateSecrets:
    - name: gooddata-cn-tls
      namespaces: [gooddata-cn]
  organizationHostnames:
    - analytics.example.com

certificateSecrets reports expiry, SANs and issuer, reading only the public half of the certificate. Each entry in organizationHostnames gets an HTTPS request from the machine running the collection, which exercises public DNS, the load balancer, the ingress and the certificate in one step. A 401 there is a good result: it proves the request reached the gateway.

Pulsar in its own namespace, or in yours

Pulsar’s pod logs and its Helm values and upgrade history are collected automatically, and the namespace is worked out from the broker host your services are already configured with (service.pulsar.host). All the usual layouts work without configuration:

The collector selects Pulsar’s pods by their release label, derived from the broker service name, so it finds your Pulsar whatever the release is called and picks up only its pods when the namespace holds anything else.

A host that is not in-cluster DNS – a managed Pulsar at pulsar.example.com, or an IP address – means Pulsar is not in this cluster, so its collectors are skipped rather than pointed at a namespace that does not exist. If Pulsar is in the cluster but reached through a name like that, name the namespace yourself:

supportBundle:
  pulsar:
    namespace: pulsar

Pulsar always gets its own collector, selected by its release label, so it stays separate from GoodData.CN’s even when the two share a namespace. To skip Pulsar’s logs or its Helm release:

supportBundle:
  pulsar:
    collectLogs: false
    collectHelmRelease: false

Check DNS resolution

The bundle can check DNS resolution, which separates a broken CoreDNS from a broken upstream resolver. It is off by default: the check runs a pod, troubleshoot always creates that pod in the default namespace rather than in the GoodData.CN one, and it does not remove it afterwards. Turn it on when a DNS problem is suspected, or when support asks:

supportBundle:
  dns:
    enabled: true

Test connectivity to the backing services

supportBundle:
  probes:
    enabled: true

This runs one short-lived pod that checks PostgreSQL, Pulsar and Redis using the same credentials the services use, injected from your Secrets and never written into the archive. It is off by default because it creates a pod in your cluster; the DNS check above is the only other step that does, and it is off by default too. Unlike that one, this pod is removed when the collection finishes.

Troubleshoot creates that pod itself, so it does not carry the chart’s commonLabels. If an admission policy requires ownership labels on every pod, leave the probes disabled or exempt that pod.

Add your own collectors and redactors

supportBundle.extraCollectors, extraAnalyzers and extraRedactors are appended to the specification verbatim, so anything in the troubleshoot documentation can be added without forking the chart. Use extraRedactors for credential formats specific to your environment.

Use a specification file instead

If you cannot use --load-cluster-specs – an older plugin version, or a workstation that may not list ConfigMaps – write the same specification to a file and pass it directly.

Extract the copy your own cluster published, which needs no editing. Set the two variables to your namespace and Helm release name:

NAMESPACE=gooddata-cn
RELEASE_NAME=gooddata-cn

kubectl -n "$NAMESPACE" get configmap "${RELEASE_NAME}-support-bundle" \
  -o go-template='{{ index .data "support-bundle-spec" }}' \
  > support-bundle-gooddata-cn.yaml
kubectl -n "$NAMESPACE" get configmap "${RELEASE_NAME}-support-bundle" \
  -o go-template='{{ index .data "redactor-spec" }}' \
  > support-bundle-redactors.yaml

Pass both files, so the GoodData-specific redaction rules are applied:

kubectl support-bundle support-bundle-gooddata-cn.yaml \
  --redactors=support-bundle-redactors.yaml

If the ConfigMap is not there – the release is not installed yet, or its chart predates this feature – render the same two documents from the chart instead. helm template prints the whole ConfigMap, so pull each key out of it:

helm template "$RELEASE_NAME" gooddata/gooddata-cn \
  --namespace "$NAMESPACE" \
  --values your-values.yaml \
  --set license.key='<your-license-key>' \
  -s templates/support-bundle.yaml > support-bundle-configmap.yaml

yq '.data."support-bundle-spec"' support-bundle-configmap.yaml \
  > support-bundle-gooddata-cn.yaml
yq '.data."redactor-spec"' support-bundle-configmap.yaml \
  > support-bundle-redactors.yaml

The chart on your cluster does not have to be the one you render from. If your release is older than this feature, pull a current chart and render the specification from it without upgrading anything:

helm pull gooddata/gooddata-cn --untar
helm template "$RELEASE_NAME" ./gooddata-cn \
  --namespace "$NAMESPACE" \
  --values your-values.yaml \
  --set license.key='<your-license-key>' \
  -s templates/support-bundle.yaml > support-bundle-configmap.yaml

If the collection fails

It times out, or takes far too long

Lower the log limits, which is what usually dominates collection time:

supportBundle:
  logs:
    maxLines: 1000
    maxBytes: 500000

client rate limiter Wait returned an error

Despite the wording, nothing is being throttled. Troubleshoot fetches container logs one at a time, and each container costs two requests – for its current and its previous instance. A collector gets 60 seconds in total for every pod it matches, not 60 seconds per pod, and when that runs out the remaining requests report the expired deadline through the rate limiter, which is where the misleading message comes from.

It means a collector was given more containers than it could read in a minute. Lower maxBytes so each log takes less time to transfer. Note the failed containers are recorded per container inside the archive, in files named *-errors.json, and the run still exits successfully – so a bundle can look complete while individual logs are missing.

Advice you may find elsewhere to raise qps and burst in the kube-proxy-config ConfigMap does not apply: that is kube-proxy’s own client configuration, unrelated to the tool collecting the bundle, which already runs at 100 QPS.

Some collectors report permission errors

The collection continues and produces a bundle regardless; the missing pieces are recorded in the archive. Grant the missing read permissions and re-run if support asks for something that was skipped. Collecting from a namespace that does not exist is harmless – it produces nothing and no error.

Send the bundle to support

Attach the archive when you open a case at GoodData Support. Include when the problem started, which organizations or workspaces are affected, and what changed before it started, such as the last helm upgrade, a configuration edit, or an infrastructure change.