Backup, Restore, and Disaster Recovery

Back up a production GoodData.CN deployment, restore it, roll back a failed upgrade, and recover from a full loss of infrastructure. This page covers the platform state that lives outside the Helm chart.

What to Back Up

The metadata PostgreSQL databases are the source of truth. Everything else is derived, rebuildable, or already in Git.

How to Back Up

Recommended: Managed Database Snapshots

With an external PostgreSQL instance (Amazon RDS, Azure Database for PostgreSQL Flexible Server, and similar), make the provider’s native backups your primary method:

• Enable automated snapshots with a retention window that meets your recovery objectives.
• Enable point-in-time recovery (PITR) to restore to a specific moment.

Snapshots are consistent across all databases on the instance and need no downtime.

Alternative: Logical Backup With pg_dump

If you cannot use snapshots, dump each database from a pod or host with the PostgreSQL client tools and network access to the instance:

export PGPASSWORD='<POSTGRES_PASSWORD>'
SOURCE_HOST=<POSTGRES_HOST>

pg_dump -h $SOURCE_HOST -p 5432 -U postgres -d md -Fc -v > /tmp/md.dump
pg_dump -h $SOURCE_HOST -p 5432 -U postgres -d automation -Fc -v > /tmp/automation.dump
pg_dump -h $SOURCE_HOST -p 5432 -U postgres -d genai -Fc -v > /tmp/genai.dump   # skip if you do not deploy the service
pg_dump -h $SOURCE_HOST -p 5432 -U postgres -d dex -Fc -v > /tmp/dex.dump       # skip if you do not deploy the service
pg_dump -h $SOURCE_HOST -p 5432 -U postgres -d gw -Fc -v > /tmp/gw.dump

Skip optional databases you do not deploy: genai requires deployGenAIService, dex requires deployDexIdP. gw is always present, because the core gateway-forge service owns it.

Object Storage

Of the three buckets, only quiver-datasource-fs holds data that cannot be regenerated: the files that users uploaded as file-based data sources. If you rely on file data sources, copy that bucket to a backup location on the same schedule as the database backup:

# AWS
aws s3 sync s3://<DATASOURCE_FS_BUCKET_NAME> s3://<BACKUP_BUCKET_NAME>/quiver-datasource-fs

# Azure
az storage blob copy start-batch \
  --source-account-name '<AZURE_STORAGE_ACCOUNT_NAME>' \
  --source-container quiver-datasource-fs \
  --account-name '<BACKUP_STORAGE_ACCOUNT_NAME>' \
  --destination-container quiver-datasource-fs \
  --auth-mode login

start-batch returns before the server-side copy finishes. Re-run this check until it prints nothing, then confirm the destination container holds the same blobs as the source:

az storage blob list \
  --account-name '<BACKUP_STORAGE_ACCOUNT_NAME>' \
  --container-name quiver-datasource-fs \
  --auth-mode login \
  --query "[?properties.copy.status=='pending'].name" -o tsv

The exports bucket holds transient artifacts, and the quiver-cache bucket regenerates automatically, so neither one needs a backup.

Before Every Upgrade

An upgrade can migrate database schemas, so the pre-upgrade backup is the only way back to the previous version. Before you run helm upgrade:

1. Take a full backup of every database GoodData.CN uses internally (md, automation, etc.). On managed PostgreSQL providers, taking a snapshot of the instance is the most convenient method. If you run PostgreSQL yourself, use a pg_dump of every database.

2. Read the release notes and Upgrade GoodData.CN for the target version and perform any extra steps they require.

3. Record the current Helm revision and chart version so you can roll back to it:

   helm history gooddata-cn -n gooddata-cn

If the upgrade causes a serious issue, follow Roll Back a Failed Upgrade.

How to Restore

Restore into a namespace that is not serving traffic. These steps restore the metadata databases and redeploy the chart from your Git-managed values.

1. Provision or restore PostgreSQL. Restore the snapshot or PITR to a new or existing instance, or provision an empty instance and create the databases md, automation, genai, dex, and gw.

2. Recreate the namespace and secrets from your secure backup:

   kubectl create namespace gooddata-cn
   kubectl -n gooddata-cn create secret generic gdcn-encryption --from-file=keySet=gdcn-keyset.json
   kubectl -n gooddata-cn create secret generic gdcn-license --from-literal=license='<YOUR_GDCN_LICENSE_KEY>'

3. Scale every deployment in the namespace to zero. Skip this step if nothing is running in the target namespace yet. Stopping everything guarantees that no service writes to the databases during the restore:

   kubectl scale deployment --all --replicas=0 -n gooddata-cn

4. Restore the databases. Skip this step if you restored a managed snapshot into a fresh instance. Restore each dump into its database:

   export PGPASSWORD='<POSTGRES_PASSWORD>'
   TARGET_HOST=<POSTGRES_HOST>
   
   pg_restore -h $TARGET_HOST -p 5432 -U postgres -d md -v /tmp/md.dump
   pg_restore -h $TARGET_HOST -p 5432 -U postgres -d automation -v /tmp/automation.dump
   pg_restore -h $TARGET_HOST -p 5432 -U postgres -d genai -v /tmp/genai.dump   # skip if you do not deploy the service
   pg_restore -h $TARGET_HOST -p 5432 -U postgres -d dex -v /tmp/dex.dump       # skip if you do not deploy the service
   pg_restore -h $TARGET_HOST -p 5432 -U postgres -d gw -v /tmp/gw.dump

   These commands assume empty target databases. If the targets still contain objects (for example during a rollback), drop and recreate each database before restoring. Do not rely on pg_restore --clean --if-exists for this: it removes only objects that exist in the archive, so anything created after the backup survives it.

   for db in md automation genai dex gw; do   # skip genai/dex if you do not deploy those services
     psql -h $TARGET_HOST -p 5432 -U postgres \
       -c "DROP DATABASE IF EXISTS $db WITH (FORCE);" \
       -c "CREATE DATABASE $db;"
   done

   A managed snapshot restore or point-in-time recovery replaces the whole instance and makes this step unnecessary. Use --clean --if-exists only when you know the target schema matches the archive.

5. Restore the uploaded data source files. Skip this step if you do not use file-based data sources. Copy the backed-up quiver-datasource-fs contents into the bucket that the restored deployment points at:

   # AWS
   aws s3 sync s3://<BACKUP_BUCKET_NAME>/quiver-datasource-fs s3://<DATASOURCE_FS_BUCKET_NAME>
   
   # Azure
   az storage blob copy start-batch \
     --source-account-name '<BACKUP_STORAGE_ACCOUNT_NAME>' \
     --source-container quiver-datasource-fs \
     --account-name '<AZURE_STORAGE_ACCOUNT_NAME>' \
     --destination-container quiver-datasource-fs \
     --auth-mode login

   On Azure, start-batch returns before the server-side copy finishes; re-run the pending-copies check from How to Back Up against <AZURE_STORAGE_ACCOUNT_NAME> until it prints nothing before you continue.

   The quiver-cache bucket needs no restore, because it rebuilds automatically as queries run. The exports bucket holds transient artifacts, so its contents are safe to lose.

6. Update the database endpoint in your Git-managed values. If the restore produced a new PostgreSQL endpoint, set service.postgres.host to the new hostname before you redeploy, along with service.postgres.port, service.postgres.username, and the password or service.postgres.existingSecret if those changed too. If you skip this, the services come back pointing at the old endpoint.

7. Redeploy the chart from those Git-managed values. The upgrade also returns every deployment to the replica counts in your values files:

   helm upgrade --install gooddata-cn gooddata/gooddata-cn \
     --namespace gooddata-cn \
     --version "<GOODDATA_CN_VERSION>" \
     --values gdcn-values.yaml \
     --values gdcn-values-sizing.yaml \
     --wait --timeout 30m

8. Re-apply your Organization custom resources from Git if they are missing.

9. Verify. All pods reach Ready, login works, dashboards load. The quiver-cache rebuilds automatically, so the first queries after a restore are slow.

Roll Back a Failed Upgrade

If an upgrade causes a serious issue, return the deployment to the previous version using the backup you took before the upgrade. Run the steps in this order:

1. Scale every deployment in the namespace to zero replicas:

   kubectl scale deployment --all --replicas=0 -n gooddata-cn

2. Restore the pre-upgrade database backup. Restore the snapshot you took before the upgrade, for example with an RDS snapshot restore, or restore the logical dumps as described in How to Restore. The databases must go back to their pre-upgrade schema, because the new chart version may have migrated them.

3. Roll the Helm release back to the pre-upgrade version. Use the revision number from helm history:

   helm rollback gooddata-cn <PRE_UPGRADE_REVISION> --namespace gooddata-cn --wait --timeout 30m

   If you restored the database to a new endpoint instead of in place, helm rollback replays the values stored with the old revision, which still carry the old service.postgres.* settings. In that case, update service.postgres.host and the other connection settings in your Git-managed values and run helm upgrade with the pre-upgrade chart version and those values, rather than helm rollback.

4. Scale the deployments back up. The rollback re-applies the chart’s replica counts, so most deployments come back on their own. Check with kubectl get deployments -n gooddata-cn and scale up anything still at zero, then verify that pods reach Ready, login works, and dashboards load.

Full Disaster Recovery

If you lose the whole environment (cluster, database, and buckets), recover in this order:

1. Reprovision infrastructure: the Kubernetes cluster, the PostgreSQL instance, the object storage buckets (exports, quiver-cache, quiver-datasource-fs), and the identity/OIDC configuration.
2. Restore the database from the latest snapshot or PITR, or from your logical dumps.
3. Restore the quiver-datasource-fs bucket contents from your object storage backup, using the same aws s3 sync or az storage blob copy start-batch command as in How to Restore. The new buckets are empty, so recreating them is not enough: without this step, every file-based data source breaks. The quiver-cache bucket does not need restoring, because it rebuilds automatically, and the exports bucket only holds transient artifacts.
4. Recreate secrets: the encryption keyset and the license, as in the restore procedure above.
5. Point your Git-managed values at the new infrastructure. Update service.postgres.* for the new database endpoint and the bucket or container names if they changed, and commit the change before you redeploy.
6. Redeploy the chart via GitOps. Apply your Git-managed Helm values and Organization custom resources.
7. Repoint DNS so the organization and auth hostnames route to the new ingress.
8. Validate: pods ready, login works, dashboards load, data sources connect.

Recovery time depends mainly on the database restore and the cache warmup, since the quiver-cache rebuilds lazily as queries run.

Organization-Level (Logical) Backup

The declarative API in Create and Restore Backups captures one organization’s configuration: workspaces, models, and data source definitions, but not secrets. Use it to migrate or clone an organization, or to snapshot before a risky change. It complements infrastructure-level backups but does not replace them: only database backups recover the whole deployment.