Rotate Encryption Keys

GoodData.CN encrypts data source credentials and organization OIDC client secrets with a Google Tink keyset that you supply to the metadata-api service. Rotating that keyset periodically is a standard security practice. For the initial keyset and the Helm values that enable encryption, see Encrypt Metadata and Data Source Credentials.

Rotation is prone to race conditions, because every metadata-api pod must know a key before anything is encrypted with it. Perform the steps in order and do not skip the restarts.

1. Add a new key
2. Promote the new key
3. Re-encrypt existing secrets
4. (Optional) Remove the oldest key

All commands use Tinkey, the Tink key management tool:

curl -fsL https://storage.googleapis.com/tinkey/tinkey-1.9.0.tar.gz | tar xzvf -

Add a New Key

Add a new key to the keyset and propagate the change to all metadata-api services before you promote it.

Steps:

1. Identify all existing keys in the keyset:

   tinkey list-keyset --in live_keyset.json |grep '  key_id:'

   The output may look something like this:

   key_id: 1088398113
   key_id: 343627806

2. Add a new key to the keyset:

   tinkey add-key --key-template AES256_GCM --in live_keyset.json --out added_key_keyset.json

3. Verify by listing the keys again:

   tinkey list-keyset --in added_key_keyset.json |grep '  key_id:'

   The output may look something like this:

   key_id: 1088398113
   key_id: 343627806
   key_id: 1292234114

   where key_id: 1292234114 is the new key.

4. Distribute added_key_keyset.json to your GoodData.CN deployments by updating the Secret referenced by metadataApi.encryptor.existingSecret, and make sure all metadata-api pods restart afterwards.

Promote the New Key

Promoting a key makes it the primary key used for encryption. Do this only after every metadata-api pod has restarted with the new key present.

tinkey promote-key --key-id <RECENTLY_ADDED_KEY_ID> --in added_key_keyset.json --out rotated_keyset.json

Replace <RECENTLY_ADDED_KEY_ID> with the ID of the key you just added (for example, 1292234114).

Distribute rotated_keyset.json the same way and restart all metadata-api pods again.

Re-encrypt Existing Secrets

With the new primary key in place, secrets stored earlier are still encrypted with the previous key. Re-encrypt them by calling the reencrypt actuator endpoint on the metadata-api pod:

kubectl exec <pod> -n <namespace> -- /bin/bash -c "curl -X POST -v http://localhost:9008/actuator/reencrypt"

For example:

kubectl -n gooddata-cn exec -c metadata-api deployments/gooddata-cn-metadata-api -- curl -X POST -v http://localhost:9008/actuator/reencrypt

(Optional) Remove the Oldest Key

The keyset grows with every rotation, so you may want to drop keys you no longer need. Before you do:

• Confirm the key does not encrypt any secret, which means the re-encryption in the previous step completed.
• Remember that SQL-level database backups taken while the key was in use become unusable once the key is gone.

tinkey delete-key --key-id <OLDEST_KEY_ID> --in rotated_keyset.json --out removed_old_key_keyset.json

Distribute the resulting keyset and restart the metadata-api pods.

Inspect a Keyset

To review the keys in a keyset and see which one is primary:

tinkey list-keyset --in keyset.json

primary_key_id: 343627806
key_info {
  type_url: "type.googleapis.com/google.crypto.tink.AesGcmKey"
  status: ENABLED
  key_id: 1088398113
  output_prefix_type: TINK
}
key_info {
  type_url: "type.googleapis.com/google.crypto.tink.AesGcmKey"
  status: ENABLED
  key_id: 343627806
  output_prefix_type: TINK
}