Harden the Deployment for Restricted Admission Policies

Both the Apache Pulsar and the GoodData.CN Helm charts install on a cluster that enforces a Pod Security Standards restricted policy set through OPA Gatekeeper. This page gives you the values files for each chart and names the few workloads that still need an exemption.

The configuration below is written against Gatekeeper constraint templates, but the settings are policy-engine agnostic. The same values satisfy equivalent Kyverno policies and the Pod Security Admission restricted controls for runAsNonRoot, allowPrivilegeEscalation: false, dropped capabilities, and seccomp. Labels, probes, group-ID constraints, and readOnlyRootFilesystem are Gatekeeper or Kyverno policy requirements; Pod Security Admission does not enforce them.

Constraints covered

K8sPSPAllowedUsers reads runAsGroup from the container securityContext when the container sets it, and from the pod otherwise, so a per-container override has to stay inside the same range. The full constraint templates are in the Gatekeeper policy library.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sRequiredProbes
metadata:
  name: must-have-probes
spec:
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    labelSelector:
      matchExpressions:
        - key: batch.kubernetes.io/job-name
          operator: DoesNotExist
  parameters:
    probes: ["readinessProbe", "livenessProbe"]
    probeTypes: ["tcpSocket", "httpGet", "exec"]

Before you start

• Apache Pulsar Helm chart 4.8.0 or later.
• GoodData.CN Helm chart 4.13.1 or later.
• A test cluster holding a copy of your data. Two of the settings below change runtime behavior and both are worth rehearsing.

Apache Pulsar

Save the following as pulsar-values-restricted.yaml and pass it after the values files that the Configuration Assistant generated.

# The bundled monitoring stack does not read the settings below, so its pods are rejected
# while Pulsar itself comes up. Deploy monitoring separately, into a namespace whose
# policy suits it.
victoria-metrics-k8s-stack:
  enabled: false

# K8sRequiredLabels. Reaches every rendered resource and every pod template, including the
# init and cleanup Jobs. Kept out of the StatefulSet selectors, which Kubernetes does not
# allow to change after creation. Replace the keys with the ones your policy names.
labels:
  example.com/owner: analytics-platform
  example.com/team: bi-team

# K8sPSPAllowedUsers. Applied to every pod the chart renders.
podSecurityContext:
  runAsNonRoot: true
  # 10000 is the uid of the `pulsar` user in the apachepulsar images.
  runAsUser: 10000
  runAsGroup: 10000
  fsGroup: 10000
  fsGroupChangePolicy: OnRootMismatch
  supplementalGroups: [10000]

# K8sPSPAllowPrivilegeEscalationContainer and K8sPSPReadOnlyRootFilesystem. Applied to
# every container and initContainer. The capabilities and seccomp settings are not
# required by those two constraints; they are what a `restricted` policy set asks for
# elsewhere, and cost nothing here.
containerSecurityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

# zookeeper, bookkeeper, broker and oxia.server ship `securityContext.fsGroup: 0`, and a
# per-component value wins over the global one, so each has to be overridden explicitly.
# `fsGroup` is added as a supplementary group, so it does not have to match `runAsGroup`.
zookeeper:
  securityContext:
    fsGroup: 10000
    fsGroupChangePolicy: OnRootMismatch
bookkeeper:
  securityContext:
    fsGroup: 10000
    fsGroupChangePolicy: OnRootMismatch
broker:
  securityContext:
    fsGroup: 10000
    fsGroupChangePolicy: OnRootMismatch
# Rendered only when the metadata store is Oxia (`components.oxia: true`).
oxia:
  server:
    securityContext:
      fsGroup: 10000
      fsGroupChangePolicy: OnRootMismatch

# K8sRequiredProbes. The autorecovery StatefulSet has no probes by default.
autorecovery:
  probe:
    liveness:
      enabled: true
    readiness:
      enabled: true

The pulsar-values.yaml that the Configuration Assistant generates already sets components.toolset: false, which is what keeps the last probe finding off the list. See Workloads that still need an exemption if you turn the toolset back on.

Install with the extra file last, so its values win:

helm upgrade --install pulsar pulsar/pulsar \
  --namespace pulsar --create-namespace \
  --version "4.8.0" \
  --values pulsar-values.yaml \
  --values pulsar-values-sizing.yaml \
  --values pulsar-values-restricted.yaml \
  --wait --timeout 30m

Writable paths under a read-only root filesystem

The Pulsar images do not run on a read-only root filesystem unaided. They rewrite their configuration under /pulsar/conf on startup, write logs under /pulsar/logs, and the JVM uses /tmp.

Setting containerSecurityContext.readOnlyRootFilesystem: true is still all you need. For each entry in the chart’s emptyDirVolumes list, the chart mounts an emptyDir at that path on every container and initContainer of the component, and for entries marked seedFromImage it prepends a copy-pulsar-conf initContainer that copies the image’s contents into the volume first. The default list describes the Pulsar images:

emptyDirVolumes:
  - path: /pulsar/conf
    seedFromImage: true
    sizeLimit: 1Gi
  - path: /pulsar/logs
    sizeLimit: 1Gi
  - path: /tmp
    sizeLimit: 1Gi

Override it per component with <component>.emptyDirVolumes, which replaces the inherited list rather than extending it. Keep a sizeLimit on each entry: an uncapped emptyDir can fill a node’s ephemeral storage and get the pod evicted.

The autorecovery probes

The autorecovery daemon does not start BookKeeper’s HTTP service, so both probes target the Prometheus stats endpoint on autorecovery.ports.http, the same endpoint the PodMonitor scrapes. If you have overridden statsProviderClass or prometheusStatsHttpPort through autorecovery.configData, that endpoint no longer serves /metrics and the probes break: liveness turns a working daemon into a restart loop, and readiness leaves the pod permanently unready and stalls the StatefulSet rollout. Confirm the endpoint responds before you enable them.

GoodData.CN

With no configuration at all, every container and initContainer that GoodData ships already sets allowPrivilegeEscalation: false, drops all Linux capabilities, and requests the RuntimeDefault seccomp profile, and every long-running workload carries both probes. K8sPSPAllowPrivilegeEscalationContainer and K8sRequiredProbes are satisfied before you change anything.

The bundled qdrant subchart is the exception, and it is rendered only when deployQdrant is true. Its ensure-dir-ownership init container escalates privileges and its main container has no liveness probe. The overlay below closes both.

The rest is opt-in, because of the two runtime changes described in Before you start. Turn it on with restrictedPodSecurity.enabled.

The chart ships a ready-made overlay as values-restricted.yaml. Extract it and use it directly:

helm repo add gooddata https://charts.gooddata.com/
helm repo update
helm pull gooddata/gooddata-cn --version "<GOODDATA_CN_VERSION>" --untar
cp gooddata-cn/values-restricted.yaml gdcn-values-restricted.yaml

Then fill in the label keys your policy requires. The complete file:

# K8sPSPAllowedUsers and K8sPSPReadOnlyRootFilesystem. Sets runAsUser, runAsGroup,
# fsGroup, supplementalGroups and readOnlyRootFilesystem on every GoodData workload.
restrictedPodSecurity:
  enabled: true

# K8sRequiredLabels. Reaches every rendered resource and every pod template, including the
# Jobs, CronJobs and helm tests. Kept out of the Deployment and StatefulSet selectors,
# which Kubernetes does not allow to change after creation.
commonLabels:
  example.com/owner: analytics-platform
  example.com/team: bi-team

# The redis-ha and qdrant subcharts are third-party and are configured through their own
# values. Helm does not template values files, so repeat the label keys here rather than
# referencing the block above.
redis-ha:
  # extraLabels, not `labels`: `labels` renders through toString, so a value like "true"
  # becomes a YAML boolean and the API server rejects the object.
  extraLabels:
    example.com/owner: analytics-platform
    example.com/team: bi-team

  # The subchart already ships runAsUser, fsGroup and runAsNonRoot. These close runAsGroup
  # and supplementalGroups.
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    fsGroup: 1000
    fsGroupChangePolicy: OnRootMismatch
    supplementalGroups:
      - 1000

  containerSecurityContext:
    runAsUser: 1000
    runAsGroup: 1000
    runAsNonRoot: true
    readOnlyRootFilesystem: true
    allowPrivilegeEscalation: false
    seccompProfile:
      type: RuntimeDefault
    capabilities:
      drop:
        - ALL

  # Only the redis and sentinel containers expose a volumeMounts hook. The other four run
  # read-only without a writable path, because nothing in the subchart's scripts writes
  # outside /data. This scratch area is here so they keep one if a future version needs it.
  extraVolumes:
    - name: scratch-tmp
      emptyDir: {}
  redis:
    extraVolumeMounts:
      - name: scratch-tmp
        mountPath: /tmp
  sentinel:
    extraVolumeMounts:
      - name: scratch-tmp
        mountPath: /tmp

# Rendered only when deployQdrant is true.
qdrant:
  # additionalLabels alone, never podLabels: the StatefulSet emits both into the same pod
  # template, so a key set in both renders twice. Helm tolerates the duplicate; a Flux
  # post-renderer rejects the whole release with `mapping key already defined`.
  additionalLabels:
    example.com/owner: analytics-platform
    example.com/team: bi-team

  podSecurityContext:
    runAsNonRoot: true
    runAsUser: 1000
    runAsGroup: 2000
    fsGroup: 3000
    fsGroupChangePolicy: OnRootMismatch
    supplementalGroups:
      - 3000

  # Merged over the subchart's own restricted defaults, which already set
  # readOnlyRootFilesystem, allowPrivilegeEscalation and the UID and GID.
  containerSecurityContext:
    capabilities:
      drop:
        - ALL
    seccompProfile:
      type: RuntimeDefault

  # Drops the ensure-dir-ownership init container, a root `chown -R` that can run neither
  # read-only nor without privilege escalation. Set this only after confirming that every
  # storage, snapshot, and restoration path is writable by UID 1000 or GID 3000 with the
  # required permissions. Test restored PVCs across every storage class you use; fsGroup
  # can set supported volume groups but does not change the file owner UID.
  updateVolumeFsOwnership: false

  # Off upstream, which would leave qdrant with a readiness probe only.
  livenessProbe:
    enabled: true

Install with the restricted overlay first, followed by the generated values files. gdcn-values-sizing.yaml is applied last, so it takes precedence for overlapping settings:

helm upgrade --install gooddata-cn gooddata/gooddata-cn \
  --namespace gooddata-cn \
  --create-namespace \
  --version "<GOODDATA_CN_VERSION>" \
  --values gdcn-values-restricted.yaml \
  --values gdcn-values.yaml \
  --values gdcn-values-sizing.yaml \
  --wait --timeout 30m

Relaxing the profile for one service

Everything the profile sets can be relaxed for a single service without forking the chart. Values under a service name win over the profile and over the chart-wide default:

tools:
  securityContext:
    readOnlyRootFilesystem: false
  scratchDirs:
    - /tmp
    - /var/cache

scratchDirs are mounted as emptyDir into every container of that service. They exist whether or not the restricted profile is on, so enabling it needs no additional writable-volume configuration. Changing the profile updates pod-template security contexts and can roll or recreate affected workloads. The default is /tmp; the nginx-based UI services also get /etc/nginx/conf.d, which their entrypoint regenerates at startup.

Verify before you install

Render each chart with your values and evaluate the output against your own constraints using Gatekeeper’s gator CLI. This catches a mismatch on a laptop instead of during a helm upgrade against a deny cluster:

If your constraints match Pod objects, include matching ExpansionTemplates for batch/v1 Jobs and CronJobs in ./your-constraints/. gator test then expands those controllers into child Pods, so it evaluates their probes too. Without those templates, these commands do not validate the Pods that Jobs and CronJobs create.

set -euo pipefail

helm template pulsar pulsar/pulsar --version "4.8.0" \
  --namespace pulsar \
  --skip-tests \
  --values pulsar-values.yaml \
  --values pulsar-values-sizing.yaml \
  --values pulsar-values-restricted.yaml \
  | gator test --filename=./your-constraints/

set -euo pipefail

helm template gooddata-cn gooddata/gooddata-cn --version "<GOODDATA_CN_VERSION>" \
  --namespace gooddata-cn \
  --skip-tests \
  --values gdcn-values-restricted.yaml \
  --values gdcn-values.yaml \
  --values gdcn-values-sizing.yaml \
  | gator test --filename=./your-constraints/

gator test reads the manifest from stdin and takes the constraint templates and constraints from --filename. It exits non-zero when it finds a violation, so the same command works as a CI gate.

Ask your platform team for the constraint and constraint-template YAML that the cluster actually runs. Evaluating against the library defaults tells you less than evaluating against the parameters your cluster enforces.

Workloads that still need an exemption

Jobs and CronJobs. Some are created during a normal install and are reported by the probe constraint, as described in Constraints covered. Scope the constraint to exclude their Pods rather than trying to add probes.

Helm tests. These run only when you invoke helm test; scope the constraint or skip the tests if they are not required.

The qdrant and redis-ha helm-test Pods. qdrant-db-test-db-interaction and redis-ha-configmap-test are bare Pods whose templates hardcode the pod spec, so no values reach them, and neither subchart offers a flag to skip its tests. Both are test hooks (helm.sh/hook: test on the qdrant Pod, the legacy test-success on the redis-ha one), so they are created only while helm test runs. If you never run helm test, they never reach the admission controller. Under deny mode, exempt them by name or skip helm test.

The Pulsar toolset. The toolset StatefulSet is a shell with pulsar-admin on the path, and the chart exposes no probe settings for it. The Configuration Assistant disables it, so it is off unless you turn it back on. If you need it, set components.toolset: true and exempt that one StatefulSet from the probe constraint. Nothing else in the Pulsar chart or in GoodData.CN depends on it.

Two things that look like gaps and are not. Qdrant’s ensure-dir-ownership init container is a root chown -R that cannot run read-only, but updateVolumeFsOwnership: false removes it entirely. And redis-ha exposes extraVolumeMounts on only its redis and sentinel containers, yet config-init, split-brain-fix, redis-exporter and the clean-failed-dumps init container all run read-only without a writable path.

Related pages

• Configuration Assistant builds the base values files that this page layers on top of.
• Production Installation walks through the full install, including the Pulsar and GoodData.CN steps referenced here.
• Helm Chart Values Reference documents every value the GoodData.CN chart accepts.