Distributed Tracing

All GoodData.CN backend services are instrumented for distributed tracing, and you configure it entirely through Helm values. There is no single on/off switch: the default state differs by service group.

• Kotlin microservices and the API Gateway are off until you set monitoring.tracing.enabled: true. That flag defaults to false.
• Python microservices are governed by monitoring.tracing.pythonOtelExporterType, which defaults to "zipkin". Most of them therefore start exporting spans as soon as they run, whatever monitoring.tracing.enabled is set to. Set the value to "" if you do not want that.

How It Works

Backend services fall into three groups. Each group has its own transport and its own endpoint values key:

The chart assembles each endpoint from the host, namespace, and port you configure:

Separately, web browsers can send traces over OTLP HTTP directly to a publicly reachable endpoint that you expose. The chart only distributes that URL to the browser; the traces do not pass through the API Gateway. See Browser Trace Ingestion.

Any backend that accepts these formats works. The reference Terraform deployment sends every group straight to Grafana Tempo, which has native OTLP, Zipkin, and Jaeger receivers, so no separate collector is required. If you already run a collector such as Grafana Alloy or the OpenTelemetry Collector, point the values at it instead.

flowchart LR
    browser["Web browser"]
    kotlin["Kotlin services"]
    gw["API Gateway (api-gw)"]
    python["Python services"]

    otlp_http["OTLP HTTP receiver 4318"]
    zipkin["Zipkin receiver 9411"]
    otlp_grpc["OTLP gRPC receiver 4317"]

    tempo["Tempo"]
    grafana["Grafana"]

    kotlin -->|"exporter: otlp (default)"| otlp_http
    kotlin -->|"exporter: zipkin"| zipkin
    gw -->|"OTLP gRPC"| otlp_grpc
    python -->|"pythonOtelExporterType: otlp-grpc"| otlp_grpc
    python -->|"pythonOtelExporterType: zipkin"| zipkin
    browser -->|"otel.public.collectorUrl"| otlp_http

    otlp_http --> tempo
    zipkin --> tempo
    otlp_grpc --> tempo
    tempo --> grafana

Kotlin Microservices and the API Gateway

Enabling

Set monitoring.tracing.enabled: true in your values.yaml:

monitoring:
  tracing:
    enabled: true

This turns on tracing in every Spring Boot service and enables the OpenTelemetry trace exporter in the API Gateway. When the flag is false, the API Gateway runs with OTEL_TRACES_EXPORTER=none.

Choosing the Exporter Format

monitoring.tracing.exporter selects the wire format for the Kotlin/Spring Boot services. It accepts otlp (the default) or zipkin. Use zipkin only for backward compatibility with an existing Zipkin-only collector.

monitoring:
  tracing:
    enabled: true
    exporter: "otlp"   # or "zipkin"

The setting does not affect the API Gateway, which always exports OTLP gRPC to monitoring.tracing.otel.grpc.

Configuration Reference

monitoring:
  tracing:
    # Master switch for Kotlin services and the API Gateway
    enabled: false

    # Wire format for Kotlin/Spring Boot services: "otlp" or "zipkin"
    exporter: "otlp"

    # Exporter for Python services, independent of "enabled" above.
    # "", "zipkin", "otlp-grpc", or "otlp-http". See Python Microservices.
    pythonOtelExporterType: "zipkin"

    # OTLP HTTP receiver — Kotlin services when exporter is "otlp"
    # Resolved endpoint: http://<host>.<namespace>:<port>/v1/traces
    otlp:
      host: alloy-cluster     # Kubernetes service name of your trace backend
      namespace: monitoring   # Kubernetes namespace of your trace backend
      port: 4318

    # Zipkin receiver — Kotlin services when exporter is "zipkin",
    # and Python services when pythonOtelExporterType is "zipkin"
    # Resolved endpoint: http://<host>.<namespace>:<port>/api/v2/spans
    zipkin:
      host: alloy-cluster
      namespace: monitoring
      port: 9411

    otel:
      # OTLP gRPC receiver — API Gateway, and Python services when
      # pythonOtelExporterType is "otlp-grpc"
      grpc:
        host: alloy-cluster
        namespace: monitoring
        port: 4317

      # Public OTLP HTTP URL — traces sent by web browsers, and by Python
      # services whose exporter type is "otlp-http".
      # Must be a publicly reachable HTTPS URL (exposed via ingress).
      # Leave empty to disable browser trace ingestion.
      public:
        collectorUrl: ''

The chart defaults (host: alloy-cluster, namespace: monitoring) assume a Grafana Alloy release named alloy-cluster in the monitoring namespace. If you send traces straight to Tempo, override host and namespace on every block your deployment uses. See Full Example.

Per-Service Override

To disable or reconfigure tracing for an individual Kotlin service, override the monitoring block at the service level using the camelCase service key:

monitoring:
  tracing:
    enabled: true        # on for all Kotlin services ...

metadataApi:
  monitoring:
    tracing:
      enabled: false     # ... except metadata-api

Any field under monitoring.tracing can be overridden per service this way.

Python Microservices

Python services are instrumented with the OpenTelemetry Python SDK and support multiple exporter backends. Their trace exporter is controlled independently from Kotlin services, by monitoring.tracing.pythonOtelExporterType.

This group covers gen-ai, gen-ai-metadata-sync, ai-lake (service and worker), tabular-exporter, agentic-workflows, mcp-server, and the Quiver nodes (cache, xtab, ml, datasource, policy).

Exporter Types

Enabling

The default value is "zipkin". Most Python services export as soon as pythonOtelExporterType is non-empty, whether or not monitoring.tracing.enabled is set, so they attempt to reach a collector even when Kotlin tracing is off. Quiver nodes are the exception: they also require monitoring.tracing.enabled: true.

Configure both flags together:

monitoring:
  tracing:
    enabled: true                    # Kotlin services, API Gateway, Quiver nodes
    pythonOtelExporterType: "zipkin" # Python services — "zipkin" is already the default

To disable Python tracing explicitly (for example, before a collector is available):

monitoring:
  tracing:
    pythonOtelExporterType: ""

Switching to OTLP gRPC

To have the API Gateway and all Python services use a single OTLP gRPC receiver:

monitoring:
  tracing:
    enabled: true
    pythonOtelExporterType: "otlp-grpc"
    otel:
      grpc:
        host: tempo
        namespace: observability
        port: 4317

Per-Service Override

Python services use a different override key from Kotlin services. Set otelExporterType directly under the service’s camelCase key:

monitoring:
  tracing:
    pythonOtelExporterType: "zipkin"  # default for all Python services

genAi:
  otelExporterType: "otlp-grpc"      # override for gen-ai only

Supported service keys: genAi, genAiMetadataSync, aiLake, tabularExporter, agenticWorkflows, agenticWorkflowsPilot, mcpServer, and quiver.

Browser Trace Ingestion

To collect traces sent by web browsers, set monitoring.tracing.otel.public.collectorUrl to the public HTTPS URL of an OTLP HTTP endpoint:

monitoring:
  tracing:
    enabled: true
    otel:
      public:
        collectorUrl: "https://collector.example.com"

The chart distributes this URL to browsers, so it must be reachable from your users’ network. You are responsible for:

• Exposing an OTLP HTTP endpoint (port 4318) through an ingress with TLS termination, fronted by a proxy or collector that authenticates and rate-limits the traffic
• Allowing CORS requests from your GoodData deployment’s domain on that endpoint

Browser ingestion is not enabled in the reference Terraform deployment; otel.public.collectorUrl is left empty there.

The following ingress shows the routing shape only. Put your authenticating proxy or collector behind backend.service, not Tempo itself. Set ingressClassName to the class your cluster uses, such as alb on AWS with the AWS Load Balancer Controller, or omit the field entirely when you route through an Istio gateway:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: otlp-http
  namespace: observability
spec:
  ingressClassName: alb
  tls:
    - hosts:
        - collector.example.com
      secretName: otlp-http-tls
  rules:
    - host: collector.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              # An authenticating proxy or collector that forwards to Tempo.
              # Do not name the Tempo service here.
              service:
                name: otlp-gateway
                port:
                  number: 4318

Full Example

The following values match what the reference Terraform deployment applies when you set enable_observability = true. Every group sends traces to Tempo in the observability namespace:

monitoring:
  tracing:
    enabled: true                       # Kotlin microservices + API Gateway
    pythonOtelExporterType: "otlp-grpc" # Python microservices
    otlp:
      host: tempo                       # Kotlin services -> http://tempo.observability:4318/v1/traces
      namespace: observability
    otel:
      grpc:
        host: tempo                     # api-gw and Python services -> http://tempo.observability:4317
        namespace: observability
        port: 4317

# Not tracing, but applied alongside it: scrape every service's metrics
podMonitor:
  enabled: true

Two defaults are deliberately left alone: monitoring.tracing.exporter stays at otlp, and monitoring.tracing.otlp.port stays at 4318, which is where Tempo’s OTLP HTTP receiver listens.

Setting Up Grafana Tempo

Tempo accepts OTLP, Zipkin, and Jaeger natively, so it can receive traces from all three service groups without a collector in front of it.

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# tempo-values.yaml
tempo:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    zipkin:
      endpoint: 0.0.0.0:9411
  # Trace retention enforced by the compactor. The PVC size below still caps
  # total storage, so traces may be evicted before this period elapses.
  retention: 168h
  # Tempo's defaults (15 MB/s rate, 20 MB burst) reject trace pushes during
  # peak windows with "RATE_LIMITED: ingestion rate limit ... exceeded".
  overrides:
    defaults:
      ingestion:
        rate_limit_bytes: 30000000
        burst_size_bytes: 45000000

persistence:
  enabled: true
  size: 5Gi

helm install tempo grafana/tempo \
  --namespace observability \
  --create-namespace \
  -f tempo-values.yaml

The release name tempo in namespace observability produces the Kubernetes service tempo.observability, which is what the values in Full Example point at.

Using a Collector Instead

Put Grafana Alloy or the OpenTelemetry Collector between the services and Tempo when you need fan-out to several backends, tail sampling, or attribute processing. Enable the OTLP gRPC (4317), OTLP HTTP (4318), and Zipkin (9411) receivers on the collector, forward everything to Tempo, and point monitoring.tracing.otlp, monitoring.tracing.zipkin, and monitoring.tracing.otel.grpc at the collector’s service instead of at Tempo. The chart’s default hosts (alloy-cluster in namespace monitoring) already assume this layout.

Visualizing Traces in Grafana

Add Tempo as a Grafana data source at http://tempo.observability.svc.cluster.local:3200, then use the Explore view to search traces by service name, trace ID, or duration.

To jump from a span to its logs, configure tracesToLogsV2 on the Tempo data source with a custom query. Grafana’s default tag mapping does not work here: every GoodData.CN microservice logs into a single Loki stream keyed by namespace, while spans carry per-service service.name values, so no span tag lines up with a Loki label. Instead, scope the query to your GoodData.CN namespace and match on the trace ID and span ID, which every service writes into its structured log lines as "traceId":"<id>","spanId":"<id>":

jsonData:
  nodeGraph:
    enabled: true
  tracesToLogsV2:
    datasourceUid: loki
    spanStartTimeShift: "-5m"
    spanEndTimeShift: "5m"
    filterByTraceID: false
    filterBySpanID: false
    customQuery: true
    # Replace gooddata-cn with your GoodData.CN namespace.
    # The doubled $$ escapes Grafana's provisioning-time variable expansion,
    # so the data source interpolates the IDs at query time.
    query: '{namespace="gooddata-cn"} |= "$${__span.traceId}" |= "$${__span.spanId}"'

Matching the span ID as well as the trace ID narrows Logs for this span to the clicked span’s service and operation rather than returning the whole trace.