AI Lake

AI Lake is GoodData’s managed data infrastructure purpose-built for analytics driven by AI. It provides managed storage and compute for the data that powers AI-driven Business Intelligence and Agentic Analytics experiences in GoodData Cloud.

Eligible customers with the AI Lake entitlement can use the public Apache Iceberg interface to ingest data in the open Iceberg table format. The ingested tables can be connected to GoodData Cloud through an AI Lake data source, added to the semantic layer, and used across metrics, visualizations, dashboards, and AI-driven analytics experiences.

Customers who manage their own data pipelines can write to AI Lake from their infrastructure with standard Iceberg-compatible tools. Customers who prefer a managed implementation can work with GoodData Professional Services.

Data Lakes and AI Lake

A data lake is a storage environment for large volumes of structured, semi-structured, and unstructured data. It provides flexible and scalable storage that can be used by different data processing and analytics tools.

AI Lake builds on this concept by combining open-format storage with managed analytical compute and GoodData’s semantic layer. It stores tables in the Apache Iceberg format and serves analytical queries directly over the stored data.

In GoodData product terminology:

• AI Lake is the managed storage and compute service for analytics.
• AI Lake entitlement determines whether an organization is eligible to use AI Lake.
• AI Lake data source connects tables stored in AI Lake to a GoodData Cloud workspace.

Why AI Lake Uses Apache Iceberg

Apache Iceberg is an open table format designed for reliable analytics on data stored in cloud object storage. It provides the following benefits:

• Multi-engine interoperability: Iceberg tables can be accessed by compatible processing engines and data engineering tools. AI Lake currently uses StarRocks as its high-performance compute engine for analytical queries.
• Separation of storage and compute: Data is stored in open Parquet files in cloud object storage, while analytical compute is provided separately.
• ACID transactions: Iceberg supports safe concurrent reads and writes without exposing partial table updates.
• Schema and partition evolution: Supported schema and partition changes can be applied through table metadata without rewriting existing data files.
• Snapshots and time travel: Iceberg keeps table snapshots that can support point-in-time reads and rollback workflows.

The public beta does not yet document every Iceberg operation as a stable supported contract. See Current Limitations.

How AI Lake Works

AI Lake exposes an Apache Iceberg REST Catalog endpoint. Your Iceberg client uses this endpoint for catalog operations such as creating, loading, and updating tables.

A typical workflow is:

1. GoodData provisions AI Lake storage for an eligible organization and provides the storage ID.
2. You create an AI Lake database instance and choose its database ID.
3. AI Lake automatically creates:
   • an Apache Iceberg namespace with the same ID as the database instance
   • a corresponding AI Lake data source in GoodData Cloud
4. You obtain a GoodData API token for the organization.
5. You configure an Apache Iceberg client with the catalog endpoint and token.
6. Your pipeline creates tables in the existing namespace and writes data in the Apache Iceberg format.
7. You add the tables from the automatically created AI Lake data source to the logical data model and use them in the semantic layer.
8. GoodData Cloud workspaces query the data through AI Lake.

The catalog endpoint has the following format:

https://<ORGANIZATION_HOST>/api/v1/ailake/database/<DATABASE_ID>/catalog

Before You Start

You need:

• a GoodData Cloud organization with the AI Lake entitlement
• an AI Lake storage ID provided by GoodData
• a GoodData API token
• the cloud region provided during onboarding
• a supported Apache Iceberg client

GoodData provisions the AI Lake storage. You then use the public API to create database instances in that storage.

The current beta has been validated with:

• PyIceberg
• Apache Spark with the Iceberg Spark runtime

Other clients that support the Iceberg REST Catalog protocol may work, but they have not been validated for the current beta.

Create an AI Lake Database and Data Source

Before you write Iceberg tables, create an AI Lake database instance.

Choose the value of DATABASE_ID. When the database instance is created, AI Lake automatically creates:

• an Iceberg namespace with the same ID
• a corresponding AI Lake data source in GoodData Cloud

Do not create the namespace separately with your Iceberg client. You also do not need to create the AI Lake data source separately.

Set the required variables:

export HOSTNAME="<ORGANIZATION_HOST>"
export TOKEN="<GOODDATA_API_TOKEN>"
export DATABASE_ID="<DATABASE_ID>"
export STORAGE_ID="<STORAGE_ID>"

Create the database instance:

curl "https://${HOSTNAME}/api/v1/ailake/database/instances" \
  -X POST \
  -H "Accept: application/json, text/plain, */*" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${TOKEN}" \
  --data-raw "{
    \"name\": \"${DATABASE_ID}\",
    \"storageIds\": [
      \"${STORAGE_ID}\"
    ]
  }"

After the database is created:

• use DATABASE_ID in the Iceberg REST Catalog URL
• use the same DATABASE_ID as the Iceberg namespace
• create tables directly in this namespace
• use the corresponding AI Lake data source to add the tables to the logical data model

For example, if DATABASE_ID is analytics, create the table as analytics.events. Do not run create_namespace("analytics") or CREATE NAMESPACE analytics, because the namespace already exists.

Write Data with PyIceberg

Install PyIceberg with the required AWS and PyArrow dependencies:

pip install "pyiceberg[pyarrow,s3fs]"

Configure the REST catalog and write sample data:

from pyiceberg.catalog import load_catalog
import pyarrow as pa

database_id = "<DATABASE_ID>"

catalog = load_catalog(
    "gooddata",
    **{
        "type": "rest",
        "uri": f"https://<ORGANIZATION_HOST>/api/v1/ailake/database/{database_id}/catalog",
        "token": "<API_TOKEN>",
        "rest.sigv4-enabled": "false",
        "s3.region": "<AWS_REGION>",
    },
)

catalog.create_table(
    f"{database_id}.events",
    schema=pa.schema(
        [
            ("event_id", pa.int64()),
            ("event_name", pa.string()),
        ]
    ),
)

# Reload the table before the first write.
table = catalog.load_table(f"{database_id}.events")

table.append(
    pa.table(
        {
            "event_id": [1, 2, 3],
            "event_name": ["created", "updated", "completed"],
        }
    )
)

Write Data with Apache Spark

Configure a Spark Iceberg catalog:

spark.sql.catalog.gooddata=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.gooddata.type=rest
spark.sql.catalog.gooddata.uri=https://<ORGANIZATION_HOST>/api/v1/ailake/database/<DATABASE_ID>/catalog
spark.sql.catalog.gooddata.token=<API_TOKEN>
spark.sql.catalog.gooddata.io-impl=org.apache.iceberg.aws.s3.S3FileIO
spark.sql.catalog.gooddata.client.region=<AWS_REGION>
spark.sql.catalog.gooddata.cache-enabled=false
spark.sql.catalog.gooddata.rest-metrics-reporting-enabled=false

Create a table in the existing namespace and insert sample data:

CREATE TABLE gooddata.<DATABASE_ID>.events (
    event_id BIGINT,
    event_name STRING
)
USING iceberg;

INSERT INTO gooddata.<DATABASE_ID>.events
VALUES
    (1, 'created'),
    (2, 'updated'),
    (3, 'completed');

Keep the Spark catalog cache disabled in the current beta so Spark reloads the table before writing to it.

Use the AI Lake Data Source

When you create an AI Lake database instance, GoodData automatically creates the corresponding AI Lake data source.

AI Lake data sources are available only to organizations with the AI Lake entitlement. You cannot create an AI Lake data source manually in the GoodData Cloud UI.

After you write tables to the database through the Iceberg REST Catalog interface, use the automatically created data source to:

• add the Iceberg tables to the logical data model
• define facts, attributes, relationships, and datasets
• create reusable metrics in the semantic layer
• use the modeled data in visualizations, dashboards, AI-driven Business Intelligence, and Agentic Analytics

The database instance, Iceberg namespace, and AI Lake data source are created as part of the same provisioning flow. The database ID determines the namespace that you use when creating Iceberg tables.

Use AI Lake in Existing Pipelines

You can integrate the Iceberg REST Catalog endpoint into orchestration and transformation workflows that run on your infrastructure. For example, you can use Airflow, Dagster, Spark, or custom code to prepare data and write it to AI Lake.

When you use the public interface directly, you manage the pipeline, scheduling, and compute on your infrastructure. GoodData Professional Services can also help with managed ingestion implementations for customers who do not want to operate the pipeline themselves.

Current Beta Scope

The current beta focuses on the public interface for ingestion in the Apache Iceberg format. The documented and validated workflow includes:

• authenticating with a GoodData API token
• creating an AI Lake database instance in storage provisioned by GoodData
• using the automatically created namespace whose ID matches the database ID
• using the automatically created AI Lake data source
• connecting to the public Iceberg REST Catalog endpoint
• creating and loading Iceberg tables
• appending data
• integrating the tables into the semantic layer
• querying the modeled data through GoodData Cloud

Overwrite, delete, and schema evolution behavior is not yet documented as a stable beta contract. Test these operations thoroughly before using them in your pipelines.

Current Limitations

The following limitations apply during the beta:

• AI Lake is available only as a GoodData-hosted capability in GoodData Cloud.
• Your organization must have the AI Lake entitlement.
• GoodData provisions the AI Lake storage and provides its storage ID. There is currently no public API for creating the storage itself.
• You create AI Lake database instances through the public API.
• AI Lake data sources are created automatically with database instances and cannot be created manually in the GoodData Cloud UI.
• Only GoodData-managed storage and catalogs are supported.
• Each AI Lake database instance has an Iceberg namespace with the same ID. Do not create that namespace separately.
• PyIceberg and Spark are the only clients currently validated.
• Streaming and sub-minute change-data-capture ingestion are not supported.
• The public ingestion interface does not itself provide orchestration, scheduling, or source-specific connectors.
• When using the public interface directly, you are responsible for the infrastructure and operation of your pipeline.
• Reload a table before its first write. For Spark, keep the catalog cache disabled.
• The supported schema evolution, overwrite, and delete contract may change.