AI Assistant Evaluation

Check whether the Assistant produces the expected results, identify responses that need improvement, and measure changes in quality over time using the gooddata-eval Python-based command-line tool.

You can evaluate visualizations, analytical answers, and selected AI Assistant skills. Evaluation results include pass/fail checks, quality scores, and response times. You can also compare models or repeat the same question set after changing your AI configuration.

Evaluations provide evidence about the questions you test. They do not guarantee that every answer to every future question will be correct.

This guide describes gooddata-eval version 1.76.0. Commands, evaluation behavior, and report fields may differ in other versions.

How Evaluations Work

An evaluation uses a question set containing representative user questions and a description of the expected result. The tool sends each question to the AI Assistant in the workspace you select, examines the response using checks appropriate to the question type, and produces a report.

The general workflow is:

1. Prepare questions that reflect how people use your AI Assistant. Define the expected outcome for each question.
2. Choose the GoodData workspace, AI agent, and model to evaluate.
3. Run the question set. You can repeat individual questions to check whether the result is consistent.
4. Review pass/fail results, quality scores, response times, and individual responses.
5. Improve your workspace’s semantic model, business terminology, AI agent configuration, or questions, as appropriate. Run the same question set again to check the effect.

A question set can be created manually, generated from existing workspace visualizations, or loaded from Langfuse. A generated question set should still be reviewed to make sure it represents actual business questions and has suitable expected results.

When to Run Evaluations

Run evaluations before you make the AI Assistant available to users and whenever a change could affect its answers:

• Before launch: Establish a baseline with representative questions and expected results that business experts have reviewed.
• After semantic model changes: Rerun the same questions after changing metrics, attributes, relationships, names, or descriptions.
• After AI configuration changes: Check the results after changing agent instructions, enabled skills, or AI Memory.
• After model changes: Run the baseline again when switching LLM providers or models, or upgrading a model.
• Before a release and after significant changes: Compare results with the previous baseline and investigate regressions before accepting the change.

Keep the question set and evaluation settings consistent when comparing runs. Add new questions as you discover gaps, but compare like-for-like results when measuring an improvement. Evaluation is one part of release validation and does not replace reviewing real user workflows.

Before You Start

You need:

• A GoodData Cloud organization with an AI Assistant and an LLM provider configured. See Configure AI Assistant.
• Access to the workspace you want to evaluate and an API token with the required permissions for the actions under test.
• Python and the uv package manager to install the command-line tool.
• A question set, or existing visualizations from which to generate one.
• An AI agent ID if you want to test a specific agent configuration. Explicitly selecting an agent is recommended when your organization has more than one agent.

Optional:

For questions scored by an additional language model, such as free-text answers and guardrail responses, you also need the optional llm-judge dependencies and an OpenAI API key. These are separate from the credentials used to access GoodData.

The LLM judge currently uses the OpenAI API, with gpt-4o as its default judge model. This requirement applies even if the AI Assistant you are evaluating uses Amazon Bedrock, Azure Foundry, or another configured LLM provider. The Assistant’s model and the judge model are separate: you can evaluate a non-OpenAI Assistant model, but free-text judging still requires OPENAI_API_KEY. Use --judge-model to select a different supported OpenAI judge model.

Install the Evaluation Tool

Install the standalone command-line tool:

uv tool install gooddata-eval

To add evaluation of free-text answers or guardrail behavior using an LLM judge, install the optional extra instead:

uv tool install 'gooddata-eval[llm-judge]'

If you are adding the package to an existing Python project, use uv add gooddata-eval or uv add 'gooddata-eval[llm-judge]' instead.

Run gd-eval --help to check the commands supported by your installed version.

Run Your First Evaluation

The following example creates evaluation questions from visualizations saved in a workspace. It uses generated, mechanical question wording and does not require a separate LLM for question phrasing.

1. Connect to GoodData

Set your GoodData API token as an environment variable:

export GOODDATA_TOKEN='<GOODDATA_API_TOKEN>'

Use your organization’s URL and a workspace ID in the commands below. Keep the API token out of scripts and shared reports.

2. Review the Available Models

gd-eval models \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>'

This shows the LLM providers and models configured in your organization. When a workspace is specified, the command also identifies its active model. If you do not specify --model in an evaluation, the tool uses that workspace’s active model.

3. Generate a Question Set

Check which visualizations can be converted into evaluation questions:

gd-eval generate \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --dataset-name workspace-quality \
  --no-phrase \
  --dry-run

If the preview looks suitable, generate the files:

gd-eval generate \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --dataset-name workspace-quality \
  --no-phrase \
  --out ./workspace-quality

The tool reads existing workspace visualizations and derives expected visualization configurations from them. It skips definitions it cannot evaluate reliably rather than guessing.

With --no-phrase, the generated question is a simple instruction such as Show Revenue by Region. This option does not require an external LLM or OPENAI_API_KEY. Without --no-phrase, the generator uses an OpenAI model to write a more natural analyst question from the visualization’s metrics, dimensions, and filters. For this, install gooddata-eval[llm-judge] and set OPENAI_API_KEY. You can select the phrasing model with --phrase-model; the default is gpt-4o. In both modes, the expected result is derived from the saved visualization, not written by the LLM.

Generation checks that the dataset has enough coverage. By default, it requires at least 15 questions (--min-questions 15), three distinct visualization shapes (--min-shapes 3), and one filtered question (--min-filtered 1). A small or limited workspace may fail one or more of these checks. You can relax them for an initial test, for example:

gd-eval generate \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --dataset-name workspace-quality \
  --no-phrase \
  --out ./workspace-quality \
  --min-questions 5 \
  --min-shapes 1 \
  --min-filtered 0

Lower thresholds only when the reduced coverage is acceptable. A successful generation does not establish that the question set is sufficiently representative for a production rollout.

Review the questions before running them. In particular, check that the saved visualizations represent correct business logic and that the wording asks for what each visualization actually contains.

4. Run the Question Set

Start with --runs 3 to check whether answers remain consistent across repeated attempts. For important decisions, use 3 to 5 runs per question and review how many attempts passed. A single run is useful for a quick connectivity check, but it is weak evidence of reliability.

Each repetition generates additional AI requests, and some evaluation types also use an LLM judge or simulated follow-up questions. Increasing --runs therefore increases execution time and potentially cost. Start small, then increase the count for critical questions.

gd-eval run \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --dataset ./workspace-quality \
  --runs 3 \
  --json ./evaluation-results.json

To target a particular AI Hub agent, add --agent-id '<AGENT_ID>'. This is important when different agents have different skills enabled. A test of a skill cannot succeed if the selected agent does not have access to that skill.

5. Create an HTML Report

gd-eval report evaluation-results.json -o evaluation-report.html

Open evaluation-report.html in a browser. The report is self-contained and does not require a GoodData server to display it. It shows the overall results and lets you examine individual questions and checks.

You can run the same dataset again after making changes, then compare the resulting reports to detect improvements or regressions.

Create Your Own Question Set

A local question set is a folder containing one JSON file per question. Each item identifies the question, the type of evaluation to perform, and the expected result.

For example, an item for evaluating a written answer can look like this:

{
  "id": "sales-definition-001",
  "dataset_name": "business-questions",
  "test_kind": "general_question",
  "question": "How does our organization define Net Sales?",
  "expected_output": "Explain the Net Sales definition used in this workspace, including the treatment of discounts and returns."
}

This is an illustrative question, not a predefined GoodData metric definition. Replace it with an expectation that is correct for your workspace. Free-text question types require the llm-judge extra and OPENAI_API_KEY to evaluate the response.

Example: Visualization Evaluation Item

Visualization questions use a structured expected result rather than a text answer. The following is a representative, anonymized example of a JSON file that gd-eval generate can produce from a saved bar chart. The metric and label identifiers must exist in your workspace.

{
  "id": "revenue-by-region",
  "dataset_name": "workspace-quality",
  "test_kind": "visualization",
  "question": "Show Revenue by Region",
  "expected_output": {
    "visualization": {
      "id": "revenue-by-region-expected",
      "type": "bar_chart",
      "query": {
        "fields": {
          "revenue": "metric/revenue",
          "region": "label/region"
        }
      },
      "metrics": [
        { "field": "revenue" }
      ],
      "view_by": ["region"]
    }
  }
}

The expected_output.visualization definition specifies the metric, the region breakdown, and the visualization type. More complex generated items can also contain filters, sorting, or segmentation. Treat this example as an illustration, not a substitute for a file generated from your workspace. Use gd-eval generate to obtain the precise structure and references for your package version, and review the resulting JSON before running it.

Different test types have different formats for expected_output. Refer to the gooddata-eval README for the supported formats before writing these items manually.

What Can You Evaluate?

--gate power is available only for supported agentic types that repeat an item K times. agentic_conversation follows its specified turn sequence once, so it does not support this gate. Single-turn types always use pass@K. If you choose --gate power for a dataset containing an unsupported type, the command stops with exit code 2.

Multi-turn types require their own expected-output structure and may use a simulated user for follow-up messages. Changing only the test_kind of a single-turn item may not produce a valid multi-turn test. See the gooddata-eval README for agentic dataset formats.

Understand the Results

The report helps you assess both whether a response meets the expected outcome and how consistently the Assistant produces it.

There is no universal passing score. Agree on an acceptable pass rate for your own question set before running it. For business-critical questions, aim for all expected checks to pass and investigate every failure. A high average quality score does not compensate for failed critical questions, skipped items, or unstable answers. Compare results against your established baseline and inspect the question-level checks rather than relying on one aggregate percentage.

Check Consistency Across Repeated Runs

An AI Assistant can produce different answers to the same question. For a more representative evaluation, repeat questions and review how many runs pass.

• --runs K executes each item multiple times.
• With the default --gate any, an item passes when at least one of its runs passes (pass@K).
• --gate power requires all K runs to pass (pass^K). It works only with agentic test kinds: vis_agentic and agentic_*, except agentic_conversation.

These scores answer different questions. An item that passes once out of five runs may satisfy pass@5, but it is not consistently correct.

The seven single-shot test kinds in What Can You Evaluate? always use pass@K. If you use --gate power with a dataset containing a single-shot item or an agentic_conversation item, gd-eval run stops with exit code 2. For this gate, set test_kind to a supported agentic kind in each dataset item, or use --kind <agentic_kind> for items that do not declare their own test_kind.

Unscored or ungraded runs are not evidence of correctness. Review any evaluator errors or ungraded results before drawing conclusions from a pass rate.

Compare Models or Evaluation Runs

To compare multiple configured models on the same question set, repeat --model:

gd-eval run \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --dataset ./workspace-quality \
  --model '<PROVIDER_A>/<MODEL_A>' \
  --model '<PROVIDER_B>/<MODEL_B>' \
  --runs 3 \
  --json model-comparison.json

Use model IDs and provider names returned by gd-eval models. If several providers expose the same model ID, prefix it with the provider name or ID to avoid ambiguity.

To compare results from separate evaluations, create an HTML report from multiple JSON files:

gd-eval report baseline.json updated.json -o comparison.html

Use the same workspace, dataset, and evaluation settings when you want changes in the results to reflect changes to the AI configuration rather than changes to the test itself.

Run Evaluations in CI

You can run the same question set in a CI pipeline before deploying a change. gd-eval run returns exit code 0 when the evaluation completes, even if questions fail, so use the JSON report to enforce your quality threshold.

The following example assumes you have installed gooddata-eval, saved the question set in ./evals, and configured GOODDATA_HOST, GOODDATA_TOKEN, and WORKSPACE_ID as CI environment variables. Use CI secrets for tokens. The 90% threshold is an example, not a GoodData recommendation for every use case.

set -e

gd-eval run \
  --host "$GOODDATA_HOST" \
  --workspace "$WORKSPACE_ID" \
  --dataset ./evals \
  --runs 3 \
  --json evaluation-results.json

python - <<'PY_CHECK'
import json
from pathlib import Path

report = json.loads(Path("evaluation-results.json").read_text(encoding="utf-8"))
models = report.get("comparison") or {}
threshold = 0.90

if not models:
    raise SystemExit("Evaluation failed: no model results found")

for model, results in models.items():
    rate = results.get("pass_rate")
    if not isinstance(rate, (int, float)) or rate < threshold:
        raise SystemExit(f"Evaluation failed: {model} pass rate {rate} is below {threshold:.0%}")
    print(f"Evaluation passed: {model} pass rate {rate:.1%}")
PY_CHECK

The pipeline stops if an evaluated model falls below the threshold. You can add separate checks for skipped items, ungraded runs, or the number of attempts passed when reliability requirements are stricter.

Review and Share Reports Safely

Reports may contain user questions, expected answers, generated responses, and diagnostic details. Review the contents before distributing a report, especially when it uses sensitive business data.

For an HTML report intended for external review, use:

gd-eval report evaluation-results.json -o customer-report.html --redact

The --redact option removes conversation and response identifiers and raw reasoning, and replaces model names with neutral labels. For agentic runs, it also removes transcripts and tool-call details. Questions, pass/fail results, and other evaluation information remain in the report. Redaction does not guarantee that the report contains no sensitive information. Inspect the resulting file before sharing it.

If you use an LLM judge or optional Langfuse integration, evaluation inputs or diagnostics may be sent to the configured external services. Review the relevant provider settings and your organization’s data-handling requirements before enabling these options.

Optional: Track Evaluations in Langfuse

If your organization uses Langfuse, gd-eval can load a question set from a Langfuse dataset and publish evaluation scores and traces as experiment results.

Configure the Langfuse connection using LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, and LANGFUSE_BASE_URL, then use --langfuse-dataset and --langfuse as documented in the package README.

export LANGFUSE_BASE_URL='https://<LANGFUSE_HOST>'
export LANGFUSE_PUBLIC_KEY='<LANGFUSE_PUBLIC_KEY>'
export LANGFUSE_SECRET_KEY='<LANGFUSE_SECRET_KEY>'

gd-eval run \
  --host 'https://<GOODDATA_HOST>' \
  --workspace '<WORKSPACE_ID>' \
  --langfuse-dataset 'workspace-quality' \
  --langfuse \
  --runs 3 \
  --json langfuse-results.json

The dataset must already exist in Langfuse. --langfuse publishes evaluation experiment results. Store the keys securely and use the dataset name from your Langfuse project.

Agentic evaluation types also check for Langfuse credentials automatically. If LANGFUSE_PUBLIC_KEY and LANGFUSE_SECRET_KEY are set in your environment, gd-eval looks for matching traces and writes evaluation scores to them even without --langfuse.

A local --dataset cannot be submitted as a named Langfuse experiment with --langfuse. Use a Langfuse dataset when you need experiment tracking across runs. Langfuse is optional; local JSON and HTML reports do not require it.

For broader usage, reliability, and token-consumption analytics across your organization, see AI Observability Workspace. This is a separate capability from evaluating a known set of questions with gd-eval.

Use Evaluation Results to Improve AI Assistant

A practical evaluation process is to start with a small, representative question set, review the failures, make targeted improvements, and rerun the same questions.

For example, if an evaluation shows that the Assistant repeatedly selects the wrong metric when asked about revenue, review the metric names and descriptions in the semantic model. If a visualization uses the wrong date or attribute, review the expected filters and the way the question is phrased. When the same question passes intermittently, evaluate consistency over several runs instead of relying on a single successful result.

Include questions that cover common tasks, important business definitions, ambiguous terms, expected filters, and requests the Assistant should not fulfill. Have domain experts review both the expected results and a sample of passing responses. A pass rate is only as useful as the questions and criteria used to calculate it.

Clean Up After Skill Evaluations

Tests of metric creation and alert creation can cause the AI Assistant to write real objects in the selected workspace. This applies to both single-turn skill tests and their agentic equivalents. Some agentic evaluators attempt to remove temporary objects after scoring, but you should not assume that every created metric or alert is deleted, particularly if an evaluation fails or is interrupted.

1. Prefer a dedicated test workspace, and record the metrics and alerts or automations that exist before the run.
2. After the run, check the item details in the JSON or HTML report for created-object identifiers, if present. Compare the workspace’s metrics and alerts or automations against your pre-run inventory.
3. Delete any objects created solely for the evaluation using the appropriate GoodData interface or API. Do not delete existing business metrics or alerts. Verify that test alerts will not continue sending notifications.
4. Before rerunning, confirm that temporary objects have been removed and that the test workspace is ready for another run.

Cleanup behavior can differ between evaluation types. Review the current package README and inspect your workspace after tests that can change its contents. The --preserve-failed option preserves failed conversations for inspection; it is not a switch for retaining or removing metrics and alerts.

Current Limitations and Considerations

• Evaluation measures behavior on a defined set of questions, not every possible user request. It is not a certification of production accuracy.
• The command-line tool requires a technical setup. It does not provide in-product question-set editing or one-click evaluation runs.
• Automatically generated question sets are limited to visualization definitions that the generator can represent and evaluate reliably. They are not a substitute for business review.
• Some evaluation types depend on an LLM judge. Judge output may vary, and failed or missing judge responses can leave questions ungraded.
• Some skill evaluations may create workspace objects. Concurrency is restricted for certain agentic test types, but you should still be careful when evaluating write-capable skills in a shared workspace. Follow the cleanup guidance after running write-capable tests.
• Evaluation runs can consume LLM requests and may incur charges from configured model providers, judges, or external observability services.
• gd-eval run may exit successfully even when some questions fail. Its exit code indicates whether the evaluation command completed, not whether the Assistant met your quality target. Examine the results before approving a release or using them as a CI quality gate.

Troubleshooting

Reference

For the current installation instructions, full CLI flags, dataset formats, evaluation criteria, and Langfuse setup, see:

• gooddata-eval on PyPI
• gooddata-eval source and README on GitHub
• GoodData Python SDK