direct: Add dashboards support #3725

shreyas-goenka · 2025-10-06T11:24:08Z

Summary

This PR adds support for Databricks Lakeview Dashboards in the direct deployment mode (DATABRICKS_BUNDLE_ENGINE=direct-exp). Dashboards can now be deployed, updated, and deleted directly via API calls without using Terraform.

Key Features

1. Full Dashboard Lifecycle Management

Create: Creates draft dashboards and automatically publishes them with warehouse and credential settings
Update: Updates both dashboard content and publish settings (warehouse_id, embed_credentials)
Delete: Safely trashes dashboards with idempotent handling
Refresh: Fetches dashboard state from both draft and published endpoints in parallel

2. Smart Field Change Detection

The implementation uses FieldTriggers to determine the correct action for each field change:

Update Actions (in-place modifications):

display_name - Dashboard name changes
warehouse_id - SQL warehouse assignment changes
embed_credentials - Credential embedding toggle
serialized_dashboard - Dashboard content/layout changes

Recreate Actions (delete + create):

parent_path - Moving dashboard to different folder requires recreation

Skip Actions (ignored for diff):

Output-only fields: dashboard_id, path, create_time, update_time, lifecycle_state
etag - Skipped for local changes (user edits), but triggers updates for remote drift detection
serialized_dashboard - Skipped for remote drift (etag is sufficient)

3. Advanced Capabilities

Parallel API Calls: Refresh operations fetch both draft and published dashboard states concurrently using goroutines for better performance.

Automatic Parent Directory Creation: If the parent directory doesn't exist, it's automatically created before dashboard creation.

Idempotent Delete: Handles already-trashed dashboards gracefully by checking the dashboard state on 403 errors.

embed_credentials Edge Case: Unlike Terraform, this implementation properly detects changes to the embed_credentials field by fetching published dashboard state separately.

Serialized Dashboard Flexibility: Supports both string and map/object formats for serialized_dashboard in bundle configuration.

4. Comprehensive Test Coverage

Added 5 new acceptance test suites testing field change behavior:

change-name - Tests display_name updates
change-warehouse-id - Tests warehouse assignment updates
change-embed-credentials - Tests credential embedding toggle
change-serialized-dashboard - Tests dashboard content updates
change-parent-path - Tests parent path changes trigger recreate

Each test verifies:

Correct API methods (POST/PATCH/DELETE) are called
Dashboard ID changes (or doesn't change) as expected
Field values are properly updated in remote state

Testing

Existing and new unit and integration tests.

## Why Used in #3725 to provide local test coverage. ## Tests Existing tests pass with slight modifications.

eng-dev-ecosystem-bot · 2025-10-13T07:38:02Z

Run: 18976428901

	Env	❌FAIL	🟨KNOWN	🔄flaky	💚RECOVERED	🙈SKIP	✅pass	🙈skip
💚	aws linux				1	1	356	592
💚	aws windows				1	1	357	591
💚	aws-ucws linux				1	1	477	487
💚	aws-ucws windows				1	1	478	486
🔄	azure linux			3		1	354	591
💚	azure windows				1	1	357	590
💚	azure-ucws linux				1	1	475	486
🔄	azure-ucws windows			3		1	474	485
🔄	gcp linux			12	1	1	343	593
❌	gcp windows	25	1			1	331	592

40 failing tests:

Test Name	aws linux	aws windows	aws-ucws linux	aws-ucws windows	azure linux	azure windows	azure-ucws linux	azure-ucws windows	gcp linux	gcp windows
TestAccept	💚R	💚R	💚R	💚R	🔄f	💚R	💚R	🔄f	💚R	🟨K
TestAccept/bundle/resources/dashboards/detect-change	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/dashboards/detect-change/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/resources/jobs/fail-on-active-runs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/jobs/fail-on-active-runs/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/jobs/shared-root-path	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/jobs/shared-root-path/DATABRICKS_BUNDLE_ENGINE=terraform	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/models/basic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/resources/models/basic/DATABRICKS_BUNDLE_ENGINE=direct	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/run/app-with-job	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S	🙈S
TestAccept/bundle/templates/default-python/combinations/classic	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=direct/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	🔄f	✅p	✅p	✅p	✅p	✅p
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=no/NBOOK=yes/PY=yes	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=no/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/classic/DATABRICKS_BUNDLE_ENGINE=terraform/DLT=yes/NBOOK=yes/PY=no	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/combinations/serverless	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p	✅p
TestAccept/bundle/templates/default-python/combinations/serverless/DATABRICKS_BUNDLE_ENGINE=direct/DLT=yes/NBOOK=no/PY=yes	🙈s	🙈s	✅p	✅p	🙈s	🙈s	✅p	🔄f	🙈s	🙈s
TestAccept/bundle/templates/default-python/integration_classic	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.10	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=direct/UV_PYTHON=3.9	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.11	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	✅p
TestAccept/bundle/templates/default-python/integration_classic/DATABRICKS_BUNDLE_ENGINE=terraform/UV_PYTHON=3.13	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	🔄f	❌F
TestSecretsPutSecretBytesValue	✅p	✅p	🙈s	🙈s	✅p	✅p	✅p	✅p	🔄f	✅p
TestSecretsPutSecretStringValue	✅p	✅p	🙈s	🙈s	✅p	✅p	✅p	✅p	🔄f	✅p
TestSyncIncrementalFileOverwritesFolder	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestSyncIncrementalSyncPythonNotebookToFile	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestExport	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestExportWithFileFlag	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestImportDirDoesNotOverwrite	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestImportDirWithOverwriteFlag	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadDir	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadDir/dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadDir/workspace_files	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadDir/workspace_files_extensions	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadWrite	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerReadWrite/dbfs	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceFilesExtensionsDelete	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceFilesExtensionsRead	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F
TestFilerWorkspaceFilesExtensionsStat	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	✅p	❌F

…ange This merge incorporates changes from PR #3741 which adds an isLocal parameter to FieldTriggers and ClassifyChange methods to differentiate between local changes (user modifications) and remote changes (drift detection). Key changes: - Renamed ActionTypeUnset to ActionTypeUndefined - Updated FieldTriggers to accept isLocal bool parameter - Updated ClassifyChange to accept isLocal bool parameter - Consolidated dashboard's separate FieldTriggersLocal/Remote methods into single FieldTriggers(isLocal bool) method - Updated adapter to store separate trigger maps for local and remote 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <[email protected]>

Resolved conflicts: - bundle/direct/dresources/adapter.go: Merged documentation for FieldTriggers method, combining detailed explanation from HEAD with note about initialization timing from main - acceptance/bundle/resources/dashboards/test.toml: Kept HEAD version with proper warehouse requirements and etag replacements for new dashboard tests

acceptance/bundle/resources/dashboards/detect-change/script

...ce/bundle/resources/dashboards/publish-failure-cleans-up-dashboard/out.dashboardrequests.txt

acceptance/bundle/resources/dashboards/publish-failure-cleans-up-dashboard/script

bundle/direct/dresources/dashboard.go

direct: Add dashboards support

92fac76

shreyas-goenka temporarily deployed to test-trigger-is October 6, 2025 11:24 — with GitHub Actions Inactive

make change detection work

cbf97d3

shreyas-goenka temporarily deployed to test-trigger-is October 6, 2025 15:30 — with GitHub Actions Inactive

shreyas-goenka added 6 commits October 8, 2025 11:35

add tests for dahsboard generate

6e1d4f2

Replace FieldTriggers with FieldTriggersLocal and FieldTriggersRemote

574939c

better comments

d2f2566

remove golangci

bf3b696

update files

7f77809

Merge remote-tracking branch 'origin' into field-trigger-remote

33e06c7

shreyas-goenka mentioned this pull request Oct 8, 2025

Improve dashboards test server to match to the real server #3743

Merged

shreyas-goenka added 6 commits October 9, 2025 10:57

address comments

b59cd5d

address comments

c62217d

address comments

3405b17

better validation

d9d2cf1

address comments

f2720d3

fix lint

046a699

github-merge-queue bot pushed a commit that referenced this pull request Oct 13, 2025

Improve dashboards test server to match to the real server (#3743)

bd47132

## Why Used in #3725 to provide local test coverage. ## Tests Existing tests pass with slight modifications.

merge

0c20964

shreyas-goenka temporarily deployed to test-trigger-is October 13, 2025 07:34 — with GitHub Actions Inactive

shreyas-goenka and others added 4 commits October 13, 2025 10:01

Merge remote-tracking branch 'origin/main' into direct/dashboard

64c6473

tests for all change; improrevements

a0bd138

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 11:18 — with GitHub Actions Inactive

-

dff5d2b

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 11:42 — with GitHub Actions Inactive

-

173c559

shreyas-goenka temporarily deployed to test-trigger-is October 14, 2025 14:44 — with GitHub Actions Inactive

shreyas-goenka added 2 commits October 29, 2025 11:41

fix test

5d94f00

merge

2eb8b86

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 10:42 — with GitHub Actions Inactive

shreyas-goenka requested a review from denik October 29, 2025 10:42

fix test

79163e8

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 10:44 — with GitHub Actions Inactive

fix test

00084c4

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 11:47 — with GitHub Actions Inactive

add omitempty back

6b0422d

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 11:49 — with GitHub Actions Inactive

denik approved these changes Oct 29, 2025

View reviewed changes

fix windows test

ddff1a1

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 12:37 — with GitHub Actions Inactive

Merge remote-tracking branch 'origin' into direct/dashboard

5661607

shreyas-goenka temporarily deployed to test-trigger-is October 29, 2025 13:02 — with GitHub Actions Inactive

shreyas-goenka added 2 commits October 30, 2025 12:36

Merge remote-tracking branch 'origin' into direct/dashboard

af20a88

address comments

d3f360e

shreyas-goenka temporarily deployed to test-trigger-is October 30, 2025 13:36 — with GitHub Actions Inactive

shreyas-goenka added 3 commits October 30, 2025 14:56

Fix some tests

758ff98

fix windows test

5c5990b

merge

a2242f0

shreyas-goenka temporarily deployed to test-trigger-is October 30, 2025 14:49 — with GitHub Actions Inactive

merge

2f971d2

shreyas-goenka temporarily deployed to test-trigger-is October 31, 2025 13:06 — with GitHub Actions Inactive

shreyas-goenka added 3 commits October 31, 2025 15:35

fix other tests

e11fd37

merge

ade72f5

-

8f0c6ad

shreyas-goenka temporarily deployed to test-trigger-is October 31, 2025 14:37 — with GitHub Actions Inactive

merge

8b0c9e2

shreyas-goenka temporarily deployed to test-trigger-is October 31, 2025 14:57 — with GitHub Actions Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

direct: Add dashboards support #3725

direct: Add dashboards support #3725

Uh oh!

shreyas-goenka commented Oct 6, 2025 •

edited

Loading

Uh oh!

eng-dev-ecosystem-bot commented Oct 13, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

direct: Add dashboards support #3725

Are you sure you want to change the base?

direct: Add dashboards support #3725

Uh oh!

Conversation

shreyas-goenka commented Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Key Features

1. Full Dashboard Lifecycle Management

2. Smart Field Change Detection

Update Actions (in-place modifications):

Recreate Actions (delete + create):

Skip Actions (ignored for diff):

3. Advanced Capabilities

4. Comprehensive Test Coverage

Testing

Uh oh!

eng-dev-ecosystem-bot commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

shreyas-goenka commented Oct 6, 2025 •

edited

Loading

eng-dev-ecosystem-bot commented Oct 13, 2025 •

edited

Loading