Databricks Connection

Before creating a connection on ABC, you need to do the following preparations on Databricks

To connect your Databricks data warehouse to ABC, it is strongly recommended to connect via a service account. The following operations will guide you on how to create a service account and grant the necessary permissions.

Create a service principal

• Navigate to the "Settings" of your workspace.
• Click on the "Identity and access" tab.
• Select "Service principals" and click "Manage".
• Click "Add service principal".
• Name the service principal "ABC Service Principal".

Create an ABC account group

• Navigate to the "Settings" of your workspace.
• Click on the "Identity and access" tab.
• Select "Groups" and click "Manage".
• Click "Add group" and name the group "ABC Service Group".
• Add "ABC Service Principal" to the group.

Add the ABC account group to the Databricks workspace

• Navigate to the "Settings" of your workspace.
• Click on the "Identity and access" tab.
• Select "Groups" and click "Manage".
• Click on "ABC Service Group", select the "Entitlements" tab.
• Enable "Databricks SQL access" and "Workspace access".

Enable personal access token permissions for the ABC account group

• Navigate to the "Settings" of your workspace.
• Click on the "Advanced" tab.
• Under "Access Control", click "Permission Settings" next to "Personal Access Tokens".
• Select "ABC Service Group" and "Can Use". Add and save.

Create a schema to store ABC cumulative exposure data

ABC will write exposure data into your Databricks SQL Warehouse, and ABC need to have create and write permissions for the specified schema. The following guide will help you create the specified schema.

Create a schema

• Navigate to the Databricks workspace.
• Click on the "catalog" in the right navigation bar.
• Select a catalog, click the "Create Schema" button in the upper right corner.
• Create a name, similar to "schema_for_abc".
• Select "schema_for_abc", click the "Permissions" option in the upper right corner.
• Click "Grant", select "ALL", and grant permissions to "ABC Service Group".

Grant ABC account group data read permissions

Grant read permissions to any catalogs/schemas/tables that ABC will use. These tables are usually behavior tables. The specific permissions are: SELECT, READ_METADATA, USAGE.

Create a connection from ABC to your Databricks SQL Warehouse

Creating a Databricks SQL Warehouse connection requires the following 5 pieces of information

• Server Hostname(3324********511.1.gcp.databricks.com)
• HTTP Path(/sql/1.0/warehouses/443b*********e4bc)
• Access Token(dapi58d93***************216b952)
• Catalog(abc_service_catalog)
• Schema(abc_schema)

Get Server Hostname information

• Navigate to your Databricks workspace.
• Select the "SQL" tab in the upper left corner.
• Click on "SQL Warehouses" in the navigation bar.
• Select the SQL warehouse you want to use for ABC from the list.
• Click the "Connection details" option.
• The Server Hostname information is at the top of the page under "Server hostname". Record the Server Hostname information.

Get HTTP Path information

• Follow the steps above to find the host information.
• Lower down, you will see "HTTP path". Record the HTTP Path information.

Get Access Token information

Databricks requires an access token to connect to the SQL warehouse. Generating an access token for the service principal account we created above is a bit complicated and requires an API call. Please follow the instructions in the document to generate a token: Access Token. Please note that we recommend choosing at least one year of validity for the token, as you need to regenerate the token and update the connection before each token expires. Record the Access Token information.

Get Catalog information

In the previous steps, you created a schema for ABC under a certain catalog. Record the Catalog name.

Get Schema information

In the previous steps, you created a schema for ABC. Record the Schema name.

Register your connection information on ABC

• Log in to ABetterChoice, click on the console in the upper right corner, and go to your space
• Click on the Metrics tab on the left
• Click on the Warehouse Native option, select the Databricks tab, and click set up
• Fill in the above information into the form, click test connection, if the test passes, click save.