> ## Documentation Index
> Fetch the complete documentation index at: https://moengage.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Databricks Exports

> Export campaign interaction events from MoEngage to your Databricks tables. Configure the integration for scalable event analytics and processing.

# Introduction

[Databricks](https://www.databricks.com/resources/ebook/maximize-your-organizations-potential-data-and-ai?scid=7018Y000001Fi1AQAS\&utm_medium=paid+search\&utm_source=google\&utm_campaign=15418435374\&utm_adgroup=130717555576\&utm_content=ebook\&utm_offer=maximize-your-organizations-potential-data-and-ai\&utm_ad=703594935116\&utm_term=databricks\&gad_source=1\&gclid=CjwKCAjwvr--BhB5EiwAd5YbXpGh4ugWZyyOwJrqbfr-c5cZhP9HVrGMKQmBuZRlVcC_BRLco-zKVxoCmZkQAvD_BwE) is a platform that allows organizations to store, analyze, and process large volumes of structured and semi-structured data in a highly scalable and efficient manner. With its unique architecture, organizations can consolidate their data, perform quick analytics, and gain valuable data-driven insights accessible to all users.

# MoEngage `<>` Databricks

With this integration, you can export data from campaign interaction events to your Databricks tables.

# Integration

<Tip>
  * Ensure you have a Databricks account with relevant permissions to create/edit Databricks instances, roles, and policies. It is optional to create a new user; you must have access to create a new user in your Databricks database.
  * If your system requires IP whitelisting, you must whitelist MoEngage IPs. For more information, refer [here](/user-guide/settings/account/security/ip-whitelisting-in-moengage).
  * Ensure you have a [Databricks Connection](https://partners.moengage.com/hc/en-us/articles/35576430608916-Databricks) on the App marketplace.
  * Exports to Databricks are part of the Streams add-on. Contact your dedicated MoEngage CSM (customer success manager) to enable them for your account.
</Tip>

If not done already, you need to set up a MoEngage `<>` Databricks integration for your account. For more information, refer to the [Databricks Integration](/user-guide/data/exports/events/data-warehouse/databricks-exports#moengage-\<>-databricks) help doc.

## Export Events from MoEngage to Databricks

### Step 1: Ensure Your User's Assigned Role Has Required Permissions

MoEngage will need `WRITE` access to your Databricks database to store data. You have the option to grant these permissions to an existing database user or create a new dedicated database user specifically for MoEngage.

Use the following code to configure write access for data storage in your Databricks database:

```text wrap theme={null}
GRANT USE CATALOG ON CATALOG `<CATALOG_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;
GRANT USE SCHEMA, CREATE TABLE ON SCHEMA `<CATALOG_NAME>`.`<SCHEMA_NAME>` TO `<EMAIL_ID_OF_THE_USER>`;
```

Remember to change the following values:

* `<CATALOG_NAME>` - The name of your catalog.
* `<SCHEMA_NAME>` - The name of your database or schema.
* `<EMAIL_ID_OF_THE_USER>` - The email ID of the user who created the token.

## Step 2: Add a New Databricks Export on the MoEngage App Marketplace

<Info>
  * For the full list of events and attributes, refer to the Data Exports Glossary.
  * The event\_time of the exported events will always be in UTC.
  * As of now, we do not support exporting Historical Data from MoEngage
</Info>

To add a new Databricks export, perform the following steps:

1. On the left navigation menu in the MoEngage dashboard, click **App marketplace**.
2. On the App Marketplace page, search for **Databricks**.
   <Frame>
     <img src="https://mintcdn.com/moengage/ug1DZn3QSotOzs8r/images/databricks.png?fit=max&auto=format&n=ug1DZn3QSotOzs8r&q=85&s=cd68453d59430bb8981e2cacfd6227a2" alt="Databricks" width="2624" height="1346" data-path="images/databricks.png" />
   </Frame>
3. Click the **Databricks** tile.
4. On the Databricks page, go to the **Integrate** tab and click **+Add Export**.
5. Enter the following details:
   | Field                     | Required | Description                                                                                                                                                                                  |
   | ------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
   | **Export name**           | Yes      | Type a name for the Databricks export.                                                                                                                                                       |
   | **Databricks connection** | Yes      | Select your Databricks connection.                                                                                                                                                           |
   | **Export into schema**    | Yes      | Select the schema that MoEngage uses to create new tables and export your events into.                                                                                                       |
   | **User properties**       | Optional | Select the user properties (user properties and device properties) you want to export with your events.                                                                                      |
   | **Select events**         | Yes      | Select the events you want to export to Databricks. You can choose to export all events, including existing and new ones coming into MoEngage, or manually select specific events to export. |

<Frame>
  <img src="https://mintcdn.com/moengage/TKMWGNv2zSN0av3O/images/wxortimagebin.png?fit=max&auto=format&n=TKMWGNv2zSN0av3O&q=85&s=61f0b8768462b6818aba8a0ef445d49e" alt="Wxortimagebin" width="962" height="780" data-path="images/wxortimagebin.png" />
</Frame>

5. Click **Save export**.

MoEngage starts dumping (appending) events to the newly created table every hour.

### Export Process and Frequency

After you save the export, MoEngage creates a destination table in your schema. New event data is then appended to the existing table at hourly intervals:

* **Initial action**: MoEngage creates the table when the first export is triggered.
* **Recurring action**: MoEngage adds (appends) data to the same table every hour.

### Sample Table Structure

MoEngage creates a single table in your schema using the following format:\
`moe_events_<table_creation_timestamp`.

**Note**: This table is created after per connection. All hourly exports and different event types configured under this connection are consolidated and appended to this specific table.

The overall structure of the table remains fixed for all event types. Any new user property or event attribute that is exported will be included in the user\_attributes or event\_attributes column, respectively. All events are consolidated in a single table containing necessary information, such as event names and event times, for easy analysis:

| Field                                     | Type      | NULL     |
| ----------------------------------------- | --------- | -------- |
| app\_name                                 | VARCHAR   | NOT NULL |
| event\_name                               | VARCHAR   | NOT NULL |
| event\_code                               | VARCHAR   | NOT NULL |
| event\_uuid                               | VARCHAR   | NOT NULL |
| event\_time                               | TIMESTAMP | NOT NULL |
| event\_type                               | VARCHAR   | NOT NULL |
| event\_source                             | VARCHAR   | NOT NULL |
| uid                                       | VARCHAR   | NULL     |
| user\_attributes\_moengage\_user\_id      | VARCHAR   | NULL     |
| device\_attributes\_moengage\_device\_id  | VARCHAR   | NULL     |
| push\_id                                  | VARCHAR   | NULL     |
| email\_address                            | VARCHAR   | NULL     |
| mobile\_number                            | VARCHAR   | NULL     |
| event\_attributes\_app\_version           | VARCHAR   | NULL     |
| event\_attributes\_sdk\_version           | VARCHAR   | NULL     |
| event\_attributes\_platform               | VARCHAR   | NULL     |
| event\_attributes\_campaign\_id           | VARCHAR   | NULL     |
| event\_attributes\_campaign\_name         | VARCHAR   | NULL     |
| event\_attributes\_campaign\_type         | VARCHAR   | NULL     |
| event\_attributes\_campaign\_channel      | VARCHAR   | NULL     |
| event\_attributes\_readable\_campaign\_id | VARCHAR   | NULL     |
| event\_attributes\_parent\_campaign\_id   | VARCHAR   | NULL     |
| event\_attributes\_parent\_flow\_id       | VARCHAR   | NULL     |
| event\_attributes\_parent\_flow\_name     | VARCHAR   | NULL     |
| event\_attributes\_variation\_id          | VARCHAR   | NULL     |
| event\_attributes\_locale\_id             | VARCHAR   | NULL     |
| event\_attributes\_locale\_name           | VARCHAR   | NULL     |
| event\_attributes\_url                    | VARCHAR   | NULL     |
| event\_attributes\_timestamp              | TIMESTAMP | NULL     |
| event\_attributes\_first\_session         | BOOLEAN   | NULL     |
| event\_attributes\_logged\_in\_status     | VARCHAR   | NULL     |
| event\_attributes                         | VARIANT   | NULL     |
| user\_attributes                          | VARIANT   | NULL     |
| device\_attributes                        | VARIANT   | NULL     |

# Frequently Asked Questions

<Accordion title="How soon will I start getting the data?">
  It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.
</Accordion>

<Accordion title="Can I export additional events at a later time?">
  Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.
</Accordion>

<Accordion title="Can I export historical data using Automated Data Exports?">
  As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports.
</Accordion>

<Accordion title="Is there any pricing involved for Automated Data Exports?">
  If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.
</Accordion>

<Accordion title="Does Data Exports guarantee the latest user attributes?">
  No, Data Exports is primarily built to export your Events in near real-time and user attributes in MoEngage are updated asynchronously so it is currently not possible to guarantee the latest values of user attributes in the exports.
</Accordion>