Introduction
BigQuery is a fully managed, serverless data warehouse available on the Google Cloud Platform (GCP). It allows you to perform complex analytics on terabytes of data in real time without managing infrastructureMoEngage <> BigQuery
The MoEngage and BigQuery integration uses MoEngage Google Cloud Storage (GCS) Data Imports and Exports to transfer user and event data to Google BigQuery for advanced analytics.
Integration
Export Events from MoEngage to BigQuery
Step 1: Ensure Your User’s Assigned Role Has Required Permissions
To successfully set up and execute a BigQuery export, your service account or user must have the following permissions based on the action being performed:- To Create a Connection: To establish a new BigQuery connection within the MoEngage App Marketplace, the user must have the BigQuery Data Viewer and BigQuery Job User roles. This allows MoEngage to validate the project and dataset details. For more information, refer here.
- To Perform Exports: To execute the actual data transfer, your service account must be assigned the BigQuery Data Editor role. This grants MoEngage the necessary permissions to read and write data to your specific BigQuery tables.
Step 2: Add a New BigQuery Export in the MoEngage App Marketplace
- For a full list of events and attributes, refer to the Data Exports Glossary.
- The
event\_timeof exported events is always in UTC. - MoEngage does not currently support exporting historical data.
- On the left navigation menu in the MoEngage UI, click App marketplace.
- On the App Marketplace page, search for BigQuery.
- Click the Google BigQuery tile.
- On the Google BigQuery page, go to the Integrate tab and click +Add Export.
- Enter the following details:
Field Required Description Export name Yes Type a name for the BigQuery export. Ensure your export name contains at least three characters. BigQuery connection Yes Select your BigQuery connection. Dataset ID Yes Select the specific dataset in BigQuery where your data will be exported. Table name Optional Enter the name for the export table. If left blank, MoEngage creates a name for you with the prefix moe_events.
Note: To avoid errors, allow MoEngage to automatically create this table.User Properties Optional Select the specific user properties you want to include in the export. Select events Yes Select the events to export to BigQuery. You can choose to export all events, including existing and new events coming into MoEngage, or manually select specific events to export. 
- Click Save export.
Export Process and Frequency
After you save the export, MoEngage creates a destination table in your dataset. New event data is then appended to this same table at hourly intervals: Initial action: MoEngage creates the table when the first export is triggered. Recurring action: MoEngage adds (appends) data to the same table every hour.Sample Table Structure
If you do not specify a table name in the export configuration, MoEngage creates a single table in your dataset using the following format:moe_events_<table_creation_timestamp>
Note: This table is created once per connection. All hourly exports and different event types configured under this connection are consolidated and appended to this specific table.
The overall structure of the table remains fixed for all event types. Any new user property or event attribute that is exported will be included in the user_attributes or event_attributes column, respectively. All events are consolidated in a single table containing necessary information, such as event names and event times, for easy analysis:
| Field | Type | NULL |
|---|---|---|
| app_name | VARCHAR | NOT NULL |
| event_name | VARCHAR | NOT NULL |
| event_code | VARCHAR | NOT NULL |
| event_uuid | VARCHAR | NOT NULL |
| event_time | TIMESTAMP | NOT NULL |
| event_type | VARCHAR | NOT NULL |
| event_source | VARCHAR | NOT NULL |
| uid | VARCHAR | NULL |
| user_attributes_moengage_user_id | VARCHAR | NULL |
| device_attributes_moengage_device_id | VARCHAR | NULL |
| push_id | VARCHAR | NULL |
| email_address | VARCHAR | NULL |
| mobile_number | VARCHAR | NULL |
| event_attributes_app_version | VARCHAR | NULL |
| event_attributes_sdk_version | VARCHAR | NULL |
| event_attributes_platform | VARCHAR | NULL |
| event_attributes_campaign_id | VARCHAR | NULL |
| event_attributes_campaign_name | VARCHAR | NULL |
| event_attributes_campaign_type | VARCHAR | NULL |
| event_attributes_campaign_channel | VARCHAR | NULL |
| event_attributes_readable_campaign_id | VARCHAR | NULL |
| event_attributes_parent_campaign_id | VARCHAR | NULL |
| event_attributes_parent_flow_id | VARCHAR | NULL |
| event_attributes_parent_flow_name | VARCHAR | NULL |
| event_attributes_variation_id | VARCHAR | NULL |
| event_attributes_locale_id | VARCHAR | NULL |
| event_attributes_locale_name | VARCHAR | NULL |
| event_attributes_url | VARCHAR | NULL |
| event_attributes_timestamp | TIMESTAMP | NULL |
| event_attributes_first_session | BOOLEAN | NULL |
| event_attributes_logged_in_status | STRING | NULL |
| event_attributes | JSON | NULL |
| user_attributes | JSON | NULL |
| device_attributes | JSON | NULL |
Frequently Asked Questions
How soon will I start getting the data?
How soon will I start getting the data?
It typically takes up to 30 minutes for the automated data exports to be set up for your account and for you to start seeing data there.
Can I export additional events at a later time?
Can I export additional events at a later time?
Yes, if you need to export new events at a later time, you can always edit your configuration on the App Marketplace and get the new events added to your exports.
Can I export historical data using Automated Data Exports?
Can I export historical data using Automated Data Exports?
As of now, you cannot export historical data using Automated Data Exports. Once configured, you will start seeing data for each event from the time you enabled your exports.
Is there any pricing involved for Automated Data Exports?
Is there any pricing involved for Automated Data Exports?
If your billing plan does not cover this already and you need to get this enabled at a later time, you would need to work with your Customer Success Manager to get this included in your billing plan. Based on the amount of data that MoEngage will place in your Data warehouse/Partner/API on a monthly basis, this could lead to additional charges.
Does Data Exports guarantee the latest user attributes?
Does Data Exports guarantee the latest user attributes?
No, Data Exports is primarily built to export your Events in near real-time, and user attributes in MoEngage are updated asynchronously, so it is currently not possible to guarantee the latest values of user attributes in the exports.