Overview Sections:
What is Google Cloud Storage (GCS)?
Crisp Google Cloud Storage Connection
Accessing Data in GCS
Crisp Data
Data Schema
Data Sync Timing
Errors
What is Google Cloud Storage (GCS)?
Google Cloud Storage (GCS) is a cloud service for storing objects of all file types and sizes. This platform has been a top three cloud platform for over a decade as it offers scalable and dynamic solutions to data storage. GCS also supports multiple development languages to build application frameworks utilizing structured and unstructured data types.
Crisp Google Cloud Storage Connection
Crisp offers a direct connection to your organization’s GCS account, allowing you to obtain the source and normalized data tables Crisp ingests from your distributors and retailers. To ensure data is securely transferred, Crisp has enabled Google OAuth and Service Account login options.
To get started, you will need a GCS account and have a storage “bucket” created. The GCS bucket is where your data from Crisp will go once the connection is established. For a guide on creating the connection in Crisp, please refer to the connector configuration steps.
Accessing Data in GCS
Unlike storing files on your computer’s hard drive, GCS enables data storage in a different hierarchy. Within the GCS account, data is stored in a bucket. Buckets can be grouped and linked to a project. A bucket is the basic container that holds data; however buckets may not be nested like files in a folder on a PC. Each file in GCS must be contained in a bucket, and each bucket must be associated with a project.
For example, to route to a data object, you would:
- Select the account
- Select the desired project
- Select the desired bucket
- Select the desired file
Users of the Crisp GCS connection will be able to determine which bucket and, if necessary, which destination path Crisp will store the data files in.
Crisp Data
Currently, Crisp allows you to export multiple source tables to your GCS bucket through a batch job sync. To do this, you'll select your desired data tables from your account's source data connections.
Exported Crisp data includes the most recent reports plus up to 2 years' worth of backfill data. Once the data is synced in GCS, you will see a folder per table selected. Each folder contains the name of the table and the timestamp the data was synced. Within those folders are the exported data files for use.
Data Schema
In the GCS connector, Crisp will export your data tables in the file format you've selected during the connector set-up.
Once the files are loaded into the GCS bucket, you can integrate them into your data analytics tools, send the files to on-prem storage, or have them opened by other cloud instances.
Please note that data schema details vary between data table sources. Crisp provides user-specific data schema information in our Knowledge Base. Here, logged-in Crisp users will find up-to-date information on exported column attributes.
Data Sync Timing
After successfully configuring your GCS connector, Crisp initiates a batch job within 10 minutes to export the first dataset that includes backfill data. Overall timing to backfill data for the first time depends on the number of tables requested. However, it should not take more than a few hours.
Once the initial data set is loaded in your cloud storage, recurring batch jobs are run on a schedule thereafter to update the data (typically taking several minutes to complete).
Since source data syncs are scheduled by the data provider, data availability varies based on the portal and ranges from daily, weekly, to monthly reports. Crisp ingests the data to normalize and export to users within six hours of the data's availability.
Errors
When connecting to the GCS connector, there are two alerts to validate a good connection. Under Connection diagnostics, Crisp alerts you if your storage container and write access are working.
After login and the connection to GCS is validated, any unexpected error with the
connections are logged and sent to Crisp engineering for immediate resolution.
Click here to book a demo and learn more about Crisp's GCS Connector