What is Microsoft Azure?
Microsoft Azure is a cloud computing platform that offers a range of software, platform and infrastructure as a service solutions. This platform has been a top five cloud platform for over a decade as it offers scalable and dynamic solutions to data storage, including connections to on-premise data warehouses. Azure also supports multiple development languages to build application frameworks utilizing structured and unstructured data types.
What is Azure Blob Storage?
Azure Blob (Binary Large Object) storage enables users to store petabytes of data onto Microsoft’s cloud infrastructure. Additionally, Blob storage offers users the capability to store a variety of data types including traditional dimensional tables as well as data that is considered unstructured such as images, video, and more.
Crisp Azure Blob Storage Connection
Crisp offers a direct connection to a user’s Azure Blob storage account that allows users to obtain Crisp’s source and normalized data tables. To ensure data is securely transferred, Crisp has enabled Azure OAuth token and SAS token login options. To get started, users will need an Azure subscription and have a storage blob container created. That container is where your data from Crisp will go once the connection is established.
For a guide on creating the connection, refer to the Azure Blob Storage connection setup guide.
Accessing Data in Blob Storage
Like storing files on your computer’s hard drive, Azure Blob storage allows users to organize their data. The storage hierarchy starts with the user’s account. Within the account level, a user can have any number of Blob containers. Within a Blob container, a user can have any number of folders and/or source files.
For instance, in the example below, to route to a data object, the user would:
- Select the account
- Select the desired Blob container
- Select the desired file
Users of the Crisp Azure Blob storage connection will be able to determine which container and, if necessary, which folder Crisp will store the data files.
When Crisp sends data to a user’s Blob container, the data will be written inside multiple folders. The number of folders is dependent on the number of source tables selected to be transferred as well as the frequency of the tables needing to be synced.
Crisp Data
Currently, Crisp allows users to export multiple source tables to a user’s Blob container through a batch job sync. Users are able to select source or normalized tables from their source data connections.
Once the data is synced in Blob storage, users will see a folder per table selected. Each folder contains the name of the table and the timestamp the data was synced. Within those folders are the exported csv files for use.
Exported Crisp data includes the most recent reports plus backfill data. Crisp backfills data up to the previous three years to date.
Data Schema
In the Azure Blob storage connector, Crisp will export the user data tables as .csv files which can be opened via Excel, integrated into data analytics tools, sent to on-prem storage or opened by other clouds.
Data schema details vary between data table sources. Crisp provides users specific data schema information in our Knowledge Base. Here, users will find up-to-date information on exported column attributes.
Data Sync Timing
Upon completion of the Azure connector configuration, Crisp initiates a batch job within 10 minutes to export the first dataset that includes backfill data. Overall timing to backfill data for the first time depends on the number of tables requested. However, it should not take more than a few hours. Once the initial data set is loaded on the user’s Blob container, recurring batch jobs are run on a schedule thereafter to update the data and typically only takes several minutes to complete.
Since source data syncs are scheduled by the data provider, data availability varies based on the portal and ranges from daily, weekly to monthly reports. Crisp ingests the data to normalize and exports to users within six hours of the data availability.
Errors
When a user is connecting to the Azure Blob storage connector, there are two alerts to validate a good connection. Under Connection diagnostics, Crisp alerts users if their storage container access is good (green with a check) as well as their write access to Azure is good (green with a check).
After login and the connection to Azure is validated, any unexpected errors with the
connections are logged and sent to Crisp engineering for immediate resolution.
Interested in learning more about this solution? Book a demo with our team to discuss!