What is Amazon S3?
Amazon Simple Storage Service (S3) is Amazon Web Service's (AWS) object cloud storage platform for all file types and sizes. This platform has been a top three cloud storage solution for over a decade as it offers scalable and dynamic data storage and services. S3 also supports multiple development languages to build application frameworks utilizing structured and unstructured data types.
Crisp Amazon S3 Cloud Storage Connection
Crisp offers a direct connection to your organization’s S3 account, allowing you to obtain the source and normalized data tables Crisp ingests from your distributors and retailers. To ensure data is securely transferred, Crisp has enabled the AWS Access Key authentication.
To get started, you will need an Amazon S3 account in an AWS cloud region along with a storage “bucket” created in the cloud region. The S3 bucket is where your data from Crisp will go once the connection is established. For a guide on creating the connection in Crisp, please refer to the connector configuration steps.
Creating an AWS Access Key
Amazon S3 enables user-controlled security measures through the AWS Access Key to your S3 data bucket. Since Crisp does not capture your S3 login, you will need to create an AWS Access Key prior to setting up the Crisp S3 connection.
To create an AWS Access Key, you'll need to log into your S3 account and then navigate to your account management dropdown options. From there, select “My Security Credentials.”
A new page will load. Here, you'll need to select the "Access keys" and then click the “Create access key” button. Download the key file and be ready to use it to connect in Crisp.
Accessing Data in Amazon S3
Unlike storing files on your computer’s hard drive, S3 enables data storage within a specific AWS cloud region. From the defined cloud region (such as AWS East), S3 data is then stored in a bucket. Unlike a folder on your computer, a bucket is the basic container that holds data and may not be nested like files.
For example, to route to a data object, you would:
- Select the AWS region
- Select the desired storage bucket
- Select the desired file
Users of the Crisp S3 connection will be able to determine which bucket and, if necessary, which destination path Crisp will store the data files in.
Currently, Crisp allows you to export multiple source tables to your S3 bucket through a batch job sync. To do this, you'll select your desired data tables from your account's inbound data connections.
Exported Crisp data includes the most recent reports plus up to 2 years' worth of backfill data. Once the data is synced in S3, you will see a folder per table selected. Each folder contains the name of the table and the timestamp the data was synced. Within those folders are the exported data files for use.
In the Amazon S3 outbound connector, Crips will export your data tables in the file format you’ve selected during the connector set-up.
Once the files are loaded into the S3 bucket, you can integrate them into your data analytics tools, send the files to on-prem storage, or have them opened by other cloud instances.
Please note that data schema details vary between data table sources. Crisp provides user-specific data schema information in our Knowledge Base. Here, logged-in Crisp users will find up-to-date information on exported column attributes.
Data Sync Timing
After successfully configuring your S3 connector, Crisp initiates a batch job within 10 minutes to export the first dataset that includes backfill data. Overall timing to backfill data for the first time depends on the number of tables requested. However, it should not take more than a few hours.
Once the initial data set is loaded in your cloud storage, recurring batch jobs are run thereafter to update the data (typically taking several minutes to complete).
Since inbound data syncs are scheduled by the data provider, data availability varies based on the portal and ranges from daily, weekly, to monthly reports. Crisp ingests the data to normalize and export to users within six hours of the data's availability.
When a user is connecting to the S3 connector, there are two alerts to validate a good connection. Under Connection diagnostics, Crisp alerts you if your storage container and write access are working properly.
After login and the connection to AWS S3 is validated, any unexpected error with the connections are logged and sent to Crisp engineering for immediate resolution.
Click here to book a demo and learn more about Crisp's AWS S3 connector