This article is an overview of the various types of data connectors available in the Stratifyd platform. For specific information on how to use each connector, please visit the individual connector pages in the Data Connector Library.
What are data connectors?
Data connectors contain individual processes that allow you to extract a data stream from a raw data source. First, let's define the various data terms we will be referencing in this article:
- Data Source – the source that holds the universe of data available
- Data Connector – the method to extract a subset of the data source
- Data Stream – the Stratifyd object that contains a subset of a data source. A data stream is what is used to build workspaces, visualizations, run AI models, and more.
Stratifyd offers a variety of connectors that will use different methods to extract data from a source. The following methods are commonly used for data connectors:
Web-Crawler
Web-Crawler connectors make real-time requests to the source to capture publicly available data at the time of your request.
Examples: Amazon, Target, Walmart
Pre-Cached Crawler
Web-Crawler connector that make requests at a specific schedule. The results will be saved in a Stratifyd DB that can be pulled by request.
- Pre-Cached data can process faster than web-crawler data because a real-time request is not needed.
- Stratifyd Tip: Pre-Cached data can have a lag depending on when a request is made. For example, our Glassdoor connector re-crawls every 6 months. So if new data appears available on Glassdoor but not in Stratifyd it could be that our crawler has not re-crawled a few months.
Automated API Method
An API connector has an automated API script that connects straight to a 3rd party source. These sources will require customers to have proper API authentication to use, and typically contain private data sources. For further questions on authentication please contact your Stratifyd rep.
- Examples: Qualtrics
Local Upload
A local upload connector allows you to drag and drop a file from your local machine and upload into the platform. These connectors are used best for a spot analysis or to test out a new potential data source.
Custom Connectors
In some circumstances, custom connectors can be developed to suit your needs. A common custom connector or data integration is an SFTP integration where a client’s internal data team will send data to an FTP location setup by Stratifyd. From that point, Stratifyd will build a process to pull the data from the FTP and ingest it directly into the Stratifyd platform. This process is common for 1st party sensitive data.
Security
Stratifyd offers a Redaction Engine to assist with security concerns. For more on redaction, please read this article.
Comments
0 comments
Please sign in to leave a comment.