Data connector details
-
Category: Social Media
-
Connector type: API
Description
The Social Intelligence connector allows you to connect with Twitter, Forums, News, Blogs, and Instagram, singly, or all in one shot.
The connector searches through the data word by word, comparing posts with the query, and crawls the post when there is an exact match. It starts crawling the data of the most recent post, and continues for the number of days specified in the schedule.
API limitations
Stratifyd can adjust the frequency of data calls to accommodate your needs. Contact your Stratifyd representative to discuss how we may best help you.
If you have issues with pulling data, try narrowing your date range on the second page of the wizard or making your query more specific.
Input fields
When setting up the connector, provide a search query and select the sources from which to pull the data.
-
Search Query: Enter your search query using the Query rules below.
-
Custom Name: Enter a name to use for the data stream. This is returned in the dc_crawler_name field.
-
Content Limit per month: Set a limit for the number of records to return per month. The default value is
-
Select: Select the sources from which you want to pull data. The options are: Twitter, Instagram, Blogs, Forums, News
You can change your date range on the second page of the wizard, or optionally Schedule data crawls to update your data stream periodically.
Data dictionary
This connector returns the following fields that you can use in analysis.
author
-
Description: User name of the content author.
-
Category: user index
-
Example: Shev1996
content
-
Description: Free-form text of the tweet, Instagram post, blog post, forum post, or news article.
-
Category: text index
-
Example: Headline news on the BBC site is automatically world news. You can change this to UK or even England news at a click and this won’t then be included. But this is world news just as the Trump impeachment has been top many times
content_link
-
Description: URL of the source of the content.
-
Category: meta
-
Example: https://www.mumsnet.com/Talk/am_i_being_unreasonable/3806366-To-think-its-odd-that-BBC-news-main-headline?pg=5#93451598
crawler_time_stamp
-
Description: Date and time the data was crawled in numeric format.
-
Category: meta
-
Example: 1580151487848
data_source
-
Description: Name of the data connector used to retrieve the data.
-
Category: meta
-
Example: Social Intelligence
dc_crawler_name
-
Description: Name supplied in the Custom Name field for the data stream.
-
Category: meta
-
Example: Social Intelligence Impeachment UK Opinions
language
-
Description: Two-letter language code of the language detected for the record.
-
Category: meta
-
Example: en
location_country
-
Description: Country of the author of the content.
-
Category: geo index
-
Example: GB
location_info
-
Description: City and state of the content author.
-
Category: geo index
-
Example: , , UK
source
-
Description: Source of the content: twitter, instagram, blog, forum, or news.
-
Category: meta
-
Example: forum
time_stamp
-
Description: Date and time on which the content was created.
-
Category: date index
-
Example: 1580090880000
title
-
Description: Free-form text of any title for the content.
-
Category: text index
-
Example: To think it's odd that BBC news main headline...
Data
-
Description: Returns all of the data fields.
Query rules
For each query, provide at least one search term (keyword or phrase).
Regardless of language, surround each search term with English double quotes. (English single-word queries do not require quotation marks.)
e.g. "Jour de la Terre" OR "earth day" OR "地球日" OR "アースデー" OR EarthDay
Chinese, Japanese, and Korean languages query by character if you do not wrap search terms (except for country) with ENGLISH double quotation marks. For accurate results, always surround words or phrases with English double quotation marks.
For example:
(("古典音乐" OR "嘻哈音乐" OR "流行音乐") AND country:China) AND NOT (country:Taiwan OR city:"Hong Kong")
Boolean operators
Connect multiple search terms with Boolean operators. Search terms are not case sensitive, but capitalizing the operators is a best practice to avoid confusion.
-
AND All of the search terms linked by this operator must exist in the post.e.g. "dog" AND "cat" returns all mentions with both dog and cat.
-
OR At least one of the search terms linked by this operator must exist in the post. e.g. "dog" OR "cat" returns all mentions with either dog or cat.
-
AND NOT None of the search terms following this operator can exist in the post.e.g. "dog" AND "cat" AND NOT "horse" returns all mentions with both dog and cat that do not include horse.
Separate plurals and word stems with the OR operator.
e.g. "bulldog" OR "bulldogs" returns
"bark" OR "barking" OR "barked"
You can also return all variations of word stems using a wildcard character, but this character cannot be enclosed in parentheses.
e.g. "bark*"
Combining operators
Use parentheses to wrap logical phrases so that you can nest queries and set the order in which the operations run.
e.g. ("tabby" OR "shorthair" OR "ragdoll") AND ("cat" OR "cats") AND NOT "mix"
("Electronic Music" AND "Pop Music") AND (country:China OR country:USA)
("Electronic Music" AND "Pop Music") AND NOT (country:China)
Proximity queries
To search for mentions where the search terms appear within a certain number (n) of words from one another, create a proximity query by adding a tilde symbol and the number of words after the search term (~n).
e.g. "british shorthair"~3 AND "shorthair" AND NOT "american shorthair"~3
Advanced operators
Express advanced operators such as country or city as country:countryName without quotation marks, unless the value contains multiple words, in which case the multi-word value must be wrapped in quotation marks.
e.g. country:France OR country:China OR country:"New Zealand"
Express advanced operators that take numeric values as postcount:1000 without quotation marks. To express a range of numeric values, use brackets.
e.g. postcount:[50 TO 100]
Further questions?
We're here to help! Don't hesitate to contact us for further assistance via chat or submit a ticket!
Comments
0 comments
Please sign in to leave a comment.