String data from Twitter imported via Datafeed Toolbox is truncated in MATLAB: Data is incomplete

See the sample dataset attached.

Problem:

• See the variable tableOfTextByTime
• That variable is a table that contains the data imported from Twitter about a sample query
• The 2nd column in the table shows the text data of the tweets
• Notice that those tweets are truncated. The text data appears truncated everywhere: variable explorer, MATLAB Command Window, even a report created with MATLAB report generator
• The data is imported using the 'search' function in Datafeed Toolbox. See details here: https://www.mathworks.com/help/datafeed/twitter.search.html

Why it's a problem:

• Incomplete data is useless to be able to perform text analysis
• Can't derive insights from data if the text is truncated by MATLAB

Question:

• Can MATLAB / Datafeed toolbox import Twitter data (using the 'search' function) while preserving the full text for each tweet?
• How can I obtain the full text of each tweet returned by the query?
• Note that the sample dataset is small. Real datasets include 10s of thousands of tweets, so the solution needs to be scalable
• I see that each tweet returns a URL pointing to the entry on Twitter, but querying each URL individually is not scalable

Thanks!

Answer 1

[Prateek]

Hello Ed,

This issue arises due to the default 140-character limit of Twitter search API.

To retrieve the complete text, the “tweet_mode” parameter must be set to “extended”:

parameters.tweet_mode = 'extended';

Please refer to the following documentation for other details: Search for Tweets - MATLAB search - MathWorks India.

Hope this helps.

Regards,

Prateek