56 - Trend Detection
Quantifying the deviation of the occurrence of words beyond the expected variability, and defining above what threshold you call this a trend.
Trending topics on Twitters streaming data is one of the best examples for the Trend Detection task. Capturing topics, thoughts and emotions over time produces a very insightful starting point for analysis.
You can quantify the deviation of a particular word count beyond the expected variability, and you can define a threshold above which you call the count a trend. If there is historical data, you can take seasonality-patterns into account in your time-series analysis.
The difficulty is that you often don’t know the scale, size or time interval of the change in advance. Depending on your use-case there are different setups possible. However, all algorithms present trade-offs, including simplicity vs. robustness, precision, recall and time-to-detection. An older python package from Twitterdev might give you a quick start.
An interesting variant is de Cold Trend Detection. This shows which topics have the highest negative change in scores and are cooling down at a certain time.
This article is part of the project Periodic Table of NLP Tasks. Click to read more about the making of the Periodic Table and the project to systemize NLP tasks.