Plotting from streaming data using a spark stream

Question

I'm new to the spark. I need to build a coincidence graph (in tweet words there will be nodes, and if words from the same tweet are added between them) from streaming data, for example twitter tweets. Can we use spark streams to plot a tweeter live match. Is sparking implied for this use case ?. I'm not sure if this can be done using sparking. If not, what are the alternatives?

Answer 1

coincidence frequency can be considered as a graph or adjacency matrix, but this is a large sparse histogram (number of frequencies) in the product space of your word list. most likely you want to detect a correlation of a moving window, so you must develop a sketch data structure to track an unusual increase or decrease in the speed of occurrence in the stream. for example, counting a flowering filter or counting a minimum sketch for each word pair - see http://twitter.imtqy.com/algebird/#com.twitter.algebird.CMSCounting