LDA Model Input

Question

I am new to python. I just started working on a project to use LDA modeling on tweets. I am trying the following code:

This example uses an online dataset. I have a csv file that includes tweets that I need to use. Can someone tell me how can I use my local file? How can I make my own vocals and titles?

I could not find a tutorial that explains how to prepare materials for the LDA. All of them assume that you already know how to do this.

from __future__ import division, print_function import numpy as np import lda import lda.datasets # document-term matrix X = lda.datasets.load_reuters() print("type(X): {}".format(type(X))) print("shape: {}\n".format(X.shape)) # the vocab vocab = lda.datasets.load_reuters_vocab() print("type(vocab): {}".format(type(vocab))) print("len(vocab): {}\n".format(len(vocab))) # titles for each story titles = lda.datasets.load_reuters_titles() print("type(titles): {}".format(type(titles))) print("len(titles): {}\n".format(len(titles))) doc_id = 0 word_id = 3117 print("doc id: {} word id: {}".format(doc_id, word_id)) print("-- count: {}".format(X[doc_id, word_id])) print("-- word : {}".format(vocab[word_id])) print("-- doc : {}".format(titles[doc_id])) model = lda.LDA(n_topics=20, n_iter=500, random_state=1) model.fit(X) topic_word = model.topic_word_ print("type(topic_word): {}".format(type(topic_word))) print("shape: {}".format(topic_word.shape)) for n in range(5): sum_pr = sum(topic_word[n,:]) print("topic: {} sum: {}".format(n, sum_pr)) n = 5 for i, topic_dist in enumerate(topic_word): topic_words = np.array(vocab)[np.argsort(topic_dist)][:-(n+1):-1] print('*Topic {}\n- {}'.format(i, ' '.join(topic_words))) doc_topic = model.doc_topic_ print("type(doc_topic): {}".format(type(doc_topic))) print("shape: {}".format(doc_topic.shape)) 

Answer 1

I know this is a little late, but hope this helps. First you need to understand that LDA applies only to DTM (Document Term Matrix). Therefore, I suggest you follow these steps:

• Download csv file
• Extract required tweets from file
• Data cleansing
• Create a dictionary containing each word of the created body
• Creating a TDM Structure
• Set the structure to the data file
• Get Dictionary - TDM Functions (Words)
• Keep using the code above

You can provide this code here to help you get started -

 token_dict = {} for i in range(len(txt1)): token_dict[i] = txt1[i] len(token_dict) print("\n Build DTM") %time tf = CountVectorizer(stop_words='english') print("\n Fit DTM") %time tfs1 = tf.fit_transform(token_dict.values()) # set the number of topics to look for num = 8 model = lda.LDA(n_topics=num, n_iter=500, random_state=1) # we fit the DTM not the TFIDF to LDA print("\n Fit LDA to data set") %time model.fit_transform(tfs1) print("\n Obtain the words with high probabilities") %time topic_word = model.topic_word_ # model.components_ also works print("\n Obtain the feature names") %time vocab = tf.get_feature_names()