Use scikit learn tfidf vectorizer starting with counts data frame

Question

I have a pandas data frame with the number of words for a number of documents. Can I apply sklearn.feature_extraction.text.TfidfVectorizerto it to return the term-document matrix?

import pandas as pd

a = [1,2,3,4]
b = [1,3,4,6]
c = [3,4,6,1]

df = pd.DataFrame([a,b,c])

How to get tfidf version of counts in df?

Answer 1

like this:

from sklearn.feature_extraction.text import TfidfTransformer
tfidf =TfidfTransformer(norm=u'l2', use_idf=True, smooth_idf=True, sublinear_tf=False)
data =tfidf.fit_transform(df.values)

This returns a sparse matrix of tfidf values. You can turn them into dense ones and return them to a data frame as follows:

pd.DataFrame(data.todense())