I use the costume tokenizer to go to TfidfVectorizer. This tokenizer depends on the external TermExtractor class, which is in another file.
Basically, I want to build TfidVectorizer based on certain terms, and not all individual words / tokens.
Here is the code for it:
from sklearn.feature_extraction.text import TfidfVectorizer from TermExtractor import TermExtractor extractor = TermExtractor() def tokenize_terms(text): terms = extractor.extract(text) tokens = [] for t in terms: tokens.append('_'.join(t)) return tokens def main(): vectorizer = TfidfVectorizer(lowercase=True, min_df=2, norm='l2', smooth_idf=True, stop_words=stop_words, tokenizer=tokenize_terms) vectorizer.fit(corpus) pickle.dump(vectorizer, open("models/terms_vectorizer", "wb"))
This works fine, but whenever I want to reuse this TfidfVectorizer and load it with pickle, I get an error:
vectorizer = pickle.load(open("models/terms_vectorizer", "rb")) Traceback (most recent call last): File "./train-nps-comments-classifier.py", line 427, in <module> main() File "./train-nps-comments-classifier.py", line 325, in main vectorizer = pickle.load(open("models/terms_vectorizer", "rb")) File "/usr/lib/python2.7/pickle.py", line 1378, in load return Unpickler(file).load() File "/usr/lib/python2.7/pickle.py", line 858, in load dispatch[key](self) File "/usr/lib/python2.7/pickle.py", line 1090, in load_global klass = self.find_class(module, name) File "/usr/lib/python2.7/pickle.py", line 1126, in find_class klass = getattr(mod, name) AttributeError: 'module' object has no attribute 'tokenize_terms'
How does Python pickle work when there are dependent classes?