Geek Answers Handbook

Spark MLlib - co-filtering training with implicit feedback - strange warnings

I'm trying to create a Collaborative filtering model for custom orders and getting useful results using ALS.train() , but I would like to try ALS.trianImplicit() , but trianImplicit() predicts only zeros in the same dataset as ASL.train() I got decent forecasts.

When using ALS.trianImplicit() to train the model, I received the following warnings:

 15/09/01 15:39:29 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS 15/09/01 15:39:29 WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS

Does this mean that it was a mistake , not a warning, and the model simply could not teach anything due to missing libraries?

+5
source share
1 answer

Since @eliasah pointed out this warning, it is not critical, but it can slow things down. Using native BLAS can lead to significant performance improvements. There are instructions on https://github.com/amplab/ml-matrix/blob/master/EC2.md on how to configure Spark + BLAS on EC2.

If Ubuntu is running on your cluster, you can install the following packages:

 libblas3gf libblas-doc libblas-dev liblapack3gf liblapack-doc liblapack-dev

However, some people have reported better performance when using http://www.openblas.net/ so you can try installing it on your workers. You also need to enable com.github.fommil.netlib with your application (Spark currently uses version 1.1.2).

+1
source

More articles:


All Articles