Why does a standalone application use socket connection in pyspark?

If I use spark in standalone applications, I donโ€™t think I need a connection to the server (of course). But why am I getting this network error message?

[ERROR] Error while sending or receiving. 
Traceback (most recent call last): 
  File "/Users/chlee021690/anaconda/lib/python2.7/site-packages/py4j/java_gateway.py", 
   line 473, in send_command 
    answer = smart_decode(self.stream.readline()[:-1]) 
  File "/Users/chlee021690/anaconda/python.app/Contents/lib/python2.7/socket.py", line 430, in readline 
    data = recv(1) 
timeout: timed out 

....

Py4JNetworkError: An error occurred while trying to connect to the Java server.

My code is as follows:

from pyspark import * 
import pyspark.mllib.recommendation as spark_rec 
filename = "./yahoo music/train_0.txt" 
sc = SparkContext('local', 'spark_rec') 
aData = sc.textFile(filename).cache() (this part was successful, but the next following lines were failtures) 
ratings = aData.map(lambda line: np.array([float(x) for x in line.split('\t')])) 
rank = 10 
numIterations = 20 
aModel = spark_rec.ALS.train(ratings, rank, numIterations) 
+4
source share

All Articles