The spark is read only in hdfs

Question

I set up the Spark cluster configuration with HDFS, and I know that the default file path will be read by Spark in the HDFS example:

/ad-cpc/2014-11-28/ Spark will read in : hdfs://hmaster155:9000/ad-cpc/2014-11-28/

Sometimes I wonder how I can get Spark to read a file locally without reConfig my cluster (so as not to use hdfs).

Please help me!!!

Answer 1

The local file system can be transferred from Spark with the prefix file: ///

Eg: sparkContext.textFile("file:///<<absolute path to file>>")

This command reads a file from the local file system.

Note. If running in a multi node cluster, then this file should be available on all nodes.