Printing Spark Streaming Content on Windows

Question

I want to just print the contents of the streams to the console. I wrote the following code, but it doesn't print anything. Anyone can help me read a text file as a stream in Spark ?? Is there a problem with the Windows system?

public static void main(String[] args) throws Exception {

     SparkConf sparkConf = new SparkConf().setAppName("My app")
        .setMaster("local[2]")
        .setSparkHome("C:\\Spark\\spark-1.5.1-bin-hadoop2.6")
        .set("spark.executor.memory", "2g");

    JavaStreamingContext jssc = new JavaStreamingContext(sparkConf, Durations.seconds(2));

    JavaDStream<String> dataStream = jssc.textFileStream("C://testStream//copy.csv");
    dataStream.print();

    jssc.start();
    jssc.awaitTermination();
}

UPDATE: contents of copy.csv

0,0,12,5,0
0,0,12,5,0
0,1,2,0,42
0,0,0,0,264
0,0,12,5,0

Answer 1

textFileStreamDesigned for monitoring compatible directories with protocol support. This operation will monitor the provided directory, and as new files are added to the provided directory, it will read / transfer data from recently added files.

text/csv textFileStream , , , , .

- ( HDFS ), textFileStream.

, "C://testStream//copy.csv" C://testStream" , Spark Streaming , copy.csv C://testStream Spark Console.

, Scala/Java-, Socket ( PORT #), socketTextStream . , .

Flume

. API.

Answer 2

Windows 7 Spark 1.6.3: ( , , )

val ssc = ...
val lines = ssc.textFileStream("file:///D:/tmp/data")
...
print 

...

D:/tmp/data, ssc -

:

• , 1.txt D:/tmp/data​​li >
• spart
• data.txt( , , , )

, , , Unix ( Notepad ++), .