Geek Answers Handbook

Spark streaming - textFileStream / fileStream - Get the file name

Using a spark stream I watch files in HDFS using the textFileStream/fileStream , how do we get fileNames that are read by these methods?

I used textFileStream , which has the contents of a file in JavaDStream , and I did not succeed with fileStream as it throws me a compilation error with intrinsically safe version 1.3.1.

Can someone tell me if we have an API function or some other way to get the names of the files that these streaming methods read?

Thanks Lokesh

+1
java apache-spark
source share

No one has answered this question yet.

See similar questions:

7
Spark streaming DStream RDD to get the file name
2
SparkStreaming: error in the Stream () file

or similar:

4
Spark Streaming does not detect new HDFS files
2
SaveAsTextFile: empty file throws Spark stream exception
2
How to split Spark DStream into several substreams
2
Spark FileStreaming does not work with foreachRDD
2
Pdf stream files using spark stream stream
one
apache spark Streaming textFileStream - reading gzip files
one
Spark streaming does not read files transferred from HDFS to HDFS
0
Spark Streaming textFileStream returns RDD.saveAsTextFile output
0
How to make Spark file stream work on Windows FS (NTFS)
0
Spark Streaming with Integer TextFiles

More articles:


All Articles