Is there a way to transfer files from subdirectories to Spark Streaming?

In the docs, I see that subdirectories are not supported by Spark StreamingContext.fileStream() . Has anyone found any workaround for this?

  streamingContext.fileStream[KeyClass, ValueClass, InputFormatClass](dataDirectory)

Link: Streaming-Programming-guide

Reference to SPARK-1795

Reference to SPARK-3586

+4

apache-spark spark-streaming

Rory byrne Aug 21 '15 at 16:35

No one has answered this question yet.

See similar questions:

eleven

Spark Streaming textFileStream does not support wildcards

or similar:

8

Spark streaming custom metrics

5

Using a text file as a Spark stream source for testing purposes

3

Get data stream from mqtt using python (pyspark) in spark version 2.2.0

3

Dynamic Distribution for Spark Streaming

2

Is there an equivalent for shortening ByKeyAndWindow in Spark Structured Streaming?

0

Using Spark fileStream using Avro Data Entry

0

Fixed streaming and kafka integration

0

Spark Streaming - Problem with CheckPointing

0

Spark streaming cannot read files from the Windows directory

All Articles