Convert JavaDStream <String> to JavaRDD <String>

Question

I have a JavaDStream that receives data from an external source. I am trying to integrate Spark Streaming and SparkSQL. JavaDStream is known to consist of JavaRDD. And I can apply the applySchema () function when I have JavaRDD. Please help me convert it to JavaRDD. I know that there are functions in scala and it is much simpler. But help me in Java.

Answer 1

You cannot convert DStream to RDD. As you mentioned, DStream contains RDD. A way to access RDD is to apply a function to each RDD DStream with foreachRDD. See Docs: https://spark.apache.org/docs/1.1.0/api/java/org/apache/spark/streaming/api/java/JavaDStreamLike.html#foreachRDD(org.apache.spark.api. java.function.Function2)

Answer 2

You must first access all the RDDs inside the DStream using forEachRDD as:

javaDStream.foreachRDD( rdd => {
    rdd.collect.foreach({
        ...
    })
})