What would be the best way to get Spark to work on a distributed database? (RDD or FileInputFormat)

Question

We have some kind of distributed data warehouse. We know all the internal components and can access data directly on disk.

I am exploring the possibility of deploying Apache Spark directly above it.

What would be the best / recommended way to do this?

• Custom RDD entry (output from RDD)
• Or through the FileInputFormat extension ?

(easier than another?) Better performance, etc.

thanks for the help

Answer 1

Spark 1.2 API Spark SQL. () SQL , . Spark SQL (, RDD API), , , , Spark SchemaRDD.

API:

• Spark SQL Optimizer API API Avro.
• API
• SPARK-3247 JIRA .

Answer 2

CustomRDD. datastax-cassandra, RDD

https://github.com/datastax/spark-cassandra-connector/blob/master/spark-cassandra-connector/src/main/scala/com/datastax/spark/connector/rdd/CassandraRDD.scala

, , , Cassandra RDD. , .