In sparks, that the parameter "minPartitions" works in SparkContext.textFile (path, minPartitions)?

Question

In Spark, either SparkContext or JavaSparkContext, there is one parameter, which is minPartitions when sc.textFile is called. what does this parameter mean?

Answer 1

minPartitionswill be transferred to Hadoop InputFormat.getSplits. A parameter is a hint, so you can get more or less sections, depending on the implementation of Hadoop InputFormat.