Why is there always 200 tasks in groupByKey?

Question

Whenever I do groupByKeyon RDD, it splits into 200 jobs, even if the source table is quite large, for example. 2k sections and tens of millions of lines.

In addition, the operation seems to be stuck in the last two tasks, which take a very long time to calculate.

Why is it 200? How to increase it and will it help?

Answer 1

This parameter comes from spark.sql.shuffle.partitions, which represents the number of partitions used in the grouping, and has a default value of 200 , but can be increased. This may help, it will depend on the cluster and data.

, , , . reduceByKey/combineByKey, groupByKey, -?