Geek Answers Handbook

Why does conf.set ("spark.app.name", appName) not specify a name in the user interface?

I'm calling

val appName : String = arguments.getNameFromConfig

val conf = new SparkConf()
conf.set("spark.driver.maxResultSize", "30G")
conf.set("spark.app.name", appName)

println("Master: " + arguments.getMaster)

conf.setMaster(arguments.getMaster)    

val sc = new SparkContext(conf)

to more easily identify my tasks in the user interface. However, he does not use this name in the scheduler. Instead, it uses the path to the main class Word2VecOnCluster:

enter image description here

The name is present only in the header:

enter image description here

My colleague actually does the same, and he works there. Here you do not see that the name of my task is slightly larger:

W2V_rtype-yelp_w2vpart-1_vsize-100_lr-0.025_dskeep-5.0perc_vocabsize-100000_epochs-1_iter-1

So maybe there is a restriction on the length of the name? If so, then this can be added to the documentation - or is there any other reason why this will be done?

+4
1

, sparkConf, , . - name {appName} spark-submit, Yarn.

+7

More articles:


All Articles