How to get the number of employees (players) in PySpark?

Question

I need to use this option because I can get the number of workers? As to Scala, I can call sc.getExecutorMemoryStatus , to get the number of workers available. But PySpark, seem to have no the API, to get this number.

Answer 1

In the scala, getExecutorStorageStatus and getExecutorMemoryStatus both return a number of artists, including the driver. Example fragment as below

 /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } active / registered executors /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } registered executors. /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } the form of host: port. /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } Seq [String] = { /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } spark.driver.host") /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } ) ( /** Method that just returns the current active/registered executors * excluding the driver. * @param sc The spark context to retrieve registered executors. * @return a list of executors each in the form of host:port. */ def currentActiveExecutors(sc: SparkContext): Seq[String] = { val allExecutors = sc.getExecutorMemoryStatus.map(_._1) val driverHost: String = sc.getConf.get("spark.driver.host") allExecutors.filter(! _.split(":")(0).equals(driverHost)).toList } 

But python api it has not been implemented

@DanielDarabos ? Answer also confirms this.

However, I am not an expert in pyspark. You can try something like this python ...

 sc.getConf.getInt("spark.executor.instances", 1)