How to get WebUI URI from SparkContext

Question

I am running pySpark on an IPython laptop. Each laptop has its own sparkContext, and each of them has an associated webUI on its own port, starting at 4040.
I would like to extract and print the port or URI so that I can get to the correct web interface. How can I get this information?

Answer 1

You can get the full URL of a SparkUI page by tunneling directly to the underlying Scala SparkContext with:

>>> sc._jsc.sc().uiWebUrl().get() u'http://192.168.0.59:6970' 

This is a bit inconvenient, so I just filed a Pull request , which adds an accessor, so you can just do:

 >>> sc.uiWebUrl u'http://192.168.0.59:6970' 

Hopefully this will be merged in the next release, but if not, you can just fix your own copy of Spark with the changes in the linked branch (or use the uglier longer form above).