Spark Master listens on several ports. Unfortunately, the used IP address / hostname scheme is different from them - and connection failure often occurs.
Then it remains for us to ask a question: how to fix connection problems: The spark itself decides to translate:
- host name
- hostname.local (on mac os / x)
- hostname.domain
- local
- localhost.localdomain
- 127.0.0.1
- external IP
- Internal IP Address (on AWS)
An important consideration: some network clients / connections require exact string matching in order to successfully contact the wizard. So in this case 127.0.0.1 does not match hostname . I saw in cases where hostname working, but hostname.local is not: this problem is Mac-oriented. But ... then the first one stops working - and I do not have enough tools for troubleshooting.
--master provides confusion on Linux when you have an internal and external IP address.
Below is an example of my Mac. I see other patterns on AWS, and others on separate clusters. This is all confusing and time consuming, as it is also not documented:
- where are the mappings
- How to achieve a consistent main address bar:
- master
- master web ui
- akka address for the master
Below we see the output when the --master option was provided for sparking.
--master spark:
Pay attention to the variety of IP addresses
http://25.xxx:4040 akka.tcp:// sparkMaster@mellyrn :7077 mellyrn/127.0.0.1:707
Here is the output on the MAC:
15/07/31 12:21:34 INFO SparkEnv: Registering OutputCommitCoordinator 15/07/31 12:21:34 INFO Utils: Successfully started service 'SparkUI' on port 4040. 15/07/31 12:21:34 INFO SparkUI: Started SparkUI at http://25.101.19.24:4040 15/07/31 12:21:34 INFO AppClient$ClientActor: Connecting to master akka.tcp:// sparkMaster@mellyrn :7077/user/Master... 15/07/31 12:21:35 WARN AppClient$ClientActor: Could not connect to akka.tcp:// sparkMaster@mellyrn :7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp:// sparkMaster@mellyrn :7077 15/07/31 12:21:35 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp:// sparkMaster@mellyrn :7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: mellyrn/127.0.0.1:7077 15/07/31 12:21:54 INFO AppClient$ClientActor: Connecting to master akka.tcp:// sparkMaster@mellyrn :7077/user/Master... 15/07/31 12:21:54 WARN AppClient$ClientActor: Could not connect to akka.tcp:// sparkMaster@mellyrn :7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp:// sparkMaster@mellyrn :7077 15/07/31 12:21:54 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp:// sparkMaster@mellyrn :7077]. Address is now gated for 5000
On Linux, a spark connection with the --master option works (although .setMaster () is not reliable). However, even on Linux, there are many wizard / driver lines created:
source share