Problem Configuring Akka Remote Route Host Name

Question

I am testing the akka remote function for the tool that I am doing. In fact, I was able to make the main and remote systems work on the same host with different ports. Please note that my remote servers work through the router as described in akka docs.

Now I'm trying to use a few azure virtual machines to do a better experiment, but I am having some problems.

The main application has the following configuration (I changed some names for security reasons):

akka.actor.deployment { /querierActor/querierPool { router = round-robin-pool nr-of-instances = 12 target.nodes = [ "akka.tcp:// [e-mail] :2560" ,"akka.tcp:// [e-mail] :2560" ,"akka.tcp:// [e-mail] :2560" ] } } // remote configuration. Use it for multiple machines calculation akka { actor { provider = "akka.remote.RemoteActorRefProvider" } remote { enabled-transports = ["akka.remote.netty.tcp"] netty.tcp { maximum-frame-size = 100MiB port = 2552 hostname = "0.0.0.0" } } } 

So far, remote hosts have the following configuration:

 akka.actor.deployment { /querierActor/querierPool { router = balancing-pool nr-of-instances = 15 } } akka { actor { provider = "akka.remote.RemoteActorRefProvider" } remote { enabled-transports = ["akka.remote.netty.tcp"] netty.tcp { maximum-frame-size = 100MiB hostname = "0.0.0.0" port = 2560 } } } 

Using this configuration, server and remote hosts can apparently communicate, but the remote host starts to log some errors:

  [ERROR] [01/17/2015 12:55:05.734] [SYSTEM-akka.remote.default-remote-dispatcher-16] [akka.tcp:// SYSTEM@0.0.0.0 :2560/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSYSTEM%400.0.0.0%3A2552-0/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp:// [e-mail] :2560/]] arriving at [akka.tcp:// [e-mail] :2560] inbound addresses are [akka.tcp:// SYSTEM@0.0.0.0 :2560] 

And after that, the server and the remote host will start to register an error and freeze.

Server error:

 [WARN] [01/17/2015 12:21:05.658] [CRAWLER-LD-akka.remote.default-remote-dispatcher-7] [akka.tcp:// SYSTEM@0.0.0.0 :2552/system/remote-watcher] Detected unreachable: [akka.tcp:// [e-mail] :2560] [WARN] [01/17/2015 12:21:05.664] [SYSTEM-akka.remote.default-remote-dispatcher-17] [Remoting] Association to [akka.tcp:// [e-mail] :2560] with unknown UID is reported as quarantined, but address cannot be quarantined without knowing the UID, gating instead for 5000 ms. (...) [INFO] [01/17/2015 12:21:05.712] [SYSTEM-akka.actor.default-dispatcher-6] [akka://SYSTEM/user/querierActor/querierPool] Message [akka.dispatch.sysmsg.DeathWatchNotification] from Actor[akka://SYSTEM/user/querierActor/querierPool#-1217916605] to Actor[akka://SYSTEM/user/querierActor/querierPool#-1217916605] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (...) 

Remote error (similar lines several times):

 (...) [ERROR] [01/17/2015 14:21:16.371] [SYSTEM-akka.remote.default-remote-dispatcher-16] [akka.tcp:// SYSTEM@0.0.0.0 :2560/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSYSTEM%400.0.0.0%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp:// [e-mail] :2560/]] arriving at [akka.tcp:// [e-mail] :2560] inbound addresses are [akka.tcp:// SYSTEM@0.0.0.0 :2560] [ERROR] [01/17/2015 14:21:17.388] [SYSTEM-akka.remote.default-remote-dispatcher-16] [akka.tcp:// SYSTEM@0.0.0.0 :2560/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSYSTEM%400.0.0.0%3A2552-2/endpointWriter] dropping message [class akka.actor.ActorSelectionMessage] for non-local recipient [Actor[akka.tcp:// [e-mail] :2560/]] arriving at [akka.tcp:// [e-mail] :2560] inbound addresses are [akka.tcp:// SYSTEM@0.0.0.0 :2560] [WARN] [01/17/2015 14:21:17.465] [SYSTEM-akka.remote.default-remote-dispatcher-16] [akka.tcp:// SYSTEM@0.0.0.0 :2560/system/endpointManager/reliableEndpointWriter-akka.tcp%3A%2F%2FSYSTEM%400.0.0.0%3A2552-2] Association with remote system [akka.tcp:// SYSTEM@0.0.0.0 :2552] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. [INFO] [01/17/2015 14:21:17.467] [SYSTEM-akka.actor.default-dispatcher-21] [akka://SYSTEM/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FSYSTEM%40186.228.120.115%3A56044-3] Message [akka.remote.transport.AssociationHandle$Disassociated] from Actor[akka://SYSTEM/deadLetters] to Actor[akka://SYSTEM/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FSYSTEM%40186.228.120.115%3A56044-3#-2070785548] was not delivered. [6] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. [INFO] [01/17/2015 14:21:17.468] [SYSTEM-akka.actor.default-dispatcher-21] [akka://SYSTEM/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FSYSTEM%40186.228.120.115%3A56044-3] Message [akka.remote.transport.ActorTransportAdapter$DisassociateUnderlying] from Actor[akka://SYSTEM/deadLetters] to Actor[akka://SYSTEM/system/transports/akkaprotocolmanager.tcp0/akkaProtocol-tcp%3A%2F%2FSYSTEM%40186.228.120.115%3A56044-3#-2070785548] was not delivered. [7] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'. (...) 

I realized that the problem could be in the host name configuration and was trying to put the host name on the server and the remote host. But in this case, the system does not even boot:

 Exception in thread "main" org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'FacadeMemory' defined in file [D:\data\development\git\semantic-web-crawler\crawlerld.core\target\classes\net\dovale\websemantics\linkedDataRecommender\facade\memory\FacadeMemory.class]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [facade.memory.FacadeMemory]: Constructor threw exception; nested exception is org.jboss.netty.channel.ChannelException: Failed to bind to: /remote-srv01.cloudapp.net:2560 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1077) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1022) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:475) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:302) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:228) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:298) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:193) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:706) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:762) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:482) at org.springframework.boot.context.embedded.EmbeddedWebApplicationContext.refresh(EmbeddedWebApplicationContext.java:109) at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:691) at org.springframework.boot.SpringApplication.run(SpringApplication.java:320) at org.springframework.boot.SpringApplication.run(SpringApplication.java:952) at org.springframework.boot.SpringApplication.run(SpringApplication.java:941) at facade.memory.GUIMain.main(GUIMain.java:23) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:483) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134) Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [facade.memory.FacadeMemory]: Constructor threw exception; nested exception is org.jboss.netty.channel.ChannelException: Failed to bind to: /remote-srv01.cloudapp.net:2560 at org.springframework.beans.BeanUtils.instantiateClass(BeanUtils.java:164) at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:89) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateBean(AbstractAutowireCapableBeanFactory.java:1070) ... 21 more Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /remote-srv01.cloudapp.net:2560 at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:393) at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:389) at scala.util.Success$$anonfun$map$1.apply(Try.scala:236) at scala.util.Try$.apply(Try.scala:191) at scala.util.Success.map(Try.scala:236) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235) at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59) at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72) at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58) at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:41) at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:401) at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) Caused by: java.net.BindException: Cannot assign requested address: bind at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:436) at sun.nio.ch.Net.bind(Net.java:428) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:372) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:296) at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 

I do not know what I am doing wrong. I tried to find information about the problem, but everything I found is related to my problem. I also opened the ports in azure configuration.

How can I make my server node communicate correctly with my remote hosts?

Answer 1

I was able to solve this problem.

After some fruitless research, I had to try different things. I make some assumptions, which may be incorrect, since I did not find any other information. If you read this answer and find any error, let me know.

The problem was that the framework (sun.nio.ch.Net.bind0, apparently, but I did not find many documents about it) allows the following ips range: 0.0.0.0 (in case you accept connections from any network interface in the machine), 127.0.0.0 (in case you work only with a local request - I am a guest) and the IP address of any of the computer network interfaces. In this latter case, requests will be allowed only for this particular interface.

The problem is that the "hostname" property is also used to address the remote Akka hosts. I mean, when the host node calls the remote node, it uses this information for identification, after which the result should be sent. In addition, if you put the hostname property with the value 0.0.0.0 and try to reach this node with your dns name (which cannot be connected to any network interface), it will fail. You must identify the machine with the same IP address as one of the network interfaces.

So my setup has changed a bit:

For host node, I made this change:

 (...) akka.actor.deployment { /sparqlQuerierMasterActor/sparqlQuerierPool { router = round-robin-pool nr-of-instances = 12 target.nodes = [ "akka.tcp:// [e-mail] :2560" ,"akka.tcp:// [e-mail] :2560" ,"akka.tcp:// [e-mail] :2560" ] } } (...) 

XXX, YYY and ZZZ are the available IP addresses of the remote nodes, which are also registered in the network interface.

Remote node device configuration changed to:

 (...) remote { enabled-transports = ["akka.remote.netty.tcp"] netty.tcp { maximum-frame-size = 100MiB hostname = "YYY.YYY.YYY.YYY" port = 2560 } } (...) 

I have not tested it, I can save the previous configuration 0.0.0.0. Perhaps it is possible.

This solution allowed me to make the host and remote sites crazy for communication =)