Shutting down the Netty server when opening client connections

Question

I am trying to shut down a Netty server with open connections to it, and it just freezes. That's what I'm doing.

• Start the server on one computer and the client on another.
• Send a message from the client to the server to which I receive a response.
• Disconnect a server using Ctrl-C

I registered a disconnect hook on a server that closes ChannelGroup and calls releaseExternalResources on ServerBootstrap (or actually I use the protobuf-pro-duplex library DuplexTcpServerBootstrap, which does just that). In any case, when turned off, the shutdown hook is called properly, but it never returns. When I take the thread of discarding what is happening, I see two interesting stacks:

java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000006b0890950> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1433) at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:103) at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:80) at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:162) at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:319) at com.googlecode.protobuf.pro.duplex.server.DuplexTcpServerBootstrap.releaseExternalResources(DuplexTcpServerBootstrap.java:132) at com.xxx.yyy.node.NodeServer$2.run(NodeServer.java:104) at java.lang.Thread.run(Thread.java:722) 

So this is a disconnect stream that never returns. And below is another thread that seems to be waiting on the channel:

  java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.interrupt(Native Method) at sun.nio.ch.EPollArrayWrapper.interrupt(EPollArrayWrapper.java:274) at sun.nio.ch.EPollSelectorImpl.wakeup(EPollSelectorImpl.java:193) - locked <0x00000006b0896660> (a java.lang.Object) at java.nio.channels.spi.AbstractSelector$1.interrupt(AbstractSelector.java:210) at java.nio.channels.spi.AbstractSelector.begin(AbstractSelector.java:216) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:80) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked <0x00000006b08964a8> (a sun.nio.ch.Util$2) - locked <0x00000006b0896498> (a java.util.Collections$UnmodifiableSet) - locked <0x00000006b0890d20> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:52) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:208) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:102) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 

I am using Netty 3.4.6.Final with Java 7.04 for Linux. Thanks!

Br, Frank.

Answer 1

Had the same "problem" with bare Netty client / server.

Thing is closing a server channel DOES NOT close open channels created for accepted client connections. It is necessary to explicitly monitor client channels on the server. This can be done with channel groups and a handler that adds client channels to this group. When it comes to shutting down a server, all channels in a group must be closed in a batch manner, not just one server channel (which can also be placed in a channel group).

The user guide has excellent documentation (9. Shutting down your application): http://static.netty.io/3.5/guide/ and the ChannelGroup API document (Simplifying the shutdown process with ChannelGroup): http://static.netty.io /3.5/api/org/jboss/netty/channel/group/ChannelGroup.html

Answer 2

I had the same problem and it was resolved. You must disable all EventLoopGroup synchronously, and then close the port.

It may take 4-5 seconds for a complete shutdown.

Here is a sample code (I think you should make a simple graphical interface with the start and stop buttons to check it):

 public class ReusePortServer { private final int port; ChannelFuture f; EventLoopGroup group; EpollEventLoopGroup bossGroup; EpollEventLoopGroup workerGroup; public ReusePortServer(int port) { this.port = port; } public void start() throws Exception { group = new EpollEventLoopGroup(); bossGroup = new EpollEventLoopGroup(); workerGroup = new EpollEventLoopGroup(); ServerBootstrap b = new ServerBootstrap(); b.group(bossGroup, workerGroup) .channel(EpollServerSocketChannel.class) .option(EpollChannelOption.SO_REUSEPORT, true) .childHandler(new ChannelInitializer<SocketChannel>() { @Override public void initChannel(SocketChannel ch) throws Exception { ch.pipeline().addLast(new ReusePortHandler()); } }); f = b.bind(port).sync(); log(String.format("%s started and listen on %s", ReusePortServer.class.getName(), f.channel().localAddress())); } private final static SimpleDateFormat datefmt = new SimpleDateFormat("HH:mm:ss "); public static void log(final String msg) { System.out.print(datefmt.format(new Date())); System.out.println(msg); System.out.flush(); } public void stop() { System.out.println("ReusePortServer.stop"); try { // shutdown EventLoopGroup bossGroup.shutdownGracefully().sync(); workerGroup.shutdownGracefully().sync(); f.channel().closeFuture().sync(); // close port } catch (InterruptedException e) { e.printStackTrace(); } } public static void main(final String[] args) throws Exception { int port = 12345; new ReusePortServer(port).start(); } } 

Note. You can change EpollEventLoopGroup to whatever EventLoopGroup you want.

Answer 3

Disabling Netty Server

• Close server channel
• Completion of work and contractor
• Release server bootstrap resource

Code example

 ChannelFuture cf = serverChannel.close(); cf.awaitUninterruptibly(); bossExecutor.shutdown(); workerExecutor.shutdown(); thriftServer.releaseExternalResources();