EDIT: I tried several things to get this to work until completion, and nothing works:
Updated neo4j-community.vmoptions file:
-server
-Xms4096m
-Xmx4096m
-XX:NewSize=1024m
Also updated my mac to handle more threads.
C02RH2U9G8WM:~ meuser$ ulimit -u
709
C02RH2U9G8WM:~ meuser$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 1
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 709
virtual memory (kbytes, -v) unlimited
Finally, a test passed to ensure that it was not limited to the hard drive causing the problems. Everything seemed to be checked, and at runtime I would check the use of the thread in another window ....
jstack -l 'pid' | grep tid | wc -l
and
ps -elfT | wc -l
There was nothing similar, so I'm very confused about why I get errors below when running spark code in scala, connecting to neo4j, which works fine with a small set of answers, but explodes when I let it copy everything.
note: sc.stop() - , , - /
, , ?
2017-10-13 01:17:45.073+0000 ERROR [o.n.b.t.SocketTransportHandler] Fatal error occurred when handling a client connection: unable to create new native thread unable to create new native thread
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at org.neo4j.kernel.impl.util.Neo4jJobScheduler.schedule(Neo4jJobScheduler.java:94)
at org.neo4j.bolt.v1.runtime.concurrent.ThreadedWorkerFactory.newWorker(ThreadedWorkerFactory.java:68)
at org.neo4j.bolt.v1.runtime.MonitoredWorkerFactory.newWorker(MonitoredWorkerFactory.java:54)
at org.neo4j.bolt.BoltKernelExtension.lambda$newVersions$1(BoltKernelExtension.java:234)
at org.neo4j.bolt.transport.ProtocolChooser.handleVersionHandshakeChunk(ProtocolChooser.java:95)
at org.neo4j.bolt.transport.SocketTransportHandler.chooseProtocolVersion(SocketTransportHandler.java:109)
at org.neo4j.bolt.transport.SocketTransportHandler.channelRead(SocketTransportHandler.java:58)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:219)
at io.netty.channel.DefaultChannelPipeline.callHandlerRemoved0(DefaultChannelPipeline.java:631)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:468)
at io.netty.channel.DefaultChannelPipeline.remove(DefaultChannelPipeline.java:428)
at org.neo4j.bolt.transport.TransportSelectionHandler.switchToSocket(TransportSelectionHandler.java:126)
at org.neo4j.bolt.transport.TransportSelectionHandler.decode(TransportSelectionHandler.java:81)
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:411)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:248)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:129)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:642)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:565)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:479)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:441)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)
at java.lang.Thread.run(Thread.java:745)
Mac ( - ). I7 16 .
scala, neo4j. , , neo4j, ( ). reset , neo4j. - -, -? Spark/Scala, neo4j configs?
org.neo4j.driver.v1.exceptions.ClientException: Failed to establish connection with server. Make sure that you have a server with bolt enabled on localhost:7687
at org.neo4j.driver.internal.connector.socket.SocketClient.negotiateProtocol(SocketClient.java:197)
at org.neo4j.driver.internal.connector.socket.SocketClient.start(SocketClient.java:76)
at org.neo4j.driver.internal.connector.socket.SocketConnection.<init>(SocketConnection.java:63)
at org.neo4j.driver.internal.connector.socket.SocketConnector.connect(SocketConnector.java:52)
at org.neo4j.driver.internal.pool.InternalConnectionPool.acquire(InternalConnectionPool.java:113)
at org.neo4j.driver.internal.InternalDriver.session(InternalDriver.java:53)
at org.neo4j.spark.Executor$.execute(Neo4j.scala:360)
at org.neo4j.spark.Neo4jRDD.compute(Neo4j.scala:408)
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:108)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)