Cassandra-stress "Failed to connect via JMX without collecting this data"

I am trying for the first time to use the cassandra-stress tool. Although I can run this tool, the output

the messages "Failed to connect via JMX, rather than collecting this data" are displayed,

Command

cassandra-stress user \ profile=./stress_write.yaml ops\(insert=1\) \ n=1000000 \ -log file=./stress_write.log \ -node node1,node2,node3,node4,node5,node6 

Exit

 WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:25 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node5) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node1) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node2) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node4) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. WARN 19:44:26 Found host with 0.0.0.0 as rpc_address, using listen_address (/node3) to contact it instead. If this is incorrect you should avoid the use of 0.0.0.0 server side. INFO 19:44:26 Using data-center name 'DC2' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor) INFO 19:44:26 New Cassandra host /node2:9042 added INFO 19:44:26 New Cassandra host /node5:9042 added Connected to cluster: MyCluster INFO 19:44:26 New Cassandra host /node4:9042 added INFO 19:44:26 New Cassandra host /node1:9042 added INFO 19:44:26 New Cassandra host /node6:9042 added Datatacenter: DC2; Host: /node4; Rack: rack1 Datatacenter: DC2; Host: /node3; Rack: rack1 Datatacenter: DC2; Host: /node6; Rack: rack1 Datatacenter: DC2; Host: /node5; Rack: rack1 Datatacenter: DC2; Host: /node1; Rack: rack1 Datatacenter: DC2; Host: /node2; Rack: rack1 INFO 19:44:26 New Cassandra host /node3:9042 added Created schema. Sleeping 6s for propagation. Failed to connect over JMX; not collecting these stats Generating batches with [1..1] partitions and [1..1] rows (of [1..1] total rows in the partitions) Failed to connect over JMX; not collecting these stats Failed to connect over JMX; not collecting these stats Improvement over 4 threadCount: 36% Failed to connect over JMX; not collecting these stats Improvement over 8 threadCount: 138% Failed to connect over JMX; not collecting these stats Improvement over 16 threadCount: 48% Failed to connect over JMX; not collecting these stats Improvement over 24 threadCount: 33% Failed to connect over JMX; not collecting these stats Improvement over 36 threadCount: 27% Failed to connect over JMX; not collecting these stats Improvement over 54 threadCount: 39% Failed to connect over JMX; not collecting these stats Improvement over 81 threadCount: 37% Failed to connect over JMX; not collecting these stats Improvement over 121 threadCount: 16% Failed to connect over JMX; not collecting these stats Improvement over 181 threadCount: 1% Failed to connect over JMX; not collecting these stats Improvement over 271 threadCount: 15% Failed to connect over JMX; not collecting these stats Improvement over 406 threadCount: 3% Failed to connect over JMX; not collecting these stats Improvement over 609 threadCount: -3% 

Is there any command line or file configuration option that I need to specify for JMX? I tested and confirmed that the connection between the stress machine and my nodes is not a problem, because I was able to establish a connection between them through jmxsh.

Another exit problem, which may or may not be due to a JMX error, is that it lacks some key parts. I am quoting sample output on this Datastax documentation page to show parts that are missing from what I received:

 WARNING: uncertainty mode (err<) results in uneven workload between thread runs, so should be used for high level analysis only Running with 4 threadCount Running WRITE with 4 threads until stderr of mean < 0.02 total ops , adj row/s, op/s, pk/s, row/s, mean, med, .95, .99, .999, max, time, stderr, gc: #, max ms, sum ms, sdv ms, mb 2552 , 2553, 2553, 2553, 2553, 1.5, 1.4, 2.5, 6.0, 12.6, 18.0, 1.0, 0.00000, 0, 0, 0, 0, 0 5173 , 2634, 2613, 2613, 2613, 1.5, 1.5, 1.8, 2.6, 8.6, 9.2, 2.0, 0.00000, 0, 0, 0, 0, 0 ... Results: op rate : 3954 partition rate : 3954 row rate : 3954 latency mean : 1.0 latency median : 0.8 latency 95th percentile : 1.5 latency 99th percentile : 1.8 latency 99.9th percentile : 2.2 latency max : 73.6 total gc count : 25 total gc mb : 1826 total gc time (s) : 1 avg gc time(ms) : 37 stdev gc time(ms) : 10 Total operation time : 00:00:59 Sleeping for 15s Running with 4 threadCount 

Notes

  • My cluster is running DSE 4.6.1 (Cassandra 2.0.12)
  • I am running a stress tool from another machine.
  • Stress Tool Version - From DS C 2.1 (Cassandra 2.1)
+5
source share
3 answers

I have the same setup (Cassandra version 2.0.12), and the stress tool is from 2.1 and saw similar problems. Finally, I had time to investigate.

I downloaded the source code and ran it in the debugger. I saw this error message being misleading. The tool connects to JMX, but has a problem with one of mBeans ( org.apache.cassandra.service:type=GCInspector ).

I saw the same thing when I conducted a stress test using the option: -log level=verbose and saw the following exception:

 java.lang.reflect.UndeclaredThrowableException at com.sun.proxy.$Proxy11.getAndResetStats(Unknown Source) at org.apache.cassandra.tools.NodeProbe.getAndResetGCStats(NodeProbe.java:385) at org.apache.cassandra.stress.util.JmxCollector.<init>(JmxCollector.java:86) at org.apache.cassandra.stress.StressMetrics.<init>(StressMetrics.java:64) at org.apache.cassandra.stress.StressAction.run(StressAction.java:187) at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:97) at org.apache.cassandra.stress.StressAction.run(StressAction.java:61) at org.apache.cassandra.stress.Stress.main(Stress.java:109) Caused by: javax.management.InstanceNotFoundException: org.apache.cassandra.service:type=GCInspector at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(Unknown Source) at .... 

I connected to Cassandra using jConsole, and version 2.0.12 does not have this mBean.

But my conclusion contains most of the data shown in the sample (excluding garbage collection statistics).

Have you tried running cassandra-stress with the default setting? Also try setting detailed logging information, perhaps this will give you some ideas.

+2
source

I also ran into the same problem (Cassandra 3.7), I ran the Cassandra-stress client with -log level = verbose and saw an exception below: java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exce4; nested exception is: java.net.ConnectException: Connection timed out] at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99) at org.apache.cassandra.stress.util.JmxCollector.(JmxCollector.java:85) at org.apache.cassandra.stress.StressMetrics.(StressMetrics.java:62) at org.apache.cassandra.stress.StressAction.run(StressAction.java:211) at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:107) at org.apache.cassandra.stress.StressAction.run(StressAction.java:60) at org.apache.cassandra.stress.Stress.run(Stress.java:133) at org.apache.cassandra.stress.Stress.main(Stress.java:61) Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmion is: java.net.ConnectException: Connection timed out] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:188) at org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:155) at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:95) ... 7 more Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 1.2.3.4; java.net.ConnectException: Connection timed out] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) at javax.naming.InitialContext.lookup(InitialContext.java:417) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ... 11 more Caused by: java.rmi.ConnectException: Connection refused to host: 1.2.3.4; nested exception is: java.net.ConnectException: Connection timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ... 16 more java.lang.RuntimeException: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exce4; nested exception is: java.net.ConnectException: Connection timed out] at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:99) at org.apache.cassandra.stress.util.JmxCollector.(JmxCollector.java:85) at org.apache.cassandra.stress.StressMetrics.(StressMetrics.java:62) at org.apache.cassandra.stress.StressAction.run(StressAction.java:211) at org.apache.cassandra.stress.StressAction.warmup(StressAction.java:107) at org.apache.cassandra.stress.StressAction.run(StressAction.java:60) at org.apache.cassandra.stress.Stress.run(Stress.java:133) at org.apache.cassandra.stress.Stress.main(Stress.java:61) Caused by: java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmion is: java.net.ConnectException: Connection timed out] at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369) at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270) at org.apache.cassandra.tools.NodeProbe.connect(NodeProbe.java:188) at org.apache.cassandra.tools.NodeProbe.(NodeProbe.java:155) at org.apache.cassandra.stress.util.JmxCollector.connect(JmxCollector.java:95) ... 7 more Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 1.2.3.4; java.net.ConnectException: Connection timed out] at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:122) at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205) at javax.naming.InitialContext.lookup(InitialContext.java:417) at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1957) at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1924) at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287) ... 11 more Caused by: java.rmi.ConnectException: Connection refused to host: 1.2.3.4; nested exception is: java.net.ConnectException: Connection timed out at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619) at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216) at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202) at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:342) at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source) at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118) ... 16 more

So, to solve this problem, I set the rpc_address property from the Cassandra.yaml file to <host_ip> and commented on broadcast_rpc_address .

This works for me and I no longer get this error.

0
source

edit conf / cassandra.yaml

change rpc_address: localhost to

rpc_address: 0.0.0.0

restart the database

0
source

Source: https://habr.com/ru/post/1216396/


All Articles