Geek Answers Handbook

Cassandra nodes cannot see each other when firewall is enabled

I created a 6-node Cassandra cluster covering two AWS regions / data centers (3 in each) and everything works fine. After I got a lot of work, I tried to enable gateway encryption, which I can’t work normally, despite the fact that I read countless documents on this subject and led endlessly.

I do not see any errors or anything unusual in the logs. I see the following line in the logs that indicates that she started the encrypted messaging service as expected:

MessagingService.java:482 - Starting Encrypted Messaging Service on SSL port 7001

I have enabled verbose logging for SSL in cassandra-env.sh , however this does not lead to errors or additional information about SSL interconnections that I can see ( update below ):

 JVM_OPTS="$JVM_OPTS -Djavax.net.debug=ssl"

I can connect to one of the node with everyone else in the encrypted messaging port 7001 using nc , so there is no problem with a firewall.

 ubuntu@ip-5-6-7-8 :~$ nc -v 1.2.3.4 7001 Connection to 1.2.3.4 7001 port [tcp/afs3-callback] succeeded!

I can connect to each node locally using cqlsh (I did not cqlsh client-server encryption) and I can request the system key space, etc.

However, if I run nodetool status , I see that the nodes cannot see each other. Only the node that I request for the cluster is present in the list. This was not the case before the firewall was turned on, they all could see each other just fine.

 ubuntu@ip-5-6-7-8 :~$ nodetool status Datacenter: us-east_A ===================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns Host ID Rack UN 1.2.3.4 144.75 KB 256 ? 992ae1bc-77e4-4ab1-a18f-4db62bb0ce6f 1b

My process was as follows:

  • Created a certificate authority for my cluster
  • Created a keystore and trust store for each node and added my CA certificate chain to
  • Generated a key pair and CSR for each node, signed it with my CA and added the received certificate to each node storage
  • Updated each node configuration as shown below.
  • Reboot all nodes

The server encryption configuration I'm using is this, with the corresponding values ​​in $variables .

 server_encryption_options: internode_encryption: all keystore: $keystore_path keystore_password: $keystore_passwd truststore: $truststore_path truststore_password: $truststore_passwd require_client_auth: true protocol: TLS algorithm: SunX509 store_type: JKS cipher_suites: [TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA]

If anyone can offer some understanding or direction to look into it, we will be very grateful.

Update: Encryption Agreement

Apparently, the logging of SSL debugging is output to standard output, which is not written to the Cassandra log files, so I have not seen this before. Running Cassandra in the foreground, I see a ton of SSL errors tracing, all of which complain about refusing a handshake, because:

 javax.net.ssl.SSLHandshakeException: no cipher suites in common

In an attempt to solve this problem, I switched to Oracle JRE (I was lazy and used OpenJDK before) and installed cryptography policy files of unlimited strength JCE to provide support for all possible ciphers.

Nothing to fix.

This is especially confusing given that all of these nodes are completely identical: hardware, OS version and version, Java provider and version, Cassandra version, and configuration file. I cannot imagine why they cannot agree to a cipher suite in these circumstances.

The following is a complete error:

 *** ClientHello, TLSv1.2 RandomCookie: GMT: 1449074039 bytes = { 205, 93, 27, 38, 184, 219, 250, 8, 232, 46, 117, 84, 69, 53, 225, 16, 27, 31, 3, 7, 203, 16, 133, 156, 137, 231, 238, 39 } Session ID: {} Cipher Suites: [TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } *** %% Initialized: [Session-3, SSL_NULL_WITH_NULL_NULL] %% Invalidated: [Session-3, SSL_NULL_WITH_NULL_NULL] ACCEPT-/1.2.3.4, SEND TLSv1.2 ALERT: fatal, description = handshake_failure ACCEPT-/1.2.3.4, WRITE: TLSv1.2 Alert, length = 2 ACCEPT-/1.2.3.4, called closeSocket() ACCEPT-/1.2.3.4, handling exception: javax.net.ssl.SSLHandshakeException: no cipher suites in common ACCEPT-/1.2.3.4, called close() ACCEPT-/1.2.3.4, called closeInternal(true) INFO 16:33:59 Waiting for gossip to settle before accepting client requests... Allow unsafe renegotiation: false Allow legacy hello messages: true Is initial handshake: true Is secure renegotiation: false ACCEPT-/1.2.3.4, setSoTimeout(10000) called ACCEPT-/1.2.3.4, READ: SSL v2, contentType = Handshake, translated length = 57 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] *** ClientHello, TLSv1.2 RandomCookie: GMT: 1449074039 bytes = { 205, 93, 27, 38, 184, 219, 250, 8, 232, 46, 117, 84, 69, 53, 225, 16, 27, 31, 3, 7, 203, 16, 133, 156, 137, 231, 238, 39 } Session ID: {} Cipher Suites: [TLS_RSA_WITH_AES_256_CBC_SHA, TLS_DHE_RSA_WITH_AES_128_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA, TLS_RSA_WITH_AES_128_CBC_SHA, TLS_DHE_RSA_WITH_AES_256_CBC_SHA, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV] Compression Methods: { 0 } *** %% Initialized: [Session-3, SSL_NULL_WITH_NULL_NULL] %% Invalidated: [Session-3, SSL_NULL_WITH_NULL_NULL] ACCEPT-/1.2.3.4, SEND TLSv1.2 ALERT: fatal, description = handshake_failure ACCEPT-/1.2.3.4, WRITE: TLSv1.2 Alert, length = 2 ACCEPT-/1.2.3.4, called closeSocket() ACCEPT-/1.2.3.4, handling exception: javax.net.ssl.SSLHandshakeException: no cipher suites in common ACCEPT-/1.2.3.4, called close() ACCEPT-/1.2.3.4, called closeInternal(true) INFO 16:33:59 Waiting for gossip to settle before accepting client requests... Allow unsafe renegotiation: false Allow legacy hello messages: true Is initial handshake: true Is secure renegotiation: false ACCEPT-/1.2.3.4, setSoTimeout(10000) called ACCEPT-/1.2.3.4, READ: SSL v2, contentType = Handshake, translated length = 57
+6
source share
1 answer

After I pushed and pushed a lot, I finally managed to get it to work. The problem is with certificates and keystore.

As a result of these problems, SSL handshaking will fail due to problems with the certificate chain or problems with the encryption agreement. Cassandra pretty uselessly throws SSL errors and writes nothing.

In any case, I managed to get everything to work by doing the following:

  • Make sure that the CA generates node certificates with attributes for using the client and server keys. Non-compliance with one or the other prevents proper authentication of the nodes to each other. This is an encryption agreement error. If you use OpenSSL to manage your CA, I have included the -extensions configuration that I used below.
  • Make sure that both the root and any intermediate CA certificates that you use (if you use an intermediate CA) are imported into both the keystore and truststore.
  • Make sure that the node certificate imported to the keystore includes a complete trust chain from the primary certificate to the root of the CA, including any intermediaries, even if you already imported these CA certificates separately into the keystore. Failure to do so is an incorrect certificate chain error.

OpenSSL CA Config

Here is my extension section for dual-client client / server certificates. You can include this in your OpenSSL configuration file and refer to it when signing by specifying -extensions dual_cert .

 [ dual_cert ] # Extensions for dual-role user/server certificates (`man x509v3_config`). basicConstraints = CA:FALSE nsCertType = client, server nsComment = "Client/Server Dual-role Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer:always keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, serverAuth

Create a PEM containing a complete trust chain

To create a single PEM file containing the complete trust chain for your node certificate, simply cat all the certificate files in the reverse order from the node certificate to the CA root.

 cat node1.crt ca-intermediate.crt ca-root.crt > node1-full-chain.crt
+2
source

More articles:


All Articles