Broker-To-Broker Network Connector With SSL And JAAS Plugin

Introduction

In my previous blogs, I have explained many topics related to enterprise messaging with ActiveMQ from simple installation to complicated configuration with Network-Of-Brokers. In this blog, I am going to demonstrate in detail how to setup a Network-Of-Brokers using SSL for network connection. Currently, there is little detailed documentation available with regard to how to setup network of brokers using SSL. One of my goals of this blog is to fill this gap.

Topology

As shown in the above picture, I have setup two clusters, name WEST and EAST. Each cluster has a master-slave pair of ActiveMQ instances. The master-slave instance are located on the same Linux server. Thus, I have two Linux servers, name SERVER01 and SERVER02.

Generate Keys and Certificates

Generate Private and Public Keys On SERVERV01

First, I created the following dir:

[amq@SERVERV01 dev1certs]$ pwd
/opt/app/activemq/cluster/master-slave/certificates/dev1certs

Generate Private Key

Then, I will need to create private key namd serverv01.ks as the following:

[amq@SERVERV01 dev1certs]$ keytool -genkey -alias serverv01 -keyalg RSA -keystore serverv01.ks
Enter keystore password:
Re-enter new password:
What is your first and last name?
  [Unknown]:  Gary Liu
What is the name of your organizational unit?
  [Unknown]:  OU
What is the name of your organization?
  [Unknown]:  OG
What is the name of your City or Locality?
  [Unknown]:  MyCity
What is the name of your State or Province?
  [Unknown]:  PA
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=Gary Liu, OU=OU, O=OG, L=MyCity, ST=PA, C=US correct?
  [no]:  yes

Enter key password for 
        (RETURN if same as keystore password):
serverv01.ks

Enter password as amqadmin@. We will need this password later. Make sure you use the alias the same as host name, so that we can update the key based on the unique alias, if necessary. As you can see, we have created a private key, name serverv01.ks. We interrogate this key by the following command:

[amq@SERVERV01 democerts]$ keytool -list -keystore serverv01.ks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

serverv01, Mar 14, 2015, PrivateKeyEntry,
Certificate fingerprint (SHA1): 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3

Generate Public Key

After generating private key, we need to create public key. Outside world will use this key to communicate with ActiveMQ broker. Here is the command to generate public key [sametimes, it is called certificates. It is the same thing.].

 keytool -export -alias server1 -keystore server1.ks -file server1_cert

Here is the details:

 [amq@SERVERV01 democerts]$ keytool -export -alias serverv01 -keystore serverv01.ks -file serverv01_cert
Enter keystore password:
Certificate stored in file 
[amq@SERVERV01 democerts]$ ls
serverv01_cert  serverv01.ks

We can also verify the public key by the following command:

keytool -printcert -file serverv01_cert

Here are the details

[amq@SERVERV01 democerts]$ keytool -printcert -file serverv01_cert
Owner: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Issuer: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Serial number: 5a47ce77
Valid from: Sat Mar 14 13:57:23 CDT 2015 until: Fri Jun 12 13:57:23 CDT 2015
Certificate fingerprints:
         MD5:  C7:90:BC:22:C6:F9:E8:D0:CA:EB:DE:55:AF:D3:90:F8
         SHA1: 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
         SHA256: B2:EE:9F:AE:89:88:00:E9:9D:E8:6C:34:BF:11:82:11:0A:A1:A9:4C:80:35:39:C2:66:08:58:02:BE:9A:89:97
         Signature algorithm name: SHA256withRSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9D E7 60 DE 70 93 6E 2B   EA 19 D0 98 44 D9 9B AE  ..`.p.n+....D...
0010: DC C4 B3 AD                                        ....
]
]

Generate Shared Keystore shared.ks

We need to have shared public key store to include all the public keys in one file. Thus client can use this file communicated with server. And ActiveMQ brokers can communicate with each other. Now, we will create a file named shared.ks using the following commands:

 keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert

And here are the details:

 [amq@SERVERV01 democerts]$ keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert
Enter keystore password:
Re-enter new password:
Owner: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Issuer: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Serial number: 5a47ce77
Valid from: Sat Mar 14 13:57:23 CDT 2015 until: Fri Jun 12 13:57:23 CDT 2015
Certificate fingerprints:
         MD5:  C7:90:BC:22:C6:F9:E8:D0:CA:EB:DE:55:AF:D3:90:F8
         SHA1: 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
         SHA256: B2:EE:9F:AE:89:88:00:E9:9D:E8:6C:34:BF:11:82:11:0A:A1:A9:4C:80:35:39:C2:66:08:58:02:BE:9A:89:97
         Signature algorithm name: SHA256withRSA
         Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9D E7 60 DE 70 93 6E 2B   EA 19 D0 98 44 D9 9B AE  ..`.p.n+....D...
0010: DC C4 B3 AD                                        ....
]
]

Trust this certificate? [no]:  yes
Certificate was added to keystore

So far I have performed 3 tasks:

• Generate private key
• Generate public key
• Create a shared keystore

And I have executed the following commands:

  keytool -genkey -alias serverv01 -keyalg RSA -keystore serverv01.ks
  keytool -export -alias serverv01 -keystore serverv01.ks -file serverv01_cert
  keytool -import -keystore shared.ks -file serverv01_cert
  keytool -list -keystore shared.ks

Generate Private And Public Keys On The Second Server

Now, we need execute the same commands on the other server. In my case, the other server is name as SERVERV02. During the execution of the commands, replace all host name by the server name. [In my case, I replace serverv01 with serverv02].

And I have executed the following commands:

  keytool -genkey -alias serverv02 -keyalg RSA -keystore serverv02.ks
  keytool -export -alias serverv02 -keystore serverv02.ks -file serverv02_cert
  keytool -import -keystore shared.ks -file serverv02_cert
  keytool -list -keystore shared.ks

At this point, on each server we have 3 files:

1. HOSTNAME.ks
2. HOSTNAME_cert
3. shared.ks

This means for each host we have a private key, a public key, and shared key. The next step is to exchange public key and update the shared key.

Exchange Public Keys

In order for the ActiveMQ brokers to authenticate each other, we need to change the public keys. So that copy the file of HOSTNAME_cert to each other's server. In my case, I use scp. Here is the command for copy from server 2 to server 1:

scp gliu@serverv02:/opt/app/activemq/cluster/master-slave/certificates/democerts/serverv02_cert .

From server 1 to server 2:

scp gliu@serverv01:/opt/app/activemq/cluster/master-slave/certificates/democerts/serverv01_cert .

Now we need to import the newly copied public key to the shared keystore, namely shared.ks. Thus on server 1[serverv01] execute the following commands:

keytool -import -alias serverv02 -keystore shared.ks -file serverv02_cert

And on the server 2[serverv02], execute the following command:

keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert

We can check what is the content of the shared keystore by following command:

[amq@SERVERV02 democerts]$ keytool -list -keystore shared.ks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 2 entries

serverv02, Mar 15, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): AB:21:A4:20:2B:81:E3:23:79:37:7D:07:45:9E:0A:D0:B6:71:4B:6B
serverv01, Mar 15, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3

From the above output, we can see that the public keys from two servers [serverv01 and serverv02] are included in the shared.ks file. Upon this point, we have completed the key generation procedure. It seems a long procedure, but it is pretty straight forward in reality.

Remember, in the large network of brokers, many servers many involved. In order for the client to connect any server in the clusters, we will need to add all the keys to the shared.ks file. With this file, client will be able to connect to the all the brokers.

ActiveMQ Configuration

For each broker, we need to copy the private key [HOSTNAME.ks] and shared.ks to the $ACTIVEMQ_BASE/conf. Once that is done, we need to update the activemq.xml file.

Update activemq.xml On The WEST Cluster

The in my case, the west cluster has two nodes [WEST-NORTH and WEST-SOUTH] playing as master-slave pair. In my design, I use WEST cluster as the active part which contains the networkConnector configuration, while the EAST cluster does not define any network connector. This is desired configuration. Firstly, it is simple to define duplex connection than define two separate one-way connection. Secondly, in the enterprise application, there may be firewall restriction which prevent us from having two one-way connections. Here is the relevant configuraiton:

        ...

        
            
        

        
            
                     
                        
                     
            
            
                     
                        
                     
            
        

        ...

         
          

          
            
          
        

        
            
            
        


        
            
        

        ...

Update activemq.xml On The EAST Cluster

The configurations of the EAST cluster are almost the same as the WEST cluster, except we do not need to define the networkConnector as explained in the previous section.

Trouble Shooting

Make Sure Ports Open On Firewall

When you run the network of brokers, make sure the ports are open. If you use 61617, for example, as ssl connection, make sure that port is open from firewallusing command of:

 sudo iptables -L -v -n]

Make Sure HOSTNAME.ks and shared.ks Aviable

In my example, I explicitly tell ActiveMQ the location of the private and public keys. In that way, I know which key I am using.

Check Log

$ACTIVEMQ_BASE/data/activemq.log contains very important server information. Alway check the log after you start the broker. If connection are established, you should see something like this:

2015-03-13 16:18:28,710 | INFO  | Establishing network connection from vm://WEST-NORTH?async=false&network=true to failover:(nio+ssl://serverv02:61617,nio+ssl://serverv02:61627)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-1
2015-03-13 16:18:28,711 | INFO  | Connector vm://WEST-NORTH started | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-1
2015-03-13 16:18:28,724 | INFO  | Establishing network connection from vm://WEST-NORTH?async=false&network=true to failover:(nio+ssl://serverv02:61617,nio+ssl://serverv02:61627)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-1
2015-03-13 16:18:29,017 | INFO  | Successfully connected to nio+ssl://serverv02:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1
2015-03-13 16:18:29,021 | INFO  | Successfully connected to nio+ssl://serverv02:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1

Test Configuration

In order to test the configuration of the Network-Of-Broker with nio+ssl, I use the sample code that comes with ActiveMQ. We need to test:

1. Failover From Both Clusters
2. Producing And Consuming Messages

Test Failover

To test failover, we need stop one ActiveMQ broker at a time. Make sure the connection between two cluster are established correctly

The above picture shows that the EAST cluster has established 2 connections with the EAST cluster on the EAST-NORHT broker. One is for queue and the other for the topic connection. If we stop the north broker from the WEST cluster, we should see the new connection establish from the WEST to the EAST cluster.

Test Message Exchanging

Here are the basic messaging changing patterns need to be tested. Basically, we need to make sure if producers and consumers are located at the same cluster, they can exchange messages. In this case, we need to make sure messages are not automatically bridged to the other cluster. Another test is to make sure if producers and consumers are located at different cluster, the brokers can bridge messages to the other brokers. Here are the 4 test cases:

1. Produce and Consumer Messages From WEST cluster
2. Produce and Consumer Messages From EAST cluster
3. Produce Messages From WEST Cluster and Consumer FROM EAST Cluster
4. Produce Messages From EAST Cluster and Consumer FROM WEST Cluster

In order to execute the above test cases, I created 4 scripts as shown in the following.

publishToDev1.sh

#!/bin/bash
# publishToDev1.sh
#
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant producer \
 -Durl="failover:(nio+ssl://serverv01.mycompany.com:61617,nio+ssl://serverv01.mycompany.com:61627)" \
 -Duser="hmuser" \
 -Dpassword="admin123@" \
 -Dtopic=false \
 -Ddurable=true \
 -Dsubject=QUEUE.NIOSSL \
 -Dmax=100 \
 -Djavax.net.debug=ssl:handshake \
 -Djavax.net.ssl.keyStore=/home/amq/dev1cert/shared.ks \
 -Djavax.net.ssl.keyStorePassword=amqadmin@ \
 -Djavax.net.ssl.trustStore=/home/amq/dev1cert/shared.ks

consumeFromDev1.sh

#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant consumer  \
 -Durl="failover:(nio+ssl://serverv01.mycompany.com:61617,nio+ssl://serverv01.mycompany.com:61627)" \
 -Duser="hmuser" \
 -Dpassword="admin123@" \
 -Dtopic=false \
 -Ddurable=true \
 -Dsubject=QUEUE.NIOSSL \
 -Djavax.net.debug=ssl:handshake \
 -Djavax.net.ssl.keyStore=/home/amq/dev1cert/shared.ks \
 -Djavax.net.ssl.keyStorePassword=amqadmin@ \
 -Djavax.net.ssl.trustStore=/home/amq/dev1cert/shared.ks

publishToDev2.sh

#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant producer \
 -Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
 -Duser="admin" \
 -Dpassword="admin" \
 -Dtopic=false \
 -Ddurable=true \
 -Dsubject=QUEUE.NIOSSL \
 -Dmax=100 \
 -Djavax.net.debug=ssl:handshake \
 -Djavax.net.ssl.keyStore=/home/amq/dev2cert/shared.ks \
 -Djavax.net.ssl.keyStorePassword=amqadmin@ \
 -Djavax.net.ssl.trustStore=/home/amq/dev2cert/shared.ks

consumeFromDev2.sh

consumeFromDev2.sh

#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
# -Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
ant consumer  \
 -Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
 -Duser="hmuser" \
 -Dpassword="admin123@" \
 -Dtopic=false \
 -Ddurable=true \
 -Dsubject=QUEUE.NIOSSL \
 -Djavax.net.debug=ssl:handshake \
 -Djavax.net.ssl.keyStore=/home/amq/dev2cert/serverv02.ks \
 -Djavax.net.ssl.keyStorePassword=amqadmin@ \
 -Djavax.net.ssl.trustStore=/home/amq/dev2cert/shared.ks

Conclusion And Summary

The main purpose of this blog is explain the ActiveMQ configuration for a Network-Of-Brokers using nio+ssl. I have included the test scripts so that admin can test the configuration without writing any new java code.