Introduction
In my previous blogs, I have explained many topics related to enterprise messaging with ActiveMQ from simple installation to complicated configuration with Network-Of-Brokers. In this blog, I am going to demonstrate in detail how to setup a Network-Of-Brokers using SSL for network connection. Currently, there is little detailed documentation available with regard to how to setup network of brokers using SSL. One of my goals of this blog is to fill this gap.
Topology
As shown in the above picture, I have setup two clusters, name WEST and EAST. Each cluster has a master-slave pair of ActiveMQ instances. The master-slave instance are located on the same Linux server. Thus, I have two Linux servers, name SERVER01 and SERVER02.
Generate Keys and Certificates
Generate Private and Public Keys On SERVERV01
First, I created the following dir:
[amq@SERVERV01 dev1certs]$ pwd
/opt/app/activemq/cluster/master-slave/certificates/dev1certs
Generate Private Key
Then, I will need to create private key namd serverv01.ks as the following:
[amq@SERVERV01 dev1certs]$ keytool -genkey -alias serverv01 -keyalg RSA -keystore serverv01.ks
Enter keystore password:
Re-enter new password:
What is your first and last name?
[Unknown]: Gary Liu
What is the name of your organizational unit?
[Unknown]: OU
What is the name of your organization?
[Unknown]: OG
What is the name of your City or Locality?
[Unknown]: MyCity
What is the name of your State or Province?
[Unknown]: PA
What is the two-letter country code for this unit?
[Unknown]: US
Is CN=Gary Liu, OU=OU, O=OG, L=MyCity, ST=PA, C=US correct?
[no]: yes
Enter key password for
(RETURN if same as keystore password):
serverv01.ks
Enter password as amqadmin@. We will need this password later. Make sure you use the alias the same as host name, so that we can update the key based on the unique alias, if necessary. As you can see, we have created a private key, name serverv01.ks. We interrogate this key by the following command:
[amq@SERVERV01 democerts]$ keytool -list -keystore serverv01.ks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 1 entry
serverv01, Mar 14, 2015, PrivateKeyEntry,
Certificate fingerprint (SHA1): 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
Generate Public Key
After generating private key, we need to create public key. Outside world will use this key to communicate with ActiveMQ broker. Here is the command to generate public key [sametimes, it is called certificates. It is the same thing.].
keytool -export -alias server1 -keystore server1.ks -file server1_cert
Here is the details:
[amq@SERVERV01 democerts]$ keytool -export -alias serverv01 -keystore serverv01.ks -file serverv01_cert
Enter keystore password:
Certificate stored in file
[amq@SERVERV01 democerts]$ ls
serverv01_cert serverv01.ks
We can also verify the public key by the following command:
keytool -printcert -file serverv01_cert
Here are the details
[amq@SERVERV01 democerts]$ keytool -printcert -file serverv01_cert
Owner: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Issuer: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Serial number: 5a47ce77
Valid from: Sat Mar 14 13:57:23 CDT 2015 until: Fri Jun 12 13:57:23 CDT 2015
Certificate fingerprints:
MD5: C7:90:BC:22:C6:F9:E8:D0:CA:EB:DE:55:AF:D3:90:F8
SHA1: 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
SHA256: B2:EE:9F:AE:89:88:00:E9:9D:E8:6C:34:BF:11:82:11:0A:A1:A9:4C:80:35:39:C2:66:08:58:02:BE:9A:89:97
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9D E7 60 DE 70 93 6E 2B EA 19 D0 98 44 D9 9B AE ..`.p.n+....D...
0010: DC C4 B3 AD ....
]
]
Generate Shared Keystore shared.ks
We need to have shared public key store to include all the public keys in one file. Thus client can use this file communicated with server. And ActiveMQ brokers can communicate with each other. Now, we will create a file named shared.ks using the following commands:
keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert
And here are the details:
[amq@SERVERV01 democerts]$ keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert
Enter keystore password:
Re-enter new password:
Owner: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Issuer: CN=Gary Liu, OU=OU, O=OG, L=Jessup, ST=PA, C=US
Serial number: 5a47ce77
Valid from: Sat Mar 14 13:57:23 CDT 2015 until: Fri Jun 12 13:57:23 CDT 2015
Certificate fingerprints:
MD5: C7:90:BC:22:C6:F9:E8:D0:CA:EB:DE:55:AF:D3:90:F8
SHA1: 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
SHA256: B2:EE:9F:AE:89:88:00:E9:9D:E8:6C:34:BF:11:82:11:0A:A1:A9:4C:80:35:39:C2:66:08:58:02:BE:9A:89:97
Signature algorithm name: SHA256withRSA
Version: 3
Extensions:
#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 9D E7 60 DE 70 93 6E 2B EA 19 D0 98 44 D9 9B AE ..`.p.n+....D...
0010: DC C4 B3 AD ....
]
]
Trust this certificate? [no]: yes
Certificate was added to keystore
So far I have performed 3 tasks:
- Generate private key
- Generate public key
- Create a shared keystore
And I have executed the following commands:
keytool -genkey -alias serverv01 -keyalg RSA -keystore serverv01.ks
keytool -export -alias serverv01 -keystore serverv01.ks -file serverv01_cert
keytool -import -keystore shared.ks -file serverv01_cert
keytool -list -keystore shared.ks
Generate Private And Public Keys On The Second Server
Now, we need execute the same commands on the other server. In my case, the other server is name as SERVERV02. During the execution of the commands, replace all host name by the server name. [In my case, I replace serverv01 with serverv02].
And I have executed the following commands:
keytool -genkey -alias serverv02 -keyalg RSA -keystore serverv02.ks
keytool -export -alias serverv02 -keystore serverv02.ks -file serverv02_cert
keytool -import -keystore shared.ks -file serverv02_cert
keytool -list -keystore shared.ks
At this point, on each server we have 3 files:
- HOSTNAME.ks
- HOSTNAME_cert
- shared.ks
-
This means for each host we have a private key, a public key, and shared key. The next step is to exchange public key and update the shared key.
Exchange Public Keys
In order for the ActiveMQ brokers to authenticate each other, we need to change the public keys. So that copy the file of HOSTNAME_cert to each other's server. In my case, I use scp. Here is the command for copy from server 2 to server 1:
scp gliu@serverv02:/opt/app/activemq/cluster/master-slave/certificates/democerts/serverv02_cert .
From server 1 to server 2:
scp gliu@serverv01:/opt/app/activemq/cluster/master-slave/certificates/democerts/serverv01_cert .
Now we need to import the newly copied public key to the shared keystore, namely shared.ks. Thus on server 1[serverv01] execute the following commands:
keytool -import -alias serverv02 -keystore shared.ks -file serverv02_cert
And on the server 2[serverv02], execute the following command:
keytool -import -alias serverv01 -keystore shared.ks -file serverv01_cert
We can check what is the content of the shared keystore by following command:
[amq@SERVERV02 democerts]$ keytool -list -keystore shared.ks
Enter keystore password:
Keystore type: JKS
Keystore provider: SUN
Your keystore contains 2 entries
serverv02, Mar 15, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): AB:21:A4:20:2B:81:E3:23:79:37:7D:07:45:9E:0A:D0:B6:71:4B:6B
serverv01, Mar 15, 2015, trustedCertEntry,
Certificate fingerprint (SHA1): 31:43:79:2B:33:3E:73:8A:C1:1C:3C:D1:EA:68:ED:00:6A:8C:F6:D3
From the above output, we can see that the public keys from two servers [serverv01 and serverv02] are included in the shared.ks file. Upon this point, we have completed the key generation procedure. It seems a long procedure, but it is pretty straight forward in reality.
Remember, in the large network of brokers, many servers many involved. In order for the client to connect any server in the clusters, we will need to add all the keys to the shared.ks file. With this file, client will be able to connect to the all the brokers.
ActiveMQ Configuration
For each broker, we need to copy the private key [HOSTNAME.ks] and shared.ks to the $ACTIVEMQ_BASE/conf. Once that is done, we need to update the activemq.xml file.
Update activemq.xml On The WEST Cluster
The in my case, the west cluster has two nodes [WEST-NORTH and WEST-SOUTH] playing as master-slave pair. In my design, I use WEST cluster as the active part which contains the networkConnector configuration, while the EAST cluster does not define any network connector. This is desired configuration. Firstly, it is simple to define duplex connection than define two separate one-way connection. Secondly, in the enterprise application, there may be firewall restriction which prevent us from having two one-way connections. Here is the relevant configuraiton:
...
...
...
Update activemq.xml On The EAST Cluster
The configurations of the EAST cluster are almost the same as the WEST cluster, except we do not need to define the networkConnector as explained in the previous section.
Trouble Shooting
Make Sure Ports Open On Firewall
When you run the network of brokers, make sure the ports are open. If you use 61617, for example, as ssl connection, make sure that port is open from firewallusing command of:
sudo iptables -L -v -n]
Make Sure HOSTNAME.ks and shared.ks Aviable
In my example, I explicitly tell ActiveMQ the location of the private and public keys. In that way, I know which key I am using.
Check Log
$ACTIVEMQ_BASE/data/activemq.log contains very important server information. Alway check the log after you start the broker. If connection are established, you should see something like this:
2015-03-13 16:18:28,710 | INFO | Establishing network connection from vm://WEST-NORTH?async=false&network=true to failover:(nio+ssl://serverv02:61617,nio+ssl://serverv02:61627)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-1
2015-03-13 16:18:28,711 | INFO | Connector vm://WEST-NORTH started | org.apache.activemq.broker.TransportConnector | ActiveMQ Task-1
2015-03-13 16:18:28,724 | INFO | Establishing network connection from vm://WEST-NORTH?async=false&network=true to failover:(nio+ssl://serverv02:61617,nio+ssl://serverv02:61627)?randomize=false&maxReconnectAttempts=0 | org.apache.activemq.network.DiscoveryNetworkConnector | ActiveMQ Task-1
2015-03-13 16:18:29,017 | INFO | Successfully connected to nio+ssl://serverv02:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1
2015-03-13 16:18:29,021 | INFO | Successfully connected to nio+ssl://serverv02:61617 | org.apache.activemq.transport.failover.FailoverTransport | ActiveMQ Task-1
Test Configuration
In order to test the configuration of the Network-Of-Broker with nio+ssl, I use the sample code that comes with ActiveMQ. We need to test:
- Failover From Both Clusters
- Producing And Consuming Messages
Test Failover
To test failover, we need stop one ActiveMQ broker at a time. Make sure the connection between two cluster are established correctly
The above picture shows that the EAST cluster has established 2 connections with the EAST cluster on the EAST-NORHT broker. One is for queue and the other for the topic connection. If we stop the north broker from the WEST cluster, we should see the new connection establish from the WEST to the EAST cluster.
Test Message Exchanging
Here are the basic messaging changing patterns need to be tested. Basically, we need to make sure if producers and consumers are located at the same cluster, they can exchange messages. In this case, we need to make sure messages are not automatically bridged to the other cluster. Another test is to make sure if producers and consumers are located at different cluster, the brokers can bridge messages to the other brokers. Here are the 4 test cases:
- Produce and Consumer Messages From WEST cluster
- Produce and Consumer Messages From EAST cluster
- Produce Messages From WEST Cluster and Consumer FROM EAST Cluster
- Produce Messages From EAST Cluster and Consumer FROM WEST Cluster
In order to execute the above test cases, I created 4 scripts as shown in the following.
publishToDev1.sh
#!/bin/bash
# publishToDev1.sh
#
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant producer \
-Durl="failover:(nio+ssl://serverv01.mycompany.com:61617,nio+ssl://serverv01.mycompany.com:61627)" \
-Duser="hmuser" \
-Dpassword="admin123@" \
-Dtopic=false \
-Ddurable=true \
-Dsubject=QUEUE.NIOSSL \
-Dmax=100 \
-Djavax.net.debug=ssl:handshake \
-Djavax.net.ssl.keyStore=/home/amq/dev1cert/shared.ks \
-Djavax.net.ssl.keyStorePassword=amqadmin@ \
-Djavax.net.ssl.trustStore=/home/amq/dev1cert/shared.ks
consumeFromDev1.sh
#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant consumer \
-Durl="failover:(nio+ssl://serverv01.mycompany.com:61617,nio+ssl://serverv01.mycompany.com:61627)" \
-Duser="hmuser" \
-Dpassword="admin123@" \
-Dtopic=false \
-Ddurable=true \
-Dsubject=QUEUE.NIOSSL \
-Djavax.net.debug=ssl:handshake \
-Djavax.net.ssl.keyStore=/home/amq/dev1cert/shared.ks \
-Djavax.net.ssl.keyStorePassword=amqadmin@ \
-Djavax.net.ssl.trustStore=/home/amq/dev1cert/shared.ks
publishToDev2.sh
#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
ant producer \
-Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
-Duser="admin" \
-Dpassword="admin" \
-Dtopic=false \
-Ddurable=true \
-Dsubject=QUEUE.NIOSSL \
-Dmax=100 \
-Djavax.net.debug=ssl:handshake \
-Djavax.net.ssl.keyStore=/home/amq/dev2cert/shared.ks \
-Djavax.net.ssl.keyStorePassword=amqadmin@ \
-Djavax.net.ssl.trustStore=/home/amq/dev2cert/shared.ks
consumeFromDev2.sh
consumeFromDev2.sh
#!/bin/bash
ACTIVEMQ_HOME=/opt/app/amq/Transport/NioSsl
# -Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
ant consumer \
-Durl="failover:(nio+ssl://serverv02.mycompany.com:61617,nio+ssl://serverv02.mycompany.com:61627)" \
-Duser="hmuser" \
-Dpassword="admin123@" \
-Dtopic=false \
-Ddurable=true \
-Dsubject=QUEUE.NIOSSL \
-Djavax.net.debug=ssl:handshake \
-Djavax.net.ssl.keyStore=/home/amq/dev2cert/serverv02.ks \
-Djavax.net.ssl.keyStorePassword=amqadmin@ \
-Djavax.net.ssl.trustStore=/home/amq/dev2cert/shared.ks
Conclusion And Summary
The main purpose of this blog is explain the ActiveMQ configuration for a Network-Of-Brokers using nio+ssl. I have included the test scripts so that admin can test the configuration without writing any new java code.
