Introduction
Enterprise messaging with ActiveMQ is very popular nowadays. It is very matured platform and it is free! Currently, the 5.10.0 release is available at http://activemq.apache.org/activemq-5100-release.html.In this blog, I will explain the detailed configuration about how to configure ActiveMQ instances to achieve high availability using master-slave paradigm with shared file system. The system diagram is shown below
Enough topology, now let's create the nodes.
Master-Slave Configurations
We install the apache-activemq-5.10.0 at /opt/app/amq, then create 2 nodes name north and south with the following command:cd /opt/app/amq/apache-activemq-5.10.0/bin [amq@SANDBOXFUSEV01 bin]$ ./activemq-admin create ../../north [amq@SANDBOXFUSEV01 bin]$ ./activemq-admin create ../../southNow we have the following file structures
[amq@SANDBOXFUSEV01 amq]$ pwd /opt/app/amq [amq@SANDBOXFUSEV01 amq]$ ls -lart total 24 drwxr-xr-x. 3 root root 4096 Jan 14 10:27 .. lrwxrwxrwx. 1 amq amq 23 Jan 14 10:29 activemq -> apache-activemq-5.10.0/ drwxr-xr-x. 11 amq amq 4096 Jan 14 10:42 apache-activemq-5.10.0 drwxrwxr-x. 3 amq amq 4096 Jan 20 12:38 tomcat drwxrwxr-x. 4 amq amq 4096 Jan 24 14:04 north drwxr-xr-x. 6 amq amqadmin 4096 Jan 24 14:06 . drwxrwxr-x. 4 amq amq 4096 Jan 24 14:06 south
Configure North
Now we need to modify the activemq.xml file with the following changes: ]17The line with useJmx="true" is for enable JMX and the line47 48 49 5859 8860 89 90 ....91 92 93 94 95
is to tell ActiveMQ to use port 11099 as JMX connection port. The rmiServerPort must be defined, otherwise, the remote access will not be possible. The reason for this is that most enterprise environment block the unspecified port.
Configure South
The South instance will be similar as the North. We just need to change the port number. For the completeness, here are the changes:1747 48 49 5859 8860 90 ....91 92 93 94 95
Do the same for the second node. The start the north node on the first server by executing the following command:
Master-Slave In Action
Now, we have done all the configuration changes. We can start all the process and to see how this paradigm of high availability works.cd /opt/app/amq/north/bin ./north start
I have created two aliases which are very useful:
alias amqjava='ps -eaf | egrep java | egrep -v grep | egrep amq' alias amqport='netstat -anp | egrep tcp | egrep java | egrep -v grep'
Where amqjava is to check java process owned by amq user, and amqport is to check port information about amq java process. Run amqport command immediately after you start the amq java process. Here the port information you will see:
tcp 0 0 :::41882 :::* LISTEN 17720/java tcp 0 0 :::1883 :::* LISTEN 17720/java tcp 0 0 :::11099 :::* LISTEN 17720/java tcp 0 0 :::44444 :::* LISTEN 17720/java tcp 0 0 :::8161 :::* LISTEN 17720/java tcp 0 0 :::5672 :::* LISTEN 17720/java tcp 0 0 :::61613 :::* LISTEN 17720/java tcp 0 0 :::61614 :::* LISTEN 17720/java tcp 0 0 :::61616 :::* LISTEN 17720/java tcp 0 0 ::ffff:10.66.13.119:37444 ::ffff:10.66.13.119:44444 ESTABLISHED 17720/java tcp 0 0 ::ffff:127.0.0.1:36187 ::ffff:127.0.0.1:11099 ESTABLISHED 17720/java tcp 0 0 ::ffff:10.66.13.119:44444 ::ffff:10.66.13.119:37444 ESTABLISHED 17720/java tcp 0 0 ::ffff:127.0.0.1:11099 ::ffff:127.0.0.1:36187 ESTABLISHED 17720/java
Another way to check these port is to use lsof command as shwon below:
[amq@SANDBOXFUSEV01 bin]$ lsof -P -iTCP COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java 17720 amq 9u IPv6 1300784 0t0 TCP *:41882 (LISTEN) java 17720 amq 111u IPv6 1300785 0t0 TCP *:11099 (LISTEN) java 17720 amq 113u IPv6 1300786 0t0 TCP *:44444 (LISTEN) java 17720 amq 125u IPv6 1300795 0t0 TCP *:61616 (LISTEN) java 17720 amq 126u IPv6 1300796 0t0 TCP *:5672 (LISTEN) java 17720 amq 127u IPv6 1300797 0t0 TCP *:61613 (LISTEN) java 17720 amq 128u IPv6 1300798 0t0 TCP *:1883 (LISTEN) java 17720 amq 129u IPv6 1300800 0t0 TCP *:61614 (LISTEN) java 17720 amq 136u IPv6 1300811 0t0 TCP *:8161 (LISTEN)
At this time, the ActiveMQ north instance is running. I can verify this by open brower with URL of sandboxfusev01:8161. This is activemq embedded web console. The default password is admin/admin
Now lets start the second instance in the folder of /opt/app/amq/south
cd /opt/app/amq/south/bin ./south start
I can see the following stdout:
[amq@SANDBOXFUSEV01 bin]$ ./south start INFO: Using default configuration (you can configure options in one of these file: /etc/default/activemq /home/amq/.activemqrc) INFO: Invoke the following command to create a configuration file /opt/app/amq/apache-activemq-5.10.0/bin/activemq setup [ /etc/default/activemq | /home/amq/.activemqrc ] INFO: Using java '/usr/bin/java' INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details INFO: pidfile created : '/opt/app/amq/south/data/activemq-SANDBOXFUSEV01.pid' (pid '17879')
When I chack amqjava, I get the following
[amq@SANDBOXFUSEV01 bin]$ amqjava amq 17720 1 0 18:52 pts/0 00:00:08 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/north/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/north/tmp -Dactivemq.classpath=/opt/app/amq/north/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/north -Dactivemq.conf=/opt/app/amq/north/conf -Dactivemq.data=/opt/app/amq/north/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start amq 17879 1 24 19:24 pts/0 00:00:02 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/south/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/south/tmp -Dactivemq.classpath=/opt/app/amq/south/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/south -Dactivemq.conf=/opt/app/amq/south/conf -Dactivemq.data=/opt/app/amq/south/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start [amq@SANDBOXFUSEV01 bin]$
This indicating there are two activemq java process are running with process ID of 17720 and 17879. Now lets check what port has been opened by these two java processes
[amq@SANDBOXFUSEV01 bin]$ amqport (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) tcp 0 0 :::41882 :::* LISTEN 17720/java tcp 0 0 :::1883 :::* LISTEN 17720/java tcp 0 0 :::11099 :::* LISTEN 17720/java tcp 0 0 :::44444 :::* LISTEN 17720/java tcp 0 0 :::44445 :::* LISTEN 17879/java tcp 0 0 :::8161 :::* LISTEN 17720/java tcp 0 0 :::12099 :::* LISTEN 17879/java tcp 0 0 :::5672 :::* LISTEN 17720/java tcp 0 0 :::36010 :::* LISTEN 17879/java tcp 0 0 :::61613 :::* LISTEN 17720/java tcp 0 0 :::61614 :::* LISTEN 17720/java tcp 0 0 :::61616 :::* LISTEN 17720/java [amq@SANDBOXFUSEV01 bin]$
Interestingly, I see only 44445 and 12099 ports opened by the second java process. You may expect 61626 and other ports are open. This is because the south instance is in standby mode. Now if I tail the log at /opt/app/amq/south/data/activemq.log, if the following output:
[amq@SANDBOXFUSEV01 bin]$ tail -10f ../data/activemq.log 2015-01-24 19:33:21,686 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:33:31,688 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:33:41,689 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:33:51,691 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:01,696 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:11,699 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:21,728 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:31,730 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:41,731 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:34:51,733 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main 2015-01-24 19:35:01,736 | INFO | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
As you can see, every 10 seconds, the south instance is throwing an IOException sayiing the lock at '/amqdata/master-slave/data/kahadb/lock' is not accessable. This is exactly what we want. This means the south instance is a slave node now. For the purpose of showing you extra information, I can use lsof utility to check who is locking the file of '/amqdata/master-slave/data/kahadb/lock' by the following command:
[amq@SANDBOXFUSEV01 kahadb]$ cd /amqdata/master-slave/data/kahadb [amq@SANDBOXFUSEV01 kahadb]$ ls db-2.log db.data db.redo lock [amq@SANDBOXFUSEV01 kahadb]$ lsof | egrep lock | egrep java java 17720 amq 118uw REG 0,21 0 4321950309 /amqdata/master-slave/data/kahadb/lock [amq@SANDBOXFUSEV01 kahadb]$ amqjava amq 17720 1 0 18:52 pts/0 00:00:09 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/north/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/north/tmp -Dactivemq.classpath=/opt/app/amq/north/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/north -Dactivemq.conf=/opt/app/amq/north/conf -Dactivemq.data=/opt/app/amq/north/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start amq 17879 1 0 19:24 pts/0 00:00:03 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/south/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/south/tmp -Dactivemq.classpath=/opt/app/amq/south/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/south -Dactivemq.conf=/opt/app/amq/south/conf -Dactivemq.data=/opt/app/amq/south/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start [amq@SANDBOXFUSEV01 kahadb]$
From above command and result, I can see that the java process with ID of 17720 is holding the lock at /amqdata/master-slave/data/kahadb/lock.
In the same maner, we can start the two ActiveMA instance in the second server. We have proved the only one instance of activemq node in the cluster is holding the lock and function
Summary
So far we have covered the basic configuration of ActiveMQ for high availability using the Master-Slave paradigm with shared file system. In this paradigm, only master will grab the lock all the ActiveMA instances in the same cluster will be in a standby mode. We call them slaves. I also explained how to setup JMX and how to check ports which are ready for connections.
Before we further explain how Master-Slave paradigm works, I would like demonstrate how to use JVisualVM and Tomcat container to monitor ActiveMA instances (nodes). I will cover these two topics in the next two blogs.
i was create two broker node , broker-1 and broker-2 and also change the activemq.xml file in broker-1 but i am not able to start the first node
ReplyDelete