Monday, January 26, 2015

ActiveMQ Configuration Explained: Master-Slave With Network Of Brokers

Introduction

In my previous posts, I have covered tutorial about ActiveMQ with the following aspects:

I have explained how master-slave paradigm with shared file system works. In this post, I will explain more advanced topic, namely, Network Of Brokers. I will demonstrate the procedures to setup a network of ActiveMQ brokers with Master-Slave pairs.

Network Topology

As shown in the above figure, I configured two clusters. The WEST cluster contains two ActiveMQ nodes, namely, north and south. These two node forms a master-slave pair. Depending on applications, the nodes number could be 2 or 3. These nodes sharing the same NFS where the kahaDB is located. The same is for the EAST cluster.

Configurations

Directory Layout

The ActiveMQ is installed at:

?
1
/opt/app/activemq/apache-activemq-5.10.0

The tomcat is installed at:

?
1
/opt/app/activemq/cluster/master-slave/tomcat/apache-tomcat-8.0.17

north and south nodes

?
1
2
/opt/app/amq/NetworkOfBrokers/cluster/north
/opt/app/amq/NetworkOfBrokers/cluster/south

activemq.xml

To setup network of brokers as shown in the introduction, we just need to update the activemq.xml file from west cluster. The file contents is as the following:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
  1 <!--xml version="1.0" encoding="UTF-8"?-->
  2
  3 <beans xmlns="http://www.springframework.org/schema/beans" 4="" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 5="" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd   http:/    /activemq.apache.org/schema/core http://activemq.apache.org/schema/core/activemq-core.xsd">
  6
  7     <bean class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer">
  8         <property name="locations">
  9             <value>file:${activemq.conf}/credentials.properties</value>
 10         </property>
 11     </bean>
 12
 13     <bean class="org.fusesource.insight.log.log4j.Log4jLogQuery" 14="" destroy-method="stop" 15="" id="logQuery" 16="" init-method="start" 17="" lazy-init="false" 18="" scope="singleton">
 19     </bean>
 20
 21     <broker xmlns="http://activemq.apache.org/schema/core" 22="" brokername="WEST-NORTH" 23="" datadirectory="${activemq.data}" 24="" usejmx="true">
 25
 26         <destinationpolicy>
 27             <policymap>
 28               <policyentries>
 29                 <policyentry queue=">">
 30                   <networkbridgefilterfactory>
 31                      <conditionalnetworkbridgefilterfactory replaywhennoconsumers="true">
 32                   </conditionalnetworkbridgefilterfactory></networkbridgefilterfactory>
 33                   <pendingmessagelimitstrategy>
 34                     <constantpendingmessagelimitstrategy limit="1000">
 35                   </constantpendingmessagelimitstrategy></pendingmessagelimitstrategy>
 36                 </policyentry>
 37                 <policyentry topic=">">
 38                   <pendingmessagelimitstrategy>
 39                     <constantpendingmessagelimitstrategy limit="1000">
 40                   </constantpendingmessagelimitstrategy></pendingmessagelimitstrategy>
 41                 </policyentry>
 42               </policyentries>
 43             </policymap>
 44         </destinationpolicy>
 45
 46         <managementcontext>
 47             <managementcontext connectorport="11099" createconnector="true" rmiserverport="44444">
 48         </managementcontext>
 49
 50         <networkconnectors>
 51             <networkconnector 52="" uri="masterslave://(tcp://sandboxfusev02:61616,tcp://sandboxfusev02:61626)" 53="" name="Queue_Connector" 54="" decreasenetworkconsumerpriority="true" 55="" conduitsubscriptions="false" 56="" suppressduplicatequeuesubscriptions="true" 57="" networkttl="2" 58="" messagettl="2" 59="" consumerttl="2" 60="" duplex="true" 61="" staticbridge="false">
 62
 63                      <dynamicallyincludeddestinations>
 64                      </dynamicallyincludeddestinations>
 65
 66                      <excludeddestinations>
 67                         <topic physicalname=">">
 68                      </topic></excludeddestinations>
 69             </networkconnector>
 70             <networkconnector 71="" uri="masterslave://(tcp://sandboxfusev02:61616,tcp://sandboxfusev02:61626)" 72="" name="Topic_Connector" 73="" decreasenetworkconsumerpriority="true" 74="" conduitsubscriptions="true" 75="" suppressduplicatequeuesubscriptions="true" 76="" networkttl="2" 77="" messagettl="2" 78="" consumerttl="2" 79="" duplex="true" 80="" staticbridge="false">
 81
 82                      <dynamicallyincludeddestinations>
 83                      </dynamicallyincludeddestinations>
 84
 85                      <excludeddestinations>
 86                         <queue physicalname=">">
 87                      </queue></excludeddestinations>
 88             </networkconnector>
 89         </networkconnectors>
 90
 91         <persistenceadapter>
 92             <kahadb directory="/amqdata/networkofbrokers/west-cluster/data">
 93         </kahadb></persistenceadapter>
 94
 95         <systemusage>
 96             <systemusage>
 97                 <memoryusage>
 98                     <memoryusage percentofjvmheap="70">
 99                 </memoryusage>
100                 <storeusage>
101                     <storeusage limit="100 gb">
102                 </storeusage>
103                 <tempusage>
104                     <tempusage limit="50 gb">
105                 </tempusage>
106             </tempusage></storeusage></memoryusage></systemusage>
107         </systemusage>
108
109         <transportconnectors>
110             <transportconnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
111             <transportconnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
112             <transportconnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
113             <transportconnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
114             <transportconnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
115         </transportconnector></transportconnector></transportconnector></transportconnector></transportconnector></transportconnectors>
116
117         <shutdownhooks>
118             <bean xmlns="http://www.springframework.org/schema/beans" class="org.apache.activemq.hooks.SpringContextHook">
119         </bean></shutdownhooks>
120
121     </managementcontext></broker>
122 </beans>

From the above configuration, you can see that I have configured two network connections, namely, Queue_Connector and Topic_Connector. Copy this activemq.xml to both north and south nodes, and leave the east cluster as it is as show my my previous blog of pure master-slave clustering.

at No comments:

Sunday, January 25, 2015

Monitoring ActiveMQ Using Tomcat Web Container

So far I have covered tutorial about ActiveMQ with the following aspects:

In this post, I will cover the topic on how to setup Apache Tomcat to monitoring ActiveMQ node in a Master-Slave cluster. In the next post, I will expain the more advanced high availability features for ActiveMQ, namely, Network Of Brokers

Installation

Current release of apache tomcat is version 8.0.17. We can download it from http://tomcat.apache.org/download-80.cgi. I am using RedHat Linux, so I downloaded tar.gz version.

After download Tomcat, we need to download two more archives:

Here are the download files I have:

?
1
2
3
4
5
6
[amq@SANDBOXFUSEV01 current]$ ll /software/TOMCAT/
total 42069
-rwxr-xr-x. 1 16777217 wheel  6371237 Jan 20 09:16 activemq-all-5.10.0.jar
-rwxr-xr-x. 1 16777217 wheel 12753574 Jan 20 09:16 activemq-web-console-5.10.0.war
-rwxr-xr-x. 1 16777217 wheel  9264418 Jan 20 09:15 apache-tomcat-8.0.17.tar.gz
[amq@SANDBOXFUSEV01 current]$

Once it is download executed the following commands:

?
1
2
cd /opt/app/amq/tomcat
tar -vxzf /software/TOMCAT/apache-tomcat-8.0.17.tar.gz

Here is the directory structures:

?
1
2
3
4
5
6
7
[amq@SANDBOXFUSEV01 tomcat]$ pwd
/opt/app/amq/tomcat
[amq@SANDBOXFUSEV01 tomcat]$ ll
total 4
drwxrwxr-x. 9 amq amq 4096 Jan 20 11:09 apache-tomcat-8.0.17
lrwxrwxrwx. 1 amq amq   20 Jan 20 12:38 current -> apache-tomcat-8.0.17
[amq@SANDBOXFUSEV01 tomcat]$

Now we need to copy the activemq-all-5.10.0.jar to tomcat's lib directory and copy activemq-web-console-5.10.0.war to webapps with the following commands:

?
1
2
cp /software/TOMCAT/activemq-all-5.10.0.jar /opt/app/amq/tomcat/current/lib/.
cp /software/TOMCAT/activemq-web-console-5.10.0.war  /opt/app/amq/tomcat/current/webapps/.

Configuration Using TCP Connector

With all the software in place, we need to modify catalina.sh file by adding the following lines: [I attach few extra lines, so that you know where are the lines in the catalina.sh file]

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# Uncomment the following line to make the umask available when using the
# org.apache.catalina.security.SecurityListener
#JAVA_OPTS="$JAVA_OPTS -Dorg.apache.catalina.security.SecurityListener.UMASK=`umask`"
 
JMX_OPTS=" -Dwebconsole.jmx.url=service:jmx:rmi:///jndi/rmi://sandboxfusev01:11099/jmxrmi"
JMX_OPTS="$JMX_OPTS,service:jmx:rmi:///jndi/rmi://sandboxfusev01:12099/jmxrmi"
 
JAVA_OPTS="-Dwebconsole.type=properties "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jms.url=\"failover:(tcp://sandboxfusev01:61616,tcp://sandboxfusev01:61626)\" "
JAVA_OPTS="$JAVA_OPTS $JMX_OPTS "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jmx.user=admin "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jmx.password=activemq"
 
 
# ----- Execute The Requested Command -----------------------------------------

After configuring the Tomcat for monitoring ActiveMQ, you may disable the embeded web console from ActiveMA. By doing this, you can simple comment out the line:

?
1
2
3
<!--
    <import resource="jetty.xml"/>
-->

Configuration Using NIO+SLL Connector

If ActiveMQ's transport connector uses nio+ssl, the connection url will have the following form:

For more details about how to configure ActiveMQ with nio+ssl, see my blog at:

In order to connect ActiveMQ brokers, we need update the catalina.sh file with the following:

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
JMX_OPTS=" -Dwebconsole.jmx.url=service:jmx:rmi:///jndi/rmi://devrhamqv01:11099/jmxrmi"
JMX_OPTS="$JMX_OPTS,service:jmx:rmi:///jndi/rmi://devrhamqv01:12099/jmxrmi"
 
JAVA_OPTS="-Dwebconsole.type=properties "
#JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jms.url=\"failover:(tcp://host-name:61616,tcp://hostname:61626)\" "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jms.url=\"failover:(nio+ssl://host-name01:61617,nio+ssl://host-name:61627)\" "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jms.user=admin"
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jms.password=admin"
 
JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStore=/opt/app/activemq/cluster/master-slave/north/conf/shared.ks "
JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.keyStorePassword=amqadmin@ "
JAVA_OPTS="$JAVA_OPTS -Djavax.net.ssl.trustStore=/opt/app/activemq/cluster/master-slave/north/conf/shared.ks "
 
JAVA_OPTS="$JAVA_OPTS $JMX_OPTS "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jmx.user=admin "
JAVA_OPTS="$JAVA_OPTS -Dwebconsole.jmx.password=activemq"

Monitoring ActiveMQ In Action

Now lets start tomcat by running startup.sh. Make sure the ActiveMQ-WEB is load successfully. You can check the log file at /opt/app/amq/tomcat/current/logs/catalina.out. And the following lines should be shown:

?
1
25-Jan-2015 04:18:48.567 INFO [localhost-startStop-1] org.apache.catalina.startup.HostConfig.deployWAR Deployment of web application archive /opt/app/amq/tomcat/apache-tomcat-8.0.17/webapps/activemq-web-console-5.10.0.war has finished in 2,863 ms

Open the web broser with the following URL:

?
1
http://sandboxfusev01:8080/activemq-web-console-5.10.0/

As shown in the above picture, tomcat container has successfully connected to the north ActiveMQ node on the host SANDBOXFUSEV01

we verify north node is running be the following:

?
1
2
3
4
5
6
7
8
9
10
[amq@SANDBOXFUSEV01 bin]$ ./north status
INFO: Using default configuration
(you can configure options in one of these file: /etc/default/activemq /home/amq/.activemqrc)
 
INFO: Invoke the following command to create a configuration file
/opt/app/amq/apache-activemq-5.10.0/bin/activemq setup [ /etc/default/activemq | /home/amq/.activemqrc ]
 
INFO: Using java '/usr/bin/java'
ActiveMQ is running (pid '17720')
[amq@SANDBOXFUSEV01 bin]$

We know that is correct. Now let's stop the north node and see what happen. After refresh the web browser I see the following:

The new master ActiveMQ node is north from SANDBOXFUSEV02. I did not find a particular order about which slave will become next master. It seems whichever slave grab the lock at a particular moment after the master dies, will be come the new master.

Enable TLS/SSL

For details about configuring TLS/SSL, please refer: https://tomcat.apache.org/tomcat-8.0-doc/ssl-howto.html. Here are the procedure I have executed:

  1. generate private key using keytool
  2. put the keystore $HOME/$HOSTNAME.ks
  3. update $TOMCAT_HOME/conf/server.xml
  4. open the port 8443 from firewall using system-config-firewall

And here is the connector configuration:

?
1
2
3
    <connector port="8443" protocol="org.apache.coyote.http11.Http11NioProtocol" maxthreads="150" sslenabled="true" scheme="https" secure="true" keystorefile="${user.home}/keystore/server.ks" keystorepass="amqadmin@" clientauth="false" sslprotocol="TLS">
 
</connector>

Summary

We have successfuly configured the tomcat to monitor ActiveMQ instance. The good thing about using Tomcat we container to monitoring activemq instance is that it automatically connects to the master node. The installation and configuration are fairly straightforward.
at 2 comments:

Monitor ActiveMQ Nodes Using JMX

Introduction

In my previous post, http://ggl-consulting.blogspot.com/2015/01/configure-activemq-for-high-available.html, I have demonstrate in detail about how to configure ActiveMQ for high-availability using Master-Slave Paradigm. In this post, I will demonstrate the process to setup jVisualVM to monitoring all ActiveMQ instances in the cluster.

jVisualVM Setup

jVisualVM is a tool from JDK. This is a very powerful visual tool to monitor java process, create thread and memory dumps, display dashboard, etc. It is much powerful than jconsole which also comes with JDK.

To start jVisualVM, simple execute the following command [assume JDK is installed and jVisualVM is in your path]:

?
1
jvisualvm

As shown in the above figure, if you run for the first time the jvisualvm, you will need to install MBean plugin by Tool-->Available Plugins. In my case, I have already installed the plugin, so that the available plugins only show 15 items.

Once the MB plugin is installed, we can add jmx connections. In my case, I will need to add the following 4 connections:

service:jmx:rmi:///jndi/rmi://sandboxfusev01:11099/jmxrmi
service:jmx:rmi:///jndi/rmi://sandboxfusev01:12099/jmxrmi
service:jmx:rmi:///jndi/rmi://sandboxfusev02:11099/jmxrm
service:jmx:rmi:///jndi/rmi://sandboxfusev02:12099/jmxrm

After that I can see connections as the following

If I browe all 4 connections, I can see only the north ActiveMQ node from sandboxfusev01 is active. The other three node will show information like this:

This is indicating that 3 ActiveMQ nodes are in standby modes or they are slaves.

at No comments:

Thursday, January 22, 2015

Configure ActiveMQ For High Available Using Master-Slave Paradigm With Shared File System

Introduction

Enterprise messaging with ActiveMQ is very popular nowadays. It is very matured platform and it is free! Currently, the 5.10.0 release is available at http://activemq.apache.org/activemq-5100-release.html.
In this blog, I will explain the detailed configuration about how to configure ActiveMQ instances to achieve high availability using master-slave paradigm with shared file system. The system diagram is shown below
As shown in the above diagram, I use two RedHat Linux boxes to configure 4 ActiveMQ broker nodes. All of these nodes share one file system [a directory where all nodes point to]. In general, we do not configure 2 activemq instance on one linux server in the production environment. Two or three ActiveMQ broker nodes to form a Master-Slave clustering is good enough to achieve high availability. You may create multiple clusters. I am going to explain that in later post to use Network Of Brokers with multiple clusters.
Enough topology, now let's create the nodes.

Master-Slave Configurations

We install the apache-activemq-5.10.0 at /opt/app/amq, then create 2 nodes name north and south with the following command:
?
1
2
3
cd /opt/app/amq/apache-activemq-5.10.0/bin
[amq@SANDBOXFUSEV01 bin]$ ./activemq-admin create ../../north
[amq@SANDBOXFUSEV01 bin]$ ./activemq-admin create ../../south
Now we have the following file structures
?
1
2
3
4
5
6
7
8
9
10
11
12
[amq@SANDBOXFUSEV01 amq]$ pwd
/opt/app/amq
 
[amq@SANDBOXFUSEV01 amq]$ ls -lart
total 24
drwxr-xr-x.  3 root root     4096 Jan 14 10:27 ..
lrwxrwxrwx.  1 amq  amq        23 Jan 14 10:29 activemq -> apache-activemq-5.10.0/
drwxr-xr-x. 11 amq  amq      4096 Jan 14 10:42 apache-activemq-5.10.0
drwxrwxr-x.  3 amq  amq      4096 Jan 20 12:38 tomcat
drwxrwxr-x.  4 amq  amq      4096 Jan 24 14:04 north
drwxr-xr-x.  6 amq  amqadmin 4096 Jan 24 14:06 .
drwxrwxr-x.  4 amq  amq      4096 Jan 24 14:06 south

Configure North

Now we need to modify the activemq.xml file with the following changes: ]
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
17     <broker brokername="north" datadirectory="${activemq.data}" usejmx="true" xmlns="http://activemq.apache.org/schema/core">
 
47         <managementcontext>
48             <managementcontext connectorport="11099" createconnector="true" rmiserverport="44444">
49         </managementcontext>
 
58         <persistenceadapter>
59             <kahadb directory="/amqdata/master-slave/data/kahadb">
60         </kahadb></persistenceadapter>
 
88         <transportconnectors>
89             <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
90             <transportconnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
91             <transportconnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
92             <transportconnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
93             <transportconnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
94             <transportconnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
95         </transportconnector></transportconnector></transportconnector></transportconnector></transportconnector></transportconnectors>
           ....
      </managementcontext></broker>
The line with useJmx="true" is for enable JMX and the line
?
1
2
<managementcontext connectorport="11099" createconnector="true" rmiserverport="44444">
</managementcontext>
is to tell ActiveMQ to use port 11099 as JMX connection port. The rmiServerPort must be defined, otherwise, the remote access will not be possible. The reason for this is that most enterprise environment block the unspecified port.

Configure South

The South instance will be similar as the North. We just need to change the port number. For the completeness, here are the changes:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
17     <broker brokername="north" datadirectory="${activemq.data}" usejmx="true" xmlns="http://activemq.apache.org/schema/core">
 
47         <managementcontext>
48             <managementcontext connectorport="12099" createconnector="true" rmiserverport="44445">
49         </managementcontext>
 
58         <persistenceadapter>
59             <kahadb directory="/amqdata/master-slave/data/kahadb">
60         </kahadb></persistenceadapter>
 
88         <transportconnectors>
90             <transportconnector name="openwire" uri="tcp://0.0.0.0:61626?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
91             <transportconnector name="amqp" uri="amqp://0.0.0.0:5682?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
92             <transportconnector name="stomp" uri="stomp://0.0.0.0:61623?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
93             <transportconnector name="mqtt" uri="mqtt://0.0.0.0:1893?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
94             <transportconnector name="ws" uri="ws://0.0.0.0:61624?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
95         </transportconnector></transportconnector></transportconnector></transportconnector></transportconnector></transportconnectors>
           ....
      </managementcontext></broker>

Do the same for the second node. The start the north node on the first server by executing the following command:

Master-Slave In Action

Now, we have done all the configuration changes. We can start all the process and to see how this paradigm of high availability works.
?
1
2
cd /opt/app/amq/north/bin
./north start

I have created two aliases which are very useful:

?
1
2
alias amqjava='ps -eaf | egrep java | egrep -v grep | egrep amq'
alias amqport='netstat -anp | egrep tcp | egrep java | egrep -v grep'

Where amqjava is to check java process owned by amq user, and amqport is to check port information about amq java process. Run amqport command immediately after you start the amq java process. Here the port information you will see:

?
1
2
3
4
5
6
7
8
9
10
11
12
13
tcp        0      0 :::41882                    :::*                        LISTEN      17720/java
tcp        0      0 :::1883                     :::*                        LISTEN      17720/java
tcp        0      0 :::11099                    :::*                        LISTEN      17720/java
tcp        0      0 :::44444                    :::*                        LISTEN      17720/java
tcp        0      0 :::8161                     :::*                        LISTEN      17720/java
tcp        0      0 :::5672                     :::*                        LISTEN      17720/java
tcp        0      0 :::61613                    :::*                        LISTEN      17720/java
tcp        0      0 :::61614                    :::*                        LISTEN      17720/java
tcp        0      0 :::61616                    :::*                        LISTEN      17720/java
tcp        0      0 ::ffff:10.66.13.119:37444   ::ffff:10.66.13.119:44444   ESTABLISHED 17720/java
tcp        0      0 ::ffff:127.0.0.1:36187      ::ffff:127.0.0.1:11099      ESTABLISHED 17720/java
tcp        0      0 ::ffff:10.66.13.119:44444   ::ffff:10.66.13.119:37444   ESTABLISHED 17720/java
tcp        0      0 ::ffff:127.0.0.1:11099      ::ffff:127.0.0.1:36187      ESTABLISHED 17720/java

Another way to check these port is to use lsof command as shwon below:

[amq@SANDBOXFUSEV01 bin]$ lsof -P -iTCP
COMMAND   PID USER   FD   TYPE  DEVICE SIZE/OFF NODE NAME
java    17720  amq    9u  IPv6 1300784      0t0  TCP *:41882 (LISTEN)
java    17720  amq  111u  IPv6 1300785      0t0  TCP *:11099 (LISTEN)
java    17720  amq  113u  IPv6 1300786      0t0  TCP *:44444 (LISTEN)
java    17720  amq  125u  IPv6 1300795      0t0  TCP *:61616 (LISTEN)
java    17720  amq  126u  IPv6 1300796      0t0  TCP *:5672 (LISTEN)
java    17720  amq  127u  IPv6 1300797      0t0  TCP *:61613 (LISTEN)
java    17720  amq  128u  IPv6 1300798      0t0  TCP *:1883 (LISTEN)
java    17720  amq  129u  IPv6 1300800      0t0  TCP *:61614 (LISTEN)
java    17720  amq  136u  IPv6 1300811      0t0  TCP *:8161 (LISTEN)

At this time, the ActiveMQ north instance is running. I can verify this by open brower with URL of sandboxfusev01:8161. This is activemq embedded web console. The default password is admin/admin

Now lets start the second instance in the folder of /opt/app/amq/south

?
1
2
cd /opt/app/amq/south/bin
./south start

I can see the following stdout:

?
1
2
3
4
5
6
7
8
9
10
[amq@SANDBOXFUSEV01 bin]$ ./south start
INFO: Using default configuration
(you can configure options in one of these file: /etc/default/activemq /home/amq/.activemqrc)
 
INFO: Invoke the following command to create a configuration file
/opt/app/amq/apache-activemq-5.10.0/bin/activemq setup [ /etc/default/activemq | /home/amq/.activemqrc ]
 
INFO: Using java '/usr/bin/java'
INFO: Starting - inspect logfiles specified in logging.properties and log4j.properties to get details
INFO: pidfile created : '/opt/app/amq/south/data/activemq-SANDBOXFUSEV01.pid' (pid '17879')

When I chack amqjava, I get the following

[amq@SANDBOXFUSEV01 bin]$ amqjava
amq      17720     1  0 18:52 pts/0    00:00:08 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/north/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/north/tmp -Dactivemq.classpath=/opt/app/amq/north/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/north -Dactivemq.conf=/opt/app/amq/north/conf -Dactivemq.data=/opt/app/amq/north/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start
amq      17879     1 24 19:24 pts/0    00:00:02 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/south/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/south/tmp -Dactivemq.classpath=/opt/app/amq/south/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/south -Dactivemq.conf=/opt/app/amq/south/conf -Dactivemq.data=/opt/app/amq/south/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start
[amq@SANDBOXFUSEV01 bin]$

This indicating there are two activemq java process are running with process ID of 17720 and 17879. Now lets check what port has been opened by these two java processes

[amq@SANDBOXFUSEV01 bin]$ amqport
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 :::41882                    :::*                        LISTEN      17720/java
tcp        0      0 :::1883                     :::*                        LISTEN      17720/java
tcp        0      0 :::11099                    :::*                        LISTEN      17720/java
tcp        0      0 :::44444                    :::*                        LISTEN      17720/java
tcp        0      0 :::44445                    :::*                        LISTEN      17879/java
tcp        0      0 :::8161                     :::*                        LISTEN      17720/java
tcp        0      0 :::12099                    :::*                        LISTEN      17879/java
tcp        0      0 :::5672                     :::*                        LISTEN      17720/java
tcp        0      0 :::36010                    :::*                        LISTEN      17879/java
tcp        0      0 :::61613                    :::*                        LISTEN      17720/java
tcp        0      0 :::61614                    :::*                        LISTEN      17720/java
tcp        0      0 :::61616                    :::*                        LISTEN      17720/java
[amq@SANDBOXFUSEV01 bin]$

Interestingly, I see only 44445 and 12099 ports opened by the second java process. You may expect 61626 and other ports are open. This is because the south instance is in standby mode. Now if I tail the log at /opt/app/amq/south/data/activemq.log, if the following output:

?
1
2
3
4
5
6
7
8
9
10
11
12
[amq@SANDBOXFUSEV01 bin]$ tail -10f ../data/activemq.log
2015-01-24 19:33:21,686 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:33:31,688 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:33:41,689 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:33:51,691 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:01,696 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:11,699 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:21,728 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:31,730 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:41,731 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:34:51,733 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main
2015-01-24 19:35:01,736 | INFO  | Database /amqdata/master-slave/data/kahadb/lock is locked... waiting 10 seconds for the database to be unlocked. Reason: java.io.IOException: File '/amqdata/master-slave/data/kahadb/lock' could not be locked. | org.apache.activemq.store.SharedFileLocker | main

As you can see, every 10 seconds, the south instance is throwing an IOException sayiing the lock at '/amqdata/master-slave/data/kahadb/lock' is not accessable. This is exactly what we want. This means the south instance is a slave node now. For the purpose of showing you extra information, I can use lsof utility to check who is locking the file of '/amqdata/master-slave/data/kahadb/lock' by the following command:

?
1
2
3
4
5
6
7
8
9
[amq@SANDBOXFUSEV01 kahadb]$ cd /amqdata/master-slave/data/kahadb
[amq@SANDBOXFUSEV01 kahadb]$ ls
db-2.log  db.data  db.redo  lock
[amq@SANDBOXFUSEV01 kahadb]$ lsof | egrep lock | egrep java
java      17720       amq  118uw     REG               0,21        0 4321950309 /amqdata/master-slave/data/kahadb/lock
[amq@SANDBOXFUSEV01 kahadb]$ amqjava
amq      17720     1  0 18:52 pts/0    00:00:09 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/north/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/north/tmp -Dactivemq.classpath=/opt/app/amq/north/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/north -Dactivemq.conf=/opt/app/amq/north/conf -Dactivemq.data=/opt/app/amq/north/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start
amq      17879     1  0 19:24 pts/0    00:00:03 /usr/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Djava.security.auth.login.config=/opt/app/amq/south/conf/login.config -Dcom.sun.management.jmxremote -Djava.awt.headless=true -Djava.io.tmpdir=/opt/app/amq/south/tmp -Dactivemq.classpath=/opt/app/amq/south/conf; -Dactivemq.home=/opt/app/amq/apache-activemq-5.10.0 -Dactivemq.base=/opt/app/amq/south -Dactivemq.conf=/opt/app/amq/south/conf -Dactivemq.data=/opt/app/amq/south/data -jar /opt/app/amq/apache-activemq-5.10.0/bin/activemq.jar start
[amq@SANDBOXFUSEV01 kahadb]$

From above command and result, I can see that the java process with ID of 17720 is holding the lock at /amqdata/master-slave/data/kahadb/lock.

In the same maner, we can start the two ActiveMA instance in the second server. We have proved the only one instance of activemq node in the cluster is holding the lock and function

Summary

So far we have covered the basic configuration of ActiveMQ for high availability using the Master-Slave paradigm with shared file system. In this paradigm, only master will grab the lock all the ActiveMA instances in the same cluster will be in a standby mode. We call them slaves. I also explained how to setup JMX and how to check ports which are ready for connections.

Before we further explain how Master-Slave paradigm works, I would like demonstrate how to use JVisualVM and Tomcat container to monitor ActiveMA instances (nodes). I will cover these two topics in the next two blogs.

at 1 comment:

Tuesday, January 13, 2015

ActiveMQ 5.10.0 Installation And Configuration

Introduction

In my previous ActiveMQ introduction, I have explained the installation process, how to check what ports are used by ActiveMQ, and how to enable JMX. In this post, I am going to further explain the ports used by ActiveMQ. However the main purpose of this post to explain how to run the examples coming with ActiveMQ. This technique is important for the future discuss on how to set up high availability features with ActiveMQ, so that we can verify the configuration.

In the coming few posts, I will explain the enterprise messaging configuration for high availability with:

  1. Master-Slave HA with Share File System
  2. Use Tomcat Web Container to Host ActiveMQ Web Console
  3. Bridging the Network Broker
  4. Network Brokers
  5. Best Practices

In the future posts, I will explain java programming techniques for writing producer and consumers for java developers, as well as advanced programing for enterprise integration using Apache Camel and ActiveMQ. Thus, we will study these examples in detail.

HTT

Now first lets discuss a bit the transport connectors come with the default configuration in activemq.xml

Default Ports Used By ActiveMQ

Take a look at activemq.xml, you will find this xml segment:

?
1
2
3
4
5
6
7
8
<transportconnectors>
    <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
    <transportconnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
    <transportconnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
    <transportconnector name="stomp" uri="stomp://0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
    <transportconnector name="mqtt" uri="mqtt://0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
    <transportconnector name="ws" uri="ws://0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600">
</transportconnector></transportconnector></transportconnector></transportconnector></transportconnector></transportconnectors>
All these protocols can be found at activemq.apache.org website. ActiveMQ support many other transport connectors, such as VM, SSL, NIO, NIO+SSL, HTTP, HTTPS, etc. The details can be found at http://activemq.apache.org/configuring-version-5-transports.html

Run ActiveMQ Examples

ActiveMQ comes with many examples. For easy testing, we can run the examples at examples/openwire/swissarmy. Under the dir, there is a readme file. Typically, we will use the following lines for the producer and consumers:

?
1
2
ant consumer -Durl=tcp://localhost:61616 -Dtopic=true -Ddurable=true -Dmax=5000
ant producer -Durl=tcp://localhost:61616 -Dtopic=true -DtimeToLive=30000 -Dmax=5000

Make sure ant is installed and the ANT_HOME and PATH are set properly. In the following post about ActiveMQ High Available Configuration with Master-Slave, I will cover more about how to run these examples.

at No comments:
Subscribe to: Posts (Atom)

Anypoint Studio Error: The project is missing Munit lIbrary to run tests

Anypoint Studio 7.9 has a bug. Even if we following the article: https://help.mulesoft.com/s/article/The-project-is-missing-MUnit-libraries-...