Clustering JBoss FSW Switchyard Using TCP Stack

Introduction

I have explained how to install JBoss Fuse Service Work and how to deploy a switchyard project in my previous blogs:

• http://ggl-consulting.blogspot.com/2014/10/jboss-service-works-explained.html
• http://ggl-consulting.blogspot.com/2014/10/build-deploy-test-and-debug-jboss-fuse.html

In this blog, I am going to explain the procedures to setup switchyard clustering in standalone mode.

Server Information

I have two Linux Servers with following information:

• OS: Red Hat Enterprise Linux Server release 6.6 (Santiago)
• JBOSS_HOME: /opt/app/fsw/jboss-eap-6.1
• java version: "1.7.0_71"

Firewall Changes

Make sure the following ports are open

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

$ sudo iptables -L -v -n | egrep NEW     2   148 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:22  3489  295K ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW udp dpt:161     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:8181     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:8080     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:631     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:25     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:4447     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:900     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5445     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:9990     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:37063     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:3528     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:9999     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:5455     0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:111     2   120 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW tcp dpt:7600     0     0 ACCEPT     udp  --  *      *       0.0.0.0/0            0.0.0.0/0           state NEW udp dpt:55200

the port 7600 and 55200 are used for tcp and udp stack repectively.if additional ports need to be open, there are two way to open new ports. The first option is to use the system-config-firewall GUI. The second one is to modify the file of /etc/sysconfig/iptables.

Modify HA Configuration File

Modify the file of ${JBOSS_HOME}/standalone/configuraiton/standalone-full-ha.xml as the following:

?

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39

<subsystem default-stack="tcp" xmlns="urn:jboss:domain:jgroups:1.1">     <stack name="udp">         <transport socket-binding="jgroups-udp" type="UDP">         <protocol type="PING">         <protocol type="MERGE3">         <protocol socket-binding="jgroups-udp-fd" type="FD_SOCK">         <protocol type="FD">         <protocol type="VERIFY_SUSPECT">         <protocol type="pbcast.NAKACK">         <protocol type="UNICAST2">         <protocol type="pbcast.STABLE">         <protocol type="pbcast.GMS">         <protocol type="UFC">         <protocol type="MFC">         <protocol type="FRAG2">         <protocol type="RSVP">     </protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></transport></stack>     <stack name="tcp">         <transport socket-binding="jgroups-tcp" type="TCP">         <protocol type="TCPPING">             <property name="initial_hosts">devrhfusev01.tmghealth.com[7600],devrhfusev02.tmghealth.com[7600]</property>             <property name="num_initial_members">2</property>             <property name="port_range">0</property>             <property name="timeout">2000</property>         </protocol>         <protocol type="MERGE2">         <protocol socket-binding="jgroups-tcp-fd" type="FD_SOCK">         <protocol type="FD">         <protocol type="VERIFY_SUSPECT">         <protocol type="pbcast.NAKACK">         <protocol type="UNICAST2">         <protocol type="pbcast.STABLE">         <protocol type="pbcast.GMS">         <protocol type="UFC">         <protocol type="MFC">         <protocol type="FRAG2">         <protocol type="RSVP">     </protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></protocol></transport></stack> </subsystem>

The out-of-box transport stack is UDP. We have to change the default-stack="udp" to default-stack="tcp". Also we add a new section:

?

1 2 3 4 5 6

<protocol type="TCPPING">      <property name="initial_hosts">devrhfusev01.tmghealth.com[7600],devrhfusev02.tmghealth.com[7600]</property>      <property name="num_initial_members">2</property>      <property name="port_range">0</property>      <property name="timeout">2000</property>  </protocol>

With that the configuration part is done. Make sure modify all the file of standalone-full-ha.xml on each host which will be participating the cluster.

Server Startup Option

Start Server One

?

1	$ bin/standalone.sh -b 10.66.13.110 -bmanagement 10.66.13.110 -c standalone-full-ha.xml

where 10.66.13.110 is et0 interface I want to bind

Start Server Two

?

1	$ bin/standalone.sh -b 10.66.13.111 -bmanagement 10.66.13.111 -c standalone-full-ha.xml

After two servers are started, you can check the ${JBOSS_HOME}/standalone/log/server.log, and you should see the something like:

?

1	10:12:55,663 INFO  [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (ServerService Thread Pool -- 66) ISPN000094: Received new cluster view: [devrhfusev02/switchyard|1] [devrhfusev02/switchyard, devrhfusev01/switchyard]

Noted that devrhfusev01 and devrhfusev02 are the node name. By default it is the host name. If you want to define the node name, you can pass the command option with -Djboss.node.name=MY-FAVORATE-NODE-NAME

Check What Ports Are Open

It is always a good practice to check how the two servers are connected. To do this, we can use the following command:

?

1 2 3 4

[fuse@DEVRHFUSEV01 log]$ lsof -iTCP -P | egrep EST | egrep -v grep java    16053 fuse  504u  IPv4 24568988      0t0  TCP devrhfusev01.tmghealth.com:7600->devrhfusev02.tmghealth.com:53845 (ESTABLISHED) java    16053 fuse  505u  IPv4 24568989      0t0  TCP devrhfusev01.tmghealth.com:38526->devrhfusev02.tmghealth.com:57600 (ESTABLISHED) java    16053 fuse  506u  IPv4 24568990      0t0  TCP devrhfusev01.tmghealth.com:57600->devrhfusev02.tmghealth.com:44482 (ESTABLISHED)

You may wonder why the devrhfusev02 using 53845 to connect devrhfusev01:7600. This is because the server on devrhfusev01 is started first. The second server can choose any port to connect the remote server. Basically, the server started latest will initiate connection.

Error Handling and Trouble Shooting

If you don't see the establish connections between the two clustering nodes, you may check the following:

• Are the ports open from the firewall? [use iptables -L -v -n to check what ports are open]
• Is the binding interface correct? [us ifconfig -a command and look eth0 interface ip address]