Saturday, February 22, 2014

A Linux Cron Job

Introduction

In my recent consulting, I was asked to help to complete a task schedule in RedHat Linux environment. Here is the requirement:
  1. The task should run every 5 minutes
  2. If a task takes longer than 5 minutes, no new task should run while a live task is running
  3. The task should not run between 5:30 am and 7:30 am

Solution

Apparently, this is a candidate for using Linux cron job based on the first statement. The requirement requires us to do a bit more than just cron job. I know that the second and third requirement can be done with the shell script. Thus, I decided to write a simple bash script. Of course, this can be done by using Perl as well. For now, I will provide the solution here.

The Cron Definition

The cron job definition is very simple as the following:

*/5     *       *       *       *       /opt/Apexus_DI/bin/ExecuteApexus_DI.sh

Linux/Unix cron job syntax can be in the following form. There are plenty references you can find in the internet. Here is the brief summary.

# * * * * *  command to execute
# ┬ ┬ ┬ ┬ ┬
# │ │ │ │ │
# │ │ │ │ │
# │ │ │ │ └───── day of week (0 - 7) (0 to 6 are Sunday to Saturday, or use names; 7 is Sunday, the same as 0)
# │ │ │ └────────── month (1 - 12)
# │ │ └─────────────── day of month (1 - 31)
# │ └──────────────────── hour (0 - 23)
# └───────────────────────── min (0 - 59)
Field name Mandatory? Allowed values Allowed special characters Remarks
Minutes Yes 0-59 * / , - -
Hours Yes 0-23 * / , - -
Day of month Yes 1-31 * / , - ? L W -
Month Yes 1-12 or JAN-DEC * / , - -
Day of week Yes 0-6 or SUN-SAT * / , - ? L # -
Year No 1970–2099 * / , - This field is not supported in standard/default implementations.

List Of The script

The complete list of the script is shown below. Line 5 - 9 is to check if a job is running. If yes, then just exit. Line 12 and 14 are to check if the time is between 5:30 am and 7:00 am.

Cron Job Log

The cron job logs are located at /var/log/cron. You need to have admin permission to view the log.

Thursday, February 13, 2014

Introduce Mule ESB

Introduction

The video for this blog is available at:
http://www.youtube.com/watch?v=wVCmik-2xAM&feature=youtu.be

Mule and JBoss Fuse ESBs are most popular open source ESBs on the market. They are the leaders. I have been working with both for numerous years. Both of them have their strengths and weakness. I discuss these further in my later blog. Today, I am going to describe Mule 101 from development point of view. I am going to cover the following aspects:

  • Install Mule Studio
  • Install Mule MMC and Runtime
  • Develop Hello World Mule Application
  • Build Mule Project With Maven
  • Mule Application Deployment

Install Mule Studio

You can download both Mule Studio and MMC from http://www.mulesoft.com/platform/soa/mule-esb-enterprise. You will need to fill a form so that Mule can track you down, and hopefully you will buy their product. I downloaded the following two file to ~/Downloads on my MacBook:

   mmc-distribution-mule-console-bundle-3.4.2.zip
   MuleStudio-for-macosx-64bit-3.5.0-201312091746.zip

As you can see the current mule run time is version 3.4.2 and Mule Studio is 3.5. Execute the following commands [My current dir is ~/Downloads]:

jar vxf mmc-distribution-mule-console-bundle-3.4.2.zip
jar vxf MuleStudio-for-macosx-64bit-3.5.0-201312091746.zip
mv MuleStudio ~/.
mv mule-enterprise-standalone-3.4.2 ~/.

Now, I have both Mule Studio and runtime under my home directory. Let's develop a Hello World Application.

Develop Hello World Mule Application

Start MuleStudio.app [Make sure it is executable]. Enable the maven setting by editing the Preferences as shown below:

To create a mule project, New -> Mule Project. Make sure create maven project as shown below:

As shown below, the MuleStudio will create a project with the following structures.

Now let's develop a simple application using HTTP component. Perform the flowing actions:

  • Double click the mule flow file under flow directory
  • Drag and drop the HTTP component
  • Drag and drop the Logger component
  • Double click the Logger icon in side the flow
  • Fill the groovy script as shown below




    
        
        
    


Now, we can run the mule application inside the studio by right click the the file hello-world.mflow file, run as, Muel application. You should see the following output in the studio console:

NFO  2014-02-15 14:33:05,294 [main] org.mule.module.management.agent.JmxAgent: Registered Connector Service with name Mule.hello-world:type=Connector,name="connector.http.mule.default.1"
INFO  2014-02-15 14:33:05,297 [main] org.mule.DefaultMuleContext: 
**********************************************************************
* Application: hello-world                                           *
* OS encoding: US-ASCII, Mule encoding: UTF-8                        *
*                                                                    *
* Agents Running:                                                    *
*   Clustering Agent                                                 *
*   JMX Agent                                                        *
**********************************************************************
INFO  2014-02-15 14:33:05,297 [main] org.mule.module.launcher.MuleDeploymentService: 
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+ Started app 'hello-world'                                +
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Open a web browser, and enter the following URL
http://localhost:8081/echo?UserName=Gary Liu
You should see the following on your browser:
/echo?UserName=Gary%20Liu

At this point, we have created a simple mule application using HTTP and Logger components coming with mule. This process is good for development and unit testing. However, we normal build the application using maven and later perform continuous integration using other utility like Jenkins. Now, let me explain the build process using maven.

Build Mule Application With Maven

In order to build maven project created through MuleStudio, we need to uncomment all the following dependencies:

 

  



  

One more thing before we can build using maven. Execute the folloing script under MuleStudio installation dir [~/MuleStudio]. This is to install licm jar file to our local maven repo.

mvn install:install-file \
 -Dfile=plugins/org.mule.tooling.server.3.4.1.ee_3.4.1.201312062152/mule/plugins/mule-plugin-debugger-3.4.1/lib/licm-1.1.3.jar \
 -DgroupId=com.mulesoft.licm \
 -DartifactId=licm \
 -Dversion=1.1.3 \
 -Dpackaging=jar

Now, we can build the Hello World mule application we just developed. Let's do the following:

cd ~/MuleStudio/workspace/hello-world
mvn clean install

The maven build process takes now time. If you check the target directory, there is a file named as: hello-world-1.0.0-SNAPSHOT.zip. This is the file we will deploy to the mule runtime. If you are curious enough and wander what are the contents inside that zip file, you can do the following:

[/Users/Gary2013/MuleStudio/workspace/hello-world]$ cd target
[/Users/Gary2013/MuleStudio/workspace/hello-world/target]$ jar vtf hello-world-1.0.0-SNAPSHOT.zip 
   989 Sat Feb 15 14:33:02 CST 2014 hello-world.xml
     0 Sat Feb 15 14:14:20 CST 2014 mule-app.properties
   119 Sat Feb 15 14:33:02 CST 2014 mule-deploy.properties
     0 Sat Feb 15 15:01:22 CST 2014 classes/
   989 Sat Feb 15 15:01:22 CST 2014 classes/hello-world.xml
     0 Sat Feb 15 15:01:22 CST 2014 classes/mule-app.properties
   119 Sat Feb 15 15:01:22 CST 2014 classes/mule-deploy.properties
[/Users/Gary2013/MuleStudio/workspace/hello-world/target]$ 

This is the beauty about mule. Its application is very clean and minimum comparing with osgi bundles. Mule will be able to figure out dynamically what jar files should be linked during runtime. Once the application zip file is build, we can deploy it to the mule runtime.

Mule Application Deployment

There is a README.txt file under mule runtime installation [in my case, it is ~/mule-enterprise-standalone-3.4.2]. In that will, the start, stop and restart procedures are described. Let's start the mule as daemon by run the following command:
[/Users/Gary2013/mule-enterprise-standalone-3.4.2]$ bin/mule start
MULE_HOME is set to /Users/Gary2013/mule-enterprise-standalone-3.4.2
Starting Mule Enterprise Edition...
[/Users/Gary2013/mule-enterprise-standalone-3.4.2]$

Now, we can check what java process is mule runtime. Here is what I see:

[/Users/Gary2013/mule-enterprise-standalone-3.4.2]$ ps -eaf | egrep java | egrep -v egrep 
  501 17266 17265   0  3:19PM ??         0:32.28 /Library/Java/JavaVirtualMachines/jdk1.7.0_40.jdk/Contents/Home/bin/java -Dmule.home=/Users/Gary2013/mule-enterprise-standalone-3.4.2 -Dmule.base=/Users/Gary2013/mule-enterprise-standalone-3.4.2 -Djava.net.preferIPv4Stack=TRUE -XX:MaxPermSize=256m -Djava.endorsed.dirs=/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/endorsed -Xmx1024m -Djava.library.path=%LD_LIBRARY_PATH%:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot -classpath %MULE_LIB%:/Users/Gary2013/mule-enterprise-standalone-3.4.2/conf:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/commons-cli-1.2.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/commons-codec-1.3-osgi.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/jul-to-slf4j-1.6.1.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/licm-1.1.4.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/log4j-1.2.16.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/mule-module-boot-ee-3.4.2.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/mule-module-logging-3.4.2.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/oscore-2.2.4.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/propertyset-1.3.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/truelicense-1.29.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/truexml-1.29.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/lib/boot/wrapper-3.5.19.jar:/Users/Gary2013/mule-enterprise-standalone-3.4.2/data-mapper -Dwrapper.key=d4wYZG88GHn1riyEuxVDypA4M4y9i5w4 -Dwrapper.port=32000 -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=17265 -Dwrapper.version=3.5.19-st -Dwrapper.native_library=wrapper -Dwrapper.arch=universal -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 -Dwrapper.lang.domain=wrapper -Dwrapper.lang.folder=../lang org.mule.module.reboot.MuleContainerBootstrap start0
[/Users/Gary2013/mule-enterprise-standalone-3.4.2]$ 

The deployment procedure is covered pretty well http://www.mulesoft.org/documentation/display/current/Application+Deployment. Simple, we can copy the zip file from target to the apps directory.

cp ~/MuleStudio/workspace/hello-world/target/hello-world-1.0.0-SNAPSHOT.zip  .

Mule has a feature called hot-deployment. Once the zip file is dropped to the apps dir, mule will unzip the archive, delete it, and create a file named like: hello-world-1.0.0-SNAPSHOT-anchor.txt. Inside that file, there is a line of text. The meaning of text is not important. Mule is just using it as key to monitor the application. To undeploy the application, you can simply delete the anchor file.

Monday, February 10, 2014

JBoss Fuse 6.0 Installation And Configuration

Trouble Shooting

Why does the URL http://localhost;8181 not work?

This could be many reasons. One of the most frequent case is the web console not install properly. To correct the issue run the following command on the command line console:

profile-edit --features fabric-webui fabric

JBoss Fuse has a good reference page at: https://access.redhat.com/site/documentation/en-US/JBoss_Fuse/6.0/html/Using_the_Management_Console/files/FMCUGAccess.html

Introduction

JBoss Fuse 6.0 and RedHat A-MQ 6.0 were introduced at March 2013. Now both JBoss Fuse and RedHat A-MQ 6.1 is on the horizon. I am little bit disappointed by the slow pace of the release, as Apache Camel 2.12.1 is already released, but JBoss Fuse 6.0 is still based on 2.10.0. Regardless this sluggishness of the update, I still believe, JBoss Fuse is the best ESB framework available in the market comparing with Mule ESB and Talend ESB.

This blog is an simple introduction of JBoss Fuse. I will focus on basic installation and basic karaf commands on Linux/MacOS. Later on, I will introduce examples on how to develop routes for the enterprise integration purposes.

Installation

The JBoss Fuse installation is very straight forward. First we need to download the binary distribution from http://www.jboss.org/products/fuse. You can choose download link based on your os. For me, I am installing on macbook pro [5.9]. I also downloaded the FuseIDE 6.0. Today, I will only introduction the runtime environment.

Once you download the zip for, you can use the following command to unzip the download file [I download the zip file to $HOME/Downloads]:

jar -vxf jboss-fuse-full-6.0.0.redhat-024.zip
mv jboss-fuse-full-6.0.0.redhat-024 ~/.
cd ~/jboss-fuse-full-6.0.0.redhat-024

The above commands unzip the downloaded zip file and put the installation dir under $HOME/jboss-fuse-full-6.0.0.redhat-024. Let's name this dir as JBOSS_FUSE_HOME. Basically, all the required installation is done. In order to start JBoss Fuse as a service, we need to create a wrapper.

Create Serice Wrapper

To create service wrapper, we can execute the following commands:

cd $JBOSS_FUSE_HOME/bin
chmod 755 *
./fuse

The above command will bring up the karaf cmmand line console. On the console, issue the following commands:

features:install wrapper
wrapper:install -n JBossFuseSerivce -d JBossFuseService -d JBoss-Fuse

Now if you list the bin dir, two new files are created:

JBossFuseSerivce-service 
JBossFuseSerivce-wrapper

You can run the service by executing the following command:

./JBossFuseSerivce-service start|stop|restart  

Where start, stop, and restart means to start, stop, and restart the JBoss Fuse service.

Create Fabric

We would like to use management console to monitor the JBoss Fuse container. In order to do this, we need to create fabric through karaf command line console by issue the following commands:

JBossFuse:karaf@root> fabric:create

Here is the output. Enter gary as user name and gary as password

No user found in etc/users.properties or specified as an option. Please specify one ...
New user name: gary
Password for gary: 
Verify password for gary:
Using specified zookeeper password:gary
JBossFuse:karaf@root> 
profile-edit --features fabric-webui fabric

The last command will ask user name and password. I just use gary/gary as user name and password, respectively.

Now you can login to the management console using the URL: localhost:8181. Note that 8181 is the default port. The default port is defined at etc/jetty.xml file. You can modify the file to change the port you wish to use

Sunday, February 9, 2014

Install and Configure ActiveMQ 5.9.0

Introduction

ActiveMQ 5.9 was release on October 23, 2013, per this web page. Many new features have been introduced since, such as LevelDB, hawtio, etc. I am going to explain these features in detail in the this and following blogs. In this blog, I am going to demonstrate how to install and configure activemq instance on MacOS [the same is for linux].

Installation

First, download the binary distribution from activemq 5.9.0 release. Then, execute the following commands to install the default configuration:

cd ~/Downloads
tar -vxzf tar -vxzf apache-activemq-5.9.0-bin.tar.gz
mv apache-activemq-5.9.0 ~/.
cd ~/apache-activemq-5.9.0

The above commands created activemq default at apache-activemq-5.9.0. Now we are ready to start activemq by execute the following commands

cd bin
./activemq start

The start command starts the activemq process as a background running process. We can check the java process by the following command

ps -eaf | egrep java | egrep -v egrep

You should see the following output:

[/Users/Gary2013/apache-activemq-5.9.0/bin]$ ps -eaf | egrep java | egrep -v egrep 
  501  9148     1   0  4:17PM ttys001    0:15.53 /Library/Java/JavaVirtualMachines/jdk1.7.0_40.jdk/Contents/Home/bin/java -Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties -Dhawtio.realm=activemq -Dhawtio.role=admins -Dhawtio.rolePrincipalClasses=org.apache.activemq.jaas.GroupPrincipal -Djava.security.auth.login.config=/Users/Gary2013/apache-activemq-5.9.0/conf/login.config -Dcom.sun.management.jmxremote.port=11099 -Dcom.sun.management.jmxremote.password.file=/Users/Gary2013/apache-activemq-5.9.0/conf/jmx.password -Dcom.sun.management.jmxremote.access.file=/Users/Gary2013/apache-activemq-5.9.0/conf/jmx.access -Dcom.sun.management.jmxremote.ssl=false -Djava.awt.headless=true -Djava.io.tmpdir=/Users/Gary2013/apache-activemq-5.9.0/tmp -Dactivemq.classpath=/Users/Gary2013/apache-activemq-5.9.0/conf; -Dactivemq.home=/Users/Gary2013/apache-activemq-5.9.0 -Dactivemq.base=/Users/Gary2013/apache-activemq-5.9.0 -Dactivemq.conf=/Users/Gary2013/apache-activemq-5.9.0/conf -Dactivemq.data=/Users/Gary2013/apache-activemq-5.9.0/data -jar /Users/Gary2013/apache-activemq-5.9.0/bin/activemq.jar start
[/Users/Gary2013/apache-activemq-5.9.0/bin]$ 

The above output from checking the java process is very important to understand what arguement have been passed to the activemq java process, such as, JDK version, default installation, etc. At this point, we can use activemq for messaging. I am going to explain this in later session together with apache camel application development. ActiveMQ comes with a web console which is very important from admin point of view. To login to console, we can using the following url:

http://localhost:8161/

Click the the link of Manage ActiveMQ broker. From this link, we will see virtually all the information about the ActiveMQ instance. If you are familiar with the old version, you can choose the link to point to the old version console. Somehow, I like the old console better. Of course, the new console provides much more information such as jmx. However, for that part, I prefer to use jvisualvm to get more information about the activemq process. In order to use jvisualvm to do so, we need to enable jmx configuration for activemq.

Enable JMX

JMX Configuration Setup

To enable JMX, we need to modify the following two files:

$ACTIVEMQ_HOME/bin/activemq
$ACTIVEMQ_HOME/conf/activemq.xml

activemq is a script which has option of start, restart, stop.
activemq.xml is the configuration file for the ActiveMQ broker.

Modify activemq

Change the activemq file for the following lines:

184  ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=11099 "
185  ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.password.file=${ACTIVEMQ_CONF}/jmx.password"
186  ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.access.file=${ACTIVEMQ_CONF}/jmx.access"
187  ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote.ssl=false"
188 #ACTIVEMQ_SUNJMX_START="$ACTIVEMQ_SUNJMX_START -Dcom.sun.management.jmxremote"

Make the the jmx.password file has the permission of 600. By default, jmx.password has the line as the following:

     admin activemq

Modify the file activemq.xml

40     

The new line has added useJmx="true"

Now we need to restart activemq by executing the following command:

$ACTIVEMQ_HOME/bin/activemq restart

OK, we have configured activemq with jmx enabled. What port is JMX running? From the configuration file, we know it should be running on 11099 based on the line of:

ACTIVEMQ_SUNJMX_START="-Dcom.sun.management.jmxremote.port=11099 "

Verifying the JMX

To verify if the JMX is working or not, we can use JVisualVM which is coming with JDK. On the command line terminal, you can type "jvisualvm", the GUI will come up. Here is the configuration I have:

When you configure the JMX, you will need to add the following properties:<

   connection url: http:localost:11099
   user name: admin
   password:  activemq

Important Ports Used By ActiveMQ

Let check what important ports activemq is running. Execute the following command:

netstat -an | egrep LISTEN

On my local MacBook Pro, I see the following ports are open:

[/Users/Gary2013/apache-activemq-5.9.0/conf]$ netstat -an | egrep LISTEN
tcp46      0      0  *.8161                 *.*                    LISTEN     
tcp46      0      0  *.61614                *.*                    LISTEN     
tcp46      0      0  *.1883                 *.*                    LISTEN     
tcp46      0      0  *.61613                *.*                    LISTEN     
tcp46      0      0  *.5672                 *.*                    LISTEN     
tcp46      0      0  *.61616                *.*                    LISTEN     
tcp46      0      0  *.51776                *.*                    LISTEN     
tcp46      0      0  *.11099                *.*                    LISTEN     
tcp46      0      0  *.51775                *.*                    LISTEN     
tcp4       0      0  127.0.0.1.37483        *.*                    LISTEN     
tcp4       0      0  *.49204                *.*                    LISTEN     
tcp6       0      0  fdfd:b5d6:df5f:c.4488  *.*                    LISTEN     
tcp46      0      0  *.3306                 *.*                    LISTEN     
tcp4       0      0  127.0.0.1.631          *.*                    LISTEN     
tcp6       0      0  ::1.631                *.*                    LISTEN     
[/Users/Gary2013/apache-activemq-5.9.0/conf]$ 

We can see 11099 port is open. That is JMX port. The other important ports are: 61613, 61613, 61616 and 8161. So, where are these ports are defined. An experience developer, use egrep command to file this out. Of course, you can look up the documentation, but that is a slow process. And many times it is not very reliable, because the most documentation are not up-to-date.

egrep -r "8161|61613|61616" . | egrep xml

We now know that 8161 is defined in the file jetty.xml and other ports are defined at activemq.xml. There are other 2 ways to check what ports are running. One is through the url: http://localhost:8161/hawtio/#/logs. The other is view the load at $ACTIVEMQ_HOME/data/activemq.log

2014-02-09 16:17:20 INFOorg.apache.activemq.store.kahadb.plist.PListStoreImpl PListStore:[/Users/Gary2013/apache-activemq-5.9.0/data/localhost/tmp_storage] started
2014-02-09 16:17:20 INFOorg.apache.activemq.broker.BrokerService Using Persistence Adapter: KahaDBPersistenceAdapter[/Users/Gary2013/apache-activemq-5.9.0/data/kahadb]
2014-02-09 16:17:21 INFOorg.apache.activemq.store.kahadb.MessageDatabase KahaDB is version 5
2014-02-09 16:17:21 INFOorg.apache.activemq.store.kahadb.MessageDatabase Recovering from the journal ...
2014-02-09 16:17:21 INFOorg.apache.activemq.store.kahadb.MessageDatabase Recovery replayed 15831 operations from the journal in 0.33 seconds.
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.BrokerService Apache ActiveMQ 5.9.0 (localhost, ID:guojiangs-mbp.home-51782-1391984241450-0:1) is starting
2014-02-09 16:17:21 INFOorg.apache.activemq.transport.TransportServerThreadSupport Listening for connections at: tcp://guojiangs-mbp.home:61616?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.TransportConnector Connector openwire started
2014-02-09 16:17:21 INFOorg.apache.activemq.transport.TransportServerThreadSupport Listening for connections at: amqp://guojiangs-mbp.home:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.TransportConnector Connector amqp started
2014-02-09 16:17:21 INFOorg.apache.activemq.transport.TransportServerThreadSupport Listening for connections at: stomp://guojiangs-mbp.home:61613?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.TransportConnector Connector stomp started
2014-02-09 16:17:21 INFOorg.apache.activemq.transport.TransportServerThreadSupport Listening for connections at: mqtt://guojiangs-mbp.home:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.TransportConnector Connector mqtt started
2014-02-09 16:17:21 INFOorg.apache.activemq.transport.ws.WSTransportServer Listening for connections at ws://guojiangs-mbp.home:61614?maximumConnections=1000&wireFormat.maxFrameSize=104857600
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.TransportConnector Connector ws started
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.BrokerService Apache ActiveMQ 5.9.0 (localhost, ID:guojiangs-mbp.home-51782-1391984241450-0:1) started
2014-02-09 16:17:21 INFOorg.apache.activemq.broker.BrokerService For help or more information please see: http://activemq.apache.org
2014-02-09 16:17:22 INFOio.hawt.jmx.JmxTreeWatcher Welcome to hawtio 1.2-M23 : http://hawt.io/ : Don't cha wish your console was hawt like me? ;-)
2014-02-09 16:17:22 INFOio.hawt.web.AuthenticationFilter Starting hawtio authentication filter, JAAS realm: "activemq" authorized role: "admins" role principal classes: "org.apache.activemq.jaas.GroupPrincipal"
2014-02-09 16:17:22 INFOio.hawt.web.UploadServlet Using file upload directory: /Users/Gary2013/apache-activemq-5.9.0/tmp/uploads
2014-02-09 16:17:22 INFO/hawtio jolokia-agent: Using access restrictor classpath:/jolokia-access.xml
2014-02-09 16:17:22 INFOorg.apache.activemq.web.WebConsoleStarter ActiveMQ WebConsole available at http://localhost:8161/

Thursday, February 6, 2014

OSGi Bundle Using Blueprint Dymisfied

Introduction

In my previous blog, I have described how to create OSGi bundle using org.apache.camel.archetypes. We can create OSGi project from scratch in that way and it works fine. In this blog I am going to describe the process to create OSGi using blueprint from scratch. In addition, I am going to demonstrate how to load property file dynamically. And how to use these properties in camel routes and spring beans.

Create OSGi Blueprint From Scratch

To create a maven OSGi maven project, I use the following script:


$ cat createCamelBlueprint.sh
mvn archetype:generate \
 -DarchetypeGroupId=org.apache.camel.archetypes \
 -DarchetypeArtifactId=camel-archetype-blueprint \
 -DarchetypeVersion=2.10.4 \
 -DgroupId=com.ggl.esb.osgi \
 -DartifactId=osgi.blueprint.demo \
 -Dversion=0.0.1-SNAPSHOT

The about script can be run on linux or mac os. If you want to use windows, you can replace the \ with ^ for line continuation and name it as createCamelBlueprint.bat. I run my script under /cygdrive/c/Camel-Dev-Demo. After executing the script, I see the following standard output:

$ ./createCamelBlueprint.sh
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building Maven Stub Project (No POM) 1
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom >>>
[INFO]
[INFO] <<< maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom <<<
[INFO]
[INFO] --- maven-archetype-plugin:2.2:generate (default-cli) @ standalone-pom ---
[INFO] Generating project in Interactive mode
[INFO] Archetype repository missing. Using the one from [org.apache.camel.archetypes:camel-archetype-blueprint:2.12.2] found in catalog remote
[INFO] Using property: groupId = com.ggl.esb.osgi
[INFO] Using property: artifactId = osgi.blueprint.demo
[INFO] Using property: version = 0.0.1-SNAPSHOT
[INFO] Using property: package = com.ggl.esb.osgi
[INFO] Using property: camel-version = 2.10.4
[INFO] Using property: log4j-version = 1.2.17
[INFO] Using property: maven-bundle-plugin-version = 2.3.7
[INFO] Using property: maven-compiler-plugin-version = 2.5.1
[INFO] Using property: maven-resources-plugin-version = 2.4.3
[INFO] Using property: slf4j-version = 1.6.6
Confirm properties configuration:
groupId: com.ggl.esb.osgi
artifactId: osgi.blueprint.demo
version: 0.0.1-SNAPSHOT
package: com.ggl.esb.osgi
camel-version: 2.10.4
log4j-version: 1.2.17
maven-bundle-plugin-version: 2.3.7
maven-compiler-plugin-version: 2.5.1
maven-resources-plugin-version: 2.4.3
slf4j-version: 1.6.6
 Y: :
[INFO] ----------------------------------------------------------------------------
[INFO] Using following parameters for creating project from Archetype: camel-archetype-blueprint:2.10.4
[INFO] ----------------------------------------------------------------------------
[INFO] Parameter: groupId, Value: com.ggl.esb.osgi
[INFO] Parameter: artifactId, Value: osgi.blueprint.demo
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: package, Value: com.ggl.esb.osgi
[INFO] Parameter: packageInPathFormat, Value: com/ggl/esb/osgi
[INFO] Parameter: maven-bundle-plugin-version, Value: 2.3.7
[INFO] Parameter: maven-resources-plugin-version, Value: 2.4.3
[INFO] Parameter: groupId, Value: com.ggl.esb.osgi
[INFO] Parameter: maven-compiler-plugin-version, Value: 2.5.1
[INFO] Parameter: slf4j-version, Value: 1.6.6
[INFO] Parameter: version, Value: 0.0.1-SNAPSHOT
[INFO] Parameter: log4j-version, Value: 1.2.17
[INFO] Parameter: camel-version, Value: 2.10.4
[INFO] Parameter: package, Value: com.ggl.esb.osgi
[INFO] Parameter: artifactId, Value: osgi.blueprint.demo
[INFO] project created from Archetype in dir: C:\Camel-Dev-Demo\osgi.blueprint.demo
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:30.173s
[INFO] Finished at: Thu Feb 06 11:05:17 CST 2014
[INFO] Final Memory: 14M/243M
[INFO] ------------------------------------------------------------------------


Note: it asks slf4j version. You can type Y or hit enter. From the command line output, you can see what version of different artifacts are used. If you list the contents under the current directory, you should see the following:

gliu@BGUNDAPA-1 /cygdrive/c/Camel-Dev-Demo
$ ls
createCamelBlueprint.sh  osgi.blueprint.demo

You can see that a new directory named: osgi.blueprint.demo has been created. At this point, a default OSGi blueprint project has been created with a default route. You can execute the following commands:

    cd osgi.blueprint.demo
    mvn clean install
    mvn camel:run

You should see the camel route print out messages every 5 second. Here is the standard output you should see:

[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building A Camel Blueprint Route 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> camel-maven-plugin:2.10.4:run (default-cli) @ osgi.blueprint.demo >>>
[INFO]
[INFO] --- maven-resources-plugin:2.4.3:resources (default-resources) @ osgi.blueprint.demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ osgi.blueprint.demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] --- maven-resources-plugin:2.4.3:testResources (default-testResources) @ osgi.blueprint.demo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 0 resource
[INFO]
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ osgi.blueprint.demo ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] <<< camel-maven-plugin:2.10.4:run (default-cli) @ osgi.blueprint.demo <<<
[INFO]
[INFO] --- camel-maven-plugin:2.10.4:run (default-cli) @ osgi.blueprint.demo ---
[INFO] Using org.apache.camel.test.blueprint.Main to initiate a CamelContext
[mel.test.blueprint.Main.main()] MainSupport                    INFO  Apache Camel 2.10.4 starting
[mel.test.blueprint.Main.main()] Activator                      INFO  Camel activator starting
[mel.test.blueprint.Main.main()] Activator                      INFO  Camel activator started
[mel.test.blueprint.Main.main()] BlueprintExtender              INFO  No quiesce support is available, so blueprint components will not participate in quiesce operations
[         Blueprint Extender: 1] BlueprintContainerImpl         INFO  Bundle osgi.blueprint.demo is waiting for namespace handlers
[         Blueprint Extender: 1] ManagementStrategyFactory      INFO  JMX enabled.
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Apache Camel 2.10.4 (CamelContext: blueprintContext) is starting
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Route: timerToLog started and consuming from: Endpoint[timer://foo?period=5000]
[         Blueprint Extender: 1] ultManagementLifecycleStrategy INFO  StatisticsLevel at All so enabling load performance statistics
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Total 1 routes, of which 1 is started.
[         Blueprint Extender: 1] BlueprintCamelContext          INFO  Apache Camel 2.10.4 (CamelContext: blueprintContext) started in 0.419 seconds
[ntext) thread #0 - timer://foo] timerToLog                     INFO  The message contains Hi from Camel at 2014-02-06 11:08:40
[ntext) thread #0 - timer://foo] timerToLog                     INFO  The message contains Hi from Camel at 2014-02-06 11:08:45

As you can see the last 2 lines are the output from the default camel route we just created. Therefore, all the ground work has been done by maven archetypes. This is really handy! At this point, if you deploy the bundle to karaf container it will work. At this point, all the ground for development OSGi bundle with blueprint is done. we can start to write our own routes, processors, components etc.

Before we do that, let's take a look what interesting source code has been generated. Let's import the newly generated project to eclipse IDE. I am using STS [Spring Tool Suite] 3.4.0. The other current version of Eclipse should work fine.

After the project is imported successfully, you should see the following directory structures:

We can inspect the blueprint.xml file. It has the following contents:




  
      
  

  
    
      
      
          
      
      
      
    
  



Load Property Files Dynamically

Now I am going to demonstrate how to update the blueprint.xml file to load a property file and how use the property in camel routes and spring beans. In real world applications, we want to have variables in our route and spring beans. And in different environments, such as, DEV, TEST, PROD, the configurations are different. The apache karaf container has a very unique feature to monitor the property files in etc directory. Thus, we want to put the property files under that directory. And when we change it, the new configurations should be loaded dynamically.

In order to demonstrate this feature, let's add a name member and change the timer uri to quartz. The new blueprint.xml are list in the following:




 
  
 

 

 
  
   
  
 


 
 
  
            
   
    
   
   
   
  

 



Before I explain the details about the new blueprint.xml. Let's deploy the bundle into karaf container. Currently, I am using Talend's ESB container which is apache karaf container. To deploy the bundle, build it first.

$ mvn clean install -Dmaven.test.skip=true

Note: we use -Dmaven.test.skip=true to skip the unit test.

Once the bundle is built successfully. We can deploy the bundle to the karaf container. Because we are using quartz component and an environment variable, we need to execute the following command on the karaf console.

    system-property -p runtime.environment DEV
    features:install camel-quartz
    osgi:install -s mvn:com.ggl.esb.osgi/osgi.blueprint.demo/0.0.1-SNAPSHOT

The first command above is to set an environment variable named: runtime.environment. The second is to install Apache camel quartz component. The last command is to deploy the bundle. Now if you tail the karaf output log, you will see error. This is because the value of cron.value used for quartz uri is not default. Consequently, the route cannot start. Now let add the file named BlueprintDemoDEV.cfg to etc dir. The content of the file is listed as below:

activemq.broker.url=failover:(tcp://uit-amq-1.corp.vha.com:61616,tcp://uit-amq-2.corp.vha.com:61616)
name.value=Gary Liu
cron.value=0/5+*+*+*+*+?

Once the file is created, the route will start and you will see message on the standard output look like:

16:03:40,001 | INFO  | Context_Worker-4 | rg.apache.camel.util.CamelLogger  176 | 174 - org.apache.camel.camel-core - 2.10.4 | The message contains Gary Liu at 2014-02-06 16:03:40 failover:(tcp://uit-amq-1.corp.vha.com:61616,tcp://uit-amq-2.corp.vha.com:61616)

Now if you change cron.value by replacing the 5 with 20. The print out from standard output will occur every 20 seconds. This means karaf container load the property file dynamically, like the hot deployment of other containers, like tomcat.

Hot Development of Properties Explained

We has demonstrated in the above session that:
  • Variables can be placed in side came routes definition and spring bean definition
  • Properties can be hot-deployed
  • Environment variable can be attached to a file name

Monday, February 3, 2014

Create Apache Camel OSGi Bundle Maven Project From Scratch

There are plenty maven projects which create Apache Camel OSGi bundles. But there is very few place I can find to simply explain how to create such a kind project from scratch. This blog will exmain the details about this process. Frankly speaking, it may seems very simple, but it took me several hours to get it work. Hopefully, people who are interested in this process do not have to go through the same processes as I did.

Create Maven Project with Apach Camel Routes

Here is the script I created for creating maven project with apache camel route [I use cygwin in my windows environment. If you run dos cmd, you can replace the \ with ^ for line continuation.]:


mvn archetype:generate \
  -DarchetypeGroupId=org.apache.camel.archetypes \
  -DarchetypeArtifactId=camel-archetype-spring  \
  -DarchetypeVersion=2.10.4 \
  -DgroupId=com.vha.esb.apexus \
  -DartifactId=ce-reject-notify \
  -Dversion=1.0.0-SNAPSHOT

You can put this into a file, namely, generateProject.sh for linux OS, or generateProject.bat. If you run under linux os or cygwind, make sure change the file permission:
    chmod 755 generatePoject.sh

Now you can run the script, it will generate all files with a defaul apache camel route, which simple copy a file from src/data to target/message/uk or target/message/others. Aparatly, apache camel community did an excellent job to make this process go smoothly. After the build process is completed, run the following commands:

    mvn clean install
    mvn camel:run
If you now check the target/messages/uk and target/messages/others, you will find an xml file existing in each directory.

Import Project to Eclipse

The projects created by the running the script is ready to be imported to eclipse IDE.

Create OSGi Bundle

To create OSGi bundle, we need to instruct maven to build bundle using the org.apache.felix maven plugin. Here is my pom.xml file




 4.0.0

 com.vha.esb.apexus
 ce-reject-notify
 bundle
 1.0.0-SNAPSHOT

 A Camel Spring Route
 http://www.myorganization.org

 
  UTF-8
  UTF-8
 

 
  
   org.apache.camel
   camel-core
   2.10.4
  
  
   org.apache.camel
   camel-spring
   2.10.4
  

  
  
   org.slf4j
   slf4j-api
   1.6.6
  
  
   org.slf4j
   slf4j-log4j12
   1.6.6
  
  
   log4j
   log4j
   1.2.17
  

  
  
   org.apache.camel
   camel-test-spring
   2.10.4
   test
  

 

 
  install

  
   
    org.apache.felix
    maven-bundle-plugin
    true
    
     target/META-INF
     
      ${project.groupId}.${project.artifactId}
      
      
       org.apache.cxf,
       org.apache.cxf.binding,
       org.apache.cxf.binding.corba,
       org.apache.cxf.binding.soap,
       org.apache.cxf.binding.soap.spring,
       org.apache.cxf.bus,
       org.apache.cxf.bus.resource,
       org.apache.cxf.bus.spring,
       org.apache.cxf.buslifecycle,
       org.apache.cxf.catalog,
       org.apache.cxf.configuration,
       org.apache.cxf.configuration.spring,
       org.apache.cxf.endpoint,
       org.apache.cxf.headers,
       org.apache.cxf.management,
       org.apache.cxf.management.jmx,
       org.apache.cxf.phase,
       org.apache.cxf.resource,
       org.apache.cxf.service.factory,
       org.apache.cxf.transport,
       org.apache.cxf.transport.http,
       org.apache.cxf.transport.http.policy,
       org.apache.cxf.transport.http_jetty,
       org.apache.cxf.transport.jms,
       org.apache.cxf.transports.http,
       org.apache.cxf.workqueue,
       org.apache.cxf.wsdl,
       org.apache.cxf.wsdl11,
       org.apache.ws.security.action,
       org.apache.ws.security.processor,
       org.springframework.beans.factory.config,
       org.springframework.mail.javamail,
       *
      
     
    
   






   
    org.apache.maven.plugins
    maven-compiler-plugin
    2.5.1
    
     1.6
     1.6
    
   
   
    org.apache.maven.plugins
    maven-resources-plugin
    2.4.3
    
     UTF-8
    
   

   
   
    org.apache.camel
    camel-maven-plugin
    2.10.4
   
  
 




Note: 2 places are changed: one packaging tag. By default, it is jar. Now it is bundle:
    bundle
The other place is the plugin:
   
    org.apache.felix
    maven-bundle-plugin
    true
    
     target/META-INF
     
      ${project.groupId}.${project.artifactId}
      
      
       org.apache.cxf,
       org.apache.cxf.binding,
       org.apache.cxf.binding.corba,
       org.apache.cxf.binding.soap,
       org.apache.cxf.binding.soap.spring,
       org.apache.cxf.bus,
       org.apache.cxf.bus.resource,
       org.apache.cxf.bus.spring,
       org.apache.cxf.buslifecycle,
       org.apache.cxf.catalog,
       org.apache.cxf.configuration,
       org.apache.cxf.configuration.spring,
       org.apache.cxf.endpoint,
       org.apache.cxf.headers,
       org.apache.cxf.management,
       org.apache.cxf.management.jmx,
       org.apache.cxf.phase,
       org.apache.cxf.resource,
       org.apache.cxf.service.factory,
       org.apache.cxf.transport,
       org.apache.cxf.transport.http,
       org.apache.cxf.transport.http.policy,
       org.apache.cxf.transport.http_jetty,
       org.apache.cxf.transport.jms,
       org.apache.cxf.transports.http,
       org.apache.cxf.workqueue,
       org.apache.cxf.wsdl,
       org.apache.cxf.wsdl11,
       org.apache.ws.security.action,
       org.apache.ws.security.processor,
       org.springframework.beans.factory.config,
       org.springframework.mail.javamail,
       *
      
     
    
   

Now we are readly to deploy the bundle to karaf container. The command is:

    osgi:install mvn:com.vha.esb.apexus/ce-reject-notify/1.0.0-SNAPSHOT
    osgi:start 244

Now, if your karaf container is at ${karaf_home}, you should see the directory ${karaf_home}/src/data. Now, let's copy the 2 xml files coming with the build to that directory. Shortly, you should see message1.xml and message2.xml are placed at ${karaf_home}/target/messages/uk and ${karaf_home}/target/messages/other directory, respectively.

Understanding Apache Camel Route

The archtetypes: camel-archetype-spring create a default camel route which copy the xml file from one place to anotehr. I add another route using quartz end point as shown blow:






 
  
 


 

 

 



 
  
   
  
            
   
    
   
   
   
  
   
   
  
   
   
    
     /person/city = 'London'
     
     
    
    
     
     
    
   
  
  
  
 



The update route shows few things. First, it is a spring configuration file. We can define beans inside the configuraion. Secondly, we can define multiple routes within a camelContext.

The camel configuration file shown above is OK, but not very practical because we hard-coded thte value. Now, i want to explain how to use property files. To use property file, we need to update the came-context.xml file. Here is the list of the file content.






       ......

 
  
 



 
  
  
  
   
  
            
   
    
   
   
   
  
   
         ......
  
  
 



As you can see, we use spring PropertyPlaceholderConfigurer bean to load file. and in the camel route you use {{my.value}}, where my.value is a variable. I deployed the file osgi.spring.demo.cfg under ${KARAF.HOME}/etc. The ${KARAF.HOME} is the karaf container installation. The file osgi.spring.demo.cfg has the following contents:

activemq.broker.url=failover:(tcp://uit-amq-1.corp.vha.com:61616,tcp://uit-amq-2.corp.vha.com:61616)
name.value=Gary Liu
cron.value=0/5+*+*+*+*+?

After my new deployment of the bundle, in the karaf log, I can see the following out put every 5 seconds, because in the quartz end point, I have defined the field for second 0/5, which is every 5 seconds.

14:48:50,003 | INFO  | amel-21_Worker-2 | rg.apache.camel.util.CamelLogger  176 | 174 - org.apache.camel.camel-core - 2.10.4 | The message contains Gary Liu at 2014-02-12 14:48:50 failover:(tcp://uit-amq-1.corp.vha.com:61616,tcp://uit-amq-2.corp.vha.com:61616)
14:48:55,001 | INFO  | amel-21_Worker-3 | rg.apache.camel.util.CamelLogger  176 | 174 - org.apache.camel.camel-core - 2.10.4 | The message contains Gary Liu at 2014-02-12 14:48:55 failover:(tcp://uit-amq-1.corp.vha.com:61616,tcp://uit-amq-2.corp.vha.com:61616)

Conclusion

Basically, to create Apache Camel route OSGi bundles from scratch, you can simply use the org.apache.camel.archetypes archetype to generate a project. Then you use the apache felix maven plugin to generate OSGi bundle.

Sunday, February 2, 2014

Understand OSGI Bundle Build And Deployment

OSGi stands for Open Service Gateway initiative. For daily development of OSGI bundles, we need to understand how it build and deployed. I have a simple camel osgi project which comes with camel 2.10.4 in examples/apache/camel/camel-example-osgi. In this short blogger, I want to explain what happened when you build and deploy the osgi bundle.

Here are the simple procedures I build and deploy the bundle:

    cd ${apache-camle-home}/examples/apache/camel/camel-example-osgi
    mvn clean install


Where the build Requests?

You should see the following lines at the end of the build.

[INFO] --- maven-bundle-plugin:2.3.7:install (default-install) @ camel-example-osgi ---
[INFO] Installing org/apache/camel/camel-example-osgi/2.10.4/camel-example-osgi-2.10.4.jar
[INFO] Writing OBR metadata
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1:11.876s
[INFO] Finished at: Sun Feb 02 16:04:17 CST 2014
[INFO] Final Memory: 26M/435M
[INFO] ------------------------------------------------------------------------


This means the build is successful. But what has maven done? if you cd to you maven repository, which normally is located at $HOME/.m2/reposity. In my case, it is under:
    C:\Users\gliu\.m2\repository\org\apache\camel\camel-example-osgi\2.10.4
You can see that:
     org.apache.camel    --> groupID
     camel-example-osgi  --> artifact ID
     2.10.4              --> version number
Under that directory you should see the following files:
   camel-example-osgi-2.10.4.pom
   camel-example-osgi-2.10.4-features.xml
   camel-example-osgi-2.10.4.jar
   _remote.repositories

How to deploy the artifact to OSGi container

The deployment is straight forward. In the karaf container command line, you can type the following:
      osgi:install mvn:org.apache.camel/camel-example-osgi/2.10.4
      start 438
Where 438 is bundle id. Here is a windows snapshot:

Anypoint Studio Error: The project is missing Munit lIbrary to run tests

Anypoint Studio 7.9 has a bug. Even if we following the article: https://help.mulesoft.com/s/article/The-project-is-missing-MUnit-libraries-...