Monday, August 31, 2015

Mule Application Development Using MySQL and ActiveMQ Connectors

Introduction

In this blog, I am going to explain how to use Mule ESB to perform integration with database and messaging brokers. I use MySQL as database engine and ActiveMQ as message broker.
The youtube video now available at https://www.youtube.com/watch?v=DAsFv045nlw&feature=youtu.be



Here is the use case:
  1. A user sends HTTP GET request to retrieve customer information based on company name
  2. A Mule flow will retrieve all the records from database table named Customer
  3. Send each record to JMS Queue
  4. Return use the counter of total records
  5. handle exceptions

Infrastructure Setup

In order achieve this, we need to setup MySQL database. Also we need to setup ActiveMQ.

Setup MySQL

Here is the procedures to setup MySQL on MacBook Pro:
  • Down load MySQL dmg
  • Install it
  • Start the mysqld by run: sudo $MYSQL_INSTALLATION/mysql/support-files/mysql.server start
After start MySQL server, you need to do the following:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$mysql -u root
 
mysql> create user 'mule'@'localhost' identified by 'mule';
 
mysql> create database dataformule;
 
$mysql -u mule -p dataformule
 
mysql> grant all privileges on * . * to 'mule'@'localhost';
 
mysql> show databases;
 
mysql> select user from mysql.user;
 
mysql -u mule -p dataformule
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
use dataformule;
show tables;
 
create table Customer (
     id INT NOT NULL AUTO_INCREMENT,
     company_name char(30) NOT NULL,
     contact_name char (30) NOT NULL,
     phone char(30),
     company_address char(30),
     PRIMARY KEY (id)
);
 
mysql> desc Customer;
+-----------------+----------+------+-----+---------+----------------+
| Field           | Type     | Null | Key | Default | Extra          |
+-----------------+----------+------+-----+---------+----------------+
| id              | int(11)  | NO   | PRI | NULL    | auto_increment |
| company_name    | char(30) | NO   |     | NULL    |                |
| contact_name    | char(30) | NO   |     | NULL    |                |
| phone           | char(30) | YES  |     | NULL    |                |
| company_address | char(30) | YES  |     | NULL    |                |
+-----------------+----------+------+-----+---------+----------------+
5 rows in set (0.01 sec)
 
 
 
select * from dataformule.Customer;
 
INSERT INTO dataformule.Customer (id, company_name, contact_name, phone, company_address) VALUES (102, 'Company_B', 'Contact_B', '999-888-7778', 'Address_B');
INSERT INTO dataformule.Customer (id, company_name, contact_name, phone, company_address) VALUES (103, 'Company_C', 'Contact_C', '999-888-7779', 'Address_C');
INSERT INTO dataformule.Customer (id, company_name, contact_name, phone, company_address) VALUES (104, 'Company_D', 'Contact_D', '999-888-7781', 'Address_D');
INSERT INTO dataformule.Customer (id, company_name, contact_name, phone, company_address) VALUES (105, 'Company_E', 'Contact_E', '999-888-7782', 'Address_E');
 
 
mysql> select * from customer;
+-----+--------------+--------------+--------------+-----------------+
| id  | company_name | contact_name | phone        | company_address |
+-----+--------------+--------------+--------------+-----------------+
| 102 | Company_B    | Contact_B    | 999-888-7778 | Address_B       |
| 103 | Company_C    | Contact_C    | 999-888-7779 | Address_C       |
| 104 | Company_D    | Contact_D    | 999-888-7781 | Address_D       |
| 105 | Company_E    | Contact_E    | 999-888-7782 | Address_E       |
+-----+--------------+--------------+--------------+-----------------+
4 rows in set (0.01 sec)
For setup ActiveMQ with SSL, you may search my previous blog. I have documented in details about the set up

The Code

The complete code is available at GitHub at:
https://github.com/garyliu1119/Mule-Development/tree/master/ActiveMQ-Messaging
The mule flow code list as the following:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
<mule version="EE-3.7.0" xmlns:db="http://www.mulesoft.org/schema/mule/db" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:jms="http://www.mulesoft.org/schema/mule/jms" xmlns:json="http://www.mulesoft.org/schema/mule/json" xmlns:scripting="http://www.mulesoft.org/schema/mule/scripting" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.mulesoft.org/schema/mule/core" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/jms http://www.mulesoft.org/schema/mule/jms/current/mule-jms.xsd
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd
http://www.mulesoft.org/schema/mule/db http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
http://www.mulesoft.org/schema/mule/scripting http://www.mulesoft.org/schema/mule/scripting/current/mule-scripting.xsd">
    <http:listener-config doc:name="HTTP Listener Configuration" host="0.0.0.0" name="HTTP_Listener_Configuration" port="8081">
    <jms:activemq-connector brokerurl="tcp://localhost:61616" doc:name="Active MQ" name="Active_MQ" validateconnections="true">
        <reconnect count="10" frequency="15000">
    </reconnect></jms:activemq-connector>
    <configuration defaultexceptionstrategy-ref="globlal_Exception_Strategy" doc:name="Configuration">
    <db:mysql-config database="dataformule" doc:name="MySQL Configuration" host="localhost" name="MySQL_Configuration" password="mule" port="3306" user="mule">
    <db:template-query doc:name="Template Query" name="Template_Query_By_Company_Name">
        <db:parameterized-query><![CDATA[SELECT * FROM Customer WHERE company_name = :companyName;]]></db:parameterized-query>
        <db:in-param defaultvalue="#[message.inboundProperties.'http.query.params'.companyName];" name="companyName">
    </db:in-param></db:template-query>
    <db:template-query doc:name="Template Query" name="Template_Query_By_Phone_Number">
        <db:parameterized-query><![CDATA[SELECT * FROM CUSTOMER WHERE phone = :phoneNumber;]]></db:parameterized-query>
        <db:in-param defaultvalue="#[message.inboundProperties.'http.query.params'.phoneNumber]" name="phoneNumber" type="CHAR">
    </db:in-param></db:template-query>
    <flow name="Mule-Use-Case-A-MainFlow">
        <http:listener allowedmethods="GET,POST,DELETE, UPDATE" config-ref="HTTP_Listener_Configuration" doc:name="HTTP" path="/json-jms">
        <choice doc:name="Choice">
            <when expression="#[message.inboundProperties.'http.query.params'.companyName != null]">
                <flow-ref doc:name="Data Retrieval by Company" name="activemq-messaging-datd-retrieval-by-company-Sub_Flow">
            </flow-ref></when>
            <when expression="#[message.inboundProperties.'http.query.params'.phoneNumber != null]">
                <flow-ref doc:name="Data Retrieval By Phone Number" name="activemq-messaging-datd-retrieval-by-phone-Sub_Flow">
            </flow-ref></when>
            <otherwise>
                <scripting:component doc:name="Groovy">
                    <scripting:script engine="Groovy"><![CDATA[throw new IllegalArgumentException('Paramenter Not Acceptable')]]></scripting:script>
                </scripting:component>
            </otherwise>
        </choice>
        <set-variable doc:name="Set Message Counter to 0" value="0" variablename="messageCounter">
        <flow-ref doc:name="activemq-messagiing-send-Sub_Flow" name="activemq-messagiing-send-Sub_Flow">
        <set-payload doc:name="Set Payload" value="{"count": #[flowVars.messageCounter]}">
        <set-property doc:name="Property" propertyname="Content-Type" value="application/json">
        <choice-exception-strategy doc:name="Choice Exception Strategy">
            <catch-exception-strategy doc:name="Catch Exception Strategy" when="#[exception.causedBy(org.mule.api.transformer.TransformerException)]">
                <set-payload doc:name="Set Payload" value="#[payload]">
                <jms:outbound-endpoint connector-ref="Active_MQ" doc:name="JMS" queue="Dead.Letter.Invalid.Data">
            </jms:outbound-endpoint></set-payload></catch-exception-strategy>
            <catch-exception-strategy doc:name="Catch Database Connection Exception" when="#[exception.causedBy(java.sql.SQLException)]">
                <set-payload doc:name="Set Payload" value="#[payload]">
                <jms:outbound-endpoint connector-ref="Active_MQ" doc:name="JMS" queue="Dead.Letter.Invalid.Data">
            </jms:outbound-endpoint></set-payload></catch-exception-strategy>
            <catch-exception-strategy doc:name="Catch ActiveMQ Connection Exception" when="#[exception.causedBy(java.lang.Exception)]">
                <set-payload doc:name="Set Payload" value="The request cannot be processed, the error is #[exception.getSummaryMessage()] ">
                <logger doc:name="Logger" level="INFO" message="Unexpected Exception: #[payload]">
            </logger></set-payload></catch-exception-strategy>
        </choice-exception-strategy>
    </set-property></set-payload></flow-ref></set-variable></http:listener></flow>
    <sub-flow name="activemq-messagiing-send-Sub_Flow">
        <foreach collection="#[payload]" doc:name="Loop List Of Messages">
            <logger doc:name="Log Single message" level="INFO" message="#[payload]">
            <json:object-to-json-transformer doc:name="Object to JSON">
            <set-variable doc:name="Set Count Variable" value="#[flowVars.counter]" variablename="messageCounter">
            <jms:outbound-endpoint connector-ref="Active_MQ" doc:name="Send To JMS" queue="Customer.Information">
        </jms:outbound-endpoint></set-variable></json:object-to-json-transformer></logger></foreach>
    </sub-flow>
    <sub-flow name="activemq-messaging-datd-retrieval-by-company-Sub_Flow">
        <db:select config-ref="MySQL_Configuration" doc:name="Database">
            <db:template-query-ref name="Template_Query_By_Company_Name">
            <db:in-param name="companyName" value="#[message.inboundProperties.'http.query.params'.companyName]">
        </db:in-param></db:template-query-ref></db:select>
        <json:object-to-json-transformer doc:name="Object to JSON">
        <json:json-to-object-transformer doc:name="JSON to Object" returnclass="java.util.List">
    </json:json-to-object-transformer></json:object-to-json-transformer></sub-flow>
    <sub-flow name="activemq-messaging-datd-retrieval-by-phone-Sub_Flow">
        <db:select config-ref="MySQL_Configuration" doc:name="Database">
            <db:template-query-ref name="Template_Query_By_Phone_Number">
            <db:in-param name="phoneNumber" type="CHAR" value="#[message.inboundProperties.'http.query.params'.phoneNumber]">
        </db:in-param></db:template-query-ref></db:select>
        <json:object-to-json-transformer doc:name="Object to JSON">
        <json:json-to-object-transformer doc:name="JSON to Object" returnclass="java.util.List">
    </json:json-to-object-transformer></json:object-to-json-transformer></sub-flow>
    <catch-exception-strategy name="globlal_Exception_Strategy">
        <set-payload doc:name="Set Payload" value="#[payload]">
        <logger doc:name="Logger" level="INFO" message="GLOBAL Exception Handller: #[payload]">
    </logger></set-payload></catch-exception-strategy>
</db:mysql-config></configuration></http:listener-config></mule>

Detailed Explanation

You can view the video for detailed explanation about this application at http://youtu.be/DAsFv045nlw 

How To Stop mysqld

Recently, I find I could not stop the mysqld process on my MacBook Pro. After some research, trial and errors, I found the following command works:

?
1
sudo launchctl unload -w /Library/LaunchDaemons/com.oracle.oss.mysql.mysqld.plist

at 6 comments:

Tuesday, August 11, 2015

Mule ESB Tutorial: Introduction To DataWeave

Introduction

MuleSoft's DataWeave available only from Anypoint Studio 3.7 since May 2015. It is a new powerful tool for data transformation. Earlier, we use DataMapper, or customer transformer to perform data transformation in addition to the out-of-box transformers, like Byte Array to String, object to JSON, XML to JSON, etc., just to name a few.

With DataWeave, the transformation become much less painful. In particular, if you don't want to write customer transform in java or scripts. Still, if you are good at writing java code, like me, customer transformer still be the best from development point of view. But it may not be the best for the long term maintenance. Once you get to the habits to write write customer transformer, you tend to use it a lot. I think it is time to change with this kind of powerful tool.

In this tutorial, I am going to demonstrate the basics about the DataWeave. Later, I will introduce more complicated cases. I will put this tutorial to Youtube.com. If you prefer to watch the usage in action.

Using Using Data Weave

The flow is like this:

With this configuration, we will use the input of JSON like the follwoing:

?
1
2
3
4
5
6
7
8
{
 "abc":
     {
     "cba":"ddd",
     "ccc":"eee",
     "aaa":["fgh","ghf","hgf"]
     }
}

Put the above file into your local somewhere name it as transform1.json. I strongly recommend a tool named: subline test 2 for edition JSON.

Now create HTTP connector. Everything is the same as simple configuration. Only extra procedure is add Metadata as the following:

Now, we can add Transform Message to the flow. Note that the "Transform Message" which is DataWeave component, it is not a transformer!

Simple Transform Doing Nothing

For the doing nothing case, the transformer will be the following:
?
1
2
3
4
5
%dw 1.0
%input payload application/json
%output application/xml
---
payload

Here is the snapshot of the transform panel:

Now we can test using PostMan with url: http://lcoalhost:8082/transform1. You should see the xml output like the following:

?
1
2
3
4
5
6
7
8
9
10
<!--xml version='1.0' encoding='UTF-8'?-->
<abc>
    <cba>ddd</cba>
    <ccc>eee</ccc>
    <aaa>
        <element>fgh</element>
        <element>ghf</element>
        <element>hgf</element>
    </aaa>
</abc>

All above is nothing fancy. Note, the payload in the transform panel. It means doing nothing. The power comes from that panel. All the logics will be input there. We will see in the next section.

Transform From JSON To XML

Now let's change the transform scripts to the following:

?
1
2
3
4
5
6
7
8
9
%dw 1.0
%input payload application/json
%output application/xml
---
items: {
 name: payload.abc.aaa[1],
 address: payload.abc.cba,
 ID: payload.abc.ccc
}

Now if you run the POST operation, you will get the following result:

?
1
2
3
4
5
6
<!--xml version='1.0' encoding='UTF-8'?-->
<items>
    <name>ghf</name>
    <address>ddd</address>
    <id>eee</id>
</items>

This is the transformation a bit interesting. Of course, you can do this with data mapper transformer. But I hope you see the power behind it. It opens up the huge possibility in manipulate data for the transformation purpose. I will cover more cases in my later blogs.

Conclusion

I think DataWeave is a powerful tool. It takes time to learn the syntaxes, but I think it worths the time spending if you want to be a good integration architect! You can visit my youtube channel to get dynamics of the DataWeave!

at 12 comments:

Monday, August 10, 2015

Mule Flow: Setup HTTPS Connector and Setup Content-type

Introduction

The most popular connector probably is HTTP connector. However, for the enterprise application, this is not good enough. We need to setup HTTPS connect. Since Anypoint Studio, previously called Mule Studio, the configuration of HTTPS has been change. In this blog, I am going document the procedures on how to setup the HTTPS connector for the on-premise application, together with few other technical aspects. Here is the flow I have setup:

In this flow, I have few interesting parts. One is about content-based-routing, and the other is how to setup response content-type. In my case, I setup it as applicaiton/json.

Generate Self-signed Keystore

For the self-signed certificates, we can use keytool coming with JDK. First created a directory name something like certs. In my case, it is located at:
?
1
/Users/Gary2013/AnypointStudio-3.7/certs
Now inside the certs directory, execute the following command:
?
1
keytool -genkey -alias mule-server -keyalg RSA -keystore keystore.jks
That is all you need to do!

Setup HTTPS Connector

The following 2 pictures show where to put the certificates. In order to setup HTTPS connector, we need to configure 2 pages from the Anypoint Studio as shown below:

Setup The Mule Flow

The flow setup is not that complicated. Here I just provide the configuration file. For the details, I will provide a video, you can my youtube video.
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<!--xml version="1.0" encoding="UTF-8"?-->
 
<mule xmlns:tracking="http://www.mulesoft.org/schema/mule/ee/tracking" xmlns:json="http://www.mulesoft.org/schema/mule/json" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns:tls="http://www.mulesoft.org/schema/mule/tls" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.7.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/tls http://www.mulesoft.org/schema/mule/tls/current/mule-tls.xsd
http://www.mulesoft.org/schema/mule/json http://www.mulesoft.org/schema/mule/json/current/mule-json.xsd
http://www.mulesoft.org/schema/mule/ee/tracking http://www.mulesoft.org/schema/mule/ee/tracking/current/mule-tracking-ee.xsd">
 <http:listener-config name="HTTPS_Listener_Configuration" protocol="HTTPS" host="0.0.0.0" port="${https.port.onpremise}" doc:name="HTTP Listener Configuration">
  <tls:context>
   <tls:trust-store>
   <tls:key-store type="jks" path="/Users/Gary2013/AnypointStudio-3.7/certs/keystore.jks" alias="mule-server" keypassword="changeit" password="changeit">
  </tls:key-store></tls:trust-store></tls:context>
 </http:listener-config>
 <http:listener-config name="HTTP_Listener_Configuration" host="0.0.0.0" port="${http.port.onpremise}" doc:name="HTTP Listener Configuration">
 <flow name="lineagelogisticsinventoryFlow">
  <http:listener config-ref="HTTPS_Listener_Configuration" path="/inventory" doc:name="HTTP" allowedmethods="GET, POST">
  <logger message="#[payload]" level="INFO" doc:name="Logger">
        <set-variable variablename="method" value="#[message.inboundProperties.'http.method']" doc:name="Variable">
        <choice doc:name="Choice">
            <when expression="#[flowVars['method'] == 'POST']">
                <set-payload value="#[payload]" doc:name="Set Payload">
            </set-payload></when>
            <otherwise>
                <set-variable variablename="name" value="#[message.inboundProperties.'http.query.params'.name]" doc:name="Variable">
                <set-payload value="{"name": "#[flowVars.name]"}" doc:name="Set Payload">
            </set-payload></set-variable></otherwise>
        </choice>
        <response>
            <set-property propertyname="Content-Type" value="application/json" doc:name="Property">
        </set-property></response>
 </set-variable></logger></http:listener></flow>
</http:listener-config></mule>
From the above xml file, you may see that I have configured a HTTP and HTTPS connectors. The HTTPS configuration can be view in the following xml form:
?
1
2
3
4
5
6
<http:listener-config name="HTTPS_Listener_Configuration" protocol="HTTPS" host="0.0.0.0" port="${https.port.onpremise}" doc:name="HTTP Listener Configuration">
 <tls:context>
  <tls:trust-store>
  <tls:key-store type="jks" path="/Users/Gary2013/AnypointStudio-3.7/certs/keystore.jks" alias="mule-server" keypassword="changeit" password="changeit">
 </tls:key-store></tls:trust-store></tls:context>
</http:listener-config>
You may note that the variable"${https.port.onpremise}". This together with other parameters are defined in the file name mule-app.properties.

Test The Flow

I use PostMan to test the flow for GET and POST methods with following URL:

To test POST method, put the following JSON into the body of the PostMan:
?
1
2
3
4
5
6
{
  "name": "Barack Obama",
  "phone": "888-555-8888",
  "address": "1600 Pennsylvania Avenue Northwest, Washington, DC 20500",
  "Title" : "US President"
}

You should get the same json back as the following:

?
1
2
3
4
5
6
{
  "name": "Barack Obama",
  "phone": "888-555-8888",
  "address": "1600 Pennsylvania Avenue Northwest, Washington, DC 20500",
  "Title": "US President"
}
To test GET method, you use the following URL:
?
1
https://localhost:8443/inventory?name=Gary Liu Jr.
You should see the following JSON output:
?
1
2
3
{
  "name": "Gary Liu Jr."
}

Conclusion

This blog explains the basic procedure to configure HTTPS connector together with few other interesting aspects of mule flow, such as, content-base-routing, response content-type setup etc.
at 1 comment:

Sunday, August 9, 2015

Change Mule Application Logging Level

Introduction

MuleSoft has changed many stuff for the last year. One of the questions developer will ask is how to change the logging level. One might think that he can just create log4j.properties file and set the logging there. Unfortunately, this will not work. We have to create a file name log4j2.xml.

Solution

Put the following file (named as: log4j2.xml) into src/main/resources:
?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
<!--xml version="1.0" encoding="UTF-8"?-->
<configuration>
  <appenders>
    <console name="STDOUT" target="SYSTEM_OUT">
      <patternlayout pattern="%-5p %d [%t] %c: %m%n">
    </patternlayout></console>
  </appenders>
   
  <!-- You can set custom log levels per-package here -->
  <loggers>
     
    <!-- CXF is used heavily by Mule for web services -->
    <logger name="org.apache.cxf" level="warn">
     
    <!-- Apache Commons tend to make a lot of noise which can clutter the log. -->
    <logger name="org.apache" level="debug">
     
    <!-- Reduce startup noise -->
    <logger name="org.springframework.beans.factory" level="warn">
     
    <!-- Mule classes -->
    <logger name="org.mule" level="debug">
    <logger name="com.mulesoft" level="info">
     
    <!-- Reduce DM verbosity -->
    <logger name="org.jetel" level="warn">
    <logger name="Tracking" level="warn">
    
    <root level="info">
      <appenderref ref="STDOUT">
    </appenderref></root>
  </logger></logger></logger></logger></logger></logger></logger></loggers>
</configuration>

Mule's Anypoint Studio comes with slf4j librarary. You can create your logger like this:

?
1
private static Logger logger = LoggerFactory.getLogger(TransformListOfObjects.class);
After you put the file of log4j2.xml in place as stated above, you can run your application from the Anypoint Studio, you will see debug messages. These message can be very valuable, in particular, to check the parameters set for various connectors.
at 59 comments:
Subscribe to: Posts (Atom)

Anypoint Studio Error: The project is missing Munit lIbrary to run tests

Anypoint Studio 7.9 has a bug. Even if we following the article: https://help.mulesoft.com/s/article/The-project-is-missing-MUnit-libraries-...