Saturday, December 15, 2018

Mule 4 Integration With Cassandra

Introduction

Mule 4 Cassandra Database Connector, current version of 3.10, has made significant improvement over the previous version. It is fairly straight forward to setup the integration to the Cassandra Database using the connector. This article is an introduction of the Cassandra Connector to Cassandra cluster.

Cassandra Cluster Configuration

I have created 2 a two node cluster as shown the in following diagram. The details on how to setup the Cassandra clustering will be covered in another post. Basically, I opened the native transport port 9042, which is the default vale on both nodes.
We need to some initial setup using cqlsh too by run the following command:
$ cqlsh -u cassandra -p casandra
cassandra@cqlsh> create keyspace if not exists dev_keyspace with replication = {'class' : 'SimpleStrategy', 'replication_factor' : 2};
The above command will create a keyspace, namely dev_keyspace. We can query the keyspaces by the following command:
cassandra@cqlsh> desc keyspaces;
You should see the following:
system_schema  system      system_distributed
system_auth    dev_keyspace  system_traces
The next step is to create emp table by the following command:
cassandra@cqlsh> create table emp (empid int primary key, emp_first varchar, emp_last varchar, emp_dept varchar);
And insert a row:
create table emp (empid int primary key, emp_first varchar, emp_last varchar, emp_dept varchar);
insert into emp (empid, emp_first, emp_last, emp_dept) values (1, 'Gary','liu','consulting');
That is all and we are ready to do the integration.

Integration Using Mule 4 Cassandra Connector

Add Cassandra Connector

First, we need to search the Exchange and add the Cassandra Connection as shown in the following snapshot:

Create CassandraDB Config

Create a mule configuration, namely, global-config.xml. Then create CasandraDB Config as the following:
Enter the General setting as the following. Note: Leave the Host empty as we use cluster configuration.
Now, click the "Advanced Settings" tab and enter the information as the following:

As you can see, we can enter the ip addresses separated by comma. In this way, we can achieve high availability from client side of view. Now, we test the connectivity. If the port of 9042 is exposed correctly, it should work fine. I will explain more in the next article on how to make sure we expose the native transport port correctly.

Create Integration Flows

Read Flow

The read flow is very straight forward. The cassandra-db:select operation uses payload as the whole query. We just need to set payload. In this case, it is "select * from emp;"

<flow name="select-objecsFlow" doc:id="8bdcc4fe-cf4e-49be-b2f9-2dfacbddaf4b" >
<http:listener doc:name="Listener" doc:id="ab57d8ab-2623-468b-b635-a0c6efd6c829" config-ref="HTTP_Listener_config" path="/cassandra/emp"/>
<set-payload value="select * from emp;" doc:name="Set Payload"  />
<cassandra-db:select doc:name="Select"  config-ref="CassandraDB_Config_cluster" />
<ee:transform doc:name="Transform Message"  >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="e6097436-1909-4d8b-8c81-27be82c11714" message="#[payload]" />
</flow>

</mule>

Insert A Row

To insert a row, we need to use insert operation as the following:

<flow name="insert-objecsFlow" doc:id="8bdcc4fe-cf4e-49be-b2f9-2dfacbddaf4b" >
<http:listener doc:name="Listener" doc:id="ab57d8ab-2623-468b-b635-a0c6efd6c829" config-ref="HTTP_Listener_config" path="/cassandra/emp/create"/>
<ee:transform doc:name="Transform Message" doc:id="0d484493-fa23-4a56-9547-58f0bb54dbd1" >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/java
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<cassandra-db:insert table="emp" doc:name="Insert" doc:id="71cb7584-981a-4475-a839-ea82ed3b9832" config-ref="CassandraDB_Config_vm1" keyspaceName="rogers_dev"/>
<ee:transform doc:name="Transform Message"  >
<ee:message >
<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
payload]]></ee:set-payload>
</ee:message>
</ee:transform>
<logger level="INFO" doc:name="Logger" doc:id="e6097436-1909-4d8b-8c81-27be82c11714" message="#[payload]" />

</flow>

Currently, we can only insert one row at a time. To insert multiple rows, we can use for loop or use batch processes for large volumes. 

About CassandraDB Connector

The detailed information can be found at the following Mulesoft Document:
https://docs.mulesoft.com/connectors/cassandra/cassandra-connector

Important Cassandra Information

When you using cqlsh to connect Cassandra cluster, you should notice the following:

cqlsh -u cassandra -p cassandra
Connected to DevelopmentCluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.11.3 | CQL spec 3.4.4 | Native protocol v4]

The v4 is the current native protocol version, which is required to configure the connector.



Saturday, August 4, 2018

Mule 4 : Dataweave 2 In Action - Use Function Modules

Introduction

In Mule 4, the mel2 functionalities are replaced by the Dataweave 2 ones. For those who worked in Mule 3, you will find the mule 4 ways of using function modules are much more advanced than the old way (see my post). In this article, I am going to present Mule 4 ways to define and use global functions.

Define Functions In Dataweave Modules

Here is my project layout:

As you can see, I place the function modules in the dir: src/main/resources/modules. And the dataweave function module is defined in CommonFunc.dwl.

%dw 2.0
fun concatName(aPerson) = aPerson.firstName ++ ' ' ++ aPerson.lastName

fun stringToDateUTC(dateString) = ((dateString as String {format: "yyyy-MM-dd'T'HH:mm:ss.SSSZ"} >> "UTC"
 )) as DateTime {format: "yyyy-MM-dd'T'HH:mm:ss.SSS"} 

I defined two functions: concatName, stringToDateUTC. The purposes are self-explained.

Use Function Modules

I place the normal dataweave modules in the dir of src/main/resources/dataweave. The usage of the functions is demonstrated at the following dwl file:

%dw 2.0
import * from modules::CommonFunc
output application/json
---
{
 "Name" : concatName(payload),
 "CreatedDate" : stringToDateUTC(payload.createdDate)
}

Here are few points:

  • import functions
    1. import modules::CommonFuc
    2. import * from modules::CommonFunc
    3. import concatName stringToDateUTC from modules::CommonFunc
  • use functions in dwl
    1. CommonFunc::concatName(payload)
    2. concatName(payload)

For those who know python, you may find the syntax is very similar between the two languages, in terms of modules, and references.

Testing Data

Here is the complete flow:
 
  
   
  
  
   
    
   
  
  
 
Here is testing data:
{
 "firstName" : "Gary",
 "lastName" : "Liu",
 "createdDate":"2018-07-17T16:18:03+00:00"
}

Wednesday, July 25, 2018

Mule 4: Dataweave 2 In Action - Using Filter

Use Cases

The requirement is to update overall status to PARTIAL if there is any failure record in the payload, i. e. item.status = "Failed"

Input:
{
    "status": "SUCCESS",
    "items": [
        {
            "accountId": "10101xyzabc",
            "status": "Success"
        },
        {
            "accountId": "10102aaabbb",
            "status": "Failed"
        }
    ]
}
Expected Output:
{
    "status": "PARTIAL",
    "items": [
        {
            "accountId": "10101xyzabc",
            "status": "Success"
        },
        {
            "accountId": "10102aaabbb",
            "status": "Failed"
        }
    ]
}

Datawave 2.0

%dw 2.0
output application/json
---
{
 status: if (sizeOf(payload.items map($) filter ($.status == "Failed")) > 0) "PARTIAL" else "SUCCESS",
 items: payload.items 
}

Complete Flow

The following is the complete flow. Please note that I have commented out the transformation payload to java. This is a big improvement of Mule 4 over mule 3.
 
  
   
  
  
  
   
    
   
   
   
  
  
 

Key Learnings

  1. sizeOf function
  2. filter function
  3. if else, in Dataweave 1.0, this will be when else
if else is a very good syntax improvement over Dataweave 1.0

Sunday, July 15, 2018

Enable JMX Authencation And SSL For Mule Runtime

Introduction

In my previous blog, I demonstrated how to change mule application logging level dynamically by using JMX MBeans. In that blog, I skipped the procedure on how to enable SSL for JMX of Mule runtimes. Apparently, in production environment, we will need to enable both authentication and SSL for the security purpose.

I will demonstrate the details about enabling SSL for on-premises Mule Runtimes. I will use local generated Cert for demonstration purpose. You may need to authorized the cert for your organization, but the basic procedures are the same.

Generate Keystore and Truststore

On mule runtime server, execute the following commands:

mkdir ${MULE_HOME}/ssl
cd ${MULE_HOME}/ssl
keytool -genkey -alias tc401 -keyalg RSA -keystore tc401_keystore.jks
keytool -export alias tc401 -file tc401_cert -keystore tc401_keystore.jks
keytool -import -alias tc401 -keystore tc401_truststore.jks -file tc401_cert

The above commands will create keystore and truststore, which will be used by Mule Runtimes. To instruct a Mule Runtime to use the keystore and truststore, we need to update wrapper.conf file.

Configure Mule Runtime with Authentication and SSL

Add the following lines to ${MULE_HOME}/conf/wrapper.conf

wrapper.java.additional.50=-Dcom.sun.management.jmxremote=true
wrapper.java.additional.51=-Dcom.sun.management.jmxremote.port=1099
wrapper.java.additional.53=-Dcom.sun.management.jmxremote.access.file=%MULE_HOME%/conf/jmxremote.access
wrapper.java.additional.54=-Dcom.sun.management.jmxremote.password.file=%MULE_HOME%/conf/jmxremote.password
wrapper.java.additional.56=-Dcom.sun.management.jmxremote.authenticate=true
wrapper.java.additional.57=-Dcom.sun.management.jmxremote.ssl=true
wrapper.java.additional.58=-Djavax.net.ssl.keyStore=%MULE_HOME%/ssl/tc401_keystore.jks
wrapper.java.additional.59=-Djavax.net.ssl.keyStorePassword=changeme
wrapper.java.additional.60=-Djavax.net.ssl.trustStore=%MULE_HOME%/ssl/tc401_keystore.jks
wrapper.java.additional.61=-Djavax.net.ssl.trustStorePassword=changeme

Note that I use jmxremote.access and jmxremote.password for the user permission and authentication. The details can be refered in my last blog.

Start jvisualvm

jvisualvm -J-Djavax.net.ssl.trustStore=./tc401_truststore.jks -J-Djavax.net.ssl.trustStorePassword=changeme
The following snapshots shows how the page of login with SSL enabled.

Friday, July 13, 2018

Dynamically Change Mule Application Logging Level At Runtime

Introduction

In many situations, we need to change the logging level from WARN to DEBUG, then change it back to WARN after a period of time. There are few ways to do so as the following. Many unexperienced developers will change log4j2.xml file. For instance, if we want to log all the requests and responses for all HTTP Listener, we can change the log4j2.xml by add the following lines:

<AsyncLogger name="org.mule.module.http.internal.HttpMessageLogger" level="INFO"/>
<AsyncLogger name="com.ning.http" level="INFO" />

This approach works, but it is often very changing if not impossible. To change code in production environment should be considered as a last resort. There are other ways such as  command line, or use web services.
< However all the above mentioned approach requires more effort the JMX approach, which is simplest way in my humble opinion. This article will demonstrate how we can achieve this.

Enable JMX For Mule Runtime

To enable jmx, we will need to update wrapper.conf, which is at ${MULE_HOME}/conf/wrapper.conf. Add the following lines:

wrapper.java.additional.50=-Dcom.sun.management.jmxremote
wrapper.java.additional.50=-Dcom.sun.management.jmxremote=true
wrapper.java.additional.51=-Dcom.sun.management.jmxremote.port=1099
wrapper.java.additional.53=-Dcom.sun.management.jmxremote.access.file=%MULE_HOME%/conf/jmxremote.access
wrapper.java.additional.54=-Dcom.sun.management.jmxremote.password.file=%MULE_HOME%/conf/jmxremote.password
wrapper.java.additional.56=-Dcom.sun.management.jmxremote.authenticate=true
wrapper.java.additional.55=-Dcom.sun.management.jmxremote.ssl=false

The above configuration requires to create two files: jmxremote.access and jmxremote.password under ${MULE_HOME}/conf. The following are examples:

$cat jmxremote.access
admin readwrite
gary  readonly

$cat jmxremote.password
admin admin
gary gary

To enable ssl requires generate certificates. I will cover the topic later

Configure jvisualvm And Change Logging Level

jvisualvm comes with Java SDK. For the details about the setup you may refer to my blog . The most important thing is to make sure you install MBeans pluging. The following snapshot shows the details:
You can traverse to the component at org.apache.logging.log4j2, under Loggers, you can change any log4j2 bean's log level as shown in the following snapshot:
In practice, you can do a lot of more with jvisualvm to inspect the mule runtime. I will cover more in my later blogs.

Tuesday, July 10, 2018

Install anypoint-cli on-premise in Off-Line mode

Introduction

anypoint-cli is a very powerful tool, which can perform a lot of operations of mule application management, environment setup, etc. Mulesoft provides a good document online: https://docs.mulesoft.com/runtime-manager/anypoint-platform-cli. However, in many on-premise environment there is no direct access to outside world. We need to install anypoint-cli in the off-line mode. Mulesoft has a documentation on this: https://support.mulesoft.com/s/article/How-to-perform-offline-installation-of-Anypoint-CLI This post is to address the details of the deployment, which includes install node, configure anypoint-cli environment.

Make Sure Your Proxy Setup Is Correct

On-premise setup mule often requires proxy setup. Here is the example of the proxy configuration:
export https_proxy=http://functional-account-name:password@YOUR-DOMAIN.com:YOUR-PORT
export no_proxy=xyz.com,localhost,,192.168.0.0/16,127.0.0.1,.xyz.com,.abc.com

Install Node

The following link is very helpful: https://tecadmin.net/install-latest-nodejs-and-npm-on-centos/ Make sure you have root access.

Install anypoint-cli

Follow the mulesoft document. https://support.mulesoft.com/s/article/How-to-perform-offline-installation-of-Anypoint-CLI. Make sure use root.
  npm install -g npm-bundle
  npm-bundle --verbose anypoint-cli
The last command will download anypoint-cli tgz file: anypoint-cli-2.3.2.tgz Once the lastest anypoint-cli package is downloaded, you can copy it to target linux node and do the following (as sudo user, don't run as root directly):
  npm install -g /tmp/anypoint-cli-2.3.2.tgz

Configuration & Test Run

Put the following in you .anypoint_env.dev file:
ANYPOINT_ENV=dev
ANYPOINT_HOST=anypoint.mulesoft.com
ANYPOINT_USERNAME=anypoint-user-name
ANYPOINT_PASSWORD=password
ANYPOINT_ORG=org-name
You can create many file like this. When you work on one, say, dev environemtn, you can source the file of .anypoint_env.dev. Once you have completed the installation you can run the following command as normal user:
anypoint-cli runtime-mgr standalone-application list
ID      Name                                        Target ID Target Name           Status  Updated

2400923 gcc-mule-services-cache-management          507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400924 gcc-mule-services-cfgmgmt                   507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400925 gcc-mule-services-cfgmgmt-ptdata-consumer   507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400926 gcc-mule-services-cfgmgmt-ptupdate-consumer 507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400927 gcc-mule-services-datacapture               507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400928 gcc-mule-services-datastore-consumer        507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400929 gcc-mule-services-fileexchange-process      507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400930 gcc-mule-services-filemanager               507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400931 gcc-mule-services-fileupload                507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400932 gcc-mule-services-fileupload-process        507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400933 gcc-mule-services-processlog-consumer       507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400934 gcc-mule-services-sfg                       507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400935 gcc-mule-services-transformation-consumer   507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400936 gcc-mule-services-transporter               507876    gcc-mule-dit1-cluster STARTED 5 hours ago
2400937 gcc-mule-services-validation-consumer       507876    gcc-mule-dit1-cluster STARTED 5 hours ago

Thursday, June 14, 2018

Install & Setup RabbitMQ on MacOS

Instroduction

Installation and setup RabbitMQ is pretty straight forward now, given the well published documentation. This post describes the basic steps for the purpose of Mule integration.

Installation

The easiest one is to use brew as the following:
$ brew install rabbitmq
$ which rabbitmqadmin
The newly installed tools are at:
liug@WRTVLMDV0002N37:~$ ls -lart /usr/local/sbin
total 0
lrwxr-xr-x   1 liug  admin    41B May 30 10:45 rabbitmqctl@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmqctl
lrwxr-xr-x   1 liug  admin    43B May 30 10:45 rabbitmqadmin@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmqadmin
lrwxr-xr-x   1 liug  admin    45B May 30 10:45 rabbitmq-server@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmq-server
lrwxr-xr-x   1 liug  admin    46B May 30 10:45 rabbitmq-plugins@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmq-plugins
lrwxr-xr-x   1 liug  admin    42B May 30 10:45 rabbitmq-env@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmq-env
lrwxr-xr-x   1 liug  admin    50B May 30 10:45 rabbitmq-diagnostics@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmq-diagnostics
lrwxr-xr-x   1 liug  admin    47B May 30 10:45 rabbitmq-defaults@ -> ../Cellar/rabbitmq/3.7.5/sbin/rabbitmq-defaults
lrwxr-xr-x   1 liug  admin    40B May 30 10:45 cuttlefish@ -> ../Cellar/rabbitmq/3.7.5/sbin/cuttlefish

Start Server

$ rabbitmq-server

  ##  ##
  ##  ##      RabbitMQ 3.7.5. Copyright (C) 2007-2018 Pivotal Software, Inc.
  ##########  Licensed under the MPL.  See http://www.rabbitmq.com/
  ######  ##
  ##########  Logs: /usr/local/var/log/rabbitmq/rabbit@localhost.log
                    /usr/local/var/log/rabbitmq/rabbit@localhost_upgrade.log

              Starting broker...
 completed with 6 plugins.

Check Listening Ports

$ lsof -iTCP -nP | egrep LISTEN | egrep beam
beam.smp  67306 liug   79u  IPv4 0x5f93ad255eddc27d      0t0  TCP *:25672 (LISTEN)
beam.smp  67306 liug   90u  IPv4 0x5f93ad255c3a127d      0t0  TCP 127.0.0.1:5672 (LISTEN)
beam.smp  67306 liug   91u  IPv6 0x5f93ad255f408cd5      0t0  TCP *:61613 (LISTEN)
beam.smp  67306 liug   92u  IPv6 0x5f93ad255f408715      0t0  TCP *:1883 (LISTEN)
beam.smp  67306 liug   93u  IPv4 0x5f93ad256141be9d      0t0  TCP *:15672 (LISTEN)
Note:
Port Purpose
5672 TCP
61613 STOMP
1883 MQTT
15672 Management - Web Console

Import Logs

The logs are located at /usr/local/var/log/rabbitmq/rabbit@localhost.log, which contains very important information like the following about different listeners.

Configurations

There are a lot of important configurations located at:

/usr/local/Cellar/rabbitmq/3.7.5/ebin

The default user ID and password are guest/guest, which is defined at rabbit.app file.

Anypoint Studio Error: The project is missing Munit lIbrary to run tests

Anypoint Studio 7.9 has a bug. Even if we following the article: https://help.mulesoft.com/s/article/The-project-is-missing-MUnit-libraries-...