Mule Application Development: Remote Debugging

This paper is published in the DZone

https://dzone.com/articles/remote-debugging-mule-application

Mule Application Development: Effective Caching

This paper has been published at DZone:

Mule Application Development: Effective Caching

Mule Dev Tricks: Dataweave Working With XML Data

The Use Case

In my earlier post, I have explained the tricks using xpath3() to extract data from xml data. In this post I will demonstrate a use case to extract data using Dataweave.

Here is the input data (I omitted the not-interesting xml data:

...
            
                        
                            anshul.gupta277@gmail.com
                            
                                
                                    
                                        836cf00ef5974ac08b786079866c946f
                                        HOME
                                    
                                
                            
                        
                        
                            sudeshna.nanda@tcs.com
                            
                                
                                    
                                        1f27f250dfaa4724ab1e1617174281e4
                                        WORK
                                    
                                
                            
                        
            
...

The desired output is an arraylist:

[
     {
         email=anshul.gupta277@gmail.com, 
         wid=836cf00ef5974ac08b786079866c946f, 
         type=HOME
     }, 
     {
         email=sudeshna.nanda@tcs.com, 
         wid=1f27f250dfaa4724ab1e1617174281e4, 
         type=WORK
     }
]

The Solution

The following is the solution:

%dw 1.0
%output application/java
%namespace ns0 urn:com.workday/bsvc
---
payload.ns0#Get_Workers_Response.ns0#Response_Data.*ns0#Worker.ns0#Worker_Data.ns0#Personal_Data.ns0#Contact_Data.*ns0#Email_Address_Data
map {
 email: $.ns0#Email_Address,
 wid: $.ns0#Usage_Data.ns0#Type_Data.ns0#Type_Reference.*ns0#ID[?($.@ns0#type == "WID")][0],
 type: $.ns0#Usage_Data.ns0#Type_Data.ns0#Type_Reference.*ns0#ID[?($.@ns0#type == "Communication_Usage_Type_ID")][0]
}

The Key Learning Points

1. Make array with wildcard *: ns0#Contact_Data.*ns0#Email_Address_Data
2. Get expect element using: ns0#ID[?($.@ns0#type == "WID")][0]
3. Remember: access attribute of a xml element using @

Mule Application Development - Advanced Topics (1): Global Functions

Introduction

In my previous post, I have talked about using Data-weave functions. Now, you will see that you may use those functions in different places. Sometimes, you may need to use the same function in MEL. For these kind of situations, the global function will resolve the reuse of functions. In this post, I am going to demonstrate how to you define these functions and how to use them.

Define Global Functions

Global functions can be defined any where in mule configurations files. However, it is better to define in a specific file. Here I have a file named: common.xml with the following contents:

     
         
             
             
             
              
                 def getReasonId2(reason) {
                   return StringUtils.splitAndTrim(reason, "-")[1];
                 } 
                 
                 def normalizeDate (aDateString) {
                  date = new DateTime(aDateString, 'MM/dd/yyyy');
             date.format('yyyy-MM-dd');
             return date;
                 }
                 
                 def mergeArray(data) {
                  sValue = StringUtils.join(data,',');
                  return sValue
                 }
              
            
         
     

In the above code, I defined 3 functions, getReasonId2, normalizeDate, and mergeArray. These functions use class of org.mule.util.StringUtils and org.mule.el.datetime.DateTime. These are imported before the global-functions element. As you can see, we are using classes from other packages. you can develop your own java classes and imported here too. We will demonstrate that in the next blog. For now, let demonstrate how to use them. Actually, the usage is the same as we use in data-weave functions. Mule runtime will be able to figure out where it is defined.

Usages

    
        
        
             {
 staff_id: record."Buyer Reference",
 Contract_End_Date: normalizeDate(record."End Date") as :date,
 Last_Day_of_Work: normalizeDate(record."End Date") as :date,
 termination_reason: getReasonId2(record."Closed Reason")
})]]>
        
        
        
    

Sample Input

[
  {
    "Buyer Reference": "207157",
    "Start Date": "05/01/2017",
    "End Date": "05/01/2017",
    "Closed Reason": "Voluntary - Assignment Completed"
  },
   {
    "Buyer Reference": "207157",
    "Start Date": "05/01/2017",
    "End Date": "05/01/2017",
    "Closed Reason": "Voluntary - Assignment Completed"
  }
]

Output From Sample Input

[
  {
    "staff_id": "207157",
    "Contract_End_Date": "2017-05-01",
    "Last_Day_of_Work": "2017-05-01",
    "termination_reason": "Assignment Completed"
  },
  {
    "staff_id": "207157",
    "Contract_End_Date": "2017-05-01",
    "Last_Day_of_Work": "2017-05-01",
    "termination_reason": "Assignment Completed"
  }
]

The above code demonstrates the usage of global functions in data-weave and MEL expression.

Further Improvement

The global-functions can be defined in a file. In this way, we separate the mule configuration files with global functions definition files. The following is the improved configuration file:

Put my_global_functions.mven under src/main/resources, or any places within classpath.

Mule Dev Tricks: Create An Array List From JSON Using DataWeave

Input

[
 {
  "id" : "11111",
  "name" : "AAAAA"
 },
 {
  "id" : "22222",
  "name": "BBBBB"
 },
 {
  "id" : "33333",
  "name" : "CCCCC"
 }
] 

Output

["11111", "22222", "33333"]

As you can see, we only want to extract the id from the input.

DataWeave Script

%dw 1.0
%output application/java
---
payload map (data) -> data.id

The map function from the Dataweave component is the mostly used function. Here we using a simple lamda expression to extract id. Another simple way is:

payload map $.id  

Mule Dev Tricks: Transform Dates Using Datawave Functions

Introduction

Processing dates is very common practice in data transformation. Dataweave's function makes this task very easy. This short blog demonstrate the procedures to do so.

Normalizing Dates

As shown in the following Dataweave code. The function normalize replace "/" and "." with "-". Note: we can use replace in a concatenation style.

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-"
---
{
 dates : {
  date1: normalize("26-JUL-16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date2: normalize("26/JUL/16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date3: normalize("26.JUL.16") as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
   }
}

In the above code, I provide two flavors for transforming the dates to the format of "yyyy-MM-dd". Apparently, date3 is a bit cleaner. The above code can be simplified by adding the data format in the function body:

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-" as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
---
{
 dates : {
  date1: normalize("26-JUL-16"),
  date2: normalize("26/JUL/16"),
  date3: normalize("26.JUL.16") 
   }
}

As you can see, now the Data-weave code is much cleaner.

Mule Dev Tricks: DataWeave - Using mapObject - Part IV

Introduction

In my last post I covered the topic of removing null object and empty strings from JSON payload. That solution is only good for two level of objects. If the JSON hierarchy has multiple level, then it would not work. In that case, we need to use recursion. This post will show the solution for this purpose.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : {
      "number" : "",
      "name" : ""
     },
     "city" : null,
     "zip" : null
    }
  }
]

Desired Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

%function skipEmpty(o) o
 unless o is :object 
    otherwise o mapObject {
        (($$): skipEmpty($)) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
 
})

Key Learning Points

In the solution given here, I used the recursion as the following:

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

This demonstrates that dataweave functions allow us to use the recursion concept. The code provide here is not perfect yet. There is a bug I am working on.

Mule Dev Tricks: DavaWeave Using mapObject - Part III

Introduction

In the message flow from different end points, many data tributes are null. In particular, if the source data are from database, where a lot of attributes can be null or empty string. In this kind of scenarios, we don't want to put all the null objects and empty strings on the wire. This post explain the procedures to remove the null object or empty string from the message payload.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : null,
     "city" : null,
     "zip" : null
    }
  }
]

Desire Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): $) when ($ != null)
    }

%function skipEmpty(o) o mapObject {
        (($$): $) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
})

Key Learning Points

This dataweave scripts is not perfect yet. More sophisticated solution will be discussed later. What we can learn from this solution is the following:

• mapObject function with conditions, which are using in the functions of skipNulls and skipEmpty.

  (($$): $) when ($ != {} and $ != '')

• mapObject is just a function, you can use it in custom defined function as well.

Mule Dev Tricks: DataWeave Using mapObject - Part II

Introduction

In the last part DataWeave Using mapObject - Part I. I presented a way to add new attribute to the payload. In this post, I am going to demonstrate the dataweave code for change keys of the JSON payload.

Input Data

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

Desired Result

[
  {
    "ew_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (ew_id : value) when key as :string == 'worker_id', 
  ((key) : value) when (key as :string != 'worker_id')
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key learning Points

From the solution I provided, the key syntax is that if the key is worker_id, we will update the key with new key value which is ew_id.

  (ew_id : value) when key as :string == 'worker_id', 
  ((key) : value) when (key as :string != 'worker_id')

Mule Dev Tricks: DataWeave Using mapObject - Part I

Introduction

Here is a problem I am facing. An incoming JSON payload misses a piece of information, the transaction type. This piece of information can be deducted from the existing information. Actually, the really application is more more complicated. In this Mule Dev Trick, I am going to demonstrate how to add new information to the payload.

This small piece of code involves several key concept of Dataweave:

• map
• mapObject
• Condition statement
• Lamda sytax

Input Payload

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

The Desired Output

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

As you can see that the "type" attribute is introduced based on whether the staff_id is empty or not. If the staff_id attribute is not empty, I set the value of this attribute to "re-hire", otherwise, it will be "new-hire".

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  "$key" : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Note: the $key is used here. There is another syntax is (key). Thus, the above code can be as the following:

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (key) : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key Learning Points

The above script seems pretty simple. It has used many key concepts of the DataWeave component. The key of the map function is to iterate a List or Array and produce a new array. And the main usage of mapObject is to iterate the attributes of an object. Both functions use Lamda expression which takes two value, $$, $. By default, $ represent the value of the current object the iterator is point to, and $$ is the index or key of the object.

Another worth noting point is that map can take single argument as well.

Mule Dev Tricks: DataWeave Using Reduction

Introduction

In this post, I am going to demonstrate how to convert an array of JSON payload to a single JSON payload.

Input

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "199508"
  }
]

Desired Output

{
  "AMAGWK00002673": "199504",
  "AMAGWK00002674": "199506",
  "AMAGWK00002675": "199508"
}

Solution

The first step is to use dataweave to extrace the worker_id. This is very straight forward. The only trick about this piece of script is to use value as key in the transformed JSON result. This is achieved by using ($.key) : $.otherKey. here key = worker_id and the otherKey is staff_id.

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) 

The above dataweave script will produce the following result:

[
  {
    "AMAGWK00002673": "199504"
  },
  {
    "AMAGWK00002674": "199506"
  },
  {
    "AMAGWK00002675": "199508"
  }
]

In order to produce the final result, we need to use reduction reduce($$ + $).

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) reduce($$ ++ $)

Note: You have to use reduce ($$ ++$). The round bracket is important! The MuleSoft documentation missed that which causes many confusions.

Mule Dev Tricks: Using xpath3

Introduction

Even though today most web services using RESTful architecture, there are still a lot of legacy systems using SOAP. Handling xml file is challenging, if not difficult. Fortunately, Mule Application platform provide several utilities to alleviate the pain. The most powerful component is Dataweave, which is very handle to build SOAP request and handle responses. However, some cases, xpath3 is an easier solution. Sometimes it may be the best solution, for instance, if we want to split the xml payload based on certain node.

In this short article, I will introduce two use cases:

1. Extract values from xml payload
2. Split the payload

Use Case One: Extract Value Of An Element

Input Example

Here is the input:

    
        31bf3ce3d6bb1025d71346c6ec276bd6
    
    
        31bf3ce3d6bb1025d71348e2f1c26bdc
        35119960
    

I want to extract the value of Position_ID in the line of:

    35119960

Solution

There are two solutions in this case. The first is:

    #[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[2]')]

The above MEL will assign position_id to 35119960. Note: XML node count starting from 1 not from 0 as in Java array.

The other better solution is the following:

#[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[@*:type=\'Position_ID\']')]

Here is the trick is \/*:ID[@*:type=\'Position_ID\'].

1. Refer element attribute using [@*:type]
2. Escape the single quote with backslash \'

Use Case Two: Use xpath3 To Split Payload

Example Input

In this use case, I would like to extract the node of StockMovementData, so that I can process the data.

    

        Warehouse
        Location
        idm1468201212
        2016-04-13T11:55:30.263+10:00
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700030011
        100
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700029911
        100
    

Here is the mule flow:

Solution

The above flow, split the xml payload using the following xpath3 expression:

Complete Mule Flow Configuration

Two Way SSL Explained

This article is posted at DZone at this location

git: Procedures To Resolve Merge Conflict

GIT Command Line Cheat Sheet

git branch -a #List branches
git branch --delete BRANCH_NAME #Delete a branch
git checkout mybranch #switch to mybranch
git merge FromBranch #merge FromBranch to the current working branch

Introduction

This post is an introduction on how to resolve merge conflict. The following procedure will create a merge conflict

As shown in the following picture, the merge conflict is caused by that the same file was change at two different branches at different time. Typically, one develop check out the code from master branch, made some change and merged to the master branch. Another developer checkout the code after the previous developer and committed the code later.

Create Merge Conflicts

$ mkdir groovy
$ cd groovy
$ vi basic_one.groovy

Copy and paste the following code to the file of basic_one.grooy, and safe it (:wq)

println('Hello World!!!')
def myList = [1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()
newList.each { item -> print item + " " }
println()

Now execute the following commands

git init
git add .
git commit -am "first commit on master"

So far we have commit a file to our local repository. Next, we will create a new branch, namely, develop. We will make changes to the same file

Now execute the following commands

git branch develop    # create develop branch
git branch -a         # list branches, you should see the * master and develop. 
                      # The start * indicates we are working on master
git checkout develop  # switch to develop branch

git branch -a

Now, I see the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git branch -a
* develop
  master
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

You can see that the current active branch is develop. We will change the content of the file to the following:

println('Hello,  Gary Liu!')
def myList = [10, 13, 15, 16, 22, 33, 44, 55] 
println newList = myList.flatten(); 
newList.each { item -> print item + " " };
println();

Now execute the following commands to commit the develop branch:

git commit -am "First commit to develop branch"

In order to create conflicts, we will need to checkout the master branch and do some modification on the file

git checkout master.

If you execute: cat basic_one.groovy, you should see the following:

println('Hello World!!!')
def myList = [1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()
newList.each { item -> print item + " " }
println()

This is where we left off. In the meantime, develop branch has changed. Let's make some change to the file and it will be looks like the following:

println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33] 
println newList = myList.flatten()

newList.each {
   item -> print item + " " 
}
println()

Remember that we are at master branch now. We will merge from develop branch to the master

git add .
git commit -am "Second commit on master"
git merge develop

Now we see the following output:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git merge develop
Auto-merging basic_one.groovy
CONFLICT (content): Merge conflict in basic_one.groovy
Automatic merge failed; fix conflicts and then commit the result.
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

We have successfully created merge conflict. If you can the file of basic_one.groovy, it looks like the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ cat basic_one.groovy 
<<<<<<< HEAD
println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()

newList.each {
   item -> print item + " "
}
println()
=======
println('Hello,  Gary Liu!')
def myList = [10, 13, 15, 16, 22, 33, 44, 55]
println newList = myList.flatten(); 
newList.each { item -> print item + " " };
println();
>>>>>>> develop
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

Resolve Merge Conflicts

The above case is a bit extreme, as the whole file is almost changed. The HEAD section is from master branch and the lower part is from develop. To resolve the merge conflict, we need to edit file. For this case I will just take the master case, and the resultant file looks like the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ vi basic_one.groovy 
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git commit -am "Resolve merge conflict"
[master d2d5ecf] Resolve merge conflict
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git merge develop
Already up-to-date.
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ cat basic_one.groovy 
println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()

newList.each {
   item -> print item + " "
}
println()
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

To resolve merge conflict, we have to manually modify the file. We can use IDE like eclipse, sourceTree, etc, but we have to make decisions on which one is the best. After that we need to commit the changes and do a merge again to make sure that all the conflicts are addressed correctly.

Mule Dev Trick: Using Message Properties Transformer

The Abuse of set-variable

Many places I saw the code like the following:

Basically, developers set a bunch of flow variables for later usage. First of all, this kind of situation reflects design flaws. We should ask our selves why do we need to set so many flow variables? Can we use bean instead?

Use Message Property Transformer

Mule platform provide a very powerful transformer, namely message property transformer. Here is an example code:

The scope can be:

  invocation: flow variables
  outbound: property
  session: session variables

By using message property transformer, you can simplify the design, and make the flow easier to review.

Mule Dev Trick: Load A Text File And Convert To ArrayList

The Challenge

Here is the task. I have a file like the following:

AAAA
BBBB
CCCC
DDDD

I need to convert this content to an ArrayList, so that I can look through them. Actually each line is an object name. I need to query those objects from web services.

Solution

Here are the steps:

• Load file and payload as string. Mule provide a construct called "parse-template", which can be used to load file. This construct will load the file and convert the content to a string payload.
• Use Groovy or MEL to convert the string to an ArrayList

Here are the source code:

Key Learning Points

Few points we learn from this:

1. MEL support java directly. This is really very powerful.
2. Groovy is very powerful tool in Mule application development. collect {it.trim()} is similar as java stream function.