Monday, May 29, 2017

Mule Application Development - Advanced Topics (1): Global Functions

Introduction

In my previous post, I have talked about using Data-weave functions. Now, you will see that you may use those functions in different places. Sometimes, you may need to use the same function in MEL. For these kind of situations, the global function will resolve the reuse of functions. In this post, I am going to demonstrate how to you define these functions and how to use them.

Define Global Functions

Global functions can be defined any where in mule configurations files. However, it is better to define in a specific file. Here I have a file named: common.xml with the following contents:

<!--xml version="1.0" encoding="UTF-8"?-->
 
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.8.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
 
     <configuration doc:name="Configuration">
         <expression-language autoresolvevariables="true">
             <import class="org.mule.util.StringUtils">
             <import class="org.mule.el.datetime.DateTime">
             <global-functions>
               
                 def getReasonId2(reason) {
                   return StringUtils.splitAndTrim(reason, "-")[1];
                 } 
                  
                 def normalizeDate (aDateString) {
                  date = new DateTime(aDateString, 'MM/dd/yyyy');
             date.format('yyyy-MM-dd');
             return date;
                 }
                  
                 def mergeArray(data) {
                  sValue = StringUtils.join(data,',');
                  return sValue
                 }
               
            </global-functions>
         </import></import></expression-language>
     </configuration>
</mule>

In the above code, I defined 3 functions, getReasonId2, normalizeDate, and mergeArray. These functions use class of org.mule.util.StringUtils and org.mule.el.datetime.DateTime. These are imported before the global-functions element. As you can see, we are using classes from other packages. you can develop your own java classes and imported here too. We will demonstrate that in the next blog. For now, let demonstrate how to use them. Actually, the usage is the same as we use in data-weave functions. Mule runtime will be able to figure out where it is defined.

Usages

<!--xml version="1.0" encoding="UTF-8"?-->
 
<mule xmlns:dw="http://www.mulesoft.org/schema/mule/ee/dw" xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
http://www.mulesoft.org/schema/mule/ee/dw http://www.mulesoft.org/schema/mule/ee/dw/current/dw.xsd">
    <flow name="global_functionsFlow">
        <http:listener config-ref="HTTP_Listener_Configuration" path="/advanced/global-function" doc:name="HTTP">
        <dw:transform-message doc:name="Transform Message">
            <dw:set-payload><!--[CDATA[%dw 1.0
%output application/json
---
payload map ((record) ---> {
 staff_id: record."Buyer Reference",
 Contract_End_Date: normalizeDate(record."End Date") as :date,
 Last_Day_of_Work: normalizeDate(record."End Date") as :date,
 termination_reason: getReasonId2(record."Closed Reason")
})]]></dw:set-payload>
        </dw:transform-message>
        <set-variable variablename="Test-Reason" value="#[getReasonId2('First - Second')]" doc:name="Variable">
        <logger message="#[payload]" level="INFO" doc:name="Logger">
    </logger></set-variable></http:listener></flow>
 
</mule>

Sample Input

[
  {
    "Buyer Reference": "207157",
    "Start Date": "05/01/2017",
    "End Date": "05/01/2017",
    "Closed Reason": "Voluntary - Assignment Completed"
  },
   {
    "Buyer Reference": "207157",
    "Start Date": "05/01/2017",
    "End Date": "05/01/2017",
    "Closed Reason": "Voluntary - Assignment Completed"
  }
]

Output From Sample Input

[
  {
    "staff_id": "207157",
    "Contract_End_Date": "2017-05-01",
    "Last_Day_of_Work": "2017-05-01",
    "termination_reason": "Assignment Completed"
  },
  {
    "staff_id": "207157",
    "Contract_End_Date": "2017-05-01",
    "Last_Day_of_Work": "2017-05-01",
    "termination_reason": "Assignment Completed"
  }
]

The above code demonstrates the usage of global functions in data-weave and MEL expression.

Further Improvement

The global-functions can be defined in a file. In this way, we separate the mule configuration files with global functions definition files. The following is the improved configuration file:

<!--xml version="1.0" encoding="UTF-8"?-->
 
<mule xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:spring="http://www.springframework.org/schema/beans" version="EE-3.8.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd
http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd">
 
     <configuration doc:name="Configuration">
         <expression-language autoresolvevariables="true">
             <import class="org.mule.util.StringUtils">
             <import class="org.mule.el.datetime.DateTime">
             <global-functions file="my_global_functions.mvel">    
            </global-functions>
         </import></import></expression-language>
     </configuration>
</mule>

Put my_global_functions.mven under src/main/resources, or any places within classpath.

Tuesday, May 23, 2017

Mule Dev Tricks: Create An Array List From JSON Using DataWeave

Input

[
 {
  "id" : "11111",
  "name" : "AAAAA"
 },
 {
  "id" : "22222",
  "name": "BBBBB"
 },
 {
  "id" : "33333",
  "name" : "CCCCC"
 }
] 

Output

1	`["11111",` `"22222",` `"33333"]`

As you can see, we only want to extract the id from the input.

DataWeave Script

%dw 1.0
%output application/java
---
payload map (data) -> data.id

The map function from the Dataweave component is the mostly used function. Here we using a simple lamda expression to extract id. Another simple way is:

payload map $.id

Tuesday, May 9, 2017

Mule Dev Tricks: Transform Dates Using Datawave Functions

Introduction

Processing dates is very common practice in data transformation. Dataweave's function makes this task very easy. This short blog demonstrate the procedures to do so.

Normalizing Dates

As shown in the following Dataweave code. The function normalize replace "/" and "." with "-". Note: we can use replace in a concatenation style.

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-"
---
{
 dates : {
  date1: normalize("26-JUL-16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date2: normalize("26/JUL/16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date3: normalize("26.JUL.16") as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
   }
}

In the above code, I provide two flavors for transforming the dates to the format of "yyyy-MM-dd". Apparently, date3 is a bit cleaner. The above code can be simplified by adding the data format in the function body:

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-" as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
---
{
 dates : {
  date1: normalize("26-JUL-16"),
  date2: normalize("26/JUL/16"),
  date3: normalize("26.JUL.16") 
   }
}

As you can see, now the Data-weave code is much cleaner.

Wednesday, May 3, 2017

Mule Dev Tricks: DataWeave - Using mapObject - Part IV

Introduction

In my last post I covered the topic of removing null object and empty strings from JSON payload. That solution is only good for two level of objects. If the JSON hierarchy has multiple level, then it would not work. In that case, we need to use recursion. This post will show the solution for this purpose.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : {
      "number" : "",
      "name" : ""
     },
     "city" : null,
     "zip" : null
    }
  }
]

Desired Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

%function skipEmpty(o) o
 unless o is :object 
    otherwise o mapObject {
        (($$): skipEmpty($)) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
 
})

Key Learning Points

In the solution given here, I used the recursion as the following:

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

This demonstrates that dataweave functions allow us to use the recursion concept. The code provide here is not perfect yet. There is a bug I am working on.

Tuesday, May 2, 2017

Mule Dev Tricks: DavaWeave Using mapObject - Part III

Introduction

In the message flow from different end points, many data tributes are null. In particular, if the source data are from database, where a lot of attributes can be null or empty string. In this kind of scenarios, we don't want to put all the null objects and empty strings on the wire. This post explain the procedures to remove the null object or empty string from the message payload.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : null,
     "city" : null,
     "zip" : null
    }
  }
]

Desire Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): $) when ($ != null)
    }

%function skipEmpty(o) o mapObject {
        (($$): $) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
})

Key Learning Points

This dataweave scripts is not perfect yet. More sophisticated solution will be discussed later. What we can learn from this solution is the following:

mapObject function with conditions, which are using in the functions of skipNulls and skipEmpty.
```
(($$): $) when ($ != {} and $ != '')
```
mapObject is just a function, you can use it in custom defined function as well.

Monday, May 1, 2017

Mule Dev Tricks: DataWeave Using mapObject - Part II

Introduction

In the last part DataWeave Using mapObject - Part I. I presented a way to add new attribute to the payload. In this post, I am going to demonstrate the dataweave code for change keys of the JSON payload.

Input Data

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

Desired Result

[
  {
    "ew_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (ew_id : value) when key as :string == 'worker_id', 
  ((key) : value) when (key as :string != 'worker_id')
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key learning Points

From the solution I provided, the key syntax is that if the key is worker_id, we will update the key with new key value which is ew_id.

1 2	`(ew_id : value) when key as :string ==` `'worker_id',` `((key) : value) when (key as :string !=` `'worker_id')`