Mule Dev Tricks: Transform Dates Using Datawave Functions

Introduction

Processing dates is very common practice in data transformation. Dataweave's function makes this task very easy. This short blog demonstrate the procedures to do so.

Normalizing Dates

As shown in the following Dataweave code. The function normalize replace "/" and "." with "-". Note: we can use replace in a concatenation style.

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-"
---
{
 dates : {
  date1: normalize("26-JUL-16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date2: normalize("26/JUL/16") as :date {format: "dd-MMM-yy"} as :string {format: "yyyy-MM-dd"} as :date,
  date3: normalize("26.JUL.16") as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
   }
}

In the above code, I provide two flavors for transforming the dates to the format of "yyyy-MM-dd". Apparently, date3 is a bit cleaner. The above code can be simplified by adding the data format in the function body:

%dw 1.0
%output application/json
%function normalize(date) date replace "/" with "-" replace "." with "-" as :date {format: "dd-MMM-yy"} as :date {format: "yyyy-MM-dd"}
---
{
 dates : {
  date1: normalize("26-JUL-16"),
  date2: normalize("26/JUL/16"),
  date3: normalize("26.JUL.16") 
   }
}

As you can see, now the Data-weave code is much cleaner.

Mule Dev Tricks: DataWeave - Using mapObject - Part IV

Introduction

In my last post I covered the topic of removing null object and empty strings from JSON payload. That solution is only good for two level of objects. If the JSON hierarchy has multiple level, then it would not work. In that case, we need to use recursion. This post will show the solution for this purpose.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : {
      "number" : "",
      "name" : ""
     },
     "city" : null,
     "zip" : null
    }
  }
]

Desired Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

%function skipEmpty(o) o
 unless o is :object 
    otherwise o mapObject {
        (($$): skipEmpty($)) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
 
})

Key Learning Points

In the solution given here, I used the recursion as the following:

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): skipNulls($)) when ($ != null and $ != "")
    }

This demonstrates that dataweave functions allow us to use the recursion concept. The code provide here is not perfect yet. There is a bug I am working on.

Mule Dev Tricks: DavaWeave Using mapObject - Part III

Introduction

In the message flow from different end points, many data tributes are null. In particular, if the source data are from database, where a lot of attributes can be null or empty string. In this kind of scenarios, we don't want to put all the null objects and empty strings on the wire. This post explain the procedures to remove the null object or empty string from the message payload.

Sample Input

[
  {
    "first_name" : "Gary",
    "middle_name" : "",
    "last_name" : "Liu",
    "address" : {
     "street" : null,
     "city" : null,
     "zip" : null
    }
  }
]

Desire Output

[
  {
    "first_name": "Gary",
    "last_name": "Liu"
  }
]

Solution

%dw 1.0
%output application/json

%function skipNulls(o) o 
    unless o is :object 
    otherwise o mapObject {
        (($$): $) when ($ != null)
    }

%function skipEmpty(o) o mapObject {
        (($$): $) when ($ != {} and $ != '')
    }

---
payload map ((person)  -> { 
 (skipEmpty(person mapObject { 
         ($$): skipNulls($)
     })
 )
})

Key Learning Points

This dataweave scripts is not perfect yet. More sophisticated solution will be discussed later. What we can learn from this solution is the following:

• mapObject function with conditions, which are using in the functions of skipNulls and skipEmpty.

  (($$): $) when ($ != {} and $ != '')

• mapObject is just a function, you can use it in custom defined function as well.

Mule Dev Tricks: DataWeave Using mapObject - Part II

Introduction

In the last part DataWeave Using mapObject - Part I. I presented a way to add new attribute to the payload. In this post, I am going to demonstrate the dataweave code for change keys of the JSON payload.

Input Data

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

Desired Result

[
  {
    "ew_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "ew_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (ew_id : value) when key as :string == 'worker_id', 
  ((key) : value) when (key as :string != 'worker_id')
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key learning Points

From the solution I provided, the key syntax is that if the key is worker_id, we will update the key with new key value which is ew_id.

  (ew_id : value) when key as :string == 'worker_id', 
  ((key) : value) when (key as :string != 'worker_id')

Mule Dev Tricks: DataWeave Using mapObject - Part I

Introduction

Here is a problem I am facing. An incoming JSON payload misses a piece of information, the transaction type. This piece of information can be deducted from the existing information. Actually, the really application is more more complicated. In this Mule Dev Trick, I am going to demonstrate how to add new information to the payload.

This small piece of code involves several key concept of Dataweave:

• map
• mapObject
• Condition statement
• Lamda sytax

Input Payload

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

The Desired Output

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

As you can see that the "type" attribute is introduced based on whether the staff_id is empty or not. If the staff_id attribute is not empty, I set the value of this attribute to "re-hire", otherwise, it will be "new-hire".

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  "$key" : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Note: the $key is used here. There is another syntax is (key). Thus, the above code can be as the following:

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (key) : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key Learning Points

The above script seems pretty simple. It has used many key concepts of the DataWeave component. The key of the map function is to iterate a List or Array and produce a new array. And the main usage of mapObject is to iterate the attributes of an object. Both functions use Lamda expression which takes two value, $$, $. By default, $ represent the value of the current object the iterator is point to, and $$ is the index or key of the object.

Another worth noting point is that map can take single argument as well.

Mule Dev Tricks: DataWeave Using Reduction

Introduction

In this post, I am going to demonstrate how to convert an array of JSON payload to a single JSON payload.

Input

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "199508"
  }
]

Desired Output

{
  "AMAGWK00002673": "199504",
  "AMAGWK00002674": "199506",
  "AMAGWK00002675": "199508"
}

Solution

The first step is to use dataweave to extrace the worker_id. This is very straight forward. The only trick about this piece of script is to use value as key in the transformed JSON result. This is achieved by using ($.key) : $.otherKey. here key = worker_id and the otherKey is staff_id.

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) 

The above dataweave script will produce the following result:

[
  {
    "AMAGWK00002673": "199504"
  },
  {
    "AMAGWK00002674": "199506"
  },
  {
    "AMAGWK00002675": "199508"
  }
]

In order to produce the final result, we need to use reduction reduce($$ + $).

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) reduce($$ ++ $)

Note: You have to use reduce ($$ ++$). The round bracket is important! The MuleSoft documentation missed that which causes many confusions.

Mule Dev Tricks: Using xpath3

Introduction

Even though today most web services using RESTful architecture, there are still a lot of legacy systems using SOAP. Handling xml file is challenging, if not difficult. Fortunately, Mule Application platform provide several utilities to alleviate the pain. The most powerful component is Dataweave, which is very handle to build SOAP request and handle responses. However, some cases, xpath3 is an easier solution. Sometimes it may be the best solution, for instance, if we want to split the xml payload based on certain node.

In this short article, I will introduce two use cases:

1. Extract values from xml payload
2. Split the payload

Use Case One: Extract Value Of An Element

Input Example

Here is the input:

    
        31bf3ce3d6bb1025d71346c6ec276bd6
    
    
        31bf3ce3d6bb1025d71348e2f1c26bdc
        35119960
    

I want to extract the value of Position_ID in the line of:

    35119960

Solution

There are two solutions in this case. The first is:

    #[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[2]')]

The above MEL will assign position_id to 35119960. Note: XML node count starting from 1 not from 0 as in Java array.

The other better solution is the following:

#[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[@*:type=\'Position_ID\']')]

Here is the trick is \/*:ID[@*:type=\'Position_ID\'].

1. Refer element attribute using [@*:type]
2. Escape the single quote with backslash \'

Use Case Two: Use xpath3 To Split Payload

Example Input

In this use case, I would like to extract the node of StockMovementData, so that I can process the data.

    

        Warehouse
        Location
        idm1468201212
        2016-04-13T11:55:30.263+10:00
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700030011
        100
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700029911
        100
    

Here is the mule flow:

Solution

The above flow, split the xml payload using the following xpath3 expression:

Complete Mule Flow Configuration