Mule Dev Tricks: DataWeave Using mapObject - Part I

Introduction

Here is a problem I am facing. An incoming JSON payload misses a piece of information, the transaction type. This piece of information can be deducted from the existing information. Actually, the really application is more more complicated. In this Mule Dev Trick, I am going to demonstrate how to add new information to the payload.

This small piece of code involves several key concept of Dataweave:

• map
• mapObject
• Condition statement
• Lamda sytax

Input Payload

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": ""
  }
]

The Desired Output

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506",
    "type": "re-hire"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "",
    "type": "new-hire"
  }
]

As you can see that the "type" attribute is introduced based on whether the staff_id is empty or not. If the staff_id attribute is not empty, I set the value of this attribute to "re-hire", otherwise, it will be "new-hire".

Solution

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  "$key" : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Note: the $key is used here. There is another syntax is (key). Thus, the above code can be as the following:

%dw 1.0
%output application/json
---
payload map ((worker) ->{
 (worker mapObject (value, key) -> {
  (key) : value
 }),
 type : "new-hire" when worker.staff_id == null or worker.staff_id == '' otherwise 're-hire'
})

Key Learning Points

The above script seems pretty simple. It has used many key concepts of the DataWeave component. The key of the map function is to iterate a List or Array and produce a new array. And the main usage of mapObject is to iterate the attributes of an object. Both functions use Lamda expression which takes two value, $$, $. By default, $ represent the value of the current object the iterator is point to, and $$ is the index or key of the object.

Another worth noting point is that map can take single argument as well.

Mule Dev Tricks: DataWeave Using Reduction

Introduction

In this post, I am going to demonstrate how to convert an array of JSON payload to a single JSON payload.

Input

[
  {
    "worker_id": "AMAGWK00002673",
    "position_id": "35119973",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199504"
  },
  {
    "worker_id": "AMAGWK00002674",
    "position_id": "35119943",
    "requisition_id": "R-32353",
    "applicant_id": "199505",
    "staff_id": "199506"
  },
  {
    "worker_id": "AMAGWK00002675",
    "position_id": "35119975",
    "requisition_id": "R-32354",
    "applicant_id": "199507",
    "staff_id": "199508"
  }
]

Desired Output

{
  "AMAGWK00002673": "199504",
  "AMAGWK00002674": "199506",
  "AMAGWK00002675": "199508"
}

Solution

The first step is to use dataweave to extrace the worker_id. This is very straight forward. The only trick about this piece of script is to use value as key in the transformed JSON result. This is achieved by using ($.key) : $.otherKey. here key = worker_id and the otherKey is staff_id.

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) 

The above dataweave script will produce the following result:

[
  {
    "AMAGWK00002673": "199504"
  },
  {
    "AMAGWK00002674": "199506"
  },
  {
    "AMAGWK00002675": "199508"
  }
]

In order to produce the final result, we need to use reduction reduce($$ + $).

payload map ((value, index) -> {
     (value.worker_id): value.staff_id 
}) reduce($$ ++ $)

Note: You have to use reduce ($$ ++$). The round bracket is important! The MuleSoft documentation missed that which causes many confusions.

Mule Dev Tricks: Using xpath3

Introduction

Even though today most web services using RESTful architecture, there are still a lot of legacy systems using SOAP. Handling xml file is challenging, if not difficult. Fortunately, Mule Application platform provide several utilities to alleviate the pain. The most powerful component is Dataweave, which is very handle to build SOAP request and handle responses. However, some cases, xpath3 is an easier solution. Sometimes it may be the best solution, for instance, if we want to split the xml payload based on certain node.

In this short article, I will introduce two use cases:

1. Extract values from xml payload
2. Split the payload

Use Case One: Extract Value Of An Element

Input Example

Here is the input:

    
        31bf3ce3d6bb1025d71346c6ec276bd6
    
    
        31bf3ce3d6bb1025d71348e2f1c26bdc
        35119960
    

I want to extract the value of Position_ID in the line of:

    35119960

Solution

There are two solutions in this case. The first is:

    #[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[2]')]

The above MEL will assign position_id to 35119960. Note: XML node count starting from 1 not from 0 as in Java array.

The other better solution is the following:

#[xpath3('/*:Create_Position_Response/*:Position_Reference/*:ID[@*:type=\'Position_ID\']')]

Here is the trick is \/*:ID[@*:type=\'Position_ID\'].

1. Refer element attribute using [@*:type]
2. Escape the single quote with backslash \'

Use Case Two: Use xpath3 To Split Payload

Example Input

In this use case, I would like to extract the node of StockMovementData, so that I can process the data.

    

        Warehouse
        Location
        idm1468201212
        2016-04-13T11:55:30.263+10:00
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700030011
        100
    
    
        YES
        2016-04-13T11:55:30.263+10:00
        30-80
        client
        7CAGL3G00
        700029911
        100
    

Here is the mule flow:

Solution

The above flow, split the xml payload using the following xpath3 expression:

Complete Mule Flow Configuration

Two Way SSL Explained

This article is posted at DZone at this location

git: Procedures To Resolve Merge Conflict

GIT Command Line Cheat Sheet

git branch -a #List branches
git branch --delete BRANCH_NAME #Delete a branch
git checkout mybranch #switch to mybranch
git merge FromBranch #merge FromBranch to the current working branch

Introduction

This post is an introduction on how to resolve merge conflict. The following procedure will create a merge conflict

As shown in the following picture, the merge conflict is caused by that the same file was change at two different branches at different time. Typically, one develop check out the code from master branch, made some change and merged to the master branch. Another developer checkout the code after the previous developer and committed the code later.

Create Merge Conflicts

$ mkdir groovy
$ cd groovy
$ vi basic_one.groovy

Copy and paste the following code to the file of basic_one.grooy, and safe it (:wq)

println('Hello World!!!')
def myList = [1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()
newList.each { item -> print item + " " }
println()

Now execute the following commands

git init
git add .
git commit -am "first commit on master"

So far we have commit a file to our local repository. Next, we will create a new branch, namely, develop. We will make changes to the same file

Now execute the following commands

git branch develop    # create develop branch
git branch -a         # list branches, you should see the * master and develop. 
                      # The start * indicates we are working on master
git checkout develop  # switch to develop branch

git branch -a

Now, I see the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git branch -a
* develop
  master
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

You can see that the current active branch is develop. We will change the content of the file to the following:

println('Hello,  Gary Liu!')
def myList = [10, 13, 15, 16, 22, 33, 44, 55] 
println newList = myList.flatten(); 
newList.each { item -> print item + " " };
println();

Now execute the following commands to commit the develop branch:

git commit -am "First commit to develop branch"

In order to create conflicts, we will need to checkout the master branch and do some modification on the file

git checkout master.

If you execute: cat basic_one.groovy, you should see the following:

println('Hello World!!!')
def myList = [1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()
newList.each { item -> print item + " " }
println()

This is where we left off. In the meantime, develop branch has changed. Let's make some change to the file and it will be looks like the following:

println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33] 
println newList = myList.flatten()

newList.each {
   item -> print item + " " 
}
println()

Remember that we are at master branch now. We will merge from develop branch to the master

git add .
git commit -am "Second commit on master"
git merge develop

Now we see the following output:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git merge develop
Auto-merging basic_one.groovy
CONFLICT (content): Merge conflict in basic_one.groovy
Automatic merge failed; fix conflicts and then commit the result.
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

We have successfully created merge conflict. If you can the file of basic_one.groovy, it looks like the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ cat basic_one.groovy 
<<<<<<< HEAD
println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()

newList.each {
   item -> print item + " "
}
println()
=======
println('Hello,  Gary Liu!')
def myList = [10, 13, 15, 16, 22, 33, 44, 55]
println newList = myList.flatten(); 
newList.each { item -> print item + " " };
println();
>>>>>>> develop
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

Resolve Merge Conflicts

The above case is a bit extreme, as the whole file is almost changed. The HEAD section is from master branch and the lower part is from develop. To resolve the merge conflict, we need to edit file. For this case I will just take the master case, and the resultant file looks like the following:

Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ vi basic_one.groovy 
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git commit -am "Resolve merge conflict"
[master d2d5ecf] Resolve merge conflict
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ git merge develop
Already up-to-date.
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ cat basic_one.groovy 
println('Hello,  World Again!');
def myList = [-1, -2, 1, 3, 4, 5, [10, 13], 1, 2, 3, [15, 16], 22, 33]
println newList = myList.flatten()

newList.each {
   item -> print item + " "
}
println()
Gary2013@Guojiangs-MacBook-Pro:~/atom-dev/groovy$ 

To resolve merge conflict, we have to manually modify the file. We can use IDE like eclipse, sourceTree, etc, but we have to make decisions on which one is the best. After that we need to commit the changes and do a merge again to make sure that all the conflicts are addressed correctly.

Mule Dev Trick: Using Message Properties Transformer

The Abuse of set-variable

Many places I saw the code like the following:

Basically, developers set a bunch of flow variables for later usage. First of all, this kind of situation reflects design flaws. We should ask our selves why do we need to set so many flow variables? Can we use bean instead?

Use Message Property Transformer

Mule platform provide a very powerful transformer, namely message property transformer. Here is an example code:

The scope can be:

  invocation: flow variables
  outbound: property
  session: session variables

By using message property transformer, you can simplify the design, and make the flow easier to review.

Mule Dev Trick: Load A Text File And Convert To ArrayList

The Challenge

Here is the task. I have a file like the following:

AAAA
BBBB
CCCC
DDDD

I need to convert this content to an ArrayList, so that I can look through them. Actually each line is an object name. I need to query those objects from web services.

Solution

Here are the steps:

• Load file and payload as string. Mule provide a construct called "parse-template", which can be used to load file. This construct will load the file and convert the content to a string payload.
• Use Groovy or MEL to convert the string to an ArrayList

Here are the source code:

Key Learning Points

Few points we learn from this:

1. MEL support java directly. This is really very powerful.
2. Groovy is very powerful tool in Mule application development. collect {it.trim()} is similar as java stream function.