julianNow: now() as String {format: "YYD"}, julianAt: "07-10-2020" as Date {format: "dd-MM-yyyy"} as String {format: "YYD"}
Tuesday, October 6, 2020
Dataweave 2.0: Convert Calendar Date To Julian Date
A Julian date is commonly used in main frame. It's format is YYDDD. DDD is the date since begining of the year.
In DetaWeave 2.0 the code should be as the following:
Tuesday, September 22, 2020
Mulesoft Send Email With CSV Attachment
Introduction
This blog explain the tricks to convert csv data to plain text for the purpuse of send csv as attachment. The Mulesoft document can be found at here1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | < file:read path = "/file.json" > < email:send config-ref = "config" > < email:to-addresses > < email:to-address value = "example@domain.com" > </ email:to-address ></ email:to-addresses > < email:body > < email:content > <!--[CDATA["<h1--> Hello this is an important message"]]></ email:content > </ email:body > < email:attachments > #[{ 'csv-attachment' : payload }] </ email:attachments > </ email:send > </ file:read > |
Code Details
As shown in the follow snapshot, we need to perform two transformations:- from csv to base64 binary
- from base64 binary to plain text>
Step 1: Initial code JSON to CSV
1 2 3 4 5 6 7 8 9 10 11 12 13 | %dw 2.0 output application/csv separator="," --- [ { name: "Gary Liu", ssn: "1234" }, { name: "John Smith", ssN: "4567" } ] |
Step 2: CSV to Base64 Binary
1 2 3 4 5 | %dw 2.0 import * from dw::core::Binaries output text/plain --- toBase64(write (payload, 'application/csv' )) |
Step 3: Base64 Binary to CSV Plain Text
1 2 3 4 5 | %dw 2.0 import fromBase64 from dw::core::Binaries output text/plain --- fromBase64(payload) |
Take Aways
The line of thinking is that Mule runtime engine internally keeps that data as array of records. This java object cannot direct convert to plain text. The code is really mimic the way we store the data to file and then read the data from file to plain text.Monday, August 10, 2020
Mule Develop Tricks: Get The Phone Number
The Requirement
The phone number from customer web site is free string. It can take the following forms:
- +1 (219) 555-8888
- +1 219-555-8888
- (219) 555 8888
- 219 555 8888
- 219 555-8888
- 219-555-8888
- -1-219-555-8888
There could be other form.
We need to use dataweave 2.0 to extract 10 digits of the phone number to: 2195558888.
Solution
%dw 2.0
output application/json
---
{
mobile: (payload.mobilePhone replace /([^0-9.]+)/ with(""))[-10 to -1]
}
Take Aways
- dataweave built in function replace / / with ()
- substring [-10 to -1] take last 10 digits
Saturday, February 15, 2020
Prototyping Mule OAuth2 Client Application
Introduction
This article explains how to prototype Mule OAuth 2.0 client application with grant type of "Client Credentials", which is most popular grant type for Mule integration. Most modern APIs enforce OAuth2.0 security policies. OAuth2.0 has the following Grant Types (The detailed explanation can be found here):- Authorization Code
- PKCE
- Client Credentials
- Device Code
- Refresh Token
- Legacy: Implicit Flow
- Legacy: Password Grant
- Using postman to retrieve access token
- Prototyping retrieve access token using cURL
- Mule flow to retrieve access token
- Sample flow using the access token with caching scope
OAuth 2.0 With Client Credential Grant Type
In order to access the OAuth2.0 enabled APIs, we first have to retrieve the access token from the Identity Providers. Then we can access the API by passing the access token. The parameters for Client Credential can be the following:- grant_type (required)
- scope (optional)
- client_id (required)
- client_secret (required)
1 | echo -n "${CLIENT_ID}:${CLIENT_SECRET}" | base64 |
1 | echo -n "${CLIENT_ID}:${CLIENT_SECRET}" | openssl enc -base64 |
Using Postman to Retrieve Access Token
The postman is the best tool to do prototyping for the OAuth 2 client. The following snapshot shows the setup of the Postman:for body: for Headers: for Authorization:
cURL Solution
Once we have the postman, the solution of cUrl is very straight forward.1 2 3 4 5 6 7 8 9 10 11 12 13 | $ cat oauth2-client.sh #!/bin/bash # CLIENT_ID=MY-CLIENT-ID-GOES-HERE-WITHOUT-QUOTE CLIENT_SECRET=YOUR-CLIENT-SECRET-GOES-HERE-WITHOUT-QUOTE OAUTH_HEADER=$( echo -n "${CLIENT_ID}:${CLIENT_SECRET}" | base64) -H "Content-Type: application/x-www-form-urlencoded" \ -H "Authorization: Basic ${OAUTH_HEADER}" \ -XPOST https: //login .microsoftonline.com /keurig .onmicrosoft.com /oauth2/v2 .0 /token |
Mule Application Solution - Retrieve access_token
The mule application flow for retrieving access token is the following: The Data-Weave transformation code is the following:1 2 3 4 5 6 7 | %dw 2.0 output application/x-www-form-urlencoded --- { grant_type: "client_credentials" , } |
1 2 3 4 5 6 7 | < http:request method = "POST" doc:name = "Request" doc:id = "65997138-84c0-48b3-8347-68abf386b3a1" config-ref = "HTTP_Request_configuration" path = "/keurig.onmicrosoft.com/oauth2/v2.0/token" > < http:headers > <!--[CDATA[#[output application/java --- { "Content-Type" : "application/x-www-form-urlencoded" }]]]--> </ http:headers > </ http:request > |
1 2 3 4 5 6 7 8 9 10 | < http:request-config name = "HTTP_Request_configuration" doc:name = "HTTP Request configuration" doc:id = "fb6e2c7d-010f-4141-bafe-a4a50c4fc540" > < http:request-connection protocol = "${oauth2.protocol}" host = "${oauth2.host}" port = "${oauth2.port}" > < reconnection > < reconnect frequency = "5000" > </ reconnect ></ reconnection > < http:authentication > < http:basic-authentication username = "${secure::oauth2.client.id}" password = "${secure::oauth2.client.secret}" > </ http:basic-authentication ></ http:authentication > </ http:request-connection > </ http:request-config > |
1 2 3 4 5 6 | { "token_type" : "Bearer" , "expires_in" : 3599, "ext_expires_in" : 3599, "access_token" : "eyJ0eXAi......" } |
Mule Application Solution - Use access_token
The following diagram shows the usage of the access_token. The access_token is passed to the server as header. As you can see, we need to use cache scope. This allow us to avoid calling the server if the token is not expired. In this case, the token will expire in one hour. Thus our object store TTL should be less then 60 minutes.Sunday, February 2, 2020
Anypoint Runtime Fabric : Part 1 - Pre-Installation Network Checking
Introduction
In the next few weeks, I will write more details about Mule runtime fabric installation, configuration, and troubleshooting. This article is about to verify the network connectivity before install Anypoint Runtime Fabric (RTF). In order to install Anypoint RTF, the network firewall must open. The details about the port and host whitelisting can be find here: https://docs.mulesoft.com/runtime-fabric/1.4/install-port-reqs.Tools for Network Connectivity Checking
The following tools should be installed and configure on the Linux system:- nslookup
- curl
- nc
- openssl
- chrony
1 | nslookup anypoint.mulesoft.com |
1 2 3 4 5 | Name: anypoint-prod-936665732.us-east-1.elb.amazonaws.com Address: 52.87.103.123 Name: anypoint-prod-936665732.us-east-1.elb.amazonaws.com Address: 3.217.25.79 ... |
1 2 3 4 5 | $ cat /etc/resolv.conf # Generated by NetworkManager search gmcr.com nameserver 10.64.2.13 nameserver 10.124.240.104 |
1 | $ curl -sk https: //anypoint .mulesoft.com:443 |
1 2 3 4 | < title >301 Moved Permanently</ title > < center >< h1 >301 Moved Permanently</ h1 ></ center > < hr >< center >nginx</ center > |
NTP - Network Time Protocol
In order for Anypoint RTF to work, we must make sure NTP is working in the system. Linux use chrony to perform NTP. Run the following command:1 | chronyc tracking |
1 2 3 4 5 6 7 8 9 10 11 12 13 | Reference ID : A3EDDA13 (toddadev.your.org) Stratum : 2 Ref time (UTC) : Mon Feb 03 00:18:55 2020 System time : 0.000235624 seconds slow of NTP time Last offset : -0.000094732 seconds RMS offset : 0.000081353 seconds Frequency : 16.987 ppm slow Residual freq : -0.003 ppm Skew : 0.062 ppm Root delay : 0.025243049 seconds Root dispersion : 0.000095447 seconds Update interval : 514.2 seconds Leap status : Normal |
1 2 3 4 5 6 7 8 | # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). #server 0.rhel.pool.ntp.org iburst #server 1.rhel.pool.ntp.org iburst #server 2.rhel.pool.ntp.org iburst #server 3.rhel.pool.ntp.org iburst server server 10.70.0.200 ... |
1 2 | sudo systemctl restart chronyd sudo systemctl enable chronyd |
1 | chronyc sources |
Network Connectivity Test Scripts
Mulesoft provided a scripts totest the network connectivity at here If you have gone through the test in the previous sections, this script should run with the following results. Once the above procedures are all done, we are read to perform the installation for Anypoint Runtime Fabric. I will cover that in the following articles.Anypoint Runtime Fabric - SSL Handshaking Troubleshooting
Introduction
Mulesoft's Anypoint Runtime Fabric is gaining momentum in the hybrid deployment model. Since version 1.4.1, the installation, configuration, and management have improved significantly. However, as it involves TLS/SSL, many things can go wrong. This article will provide some technical insights on how to diagnose those issues. In many cases, it may not be the configuration issues. Sometimes, the issues could be related to client code or network configuration. In later section, I have provided java code to test HTTPS REST API. First let's review the architecture of Anypoint Runtime Fabric. As shown from the above diagram, the TLS/SSL is applied to all the controllers. Let's say the IP addresses are:10.64.6.65
10.64.6.66
10.64.6.67
On all of the above controllers, the port 443 should be open. And we should be able to connect to all these controllers by using networking tools like nc, openssl, etc.
Verify Controller's TLS/SSL
First, from the client point of view, we need to make sure the port 443 is open and reachable. To do so, we can execute the following command:1 2 | $ nc -zv 10.64.6.65 443 Connection to 10.64.6.65 port 443 [tcp /https ] succeeded! |
1 | $ openssl s_client -connect 10.64.6.65:443 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 | $ openssl s_client -connect 10.64.6.65:443 CONNECTED(00000003) depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA verify return :1 depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA verify return :1 depth=0 C = US, ST = Texas, L = Plano, O = "Keurig Dr. Pepper, Inc." , CN = *.gmcr.com verify return :1 --- Certificate chain 0 s: /C =US /ST =Texas /L =Plano /O =Keurig Dr. Pepper, Inc. /CN =*.gmcr.com i: /C =US /O =DigiCert Inc /OU =www.digicert.com /CN =DigiCert SHA2 High Assurance Server CA 1 s: /C =US /O =DigiCert Inc /OU =www.digicert.com /CN =DigiCert SHA2 High Assurance Server CA i: /C =US /O =DigiCert Inc /OU =www.digicert.com /CN =DigiCert High Assurance EV Root CA --- Server certificate -----BEGIN CERTIFICATE----- ... -----END CERTIFICATE----- subject= /C =US /ST =Texas /L =Plano /O =Keurig Dr. Pepper, Inc. /CN =*.gmcr.com issuer= /C =US /O =DigiCert Inc /OU =www.digicert.com /CN =DigiCert SHA2 High Assurance Server CA --- No client certificate CA names sent Server Temp Key: ECDH, X25519, 253 bits --- SSL handshake has read 4087 bytes and written 289 bytes --- New, TLSv1 /SSLv3 , Cipher is ECDHE-RSA-AES128-GCM-SHA256 Server public key is 4096 bit Secure Renegotiation IS supported Compression: NONE Expansion: NONE No ALPN negotiated SSL-Session: Protocol : TLSv1.2 Cipher : ECDHE-RSA-AES128-GCM-SHA256 Session-ID: 820EF7CA565E40E495662B0D1AAAB24E2AE431B660A86DF7659D2DEFB7B986B1 Session-ID-ctx: Master-Key: 98F3D990B13A2847041E6275682C74F3D371A15C8D0358434957D6B87B865DA9E839AB95950FE6F24E9D5CDD0DE59F93 TLS session ticket lifetime hint: 7200 (seconds) TLS session ticket: ... Start Time: 1580667077 Timeout : 7200 (sec) Verify return code: 0 (ok) --- closed |
- CN = *.gmcr.com -- This dictate how client should invoke the service
- SHA2 High Assurance Server CA -- SHA2 is Secure Hashing Algorithm 2.
- Protocol : TLSv1.2 -- We are using TLSv1.2, now Anypoint RTF also support TLSv1.3
- Cipher : ECDHE-RSA-AES128-GCM-SHA256
- Verify return code: 0 (ok) -- This is very important.
1 | Verify return code: 18 (self signed certificate) |
Verify Server's Cipher Suites
It is not very uncommon that server and client could not find the common supported ciphers. To handle this, we need to scan server's cipher suite. I use two tools: nmap and cipherscan. There are many free software available, but I find they are very easy to use and very powerful. Wireshark is another very popular tool. If all of the simple tools are exhausted, we can use Wireshark. Here is example using cipherscan:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | $ . /cipherscan 10.64.6.65 Warning: target is not a FQDN. SNI was disabled. Use a FQDN or '-servername <fqdn>' ............... Target: 10.64.6.65:443 prio ciphersuite protocols pfs curves 1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1,secp384r1,secp521r1 2 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1,secp384r1,secp521r1 3 AES256-GCM-SHA384 TLSv1.2 None None 4 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,2048bits None 5 AES128-GCM-SHA256 TLSv1.2 None None Certificate: trusted, 4096 bits, sha256WithRSAEncryption signature TLS ticket lifetime hint: 7200 NPN protocols: None OCSP stapling: not supported Cipher ordering: server Curves ordering: server - fallback: no Server supports secure renegotiation Server supported compression methods: NONE TLS Tolerance: yes Intolerance to: SSL 3.254 : absent TLS 1.0 : PRESENT TLS 1.1 : PRESENT TLS 1.2 : absent TLS 1.3 : absent TLS 1.4 : absent < /fqdn > |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 | $ nmap -sV --script ssl-enum-ciphers -p 443 10.64.6.65 Starting Nmap 7.80 ( https: //nmap .org ) at 2020-02-02 13:11 CST Nmap scan report for hello-earth.kdrp.com (10.64.6.65) Host is up (0.086s latency). PORT STATE SERVICE VERSION 443 /tcp open ssl /https | fingerprint-strings: | FourOhFourRequest, GetRequest, HTTPOptions: | HTTP /1 .1 404 NOT FOUND | Content-Length: 0 | Connection: Close | RTSPRequest, SIPOptions: | HTTP /1 .1 400 BAD REQUEST - bad version | Content-Length: 0 |_ Connection: Close | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 4096) - A | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 4096) - A | compressors: | NULL | cipher preference: server | warnings: | Key exchange (dh 2048) of lower strength than certificate key | Key exchange (ecdh_x25519) of lower strength than certificate key |_ least strength: A 1 service unrecognized despite returning data. If you know the service /version , please submit the following fingerprint at https: //nmap .org /cgi-bin/submit .cgi?new-service : SF-Port443-TCP:V=7.80%T=SSL%I=7%D=2 /2 %Time=5E371EE9%P=x86_64-apple-darwin1 SF:7.7.0%r(GetRequest,40,"HTTP /1 \.1\x20404\x20NOT\x20FOUND\r\nContent-Leng SF:th:\x200\r\nConnection:\x20Close\r\n\r\n ")%r(HTTPOptions,40," HTTP /1 \.1\ SF:x20404\x20NOT\x20FOUND\r\nContent-Length:\x200\r\nConnection:\x20Close\ SF:r\n\r\n ")%r(FourOhFourRequest,40," HTTP /1 \.1\x20404\x20NOT\x20FOUND\r\nC SF:ontent-Length:\x200\r\nConnection:\x20Close\r\n\r\n")%r(RTSPRequest,50, SF:"HTTP /1 \.1\x20400\x20BAD\x20REQUEST\x20-\x20bad\x20version\r\nContent-L SF:ength:\x200\r\nConnection:\x20Close\r\n\r\n ")%r(SIPOptions,50," HTTP /1 \. SF:1\x20400\x20BAD\x20REQUEST\x20-\x20bad\x20version\r\nContent-Length:\x2 SF:00\r\nConnection:\x20Close\r\n\r\n"); Service detection performed. Please report any incorrect results at https: //nmap .org /submit/ . Nmap done : 1 IP address (1 host up) scanned in 46.51 seconds |
Verify REST API
From network point view, everything seems working, but client may still have problem to invoke the REST API exposed on the Anypoint RTF. In these cases, we need to check the following. Using postman, if the certificate applied to Anypoint RTF is self-signed, we need to make sure to turn off the ssl verification on the postman. Still some client are using plain java to invoke REST API. In this case, we can use the following code:1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | import org.apache.http.Header; import org.apache.http.HttpEntity; import org.apache.http.client.methods.CloseableHttpResponse; import org.apache.http.client.methods.HttpGet; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.apache.http.util.EntityUtils; public class HttpsHelloWorld { public static void main(String[] args) { // TODO Auto-generated method stub System.out.println( "Hello, World" ); request.addHeader( "Host" , "hello-earth-one.gmcr.com" ); CloseableHttpClient httpClient = HttpClients.createDefault(); try { CloseableHttpResponse response = httpClient.execute(request); // Get HttpResponse Status System.out.println(response.getStatusLine().toString()); HttpEntity entity = response.getEntity(); Header headers = entity.getContentType(); System.out.println(headers); if (entity != null ) { // return it as a String String result = EntityUtils.toString(entity); System.out.println(result); } } catch (Exception e) { e.printStackTrace(); } } } |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | < dependency > < groupid >org.apache.httpcomponents</ groupid > < artifactid >httpclient</ artifactid > < version >4.5.10</ version > </ dependency > < dependency > < groupid >org.springframework</ groupid > < artifactid >spring-core</ artifactid > < version >5.2.3.RELEASE</ version > </ dependency > < dependency > < groupid >org.springframework</ groupid > < artifactid >spring-web</ artifactid > < version >5.2.3.RELEASE</ version > </ dependency > |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 | import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.net.URI; import java.net.URL; import javax.net.ssl.HttpsURLConnection; import org.apache.http.conn.ssl.NoopHostnameVerifier; import org.apache.http.impl.client.CloseableHttpClient; import org.apache.http.impl.client.HttpClients; import org.springframework.http.HttpEntity; import org.springframework.http.HttpHeaders; import org.springframework.http.HttpMethod; import org.springframework.http.RequestEntity; import org.springframework.http.ResponseEntity; import org.springframework.http.client.HttpComponentsClientHttpRequestFactory; import org.springframework.util.LinkedMultiValueMap; import org.springframework.util.MultiValueMap; import org.springframework.web.client.HttpClientErrorException; import org.springframework.web.client.RestTemplate; public class HttpsPostMes { public static void main(String[] args) { try { httpsRequestCall3(); } catch (Exception e) { e.printStackTrace(); } } public static void httpsRequestCall3() { CloseableHttpClient httpClient = HttpClients.custom() .setSSLHostnameVerifier( new NoopHostnameVerifier()) .build(); HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(); requestFactory.setHttpClient(httpClient); HttpHeaders headers = new HttpHeaders(); headers.add( "Host" , "hello-earth-one.gmcr.com" ); HttpEntity<string> entity = new HttpEntity<>( "header" , headers); ResponseEntity<string> response = new RestTemplate(requestFactory).exchange(urlOverHttps, HttpMethod.GET, entity, String. class ); org.springframework.http.HttpStatus code = response.getStatusCode(); System.out.println(code); System.out.print(response); } } </string></string> |
Take Aways
In this article, I have covered the following:- Basic Anypoint runtime fabric's controllers
- Tools to verify servers ports, SSL protocols, and ciphers
- nc
- openssl
- nmap
- ciphjerscan
- java code to invoke REST API using https
Subscribe to:
Posts (Atom)
Anypoint Studio Error: The project is missing Munit lIbrary to run tests
Anypoint Studio 7.9 has a bug. Even if we following the article: https://help.mulesoft.com/s/article/The-project-is-missing-MUnit-libraries-...
-
Introduction MuleSoft has changed many stuff for the last year. One of the questions developer will ask is how to change the logging level....
-
Introduction The video for this blog is available at: http://www.youtube.com/watch?v=wVCmik-2xAM&feature=youtu.be Mule and JBoss ...
-
Congratulation Gary! First of all, I must congratulate myself for getting this done. For almost a year, I have been thinking to take the e...