Introduction
Mulesoft's Anypoint Runtime Fabric is gaining momentum in the hybrid deployment model. Since version 1.4.1, the installation, configuration, and management have improved significantly. However, as it involves TLS/SSL, many things can go wrong. This article will provide some technical insights on how to diagnose those issues. In many cases, it may not be the configuration issues. Sometimes, the issues could be related to client code or network configuration. In later section, I have provided java code to test HTTPS REST API. First let's review the architecture of Anypoint Runtime Fabric.
10.64.6.65
10.64.6.66
10.64.6.67
On all of the above controllers, the port 443 should be open. And we should be able to connect to all these controllers by using networking tools like nc, openssl, etc.
Verify Controller's TLS/SSL
First, from the client point of view, we need to make sure the port 443 is open and reachable. To do so, we can execute the following command:$ nc -zv 10.64.6.65 443 Connection to 10.64.6.65 port 443 [tcp/https] succeeded!As you can see, we can connect the port 443 successfully. nc is a very light and powerful tool for quickly scan the port of the server. Secondly, we can use openssl to verify the TLS/SSL:
$ openssl s_client -connect 10.64.6.65:443openssl is a heavy weight tool. It can do a lot things, such as, connection verification, ssl certification generation and conversion, etc. The above command will print out information about the server's TLS/SSL certificates, ssh handshaking, cipher, etc. Here are an example:
$ openssl s_client -connect 10.64.6.65:443
CONNECTED(00000003)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 High Assurance Server CA
verify return:1
depth=0 C = US, ST = Texas, L = Plano, O = "Keurig Dr. Pepper, Inc.", CN = *.gmcr.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Texas/L=Plano/O=Keurig Dr. Pepper, Inc./CN=*.gmcr.com
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert High Assurance EV Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
subject=/C=US/ST=Texas/L=Plano/O=Keurig Dr. Pepper, Inc./CN=*.gmcr.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 4087 bytes and written 289 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: 820EF7CA565E40E495662B0D1AAAB24E2AE431B660A86DF7659D2DEFB7B986B1
Session-ID-ctx:
Master-Key: 98F3D990B13A2847041E6275682C74F3D371A15C8D0358434957D6B87B865DA9E839AB95950FE6F24E9D5CDD0DE59F93
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
...
Start Time: 1580667077
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
closed
The most important information client should notice are the following:
- CN = *.gmcr.com -- This dictate how client should invoke the service
- SHA2 High Assurance Server CA -- SHA2 is Secure Hashing Algorithm 2.
- Protocol : TLSv1.2 -- We are using TLSv1.2, now Anypoint RTF also support TLSv1.3
- Cipher : ECDHE-RSA-AES128-GCM-SHA256
- Verify return code: 0 (ok) -- This is very important.
Verify return code: 18 (self signed certificate)
Verify Server's Cipher Suites
It is not very uncommon that server and client could not find the common supported ciphers. To handle this, we need to scan server's cipher suite. I use two tools: nmap and cipherscan. There are many free software available, but I find they are very easy to use and very powerful. Wireshark is another very popular tool. If all of the simple tools are exhausted, we can use Wireshark. Here is example using cipherscan:$ ./cipherscan 10.64.6.65 Warning: target is not a FQDN. SNI was disabled. Use a FQDN or '-servernameAs you can see, it provide supported cipher and TLS protocols. You can down cipherscan from github: https://github.com/mozilla/cipherscan. nmap is another very powerful and easy to use tool. nmap is a bit slow.' ............... Target: 10.64.6.65:443 prio ciphersuite protocols pfs curves 1 ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 ECDH,P-256,256bits prime256v1,secp384r1,secp521r1 2 ECDHE-RSA-AES256-GCM-SHA384 TLSv1.2 ECDH,P-256,256bits prime256v1,secp384r1,secp521r1 3 AES256-GCM-SHA384 TLSv1.2 None None 4 DHE-RSA-AES128-GCM-SHA256 TLSv1.2 DH,2048bits None 5 AES128-GCM-SHA256 TLSv1.2 None None Certificate: trusted, 4096 bits, sha256WithRSAEncryption signature TLS ticket lifetime hint: 7200 NPN protocols: None OCSP stapling: not supported Cipher ordering: server Curves ordering: server - fallback: no Server supports secure renegotiation Server supported compression methods: NONE TLS Tolerance: yes Intolerance to: SSL 3.254 : absent TLS 1.0 : PRESENT TLS 1.1 : PRESENT TLS 1.2 : absent TLS 1.3 : absent TLS 1.4 : absent
$ nmap -sV --script ssl-enum-ciphers -p 443 10.64.6.65 Starting Nmap 7.80 ( https://nmap.org ) at 2020-02-02 13:11 CST Nmap scan report for hello-earth.kdrp.com (10.64.6.65) Host is up (0.086s latency). PORT STATE SERVICE VERSION 443/tcp open ssl/https | fingerprint-strings: | FourOhFourRequest, GetRequest, HTTPOptions: | HTTP/1.1 404 NOT FOUND | Content-Length: 0 | Connection: Close | RTSPRequest, SIPOptions: | HTTP/1.1 400 BAD REQUEST - bad version | Content-Length: 0 |_ Connection: Close | ssl-enum-ciphers: | TLSv1.2: | ciphers: | TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (ecdh_x25519) - A | TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (ecdh_x25519) - A | TLS_RSA_WITH_AES_256_GCM_SHA384 (rsa 4096) - A | TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 (dh 2048) - A | TLS_RSA_WITH_AES_128_GCM_SHA256 (rsa 4096) - A | compressors: | NULL | cipher preference: server | warnings: | Key exchange (dh 2048) of lower strength than certificate key | Key exchange (ecdh_x25519) of lower strength than certificate key |_ least strength: A 1 service unrecognized despite returning data. If you know the service/version, please submit the following fingerprint at https://nmap.org/cgi-bin/submit.cgi?new-service : SF-Port443-TCP:V=7.80%T=SSL%I=7%D=2/2%Time=5E371EE9%P=x86_64-apple-darwin1 SF:7.7.0%r(GetRequest,40,"HTTP/1\.1\x20404\x20NOT\x20FOUND\r\nContent-Leng SF:th:\x200\r\nConnection:\x20Close\r\n\r\n")%r(HTTPOptions,40,"HTTP/1\.1\ SF:x20404\x20NOT\x20FOUND\r\nContent-Length:\x200\r\nConnection:\x20Close\ SF:r\n\r\n")%r(FourOhFourRequest,40,"HTTP/1\.1\x20404\x20NOT\x20FOUND\r\nC SF:ontent-Length:\x200\r\nConnection:\x20Close\r\n\r\n")%r(RTSPRequest,50, SF:"HTTP/1\.1\x20400\x20BAD\x20REQUEST\x20-\x20bad\x20version\r\nContent-L SF:ength:\x200\r\nConnection:\x20Close\r\n\r\n")%r(SIPOptions,50,"HTTP/1\. SF:1\x20400\x20BAD\x20REQUEST\x20-\x20bad\x20version\r\nContent-Length:\x2 SF:00\r\nConnection:\x20Close\r\n\r\n"); Service detection performed. Please report any incorrect results at https://nmap.org/submit/ . Nmap done: 1 IP address (1 host up) scanned in 46.51 seconds
Verify REST API
From network point view, everything seems working, but client may still have problem to invoke the REST API exposed on the Anypoint RTF. In these cases, we need to check the following. Using postman, if the certificate applied to Anypoint RTF is self-signed, we need to make sure to turn off the ssl verification on the postman.
import org.apache.http.Header;
import org.apache.http.HttpEntity;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.util.EntityUtils;
public class HttpsHelloWorld {
public static void main(String[] args) {
// TODO Auto-generated method stub
System.out.println("Hello, World");
HttpGet request = new HttpGet("https://kknxplvmsftct.gmcr.com/earth/hello");
request.addHeader("Host", "hello-earth-one.gmcr.com");
CloseableHttpClient httpClient = HttpClients.createDefault();
try {
CloseableHttpResponse response = httpClient.execute(request);
// Get HttpResponse Status
System.out.println(response.getStatusLine().toString());
HttpEntity entity = response.getEntity();
Header headers = entity.getContentType();
System.out.println(headers);
if (entity != null) {
// return it as a String
String result = EntityUtils.toString(entity);
System.out.println(result);
}
} catch (Exception e) {
e.printStackTrace();
}
}
}
add the following dependencies to pom.xml
Or if you prefer to use REST template, you can use the following example:org.apache.httpcomponents httpclient 4.5.10 org.springframework spring-core 5.2.3.RELEASE org.springframework spring-web 5.2.3.RELEASE
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.URI;
import java.net.URL;
import javax.net.ssl.HttpsURLConnection;
import org.apache.http.conn.ssl.NoopHostnameVerifier;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.HttpMethod;
import org.springframework.http.RequestEntity;
import org.springframework.http.ResponseEntity;
import org.springframework.http.client.HttpComponentsClientHttpRequestFactory;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.RestTemplate;
public class HttpsPostMes {
public static void main(String[] args) {
try {
httpsRequestCall3();
} catch(Exception e) {
e.printStackTrace();
}
}
public static void httpsRequestCall3() {
CloseableHttpClient httpClient
= HttpClients.custom()
.setSSLHostnameVerifier(new NoopHostnameVerifier())
.build();
HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory();
requestFactory.setHttpClient(httpClient);
String urlOverHttps = "https://kknxplvmsftct.gmcr.com/earth/hello";
HttpHeaders headers = new HttpHeaders();
headers.add("Host", "hello-earth-one.gmcr.com");
HttpEntity entity = new HttpEntity<>("header", headers);
ResponseEntity response = new RestTemplate(requestFactory).exchange(urlOverHttps, HttpMethod.GET, entity, String.class);
org.springframework.http.HttpStatus code = response.getStatusCode();
System.out.println(code);
System.out.print(response);
}
}
Take Aways
In this article, I have covered the following:- Basic Anypoint runtime fabric's controllers
- Tools to verify servers ports, SSL protocols, and ciphers
- nc
- openssl
- nmap
- ciphjerscan
- java code to invoke REST API using https
