Runtime Monitor fail with SSLHandshakeException

Description

In remote execution mode (when non-native compute profile is used), program run occasionally stuck in "Starting" phase, even the actual program execution already completed in the target cluster.

It is caused by the RuntimeMonitor not able to fetch metadata from the RemoteRuntimeServer. The underlying failure is caused by SSLHandshakeException, which unfortunately didn't get log due to CDAP-15505.

The reason of the SSL exception is because occasionally the remote runtime server binds to a port that was already used by the NFS kernel server. Running the netstat command confirming it:

This can happen because NFS binds with the IPV6_V6ONLY socket option. When the remote runtime monitor server starts up, the OS is allowed to give it the same port for binding to the IPv4 127.0.0.1 address. When this happen, HTTPS calls to "localhost:port" will be handled by the NFS server, hence resulting in connection close by the server, hence the SSL handshake error.

I first search for why there is no process ID from the netstat output and saw a lot of references referring to NFS kernel server. After I shutdown the NFS kernel server, the RuntimeMonitor was able to fetch metadata and proceed.

Ref: https://unix.stackexchange.com/questions/97752/how-to-identify-a-process-which-has-no-pid
Ref2: https://unix.stackexchange.com/questions/536634/linux-gives-an-ephemeral-port-that-is-already-used-and-bind-on-any-interface

Release Notes

None

Activity

Show:
Terence Yim
August 21, 2019, 5:19 PM
Terence Yim
August 21, 2019, 9:39 PM
Fixed

Assignee

Terence Yim

Reporter

Terence Yim

Labels

None

Docs Impact

None

UX Impact

None

Fix versions

Priority

Blocker
Configure