When programs are run in a remote cluster, the runtime monitor sets up SSH port forwarding information on the remote box by scp-ing a file. While doing the scp, the runtime monitor thread can hang waiting to read scp response. A single hanging runtime monitor thread will block all other runtime monitors due to a shared lock, leading to no new programs getting executed.

I have attached a stack trace showing this deadlock.