Stopping of preview run on branch 6.2 is failing. /stop REST api does not return. I am also seeing multiple runner pods getting terminated as a result of stop. Ideally it should only terminate the one which is running preview.
Looks like when the exception is thrown while stopping such as
Controller is terminated
This results in re-creation of preview runner statefulset and hence all the runner pods are restarted.
The conflict error was due to race condition on stopping pod and deletion of statefulset due to CDAP timeout waiting for all instances up and running, which is expected.
However, there is another problem discovered in preview stop that it was not shutting down gracefully.