The pipelines fail with the following exception in Spark 2.2 -
Fixed an issue where Spark 2.2 batch pipelines with HDFS sinks would fail with delegation token issue error
This is because in Spark 2, rdd.saveAsNewAPIHadoopDataset(conf) expects the conf to be a JobConf object, and to contain the credentials. If a regular configuration object is passed, then Spark creates a JobConf object out of it. And when this happens, the credentials in the JobConf object would be set to null. Hence the error.