If you use the StreamBatchSource with a format and schema, it does not work using the Spark engine.
It seems like the format spec from:
is not used in spark, but is in mapreduce.
As a result, we get an exception like:
Added support for FormatSpecification in Spark when consuming data from Stream
I think it's the same behavior in the old Spark runtime as well. It's a new feature we add to Spark in CDAP.