Same issue is being faced in 6.9.2 with a Snowflake plugin when using the query as a parameter. More context

The customer has recently updated his Snowflake plugin from version 1.1.1 to 1.1.2. Moreover, he did this to address an issue that he was experiencing when trying to use OAuth and after updating the plugin, his pipeline fails with the error [1].

Upon further investigation, it appears that the pipeline fails when the customer passes the source import query to the pipeline using a json file. If the customer hard code the query, the pipeline runs successfully.

I have investigated the logs provided and I was able to find other errors like [2] and [3], but the logs are not very informative about what can be the issue.

[1]:
Spark program 'phase-2' failed with error: Application application_1706017607571_0002 finished with failed status. Please check the system logs for more details.

[2]:
2024-01-23 13:51:07.474]Container exited with a non-zero exit code 1. Error file: prelaunch.err.
Last 4096 bytes of prelaunch.err :
Last 4096 bytes of stderr :

[3]:
2024-01-23 13:51:02,314 - ERROR [Driver:o.a.s.d.y.ApplicationMaster@94] - User class threw exception: java.lang.NullPointerException
java.lang.NullPointerException: null
at io.cdap.cdap.etl.spark.batch.SparkSQLEngine.getRelation(SparkSQLEngine.java:89)
at io.cdap.cdap.etl.spark.batch.BatchSQLEngineAdapter.lambda$tryRelationalTransform$4(BatchSQLEngineAdapter.java:799)
at java.util.stream.Collectors.lambda$toMap$58(Collectors.java:1321)
.. . . . .. . . . . . . .. . . . . . .. . . . . . . .. . . . . . . . . .. . . . . . . . .. .. .
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:732)

attached the config file that need to be put on a GCS bucket for GCS argument setter plugin to read config from and data file a.gz which should also be put on the GCS for GCS source plugin to read data from

attached two pipeline jsons, the major difference is the plugin version