If one of the stages going into a join has two columns that are the same but with different casing, the pipeline will fail with an error of the form:
This particular example happened when I had a stage that had 'region' and 'REGION' as field names. This is a backwards incompatible change from before, where they were treated as different fields without a problem.
I think we just need to set 'spark.sql.caseSensitive' to true.