We're updating the issue view to help you get more done. 

AutoJoin should be case sensitive when selecting fields

Description

If one of the stages going into a join has two columns that are the same but with different casing, the pipeline will fail with an error of the form:

This particular example happened when I had a stage that had 'region' and 'REGION' as field names. This is a backwards incompatible change from before, where they were treated as different fields without a problem.

I think we just need to set 'spark.sql.caseSensitive' to true.

Release Notes

None

Activity

Show:

Assignee

Albert Shau

Reporter

Albert Shau

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Major
Configure