BQ Sink without input schema or table fails with NPE
Description
Steps to reproduce,
1. Create a simple BQ source -> BQ Sink pipeline
2. Configure source to have macros in place of dataset and table. This will result in no schema available during configure
3. Configure BQ sink with dataset and table. Make sure the table doesn't exists already in BQ.
4. Deploy and run the pipeline
After step 4 the pipeline fails with NPE
The reason this fails is because,
1. BQ sink relies on either input schema or the table schema to write data.
2. Since the BQ source is configured with macro it is not possible to generate schema for input source. So input schema is unavailable
3. Since the table doesn't exist in BQ yet it does not have table schema either.
The BQ sink used was 0.15.3 version of google-cloud.
Release Notes
None
Activity
Show:
Ron
3 days ago
PR:
Assignee
Reporter
Labels
None
Dev Complete Date
None
Publish Date
None