BQ Sink without input schema or table fails with NPE

Description

Steps to reproduce,

1. Create a simple BQ source -> BQ Sink pipeline
2. Configure source to have macros in place of dataset and table. This will result in no schema available during configure
3. Configure BQ sink with dataset and table. Make sure the table doesn't exists already in BQ.
4. Deploy and run the pipeline

After step 4 the pipeline fails with NPE

The reason this fails is because,
1. BQ sink relies on either input schema or the table schema to write data.
2. Since the BQ source is configured with macro it is not possible to generate schema for input source. So input schema is unavailable
3. Since the table doesn't exist in BQ yet it does not have table schema either.

The BQ sink used was 0.15.3 version of google-cloud.

Release Notes

None

Activity

Show:
Ron
3 days ago

PR:

Assignee

Ron

Reporter

Ajai Narayanan

Labels

None

Dev Complete Date

None

Publish Date

None

Reviewer

None

Components

Sprint

Priority

Major