Macro can't be used by BQ source plugin as dataset or table name parameters.

Description

Even if output schema defined and ${bq.input.table} are set on system level. Preview run and deploying fail
...
Caused by: io.cdap.cdap.etl.validation.InvalidPipelineException: Error configuring stage 'BQ_source': null
at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:315) ~[na:na]
at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:207) ~[na:na]
at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStages(PipelineSpecGenerator.java:155) ~[na:na]
at io.cdap.cdap.etl.batch.BatchPipelineSpecGenerator.generateSpec(BatchPipelineSpecGenerator.java:56) ~[na:na]
at io.cdap.cdap.datapipeline.SmartWorkflow.configure(SmartWorkflow.java:175) ~[na:na]
... 32 common frames omitted
Caused by: java.lang.NullPointerException: null
at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:770) ~[com.google.guava.guava-13.0.1.jar:na]
at com.google.cloud.bigquery.TableId.of(TableId.java:86) ~[na:na]
at io.cdap.plugin.gcp.bigquery.util.BigQueryUtil.getBigQueryTable(BigQueryUtil.java:130) ~[na:na]
at io.cdap.plugin.gcp.bigquery.source.BigQuerySource.getSchema(BigQuerySource.java:180) ~[na:na]
at io.cdap.plugin.gcp.bigquery.source.BigQuerySource.configurePipeline(BigQuerySource.java:89) ~[na:na]
at io.cdap.cdap.etl.spec.PipelineSpecGenerator.configureStage(PipelineSpecGenerator.java:296) ~[na:na]
... 36 common frames omitted

Release Notes

Fixed a bug in the BigQuery source that caused pipelines to fail deploy if the dataset or table contained a macro.

Activity

Show:
Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Components

Fix versions

Priority

Created May 20, 2019 at 3:36 PM
Updated July 17, 2020 at 12:28 AM
Resolved May 20, 2019 at 7:49 PM