Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Pipeline with bigquery sink fails with INVALID_ARGUMENT exception if the range specified is a macro

Description

Pipeline with bigquery sink fails with INVALID_ARGUMENT exception if the range specified is a macro. Steps to reproduce:
1. Import the pipeline attached
2. Deploy it
3. Run it

Release Notes

None

Attachments

1

Activity

Show:

Ardian Saidi March 1, 2021 at 10:40 AM

Cherry picks:

0.14 -

0.15 -

0.16 -

Ardian Saidi February 15, 2021 at 11:35 AM

Fixed:

PR:

Integration test:

Vinisha Shah February 11, 2021 at 12:30 AM

I see. In the attached pipeline multiple fields are macro.

Please also create an integration test for this item

Ardian Saidi February 10, 2021 at 3:35 PM
Edited

We managed to replicate the issue. The issue is not a consequence of the range specified as macro. Instead, in the provided pipeline 'Partition Field' is not specified, which must be specified when the table does not exist.

We will add the validation for this specific scenario.

Logs for this test:

Spark program 'phase-1' failed with error: 400 Bad Request POST https://bigquery.googleapis.com/bigquery/v2/projects/***/jobs { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Field must be specified for range partition.", "reason" : "invalid" } ], "message" : "Field must be specified for range partition.", "status" : "INVALID_ARGUMENT" }. Please check the system logs for more details.

Bajram Bojku February 9, 2021 at 7:18 PM

try and reproduce since this may be linked to a Jira item that Vinisha has closed already. Add results here once tested

Unresolved

Details

Assignee

Reporter

Original estimate

Time tracking

No time logged2w remaining

Components

Sprint

Fix versions

Affects versions

Priority

Created December 22, 2020 at 6:49 PM
Updated March 1, 2021 at 11:21 PM
Configure
Loading...