Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Pipeline with bigquery sink fails with INVALID_ARGUMENT exception if the range specified is a macro

Description

Pipeline with bigquery sink fails with INVALID_ARGUMENT exception if the range specified is a macro. Steps to reproduce:
1. Import the pipeline attached
2. Deploy it
3. Run it

Release Notes

None

Attachments

1
  • 22 Dec 2020, 06:48 PM

Activity

Show:

Ardian Saidi March 1, 2021 at 10:40 AM

Ardian Saidi February 15, 2021 at 11:35 AM

Fixed:

1. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table does not exist 2. Throw validation issue in configuration stage when partition field is empty and partition type is Integer and destination table exist 3. Throw validation issue in prepare run stage when partition field or partition type is macro and partition field is empty

PR: https://github.com/data-integrations/google-cloud/pull/553

Integration test: https://github.com/cdapio/cdap-integration-tests/pull/1094

Vinisha Shah February 11, 2021 at 12:30 AM

I see. In the attached pipeline multiple fields are macro.

Please also create an integration test for this item

Ardian Saidi February 10, 2021 at 3:35 PM
Edited

We managed to replicate the issue. The issue is not a consequence of the range specified as macro. Instead, in the provided pipeline 'Partition Field' is not specified, which must be specified when the table does not exist.

We will add the validation for this specific scenario.

Logs for this test:

Spark program 'phase-1' failed with error: 400 Bad Request POST https://bigquery.googleapis.com/bigquery/v2/projects/***/jobs { "code" : 400, "errors" : [ { "domain" : "global", "message" : "Field must be specified for range partition.", "reason" : "invalid" } ], "message" : "Field must be specified for range partition.", "status" : "INVALID_ARGUMENT" }. Please check the system logs for more details.

Bajram Bojku February 9, 2021 at 7:18 PM

try and reproduce since this may be linked to a Jira item that Vinisha has closed already. Add results here once tested

Unresolved
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Original estimate

Time tracking

No time logged2w remaining

Components

Sprint

Fix versions

Affects versions

Priority

Created December 22, 2020 at 6:49 PM
Updated March 1, 2021 at 11:21 PM
Configure