Versions Compared

Old Version 9

changes.mady.by.user Robin Rielley

Saved on

compared with

New Version 10

changes.mady.by.user Robin Rielley

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The final target tables will include all the original columns from the source table plus one additional _sequence_num column. The sequence number is used to ensure that data is not duplicated or missed in replicator failure scenarios.

Retaining Staging Tables

You can add a system preference to retain the staging tables in BigQuery to help you debug issues. For example, if there’s an issue with the Replication job, you can run a diff on the staging table to help you identify the issue.

To retain staging tables in BigQuery, follow these steps:

  1. Click System Admin > Configuration.

  2. Click Edit System Preferences.

  3. In the Key field, enter retain.staging.table.

  4. In the Value field, enter true.

    Image Removed

  5. Click Save & Close.

Now, when you run a Replication job, CDAP retains staging tables in BigQuery.

Credentials

If the plugin is run on a Google Cloud Dataproc cluster, the service account key does not need to be provided and can be set to 'auto-detect'. Credentials will be automatically read from the cluster environment.

...