Replication Microservices
- Robin Rielley
- robin
Use the CDAP Replication Microservices to manage replication jobs.
All methods or endpoints described in this API have a base URL (typically http://<host>:11015 or https://<host>:10443) that precedes the resource identifier, as described in the Microservices Conventions. These methods return a status code, as listed in the Microservices Status Codes.
Assess a Draft Replication Job
To assess a replication job in draft mode based on the source and target tables, submit an HTTP POST request:
POST /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>/assessPipelineParameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
Assess a Table in a Draft Replication Job
To assess a specific table in a replication job in draft mode, submit an HTTP POST request:
POST /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>/assessPipelineParameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
The request body is a JSON object specifying the database and table name in the draft replication job.
For example:
{
"database": "db1",
"table": "table1"
}Create or Update a Draft Replication Job
To add a new replication job in draft mode or overwrite an existing one, submit an HTTP PUT request:
PUT /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>Parameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
The request body is a JSON object specifying the draft replication job configuration.
For example:
{
"name": "<app_name>",
"artifact": {
"name": "delta-app",
"version": "0.6.1",
"scope": "SYSTEM"
},
"config": {
"tables": [
{
"database": "testreplication",
"table": "persons",
"schema": "dbo"
}
],
"stages": [
{
"name": "Microsoft SQLServer",
"plugin": {
"name": "sqlserver",
"type": "cdcSource",
"properties": {
"port": "1433",
"serverTimezone": "UTC",
"replicateExistingData": "true",
"user": "<>",
"password": "<>",
"host": "1.2.3.4",
"jdbcPluginName": "sqlserver42",
"database": "testreplication"
},
"artifact": {
"name": "sqlserver-delta-plugins",
"version": "0.6.1",
"scope": "SYSTEM"
}
}
},
{
"name": "BigQuery",
"plugin": {
"name": "bigquery",
"type": "cdcTarget",
"properties": {
"project": "auto-detect",
"serviceAccountKey": "auto-detect",
"stagingBucketLocation": "us",
"loadInterval": "90",
"stagingTablePrefix": "staging",
"requireManualDrops": "false",
"datasetName": "testreplication1"
},
"artifact": {
"name": "bigquery-delta-plugins",
"version": "0.6.1",
"scope": "SYSTEM"
}
}
}
],
"tableTransformations": [],
"parallelism": {
"numInstances": 1
},
"description": "<>"
}
}Delete a Draft Replication Job
To delete an existing replication job in draft mode, submit an HTTP DELETE request:
DELETE /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>Parameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
Details of a Source Table in a Replication Job Draft
To list the details of a source table in replication job draft, submit an HTTP POST request:
POST /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>/describeTableParameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
The request body is a JSON object specifying the database and table name in the draft replication job.
For example:
{
"database": "db1",
"table": "table1"
}Details of a Replication Job Draft
To list the details of a replication job draft, submit an HTTP GET request:
GET /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>Parameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
Draft Replication Jobs
To list all replication jobs in Draft mode in a namespace, submit an HTTP GET request:
GET /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/draftsParameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
The information will be returned in the body of the response. It list the draft objects, where a draft contains the full job configuration. For example:
[
{
"name": "Draft1",
"created": timestamp
"config": {
"stages": [
{
"name": "source",
"plugin": {
"name": "oracle",
"type": "cdcSource",
"properties": {
"host": "[hostname]",
"port": "1433",
"user": "repluser",
"password": "${secure(kms-key)}"
}
}
},
...
]
},
}
]Generation Number for the Replication Job (CDAP 6.7+)
To get the generation number for a replication job, submit an HTTP GET request:
GET /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/{namespace}/apps/{app_name}/maxgenerationParameter | Description |
|---|---|
| The namespace for your application. |
| The name of the replication job. |
Offset Information for each Task Instance (CDAP 6.7+)
The offset is maintained at for every task instance. When you configure a replication job, there is an option to set the number of tasks and each task has its own offset information.
To get the offset information for each task instance for a replication job, submit an HTTP GET request:
GET /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/{namespace}/apps/{app_name}/generations/{generation_num}/instances/{instance_id}/offsetParameter | Description |
|---|---|
| The namespace for the deployed replication job. |
| The name of the deployed replication job. |
| The generation number for the deployed replication job. It’s generally a LONG type value. |
| The ID of the task. |
Replication Tables Mapped to each Task / Instance_id (CDAP 6.7+)
Each Task/instance handles a particular set of tables. It is responsible to snapshot and replication process for those tables.
To list the replication tables mapped to an Instance_id for a deployed replication job, submit an HTTP GET request:
GET v3/namespaces/system/apps/{app_name}/workers/DeltaWorkerParameter | Description |
|---|---|
| The name of the deployed replication job. |
In the Response body, look for properties . table.assignments.
Source Tables for a Draft Replication Job
To list the source tables in a replication job in draft mode, submit an HTTP POST request:
POST /v3/namespaces/system/apps/delta/services/assessor/methods/v1/contexts/<namespace>/drafts/<draft-id>/listTablesParameter | Description |
|---|---|
| The namespace for the draft replication jobs. |
| The Draft ID of the replication job. The Draft ID can be any alphanumeric string. It must be unique in the namespace. |
Related content
Created in 2020 by Google Inc.