We're updating the issue view to help you get more done. 

Don't generate sequence number for DDL events

Description

sequence number is used for :

Avoid duplicated – events in some case a batch of events were merged to target table while offset was not committed, when restarting from last offset, we only merge those events that have sequence number greater than the largest sequence num we've ever seen in target table.

So sequence number should be consistent for all DML events even the source or target restarts from last committed offset.

And the current implementation assign sequence number to both DML and DDL events in incremental way. While some source plugin may generate different amount DDL events (eventual consistency) if restarting from last committed offset. which will cause inconsistent sequence number assign to DML.

So we don't need to assign sequence number to DDL events or assign a sequence number that is independent of DML event.

The impact of not adding this feature is :

if there are some DDL changes between the last committed Offset and current time and replicator restarts from last committed Offset, users may lose DML events or have duplicated DML events applied on the target table.

Release Notes

None

Activity

Show:
Sean Zhou
6 days ago

Discussed with Sagar, it’s kind of corner case.

Bhooshan, need your confirmation that we will do it for 6.4.0 instead of 6.3.0

Assignee

Bhooshan Mogal

Reporter

Sean Zhou

Labels

None

Docs Impact

None

UX Impact

None

Affects versions

Priority

Major
Configure