Expose committed offset in DeltaContext so Source can read directly.
Description
Currently,
In the source plugins we fetch the event, and tell debezium that it’s committed. This is not true as it’s not yet committed to BQ. In this case there can be a discrepancy between the understanding between SOURCE DB <> DEBEZIUM . Instead when there is a failure, we reset and start from an earlier offset ( even though we have told it;s committed) . Even though this would work for some sources, but some works might not function in this way. ( Example : Oracle xstream won't allow to restart from an earlier offset than the committed one)
So, we need to get the committed offset from the BQ side, and the publish the commitment to the debezium .
For this we need to expose something like getCommittedOffset in the delta context so the source can read it .
Release Notes
None
Activity
Show:
Sanket SahuMay 10, 2022 at 3:26 PM
PR :
Sebastian EchegarayMay 9, 2022 at 5:00 PM
Can you add a due date please?
Sanket SahuMay 7, 2022 at 7:38 AM
This is not a blocker, but an Internal Change. The change is small and doesn’t affect UX or any component as of now. But wanted to keep it in 6.7 so that future development of other plugins can utilise this change.
Prerna BellaraMay 6, 2022 at 5:43 PM
Can you please look into it to evaluate if its a blocker for 6.7?
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Currently,
In the source plugins we fetch the event, and tell debezium that it’s committed. This is not true as it’s not yet committed to BQ.
In this case there can be a discrepancy between the understanding between SOURCE DB <> DEBEZIUM .
Instead when there is a failure, we reset and start from an earlier offset ( even though we have told it;s committed) . Even though this would work for some sources, but some works might not function in this way. ( Example : Oracle xstream won't allow to restart from an earlier offset than the committed one)
So, we need to get the committed offset from the BQ side, and the publish the commitment to the debezium .
For this we need to expose something like
getCommittedOffsetin the delta context so the source can read it .