Thread leak in replication pipelines due to DeltaWorker not closing source plugin's event reader in error scenarios
Description
In case of error in source plugin, DeltaWorker tries to stop the event reader The particular variant stop(StopContext context) has a default no-op implementation in the interface but none of the event readers in any source plugin override this variant , hence stop is a no-op leading to dormant threads which with keep polling the source.
Impact - e.g. in case of Oracle/Datastream replication, if there is a persistent network error due to unavailability of source DB, a lot EventReader instances are spawned each having a thread to poll Datastream service, which creates a huge load on the service (around 70 request per second).
Fixed an issue where the replication source plugin's event reader was not stopped by the Delta worker in case of errors, leading to leakage of thread leak
Activity
Show:
Sumit Jain February 20, 2023 at 5:13 AM
Cherry-picks
6.7.3
6.8.1
Sumit Jain February 20, 2023 at 5:03 AM
Develop:
Fixed
Pinned fields
Click on the next to a field label to start pinning.
In case of error in source plugin, DeltaWorker tries to stop the event reader The particular variant
stop(StopContext context)has a default no-op implementation in the interface but none of the event readers in any source plugin override this variant , hence stop is a no-op leading to dormant threads which with keep polling the source.Impact - e.g. in case of Oracle/Datastream replication, if there is a persistent network error due to unavailability of source DB, a lot EventReader instances are spawned each having a thread to poll Datastream service, which creates a huge load on the service (around 70 request per second).
PR where the
stop(StopContext context)was introduced https://github.com/data-integrations/delta/pull/135/files