Update mySql plugin so that rewriteBatchedStatements is set to true
Description
In the Connection Arguments of Advanced Parameters for the MySQL plugin, reWriteBatchedStatements should be set to true.
While running pipelines which makes use of the mySQL plugin sink, it was noticed that writing 300k records takes around 3 hours without this argument, whereas with this argument included it takes around 7-8 minutes.
This may cause issues for users of the plugin, if this argument is not added to the default setting.
For reference, the stack overflow post which recommends this -
Release Notes
Improved performance for batch pipelines with MySQL sinks.
Activity
Show:
Ameya Khandekar October 26, 2022 at 5:47 PM
Cherry picks Merged, closing the issue.
Ameya Khandekar October 19, 2022 at 9:04 PM
Created Cherry Pick PRs for release/1.7 and release/1.8 (release/1.9 already has these changes) -
1.7 -
1.8 -
Ameya Khandekar September 16, 2022 at 4:22 PM
Edited
Will add cherry Pick PRs by Wednesday.
Ameya Khandekar September 9, 2022 at 7:02 AM
Committed to develop branch -
Vitalii Tymchyshyn September 7, 2022 at 5:07 PM
For a note, without this setting MySQL sink has a performance of 20-30 records per second written by a single executor. This makes it almost unusable for bigger datasets (1M+ records)
Unresolved
Pinned fields
Click on the next to a field label to start pinning.
In the Connection Arguments of Advanced Parameters for the MySQL plugin, reWriteBatchedStatements should be set to true.
While running pipelines which makes use of the mySQL plugin sink, it was noticed that writing 300k records takes around 3 hours without this argument, whereas with this argument included it takes around 7-8 minutes.
This may cause issues for users of the plugin, if this argument is not added to the default setting.
For reference, the stack overflow post which recommends this -