We're updating the issue view to help you get more done. 

The data pipeline app is not setting the partitions correctly for aggregators

Description

The AbstractSparkPreparer is not setting the partitions for both the old and new aggregators, which makes the aggregator always use the same number of partitions as its previous stages. 

Release Notes

Fixed a bug that caused the number of partitions set by aggregator plugins to be ignored.

Activity

Show:
Yaojie Feng
November 16, 2020, 6:57 PM

As a workaround, people can set spark.default.parallelism to the desired partitions in engine config for this to take effect

Albert Shau
3 days ago

Fixed

Assignee

Albert Shau

Reporter

Yaojie Feng

Labels

None

Docs Impact

None

UX Impact

None

Fix versions

Priority

Major
Configure