System Workers

Starting CDAP 6.7, CDAP introduces system worker pods which (if enabled) can be used for starting programs. This enables horizontal scalability of starting programs.

More specifically, instead of appfabric starting programs (e.g., pipelines), appfabric dispatches start requests to one of the system worker pods, and the system worker pod (which receives the request) is responsible for starting programs.

Note that a start program request/endpoint has not been changed with introduction of system workers.

Configurations

All following configurations start with system.worker.

Property	Default value	Description

Property	Default value	Description
pool.enable	false	If true, system worker pods will be created when appfabric starts.
container.count	2	Number of system workers for task execution.
cleanup.threads	1	Size of the thread pool for performing cleanup tasks.
container.num.cores	1	Number of virtual cores for the system worker.
container.memory.mb	400	Memory in megabytes for each system worker.
bind.address	0.0.0.0	The bind address for system worker service.
bind.port	11020	The port to bind the system worker service to.
request.limit	10	Number of requests accepted by a system worker pod.
dispatch.program.types	workflow	comma separated list of ProgramTypes to be used for dispatching to system workers. Missing program types will be launched by appfabric.
http.client.read.timeout.ms	300000	Read timeout in milliseconds for system worker HTTP requests.
http.client.connection.timeout.ms	60000	Connection timeout in milliseconds for system worker HTTP requests.