System Workers

Starting CDAP 6.7, CDAP introduces system worker pods which (if enabled) can be used for starting programs. This enables horizontal scalability of starting programs.

More specifically, instead of appfabric starting programs (e.g., pipelines), appfabric dispatches start requests to one of the system worker pods, and the system worker pod (which receives the request) is responsible for starting programs.

Note that a start program request/endpoint has not been changed with introduction of system workers.

Configurations

All following configurations start with system.worker.

Property

Default value

Description

Property

Default value

Description

pool.enable

false

If true, system worker pods will be created when appfabric starts.

 

container.count

2

Number of system workers for task execution.

cleanup.threads

1

Size of the thread pool for performing cleanup tasks.

container.num.cores

1

Number of virtual cores for the system worker.

container.memory.mb

400

Memory in megabytes for each system worker.

bind.address

0.0.0.0

The bind address for system worker service.

bind.port

11020

The port to bind the system worker service to.

request.limit

10

Number of requests accepted by a system worker pod.

dispatch.program.types

workflow

comma separated list of ProgramTypes to be used for dispatching to system workers. Missing program types will be launched by appfabric.

http.client.read.timeout.ms

300000

Read timeout in milliseconds for system worker HTTP requests.

http.client.connection.timeout.ms

60000

Connection timeout in milliseconds for system worker HTTP requests.

Created in 2020 by Google Inc.