Starting CDAP 6.7, CDAP introduces system worker pods which (if enabled) can be used for starting programs. This enables horizontal scalability of starting programs.
More specifically, instead of appfabric starting programs (e.g., pipelines), appfabric dispatches start requests to one of the system worker pods, and the system worker pod (which receives the request) is responsible for starting programs.
Note that a start program request/endpoint has not been changed with introduction of system workers.
Configurations
All following configurations start with system.worker.
Property | Default value | Description |
|---|---|---|
pool.enable | false | If true, system worker pods will be created when appfabric starts |
container.count | 2 | Number of system workers for task execution |
cleanup.threads | 1 | Size of the thread pool for performing cleanup tasks |
container.num.cores | 1 | Number of virtual cores for the system worker |
container.memory.mb | 400 | Memory in megabytes for each system worker |
bind.address | 0.0.0.0 | The bind address for system worker service. |
bind.port | 11020 | The port to bind the system worker service to. |
request.limit | 10 | Number of requests accepted by a system worker pod. |
dispatch.program.types | workflow | comma separated list of ProgramTypes to be used for dispatching to system workers. Missing program types will be launched by appfabric. |
http.client.read.timeout.ms | 300000 | Read timeout in milliseconds for system worker HTTP requests. |
http.client.connection.timeout.ms | 60000 | Connection timeout in milliseconds for system worker HTTP requests. |