You can assign compute profiles to batch pipelines in the following ways:
Assign a default profile for all of CDAP
Assign a default profile for a specific namespace.
Assign a profile to a batch pipeline to use for runs that are started manually.
Assign a profile to a pipeline schedule.
The profile that is used at runtime is determined by walking the CDAP hierarchy from most specific to most general.
If a profile is set on the schedule that triggered a run, that profile is used. If no profile is set, the default profile for the namespace is used. If no default is set on the namespace, the default for CDAP is used. If no CDAP default is set, the native profile is used.
Similarly, if a pipeline is manually run and there is a profile assigned to that pipeline, that profile is used. If no profile is set, the default profile for the namespace is used. If no default is set on the namespace, the default for CDAP is used. If no CDAP default is set, the native profile is used.
Note: If you have deployed your own custom applications and are using workflows, you can also use profiles to run your workflow in a cloud environment. Batch pipelines are just workflows with special UI support that makes profiles are easy to use.
Note: If you have deployed your own custom applications and are using workflows, you can also use profiles to run your workflow in a cloud environment. Batch pipelines are just workflows with special UI support that makes profiles are easy to use.
You can assign default profiles to a namespace and to the CDAP instance through the UI. Navigate to the Administration page and click the star next to the profile.
Alternatively, you can use the Preferences Microservices to set default profiles.
To set the default profile for all of CDAP, set a preference on the CDAP instance with key system.profile.name and value system:<profile-name>.
To set the default profile for a namespace, set a preference on the desired namespace with key system.profile.name and value <scope>:<profile-name>.
You can assign a profile to use for manual pipeline runs. Navigate to the pipeline detail page and click Configure > Compute config. Next, select a profile and click Save. The selected profile is used any time the pipeline is run manually.
Alternatively, you can use the Preferences Microservices to set the profile for manual runs by setting preference on the DataPipelineWorkflow entity with key system.profile.name and value <scope>:<profile-name>.
Any time you create a schedule for a pipeline, you can assign a profile to it. Whenever the schedule triggers a pipeline run, it will use that profile for the run. This is true for time schedules and schedules that other pipelines trigger.
Alternatively, you can use the Schedules Microservices to assign a profile to a schedule.
When a profile is created, each configuration settings can be made immutable by locking it. All other configuration settings can be overridden at runtime. To override profile configuration, follow these steps:
From the Pipeline List page, select the deployed pipeline you want to run.
From the Pipeline Details page, click Configure.
Locate the compute profile you want to use and click Customize.
Change any settings and click Save.
You can use runtime arguments and schedule properties to modify the size of the cluster or other important settings.
To override the profile used, set a runtime argument with key system.profile.name and value <scope>:<profile-name>.
To override a profile property, set a runtime argument with key system.profile.properties.<property-name> and value equal to the desired value for that property. For example, to override the numWorkerssetting to a value of 10, set a preference or runtime argument with key system.profile.properties.numWorkers and value 10.