...
For instance, you might want a pipeline that reads from a database (a source) and writes to a table file (a sink). The name of the database source and name of the table file sink might change from run to run and you need to specify those values as input before starting a run.
You might want to create a pipeline with a particular action at the start of the run. The action could, based on some logic, provide the name of the database to use as a source and the name of the table file to write as a sink. The next stage in the pipeline might use this information to read and write from appropriate sources and sinks.
...
These arguments and preferences can be set with the Pipeline Studio, the CDAP CLI, or the Program Lifecycle and Preferences Microservicesthe Lifecycle Microservices, or Preferences Microservices.
For more information, see Setting Preferences and Using Macros to Create a Dynamic Data Pipeline.
...
Logs for a pipeline can be obtained using the same tools as any other CDAP application and program, and are described in Logging and Monitoring and the Logging Microservices.
Script transform steps can write to logs, as described in the section in developing plugins on script transformations.
Metrics
As pipelines run, they create both system and user metrics.
System metrics for a pipeline can be obtained using the same tools as any other CDAP application and program, and are described in Metrics and Metrics Microservices.
Script transform steps can create metrics, as described in the section in developing plugins on script transformations.
For instance, if you have a real-time pipeline named "demoPipeline" with three stages (File, JavaScript, and Table), you can discover the available metrics using a curl command, such as (reformatted for display):
...
Resources for pipelines can be configured as any other CDAP workflow application.
See Scaling Instances and Resource Guarantees for CDAP Programs in YARN.