Creating a simple batch data pipeline

To create a simple batch data pipeline, follow these steps:

  1. Open the Pipeline Studio.

  2. Choose Data Pipeline - Batch.

     

  3. Name the data pipeline and add a description. The name cannot include spaces.
    For more information about supported characters, see Naming Conventions.

     

  4. In the plugin palette, click the sources needed in the pipeline.

     

  5. To configure each source, hover your mouse over the source plugin. The Properties button will be displayed. Click the Properties button. 

After you configure the source properties, click Validate to ensure the source is error-free. Validating the source is especially helpful if you change any of the fields in the Output Schema.

If you forget to validate the source and there are any errors, the number of errors will display in the upper-right corner of the source.

7. To add transformations to the data pipeline, click on transformations in the Transform area and the Analytics area.

8. Connect the source to the transformation.

9. To configure the transformation, click the Properties button.

10. After you configure the transformation, click Validate to ensure the transformation is error-free. 

11. In the Sink area, click on the Sink required for the data pipeline. Then connect the Sink to the upstream transformation and configure the Sink properties. 

12. Click Save. After you save the data pipeline, you can find it in the List page under Drafts.

Created in 2020 by Google Inc.