Page Comparison

...

In the CDAP Pipelines UI, all Batch Aggregator, Batch Joiner, Spark Compute, and Spark Sink plugins are grouped under the Analytics section. All Transformation and Windower plugins are grouped under the Transforms section.

Maven Archetypes

To get started on creating a custom plugin, you can use the Maven archetype to create your project:

• cdap-data-pipeline-plugins-archetype (contains batch, Spark plugin, and other types)

This command will create a project from an archetype:

$ mvn archetype:generate \
      -DarchetypeGroupId=io.cdap.cdap \
      -DarchetypeArtifactId=cdap-data-pipeline-plugins-archetype \
      -DarchetypeVersion=6.2.0 \
      -DgroupId=org.example.plugin

Note: Replace the groupId parameter (org.example.plugin) with your own organization, but it must not be replaced with io.cdap.cdap.

Complete examples for each archetype:

$ mvn archetype:generate -DarchetypeGroupId=io.cdap.cdap -DarchetypeArtifactId=cdap-data-pipeline-plugins-archetype -DarchetypeVersion=6.2.0 -DgroupId=org.example.plugin

Maven supplies a guide to the naming convention used above at https://maven.apache.org/guides/mini/guide-naming-conventions.htmlcopy the code at https://github.com/data-integrations/example-transform as a starting point for your project.

Class Annotations

These annotations are used for plugin classes:

• @Plugin: The class to be exposed as a plugin needs to be annotated with the @Plugin annotation and the type of the plugin must be specified.

• @Name: Annotation used to name the plugin.

• @Description: Annotation used to add a description of the plugin.

• @Requirements: Annotation used to specify the Requirements needed the requirements needed by a plugin to run successfully.

...

LocalDate localDate = record.getDate("date");
LocalTime localTimeMillis = record.getTime("time-millis");
LocalTime localTimeMicros = record.getTime("time-micros");
ZonedDateTime timestampMillis = record.getTimestamp("ts-millis");
ZonedDateTime timestampMicros = record.getTimestamp("ts-micros");

Please note Note that logical types such as Date, Time, and Timestamp are internally represented as primitive types. Therefore, these types can be set or retrieved as primitive types in structured record. For example:

...