Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

If you wanted, you could add to the transform method a user metric indicating the number of fields changed. The user metrics can be queried by using the CDAP Metrics HTTP RESTful APIMicroservices:

Code Block
public void transform(StructuredRecord input, Emitter<StructuredRecord> emitter) throws Exception {
  int fieldsChanged = 0;
  . . .
    builder.set(fieldName, record.get(fieldName). . .
    fieldsChanged += 1;
  . . .
  getContext().getMetrics().count("fieldsChanged", fieldsChanged);
}

...

  1. In the joinOn step, the joiner creates a join key for each input record. the The CDAP pipeline will then take all records that have the same join key and collect them into a group.

  2. The merge step is then called. In this step, the plugin receives a list of all the records with same join key based on the type of join (either an inner or outer join). It is then up to the plugin to decide what to emit, in what becomes the final output of the stage.

...