Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

This feature can be used to perform metadata based processing. For example, suppose you have a dataset which contains sensitive information in one of its fields. Your organization also has a policy that requires that all sensitive information be masked. You can annotate the field with a tag to represent that it contains sensitive information. You can then develop a plugin which reads the metadata of all the fields in the dataset and masks the contents of a field, if it is annotated with a particular tag.

Note:

...

Reading metadata is not supported in a program or a plugin running in a cloud environment. However, metadata can still be added, updated, or deleted in this case.

Programs

Metadata can be accessed from MapReduce, Spark, Workers and Service through methods from the MetadataReader object, which are available via the appropriate program context object in your program. The program context object can be obtained by invoking the getContext() method inside the initialize method of your program.

...