Introduction 

There are a few use cases that we want to support with this feature. The first is to support a use case where the pipeline is a geofencing use case, where the pipeline is processing user locations, and wants to send notifications when a user enters or leaves a geo fence. Similar use cases include pipelines that are reading user biometric data that need to send notifications if user heart rate exceeds a certain value, or pipelines that read machine cpu usage that need to send notification if cpu usage exceeds a certain value. In these use cases, some uncommon event occurs that must be acted on by a separate system. The events that trigger these notifications still need to be processed by the rest of the pipeline, there is just some side effect that is triggered once they are observed.

Goals

The goal is to allow stages in a pipeline to emit notifications, which can be configured to be published to TMS or to Kafka.

User Stories 

• As a pipeline developer, I want to be able to create a pipeline where notifications are published to Kafka or TMS when some condition is met
• As a pipeline developer, I want to be able to create some pipelines that publish notifications and some that do not, even when the conditions are met
• As a pipeline developer, I want to be able to configure which topic notifications are published to
• As a pipeline developer, I want to be able to tell which plugins can emit notifications and which plugins cannot
• As a cluster administrator, I want to be able to see how many notifications were published for a pipeline run
• As a plugin developer, I want to be able to write arbitrary logic to control when to publish notifications
• As a plugin developer, I want to be able to indicate which plugins can emit notifications

Design

At a high level, we would like each existing plugin type (except sinks) to be able to emit notifications. A notification is not an arbitrary record, but must conform to a specific schema. Each plugin will indicate whether it can emit notifications or not, which can be reflected in the UI by an additional 'port'. When a pipeline stage is connected to a new 'NotificationPublisher' plugin type, any notifications emitted by that stage will be sent to the NotificationPublisher for actual publishing.

When the actual notifications are published is left up to the pipeline. If a NotificationEmitter plugin is not attached to a stage, nothing will be published.

Approach

Approach #1

Approach #2

API changes

New Programmatic APIs

New Java APIs introduced (both user facing and internal)

Deprecated Programmatic APIs

New REST APIs

Deprecated REST API

None

CLI Impact or Changes

• None

UI Impact or Changes

• UI must be able to detect which plugins can emit notifications, and display a corresponding port
• UI must display metrics for notifications emitted

Security Impact 

This feature will use TMS, so any authorization added to TMS will affect this feature.

Impact on Infrastructure Outages 

None

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work