Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
YesIntroduction
Confluent Cloud is a streaming data service with Apache Kafka delivered as a managed service.
User Storie(s)
- As a pipeline developer, I would like to stream data real time from Confluent cloud cloud based on the specified schema by the user
- As a pipeline developer, I would like to capture records that are not delivered downstream for analysis stream data real time to Confluent Cloud and perform serialization during event streaming
- As a pipeline developer , I would like to stream data real time capture records that did not get delivered downstream to Confluent Cloud for analysis
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Prerequisites
- Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
- Connect: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
- Java 1.8
Properties
Real time source
| Property | description | Mandatory or Optional |
|---|---|---|
| Kafka cluster credential API key | API key in order to connect to the cluster | Yes |
| Kafka cluster credential secret | Secret in order to connect to the cluster | Yes |
| Kafka Zookeeper | The connect string location of ZooKeeper. Either that or the list of brokers is required | Required if brokers not specified |
| Kafka brokers | Comma-separated list of Kafka brokers. Either that or the ZooKeeper quorum is required | Required if zookeeper not specified |
| Kafka partition | Number of partitions | Yes |
| Kafka topic | List of topics which we are listening to for streaming | Yes |
| Schema registry URL | URL endpoint for the schema registry on Confluent Cloud or self hosted schema registry URL | No |
| Schema registry API key | API key | No |
| Schema registry secret | Secret | No |
Real time sink
| Property | description |
|---|---|
| Kafka topic | List of topics to which the data should be published to |
| Schema registry URL | URL endpoint for the schema registry on Confluent Cloud |
| Schema registry API key | API key for |
| Schema registry secret |
References
Design / Implementation Tips
- Tip #1
- Tip #2
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature