Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
Use case(s)Confluent Cloud is a streaming data service with Apache Kafka delivered as a managed service.
User Storie(s)
- As a pipeline developer, I would like to stream data real time from Confluent cloud
- As a pipeline developer I would like to capture records that are not delivered downstream for analysis
- As a pipeline developer, I would like to stream data real time to Confluent Cloud
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Prerequisites
Properties
Real time source
| Property | description |
|---|---|
| Kafka topic | List of topics which we are listening to for streaming |
| Schema registry URL | URL endpoint for the schema registry on Confluent Cloud |
| Schema registry API key | |
| Schema registry secret |
Real time sink
| Property | description |
|---|---|
| Kafka topic | List of topics to which the data should be published to |
| Schema registry URL | URL endpoint for the schema registry on Confluent Cloud |
| Schema registry API key | |
| Schema registry secret |
References
Design / Implementation Tips
- Tip #1
- Tip #2
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature