Versions Compared
compared with
Key
- This line was added.
- This line was removed.
- Formatting was changed.
Introduction
Confluent Cloud is a resilient, scalable streaming data service based on Apache Kafka®, delivered as a fully managed service. Confluent Cloud has a web interface and local command line interface. You can manage cluster resources, settings, and billing with the web interface.
Use case(s)
User Storie(s)
- As a pipeline developer, I would like to stream data from Confluent cloud
- As a pipeline developer I would like to capture records that are not delivered downstream for analysis
Plugin Type
- Batch Source
- Batch Sink
- Real-time Source
- Real-time Sink
- Action
- Post-Run Action
- Aggregate
- Join
- Spark Model
- Spark Compute
Configurables
Following fields must be configurable for the plugin. The plugin should be created as a wrapper on HTTPSink with additional attributes required for Splunk HTTP Event Collector
| User Facing Name | Type | Description | Constraints | Macro Enabled? |
|---|---|---|---|---|
| URL | String | Required. The URL to post data to. | yes | |
| HEC Token | String | Required . Specify value of token created for authentication to Splunk | ||
| Authentication Type | Select |
| ||
| Batch Size | Number - with upper bound | The number of messages to batch before sending | > 0, default 1 (no batching) | yes |
| Format | Number with upper limit | The format to send the message in. JSON will format the entire input record to json and send it as a payload. Form will convert the input message to a query string and send it in the payload. Custom will leverage the request body field to send. | JSON, Form, Custom | |
| Request Body | String | Optional request body. Only required if Custom format is specified. | yes | |
| Content Type | String | Used to specify the Content-Type header. | yes | |
| Channel Identifier Header | KeyValue | If your request includes raw events, you must include an header field in the event, and it must be set to a unique channel identifier (a GUID). curl https://http-inputs-<customer>.splunkcloud.com/services/collector/raw -H "X-Splunk-Request-Channel: FE0ECFAD-13D5-401B-847D-77833BD77131" -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d '<raw data string>' -v Alternatively, the curl https://http-inputs-<customer>.splunkcloud.com/services/collector/raw?channel=FE0ECFAD-13D5-401B-847D-77833BD77131 -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d '<raw data string>' -v | yes | |
| Should Follow Redirects? | Toggle | Whether to automatically follow redirects. Defaults to true. | true,false | |
| Number of Retries | Toggle | The number of times the request should be retried if the request fails. Defaults to 3. | 0,1,2,3,4,5,6,7,8,9,10 | |
| Connect Timeout | String | The time in milliseconds to wait for a connection. Set to 0 for infinite. Defaults to 60000 (1 minute). | ||
| Read Timeout | String | The time in milliseconds to wait for a read. Set to 0 for infinite. Defaults to 60000 (1 minute). | ||
| Use Proxy | Toggle | True or false to enable HTTP proxy to connect to system. Defaults to false | true, false | |
| Proxy URI | String | Proxy URI | ||
| Proxy Username | String | Username | ||
| Proxy Password | String | Password |
References
https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/FormateventsforHTTPEventCollector
Design / Implementation Tips
- Tip #1
- Tip #2
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
- Some future work – HYDRATOR-99999
- Another future work – HYDRATOR-99999
Test Case(s)
- Test case #1
- Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
Table of Contents
Table of Contents style circle
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
- Examples and guides
- Integration tests
- Documentation for feature
- Short video demonstrating the feature