Confluent Cloud is a resilient, scalable streaming data service based on Apache Kafka®, delivered as a fully managed service. Confluent Cloud has a web interface and local command line interface. You can manage cluster resources, settings, and billing with the web interface.
Use case(s)
User Storie(s)
As a pipeline developer, I would like to stream data from Confluent cloud
As a pipeline developer I would like to capture records that are not delivered downstream for analysis
Plugin Type
Batch Source
Batch Sink
Real-time Source
Real-time Sink
Action
Post-Run Action
Aggregate
Join
Spark Model
Spark Compute
Configurables
Following fields must be configurable for the plugin. The plugin should be created as a wrapper on HTTPSink with additional attributes required for Splunk HTTP Event Collector
User Facing Name
Type
Description
Constraints
Macro Enabled?
URL
String
Required. The URL to post data to.
yes
HEC Token
String
Required . Specify value of token created for authentication to Splunk
Authentication Type
Select
Basic Authentication
Example: -u "x:<hec_token>"
HTTP Authentication
Example: "Authorization: Splunk <hec_token>"
Query String (Splunk Cloud Only)
Example: ?token=<hec_token>
Pre-requisities for Query String URL
You must also enable query string authentication on a per-token basis. On your Splunk server, request Splunk Support to edit the file at $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf. Your tokens are listed by name in this file, in the form http://<token_name>
Within the stanza for each token you want to enable query string authentication, add the following setting (or change the existing setting, if applicable):allowQueryStringAuth = true
Batch Size
Number - with upper bound
The number of messages to batch before sending
> 0, default 1 (no batching)
yes
Format
Number with upper limit
The format to send the message in. JSON will format the entire input record to json and send it as a payload. Form will convert the input message to a query string and send it in the payload. Custom will leverage the request body field to send.
JSON, Form, Custom
Request Body
String
Optional request body. Only required if Custom format is specified.
yes
Content Type
String
Used to specify the Content-Type header.
yes
Channel Identifier Header
KeyValue
If your request includes raw events, you must include an X-Splunk-Request-Channel
header field in the event, and it must be set to a unique channel identifier (a GUID). curl https://http-inputs-<customer>.splunkcloud.com/services/collector/raw -H "X-Splunk-Request-Channel: FE0ECFAD-13D5-401B-847D-77833BD77131" -H "Authorization: Splunk BD274822-96AA-4DA6-90EC-18940FB2414C" -d '<raw data string>' -v Alternatively, the X-Splunk-Request-Channel header field can be sent as a URL query parameter, as shown here: