Introduction

Introduction

Splunk Enterprise is a fully featured, powerful platform for collecting, searching, monitoring and analyzing machine data. Splunk Enterprise is easy to deploy and use. It turns machine data into rapid visibility, insight and intelligence. 

Use case(s)

User Storie(s)

• As a pipeline developer, I would like to write relevant and filtered machine level data to Splunk for log analysis
• As a pipeline developer, I would like to sent The destination sends HTTP POST requests to the HEC endpoint using the JSON data format. The destination generates one HTTP request for each batch, sending multiple records at a timeto filter and transform relevant fields to send to Splunk for analysis
• As a pipeline developer I would like to ensure the records and relevant metadata are transformed into the correct format needed by Splunk HTTP Event Collector (Splunk HEC). Each record must contain the event data and optionally the event optional metadata in the format required by Splunk.required format.
• As a pipeline developer, I would like to send HTTP POST requests to Spunk HEC in JSON. One request is generated for one batch of records to Splunk HEC
• As a pipeline developer, I would like to get an error if the data is not written successfully to Spunk HTTP Event Collector (Splunk HEC)

Plugin Type

•  Batch Source
•  Batch Sink 
•  Real-time Source
•  Real-time Sink
•  Action
•  Post-Run Action
•  Aggregate
•  Join
•  Spark Model
•  Spark Compute

Configurables

This integration should be built using two plugins:

Zendesk single object

Used when you want to pull only a single object from Zendesk

Following fields must be configurable for the plugin. The plugin should be created as a wrapper on HTTPSink with additional attributes required for Splunk HTTP Event Collector

Batch Sink

The plugin should output one record per row in Zendesk, with an additional field specifying the object where the record came from. Refer to the Salesforce multi-objects plugin for examples

Design / Implementation Tips

User Facing Name

Type

Description

Zendesk multi-object

Used when you want to pull only a single object from Zendesk

URL

String

Required. The URL to post data to.

Authentication Type

Select

HEC Token

String

Required. Specify value of token created for authentication to Splunk.

Username

String

Required for Basic Authentication. Login name for authentication to the Splunk API.

Endpoint

Select

Splunk endpoint to send data to.

Batch Size

Number - with upper bound

The number of messages to batch before sending

Event metadata

String

Optional event metadata string in the JSON export for destination

Channel Identifier Header

String

GUID for Channel Identifier

Connect Timeout

Number - with upper bound

The time in milliseconds to wait for a connection. Set to 0 for infinite. Defaults to 60000 (1 minute).

Read Timeout

Number

The time in milliseconds to wait for a read. Set to 0 for infinite. Defaults to 60000 (1 minute).

Number of Retries

Toggle

The number of times the request should be retried if the request fails. Defaults to 3.

Max Retry Wait

Number

Maximum time in milliseconds retries can take. Set to 0 for infinite. Defaults to 60000 (1 minute).

Max Retry Jitter Wait

Number

Maximum time in milliseconds added to retries. Defaults to 100.

Streaming Source or Batch Source using Splunk Search Jobs API

For streaming data from Splunk, real time searches need to be performed. The searches are performed against the incoming events within a sliding time window by using the search criteria that has been defined in Splunk. Option to run search is normal or realtime exists

Required. Specify value of token created for authentication to Splunk

Username

Required. Login name for authentication to the Splunk API.

Password

Required. Password for authentication to the Splunk API.

References

https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/FormateventsforHTTPEventCollector

https://docs.splunk.com/Documentation/Splunk/7.3.1/RESTREF/RESTsearch#search.2Fjobs.2Fexport

https://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTsearch#search.2Fjobs

https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Aboutrealtimesearches

Design / Implementation Tips

• Plugin will be implemented using Splunk Java SDK.
• Authentication will be performed using Username/Password or API Token.
• Output schema will be automatically generated from specified Search String or Search Id.

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• Test case #1
• Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2