Versions Compared

Old Version 12

changes.mady.by.user Vasudha Gupta

Saved on

compared with

New Version 13

changes.mady.by.user Vasudha Gupta

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Introduction

Splunk Enterprise is a fully featured, powerful platform for collecting, searching, monitoring and analyzing machine data. Splunk Enterprise is easy to deploy and use. It turns machine data into rapid visibility, insight and intelligence. 

Use case(s)

User Storie(s)

  • As a pipeline developer, I would like to write relevant machine level data to Splunk for analysis
  • As a pipeline developer, I would like to filter and transform relevant fields to send to Splunk for analysis
  • As a pipeline developer I would like to ensure the records and relevant metadata are transformed into the correct format needed by Splunk HTTP Event Collector (Splunk HEC). Each record must contain event data and optional metadata in required format.
  • As a pipeline developer, I would like to send HTTP POST requests to Spunk HEC in JSON. One request is generated for one batch of records to Splunk HEC
  • As a pipeline developer, I would like to get an error if the data is not written successfully to Spunk HTTP Event Collector (Splunk HEC)

Plugin Type

  •  Batch Source
  •  Batch Sink 
  •  Real-time Source
  •  Real-time Sink
  •  Action
  •  Post-Run Action
  •  Aggregate
  •  Join
  •  Spark Model
  •  Spark Compute

Configurables

Following fields must be configurable for the plugin. The plugin should be created as a wrapper on HTTPSink with additional attributes required for Splunk HTTP Event Collector


User Facing Name
Type
Description
ConstraintsMacro Enabled?
URL
String
Required. The URL to post data to.

yes
HEC Token
String
Required . Specify value of token created for authentication to Splunk


Authentication Type
Select
  • Basic Authentication
    • Example: -u "x:<hec_token>"
  • HTTP Authentication
    • Example: "Authorization: Splunk <hec_token>"
  • Query String (Splunk Cloud Only)
    • Example: ?token=<hec_token>
    Pre-requisities for Query String URL
  • You must also enable query string authentication on a per-token basis. On your Splunk server, request Splunk Support to edit the file
    at $SPLUNK
     at $SPLUNK_HOME/etc/apps/splunk_httpinput/local/inputs.conf. Your tokens are listed by name in this file, in the
    form http
     form http://<token_name>
  • Within the stanza for each token you want to enable query string authentication, add the following setting (or change the existing setting, if applicable):allowQueryStringAuth = true


Batch Size
Number - with upper bound
The number of messages to batch before sending
> 0, default 1 (no batching)yes
Format
Number with upper limit
The format to send the message in. JSON will format the entire input record to json and send it as a payload. Form will convert the input message to a query string and send it in the payload. Custom will leverage the request body field to send.
JSON, Form, Custom
Request Body
String
Optional request body. Only required if Custom format is specified.

yes
Content Type
String
Used to specify the Content-Type header.

yes
Channel Identifier Header
KeyValue
GUID for Channel Identifier

yes
Should Follow Redirects?
Toggle
Whether to automatically follow redirects. Defaults to true.
true,false
Number of Retries
Toggle
The number of times the request should be retried if the request fails. Defaults to 3.
0,1,2,3,4,5,6,7,8,9,10
Connect Timeout
String
The time in milliseconds to wait for a connection. Set to 0 for infinite. Defaults to 60000 (1 minute).


Read Timeout
String
The time in milliseconds to wait for a read. Set to 0 for infinite. Defaults to 60000 (1 minute).


Use Proxy
Toggle
True or false to enable HTTP proxy to connect to system. Defaults to false
true, false
Proxy URI
String
Proxy URI


Proxy Username
String
Username


Proxy Password
String
Password



References

https://docs.splunk.com/Documentation/Splunk/7.1.1/Data/FormateventsforHTTPEventCollector

Design / Implementation Tips

  • Tip #1
  • Tip #2

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

  • Some future work – HYDRATOR-99999
  • Another future work – HYDRATOR-99999

Test Case(s)

  • Test case #1
  • Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2



Table of Contents

Table of Contents
stylecircle

Checklist

  •  User stories documented 
  •  User stories reviewed 
  •  Design documented 
  •  Design reviewed 
  •  Feature merged 
  •  Examples and guides 
  •  Integration tests 
  •  Documentation for feature 
  •  Short video demonstrating the feature