Page Comparison

Reference Name

No

Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.

Application Name

Yes

Required. The name of the Kinesis application. The application name that is used to checkpoint the Kinesis sequence numbers in DynamoDB table.

Stream Name

Yes

Required. The name of the Kinesis stream to the get the data from. The stream should be active. 

Kinesis Endpoint url

Yes

Required. Valid Kinesis endpoint URL, for example, Kinesis.us-east-1.amazonaws.com. 

Region

Yes

Required. Valid Kinesis region URL, for example, ap-south-1.

Default is us-east-1.

Checkpoint Interval Duration

Yes

Required. The interval in milliseconds at which the Kinesis Client Library saves its position in the stream

Initial Position in Stream

Yes

Required. Initial position in the stream. Can be either TRIM_HORIZON or LATEST.

Default is LATEST.

AWS Access Key ID

Yes

Required. The access Id provided by AWS required to access the Kinesis streams. The Id can be stored in CDAP secure store and can be provided as macro configuration. 

AWS Access Secret

Yes

Required. AWS access key secret having access to Kinesis streams. The key can be stored in CDAP secure store and can be provided as macro configuration.

Format

No

Optional. Format of the Kinesis shard payload. Any format supported by CDAP is supported. For example, a value of ‘csv’ will attempt to parse Kinesis payloads as comma-separated values. If no format is given, Kinesis payloads will be treated as bytes.