Avro Snapshot Dataset Batch Source (Deprecated)

Note: Datasets and the Avro Snapshot Dataset Batch Source are deprecated and will be removed in CDAP 7.0.0.

A batch source that reads from a corresponding Avro Snapshot Dataset sink. The source will only read the most recent snapshot written to the sink.

This source is used whenever you want to read data written to the corresponding Avro Snapshot Dataset sink. It will read only the last snapshot written to that sink. For example, you might want to create daily snapshots of a database by reading the entire contents of a table and writing it to a Avro Snapshot Dataset sink. You might then want to use this source to read the most recent snapshot and run some data analysis on it.

Configuration

Property	Macro Enabled?	Description

Property	Macro Enabled?	Description
Dataset Name	Yes	Required. Name of the PartitionedFileSet to which records are written. If it doesn't exist, it will be created.
Snapshot Base Path	Yes	Optional. Base path for the PartitionedFileSet. Defaults to the name of the dataset.
FileSet Properties	Yes	Optional. Advanced feature to specify any additional properties that should be used with the sink, specified as a JSON object of string to string. These properties are set on the dataset if one is created. The properties are also passed to the dataset at runtime as arguments.
Output Schema	Yes	Required. The Avro schema of the record being written to the sink as a JSON object.

Example

This example will read from a SnapshotFileSet named ‘users’. It will read data in Avro format using the given schema. Every time the pipeline runs, only the most recently added snapshot will be read:

{
    "name": "SnapshotAvro",
    "type": "batchsource",
    "properties": {
        "name": "users",
        "schema": "{
            \"type\":\"record\",
            \"name\":\"user\",
            \"fields\":[
                {\"name\":\"id\",\"type\":\"long\"},
                {\"name\":\"name\",\"type\":\"string\"},
                {\"name\":\"birthyear\",\"type\":\"int\"}
            ]
        }"
    }
}