...
This source is used whenever you want to read data written to the corresponding Avro Snapshot Dataset sink. It will read only the last snapshot written to that sink. For example, you might want to create daily snapshots of a database by reading the entire contents of a table and writing it to a Avro Snapshot Dataset sink. You might then want to use this source to read the most recent snapshot and run some data analysis on it.
Configuration
Property | Macro Enabled? | Description |
|---|---|---|
Dataset Name | Yes | Required. Name of the PartitionedFileSet to which records are written. If it doesn't exist, it will be created. |
Snapshot Base Path | Yes | Optional. Base path for the PartitionedFileSet. Defaults to the name of the dataset. |
FileSet Properties | Yes | Optional. Advanced feature to specify any additional properties that should be used with the sink, specified as a JSON object of string to string. These properties are set on the dataset if one is created. The properties are also passed to the dataset at runtime as arguments. |
Output Schema | Yes | Required. The Avro schema of the record being written to the sink as a JSON object. |
Example
This example will read from a SnapshotFileSet named ‘users’. It will read data in Avro format using the given schema. Every time the pipeline runs, only the most recently added snapshot will be read:
...