...
Property | Macro Enabled? | Description |
|---|---|---|
Dataset Name | Yes | Required. Name of the |
Dataset Base Path | Yes | Optional. Base path for the |
Partition Directory Format | Yes | Optional. Format for the time partition, as used by |
Time Zone | Yes | Optional. The string ID for the time zone to format the date in. Defaults to using UTC. This setting is only used if |
Partition Offset | Yes | Optional. Amount of time to subtract from the pipeline runtime to determine the output partition. Defaults to 0m. The format is expected to be a number followed by an 's', 'm', 'h', or 'd' specifying the time unit, with 's' for seconds, 'm' for minutes, 'h' for hours, and 'd' for days. For example, if the pipeline is scheduled to run at midnight of January 1, 2016, and the offset is set to '1d', data will be written to the partition for midnight Dec 31, 2015." |
Clean Partitions Older Than | Optional. Configures the sink to delete partitions older than a specified date-time after a successful run. If set, when a run successfully finishes, the sink will subtract this amount of time from the runtime and delete any partitions for time partitions older than that. The format is expected to be a number followed by an 's', 'm', 'h', or 'd' specifying the time unit, with 's' for seconds, 'm' for minutes, 'h' for hours, and 'd' for days. For example, if the pipeline is scheduled to run at midnight of January 1, 2016, and this property is set to 7d, the sink will delete any partitions for time partitions older than midnight Dec 25, 2015. | |
Compression Codec | No | Default is None. Determines the compression codec to use on the resulting data. Valid values are None, Snappy, GZip, and LZO. |
Output Schema | Yes | OptionalRequired. The Avro schema of the record being written to the sink as a JSON Object. |
...