File Appender Sink
The File Appender sink is available in the Hub.
Writes to a CDAP FileSet in text format. HDFS append must be enabled for this to work. One line is written for each record sent to the sink. All record fields are joined using a configurable separator. Each time a batch is written, the sink will examine all existing files in the output directory. If there are any files that are smaller in size than the size threshold, or more recent than the age threshold, new data will be appended to those files instead of written to new files.
Configuration
Property | Macro Enabled? | Property |
|---|---|---|
FileSet Name | No | Required. The name of the FileSet to write to. |
Field separator | No | Optional. The separator to join input record fields on. Default is a comma. |
Output Directory | No | Optional. The output directory to write to. |
File Name Prefix | No | Required. The prefix for all files written to the FileSet. |
Size Threshold (megabytes) | No | Optional. Files greater than this size (in megabytes) will be closed and a new file will be written to. 0 means there is no threshold. Default is 100. |
Age Threshold (minutes) | No | Optional. Files older than this size (in minutes) will not be appended to. 0 means there is no threshold. Default is 60. |
Created in 2020 by Google Inc.