...
| Section | Field | Type | Description |
|---|---|---|---|
Basic Configuration | Path | String | Path - Provide the path for the File or Directory. ( Text Field) This should also support other file sources like FTP / SFTP etc |
| Recursive Processing | Boolean | List Files Recursively ( Boolean ) True / False | |
Output Schema | FileName | String | Record with FileName with full URI |
Queries
- If SFTP or FTP needs to be supported then its not clear how the credential information can be shared to the next step in the process.
FileCompressEncrypt Plugin
...
| Section | Field | Type | Description |
|---|---|---|---|
Basic Configuration | Input FileName | String | Full Name of File including path ( URI) |
| Compress File | Boolean | True / False | |
| Compression Algorithm | String ( List) | Gzip / Snappy. Applicable only if above Compress File is set to true | |
| Encrypt File | Boolean | True | |
| PGP Public Key Path | String | Location of PGP public key. Path to File | |
| PGP Public Key Access Userid | String | Userid to access the public key incase security is enabled | |
PGP Public Key Access password | String | Password to access the key file | |
| OutFilePath | String | Path to store the output file from sync. The output filename will follow the format of <InputfileName Suffix>.gz.pgp The file path URI can contain filesystem , Hdfs, gcs - google file system or cloud store. | |
| MoveInput | Boolean | True / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed. | |
| MoveFilePath | String | Path to move the input on successful processing of the file. |
Queries
- What is the best approach to track processed files so they are not processed again. Proposing moving the input files after successful processing to a different directory so they dont get processed again in the next run.
Usecase 2
FileDecompressDecrypt Plugin
The plugin support decrypting files using PGP Public key and decompress file.
Plugin Properties
| Section | Field | Type | Description |
|---|---|---|---|
Basic Configuration | Path | String | Path containing file name or directory of files. |
| Recursive Processing | Boolean | True / False | |
| DeCompress File | Boolean | True / False | |
| DeCompression Algorithm | String ( List) | Gzip / Snappy. Applicable only if above Compress File is set to true | |
| DeEncrypt File | Boolean | True | |
| PGP Private Key Path | String | Location of PGP public key. Path to File | |
| PGP Private Key Access Userid | String | Userid to access the public key incase security is enabled | |
PGP Private Key Access password | String | Password to access the key file | |
| MoveInput | Boolean | True / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed. | |
| MoveFilePath | String | Path to move the input on successful processing of the file. | |
Output Schema | Output | String | Each Row from the file read. |
...