...
| Section | Field | Type | Description |
|---|---|---|---|
Basic Configuration | Path | String | Path - Provide the path for the File or Directory. ( Text Field) This should also support other file sources like FTP / SFTP etc |
| Recursive Processing | Boolean | List Files Recursively ( Boolean ) True / False | |
Output Schema | FileName | String | Record with FileName with full URI |
Queries
- If SFTP or FTP needs to be supported then its not clear how the credential information can be shared to the next step in the process.
FileCompressEncrypt Plugin
...
| Section | Field | Type | Description |
|---|---|---|---|
Basic Configuration | Input FileName | String | Full Name of File including path ( URI) |
| Compress File | Boolean | True / False | |
| Compression Algorithm | String ( List) | Gzip / Snappy. Applicable only if above Compress File is set to true | |
| Encrypt File | Boolean | True | |
| PGP Public Key Path | String | Location of PGP public key. Path to File | |
| PGP Public Key Access Userid | String | Userid to access the public key incase security is enabled | |
PGP Public Key Access password | String | Password to access the key file | |
| OutFilePath | String | Path to store the output file from sync. The output filename will follow the format of <InputfileName Suffix>.gz.pgp The file path URI can contain filesystem , Hdfs, gcs - google file system or cloud store. | |
| MoveInput | Boolean | True / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed. | |
| MoveFilePath | String | Path to move the input on successful processing of the file. |
Queries
- What is the best approach to track processed files so they are not processed again. Proposing moving the input files after successful processing to a different directory so they dont get processed again in the next run.
Usecase 2
FileDecompressDecrypt Plugin
...
| Section | Field | Type | Description | ||||||
|---|---|---|---|---|---|---|---|---|---|
Basic Configuration | FileNamePath | String | Full Name of File including path ( URI) | Location of PGK Public Key | String | Key Access ( userid ) | String | Password | String |
Output Schema | Decrypt FileName (TBD) | String | Record with FileName with full URI | ||||||
| Decrypt File contents Content (TBD) Path containing file name or directory of files. | |||||||||
| Recursive Processing | Boolean | True / False | |||||||
| DeCompress File | Boolean | True / False | |||||||
| DeCompression Algorithm | String ( List) | Gzip / Snappy. Applicable only if above Compress File is set to true | |||||||
| DeEncrypt File | Boolean | True | |||||||
| PGP Private Key Path | String | Location of PGP public key. Path to File | |||||||
| PGP Private Key Access Userid | String | Userid to access the public key incase security is enabled | |||||||
PGP Private Key Access password | String | Password to access the key file | |||||||
| MoveInput | Boolean | True / False - Move the source input file to a different path so the next run of the pipeline the same file will not be processed. | |||||||
| MoveFilePath | String | Path to move the input on successful processing of the file. | |||||||
Output Schema | Output | String | Each Row from the file read. | ||||||