Goal 

This is a source plugin that would allow users to read and process mainframe files defined using COBOL Copy Book. This should be basic first implementation.

...

Input Format implementation : here 

Design

Examples

Properties : 

cobolFilePath : hdfs path of .cbl file to be read
binaryFilePath : hdfs path of .bin file to be read
isCompressed : check if it is a compressed file.User can also specify a Native Compressed Codec as input.
outputSchema : list of fields in the output file

Example :

{

"name": "CopyBookReader",
"plugin": {

"name": "CopyBookReader",

"type": "batchsource",

"properties": {

"cobolFilePath": "/data/sales/sales.cbl",
"binaryFilePath" "/data/sales/sale.bin"
"isCompressed" : "true/false",
"outputSchema" : {},
"uploadFileProperties": {}

}

}

}
This source plugin will read fixed length flat file sale.bin and the CobolCopyBook file stored at the hdfs location hdfs://data/sales/sales.bin and hdfs://data/sales/sales.cbl respectively. The plugin will output a Base64 encoded data having the schema as defined by the user.

Sample .cbl file:

000600*
000700* RECORD LENGTH IS 27.
000800*
000900 03 DTAR020-KCODE-STORE-KEY.
001000 05 DTAR020-KEYCODE-NO PIC X(08).
001100 05 DTAR020-STORE-NO PIC S9(03) COMP-3.
001200 03 DTAR020-DATE PIC S9(07) COMP-3.
001300 03 DTAR020-DEPT-NO PIC S9(03) COMP-3.
001400 03 DTAR020-QTY-SOLD PIC S9(9) COMP-3.
001500 03 DTAR020-SALE-PRICE PIC S9(9)V99 COMP-3.

The source plugin will read the above file as well as the data present in the .bin file and generate Base64 encoded output. The schema for the output will depend on the output schema as defined by the user.

Sample Base64 encoded output:

9vn2+PT19fgCDABAEYwoDAAAAAAcAAAAAZAM9vn2+PT19fgCDABAEYwoDAAAAAAdAAAAAZAN9vn2+PT19fgCDABAEYwoDAAAAAAcAAAAAFAc9vn2+fTx9fgCDABAEYwoDAAAAAAcAAAAAZA
M9vn2+fTx9fgCDABAEYwoDAAAAAAdAAAAAZAN9vn2+fTx9fgCDABAEYwoDAAAAAAcAAAAAFAc9vP28PT48PgCDABAEYwXDAAAAAAcAAAAAEh89vL2+PT29/ECDABAEYxoXAAAAAAcAAAABp
mc9vL2+PT29/ECDABAEYxoXAAAAAAdAAAABpmd9vT28/T08vkCDABAEYyVfAAAAAAcAAAAADmc9vb28vT09fgCDABAEYyVfAAAAAAcAAAAAAic9vP29/T49vECDABAEYyVfAAAAAEMAAAAA
CcM9vX29/T18/ICDABAEYySnAAAAAAcAAAAADWc9vT28fT08PEFnABAEYyVfAAAAAAcAAAAABmc9vT28fT08PEFnABAEYyVfAAAAAAcAAAAABmc9vH29vT38fMFnABAEYwzXAAAAAAcAAAA
AXmc9vH29vT38fMFnABAEYwzXAAAAAAdAAAAAXmd9vj28/T39fIFnABAEYxBDAAAAAAcAAAAAImc9vD28fT0+PcFnABAEYyHjAAAAAAcAAAAAFlc....