Goal 

This is a source plugin that would allow users to read and process mainframe files defined using COBOL Copy Book. This should be basic first implementation.

...

Input Format implementation : here 

Design

Examples

Properties : 

...

}
This source plugin will read fixed length flat file sale.bin and the CobolCopyBook file stored at the hdfs location hdfs://data/sales/sales.bin and hdfs://data/sales/sales.cbl respectively. The plugin will output a Base64 encoded data having the schema as defined by the user.

Sample .cbl file:

000600*
000700* RECORD LENGTH IS 27.
000800*
000900 03 DTAR020-KCODE-STORE-KEY.
001000 05 DTAR020-KEYCODE-NO PIC X(08).
001100 05 DTAR020-STORE-NO PIC S9(03) COMP-3.
001200 03 DTAR020-DATE PIC S9(07) COMP-3.
001300 03 DTAR020-DEPT-NO PIC S9(03) COMP-3.
001400 03 DTAR020-QTY-SOLD PIC S9(9) COMP-3.
001500 03 DTAR020-SALE-PRICE PIC S9(9)V99 COMP-3.

The source plugin will read the above file as well as the data present in the .bin file and generate Base64 encoded output. The schema for the output will depend on the output schema as defined by the user.

Sample Base64 encoded output:

9vn2+PT19fgCDABAEYwoDAAAAAAcAAAAAZAM9vn2+PT19fgCDABAEYwoDAAAAAAdAAAAAZAN9vn2+PT19fgCDABAEYwoDAAAAAAcAAAAAFAc9vn2+fTx9fg
CDABAEYwoDAAAAAAcAAAAAZAM9vn2+fTx9fgCDABAEYwoDAAAAAAdAAAAAZAN9vn2+fTx9fgCDABAEYwoDAAAAAAcAAAAAFAc9vP28PT48PgCDABAEYwXDA
AAAAAcAAAAAEh89vL2+PT29/ECDABAEYxoXAAAAAAcAAAABpmc9vL2+PT29/ECDABAEYxoXAAAAAAdAAAABpmd9vT28/T08vkCDABAEYyVfAAAAAAcAAAAA
Dmc9vb28vT09fgCDABAEYyVfAAAAAAcAAAAAAic9vP29/T49vECDABAEYyVfAAAAAEMAAAAACcM9vX29/T18/ICDABAEYySnAAAAAAcAAAAADWc9vT28fT0
8PEFnABAEYyVfAAAAAAcAAAAABmc9vT28fT08PEFnABAEYyVfAAAAAAcAAAAABmc9vH29vT38fMFnABAEYwzXAAAAAAcAAAAAXmc9vH29vT38fMFnABAEYw
zXAAAAAAdAAAAAXmd9vj28/T39fIFnABAEYxBDAAAAAAcAAAAAImc9vD28fT0+PcFnABAEYyHjAAAAAAcAAAAAFlc....