Goal
This is a source plugin that would allow users to read and process mainframe files defined using COBOL Copybook. This should be basic first implementation.
Checklist
- User stories documented
- User stories reviewed
- Design documented
- Design reviewed
- Feature merged
-
Examples and guides - Integration tests
- Documentation for feature
- Short video demonstrating the feature
...
Input Format implementation : here
Design
- If "AbstractFieldValue"(JRecord) type is binary, the data will be encoded to Base64 format.
Integer.parseInt(new String(Base64.decodeBase64(Base64.encodeBase64(value.toStringasString().getBytes())).toString(), 2);
or
Base64.decodeInteger(Base64.encodeInteger(value.asBigInteger()));
It will depend on the field data type(int or BigInteger)
JRecord AbstractFieldValue type to JAVA primitive data type
| JRecord AbstractFieldValue type | JAVA primitive data type | Description | Comments |
|---|---|---|---|
char, char just right , char null terminated, char null padded | java.lang.String | ||
num left justified, num right justified , num zero padded | int | ||
| binary int, binary int positive, positive binary int fields | int | decode it using BASE64 format and then retrieve it. Integer.parseInt(new String(Base64.decodeBase64( Base64.encodeBase64(filedValue value.toString asString().getBytes())).toString( ), 2) | The Base64.decodeBase64() accepts either binary or String data, and therefore, first encoding and then decoding the values Decoding it directly results in improper values |
decimal, Mainframe Packed Decimal, Mainframe Packed Decimal, Mainframe Zoned Numeric Numeric, Mainframe Packed Decimal(+ve)
| java.math.BigDecimal | Since CDAP Schema.Type does not have a BigDecimal data type, converting everything to DOUBLE | |
Binary Integer Big Endian (Mainframe, AIX etc) -
| java.math.BigDecimal | decode it using BASE64 format and then retrieve it. Base64.decodeInteger(Base64.encodeInteger(filedValue.asBigInteger())) | The Base64.decodeBase64() accepts either binary or String data, and therefore, first encoding and then decoding the values Decoding it directly resulted in improper values Since CDAP Schema.Type does not have BigInteger converting this to LONG |
| num any decimal, positive num any decimal, assumed decimal | double | ||
| Hex | long | Long.parseLong(value.asHex(), 16) | Since CDAP Schema.Type does not have Hex converting this to LONG |
| Boolean / (Y/N) | java.lang.Boolean | ||
| Default | java.lang.String |
Examples
Properties :
referenceName : This will be used to uniquely identify this source for lineage, annotating metadata, etc.
copybookContents : Contents of the COBOL copybook file which will contain the data structure
binaryFilePath : Complete path of the .bin to be read.This will be a fixed length binary format file,that matches the copybook.
drop : Comma-separated list of fields to drop. For example: 'field1,field2,field3'.
maxSplitSize : Maximum split-size for each mapper in the MapReduce. \n Job. Defaults to 128MB.
...
"binaryFilePath": "file:///home/cdap/DTAR020_FB.bin",
"maxSplitSize": "5"
}
}
}