In the Oracle Batch Source (version 1.8.4 and earlier), when the source data included fields with the NUMBER data type with undefined precision and scale, CDAP read it as decimal and set the precision to 38 and the scale to 0. If any values in the field had scale other than 0, CDAP rounded these values, which could have resulted in data loss. If the scale for this field is overridden in the plugin output schema, the pipeline fails.
In the Oracle batch Source (version 1.8.5), you can edit the scale of the CDAP decimal data type in the output schema, and the overridden value is used to map the Oracle NUMBER data type without failing the pipeline. If there are any Numbers present in the fields with the scale greater than the scale defined in the plugin, CDAP rounds the values based on the scale set by the user in the output schema. For example, if you specify precision=10, scale = 3 and the value 123.4567 comes in, CDAP rounds to 123.457. For more information about setting precision and scale in a plugin, see Changing the precision and scale for decimal fields in the output schema.
During pipeline execution when we encounter any Oracle Number (decimal) type whose scale does not match with the scale provided by the User in the output schema, then we throw any IllegalStateException stating that the scale in the decimal does not match with the scale set in the output schema for that field.
This check ensures that any input decimal is of the same scale as stated in the output schema for its field. The assumption here is that the input decimal scale would match with the output field schema scale. But this assumption does not hold true in case the decimal is coming from Oracle Number type which do not have a precision and scale(oracle DB allows such types). In this case every value for the field type would have its own scale and would not match with the output schema set for the field.
Due to current state of things, these types only get mapped against the default precision and scale of 38,0 and any other value set in the output schema for the field result into error. Due to this there is always a precision loss seen for these Oracle Numbers in the pipeline.
Hence creating this bug to allow any user to provide a scale and allow rounding of such numbers based on the scale provided and not throw exception while running.