When using a connection and importing this as AVRO in wrangler, It doesn’t show any data.
On checking the wrangler service we find :
2023-09-14 07:25:19,952 - WARN [service-http-executor-56:i.c.c.e.p.c.SampleResponseCodec@68] - Error converting the json string {"SequenceNumber":14208103,"Offset":"73362348596784","EnqueuedTimeUtc":"3/12/2023 2:17:24 AM","SystemProperties":{"x-opt-enqueued-time":1678587444563},"Properties":{},"Body": [large byte array removed for this comment]} to StructuredRecord
java.io.IOException: Unable to convert string to type UNION
at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.convertKey(JsonStructuredRecordDatumReader.java:209)
at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decodeMap(JsonStructuredRecordDatumReader.java:133)
at io.cdap.cdap.format.io.StructuredRecordDatumReader.decode(StructuredRecordDatumReader.java:76)
at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decode(JsonStructuredRecordDatumReader.java:98)
at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decodeRecord(JsonStructuredRecordDatumReader.java:156)
at io.cdap.cdap.format.io.StructuredRecordDatumReader.read(StructuredRecordDatumReader.java:40)
at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.read(JsonStructuredRecordDatumReader.java:87)
at io.cdap.cdap.format.StructuredRecordStringConverter.fromJsonString(StructuredRecordStringConverter.java:61)
at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.lambda$deserialize$0(SampleResponseCodec.java:64)
at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:901)
at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.deserialize(SampleResponseCodec.java:61)
at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.deserialize(SampleResponseCodec.java:43)
at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58)
at com.google.gson.Gson.fromJson(Gson.java:810)
at com.google.gson.Gson.fromJson(Gson.java:775)
at com.google.gson.Gson.fromJson(Gson.java:724)
at com.google.gson.Gson.fromJson(Gson.java:696)
at io.cdap.wrangler.service.directive.ConnectionDiscoverer.retrieveResult(ConnectionDiscoverer.java:202)
at io.cdap.wrangler.service.directive.ConnectionDiscoverer.execute(ConnectionDiscoverer.java:134)
at io.cdap.wrangler.service.directive.ConnectionDiscoverer.retrieveSample(ConnectionDiscoverer.java:106)
at io.cdap.wrangler.service.directive.WorkspaceHandler.lambda$createWorkspace$1(WorkspaceHandler.java:153)
at io.cdap.wrangler.service.common.AbstractWranglerHandler.respond(AbstractWranglerHandler.java:202)
at io.cdap.wrangler.service.directive.WorkspaceHandler.createWorkspace(WorkspaceHandler.java:136)
at io.cdap.wrangler.service.directive.WorkspaceHandlerf667fa1c55614e7a62f467e95f30ef13.lambda$createWorkspace_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source)
at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:608)
at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:332)
at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:69)
at io.cdap.wrangler.service.directive.WorkspaceHandlerf667fa1c55614e7a62f467e95f30ef13.createWorkspace(Unknown Source)
We should throw error message on UI that we don’t support UNION type.
Release Notes
None
Activity
Show:
Pinned fields
Click on the next to a field label to start pinning.
Repro steps :
- a avro file while a field of schema union
Example field schema :
{ "name" : "Body", "type" : [ "null", "bytes" ] } ]
When using a connection and importing this as AVRO in wrangler,
It doesn’t show any data.
On checking the wrangler service we find :
2023-09-14 07:25:19,952 - WARN [service-http-executor-56:i.c.c.e.p.c.SampleResponseCodec@68] - Error converting the json string {"SequenceNumber":14208103,"Offset":"73362348596784","EnqueuedTimeUtc":"3/12/2023 2:17:24 AM","SystemProperties":{"x-opt-enqueued-time":1678587444563},"Properties":{},"Body": [large byte array removed for this comment]} to StructuredRecord java.io.IOException: Unable to convert string to type UNION at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.convertKey(JsonStructuredRecordDatumReader.java:209) at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decodeMap(JsonStructuredRecordDatumReader.java:133) at io.cdap.cdap.format.io.StructuredRecordDatumReader.decode(StructuredRecordDatumReader.java:76) at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decode(JsonStructuredRecordDatumReader.java:98) at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.decodeRecord(JsonStructuredRecordDatumReader.java:156) at io.cdap.cdap.format.io.StructuredRecordDatumReader.read(StructuredRecordDatumReader.java:40) at io.cdap.cdap.format.io.JsonStructuredRecordDatumReader.read(JsonStructuredRecordDatumReader.java:87) at io.cdap.cdap.format.StructuredRecordStringConverter.fromJsonString(StructuredRecordStringConverter.java:61) at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.lambda$deserialize$0(SampleResponseCodec.java:64) at java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:901) at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.deserialize(SampleResponseCodec.java:61) at io.cdap.cdap.etl.proto.connection.SampleResponseCodec.deserialize(SampleResponseCodec.java:43) at com.google.gson.TreeTypeAdapter.read(TreeTypeAdapter.java:58) at com.google.gson.Gson.fromJson(Gson.java:810) at com.google.gson.Gson.fromJson(Gson.java:775) at com.google.gson.Gson.fromJson(Gson.java:724) at com.google.gson.Gson.fromJson(Gson.java:696) at io.cdap.wrangler.service.directive.ConnectionDiscoverer.retrieveResult(ConnectionDiscoverer.java:202) at io.cdap.wrangler.service.directive.ConnectionDiscoverer.execute(ConnectionDiscoverer.java:134) at io.cdap.wrangler.service.directive.ConnectionDiscoverer.retrieveSample(ConnectionDiscoverer.java:106) at io.cdap.wrangler.service.directive.WorkspaceHandler.lambda$createWorkspace$1(WorkspaceHandler.java:153) at io.cdap.wrangler.service.common.AbstractWranglerHandler.respond(AbstractWranglerHandler.java:202) at io.cdap.wrangler.service.directive.WorkspaceHandler.createWorkspace(WorkspaceHandler.java:136) at io.cdap.wrangler.service.directive.WorkspaceHandlerf667fa1c55614e7a62f467e95f30ef13.lambda$createWorkspace_Lio_cdap_cdap_api_service_http_HttpServiceRequest_Lio_cdap_cdap_api_service_http_HttpServiceResponder_Ljava_lang_String__V(Unknown Source) at io.cdap.cdap.internal.app.runtime.AbstractContext.execute(AbstractContext.java:608) at io.cdap.cdap.internal.app.services.ServiceHttpServer$HandlerDelegatorContext$1.execute(ServiceHttpServer.java:332) at io.cdap.cdap.internal.app.runtime.service.http.AbstractHttpHandlerDelegator.execute(AbstractHttpHandlerDelegator.java:69) at io.cdap.wrangler.service.directive.WorkspaceHandlerf667fa1c55614e7a62f467e95f30ef13.createWorkspace(Unknown Source)
We should throw error message on UI that we don’t support UNION type.