Improve error message in database source when there is a schema mismatch
Activity
Show:
Vinisha Shah September 5, 2019 at 6:37 AM
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Vinisha Shah
Vinisha ShahReporter
Bhooshan Mogal
Bhooshan MogalLabels
Components
Fix versions
Priority
MajorCreated March 30, 2019 at 12:45 PM
Updated July 17, 2020 at 12:28 AM
Resolved September 5, 2019 at 4:53 PM
Not sure if this is a schema mismatch issue. Here's the scenario:
1. I was using an older version of the plugin that does not support date types.
2. One of the input fields is a date, and the UI allows you to specify a date, so I chose it
3. When the pipeline is run, you get the following error message, which indicates that there is a schema mismatch somewhere. As a user, my first reaction was to check the DB, then check my configuration, but there are no clues, since everything seems to be correct. Perhaps this error message should be improved to indicate what the plugin is trying to do when it is matching these schemas, and recommend a solution to users. This could save a lot of debugging time, since this message could potentially send users in completely tangential directions for a fix.
java.lang.IllegalArgumentException: Schema field L_SHIPDATE has type INT but in input record found type LONG at co.cask.hydrator.plugin.DBUtils.getSchemaFields(DBUtils.java:136) ~[1553922916040-0/:na] at co.cask.hydrator.plugin.DBRecord.readFields(DBRecord.java:101) ~[1553922916040-0/:na] at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244) ~[sqoop-1.4.7-hadoop260.jar:na] at co.cask.hydrator.plugin.db.batch.source.DataDrivenETLDBInputFormat$1.nextKeyValue(DataDrivenETLDBInputFormat.java:131) ~[1553922916040-0/:na] at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:207) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) ~[spark-core_2.11-2.2.3.jar:2.2.3] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:150) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:148) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1374) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:163) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:93) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:88) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.scheduler.Task.run(Task.scala:109) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:344) [spark-core_2.11-2.2.3.jar:2.2.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]