Improve error message in database source when there is a schema mismatch

Description

Not sure if this is a schema mismatch issue. Here's the scenario:
1. I was using an older version of the plugin that does not support date types.
2. One of the input fields is a date, and the UI allows you to specify a date, so I chose it
3. When the pipeline is run, you get the following error message, which indicates that there is a schema mismatch somewhere. As a user, my first reaction was to check the DB, then check my configuration, but there are no clues, since everything seems to be correct. Perhaps this error message should be improved to indicate what the plugin is trying to do when it is matching these schemas, and recommend a solution to users. This could save a lot of debugging time, since this message could potentially send users in completely tangential directions for a fix.

java.lang.IllegalArgumentException: Schema field L_SHIPDATE has type INT but in input record found type LONG at co.cask.hydrator.plugin.DBUtils.getSchemaFields(DBUtils.java:136) ~[1553922916040-0/:na] at co.cask.hydrator.plugin.DBRecord.readFields(DBRecord.java:101) ~[1553922916040-0/:na] at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:244) ~[sqoop-1.4.7-hadoop260.jar:na] at co.cask.hydrator.plugin.db.batch.source.DataDrivenETLDBInputFormat$1.nextKeyValue(DataDrivenETLDBInputFormat.java:131) ~[1553922916040-0/:na] at org.apache.spark.rdd.NewHadoopRDD$$anon$1.hasNext(NewHadoopRDD.scala:207) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37) ~[spark-core_2.11-2.2.3.jar:2.2.3] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:439) ~[scala-library-2.11.8.jar:na] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:150) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$4.apply(SparkHadoopMapReduceWriter.scala:148) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1374) ~[spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$.org$apache$spark$internal$io$SparkHadoopMapReduceWriter$$executeTask(SparkHadoopMapReduceWriter.scala:163) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:93) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.internal.io.SparkHadoopMapReduceWriter$$anonfun$3.apply(SparkHadoopMapReduceWriter.scala:88) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.scheduler.Task.run(Task.scala:109) [spark-core_2.11-2.2.3.jar:2.2.3] at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:344) [spark-core_2.11-2.2.3.jar:2.2.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_181] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_181] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]

Release Notes

None

Cask Community OSS Builds

Activity

Show:

Vinisha Shah September 5, 2019 at 6:37 AM

Fixed
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Components

Fix versions

Priority

Created March 30, 2019 at 12:45 PM
Updated July 17, 2020 at 12:28 AM
Resolved September 5, 2019 at 4:53 PM