Issues
- Failure during program launch doesn't show up in program logsCDAP-14842Resolved issue: CDAP-14842Mikkin Patel
- Remote runtime stuckCDAP-14776Resolved issue: CDAP-14776Albert Shau
- Metadata search will be broken if the search result contains schedule type with authorization enabledCDAP-14705Yaojie Feng
- Columns without data (nulls) doesn't propagated in output schemaCDAP-14597Mikkin Patel
- Import pipeline button overlaps textCDAP-14564Priyanka Nambiar
- Realtime/Spark Streaming pipelines ignore config and always run with checkpointing onCDAP-14558Resolved issue: CDAP-14558Jon Gray
- UI should not force you to update pipeline plugin versionsCDAP-14555Resolved issue: CDAP-14555Edwin Elia
- Unexpected WARN log messages seen when running pipelines on dataproc on a clusterCDAP-14547Priyanka Nambiar
- BigQuery plugins need an extra job project propertyCDAP-14542Resolved issue: CDAP-14542Albert Shau
- kafka-plugin in development branch doesn't load properly in cdap 5.1.0 - It is not shown in the UICDAP-14538Resolved issue: CDAP-14538Albert Shau
- BigQuerySink sink schema validation should not depend on table existenceCDAP-14532Resolved issue: CDAP-14532Vinisha Shah
- BigQuery Sink NPE if schema is mismatchedCDAP-14531Resolved issue: CDAP-14531Vinisha Shah
- Clicking on About CDAP in Log viewer does not do anything.CDAP-14527Resolved issue: CDAP-14527Edwin Elia
- Welcome modal header image doesn't span whole modal widthCDAP-14518Ajai Narayanan
- Scheduler should emit metrics about the job queueCDAP-14517Priyanka Nambiar
- After upgrading cluster from 5.0 to 5.1, no longer able to create pipelines.CDAP-14515Resolved issue: CDAP-14515Rohit Sinha
- Update docs to drop Java 7 supportCDAP-14514Resolved issue: CDAP-14514Rohit Sinha
- Split cdap-security-extension repoCDAP-14513Rohit Sinha
- Scheduler should log all job creation, execution and cancellationCDAP-14512Priyanka Nambiar
- commons-lang and few other jar missing from ranger pluginsCDAP-14507Rohit Sinha
- Configuration changes/updates for CSD and AmbariCDAP-14495Resolved issue: CDAP-14495Matt Wuenschel
- Remove Datapacks from Hub UICDAP-14471Resolved issue: CDAP-14471Edwin Elia
- Applications/Pipeline tags should not have 'cdap' in themCDAP-14388Rohit Sinha
- CDAP navigation bar - add caret next to admin icon, add spatial definition around Namespace drop downCDAP-14380Resolved issue: CDAP-14380Ajai Narayanan
- GCSSource does not work if Regex Path Filter is not specifiedCDAP-14353Resolved issue: CDAP-14353Rohit Sinha
- Spanner source created through DataPrep has incorrect labelCDAP-14351Resolved issue: CDAP-14351Bhooshan Mogal
- Exploring parquet fileset with timestamp throws ClassCastExceptionCDAP-14346Resolved issue: CDAP-14346Priyanka Nambiar
- ExploreTest is consistently failing on MapR4.1CDAP-14342Resolved issue: CDAP-14342Vinisha Shah
- CrossNSAppImpersonationTest can only run with spark2CDAP-14328Resolved issue: CDAP-14328Rohit Sinha
- Strange warnings in dataprep service logCDAP-14326Resolved issue: CDAP-14326Vinisha Shah
- Strange error from Dataprep file connectionCDAP-14325Resolved issue: CDAP-14325Vinisha Shah
- SparkTest.testStreamFormatSpec() is flakyCDAP-14323Resolved issue: CDAP-14323Albert Shau
- upgrade tool fails with missing propertyCDAP-14314Resolved issue: CDAP-14314Vinisha Shah
- Cyclic HTTP call loop in Dataset ServiceCDAP-14298Resolved issue: CDAP-14298Andreas Neumann
- Improve theme file spec to only have one logo field instead of twoCDAP-14297Resolved issue: CDAP-14297Tony Bach
- Start dataprep bootstrap step logs a warnCDAP-14295Resolved issue: CDAP-14295Albert Shau
- KafkaOffsetResolverTest is flakyCDAP-14276Resolved issue: CDAP-14276Terence Yim
- Unable to wrangle mysql tableCDAP-14262Resolved issue: CDAP-14262Vinisha Shah
- LevelDBQueueTest is flakyCDAP-14254Resolved issue: CDAP-14254Ali Anwar
- Move cask.common.http back into cdapCDAP-14240Resolved issue: CDAP-14240Andreas Neumann
- Race condition in Netty can cause HTTP time outCDAP-14201Resolved issue: CDAP-14201Terence Yim
- Instrument all HTTP handlers and clients to trace-log all requestsCDAP-14193Priyanka Nambiar
- core scheduler, app fabric and dataset service should use fewer threads in testsCDAP-14189Resolved issue: CDAP-14189Andreas Neumann
- Localize only artifact jar needed by a programCDAP-14188Priyanka Nambiar
- All code should use the same way to make HTTP requestsCDAP-14180Bhooshan Mogal
- Flaky unit testsCDAP-14178Bhooshan Mogal
- Column selection dropdown in data preparation gets cut off partiallyCDAP-14174Ajai Narayanan
- Fix 5.1 docs buildCDAP-14170Resolved issue: CDAP-14170Sreevatsan Raman
- Always display current dataset schema in FLL regardless of time period selectedCDAP-14168Resolved issue: CDAP-14168Rohit Sinha
- Field Lineage fails for a Join if the join key is not written to a sinkCDAP-14167Resolved issue: CDAP-14167Yaojie Feng
50 of 72
Failure during program launch doesn't show up in program logs
Fixed
Description
Release Notes
None
Attachments
1
relates to
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Mikkin Patel
Mikkin Patel
Reporter
Ali Anwar
Ali Anwar
Labels
Affects versions
Components
Fix versions
Priority
MajorCreated February 1, 2019 at 1:57 AM
Updated December 11, 2020 at 8:04 PM
Resolved December 11, 2020 at 8:04 PM
Activity
Show:
Ali Anwar September 9, 2019 at 9:23 PM
Ok, reopened this for now then, since it is not a duplicate to that other issue.
Albert Shau September 9, 2019 at 8:10 PM
This particular issue seems to be different than CDAP-15757. This was caused by an errors during provisioning being logged as debug instead of warn or error. The other jira caused all logs from spark/mapreduce on the remote cluster to be lost, regardless of the level.
Ali Anwar September 7, 2019 at 1:21 AM
This has the same symptoms as in CDAP-15757, and I believe that it is the resolved by the fix in that bug. Marking as duplicate.
@Albert Shau Let me know if you think otherwise.
I ran a program and it failed. The only logs I saw in program logs were DEBUG-level logs about provisioning.
I had to look into the cdap-debug.log file to gain insight about why the program failed.
Below is a relevant snippet from cdap-debug.log:
2019-01-31 22:57:07,397 - DEBUG [LocalLogProcessor-cdap:c.c.c.l.m.FileMetaDataWriter@60] - Writing meta data for logging context default:CampaignPipeline_v2:DataPipelineWorkflow: with startTimeMs 1548975427394 sequence Id 1548975427395 and location /opt/cdap/sandbox-5.1.3/data/logs/default/2019-01-31/CampaignPipeline_v2/DataPipelineWorkflow/1548975427395.avro2019-01-31 22:57:07,407 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@106] - Executing PROVISION subtask REQUESTING_CREATE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:57:09,407 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@110] - Completed PROVISION subtask REQUESTING_CREATE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:57:09,417 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@106] - Executing PROVISION subtask POLLING_CREATE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:30,242 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@110] - Completed PROVISION subtask POLLING_CREATE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:30,252 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@106] - Executing PROVISION subtask INITIALIZING for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:32,534 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@110] - Completed PROVISION subtask INITIALIZING for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:32,542 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@125] - Completed PROVISION task for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:33,516 - ERROR [program.status:c.c.c.a.r.AbstractProgramRuntimeService@173] - Exception while trying to run programjava.io.FileNotFoundException: /opt/cdap/sandbox-5.1.0/data/namespaces/system/artifacts/google-cloud/0.11.1-SNAPSHOT.b22caaaa-2cde-4363-afc0-4f30889c41c0.jar (No such file or directory) at java.io.FileInputStream.open0(Native Method) ~[na:1.8.0_191] at java.io.FileInputStream.open(FileInputStream.java:195) ~[na:1.8.0_191] at java.io.FileInputStream.<init>(FileInputStream.java:138) ~[na:1.8.0_191] at org.apache.twill.filesystem.LocalLocation.getInputStream(LocalLocation.java:86) ~[org.apache.twill.twill-common-0.13.0.jar:0.13.0] at co.cask.cdap.common.io.Locations.linkOrCopy(Locations.java:228) ~[na:na] at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.copyArtifact(AbstractProgramRuntimeService.java:305) ~[na:na] at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.createPluginSnapshot(AbstractProgramRuntimeService.java:284) ~[na:na] at co.cask.cdap.app.runtime.AbstractProgramRuntimeService.run(AbstractProgramRuntimeService.java:160) ~[na:na] at co.cask.cdap.internal.app.services.ProgramLifecycleService.startInternal(ProgramLifecycleService.java:525) [na:na] at co.cask.cdap.internal.app.services.ProgramNotificationSubscriberService.lambda$handleClusterEvent$0(ProgramNotificationSubscriberService.java:425) [na:na] at co.cask.cdap.internal.app.services.ProgramNotificationSubscriberService.postProcess(ProgramNotificationSubscriberService.java:155) [na:na] at co.cask.cdap.messaging.subscriber.AbstractMessagingSubscriberService.runTask(AbstractMessagingSubscriberService.java:185) ~[na:na] at co.cask.cdap.common.service.AbstractRetryableScheduledService.runOneIteration(AbstractRetryableScheduledService.java:155) ~[na:na] at com.google.common.util.concurrent.AbstractScheduledService$1$1.run(AbstractScheduledService.java:170) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractScheduledService$CustomScheduler$ReschedulableCallable.call(AbstractScheduledService.java:355) ~[com.google.guava.guava-13.0.1.jar:na] at com.google.common.util.concurrent.AbstractScheduledService$CustomScheduler$ReschedulableCallable.call(AbstractScheduledService.java:321) ~[com.google.guava.guava-13.0.1.jar:na] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_191] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) ~[na:1.8.0_191] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ~[na:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_191] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_191] at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_191]2019-01-31 22:59:33,548 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@76] - Starting DEPROVISION task for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:33,559 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@106] - Executing DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:34,113 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@110] - Completed DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 22:59:34,123 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@106] - Executing DEPROVISION subtask POLLING_DELETE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 23:00:00,001 - DEBUG [stream-cleanup:c.c.c.d.s.s.LocalStreamFileJanitorService$1@65] - Execute stream file cleanup.2019-01-31 23:00:00,002 - DEBUG [stream-cleanup:c.c.c.d.s.s.LocalStreamFileJanitorService$1@69] - Completed stream file cleanup.2019-01-31 23:00:00,002 - DEBUG [stream-cleanup:c.c.c.d.s.s.LocalStreamFileJanitorService$1@80] - Schedule stream file cleanup in 299998 ms2019-01-31 23:00:51,054 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@110] - Completed DEPROVISION subtask POLLING_DELETE for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.2019-01-31 23:00:51,064 - DEBUG [provisioning-service-3:c.c.c.i.p.t.ProvisioningTask@125] - Completed DEPROVISION task for program run program_run:default.CampaignPipeline_v2.-SNAPSHOT.workflow.DataPipelineWorkflow.87a0d2d4-25ab-11e9-a159-024265a1ac28.