Dataproc provisioners broken when no labels given
Description
Release Notes
None
Activity
Show:
Arjan Bal August 10, 2023 at 4:43 AM
This Jira doesn’t require a release note because the issue was caused by a change that was not in any release branch only in develop. The bug was introduced while adding this feature for 6.9.2 and the fix also goes into 6.9.2https://cdap.atlassian.net/browse/CDAP-20698
Olivia Emery August 9, 2023 at 8:55 PM
@Arjan Bal Just confirming, does this need a release note? If so, can you add one in this format: "Fixed an issue causing <error> in <versions>."
Albert Shau August 2, 2023 at 10:42 PM
This was caused by https://github.com/cdapio/cdap/pull/15237
Fixed
Pinned fields
Click on the next to a field label to start pinning.
Details
Details
Assignee
Arjan Bal
Arjan BalReporter
Albert Shau
Albert ShauTriaged
Yes
Size
S
Components
Fix versions
Due date
Aug 07, 2023
Priority
BlockerCreated August 2, 2023 at 10:37 PM
Updated August 10, 2023 at 4:43 AM
Resolved August 8, 2023 at 6:19 AM
To reproduce, run any pipeline that uses the existing dataproc profile when there aren’t any labels given. It will fail with:
Fail to start program run program_run:default.dg-trash.6c12e4a1-317b-11ee-988b-66d48fe5f7c3.workflow.DataPipelineWorkflow.0fbcf572-3184-11ee-a144-1a42eee1e305 io.cdap.cdap.runtime.spi.provisioner.dataproc.DataprocRuntimeException: Error while launching job default_dg-trash_DataPipelineWorkflow_0fbcf572-3184-11ee-a144-1a42eee1e305 on cluster dp-2-1. at io.cdap.cdap.runtime.spi.runtimejob.DataprocRuntimeJobManager.launch(DataprocRuntimeJobManager.java:380) at io.cdap.cdap.internal.provision.ProvisioningService$RuntimeJobManagerCallWrapper.launch(ProvisioningService.java:1031) at io.cdap.cdap.internal.app.runtime.distributed.remote.RuntimeJobTwillPreparer.launch(RuntimeJobTwillPreparer.java:185) at io.cdap.cdap.internal.app.runtime.distributed.remote.AbstractRuntimeTwillPreparer.lambda$start$1(AbstractRuntimeTwillPreparer.java:472) at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService$ControllerFactory.lambda$create$0(RemoteExecutionTwillRunnerService.java:613) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: com.google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Not a valid key: "". Only lowercase letters, numbers, and dashes are allowed. The key must start with a lowercase letter and end with a lowercase letter or number. at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:47) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72) at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60) at com.google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97) at com.google.api.core.ApiFutures$1.onFailure(ApiFutures.java:67) at com.google.common.util.concurrent.Futures$4.run(Futures.java:1123) at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:435) at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:900) at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:811) at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:675) at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:563) at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533) at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39) at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) at com.google.api.gax.grpc.ChannelPool$ReleasingClientCall$1.onClose(ChannelPool.java:535) at io.grpc.internal.DelayedClientCall$DelayedListener$3.run(DelayedClientCall.java:463) at io.grpc.internal.DelayedClientCall$DelayedListener.delayOrExecute(DelayedClientCall.java:427) at io.grpc.internal.DelayedClientCall$DelayedListener.onClose(DelayedClientCall.java:460) at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562) at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743) at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722) at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133) ... 3 common frames omitted Suppressed: com.google.api.gax.rpc.AsyncTaskException: Asynchronous task failed at com.google.api.gax.rpc.ApiExceptions.callAndTranslateApiException(ApiExceptions.java:57) at com.google.api.gax.rpc.UnaryCallable.call(UnaryCallable.java:112) at com.google.cloud.dataproc.v1.JobControllerClient.submitJob(JobControllerClient.java:266) at io.cdap.cdap.runtime.spi.runtimejob.DataprocRuntimeJobManager.launch(DataprocRuntimeJobManager.java:346) at io.cdap.cdap.internal.provision.ProvisioningService$RuntimeJobManagerCallWrapper.launch(ProvisioningService.java:1031) at io.cdap.cdap.internal.app.runtime.distributed.remote.RuntimeJobTwillPreparer.launch(RuntimeJobTwillPreparer.java:185) at io.cdap.cdap.internal.app.runtime.distributed.remote.AbstractRuntimeTwillPreparer.lambda$start$1(AbstractRuntimeTwillPreparer.java:472) at io.cdap.cdap.internal.app.runtime.distributed.remote.RemoteExecutionTwillRunnerService$ControllerFactory.lambda$create$0(RemoteExecutionTwillRunnerService.java:613) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) ... 3 common frames omitted Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: Not a valid key: "". Only lowercase letters, numbers, and dashes are allowed. The key must start with a lowercase letter and end with a lowercase letter or number. at io.grpc.Status.asRuntimeException(Status.java:535) ... 17 common frames omittedThis is because
DataprocUtils.parseKeyValueConfig("", ...)returns a Map with empty string as a key and value.