Issues
- Create OAuth Provider reusing existing secrets in GCP Secret ManagerCDAP-21006Resolved issue: CDAP-21006Vaibhav Sethi
- String sorting is not supported by NoSQL StructuredTableCDAP-21005
- Migrate scm metadata from app_spec to new metadata tab;eCDAP-21004Resolved issue: CDAP-21004Samik Gupta
- Test Windows Share Copy Action plugin as it might be broken.CDAP-21003
- GCS Source fails with NO INPUT FOUND with input path ending with a directory name starting with hypen _CDAP-21002Sanket Sahu
- [Wrangler] directives which update the schema do not work as expectedCDAP-21001
- Pipeline run gets stuck in stopping state endlesslyCDAP-21000Avinash Achar
- Schedule Client not correctly encoding the URL resulting in pre-upgrade job failureCDAP-20999Resolved issue: CDAP-20999Ankit Jain
- Wrangle option available in the Wrangler Plugin not reflecting the wrangle changes in the output schemaCDAP-20998Sagar Ahire
- Remove access to CDAP_Auth_Token in frontend and mark it httpOnlyCDAP-20997Resolved issue: CDAP-20997Ganesh Sahoo
- CDAP Controller fails to reconcile objects when instance name length exceeds 28 characters.CDAP-20996Resolved issue: CDAP-20996Ankit Jain
- Write TMS tests which was previously written in HBASECDAP-20995
- CVE-2019-20444 in netty-3.6.6.Final.jarCDAP-20994
- Periodically refresh sync status for namespace appsCDAP-20993
- Migrate source control metadata from appSpec table to new tableCDAP-20992
- Add API implementation for Namespace and Repository source control metadataCDAP-20991
- Add new database schemas, API schema and store methodsCDAP-20990
- Add implementation for Namespace and Repository pipelines APICDAP-20989Resolved issue: CDAP-20989Adrika Gupta
- Scheduling pipelines with maxConcurrentRuns does not work as expectedCDAP-20988Resolved issue: CDAP-20988Ganesh Sahoo
- OAuthHandler should return response details in the exception other than response code when request failsCDAP-20987Resolved issue: CDAP-20987Priyanshu Bhardwaj
- We should properly log an error instead of calling printstacktaceCDAP-20986Resolved issue: CDAP-20986Avinash Achar
- Replication logs sometimes show null error message on top of UI without detailsCDAP-20985
- Add all supported dataproc regions in custom profile optionsCDAP-20984Resolved issue: CDAP-20984Vaibhav Sethi
- Compute Profile cannot be deleted after a new version of Pipeline is created after EditingCDAP-20983Avinash Achar
- Remove HBASE dependency from CDAP code baseCDAP-20982Sanket Sahu
- Add SCM Sync Status featureCDAP-20981
- Task worker requests are getting retried even for internal errorsCDAP-20980Resolved issue: CDAP-20980Ankit Jain
- Add data-testids for UI elements in the control center pageCDAP-20979Resolved issue: CDAP-20979Ganesh Sahoo
- Add support for managing data-testids in cdap-ui centrallyCDAP-20978Resolved issue: CDAP-20978Ganesh Sahoo
- Add data-testids for UI elements as reported by the test engineersCDAP-20977
- Add support for getting unique xpaths to elements based on data-testids in the chrome extensionCDAP-20976Resolved issue: CDAP-20976Ganesh Sahoo
- Add chrome extension for reporting missing data-testids easilyCDAP-20975Resolved issue: CDAP-20975Ganesh Sahoo
- Make e2e tests locate UI elements in a stable manner by relying only on data-testids.CDAP-20974Ganesh Sahoo
- Add bitbucket support for SCMCDAP-20973Resolved issue: CDAP-20973Samik Gupta
- Delay in able to delete the compute profile after removing its association from the pipelines and disabling itCDAP-20972
- Some invalid condition branches are not validatedCDAP-20971Ankit Jain
- Plugin version not updating in every plugin in studioCDAP-20970Ganesh Sahoo
- Add gitlab support for SCMCDAP-20969Resolved issue: CDAP-20969Samik Gupta
- Not enough information when connection macro evaluation fails due to timeoutCDAP-20968Resolved issue: CDAP-20968Ankit Jain
- k8s deployments managed by appfabric don't get recreated when image is changed in cdap-operatorCDAP-20967
- Bug in data plane audit logging class loaderCDAP-20966Resolved issue: CDAP-20966Sanket Sahu
- Pipeline fails due to Twill Program failure when appfabric restarts as soon as pipeline execution reaches STARTING stateCDAP-20965
- Update google-cloud-bom in Runtime Dataproc Extension to latestCDAP-20964Resolved issue: CDAP-20964Ankit Jain
- CDAP UI - Add missing data test ids in system administration UICDAP-20963Radhika Vissamsetty
- TMS message publish do not retry on socket timeoutCDAP-20962Samik Gupta
- LOG current yarn memory usage when an Application is Submitted or accepted.CDAP-20961Avinash Achar
- Implement backend changes to support Sync StatusCDAP-20960Resolved issue: CDAP-20960
- CDAP on k8s pipelines fail when reading from HDFSCDAP-20959Resolved issue: CDAP-20959Prashant Jaikumar
- Add metrics which expose number of available appworkersCDAP-20958
- Add oAuth2 scopes support in Remote AuthenticatorCDAP-20957Resolved issue: CDAP-20957Ankit Jain
- No retry when system worker is temporarily unavailable due to error 5xxCDAP-20956Resolved issue: CDAP-20956Ankit Jain
- Request to the metadata server that lives on the VM fails on first attempt without retriesCDAP-20955Resolved issue: CDAP-20955Dvij Hirenkumar Joshi
- Error while trying to overwrite non cacheable files while uploading to GCSCDAP-20954Resolved issue: CDAP-20954Ankit Jain
- HTTP Plugin (Source and Sink) || Discrepancy in Measurement unit for ReadTimeout and connect TimeoutCDAP-20953Pushpender Saini
- Spark SQL relation breaks on Argument SetterCDAP-20952Resolved issue: CDAP-20952Sanket Sahu
- Assessment Handler fails with NPE when running operations in task worker.CDAP-20951Resolved issue: CDAP-20951Ankit Jain
- Have an INFO log when namespaces are deletedCDAP-20949
- BigQuery Delta Plugin: Unable to fetch maximum sequence number from existing tablesCDAP-20948Resolved issue: CDAP-20948Shivam Varshney
- Skip running metadata consumer subscriber if no extensions are enabledCDAP-20947Resolved issue: CDAP-20947Ankit Jain
- Field lineage should not be published to message service if the Data Lineage Integration Feature is disabledCDAP-20946Resolved issue: CDAP-20946
- Add UI modal for a new field in WRANGLER UI for XMLTOJSONCDAP-20945Resolved issue: CDAP-20945Ganesh Sahoo
- Validate Reference name in MultiTableDatabase pluginCDAP-20944
- MetadataConsumerSubscriberService and MetadataSubscriberService share same topic pointerCDAP-20943Resolved issue: CDAP-20943Ankit Jain
- cdap.twill.app is not normalized correctly when length increases greater than 63 when running in k8s modeCDAP-20942Resolved issue: CDAP-20942Prashant Jaikumar
- Retry socket timeout exceptions while publishing metricsCDAP-20941Avinash Achar
- Add imageVersion in CDAP Provisioner metricCDAP-20940Resolved issue: CDAP-20940Ankit Jain
- Unable to edit pipeline json when a stage name contains a periodCDAP-20939Radhika Vissamsetty
- Messaging pod keeps crashing for OOM due to large message variable in memoryCDAP-20938Avinash Achar
- RunRecord TTL service should also adjust run countsCDAP-20937Resolved issue: CDAP-20937Ankit Jain
- [Code cleanup] remove unused code in cdap-uiCDAP-20936Ganesh Sahoo
- Pipeline trigger fails at undefined runtime arguments because of slow asynchronous metadata messageCDAP-20935
- Wrangle XML to Json parser incorrectly converts string valueCDAP-20934Resolved issue: CDAP-20934Sanket Sahu
- CDAP on k8s: cdap.twill.app label value should not end in '-'CDAP-20933Resolved issue: CDAP-20933Prashant Jaikumar
- The committer id in scm is base64encoded in rbac instanceCDAP-20932Resolved issue: CDAP-20932Samik Gupta
- Listing Applications Uses Dataset Entity for Permission ChecksCDAP-20931Resolved issue: CDAP-20931Ankit Jain
- CDAP Services Do Not Properly Set URI Scheme During Discoverable AnnouncementCDAP-20930Resolved issue: CDAP-20930Arjan Bal
- Executors fail due to error: Netty direct memory should at least be bigger than 'spark.network.maxRemoteBlockSizeFetchToMem'CDAP-20929
- Unable to evaluate connection macros when internal router is enabledCDAP-20928Resolved issue: CDAP-20928Arjan Bal
- Help content area should show without scroll in service accounts edit dialogCDAP-20927Resolved issue: CDAP-20927Radhika Vissamsetty
- Add Metric for Tracking Number of Concurrent UsersCDAP-20926
- [UI] cleanup test page /ns/:namespace/kitchenCDAP-20925Resolved issue: CDAP-20925Ganesh Sahoo
- Unable to deploy pipeline in instance if internal router is enabled.CDAP-20924Resolved issue: CDAP-20924Arjan Bal
- Same pipeline failed on 6.9.2 but succeeded on 6.8.3CDAP-20923
- Reuse Spark Class Loader while loading artifacts in appfabric and task workersCDAP-20922Terence Yim
- Restrict single git operations when multi operations are runningCDAP-20921Samik Gupta
- We should enable spark autotuning for predefined autoscalingCDAP-20920Resolved issue: CDAP-20920Vitalii Tymchyshyn
- Vulnerability issue in glibcCDAP-20919Resolved issue: CDAP-20919Arjan Bal
- Add more git metadataCDAP-20918Resolved issue: CDAP-20918Samik Gupta
- UX improvements: Add sync status, more providers and better error messagesCDAP-20917Ganesh Sahoo
- Add metrics for pipeline pull/push countCDAP-20916Resolved issue: CDAP-20916Samik Gupta
- Executing Native on Hadoop throws NoSuchMethodExceptionCDAP-20915Resolved issue: CDAP-20915Vitalii Tymchyshyn
- Update error message in deploy when plugin is missingCDAP-20914Resolved issue: CDAP-20914Samik Gupta
- Pipeline run fails after appfabric is restartedCDAP-20913Resolved issue: CDAP-20913Prashant Jaikumar
- replication retry of LOAD in STAGING table may give duplicatesCDAP-20912Resolved issue: CDAP-20912
- ClassCastException when doing unions with DatasetCollectionCDAP-20911Resolved issue: CDAP-20911Vitalii Tymchyshyn
- Unclosed URLClass loaders that can leak resourcesCDAP-20910Resolved issue: CDAP-20910Arjan Bal
- CDAP on Hadoop Fails to Start due to Dependency ConflictCDAP-20909Resolved issue: CDAP-20909Dennis Li
- Pipeline Sync Issue: Pushing Pipelines with no change results in an unexpected new commitCDAP-20908Samik Gupta
- Pipeline Sync Issue: Updated Descriptions Not Reflecting After Pulling New VersionsCDAP-20907Resolved issue: CDAP-20907Samik Gupta
- Endtime not getting populated for operationsCDAP-20906Resolved issue: CDAP-20906Samik Gupta
Finished loading more issues at the top of the list.
100 of
support submitting jobs on dataproc driver node groups cluster
Fixed
Description
Release Notes
Added support for Dataproc driver node groups. To use Dataproc driver node groups, when you create the Dataproc cluster, configure the following properties:
yarn:yarn.nodemanager.resource.memory.enforced=false
yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME
Note: The single quotes are important in the property when using gcloud CLI to create the cluster ('yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME') so that the shell doesn't try to resolve the $ locally before submitting.
Created January 18, 2023 at 10:53 AM
Updated September 13, 2023 at 10:01 AM
Resolved February 1, 2023 at 5:05 PM
Activity
Show:
While creating the Dataproc cluster, properties
yarn:yarn.nodemanager.resource.memory.enforced=falseandyarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOMEare needed.Note - The single quotes are important in the property when using gcloud CLI to create the cluster (
'yarn:yarn.nodemanager.admin-env.SPARK_HOME=$SPARK_HOME') so that the shell doesn't try to resolve the$locally before submitting.