Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Python plugin fails when when used in Native mode.

Description

Python plugin fails when used in the Native mode. Following error is seen in the logs

Release Notes

None

Activity

Show:

Ankit JainMarch 27, 2024 at 7:42 AM
Edited

Used these configurations in python-transform 2.3.0 in native mode.

Pipeline was successful when run on dataproc cluster.

Sagar KapareOctober 15, 2021 at 6:06 PM

For Native mode we have to provide PYTHONPATH config containing path to the py4j-<version>-src.zip For example on my Hadoop cluster zip was located at following path: /usr/lib/spark/python:/usr/lib/spark/python/lib/py4j-0.10.9-src.zip:$PYTHONPATH 

Even after giving PYTHONPATH, plugin now fails with exception mentioned in the bug.

We tried different memory configurations for the job based on some online research but didnt help.

Unresolved
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Labels

Priority

More fields

Created October 15, 2021 at 6:03 PM
Updated March 27, 2024 at 7:43 AM