We're updating the issue view to help you get more done. 

Classloading issues between google-cloud and core-plugins

Description

Because both of the plugins export avro and parquet, we run into the familiar classloading issues we've seen with other plugins if plugins from both artifacts are used. For example, a gcs source reading parquet and a file sink writing parquet. I don't believe the problem shows up if sources and sinks are not mixed in a pipeline.

One short term fix could be to shade the relevant classes. Not sure how/if that will work with the maven bundle plugin.

Release Notes

Fixed an issue that caused avro, parquet, and orc classes across file, gcs, and s3 plugins to clash and cause pipeline failures.

Activity

Show:
Albert Shau
October 2, 2018, 3:41 AM

shading appears to work, though I had to move the manifest creation to shade using org.apache.maven.plugins.shade.resource.ManifestResourceTransformer and remove the maven-bundle-plugin. The jar size also increased from 51M to 60M

If everything works after more testing, will probably just shade it. We really need to move these formats to plugins sometime soon to avoid these issues.

Albert Shau
October 2, 2018, 8:12 PM

going to try to do this properly instead of shading.

Albert Shau
October 25, 2018, 9:11 PM
Albert Shau
October 31, 2018, 9:02 PM
Albert Shau
November 2, 2018, 11:38 PM

Assignee

Albert Shau

Reporter

Albert Shau

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Critical
Configure