We're updating the issue view to help you get more done. 

Hive table partitions are not deleted when a partitioned fileset is truncated

Description

This

Release Notes

None

Activity

Show:
Gary Helmling
March 20, 2015, 10:51 PM

I'm pretty sure this only happens when the dataset is truncated. PartitionedFileSetDataset.dropPartition() does have code to drop the partition from explore when it is called explicitly.

However, in the truncate case, we are just delegating to the wrapped dataset admins. So for the partition data, it is just calling HBaseTableAdmin.truncate(). We will need to add a PartitionedFileSetAdmin to implement the truncate properly and drop and recreate the Hive table.

Andreas Neumann
March 21, 2015, 12:22 AM

That means the truncate() would have to drop all existing partitions form Hive. That can take quite a long time. It could also disable and re-enable for explore. Not sure whether it is possible to do that from the admin method. Will check.

Gary Helmling
March 21, 2015, 1:15 AM
Gary Helmling
March 21, 2015, 1:19 AM
Gary Helmling
March 21, 2015, 6:36 AM

Merged to release/2.8

Assignee

Gary Helmling

Reporter

Julien Guery

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Blocker
Configure