Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Partitioned file set should send a warning if not all partitions are in Hive after enabling explore

Description

After enabling explore for a dataset, the user expects that all partitions are available in Hive. However, enableExplore only creates an empty Hive table without partitions.

Release Notes

None

Activity

Show:

NitinMJune 9, 2020 at 1:37 AM

Closing jiras older than 2+ years.

Andreas NeumannMay 12, 2017 at 10:48 PM

This can't be done in 4.2, because it depends on a change in dataset framework so that dataset admin ops have access to transactions.

Andreas NeumannDecember 8, 2016 at 6:11 AM

Actually, this is not possible, because the ExploreTableManager does not have a way to start a transaction. That, however, is required to list the partitions. This can only be done after the dataset framework has been refactored to allow dataset admin operations to have transactions.

Andreas NeumannNovember 21, 2016 at 11:19 PM

For now we will emit a warning that states that some partitions may not be available in Hive.

Andreas NeumannSeptember 8, 2016 at 7:57 PM

I think we need a way to do this as a long-running process. In 3.5 we added a worker that can be used to sync the Hive table with all partitions. This is still something the user has to include in his app and then start the worker. The dataset framework currently has no way to perform a long-running maintenance task on a dataset.

Won't Fix
Pinned fields
Click on the next to a field label to start pinning.

Details

Assignee

Reporter

Affects versions

Components

Priority

Created July 8, 2016 at 2:34 PM
Updated June 9, 2020 at 1:37 AM
Resolved June 9, 2020 at 1:37 AM