We're updating the issue view to help you get more done. 

TPFS sinks should have an option to delete partitions older than some amount of time after they are done

Description

One common use case is somebody is dumping snapshots of a database into a tpfs. But they don't want the tpfs to grow infintely, but only want to keep x days of data in the dataset. This setting would allow them to cap how many days of data are in it by deleting older partitions after the pipeline run.

Release Notes

Added a configuration property to Hydrator TPFS sinks that will clean data that is older than a threshold amount of time.

Activity

Show:
Albert Shau
February 26, 2016, 7:10 PM
Andreas Neumann
February 26, 2016, 8:03 PM

In a way, we are implementing a dataset feature (TTL) in the sink here. Is that the correct approach?

Albert Shau
February 26, 2016, 10:02 PM

I would say its a useful application stopgap that is good enough for a lot of use cases. Since proper fileset ttl is much more involved, I think it makes sense to have this now until the platform feature is available.

Andreas Neumann
February 26, 2016, 10:12 PM

that works for me. Do we have a Jira for supporting TTL in FileSets?

Albert Shau
February 26, 2016, 10:46 PM

opened

Assignee

Albert Shau

Reporter

Albert Shau

Labels

Docs Impact

None

UX Impact

None

Components

Fix versions

Priority

Major
Configure