Tag the Elasticsearch index with the CDAP version

Description

For future upgrades, it is useful to know what version of CDAP was used to create an index. That will help determine whether an upgrade step or migration is required. 

Release Notes

None

Activity

Show:
Andreas Neumann
March 12, 2019, 4:12 PM

Turns out that Elastic's way to associate metadata is a mapping for the special field "_meta". See https://www.elastic.co/guide/en/elasticsearch/reference/6.6/mapping-meta-field.html

PR: https://github.com/cdapio/cdap/pull/11191

Andreas Neumann
March 11, 2019, 8:00 PM
Edited

There are two ways to possibly do this:

  1. add a custom property to the index settings that can be retrieved. It appears that Elasticsearch does not allow this, it rejects any settings that are not known to the indexer with illegal argument exceptions. There may still be a way to do this as a "description" or other tag, research is ongoing. https://discuss.elastic.co/t/is-it-possible-to-annotate-an-index-with-custom-properties/171829

  2. store a special document that has this information. This document would have to be stored but not indexed, or in some other way be excluded from searches. 

Fixed

Assignee

Andreas Neumann

Reporter

Andreas Neumann

Labels

None

Docs Impact

None

UX Impact

None

Components

Fix versions

Affects versions

Priority

Major