Checklist
The current metadata search does not allow for users to search for entities by dates. The feature presented in this design doc aims to allow users to query entities by their creation date as well as accepted forms of user-defined date properties. A new search syntax will be introduced to facilitate this. The feature will be implemented through Elasticsearch.
Allow users to search for entities based on a date property.
As a pipeline developer, I want to be able to see entities that were created before or after a certain date.
As a pipeline developer, I want to be able to see entities that I made between one date and another.
As a pipeline developer, I want to be able to see entities that are set for deployment on a specific date.
As a pipeline developer, I defined my own property with a date value, X, and now I want to get entities whose value X is after a specific date.
To accommodate for this type of search, a new syntax will be introduced. Relevant methods in the QueryParser
class will be modified to be able to detect this syntax, and the QueryTerm
class will include a new field to indicate if the term is a date search.
Modify the current search method such that when a date query is supplied, entities created with a date-value condition are found.
In case there is a similarity between date-search syntax and other user-created metadata labels, the program will conduct a regular string search for the query in addition to the date search.
Currently the QueryBuilder
object used for searches is a BoolQueryBuilder
, which has been used to distinguish which terms are required and optional in the search results. ElasticSearch also has a RangeQueryBuilder
which will be applied to search for creation times as a range over an hour, day, or multiple days.
A date query will be assumed to be a required query and so the RangeQueryBuilder
object can be added into the main BoolQueryBuilder
’s required searches. Since the RangeQueryBuilder
will be embedded in the BoolQueryBuilder
, there will still be only one search object for the entirety of the user's query.
The user’s search date / time will be converted from the predetermined UI syntax into a long to represent the millis since the epoch (unix time). This is the format that is currently used to store creation time of entities and is supported by Elasticsearch.
The metadata Property
class will now include an additional field called date
which will be a Long
. If a user defines a metadata property with a date-format value, the value will be parsed into unix time and stored as date
. This will allow for a date search over user defined dates.
QueryTerm
class with an enum: SearchType
createMainQuery()
in ElasticsearchMetadata
so that it creates a date subquery when necessaryQueryTerms
, if it is a DATE SearchType, call dateQuery = createDateQuery
and add the result to create a new range query builder. termQuery = createTermQuery
as earlier, for possible conflict reasons as mentioned in the design section.dateQuery
and termQuery
) into the boolQuery
search, checking, as in the current implementation, to see if the query is required or optional. createDateQuery()
QueryTerm
. The QueryTerm
’s term (string) will be parsed to extract the date range that is meant to be searched for. The extracted range will be in the form of a long representing unix time. Introduce the following fields:
public enum SearchType { STANDARD, DATE }
private SearchType searchType;
private Long date;
Accommodate a new constructor that takes a SearchType
as a parameter. Existing constructor will default the searchType
to be SearchType.STANDARD
Create a check for the new keywords (using an if-statement, similar to checking for required terms). If the term contains the new keywords, return a QueryTerm
indicating SearchType.DATE
in its construction, otherwise SearchType
should be STANDARD
.
Users create their date query starting with "DATE:"
The colon can be followed by a date-field name to search over, followed by another colon
The date can immediately follow "DATE:" or it can follow the field name and colon
Examples searches and their results:
“DATE:2019-07-01” or "DATE:==2019-07-01"
“DATE:<2019-07-01”
Returns entities created before July 1st, 2019 or with equivalent date property values.
“DATE:<=2019-07-01”
Returns entities created on or before July 1st, 2019 or with equivalent date property values.
“DATE:>2019-07-01”
Returns entities created after July 1st, 2019 or with equivalent date property values.
“DATE:>=2019-07-01”
Returns entities created on or after July 1st, 2019 or with equivalent date property values.