Introduction 

Use-case

User Stories

• User should have ability to specify single or composite field for sorting the records in the output
• User should be able to specify field (basic type) from a nested structure for sorting the records
• User should specify for each field how the records should be sorted
• User can specify only basic types - String, Int, Long, Short, Float, Double, Byte as key, in case any other types are specified then error is thrown to notify the user

Example

Following is a simple example showing how Order By would work.

Input

• Input Schema
  • First Name, String
  • Last Name, String
  • Age, Int
  • Zip Code, Long
• Sort by 
  • Last Name, Ascending
  • Age, Ascending
  • Zipcode, Descending
• Output Schema
  • First Name, String
  • Last Name, String
  • Age, Int
  • Zip Code, Long

Output is as follows

Implementation Tips

• Investigate how ‘Group Comparator’ and ‘Sort Comparator’ work together and be used to achieve the functionality for this plugin.
• Build a simple map-reduce program to show understand how the above functionality work — Implement Sort Comparator using StructuredRecord
• If the above works, then Data Pipeline Application Template need to be modified to the set the sort class comparator and this shouldn’t affect the other plugins.

Design