Introduction

Use-Case

• Transform input features(tokens in array form) into n-grams using parameter for number of terms in each n-gram.
• Transformed output will be an array of n-grams where each n-gram is represented by a space-delimited string of n consecutive words.

User Stories

• As a Hydrator user,I want to transfom input features data in a column from source schema into output schema which will have a single column having n-gram data.
• As a Hydrator user I want to have configuration for specifying the column name from input schema on which transformation has to be performed.
• As a Hydrator user I want to have configuration to specify the no of terms which would be used for transformation of input features into n-grams.
• As a Hydrator user I want to have configuration to specify output column name wherein ngrams will be emitted.

Conditions

Example

Input source:

NGramTransform:

Mandatory inputs from user:

Output:

End to End Example pipeline:       

Input source:

Mandatory inputs from user:

• • Column on which tokenization to be done:”sentence”
  • Delimiter for tokenization:” ”
  • Output column name for tokenized data:”tokens”

NGramTransform:

• A bio data scientist wants to  study the sequence of the nucleotides using the input stream of DNA sequencing to identify the bonds.
  The input Stream contains the DNA sequence eg AGCTTCGA. The output contains the bigram sequence AG, GC, CT, TT, TC, CG, GA

  Input source: 

  DNASequence
  AGCTTCGA

  Mandatory inputs from user:NGramTransform: 

  • Field to be used to transform input features into n-grams:
”tokens”
• • ”DNASequence”
  • Number of terms in each n-gram:”2”
  • Transformed field for sequence of n-gram:
”ngrams”

TPFSAvro Output

Design

Properties:

• **fieldToBeTransformed:** Column to be used to transform input features into n-grams.

• **noOfTerms:** No of terms in each n-gram.

• **outputField:** Transformed column for sequence of n-gram.

Input JSON:

         {
           "name": "NGramTransform",
           "type": "sparkcompute",
           "properties": {
                                   "fieldToBeTransformed": "tokens",
                                   "noOfTerms": "2",
                                   "outputField": "ngrams"
                                }
          }

Introduction

Use-Case

• Transform input features(tokens in array form) into n-grams using parameter for number of terms in each n-gram.
• Transformed output will be an array of n-grams where each n-gram is represented by a space-delimited string of n consecutive words.

User Stories

• • ”bigram” 
  • Tokenization unit used to tokenize the input string before n-gram could be created:"Character" 

  Output: 

  DNASequence	bigram
  AGCTTCGA	[AG, GC, CT, TT, TC, CG, GA]

User Stories

• As a Hydrator user,I want to transfom input features data in a column from source schema into output schema which will have a single column having n n-gram data in one of the columns in output schema.
• As a Hydrator user I want to have configuration for specifying the column name from input schema on which transformation has to be performed.
• As a Hydrator user I want to have configuration to specify the no of terms which would be used for transformation of input features into n-grams.
• As a Hydrator user I want to have configuration to specify output column name wherein ngrams will be emitted.
• As a Hydrator user I want to specify the tokenization unit for the input to be tokenized before it could be converted to n-gram

Conditions

• Source field ,to be transformed,can be of only type string array.
• User can transform single field only from the source schema.Output schema will have a single field of type string array.
• If the input sequence contains fewer than n strings, no output is produced.

End to End Example pipeline:       

Input source:

NGramTransform:

Mandatory inputs from user:

• • Field to be used to transform input features into n-grams:”tokens”
  • Number of terms in each n-gram:”2”
  • Transformed field for sequence of n-gram:”ngrams”

Output:

• • Tokenization unit: "words"

TPFSAvro Output

End to End Example pipeline

Input source:

topic

sentence

Java

Hello world / is the /basic application

HDFS

HDFS/ is a /file system

Spark

Spark /is engine for /bigdata processing

NGramTransform:

Mandatory inputs from user:

Output:

Design

This is a sparkcompute type of plugin and is meant to work with Spark only.

Properties:

• **fieldToBeTransformed:** Column to be used to transform input features into n-grams.

• **noOfTermsnumberOfTerms:** No Number of terms in each n-gram.

• **outputField:** Transformed column for sequence of n-gram.
• **tokenizationUnit** Unit into which the input string will be tokenized.

Input JSON:

         {
           "name": "NGramTransform",
           "type": "sparkcompute",
           "properties": {
                                   "fieldToBeTransformed": "tokens",
                                   "noOfTermsnumberOfTerms": "2",

                                    "tokenizationUnit":"word",

                                   "outputField": "ngrams"
                                }
          }