Introduction

Use-Case

User Stories

• User should be able to specify the fully full path to binary or just the binary
• User should be able to specify the arguments for the binary to be executed
• User should be provided a specification about how the record is passed to binary (need to be designed)
• If binary executable doesn’t exist or not in path or not executable, user should be notified appropriately during runtime
• User is able to see the errors in log if the executable writes the errors to STDERR
• User executable is able to read the record from STDIN
• User executable is able to write the record to STDOUT
• User executable is able to write the error records to a different FILE descriptor 
• User should be able to specify the error dataset to which data returned from executable special FILE descriptor
• User will make sure the binary and it’s dependencies are available on all machines of the cluster and no capability needs to be added to the plugin for marshaling the executable
  

Conditions

• Binary and it’s dependencies must be available on all the machines of the cluster, prior to the execution of binary.

• Arguments should be in the proper sequence and the supported format. Any mismatch in the sequence of the arguments will result into the failure of execution.

Design

For both the design approaches, following are the assumptions/considerations:

Design Approach Assumptions/Considerations:

1. Types of binary executable that will be supported by plugin are:

Run Plugin Properties:

• binaryType : Type of the binary to be executed. For example, .jar or .bat or .sh or .exe.

• binaryName : Full path to the binary to be executed.

• inputSource : Source to read the data for runtime/variable input arguments for binary to be executed. For example, Hydrator Source Stage or STDIN

1. "bat, jar, sh and exe".

2. Path to the executable binary, specified in 'commandToExecute' property, should be an absolute path not the URI path i.e. should not start with hdfs:// or file:///.

3. Executable binary will always read the input through STDIN and should generate the STDOUT for each input record. Also, errors emitted by the executable through STDERR will be captured in logs.

4. Executable binary can take 0 to N inputs. Source for the varying inputs will always be the structured records coming through the Hydrator source stage and will passed to the binary through STDIN. Required fields can be provided using 'fieldsToProcess' property.

5. Fixed inputs (if any), will always be followed by the varying inputs. All the inputs will be passed as space separated sequence to the executable binary through STDIN. This will be the format for sending the inputs to the executable binary.

6. The output of the binary execution will be stored into the target/output field which will be provided by user. Final output will include the output fields as well as the input fields coming from the previous stage.
7. If the binary does not exists, then it will result into the failure during runtime. (after pipeline is published).
8. Supported schema types for output field are: "boolean, bytes, double, float, int, long and string".
9. Plugin will read the standard output and error streams, with UTF_8 encoding.
   
   

Run Plugin Properties:

• commandToExecute : Command that will contain the full path to the executable binary present on the local filesystem of the Hadoop nodes as well as how to execute that binary. For example, java -jar /home/user/ExampleRunner.jar, if the binary to be executed is of type jar.

• fieldsToProcess: A comma-separated sequence of the fields to be used as variable command line arguments for binary to be executed. For example, 'firstname' or 'firstname,lastname' in case of multiple arguments. Please make sure that the sequence of fields/arguments is proper.in the order as expected by binary. (Macro Enabled)

• fixedArguments :A space-separated sequence of the fixed input command line arguments that will be passed to the executable binary to be executed. Please make sure that the sequence of the arguments is properin the order as expected by binary. All the fixed input command line arguments are will be followed by the runtime/ variable input arguments.
  
• outputFields : A comma-separated sequence of field name and its type which will be used to store the final output.

• command line arguments, provided through 'Fields to Process for Variable Arguments'. (Macro enabled)

• outputField : The field name that holds the output of the executable binary.
  
  
• outputFieldType: Schema type of the 'Output Field'. Supported types are: boolean, bytes, double, float, int, long and string.
  
  

Run Input Json Format:

{

  "name": "Run",

  "type": "transform",

    "properties": {

      "commandToExecute": "

java -jar /opt/cdap/Runner.jar",

      "fieldsToProcess": "

Firstname,Lastname",

      "fixedArguments": "256 1024 -

Dcheckstyle.skip=true",

         "outputFields": "FinalOutput:string"

     }

 }

      "outputField": "target",
      "outputFieldType": "string"
    }
}

Implementation Tips

• Please reuse and/or modify ExternalProgramExecutor (https://github.com/caskdata/cdap-apps/blob/develop/TwitterSentiment/src/main/java/co/cask/cdap/apps/flowlet/ExternalProgramExecutor.java)