Introduction

Use-Case

User Stories

• User should be able to specify the fully path to binary or just the binary
• User should be able to specify the arguments for the binary to be executed
• User should be provided a specification about how the record is passed to binary (need to be designed)
• If binary executable doesn’t exist or not in path or not executable, user should be notified appropriately during runtime
• User is able to see the errors in log if the executable writes the errors to STDERR
• User executable is able to read the record from STDIN
• User executable is able to write the record to STDOUT
• User executable is able to write the error records to a different FILE descriptor 
• User should be able to specify the error dataset to which data returned from executable special FILE descriptor
• User will make sure the binary and it’s dependencies are available on all machines of the cluster and no capability needs to be added to the plugin for marshaling the executable
  

Conditions

• Binary and it’s dependencies must be available on all the machines of the cluster, prior to the execution of binary.

• Arguments should be in the proper sequence and the supported format. Any mismatch in the sequence of the arguments will result into the failure of execution.

Design

For both the design approaches, following are the assumptions/considerations:

Design Approach Assumptions/Considerations:

1. Types of binary executable that will be supported by plugin are:

1. "bat, jar, sh and exe".
   
   
2. Binaries can take 0 to N arguments. Source for the variable arguments will always be the structured records coming through the Hydrator Source stage.(As it is a transform plugin)
   
   
3. Sequence of arguments is important. Any mismatch will result into the failure of the execution. Also, the fixed arguments will always be followed by the variable input arguments. This will be the format of the arguments for any binary to be executed. For example, 
   java -jar <Example.jar> <runtime/variable

1. arg1> <runtime/variable

Run Plugin Properties:

• binaryType : Type of the binary to be executed. For example, .jar or .bat or .sh or .exe.

• binaryName : Full path to the binary to be executed.

• inputSource : Source to read the data for runtime/variable input arguments for binary to be executed. For example, Hydrator Source Stage or STDIN

1. arg2>..... <Other fixed/static named & unnamed arguments>
   

1. Errors will be written to the cdap logs and the respective record will be stored in the Error Dataset, if the executable binary writes the errors to STDERR.
   
   
2. The output of the binary execution will be stored into the target/output fields which will be provided by user. Final output will include the output fields as well as the input fields coming from the previous stage.
   
   
3. In case, if the binaries do not produce any output, then plugin will write empty string to the target/output field.

Run Plugin Properties:

• commandToExecute : Command that will contain the path to the binary as well as how to execute that binary. For example, java -jar /home/user.ExampleRunner.jar, if the binary to executed is of type jar.

• variableArguments : A comma-separated sequence of the fields which will be used as input source for the runtime/variable arguments. For example, firstname or firstname,lastname in case of multiple arguments. Please make sure that the sequence of arguments is proper.

• fixedArguments : A space-separated sequence of the fixed input arguments that will be passed to the binary to be executed. Please make sure that the sequence of the arguments is proper. All the fixed input arguments are followed by the runtime/variable input arguments.

• outputFields : A comma-separated sequence of field name and its type which will be used to store the final output of the binary.

Run Input Json Format:

{

  "name": "Run",

  "type": "transform",

    "properties": {

      "commandToExecute": "

java -jar /opt/cdap/Runner.jar",

         "inputSource": "HydratorSource",

      "variableArguments": "

Firstname,

Lastname",

      "fixedArguments": "256 1024 -

Dcheckstyle.skip=true",

      "outputFields": "FinalOutput:string",

    }

}

Note: More details will be added based on the findings.

Implementation Tips

• Please reuse and/or modify ExternalProgramExecutor (https://github.com/caskdata/cdap-apps/blob/develop/TwitterSentiment/src/main/java/co/cask/cdap/apps/flowlet/ExternalProgramExecutor.java)