User Stories
As a developer of the Workflow application -
- I want the ability to pass the custom data (such as metric, status, error codes etc.) from one program in the Workflow to the next subsequent programs in the form of token.
- At any node in the Workflow, I want ability to query the data from the token.
- I want ability to fetch the data from the token which was set by a specific node.
- I want ability to find the name of the node which most recently set the token value for the specific key e.g. node who last set the ERROR flag in the token, so that I can take appropriate action (such as logging or improving its code) on it.
- I want to have the conditional execution in the Workflow based on the information contained in the token.
- I want to terminate the execution if some node in the Workflow produce unexpected results.
As an admin/support person/developer of the Workflow application -
- I want ability to query the WorkflowToken from the past runs for running analysis such as which node is executed more frequently and why.
- I want ability to query the token values which were added by the specific node in the Workflow to debug the flow of execution.
Requirements
Generic Token support (not only MR counters)
- Ability for the user to access the token set by any node in the Workflow
- Persist token per Workflow run in the MDS
- View token set by specific node in the Workflow
Assumptions
Generic information in the token is of the key-value type, each represented by String.
Potential Changes
WorkflowToken interface changes
/** * Interface to represent the data that is transferred from one node to the next nodes in the {@link Workflow}. */ public interface WorkflowToken { /** * Put the specified key-value entry into the {@link WorkflowToken}. * The token may store additional information about the context in which * this key is being set, for example, the unique name of the workflow node. * @param key the key representing the entry * @param value the value for the key */ void put(String key, String value); /** * Get the most recent value for the specified key. * @param key the key to be searched * @return the {@link Value} for the key */ @Nullable Value get(String key); /** * Get the value set for the specified key by the specified node. * @param key the key to be searched * @param nodeName the name of the node * @return the {@link Value} set for the key by nodeName */ @Nullable Value get(String key, String nodeName); /** * Same key can be added to the WorkflowToken by multiple nodes. * This method returns the {@link List} of {@link NodeValueEntry}, where * each entry represents the unique node name and the value that it set * for the specified key. * <p> * The list maintains the order in which the values were * inserted in the WorkflowToken for a specific key except in the case of fork * and join. In case of fork in the Workflow, copies of the WorkflowToken are made * and passed along each branch. At the join, all copies of the * WorkflowToken are merged together. While merging, the order in which the values were * inserted for a specific key is guaranteed within the same branch, but not across * different branches. * @param key the key to be searched * @return the list of {@link NodeValueEntry} from node name to the value that node * added for the input key */ List<NodeValueEntry> getAll(String key); /** * Get the {@link Map} of key-values that were added to the {@link WorkflowToken} * by specific node. * @param nodeName the unique name of the node * @return the map of key-{@link Value} that were added by the specified node */ Map<String, Value> getAllFromNode(String nodeName); /** * Get the system values that were added to the {@link WorkflowToken} by specific node. * @param nodeName the unique name of the node * @return the map of key-{@link Value} that were added by the specific node */ Map<String, Value> getAllSystemValuesFromNode(String nodeName); /** * This method is deprecated as of release 3.1. Instead to get the * MapReduce counters from the WorkflowToken, use the flatten key prefixed * by 'mr.counters'. * <p> * Example: * <p> * <ul> * <li> * To get the most recent value of counter with group name * 'org.apache.hadoop.mapreduce.TaskCounter' and counter name 'MAP_INPUT_RECORDS' * <pre> * <code> * String flattenCounterKey = "mr.counters.org.apache.hadoop.mapreduce.TaskCounter.MAP_INPUT_RECORDS"; * workflowToken.getValue(flattenCounterKey); * </code> * </pre> * </li> * * <li> To get the value of counter with group name 'org.apache.hadoop.mapreduce.TaskCounter' * and counter name 'MAP_INPUT_RECORDS' as set by MapReduce program with unique name 'PurchaseHistoryBuilder' * <pre> * <code> * String flattenCounterKey = "mr.counters.org.apache.hadoop.mapreduce.TaskCounter.MAP_INPUT_RECORDS"; * workflowToken.getValue(flattenCounterKey, "PurchaseHistoryBuilder"); * </code> * </pre> * </li> * </ul> * <p> * Get the Hadoop counters from the previous MapReduce program in the Workflow. * The method returns null if the counters are not set. * @return the Hadoop MapReduce counters set by the previous MapReduce program */ @Deprecated @Nullable Map<String, Map<String, Long>> getMapReduceCounters(); /** * Return true if the {@link WorkflowToken} contains the specified key. * @param key the key to be tested for the presence in the {@link WorkflowToken} * @return the result of the test */ boolean containsKey(String key); }
The method getAll(String key) in the above interface returns the List of NodeValueEntry objects. NodeValueEntry class represents nodeName and value that the node put for the specific key./** * Multiple nodes in the Workflow can add the same key to the {@link WorkflowToken}. * This class provides a mapping from node name to the value which was set for the * specific key. */ public final class NodeValueEntry { private final String nodeName; private final Value value; public NodeValueEntry(String nodeName, String value) { this.nodeName = nodeName; this.value = value; } public String getNodeName() { return nodeName; } public Value getValue() { return value; } ... // other methods like toString(), equals() and hashCode() ... }
The method getAllSystemValuesFromNode(String nodeName) returns the map of string keys to the Value class. The details of the Value class are as follows:/** * Represents the value for the specific key in the {@link WorkflowToken} */ public final class Value { private final String value; public Value(String value) { this.value = value; } public String getAsString() { return value; } public long getAsLong() { return Long.parseLong(value); } public int getAsInt() { return Integer.parseInt(value); } }Ability to include same MapReduce program multiple times in the Workflow
Potential Use Case: Email campaign generates two categories of events - send events (SUCCESS, FAIL) and tracking events (OPEN, CLICK etc.). Records representing the send event and tracking event have different number of fields. EventParser MapReduce program should be able to parse these two categories of the events and create the list Event object per audience id.
Tracking event format:
audience_id,event_sub_type,ip_address,device_type,event_time,link
Example records:
bob,CLICK,192.168.29.10,android,1436311150092,http://www.somedomain.com
adam,CLICK,192.168.29.18,ipad,1436311232276,http://www.anotherdomain.com
Send event format:
audience_id,event_sub_type,ip_address,deliveryCode,event_time
Example records:
bob,SEND,192.168.29.10,SUCCESS,1436311232276
adam,SEND,192.168.29.10,FAILED,1436311450437
Approach 1: Add same MapReduce/Spark program multiple times in the Application.
API changes for the ApplicationConfigurer to allow adding MapReduce/Spark program in the Application with the explicit name.
/** * Adds a {@link MapReduce} to the Application. * @param name the name to be given to the {@link MapReduce} program * @param mapReduce the {@link MapReduce} program to be included in the Application */ void addMapReduce(String name, MapReduce mapReduce); /** * Adds a {@link Spark} to the Application. * @param name the name to be given to the {@link Spark} program * @param spark the {@link Spark} program to be included in the Application */ void addSpark(String name, Spark spark);EventParser application:
public class EventParserApp extends AbstractApplication { @Override public void configure() { // Stream to receive send events addStream(new Stream("send")); // Stream to receive tracking events addStream(new Stream("tracking")); // Add EventParser MapReduce program multiple times in the application with different properties Map<String, String> properties = Maps.newHashMap(); properties.put("input.stream", "tracking"); // 'trackingParser' is instance of the EventParser which will read the 'tracking' stream addMapReduce("trackingParser", new EventParser(properties)); properties = Maps.newHashMap(); properties.put("input.stream", "success"); // 'sendParser' is instance of the EventParser which will read the 'send' stream addMapReduce("sendParser", new EventParser(properties)); // Add Workflow which will process the tracking and send events in parallel addWorkflow(new EventParserWorkflow()); } }EventParser MapReduce program:
public class EventParser extends AbstractMapReduce { private final Map<String, String> properties; public EventParser(Map<String, String> properties) { this.properties = properties; } @Override public void configure() { setDescription("MapReduce program for parsing the email events and storing them in the dataset."); // Serialize the properties setProperties(properties); setOutputDataset("events"); } @Override public void beforeSubmit(MapReduceContext context) throws Exception { Job job = context.getHadoopJob(); job.setMapperClass(EventParserMapper.class); job.setReducerClass(EventParserReducer.class); job.setMapOutputKeyClass(Text.class); job.setOutputValueClass(Event.class); job.setNumReduceTasks(1); String streamToVerify = context.getSpecification().getProperties().get("input.stream"); job.getConfiguration().set("input.stream", streamToVerify); // Read the purchase events from the last 60 minutes as input to the mapper. final long endTime = context.getLogicalStartTime(); final long startTime = endTime - TimeUnit.MINUTES.toMillis(60); StreamBatchReadable.useStreamInput(context, streamToVerify, startTime, endTime); } } // EventParserMapper public static class EventParserMapper extends Mapper<LongWritable, Text, Text, Event> { @Override public void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String logEvent = value.toString(); if (logEvent.isEmpty()) { return; } String inputStream = context.getConfiguration().get("input.stream"); Event event; if(inputStream.equals("send")) { event = getSendEvent(logEvent); } else { event = getTrackingEvent(logEvent); } if (event != null) { context.write(new Text(event.getAudienceId()), event); } } private Event getSendEvent(String logEvent) { String seperator = "::"; int fieldLength = 5; String[] fields = logEvent.split(seperator); if (fields.length != fieldLength) { return null; } String audienceId = fields[0]; String eventType = fields[1]; String ipAddress = fields[2]; String deliveryCode = fields[3]; String eventTime = fields[4]; return new Event(audienceId, eventType, ipAddress, eventTime, deliveryCode) } private Event getTrackingEvent(String logEvent) { String seperator = ","; int fieldLength = 6; String[] fields = logEvent.split(seperator); if (fields.length != fieldLength) { return null; } String audienceId = fields[0]; String eventType = fields[1]; String ipAddress = fields[2]; String deviceType = fields[3]; String eventTime = fields[4]; String link = fields[5]; return new Event(audienceId, eventType, ipAddress, eventTime, deviceType + "&&" + link); } }Approach 2: WorkflowConfigurer interface changes
We generate unique numeric node ids for each node when the application is deployed. However, while writing the Workflow, users will not be aware of the node id associated with each node in the Workflow. Since WorkflowToken stores the MapReduce counters and other information per node level, users should be able to get the value of a particular key from the token as set by the particular program in the Workflow.
If the program is used only once in a Workflow, then the user can use its name to query for the token information. However, we allow the same program to occur multiple times in a Workflow. In that case, the program name will not be sufficient to access the token.
The WorkflowConfigurer API can be updated to allow a user to set a unique name for the program, if it occurs multiple times in a Workflow and use that unique name to retrieve the token.
/** * Add MapReduce program to the {@link Workflow}. * @param uniqueName the unique name for the MapReduce program which will be used * to identify particular occurrence of the program in the Workflow * @param mapReduceName the name of the MapReduce program */ void addMapReduce(String uniqueName, String mapReduceName);
WorkflowToken can also be updated from a predicate on the condition node. In the presence of multiple condition nodes in a Workflow, we will need the ability to specify unique names for the conditions as well so that token values from specific condition nodes can be fetched./** * Adds a condition with the unique name to the {@link Workflow}. * @param conditionName the unique name to be assigned to the condition * @param condition the {@link Predicate} to be evaluated for the condition * @return the configurer for the condition */ WorkflowConditionConfigurer<? extends WorkflowConfigurer> condition(String conditionName, Predicate<WorkflowContext> condition);Provide ability to set and get information in the WorkflowToken
1. MapReduce program: Users should be able to access and modify WorkflowToken from "beforeSubmit" and "onFinish" methods of the MapReduce program. Since these methods get the MapReduceContext, we will need to update the MapReduceContext interface to get the WorkflowToken./** * If {@link MapReduce} program is executed as a part of the {@link Workflow} * then get the {@link WorkflowToken} associated with the current run, otherwise return null. * @return the {@link WorkflowToken} if available */ @Nullable WorkflowToken getWorkflowToken();Consider the following code sample to update the WorkflowToken in the MapReduce program:
@Override public void beforeSubmit(MapReduceContext context) throws Exception { ... WorkflowToken workflowToken = context.getWorkflowToken(); if (workflowToken != null) { // Put the action type in the WorkflowToken workflowToken.put("action_type", "MAPREDUCE"); // Put the start time for the action workflowToken.put("startTime", String.valueOf(System.currentTimeMillis())); } ... } @Override public void onFinish(boolean succeeded, MapReduceContext context) throws Exception { ... WorkflowToken workflowToken = context.getWorkflowToken(); if (workflowToken != null) { // Put the end time for the action workflowToken.put("endTime", String.valueOf(System.currentTimeMillis())); } ... }2. Spark program: Users should be able to access and modify WorkflowToken from "beforeSubmit" and "onFinish" methods of the Spark program. Since these methods get the SparkContext, we will need to update the SparkContext interface to get the WorkflowToken.
/** * If {@link Spark} program is executed as a part of the {@link Workflow} * then get the {@link WorkflowToken} associated with the current run, otherwise return null. * @return the {@link WorkflowToken} if available */ @Nullable WorkflowToken getWorkflowToken();3. Custom Workflow action: Since custom workflow actions already receive WorkflowContext, no changes are anticipated in the interface.
Following is the sample code to get values from the WorkflowToken in custom action:
@Override public void run() { ... WorkflowToken token = getContext().getToken(); // set the type of the action of the current node token.put("action_type", "CUSTOM_ACTION"); // Assume that we have the following Workflow // |------->PurchaseByCustomer------->| // True | | // Start---->RecordVerifier---->Predicate-------->| |------------->StatusReporter----->End // | | | | // | False |------->PurchaseByProduct-------->| | // | | // |--------------------->ProblemLogger--------------------->| // Use case 1: Predicate can add the key "branch" in the WorkflowToken with value as "true" if true branch will be executed // or "false" otherwise. In "StatusReporter" in order to get which branch in the Workflow was executed boolean bTrueBranch = Boolean.parseBoolean(token.get("branch")); // Use case 2: User may want to compare the counters emitted by "PurchaseByCustomer" and "PurchaseByProduct", in order to find which job // is processing more records. String purchaseByCustomerCounterValue = token.get("MapReduceCounterName", "PurchaseByCustomer"); String purchaseByProductCounterValue = token.get("MapReduceCounterName", "PurchaseByProduct"); // Use case 3: Since Workflow can have multiple complex conditions and forks in its structure, in the "StatusReporter", // user may want to know how many actions were executed as a part of this run. If the number of nodes executed were below // certain threshold send an alert. Assuming that every node in the Workflow adds the key "action_type" with the value as action // type for node in the WorkflowToken, user can further figure out the break down by action type in the particular Workflow run. List<NodeValueEntry> nodeValues = token.getAll("action_type"); int totalNodeExecuted = nodeValues.size(); int mapReduceNodes = 0; int sparkNodes = 0; int customActionNodes = 0; int conditions = 0; for (NodeValueEntry entry : nodeValues) { if (entry.getValue().equals("MAPREDUCE")) { mapReduceNodes++; } if (entry.getValue().equals("SPARK")) { sparkNodes++; } if (entry.getValue().equals("CUSTOM_ACTION")) { customActionNodes++; } if (entry.getValue().equals("CONDITION")) { conditions++; } } // Use case 4: To get the name of the last node which set the "ERROR" flag in the WorkflowToken List<NodeValueEntry> errorNodeValueList = token.getAll("ERROR"); String nodeNameWhoSetTheErrorFlagLast = errorNodeValueList.get(errorNodeValueList.size() - 1); // To get the start time of the MapReduce program with unique name "PurchaseHistoryBuilder" String startTime = token.get("startTime", "PurchaseHistoryBuilder"); // To get the most recent value of counter with group name // 'org.apache.hadoop.mapreduce.TaskCounter' and counter name 'MAP_INPUT_RECORDS' String flattenCounterKey = "mr.counters.org.apache.hadoop.mapreduce.TaskCounter.MAP_INPUT_RECORDS"; workflowToken.get(flattenCounterKey); // To get the value of counter with group name 'org.apache.hadoop.mapreduce.TaskCounter' // and counter name 'MAP_INPUT_RECORDS' as set by MapReduce program with unique name 'PurchaseHistoryBuilder' workflowToken.get(flattenCounterKey, "PurchaseHistoryBuilder"); ... }- WorkflowToken in presence of Fork and Join
When a fork is encountered in the Workflow, we make a copy of the WorkflowToken and pass it along to each branch. At the join, we create a new WorkflowToken, which will be a merge of the WorkflowTokens associated with each of the branches of the fork. Since we are storing the information in the token at the node level, there will not be any conflicts during the merge process. - Persisting the WorkflowToken
The RunRecord for the Workflow will contain the WorkflowToken as its property. This token will be updated before the execution of the action in the Workflow. We will add a version field to the RunRecord itself which will help in the upgrade process. - RESTful end-points to access the value of the WorkflowToken that was received by an individual node in the WorkflowWe will expose a RESTful end point to retrieve the token values that were set by a particular node as identified by its unique name.
/apps/{app-id}/workflows/{workflow-name}/runs/{run-id}/nodes/{unique-node-name}/token