Data Wrangler - CDAP 4.0
Goals
Adding a Data Wrangler will improve the overall user experience for creating schemas, and facilitate easier of importing data.
Checklist
User Stories.
As a Hydrator user in my pipeline after a source plugin, I want a transform node that allows me to graphically build a schema to be used in my pipeline.
As a Hydrator user I want the new transform node type to be able to operate anywhere in my pipeline after a Source is defined.
As a Hydrator user I want the nmentedew transform node to make a best effort to determine if the first row of a file I’m importing is different so that I can quickly determine if a header row exists.
As a Hydrator user I want the new transform node to understand common delimiters so that I can parse my data into columns and fields.
As a Hydrator user I want the new transform node to allow configuration of column fields (name/type/reorder/include/drop/merge/split) from sources in my pipeline, using a graphical interface
As a Hydrator user I want the new transform node to provide easily visible statistics for data quality and a histogram for distribution. I want these to be viewable at the column level for each field in my source.
As a Hydrator user I want the new transform node to provide a history of all steps I perform on a document to be available to me.
Requirements
General
The new tool can be instantiated as a new transform node type from inside Hydrator.
The tool should also be accessible outside of Hydrator.
The input for the tool should be schema OR a json representation of sampled data in JSON. The new tool should not configure data sources.
The output for the tool should be an output schema in JSON and the DSL for performing the transformations.
State, including sample, should be preserved upon "saving" and returning from within the pipeline.
Supported Operations
Sample Data/Schema inference
Column Operations
Bulk Column Editing Operations should be possible from a single view to:
Steps viewer to:
Future Considerations
Date/time support as a field type, and date/time functions
Design
Check for header workflow:
Comments
Can you explain in more detail what data quality and histogram requirements are? How are these going to be collected?
If there are bad records that cannot be wrangled, what happen to them? Are they emitted to a "quarantine" dataset? or dropped? How is that configured at the plugin level? Or does this require more work at the pipeline level? For example, connecting to another filter/sink for bad records?
For the steps viewer, is this for within my session? Or can I save a pipeline, then if I run the pipeline and find something wrong, can I edit the wrangle plugin and undo the last step? That would be a typical use case, right? Iterative configure wrangler, run pipeline (possibly in preview), refine wrangler, run again, etc.
I am not sure how this will work. Suppose I have a source that produces a schema with 178 fields, but I don't know exactly what these look like.
How am I able to get sample data? For that I would have to configure a source, but that requires that I know the schema?
What is the input schema for this plugin? It's schema from the source?
Or does the source emit each line of the file as a blob? What is the format is not text, but json or xml? Can I still use data wrangler?
Is the wrangler plugin a new plugin, or is it a Javascript plugin, where the JS code is generated by the UI?
If there is a header, how is this going to be used in the pipeline? Does Hydrator have the capability to remove headers?
Is this going to work for streaming and for batch? What is the meaning of a "header" in streaming?