Goals

• The market will provide an application store for CDAP users to quickly and efficiently add capabilities into the platform.   

• The Cask Market will be a read only for 4.0 and include provisions for working offline.   

Checklist

User Stories

• As a CDAP User I want a plus button on the navbar so that I can an add new capabilities from anywhere in the platform

• As a CDAP User I want to be able to add new artifacts from the market into the system to speed up development and improve the overall developer experience.

• As a Hydrator User I want to be able to add plugins from the market to expand the capabilities of my pipeline.  

• As a Hydrator User from within the Hydrator Studio I want to be able to add new pipelines or applications from within Hydrator, and upon adding, remain in the Hydrator context.

• As a CDAP user I want to be able to add examples from the market.  

• As a CDAP user from within the market I want to be able to know what I've added into the platform and have those special cased when browsing the market.  

• AS a CDAP user I want to have access to the market in offline mode, so that I can add capabilities into the system from where-ever my cluster is deployed.  

Use Cases

Scenario 1

[1] I'm in hydrator studio, in CDAP 4.0

[2] I click the + button

[3] From the market I choose pipeline, and add "SFDC Lead dump" 

[4] The wizard walks me through the rep defined steps to add the pipeline

[5] I return to the hydrator context, and it's in the pipeline studio, AND NOT deployed as an app

Scenario 2

[1] In CDAP, and I click the + button

[2] From the market I choose pipeline, and add "SFDC Lead dump" 

[3] The wizard walks me through the repo defined steps to add the pipeline

[4] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app

Scenario 3

[1] In CDAP, and I click the + button

[2] From the market I choose application, "Stock Market Trainer" 

[3] I will see the option "clone in hydrator".

[4] If I click "clone in hydrator" the wizard walks me through the repo defined steps to add the pipeline

[5] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app

Scenario 4

[1] In CDAP, and I click the + button

[2] From the market I choose application, "Stock Market Trainer" 

[3] I will see the option "clone in hydrator".

[4] If I click "add", the wizard walks me through the repo defined steps to deploy the application

[5] The app will be deployed and I will see in detailed pipeline view.  

Scenario 5

[1] In CDAP, and I click the + button

[2] From the market I choose plugin, "SFDC Source" 

[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  

[4] The plugin is added as a resource, and the resource center shows this as included, with the specific entity version.

Scenario 6

[1] In hydrator studio, and I click the + button

[2] From the market I choose plugin, "SFDC Source" 

[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  

[4] The plugin is added and will show up on in the plugin library for immediate use

Scenario 7

Scenario 8

Scenario 8 Workflow Diagram

Open

Requirements

Market Requirements

• The market should be accessible from anywhere within CDAP.

• The market should support adding:

  • Artifacts

    • Applications

    • Native Plugins

    • Third Party Plugins

  • Pipelines

  • DataPacks

• The market should provide context when browsing to know if I've already added an entity.  I should still be able to create another instance and the market should keep track.  

• The market should instantiate the new entity wizard where applicable.  

• The market should be browsable independently of the product, from the website.  

• The market on the website should link to localhost and the cdap port to indicate "open in cdap" to add entities.  

• The market should be supported by default in both Distributed and Standalone, with the option to be disabled.  

• The market should work in offline mode after syncing the repository onto the cluster.  

• The market should be an independent entity outside of CDAP.

• CDAP should track what has been added previously

• The market should be read only for the first iteration.   

Market Repository Requirements

• The repo should include metadata only.

• The contract between the repo and the entities needs to be versioned to support changes in repo behavior or meta data specifications for entities.

• The repo should be updatable/installable from the UI, REST API or CMD Line.  

• The repo should include versioning information for entities.

• The repo should include a checksum for entities.

• The repo should include information for third party entities.

• The repo should include author and licensing information for entities.

• The repo should include dependency information for entities

• The repo should include abstraction information (service descriptor) for how an entity will interact with the market wizard workflow.   The wizard will consist of 1 to N steps and must be able to abstract all information necessary from the entity descriptor for how an entity is added including dependencies (internal/external) precedence of steps.

Market Datapack Requirements

Datapacks are sources of data that will improve the bootstrapping and getting started experience on the platform.  

• Datapacks should be able to be added for:

  • SQL datapacks 

    • MYSQL .sql files

    • SQLITE .sqlite

  • S3 datapacks

    • Text files

    • csv examples

  • HTTP (for hydrator realtime pipelines)

    • Endpoints for various supported web services

• Datapacks should be available to be added as new sources plugin templates in hydrator.  

• Datapacks should be able available for custom applications.