Cask Market - CDAP 4.0

Owned by Todd Greenstein
Last updated: Sept 02, 2016
5 min read

Goals

  • The market will provide an application store for CDAP users to quickly and efficiently add capabilities into the platform.   
  • The Cask Market will be a read only for 4.0 and include provisions for working offline.   

Checklist

  • User stories documented(Todd)
  • Requirements documented(Todd)
  • Requirements Reviewed
  • Mockups Built
  • Design Built
  • Design Accepted

User Stories

  • As a CDAP User I want a plus button on the navbar so that I can an add new capabilities from anywhere in the platform
  • As a CDAP User I want to be able to add new artifacts from the market into the system to speed up development and improve the overall developer experience.
  • As a Hydrator User I want to be able to add plugins from the market to expand the capabilities of my pipeline.  
  • As a Hydrator User from within the Hydrator Studio I want to be able to add new pipelines or applications from within Hydrator, and upon adding, remain in the Hydrator context.
  • As a CDAP user I want to be able to add examples from the market.  
  • As a CDAP user from within the market I want to be able to know what I've added into the platform and have those special cased when browsing the market.  
  • AS a CDAP user I want to have access to the market in offline mode, so that I can add capabilities into the system from where-ever my cluster is deployed.  

Use Cases

Scenario 1
[1] I'm in hydrator studio, in CDAP 4.0
[2] I click the + button
[3] From the market I choose pipeline, and add "SFDC Lead dump" 
[4] The wizard walks me through the rep defined steps to add the pipeline
[5] I return to the hydrator context, and it's in the pipeline studio, AND NOT deployed as an app
Scenario 2
[1] In CDAP, and I click the + button
[2] From the market I choose pipeline, and add "SFDC Lead dump" 
[3] The wizard walks me through the repo defined steps to add the pipeline
[4] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 3
[1] In CDAP, and I click the + button
[2] From the market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4] If I click "clone in hydrator" the wizard walks me through the repo defined steps to add the pipeline
[5] Hydrator is opened and it's in the pipeline studio, AND NOT deployed as an app
Scenario 4
[1] In CDAP, and I click the + button
[2] From the market I choose application, "Stock Market Trainer" 
[3] I will see the option "clone in hydrator".
[4] If I click "add", the wizard walks me through the repo defined steps to deploy the application
[5] The app will be deployed and I will see in detailed pipeline view.  
Scenario 5
[1] In CDAP, and I click the + button
[2] From the market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added as a resource, and the resource center shows this as included, with the specific entity version.
Scenario 6
[1] In hydrator studio, and I click the + button
[2] From the market I choose plugin, "SFDC Source" 
[3] The wizard walks me through the steps to add the plugin, including the option to have it added as plugin template.  
[4] The plugin is added and will show up on in the plugin library for immediate use
Scenario 7

[1] In hydrator studio, and I click the + button
[2] From the market I choose add a third party plugin, "MYSQL JDBC"
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the plugin.
[5] In the case of an external dependency, the license agreement and download should be incorporated into the wizard.
[6] The option to have it added as plugin template should be part of the wizard.
[7] The plugin is added and will show up on in the plugin library for immediate use.

Scenario 8

[1] In CDAP, and I click the + button
[2] From the market I choose add "Traffic Incidents"  mysql data 
[3] If multiple versions are available, I should be able to choose which version I want from a dropdown.
[4] The wizard walks me through the steps to add the datapack locally and into my development environment 

Scenario 8 Workflow Diagram

Requirements

Market Requirements

  • The market should be accessible from anywhere within CDAP.
  • The market should support adding:
    • Artifacts
      • Applications
      • Native Plugins
      • Third Party Plugins
    • Pipelines
    • DataPacks
  • The market should provide context when browsing to know if I've already added an entity.  I should still be able to create another instance and the market should keep track.  
  • The market should instantiate the new entity wizard where applicable.  
  • The market should be browsable independently of the product, from the website.  
  • The market on the website should link to localhost and the cdap port to indicate "open in cdap" to add entities.  
  • The market should be supported by default in both Distributed and Standalone, with the option to be disabled.  
  • The market should work in offline mode after syncing the repository onto the cluster.  
  • The market should be an independent entity outside of CDAP.
  • CDAP should track what has been added previously
  • The market should be read only for the first iteration.   

Market Repository Requirements

  • The repo should include metadata only.
  • The contract between the repo and the entities needs to be versioned to support changes in repo behavior or meta data specifications for entities.
  • The repo should be updatable/installable from the UI, REST API or CMD Line.  
  • The repo should include versioning information for entities.
  • The repo should include a checksum for entities.
  • The repo should include information for third party entities.
  • The repo should include author and licensing information for entities.
  • The repo should include dependency information for entities
  • The repo should include abstraction information (service descriptor) for how an entity will interact with the market wizard workflow.   The wizard will consist of 1 to N steps and must be able to abstract all information necessary from the entity descriptor for how an entity is added including dependencies (internal/external) precedence of steps.

Market Datapack Requirements

Datapacks are sources of data that will improve the bootstrapping and getting started experience on the platform.  

  • Datapacks should be able to be added for:
    • SQL datapacks 
      • MYSQL .sql files
      • SQLITE .sqlite
    • S3 datapacks
      • Text files
      • csv examples
    • HTTP (for hydrator realtime pipelines)
      • Endpoints for various supported web services
  • Datapacks should be available to be added as new sources plugin templates in hydrator.  
  • Datapacks should be able available for custom applications.  

 

Created in 2020 by Google Inc.