Introduction 

Enhancements to wrangler. Ability to read connect to external sources like Database, Kafka and perform wrangler directives on them for data preparation. 

Goals

Ability to dynamically load plugins from CDAP Service. 

User Stories 

• User wants to load the database driver artifact in wranger and execute database commands to load data. Once data is loaded they want to execute wrangler directives on the loaded data.
• User adds kafka artifact, immediately wants to load the kafka artifact in wrangler and read data from a kafka topic, once data is loaded, they want to execute wrangler directives on the loaded data.
• User Story #3

Design

Background :

Cover details on assumptions made, design alternatives considered, high level design

Approach

Approach #1

Approach #2

API changes

New Programmatic APIs

New Java APIs introduced (both user facing and internal)

public interface Admin extends DatasetManager, SecureStoreManager, MessagingAdmin, ArtifactManager {

} 

/**
 * Provides access to information about artifacts
 */
interface ArtifactManager {

/**
 * returns the list of artifacts available in current namespace and system namespace.
 */
List<ArtifactDetail> listArtifacts();

/**
 * Create and return class loader with the contents of artifactDetail, with the parentClassLoader as parent, 
 * if parentClassLoader is null we will use boot strap class loader as the parent.
 * @return ClassLoader
 */
ClassLoader createClassLoader(ArtifactDetail artifactDetail, @Nullable ClassLoader parentClassLoader);

...
} 

Notes :

1) ArtifactId class is available in cdap-api, we might want to move ArtifactDetail or create another object in cdap-api exposing artifact details, With Admin, CDAP developers can access the ArtifactManager and get information about artifacts

2) How the user can instantiate the plugin ?

 Since InstantatiorFactory is available as part of cdap-common, Once they have loaded the desired plugin class, they can use the instantiatorFactory to instantiate the plugin.

Deprecated Programmatic APIs

New REST APIs

Deprecated REST API

CLI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

UI Impact or Changes

• Impact #1
• Impact #2
• Impact #3

Security Impact 

What's the impact on Authorization and how does the design take care of this aspect

Impact on Infrastructure Outages 

System behavior (if applicable - document impact on downstream [ YARN, HBase etc ] component failures) and how does the design take care of these aspect

Test Scenarios

Releases

Release X.Y.Z

Release X.Y.Z

Related Work

• Work #1
• Work #2
• Work #3

Future work