Overview

The purpose of this page is to illustrate the plan for ApplicationTemplate and Application consolidation.  This work is being tracked in 

Motivation

Why do we want to consolidate templates and applications? In CDAP 3.0, an ApplicationTemplate is a way for somebody to write an Application that can be given some configuration to create an Adapter. The story is confusing; one would expect an ApplicationTemplate to create... Applications. Instead, we use the term Adapter because Application means something else already. In addition an ApplicationTemplate can only include a single workflow or a single worker, giving people different experiences for templates and applications. 

Really, the goal of templates was to be able to write one piece of Application code that could be used to create multiple Applications. To do this requires that an Application can be configured at creation time instead of at compile time. For example, a user should be able to set the name of their dataset based on configuration instead of hardcoding it in the code. To support this, we plan on making it possible to get a configuration object from the ApplicationContext available in Application's configure() method. This allows somebody to pass in a config when creating an Application through the RESTful API, which can be used to configure an Application. The relevant programmatic API changes are shown below.

Definitions

Artifact - A jar file containing classes that can be used by CDAP.

...

Plugin - An extension to an Artifact. Usually implements an interface used by Application Classes in the Artifact.

Use Case Walkthrough

1. Create an Application that uses config

1.1 Deploying the Artifact

A developer writes a configurable Application Class that uses a Flow to read from a stream and write to a Table.

...

GET /namespaces/default/classes/apps
[
  {
    "className": "co.cask.cdap.examples.myapp.MyApp",
    "artifact": {
      "name": "myapp",
      "version": "1.0.0"
    }
  }
]

1.2 Creating an Application

The user decides to create an application from the deployed artifact. From the calls above, the user gathers that input and output are both configurable. The user decides to create an Application that reads from the 'purchases' stream and writes to the 'events' table.

...

The Application now shows up in all the normal RESTful APIs, with all its programs, streams, and datasets.

1.3 Updating an Application

A bug is found in the code, a fix is provided, and a 'myapp-1.0.1.jar' release is made. The artifact is deployed:

...

The config section is optional. If none is given, the previous config will be used. If it is given, it will replace the old config (no merging is done).

1.4 Rolling Back an Application

Actually, version 1.0.1 has a bug that's even worse and needs to be rolled back. The same update call can be made:

POST /namespaces/default/apps/purchaseDump/update -d '
{ 
  "artifact": {
    "name": "myapp",
    "version": "1.0.0"
  },
  "config": {
    "stream": "purchases",
    "table": "events" 
  }
}'

1.5 Deploying an Artifact and Creating an App in one step

For backwards compatibility, the deploy app API will remain the same and will internally deploy an artifact and create the app in one call. An additional header will be supported specifying the Application Config.

POST /namespaces/default/apps --data-binary @myapp-1.0.0.jar -H 'X-App-Config: { "stream": "purchases", "table": "events" }'

2. Create an Application that uses plugins

2.1 Application Class changes

Now the user decides to update the MyApp Application Class to support pluggable ways of reading from a stream. This is done by introducing a 'StreamReader' interface in their project:

...

GET /namespaces/default/artifacts/myapp/versions/2.0.0
{
  "name": "myapp",
  "version": "2.0.0",
  "classes": {
    "apps": [
      {
        "className": "co.cask.cdap.examples.myapp.MyApp",
        "properties": {
          "stream": { 
            "name": "stream", 
            "description": "The name of the stream to read from. Defaults to 'A'.", 
            "type": "string", 
            "required": false 
          },
          "table": {
            "name": "table",
            "description": "The name of the table to write to. Defaults to 'X'.",
            "type": "string",
            "required": false,
          },
          "readerPlugin": {
            "name": "readerPlugin",
            "description": "The name of the streamreader plugin to use.",
            "type": "string",
            "required": true
          },
          "readerPluginProperties": {
            "name": "readerPluginProperties",
            "description": "Properties to send to the streamreader plugin.",
            "type": "plugin",
            "plugintype": "streamreader",
            "required": false
          }
        }
      }
    ],
    "flows": [ ... ],
    "flowlets": [ ... ],
    "datasetModules": [ ... ]
  }
}

2.2 Adding plugins

A default implementation of the streamreader plugin is created to implement the previous logic:

...

GET /namespaces/default/artifacts/myapp/versions/2.0.0/extensions
[ "streamreader" ]
 
GET /namespaces/default/artifacts/myapp/versions/2.0.0/extensions/streamreader
[
  {
    "name": "default",
    "type": "reader",
    "description": "Writes timestamp and body as two columns and expects the row key to come as a header in the stream event.",
    "className": "co.cask.cdap.examples.myapp.plugins.DefaultStreamReader",
    "artifact": {
      "name": "streamreaders",
      "version": "1.0.0"
    }
  }
]
 
GET /namespaces/default/artifacts/myapp/versions/2.0.0/extensions/streamreader/plugins/default
[
  {
    "name": "default",
    "type": "reader",
    "description": "Writes timestamp and body as two columns and expects the row key to come as a header in the stream event.",
    "className": "co.cask.cdap.examples.myapp.plugins.DefaultStreamReader",
    "properties": {
      "rowkey": {
        "name": "rowkey",
        "description": "The header that should be used as the row key to write to. Defaults to 'rowkey'.",
        "type": "string",
        "required": false
      }
    }
    "artifact": {
      "name": "streamreaders",
      "version": "1.0.0"
    }
  }
]

2.3 Creating an Application that uses plugins

With this information a user is now able to create an Application that uses a Plugin:

PUT /namespaces/default/apps/userDump -H 'Content-Type: application/json' -d '
{ 
  "artifact": {
    "name": "myapp",
    "version": "2.0.0"
  },
  "config": {
    "stream": "users",
    "table": "events"
    "readerPlugin": "default",
    "readerPluginProperties": {
      "rowkey": "user-id"
    } 
  }
}'

3. System Artifacts

System artifacts are special artifacts that can be accessed in other namespaces. They cannot be deployed through the RESTful API unless a conf setting is set. Instead, they are placed in a directory on the CDAP master host. When CDAP starts up, the directory will be scanned and those artifacts will be added to the system. Example uses for system artifacts are the ETLBatch and ETLRealtime applications that we want to include out of the box.

...

PUT /namespaces/default/apps/somePipeline -d '
{ 
  "artifact": {
    "name":"ETLBatch",
    "version":"3.1.0",
    "isSystem": true
  },
  "config": { ... }
}'

4. Deleting an Artifact

Non-snapshot artifacts will be immutable. Advanced users can delete an existing artifact, but the assumption will be that they know exactly what they are doing. Deleting an artifact may cause programs that are using it to fail.

5. CDAP Upgrade

The programmatic API changes are all backwards compatible, so existing apps will not need to be recompiled. They will, however, need to be added to the artifact repository as part of the upgrade tool (or force people to redeploy their existing apps).

Any existing adapters will need to be migrated. Ideally, the upgrade tool will create matching applications based on the adapter conf, but at a minimum we will simply delete existing adapters and templates.

RESTful API changes

Application APIs

...

6. Application Versioning

This was mentioned in stories 1 and 2, but versioning is now explicitly managed by CDAP.

Suppose a development team is working on a search application. There is a dev instance of CDAP running, and an initial version 0.1.0-SNAPSHOT of the artifact is deployed, and a corresponding application is created from it:

POST /namespaces/default/artifacts/searchapp --data-binary @searchapp-0.1.0-SNAPSHOT.jar
 
PUT /namespaces/default/apps/search -H 'Content-Type: application/json' -d '
{ 
  "artifact": {
    "name": "searchapp",
    "version": "0.1.0-SNAPSHOT"
  },
  "config": { 
    "stream": "docs"
  }
}'

During development, every day, a new version of the artifact is built and deployed:

POST /namespaces/default/artifacts/searchapp --data-binary @searchapp-0.1.0-SNAPSHOT.jar

This replaces the version of the artifact that was there before.  Any running programs will be using the old code, but any new programs started after the artifact is added will use the new code.  Therefore, as part of the deployment process, application programs are restarted.  After some time, the initial version of the application code is deemed ready for release.  The project version is bumped to version 0.1.0, an artifact is built and deployed to CDAP:

POST /namespaces/default/artifacts/searchapp --data-binary @searchapp-0.1.0.jar

This adds a newer version of the artifact.  This version is not a snapshot version and is therefore immutable.  Attempts to re-deploy it will fail.  Deploying the artifact has no impact on existing applications.  The 'search' application will continue to use version 0.1.0-SNAPSHOT of the artifact until it is updated to the new version:

POST /namespaces/default/apps/search/update -d '
{ 
  "artifact": {
    "name": "searchapp",
    "version": "0.1.0"
  }
}

If no config is given, the existing config will be used. Otherwise, if a config is given, it will entirely replace the existing config. Artifact version cannot be changed unless all running programs are stopped.

After some time, some bugs are found and version 0.1.1 is developed and released. The jar is built and deployed to CDAP:

POST /namespaces/default/artifacts/searchapp --data-binary @searchapp-0.1.1.jar

The application is updated to use the new version of the artifact with the bug fixes:

POST /namespaces/default/apps/search/update -d '
{ 
  "artifact": {
    "name": "searchapp",
    "version": "0.1.1"
  }
}

After some more time, additional features are added and version 0.2.0 is built and released, and the application is changed to use the new version of the artifact:

POST /namespaces/default/artifacts/searchapp --data-binary @searchapp-0.2.0.jar
 
POST /namespaces/default/apps/search/update -d '
{ 
  "artifact": {
    "name": "searchapp",
    "version": "0.2.0"
  }
}

If there is any schema evolution happening or any other backwards impacting changes, they must be handled correctly by the application logic. CDAP will not migrate data or have any guarantees of compatibility between artifact versions.

During the release, a serious bug is discovered and the application is rolled back to use the previous artifact version:

POST /namespaces/default/apps/search/update -d '
{ 
  "artifact": {
    "name": "searchapp",
    "version": "0.1.1"
  }
}

Again, no compatibility guarantees are made by CDAP.  This operation may not be safe if the application logic does not make it safe, for example if there is data written in a new format that the old code cannot understand.

RESTful API changes

Application APIs

{ 
  'artifact': {'name':<name>, 'version':<version>}, 
  'config': { ... } 
}

{ 
  'artifact': {'name':<name>, 'version':<version>}, 
  'config': { ... } 
}

Artifact APIs

{
  "className": "co.cask.cdap.example.MyPlugin",
  "description": "My Plugin",
  "name": "MyPlugin",
  "properties": {
    "threshold": { "name": "thresh", "type": "int", "required": false },
    "user": { "name": "user", "type": "config", "required": true,
      "fields": {
        "id": { "name": "id", "type": "long", "required": true },
        "digits": { "name": "phoneNumber", "type": "string", "required": true }
      }
    }
  }
}

Template APIs (will be removed)