Motivation

CDAP has - roughly speaking - two sets of APIs: the programming APIs that are available for developing applications (and plugins, datasets, tests, etc.), and administrative or operational APIs that are available to external clients through REST calls. There is a little (but not much) overlap between the two sets of APIs, and that has its foundation in the fact that the two APIs are targeted for distinctly different audiences: one is for developers, the other is for operations. 

However, as we evolve the platform the use cases that it solves, the line between the two sets of APIs becomes fuzzy. For example, data operations on datasets are only available within application code. But what if an operator needs to inspect some data in order to investigate a problem? There is no easy way to perform a Get on a row of a table through the REST APIs. If the operator is lucky, the Table is explorable and he can issue a SQL query. Or he can develop a service that exposes the Table through its REST API. The answer to particular problem is "Data as a Service" and it is beyond the scope of this document. Instead we will identify more use cases, where it is desirable to call operational APIs from inside an application. 

What does the operational API include? Here is an (incomplete) list:

• Dataset administration: Create, drop, alter, truncate datasets;
• Program lifecycle: Start, stop, pause programs
• Application lifecycle: Create app (from artifact), drop app
• Metrics+Logs: Retrieve/query operational metrics and logs from an application/program/dataset
• Lineage and Metadata: define and query metadata for apps, programs, datasets etc; query the (system-generated) lineage metadata
• etc.

Typically, these kinds of operations are not done from inside an application. But here are some reasons why we may want to allow that: 

Uber-Apps