Change log level dynamically

Problems: Currently if we have to change CDAP program's log-level, we have to change the log level in logback.xml in (src/main/resources) of the app and redeploy the app.If the app's program is already running, it has to be restarted to get the logback changes in the app.

Reason: Currently when we start a program, we will look for “logback.xml” in the program classLoader. If that is not provided, we will use the default file “logback-container.xml”. Therefore, if we want to change the log level now, we will have to make changes to the “logback.xml” and have to redeploy the app to reflect the changes.

Requirement/Goals: Our goal is to make log level change dynamically at runtime if there is a request.

Use cases: 

Users can change log level of any applications or programs dynamically through the API. They should be able to see the log level change at runtime without redeploying the applications or restarting the programs. They should also be able to change log level at class level to specify which level they want to see for each class.

Proposal:

 We will have two parts to fulfill this goal. Part 1: we will enable log level change per program run without redeploying the app. With this, we should be able to see log level change after restarting the program. Part 2: we will enable the ability to update the aggregate log level for a particular runnable in a Twill application.

For Part 1: Enable log level change per program run without redeploying the app

Step 1: Use the preferences to pass log level information

Pass the log level information as preferences and change the log level correspondingly once the program starts. With the property of the preferences, the log level can be set through namespace, application and program level. The key of the preferences should be “system.log.level”, and the value will be one of the valid log levels. (ALL, OFF, TRACE, DEBUG, INFO, WARN, ERROR) If the value is invalid log levels, this log level change will be ignored and the default log level "INFO" will be used.

In AbstractProgramTwillRunnable, DistributedMapReduceTaskContextProvider, and SparkRuntimeContextProvider, we get the log level information by the key “system.log.level” from ProgramOptions. Make LogAppenderInitializer’s initialize method accept the log level and change the root logger’s log level.

After step 1, the log level should be correctly changed for all classes in programs except for classes with same namespace prefixes in logback-container.xml. Changing the root logger is not enough since the loggers for these classes will get overridden and therefore their log level will be INFO.

Step 2: Enable the log level for class by passing the class name with prefix “system.log.level”.

Extract the prefix “system.log.level”, get all the classes and levels. Then pass them to LogAppenderInitializer to change the log level for their loggers. For a flow, preferences with key format "flowlet.[flowletname].system.log.level.[class]" can be passed to change the log level of a specific flowlet. The idea is like how scoping works for system.resources.memory, we extract the scope of flowlet.name, and set the log level in FlowTwillRunnable.

Step 3: Remove some settings in logback-container.xml

For these loggers in logback-container.xml, it is unclear to the user when they want to change the log levels related to these loggers. So it is better to remove them from the xml and add them to system preferences.

For Part 2: Enable log level change dynamically

REST API changes: The changes will be made in PreferencesHttpHandler and happen in application or program level:

GET /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel

GET /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel

Get the current log level of the application or the program. The response should be a JSON string map of the log level: {"loglevel":"value1"}

PUT  /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel {request}

PUT /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel {request}

Set preferences for the application or programs, the request body should be a json related to the log level, one out of TRACE, DEBUG, INFO, WARN, ERROR, FATAL

DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/loglevel

DELETE /v3/namespaces/<namespace-id>/apps/<app-id>/<program-type>/<program-id>/loglevel

 Delete the preferences of the application or programs, the program should use the configuration in “logback.xml” as the log level.

Rest: To be decided...