Audit Logging
Audit logging provides a chronological ledger containing evidence of operations or changes on CDAP entities. This information can be used to capture a trail of the activities that determined the state of an entity at a given point in time. These activities include the creation, modification, and deletion of an entity. It also includes modification of the entity's metadata. For data entities (datasets), it includes access information used to generate the entity's lineage.
Please note that audit logs are not published during a CDAP upgrade, as CDAP services are not available. Hence, any application which uses CDAP audit logs to sync metadata will go out of sync with respect to changes made during the upgrade. See CDAP-5954 for details.
Supported Audit Events
These audit events are supported in CDAP:
Audit Event Type | Supported Entities |
---|---|
CREATE |
|
UPDATE |
|
DELETE |
|
TRUNCATE |
|
ACCESS |
|
METADATA_CHANGE |
|
Configuring Audit Publishing
Audit publishing is controlled by these properties set in the cdap-site.xml
, as described in the Administration Manual:
audit.enabled
: Determines if publishing of audit logs is enabled; defaults toÂtrue
audit.topic
: Determines the topic to publish to; defaults toÂaudit
Consuming Audit Events
When audit publishing is enabled, for every audit event, a message is published to CDAP Kafka to the configured Kafka topic.
The contents of the message are a JSON representation of the AuditMessage class.
Here are some example JSON messages, pretty-printed:
Dataset Creation
{
"version": 1,
"time": 1000,
"entityId": {
"namespace": "ns1",
"dataset": "ds1",
"entity": "DATASET"
},
"user": "user1",
"type": "CREATE",
"payload": {}
}
Application Metadata Change
{
"version": 1,
"time": 3000,
"entityId": {
"namespace": "ns1",
"application": "app1",
"entity": "APPLICATION"
},
"user": "user1",
"type": "METADATA_CHANGE",
"payload": {
"previous": {
"USER": {
"properties": {
"uk": "uv",
"uk1": "uv2"
},
"tags": ["ut1", "ut2"]
},
"SYSTEM": {
"properties": {
"sk": "sv"
},
"tags": []
}
},
"additions": {
"SYSTEM": {
"properties": {
"sk": "sv"
},
"tags": ["t1", "t2"]
}
},
"deletions": {
"USER": {
"properties": {
"uk": "uv"
},
"tags": ["ut1"]
}
}
}
}
Â
Created in 2020 by Google Inc.