Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

Parameter Name

Default Value

Description

log.base.dir

/logs/avro

In Distributed CDAP, the HDFS directory under which the system log pipeline saves log files.

log.buffer.base.dir

${local.data.dir}/logs.buffer

Base directory for logs buffer.

log.buffer.max.file.size.bytes

104857600

Maximum size in bytes of a log buffer file.

log.buffer.server.bind.address

0.0.0.0

Log buffer HTTP service bind address.

log.buffer.server.bind.port

0

Log buffer HTTP service bind port.

app.program.log.appender.extensions.dir

/opt/cdap/master/ext/log/appenders

Directory for log appender jars.

app.program.log.appender.provider

 

Provider for log appender. The log appender provided by this provider will be loaded in each program container.

app.program.log.appender.provisioners

native

Comma-separated list of provisioners to enable log appender provider for program executions.

log.buffer.recovery.batch.size

5000

Log buffer recovery batch size in number of log events read. This will be used to recover logs in batches from log buffer upon log service restart.

log.kafka.topic

logs.user-v2

Kafka topic name used to publish logs.

log.tms.topic.prefix

logs

TMS topic prefix used to publish logs.

log.queue.size

2048

The buffer size used in Log Appender.

log.pipeline.cdap.dir.permissions

700

Permissions used by the system log pipeline when creating directories.

log.pipeline.cdap.file.cleanup.interval.mins

1440

Time in minutes between runs of the log cleanup thread.

log.pipeline.cdap.file.cleanup.batch.size

10000

Batch size to clean up log metadata table.

log.pipeline.cdap.file.max.lifetime.ms

21600000

Maximum time span in milliseconds of a log file created by the system log pipeline.

log.pipeline.cdap.file.max.size.bytes

104857600

Maximum size in bytes of a log file created by the system log pipeline.

log.pipeline.cdap.file.permissions

600

Permissions used by the system log pipeline when creating files.

log.pipeline.cdap.file.retention.duration.days

7

Time in days a log file is retained.

log.pipeline.cdap.file.sync.interval.bytes

10485760

Number of bytes for the sync interval setting of the Avro file written by the system log pipeline.

log.process.pipeline.auto.buffer.ratio

0.7

The ratio of total memory used for determining processing pipeline buffer size. This property is only used if${log.process.pipeline.buffer.size} is set to zero and this value should be greater than zero and less than one.

log.process.pipeline.buffer.size

0

The internal buffer size in bytes for each log processing pipeline. Setting it to zero means the system will determine it dynamically based on the container size as given by ${log.saver.container.memory.mb}.

log.process.pipeline.checkpoint.interval.ms

10000

The time between log processing pipeline checkpoints in milliseconds.

log.process.pipeline.config.dir

/opt/cdap/master/ext/logging/config

A local directory on the CDAP Master that is scanned for log processing pipeline configurations. Each pipeline is defined by a file in the logback XML format, with ".xml" as the file name extension.

log.process.pipeline.event.delay.ms

2000

The time a log event stays in the log processing pipeline buffer before writing out to log appenders in milliseconds. A longer delay will result in better time ordering of log events before presenting to log appenders but will consume more memory.

log.process.pipeline.kafka.fetch.size

1048576

The buffer size in bytes, per topic partition, for fetching log events from Kafka.

log.process.pipeline.lib.dir

/opt/cdap/master/ext/logging/lib

Comma-separated list of local directories on the CDAP Master scanned for additional library JAR files to be included for log processing.

log.process.pipeline.logger.cache.expiration.ms

300000

Logger cache entry expiration time in milliseconds

log.process.pipeline.logger.cache.size

1000

The number of loggers that each log processing pipeline will cache.

log.publish.num.partitions

10

Number of CDAP Kafka service partitions to publish the logs to.

log.publish.partition.key

program

Publish logs from an application or a program to the same partition. Valid values are "application" or "program". If set to "application", logs from all the programs of an application go to the same partition. If set to "program", logs from the same program go to the same partition. Changes to this property requires restarting of all CDAP applications.

log.query.server.bind.address

0.0.0.0

Log query HTTP service bind address.

log.query.server.bind.port

0

Log query HTTP service bind port.

log.saver.container.memory.mb

${master.service.memory.mb}

Memory in megabytes for each log saver instance to run in YARN.

log.saver.container.num.cores

2

Number of virtual cores for each log saver instance in YARN.

log.saver.max.instances

${master.service.max.instances}

Maximum number of log saver instances to run in YARN.

log.saver.num.instances

1

Number of log saver instances to run in YARN.

log.saver.status.bind.address

0.0.0.0

Log saver HTTP service bind address.