{"type":"doc","content":[{"type":"paragraph","content":[{"text":"This topic outlines the new metrics CDAP collects for BigQuery Pushdown pipelines in CDAP 6.7.2","type":"text"},{"text":" ","type":"text","marks":[{"type":"strong"}]},{"text":"and later releases.","type":"text"}]},{"type":"heading","attrs":{"level":2},"content":[{"text":"New metrics for BigQuery Pushdown pipelines","type":"text"}]},{"type":"paragraph","content":[{"text":"Using the Metrics Context supplied by the CDAP platform, we have added logic to capture additional metrics from the portions of the pipeline executed in BigQuery.","type":"text"}]},{"type":"paragraph","content":[{"text":"These metrics all share a common prefix:","type":"text"}]},{"type":"paragraph","content":[{"text":"user.pushdown.","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"For the BigQuery SQL Engine implementation, this results in:","type":"text"}]},{"type":"paragraph","content":[{"text":"user.pushdown.BigQueryPushdownEngine","type":"text","marks":[{"type":"code"}]}]},{"type":"paragraph","content":[{"text":"Additionally, metrics are further split in 2 further categories:","type":"text"}]},{"type":"orderedList","attrs":{"order":1},"content":[{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Pipeline metrics are collected in the platform, and relate to the number of stages that get pushed down, and the number of records that get pushed down.","type":"text"},{"type":"hardBreak"},{"text":".pipeline.records.in -> # input records to pushdown stages","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.records.out -> # output records from pushdown stages","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.records.pull -> # pushed records into the engine","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.records.push -> # pulled records from the engine","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.stages.count -> # executed stages in the engine","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.stages.count.join -> # executed join stages in the engine","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".pipeline.stages.count.transform -> # executed transform in the engine","type":"text","marks":[{"type":"code"}]}]}]},{"type":"listItem","content":[{"type":"paragraph","content":[{"text":"Engine metrics are collected by the engine itself, and can be used to collect resource utilization, cost and other metrics emitted by the engine directly.","type":"text"},{"type":"hardBreak"},{"text":".engine.bytes.billed -> Bytes Billed by BigQuery","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".engine.bytes.processed -> Bytes Processed by BigQuery","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":".engine.slot.ms -> Slot usage for BigQuery jobs","type":"text","marks":[{"type":"code"}]}]}]}]},{"type":"paragraph","content":[{"text":"A ","type":"text"},{"text":"non-exhaustive","type":"text","marks":[{"type":"strong"}]},{"text":" list of the metrics we collect from BigQuery engine:","type":"text"}]},{"type":"paragraph","content":[{"text":"POST v3/metrics/search?target=metric&tag=namespace:default&tag=app:the_pipeline_nameuser.pushdown.BigQueryPushdownEngine.engine.bytes.billed","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.engine.bytes.processed","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.engine.slot.ms","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.records.in","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.records.out","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.records.pull","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.records.push","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.join","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.transform","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.pull","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.push","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.spark_pull","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.spark_push","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.read","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"text":"user.pushdown.BigQueryPushdownEngine.pipeline.stages.count.write","type":"text","marks":[{"type":"code"}]},{"type":"hardBreak"},{"type":"hardBreak"}]}],"version":1}

Browser not supported