Introduction

Doubleclick campaign manager is a web-based ad management system for advertisers and agencies. It helps you manage your digital campaigns across websites and mobile. The plugins will allow users to perform ETL over reporting data for their display ad campaigns.

Use case(s)

• Read and replicate all of the Doubleclick Campaign Manager report data from your Google account. This allows you to merge ad data with other data sources such as customer information enrich it.

User Storie(s)

• As a data pipeline developer, I should be able to pull data from Doubleclick Campaign Manager, so that I can use it in my analytics pipelines with other data sources.
• As a data pipeline developer, I should be able to specify parameters for generating a report, and use the report data as a source in my pipeline
• As a data pipeline developer, I should not be exposed to the details such as where the report is stored, etc. It should be automatically managed by CDAP. I should only see the data from the report as output from my source.
• As a data pipeline developer, once I've specified parameters for Doubleclick Campaign Manager data, Data Fusion should handle and populate schema automatically, so I do not need any manual work.
• As a data pipeline developer, I should be able to pick the type of report to generate. It should be one of Instant, Standard, Floodlight, Path to Conversion (P2C), Reach, Cross-Dimension Reach or GRP. It should default to Standard.
• As a data pipeline developer, I should be able to specify the time range to pull the DCM data for. It should default to the last month's data. The possible values should be "LAST_14_DAYS", "LAST_24_MONTHS", "LAST_30_DAYS", "LAST_365_DAYS", "LAST_60_DAYS", "LAST_7_DAYS", "LAST_90_DAYS", "MONTH_TO_DATE", "PREVIOUS_MONTH", "PREVIOUS_QUARTER", "PREVIOUS_WEEK", "PREVIOUS_YEAR", "QUARTER_TO_DATE", "TODAY", "WEEK_TO_DATE", "YEAR_TO_DATE", "YESTERDAY".
• As a data pipeline developer, I should be able to specify dimensions to be included in the DCM data. I should be able to select the dimensions based on the chosen report type.
• As a data pipeline developer, I should be able to specify the metrics to be included in the DCM data. I should be able to select the metrics depending on the chosen report type. Full list here
• As a data pipeline developer, I should be able to read data from an existing report, so I don't have to spend upfront time in generating a new report.
• As a data pipeline developer, I should be able to use a data transfer set up by Google in GCS, so that I do not have to generate a report to query DCM data.

Plugin Type

• Batch Source
• Batch Sink 
• Real-time Source
• Real-time Sink
• Action
• Post-Run Action
• Aggregate
• Join
• Spark Model
• Spark Compute

Configurables

The Campaign manager sources should support two different plugins:

Campaign Manager Reporting plugin

This plugin should be used when you want to directly query the DCM APIs to run a report, and fetch the data. 

Pro: Doesn't require any extra set up from users. Handles the report generation itself.

Con: There is some upfront work involved in generating the report, so it may take some time.

Campaign Manager Transfer plugin

This plugin should be used when you want to read raw DCM data that is already uploaded on GCS

Pro: No upfront work involved in generating a new report

Con: Need to work with Google to get DCM files uploaded to GCS

Note: This plugin should be a wrapper over the existing GCS source, that hides some details from the user, but also auto-populates schemas.

Design / Implementation Tips

• It seems like the flow for the reporting plugin will be:

1. 1. Once the pipeline starts, request to create a report - https://developers.google.com/doubleclick-advertisers/v3.2/reports/insert
   2. Once the report is created, make an async request to run the report - https://developers.google.com/doubleclick-advertisers/v3.2/reports/run
   3. Keep polling for the report to be generated, then download it using https://developers.google.com/doubleclick-advertisers/guides/download_reports
   4. Once downloaded, read the CSV report and convert it into StructuredRecords for parallel execution.

Design

Approach(s)

Properties

Security

Limitation(s)

Future Work

• Some future work – HYDRATOR-99999
• Another future work – HYDRATOR-99999

Test Case(s)

• Test case #1
• Test case #2

Sample Pipeline

Please attach one or more sample pipeline(s) and associated data. 

Pipeline #1

Pipeline #2

References

https://developers.google.com/doubleclick-advertisers/getting_started