Salesforce Marketing Cloud Batch Source
Salesforce Marketing Cloud is a leading provider of Digital Marketing automation and analytics. It is used to create and manage marketing campaigns. Users can use the platform and the marketing APIs to create mailing lists, send emails, etc. This plugin would allow users to retrieve data from their Salesforce Marketing Cloud instance on objects such as accounts, activities, campaigns etc.
This plugin would support two modes of data retrieval:
Multi-object data extraction - In this mode, the user will be allowed to select multiple objects and get data associated with those objects in separate files/tables based on the sink choice. This mode can only be used with one of CDAP's Multi-* sinks.
Single-object data extraction - This mode would user to select a single object and extract the data associated with the specified object.
Use Cases
User would like to specify a Salesforce Marketing Cloud object so that she can get the entire dataset, perform intermediate transformations and load it in data warehouse for reporting and analytics purposes.
User would like to retrieve SFMC data for a list of objects to perform an extract and load pattern, so that all the needed objects are available in the sink for complex analytical queries.
User would like retrieve SFMC data so she can connect this data with other data sources to get full understanding of organization's digital marketing initiatives.
User would like to get SFMC data for a specific object so that it can be transformed and used for specific types of reports such as campaign monitoring, user access and security etc.
Plugin Type
User Configurations
Section | User Configuration Label | Description | Default | User Widget | Notes |
|---|---|---|---|---|---|
Authentication | Client ID | OAuth client ID for authentication |
| Text Box |
|
Client Secret | OAuth client ID for authentication |
| Password |
| |
Basic | Data retrieval mode | Select Multi-object to retrieve data on multiple objects; Select Single-object for extracting and transforming data for a specific object |
| Drop-down |
|
Object | Salesforce Marketing Cloud object name - ex: Campaign, Account, Emaiil etc. |
| Drop-down | Enabled only when data retrieval mode is set to Single-object | |
Objects list | List of Salesforce Marketing Cloud objects to retrieve |
| Text Box | Enabled only when data retrieval mode is set to Multi-object | |
Advanced | Error handling | Possible values: "Skip on error" - ignores any reports about records not inserted. Simply prints an error log.
| Skip on error | Select |
|
Design / Implementation Tips
Data retrieve since last batch option: https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/retrieving_all_records_since_last_batch.htm
API Integration: https://developer.salesforce.com/docs/atlas.en-us.mc-app-development.meta/mc-app-development/api-integration.htm
Design
Approach(s)
Properties
Security
Limitation(s)
Future Work
Test Case(s)
Test case #1
Test case #2
Sample Pipeline
Please attach one or more sample pipeline(s) and associated data.
Pipeline #1
Pipeline #2
References
Salesforce Marketing Cloud objects can be found at https://developer.salesforce.com/docs/atlas.en-us.noversion.mc-apis.meta/mc-apis/account.htm
Comments
@Shiva Gokaram - We should ask customer's Salesforce resource on #1 above. SFMC is heavily integrated with other applications and Salesforce team can guide us here. Are there options for pagination along with larger timeout that can help here?
I need to look into #2, but think that it can also be something SF can guide us in terms of best practices for integration. I do see that they have a concept of deltas with DataExtensions.
@Prateek Duble @Bhooshan Mogal
We have been working with the API's and wanted to share some findings and need some guidance.
Data Volumes and Timeouts - We have been playing with the API's and what we noticed is that when we use the MultiObject i.e retrive all the data for say Data Extensions , in the colgate environment they have about 400 Data extensions and each of them has thousands to millions of rows. Just retrieving 3 objects we hit the default transaction timeout and had to increase it. I cannot image how big of a time out value we need to provide if we need to pull all the data out.
I can see how this works for a one time usecase of load all the data from SFMC to Google cloud but how about deltas, there is no data time filter to support incremental daily loads. Even to support this , probably we can look into the standard objects for the time filter but for Data extension objects since this is all custom objects we dont even know how to figure out delta's between runs.
As discussed on the call looks like majority of the clients only use Data Extensions and was confirmed by the Sales Force Sales and PS teams. So let me know how you want to proceed.