Plugin version: 0.22.0
This sources reads from a Google Cloud Pub/Sub subscription in realtime. Cloud Pub/Sub brings the scalability, flexibility, and reliability of enterprise message-oriented middleware to the cloud. By providing many-to-many, asynchronous messaging that decouples senders and receivers, it allows for secure and highly available communication between independently written applications.
Credentials
If the plugin is run on a Google Cloud Dataproc cluster, the service account key does not need to be provided and can be set to 'auto-detect'. Credentials will be automatically read from the cluster environment.
If the plugin is not run on a Dataproc cluster, the path to a service account key must be provided. The service account key can be found on the Dashboard in the Cloud Platform Console. Make sure the account key has permission to access Google Cloud Pub/Sub. The service account key file needs to be available on every node in your cluster and must be readable by all users running the job.
Configuration
Property | Macro Enabled? | Version Introduced | Description |
---|---|---|---|
Reference Name | No | Required. Name used to uniquely identify this source for lineage, annotating metadata, etc. | |
Project ID | Yes | Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console. Default is auto-detect. | |
Subscription | Yes | Required. Name of the Google Cloud PubSub subscription to subscribe. If the subscription needs to be created then the topic to which the subscription will belong must be provided. Naming Convention for Subscription:
| |
Topic | Yes | Optional. Name of the Google Cloud PubSub topic to subscribe to. If a topic is provided and the given subscriber does not exists it will be created. If a subscriber does not exists and is created only the messages arrived after the creation of subscriber will be received. | |
Format | Yes | 6.4.0 / 0.17.0 | Optional. Format of the data to read. Supported formats are avro, blob, tsv, csv, delimited, json, parquet, and text. Default is text. |
Service Account Type | Yes | 6.3.0 / 0.16.0 | Optional. Select one of the following options:
|
Service Account File Path | Yes | Optional. Path on the local file system of the service account key used for authorization. Can be set to 'auto-detect' when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster. Default is auto-detect. | |
Service Account JSON | Yes | 6.3.0 / 0.16.0 | Optional. Content of the service account. |
Number of Readers | Yes | 6.4.0 / 0.17.0 | Optional. Number of Pub/Sub reader workers to run in parallel for this source. Each reader requires a worker in the cluster. The default number of readers per Pub/Sub Streaming Source is 1. |