Google Firestore Sink

The Google Firestore sink is available for Preview in the Hub.

Plugin version: 1.1.0

This sink writes data to Google Cloud Firestore.

Credentials

If the plugin is run on a Google Cloud Dataproc cluster, the service account key does not need to be provided and can be set to auto-detect. Credentials will be automatically read from the cluster environment.

If the plugin is not run on a Dataproc cluster, the path to a service account key must be provided. The service account key can be found on the Dashboard in the Cloud Platform Console. Make sure the account key has permission to access Google Cloud Firestore. The service account key file needs to be available on every node in your cluster and must be readable by all users running the job.

Configuration

Property

Macro Enabled?

Version Introduced

Description

Property

Macro Enabled?

Version Introduced

Description

Reference Name

No

 

Required. Name used to uniquely identify this sink for lineage, annotating metadata, etc.

Database Id

Yes

1.1.0

Optional. Firestore database name. If no value is provided, then (default) will be used.

Collection

Yes

 

Required. Name of the collection to write data. If the collection name does not exist in Firestore then new collection will be created first and then data written to it.

Document Id Type

Yes

 

Required. Type of ID(Key) assigned to documents written to the Cloud Firestore. The type can be one of two values:

  • Auto-generated id. ID(Key) will be generated by Cloud Firestore as a Alpha Numeric ID.

  • Custom name. ID(Key) will be provided as a field from the input records. The id field must not be nullable and must be of type STRING.

Document Id Alias

Yes

 

Optional. The field that will be used as the document id when writing to Cloud Firestore. This must be provided when the Document Id Type is not auto generated.

Service AccountType

Yes

 

Required. Service Account type, file path where the service account is located or the JSON content of the service account.

Service Account File Path

Yes

 

Optional. Path on the local file system of the service account key used for authorization. Can be set to auto-detect when running on a Dataproc cluster. When running on other clusters, the file must be present on every node in the cluster.

Default is auto-detect.

Service Account JSON

Yes

 

Optional. Content of the Service Account file.

Project ID

Yes

 

Optional. Google Cloud Project ID, which uniquely identifies a project. It can be found on the Dashboard in the Google Cloud Platform Console.

Default is auto-detect.

Batch Size

Yes

 

Required. Maximum number of documents that can be passed in one batch to a Commit operation. The minimum value is 1 and maximum value is 500.

Default is 25.

Created in 2020 by Google Inc.