MongoDB Batch Source

The MongoDB Batch source plugin is available in the Hub.

Plugin version: 2.0.1

Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema. The user can optionally provide input query.

Configuration

Property

Macro Enabled?

Description

Property

Macro Enabled?

Description

Reference Name

No

Required. Name used to uniquely identify this source for lineage, annotating metadata, etc.

Host

Yes

Required. Host that MongoDB is running on. Enter the hostname only. The connection string is constructed by the plugin.

Port

Yes

Required. Port that MongoDB is listening to.

Default is 27017.

Database

Yes

Required. MongoDB database name.

Collection

Yes

Required. Name of the database collection to read from.

Input Query

Yes

Optional. Optionally filter the input collection with a query. This query must be represented in JSON format and use the MongoDB extended JSON format to represent non-native JSON data types.

Username

Yes

Optional. User identity for connecting to the specified database.

Password

Yes

Optional. Password to use to connect to the specified database.

Authentication Connection String

Yes

Optional. MongoDB connection string to connect to the ‘config’ database of a sharded cluster. It can be omitted if username and password do not differ from the previously provided ones or if ‘config’ database does not require authentication.

On Record Error

Yes

Required. Specifies how to handle error in record processing. An error will be thrown if failed to parse value according to a provided schema.

Default is Fail pipeline.

Connection Arguments

Yes

Optional. A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments.

Data Type Mapping

MongoDB Data Type

CDAP Schema Data Type

Comments

MongoDB Data Type

CDAP Schema Data Type

Comments

Boolean

boolean

 

Binary data

bytes

 

ObjectId

bytes

 

Double

double

 

Decimal128

decimal

 

32-bit integer

int

 

64-bit integer

long

 

String

string

 

Symbol

string

 

Date

timestamp

 

Array

array

 

Object

record, map

Map keys must be a non-nullable string.

Regular expression

 

Not supported.

DBPointer

 

Not supported.

JavaScript

 

Not supported.

JavaScript (with scope)

 

Not supported.

Timestamp

 

Not supported. Special type for internal MongoDB use.

Min key

 

Not supported.

Max key

 

Not supported.

Created in 2020 by Google Inc.