MongoDB Batch Source
The MongoDB Batch source plugin is available in the Hub.
Plugin version: 2.0.1
Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema. The user can optionally provide input query.
Configuration
Property | Macro Enabled? | Description |
---|---|---|
Reference Name | No | Required. Name used to uniquely identify this source for lineage, annotating metadata, etc. |
Host | Yes | Required. Host that MongoDB is running on. Enter the hostname only. The connection string is constructed by the plugin. |
Port | Yes | Required. Port that MongoDB is listening to. Default is 27017. |
Database | Yes | Required. MongoDB database name. |
Collection | Yes | Required. Name of the database collection to read from. |
Input Query | Yes | Optional. Optionally filter the input collection with a query. This query must be represented in JSON format and use the MongoDB extended JSON format to represent non-native JSON data types. |
Username | Yes | Optional. User identity for connecting to the specified database. |
Password | Yes | Optional. Password to use to connect to the specified database. |
Authentication Connection String | Yes | Optional. MongoDB connection string to connect to the ‘config’ database of a sharded cluster. It can be omitted if username and password do not differ from the previously provided ones or if ‘config’ database does not require authentication. |
On Record Error | Yes | Required. Specifies how to handle error in record processing. An error will be thrown if failed to parse value according to a provided schema. Default is Fail pipeline. |
Connection Arguments | Yes | Optional. A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments. |
Data Type Mapping
MongoDB Data Type | CDAP Schema Data Type | Comments |
---|---|---|
Boolean | boolean |
|
Binary data | bytes |
|
ObjectId | bytes |
|
Double | double |
|
Decimal128 | decimal |
|
32-bit integer | int |
|
64-bit integer | long |
|
String | string |
|
Symbol | string |
|
Date | timestamp |
|
Array | array |
|
Object | record, map | Map keys must be a non-nullable string. |
Regular expression |
| Not supported. |
DBPointer |
| Not supported. |
JavaScript |
| Not supported. |
JavaScript (with scope) |
| Not supported. |
Timestamp |
| Not supported. Special type for internal MongoDB use. |
Min key |
| Not supported. |
Max key |
| Not supported. |
Created in 2020 by Google Inc.