The MongoDB Batch source plugin is available in the Hub.
Plugin version: 2.0.1
Reads documents from a MongoDB collection and converts each document into a StructuredRecord with the help of a specified schema. The user can optionally provide input query.
Configuration
Property | Macro Enabled? | Description |
|---|---|---|
Reference Name | No | Required. Name used to uniquely identify this source for lineage, annotating metadata, etc. |
Host | Yes | Required. Host that MongoDB is running on. Enter the hostname only. The connection string is constructed by the plugin. |
Port | Yes | Required. Port that MongoDB is listening to. Default is 27017. |
Database | Yes | Required. MongoDB database name. |
Collection | Yes | Required. Name of the database collection to read from. |
Input Query | Yes | Optional. Optionally filter the input collection with a query. This query must be represented in JSON format and use the MongoDB extended JSON format to represent non-native JSON data types. |
Username | Yes | Optional. User identity for connecting to the specified database. |
Password | Yes | Optional. Password to use to connect to the specified database. |
Authentication Connection String | Yes | Optional. MongoDB connection string to connect to the ‘config’ database of a sharded cluster. It can be omitted if username and password do not differ from the previously provided ones or if ‘config’ database does not require authentication. |
On Record Error | Yes | Required. Specifies how to handle error in record processing. An error will be thrown if failed to parse value according to a provided schema. |
Connection Arguments | Yes | Optional. A list of arbitrary string key/value pairs as connection arguments. See Connection String Options for a full description of these arguments. |
Data Type Mapping
MongoDB Data Type | CDAP Schema Data Type | Comments |
|---|---|---|
Boolean | boolean | |
Binary data | bytes | |
ObjectId | bytes | |
Double | double | |
Decimal128 | decimal | |
32-bit integer | int | |
64-bit integer | long | |
String | string | |
Symbol | string | |
Date | timestamp | |
Array | array | |
Object | record, map | Map keys must be a non-nullable string. |
Regular expression | Not supported. | |
DBPointer | Not supported. | |
JavaScript | Not supported. | |
JavaScript (with scope) | Not supported. | |
Timestamp | Not supported. Special type for internal MongoDB use. | |
Min key | Not supported. | |
Max key | Not supported. |