Copy data from or to MongoDB by using Azure Data Factory or Synapse Analytics

APPLIES TO: Azure Data Factory Azure Synapse Analytics

This article outlines how to use the Copy Activity in Azure Data Factory and Synapse Analytics pipelines to copy data from and to a MongoDB database. It builds on the copy activity overview article that presents a general overview of copy activity.

Supported capabilities

This MongoDB connector supports the following capabilities:

① Azure integration runtime ② Self-hosted integration runtime

For a list of data stores that are supported as sources and sinks, see the Supported data stores table.

Specifically, this MongoDB connector supports versions up to 4.2. If your work requires versions newer than 4.2, consider using MongoDB Atlas with the MongoDB Atlas connector, which provides more comprehensive support and features.

Prerequisites

If your data store is located inside an on-premises network, an Azure virtual network, or Amazon Virtual Private Cloud, you need to configure a self-hosted integration runtime to connect to it.

If your data store is a managed cloud data service, you can use the Azure Integration Runtime. If the access is restricted to IPs that are approved in the firewall rules, you can add Azure Integration Runtime IPs to the allow list.

You can also use the managed virtual network integration runtime feature in Azure Data Factory to access the on-premises network without installing and configuring a self-hosted integration runtime.

For more information about the network security mechanisms and options supported by Data Factory, see Data access strategies.

Getting started

To perform the copy activity with a pipeline, you can use one of the following tools or SDKs:

• Copy Data tool
• Azure portal
• .NET SDK
• Python SDK
• Azure PowerShell
• REST API
• Azure Resource Manager template

Create a linked service to MongoDB by using the UI

Use the following steps to create a linked service to MongoDB in the Azure portal UI.

1. Browse to the Manage tab in your Azure Data Factory or Synapse workspace and select Linked Services. Then select New:

   • Azure Data Factory
   • Azure Synapse

   

2. Search for MongoDB and select the MongoDB connector.

   

3. Configure the service details, test the connection, and create the new linked service.

   

Connector configuration details

The following sections provide details about properties that are used to define Data Factory entities specific to MongoDB connector.

Linked service properties

The following table lists the supported properties for a MongoDB linked service:

You can also put a connection string in Azure Key Vault. For more details, see Store credentials in Azure Key Vault. |Yes | | database | Name of the database that you want to access. | Yes | | connectVia | The Integration Runtime to use to connect to the data store. To learn more, see the Prerequisites section. If you don't specify this property, the default Azure Integration Runtime is used. |No |

Example:

{
    "name": "MongoDBLinkedService",
    "properties": {
        "type": "MongoDbV2",
        "typeProperties": {
            "connectionString": "mongodb://[username:password@]host[:port][/[database][?options]]",
            "database": "myDatabase"
        },
        "connectVia": {
            "referenceName": "<name of Integration Runtime>",
            "type": "IntegrationRuntimeReference"
        }
    }
}

Dataset properties

For a full list of sections and properties that you can use to define datasets, see Datasets and linked services. The following table lists the supported properties for a MongoDB dataset:

Example:

{
    "name": "MongoDbDataset",
    "properties": {
        "type": "MongoDbV2Collection",
        "typeProperties": {
            "collectionName": "<Collection name>"
        },
        "schema": [],
        "linkedServiceName": {
            "referenceName": "<MongoDB linked service name>",
            "type": "LinkedServiceReference"
        }
    }
}

Copy activity properties

For a full list of sections and properties available for defining activities, see the Pipelines article. This section provides a list of properties supported by MongoDB source and sink.

MongoDB as source

The copy activity source section supports the following properties:

Example:

"activities":[
    {
        "name": "CopyFromMongoDB",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<MongoDB input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "MongoDbV2Source",
                "filter": "{datetimeData: {$gte: ISODate(\"2018-12-11T00:00:00.000Z\"),$lt: ISODate(\"2018-12-12T00:00:00.000Z\")}, _id: ObjectId(\"5acd7c3d0000000000000000\") }",
                "cursorMethods": {
                    "project": "{ _id : 1, name : 1, age: 1, datetimeData: 1 }",
                    "sort": "{ age : 1 }",
                    "skip": 3,
                    "limit": 3
                }
            },
            "sink": {
                "type": "<sink type>"
            }
        }
    }
]

MongoDB as sink

The Copy Activity sink section supports the following properties:

Example

"activities":[
    {
        "name": "CopyToMongoDB",
        "type": "Copy",
        "inputs": [
            {
                "referenceName": "<input dataset name>",
                "type": "DatasetReference"
            }
        ],
        "outputs": [
            {
                "referenceName": "<Document DB output dataset name>",
                "type": "DatasetReference"
            }
        ],
        "typeProperties": {
            "source": {
                "type": "<source type>"
            },
            "sink": {
                "type": "MongoDbV2Sink",
                "writeBehavior": "upsert"
            }
        }
    }
]

Import and export JSON documents

Use this MongoDB connector to easily:

• Copy documents between two MongoDB collections as-is.
• Import JSON documents from various sources to MongoDB, including from Azure Cosmos DB, Azure Blob storage, Azure Data Lake Storage, and other supported file-based stores.
• Export JSON documents from a MongoDB collection to various file-based stores.

To achieve such schema-agnostic copy, skip the "structure" (also called schema) section in dataset and schema mapping in copy activity.

Data type mapping for MongoDB

When you copy data from MongoDB, the service uses the following mappings from MongoDB data types to interim data types. For more information about how copy activity maps the source schema and data type to the sink, see Schema and data type mappings.

MongoDB connector lifecycle and upgrade

The following table shows the release stage and change logs for different versions of the MongoDB connector:

Upgrade the MongoDB linked service

Create a new MongoDB linked service and configure it by referring to Linked service properties.

Related content

For a list of data stores supported as sources and sinks by the copy activity, see supported data stores.