Solutions for periodic data transfer
This article provides an overview of the data transfer solutions when you are transferring data periodically. Periodic data transfer over the network can be categorized as recurring at regular intervals or continuous data movement. The article also describes the recommended data transfer options and the respective key capability matrix for this scenario.
To understand an overview of all the available data transfer options, go to Choose an Azure data transfer solution.
Recommended options
The recommended options for periodic data transfer fall into two categories depending on whether the transfer is recurring or continuous.
Scripted/programmatic tools – For data transfer that occurs at regular intervals, use the scripted and programmatic tools such as AzCopy and Azure Storage REST APIs. These tools are targeted towards IT professionals and developers.
- AzCopy - Use this command-line tool to easily copy data to and from Azure Blobs, Files, and Table storage with optimal performance. AzCopy supports concurrency and parallelism, and the ability to resume copy operations when interrupted.
- Azure Storage REST APIs/SDKs – When building an application, you can develop the application against Azure Storage REST APIs and use the Azure SDKs offered in multiple languages. The REST APIs can also leverage the Azure Storage Data Movement Library designed especially for the high-performance copying of data to and from Azure.
Continuous data ingestion tools – For continuous, ongoing data ingestion, you can select one of the following options.
- Object replication - Object replication asynchronously copies block blobs between containers in a source and destination storage account. Use object replication as a solution to keep containers in two different storage accounts in sync.
- Azure Data Factory – Data Factory should be used to scale out a transfer operation, and if there is a need for orchestration and enterprise grade monitoring capabilities. Use Azure Data Factory to set up a cloud pipeline that regularly transfers files between several Azure services, on-premises, or a combination of the two. Azure Data Factory lets you orchestrate data-driven workflows that ingest data from disparate data stores and automate data movement and data transformation.
Azure Data Factory is set up by IT professionals and can transparently automate data transfer.
Comparison of key capabilities
The following table summarizes the differences in key capabilities.
Scripted/Programmatic network data transfer
| Capability | AzCopy | Azure Storage REST APIs |
|---|---|---|
| Form factor | Command-line tool from Microsoft | Customers develop against Storage REST APIs using Azure client libraries |
| Initial one-time setup | Minimal | Moderate, variable development effort |
| Data Format | Azure Blobs, Azure Files, Azure Tables | Azure Blobs, Azure Files, Azure Tables |
| Performance | Already optimized | Optimize as you develop |
| Pricing | Free, data egress charges apply | Free, data egress charges apply |
Continuous data ingestion over network
| Feature | Azure Data Factory |
|---|---|
| Form factor | Service in Azure portal, agent on-premises |
| Hardware | NA |
| Initial setup effort | Large (~days) |
| Data Format | Supports 70+ data connectors for data stores and formats |
| Data pre-processing | Yes |
| Local cache (to store on-premises data) |
No |
| Transfer from other clouds | Yes |
| Pricing | Pricing |