Select a data ingestion tool

  • Article

This article describes a set of different tools used to transfer your historical data to the selected target platform. This table lists the tools available for each target platform, and general tools to help you with the ingestion process.

Azure Blob Storage

You can ingest data to Azure Blob Storage in several ways.

Review the Azure Data Factory (ADF) and Azure Synapse methods, which are better tailored to the data migration use case.

Azure Data Factory or Azure Synapse

To use the Copy activity in Azure Data Factory (ADF) or Synapse pipelines:

  1. Create and configure a self-hosted integration runtime. This component is responsible for copying the data from your on-premises host.
  2. Create linked services for the source data store (filesystem and the sink data store blob storage.
  3. To copy the data, use the Copy data tool. Alternatively, you can use method such as PowerShell, Azure portal, a .NET SDK, and so on.

AzCopy

AzCopy is a simple command-line utility that copies files to or from storage accounts. AzCopy is available for Windows, Linux, and macOS. Learn how to copy on-premises data to Azure Blob storage with AzCopy.

You can also use these options to copy the data:

Azure Data Box

In a scenario where the source SIEM doesn't have good connectivity to Azure, ingesting the data using the tools reviewed in this section might be slow or even impossible. To address this scenario, you can use Azure Data Box to copy the data locally from the customer's data center into an appliance, and then ship that appliance to an Azure data center. While Azure Data Box isn't a replacement for AzCopy or LightIngest, you can use this tool to accelerate the data transfer between the customer data center and Azure.

After you complete the migration, the data is available in a storage account under one of your Azure subscriptions. You can then use AzCopy, or ADF to ingest data from the storage account.

SIEM data migration accelerator

In addition to selecting an ingestion tool, your team needs to invest time in setting up the foundation environment. To ease this process, you can use the SIEM data migration accelerator, which automates the following tasks:

  • Deploys a Windows virtual machine that will be used to move the logs from the source to the target platform
  • Downloads and extracts the following tools into the virtual machine desktop:
    • AzCopy: Used to migrate data to Azure Blob Storage
  • Deploys the target platform that will host your historical logs:
    • Azure Storage account (Azure Blob Storage)
    • Azure Data Explorer cluster and database
    • Azure Monitor Logs workspace (Basic Logs; enabled with Microsoft Sentinel)

To use the SIEM data migration accelerator:

  1. From the SIEM data migration accelerator page.
  2. Select Basics, select your resource group and location, and then select Next.
  3. Select Migration VM, and do the following:
    • Type the virtual machine name, username and password.
    • Select an existing vNet or create a new vNet for the virtual machine connection.
    • Select the virtual machine size.
  4. Select Target platform and do one of the following:
    • Skip this step.
    • Provide the ADX cluster and database name, SKU, and number of nodes.
    • For Azure Blob Storage accounts, select an existing account. If you don't have an account, provide a new account name, type, and redundancy.
    • For Azure Monitor Logs, type the name of the new workspace.

Next steps

In this article, you learned how to select a tool to ingest your data into the target platform.

Ingest your data