Ingest historical data into your target platform

  • Article

In previous articles, you selected a tool to transfer your data and stored the historical data in a staging location. You can now start to ingest the data into the target platform.

This article describes how to ingest your historical data into your selected target platform.

Export data from the legacy SIEM

In general, SIEMs can export or dump data to a file in your local file system, so you can use this method to extract the historical data. It's also important to set up a staging location for your exported files. The tool you use to transfer the data ingestion can copy the files from the staging location to the target platform.

To export data from your current SIEM, see one of the following sections:

Ingest data to Microsoft Sentinel Basic Logs

To ingest your historical data into Microsoft Sentinel Basic Logs:

  1. If you don't have an existing Log Analytics workspace, create a new workspace and install Microsoft Sentinel.

  2. Create an App registration to authenticate against the API.

  3. Create a custom log table to store the data, and provide a data sample. In this step, you can also define a transformation before the data is ingested.

  4. Collect information from the data collection rule and assign permissions to the rule.

  5. Change the table from Analytics to Basic Logs.

  6. Run the Custom Log Ingestion script. The script asks for the following details:

    • Path to the log files to ingest
    • Microsoft Entra tenant ID
    • Application ID
    • Application secret
    • DCE endpoint (Use the logs ingestion endpoint URI for the DCR)
    • DCR immutable ID
    • Data stream name from the DCR

    The script returns the number of events that have been sent to the workspace.

Ingest to Azure Blob Storage

To ingest your historical data into Azure Blob Storage:

  1. Install and configure AzCopy on the system to which you exported the logs. Alternatively, install AzCopy on another system that has access to the exported logs.
  2. Create an Azure Blob Storage account and copy the authorized Microsoft Entra ID credentials or Shared Access Signature token.
  3. Run AzCopy with the folder path that includes the exported logs as the source, and the Azure Blob Storage connection string as the output.

Next steps

In this article, you learned how to ingest your data into the target platform.

Convert your dashboards to workbooks