Data retention and archive in Azure Monitor Logs

Article
07/01/2024

Azure Monitor Logs retains data in two states:

Interactive retention: Lets you retain Analytics logs for interactive queries of up to 2 years.
Archive: Lets you keep older, less used data in your workspace at a reduced cost. You can access data in the archived state by using search jobs and restore. You can keep data in archived state for up to 12 years.

The default free-tier retention period is "interactive retention" for 30 days.

This article describes how to configure data retention and archiving.

How retention and archiving work

Each workspace has a default retention setting that's applied to all tables. You can configure a different retention setting on individual tables.

During the interactive retention period, data is available for monitoring, troubleshooting, and analytics. When you no longer use the logs, but still need to keep the data for compliance or occasional investigation, archive the logs to save costs.

Archived data stays in the same table, alongside the data that's available for interactive queries. When you set a total retention period that's longer than the interactive retention period, Log Analytics automatically archives the relevant data immediately at the end of the retention period.

You can access archived data by running a search job or restoring archived logs.

Note

The archive period can only be set at the table level, not at the workspace level.

Adjustments to retention and archive settings

When you shorten an existing retention setting, Azure Monitor waits 30 days before removing the data, so you can revert the change and avoid data loss in the event of an error in configuration. You can purge data immediately when required.

When you increase the retention setting, the new retention period applies to all data that's already been ingested into the table and hasn't yet been purged or removed.

If you change the archive settings on a table with existing data, the relevant data in the table is also affected immediately. For example, you might have an existing table with 180 days of interactive retention and no archive period. You decide to change the retention setting to 90 days of interactive retention without changing the total retention period of 180 days. Log Analytics immediately archives any data that's older than 90 days and none of the data is deleted.

What happens to data when you delete a table in a Log Analytics workspace

A Log Analytics workspace can contain several types of tables. What happens when you delete the table is different for each:

Table type	Data retention	Recommendations
Azure table	An Azure table holds logs from an Azure resource or data required by an Azure service or solution and cannot be deleted. When you stop streaming data from the resource, service, or solution, data remains in the workspace until the end of the retention period defined for the table or for the default workspace retention, if you do not define table-level retention.	To minimize charges, set table-level retention to four days before you stop streaming logs to the table.
Restored table `(table_RST`)	Deletes the hot cache provisioned for the restore, but source table data isn't deleted.
Search results table (`table_SRCH`)	Deletes the table and data immediately and permanently.
Custom log table (`table_CL`)	Soft deletes the table until the end of the table-level retention or default workspace retention period. During the soft delete period, you continue to pay for data retention and can recreate the table and access the data by setting up a table with the same name and schema. Fourteen days after you delete a custom table, Azure Monitor removes the table-level retention configuration and applies the default workspace retention.	To minimize charges, set table-level retention to four days before you delete the table.

Permissions required

Action	Permissions required
Configure data retention and archive policies for a Log Analytics workspace	`Microsoft.OperationalInsights/workspaces/write` and `microsoft.operationalinsights/workspaces/tables/write` permissions to the Log Analytics workspace, as provided by the Log Analytics Contributor built-in role, for example
Get the retention and archive policy by table for a Log Analytics workspace	`Microsoft.OperationalInsights/workspaces/tables/read` permissions to the Log Analytics workspace, as provided by the Log Analytics Reader built-in role, for example
Purge data from a Log Analytics workspace	`Microsoft.OperationalInsights/workspaces/purge/action` permissions to the Log Analytics workspace, as provided by the Log Analytics Contributor built-in role, for example

Configure the default workspace retention

You can set a Log Analytics workspace's default retention in the Azure portal to 30, 31, 60, 90, 120, 180, 270, 365, 550, and 730 days. You can apply a different setting to specific tables by configuring retention and archive at the table level. If you're on the free tier, you need to upgrade to the paid tier to change the data retention period.

Important

Workspaces with a 30-day retention might keep data for 31 days. If you need to retain data for 30 days only to comply with a privacy policy, configure the default workspace retention to 30 days using the API and update the immediatePurgeDataOn30Days workspace property to true. This operation is currently only supported using the Workspaces - Update API.

To set the default workspace retention:

From the Log Analytics workspaces menu in the Azure portal, select your workspace.
Select Usage and estimated costs in the left pane.
Select Data Retention at the top of the page.
Move the slider to increase or decrease the number of days, and then select OK.

To set the retention and archive duration for a table, call the Workspaces - Create Or Update API:

PATCH https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}?api-version=2023-09-01

Request body

The request body includes the values in the following table.

Name	Type	Description
`properties.retentionInDays`	integer	The workspace data retention in days. Allowed values are per pricing plan. See pricing tiers documentation for details.
`location`	string	The geo-location of the resource.
`immediatePurgeDataOn30Days`	boolean	Flag that indicates whether data is immediately removed after 30 days and is non-recoverable. Applicable only when workspace retention is set to 30 days.

Example

This example sets the workspace's retention to the workspace default of 30 days and ensures that data is immediately removed after 30 days and is non-recoverable.

Request

PATCH https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}?api-version=2023-09-01

{
  "properties": {
    "retentionInDays": 30,
    "features": {"immediatePurgeDataOn30Days": true}
    },
"location": "australiasoutheast"
}

**Response**

Status code: 200

```http
{
  "properties": {
    ...
    "retentionInDays": 30,
    "features": {
      "legacy": 0,
      "searchVersion": 1,
      "immediatePurgeDataOn30Days": true,
      ...
    },
    ...

To set the retention and archive duration for a table, run the az monitor log-analytics workspace update command and pass the --retention-time parameter.

This example sets the table's interactive retention to 30 days, and the total retention to two years, which means that the archive duration is 23 months:

az monitor log-analytics workspace update --resource-group myresourcegroup --retention-time 30 --workspace-name myworkspace

Use the Set-AzOperationalInsightsWorkspace cmdlet to set the retention for a workspace. This example sets the workspace's retention to 30 days:

Set-AzOperationalInsightsWorkspace -ResourceGroupName "myResourceGroup" -Name "MyWorkspace" -RetentionInDays 30

Configure retention and archive at the table level

By default, all tables in your workspace inherit the workspace's interactive retention setting and have no archive. You can modify the retention and archive settings of individual tables, except for workspaces in the legacy trial pricing tier.

The Analytics log data plan includes 31 days of interactive retention for workspaces in current-generation pricing tiers (Pay-as-you-go and Commitment Tiers as well as the legacy Standalone and Per Node tiers). You can increase the interactive retention period to up to 730 days at an additional cost. If needed, you can reduce the interactive retention period to as little as four days using the API or CLI. However, since 31 days of interactive retention are included in the ingestion price, lowering the retention period below 31 days doesn't reduce costs. You can set the archive period to a total retention time of up to 4,383 days (12 years).

Note

Currently, you can set total retention to up to 12 years through the Azure portal and API. CLI and PowerShell are limited to seven years; support for 12 years will follow.

To set the retention and archive duration for a table in the Azure portal:

From the Log Analytics workspaces menu, select Tables.

The Tables screen lists all the tables in the workspace.
Select the context menu for the table you want to configure and select Manage table.
Configure the retention and archive duration in the Data retention settings section of the table configuration screen.

To set the retention and archive duration for a table, call the Tables - Update API:

PATCH https://management.chinacloudapi.cn/subscriptions/{subscriptionId}/resourcegroups/{resourceGroupName}/providers/Microsoft.OperationalInsights/workspaces/{workspaceName}/tables/{tableName}?api-version=2022-10-01

Note

You don't explicitly specify the archive duration in the API call. Instead, you set the total retention, which is the sum of the interactive retention plus the archive duration.

You can use either PUT or PATCH, with the following difference:

The PUT API sets retentionInDays and totalRetentionInDays to the default value if you don't set non-null values.
The PATCH API doesn't change the retentionInDays or totalRetentionInDays values if you don't specify values.

Request body

The request body includes the values in the following table.

Name	Type	Description
properties.retentionInDays	integer	The table's data retention in days. This value can be between 4 and 730. Setting this property to null applies the workspace retention period. For a Basic Logs table, the value is always 8.
properties.totalRetentionInDays	integer	The table's total data retention including archive period. This value can be between 4 and 730; or 1095, 1460, 1826, 2191, 2556, 2922, 3288, 3653, 4018, or 4383. Set this property to null if you don't want to archive data.

Example

This example sets the table's interactive retention to the workspace default of 30 days, and the total retention to two years, which means that the archive duration is 23 months.

Request

PATCH https://management.chinacloudapi.cn/subscriptions/00000000-0000-0000-0000-00000000000/resourcegroups/testRG/providers/Microsoft.OperationalInsights/workspaces/testWS/tables/CustomLog_CL?api-version=2022-10-01

Request body

{
    "properties": {
        "retentionInDays": null,
        "totalRetentionInDays": 730
    }
}

Response

Status code: 200

{
    "properties": {
        "retentionInDays": 30,
        "totalRetentionInDays": 730,
        "archiveRetentionInDays": 700,
        ...        
    },
   ...
}

To set the retention and archive duration for a table, run the az monitor log-analytics workspace table update command and pass the --retention-time and --total-retention-time parameters.

This example sets the table's interactive retention to 30 days, and the total retention to two years, which means that the archive duration is 23 months:

az monitor log-analytics workspace table update --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name AzureMetrics --retention-time 30 --total-retention-time 730

To reapply the workspace's default interactive retention value to the table and reset its total retention to 0, run the az monitor log-analytics workspace table update command with the --retention-time and --total-retention-time parameters set to -1.

For example:

az monitor log-analytics workspace table update --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name Syslog --retention-time -1 --total-retention-time -1

Use the Update-AzOperationalInsightsTable cmdlet to set the retention and archive duration for a table. This example sets the table's interactive retention to 30 days, and the total retention to two years, which means that the archive duration is 23 months:

Update-AzOperationalInsightsTable -ResourceGroupName ContosoRG -WorkspaceName ContosoWorkspace -TableName AzureMetrics -RetentionInDays 30 -TotalRetentionInDays 730

To reapply the workspace's default interactive retention value to the table and reset its total retention to 0, run the Update-AzOperationalInsightsTable cmdlet with the -RetentionInDays and -TotalRetentionInDays parameters set to -1.

For example:

Update-AzOperationalInsightsTable -ResourceGroupName ContosoRG -WorkspaceName ContosoWorkspace -TableName Syslog -RetentionInDays -1 -TotalRetentionInDays -1

Get retention and archive settings by table

To view the retention and archive duration for a table in the Azure portal, from the Log Analytics workspaces menu, select Tables.

The Tables screen shows the interactive retention and archive period for all the tables in the workspace.

To get the retention setting of a particular table (in this example, SecurityEvent), call the Tables - Get API:

GET /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables/SecurityEvent?api-version=2022-10-01

To get all table-level retention settings in your workspace, don't set a table name.

For example:

GET /subscriptions/00000000-0000-0000-0000-00000000000/resourceGroups/MyResourceGroupName/providers/Microsoft.OperationalInsights/workspaces/MyWorkspaceName/Tables?api-version=2022-10-01

To get the retention setting of a particular table, run the az monitor log-analytics workspace table show command.

For example:

az monitor log-analytics workspace table show --subscription ContosoSID --resource-group ContosoRG --workspace-name ContosoWorkspace --name SecurityEvent

To get the retention setting of a particular table, run the Get-AzOperationalInsightsTable cmdlet.

For example:

Get-AzOperationalInsightsTable -ResourceGroupName ContosoRG -WorkspaceName ContosoWorkspace -tableName SecurityEvent

Tables with unique retention periods

By default, two data types, Usage and AzureActivity, keep data for at least 90 days at no charge. When you increase the workspace retention to more than 90 days, you also increase the retention of these data types. These tables are also free from data ingestion charges.

Tables related to Application Insights resources also keep data for 90 days at no charge. You can adjust the retention of each of these tables individually:

AppAvailabilityResults
AppBrowserTimings
AppDependencies
AppExceptions
AppEvents
AppMetrics
AppPageViews
AppPerformanceCounters
AppRequests
AppSystemEvents
AppTraces

Pricing model

The charge for maintaining archived logs is calculated based on the volume of data you archive, in GB, and the number or days for which you archive the data. Log data that has _IsBillable == false is not subject to retention or archive charges.

For more information, see Azure Monitor pricing.

Next steps

Learn more about:

Data retention and archive in Azure Monitor Logs

How retention and archiving work

Adjustments to retention and archive settings

What happens to data when you delete a table in a Log Analytics workspace

Permissions required

Configure the default workspace retention

Configure retention and archive at the table level

Get retention and archive settings by table

Tables with unique retention periods

Pricing model

Next steps

Additional resources