Set autopurge retention policies for Azure Functions Durable Task Scheduler (preview)

To prevent reaching the memory limit of a capacity unit (CU), it's best practice to periodically purge orchestration history data. The Durable Task Scheduler offers a lightweight, configurable autopurge feature that helps you manage orchestration data clean-up without manual intervention.

Autopurge operates asynchronously in the background, optimized to minimize system resource usage and prevent interference with other Durable Task operations. Although autopurge doesn't adhere to a strict schedule, its clean-up rate generally aligns with your orchestration scheduling rate.

How it works

Autopurge is an opt-in feature. You can enable it by defining retention policies that control how long to keep the data of orchestrations in certain statuses. The autopurge feature purges orchestration data associated with terminal statuses. "Terminal" refers to orchestrations that have reached a final state with no further scheduling, event processing, or work item generation. Terminal statuses include:

  • Completed
  • Failed
  • Canceled
  • Terminated

The orchestration instances eligible for autopurge match those targeted by the Durable SDK PurgeInstancesAsync API.

Autopurge ignores orchestration data associated with non-terminal statuses. "Non-terminal" statuses indicate that the orchestration instance is either actively executing, paused, or in a state where it may resume in the future (waiting for external events or timers). These orchestrations that are continuing as new, where the current execution is completed, but a new instance has been started as a continuation.

These statuses include:

  • Pending
  • Running
  • Suspended
  • Continued_As_New

Once enabled, autopurge periodically deletes orchestration data older than the retention period you set. Autopurge only

Note

Retention policies you define are applied to all task hubs in a scheduler.

Policy value

Retention value can range from 0 (purge as soon as possible) to the maximum integer value, with the unit being days.

The retention period refers to the time period since the orchestration entered terminal state. For example, you set a retention value of 1 day. If the orchestration takes 10 days to finish, autopurge won't delete it until the following day. Autopurge isn't triggered until the orchestration finishes.

Although retention periods have no maximum limit, we recommend you avoid retaining large volumes of stale orchestration data for extended periods. This practice ensures efficient use of storage resources and maintains optimal app performance.

Types of policies

When configuring an autopurge retention policy, you can set either a specific or a default policy.

  • Default policy purges orchestration data regardless of orchestrationState. The following policy purges orchestration data of all statuses covered by the feature after 2 days:

    {
  "retentionPeriodInDays": 2
}

  • Specific policy defines purging of orchestration data for specific orchestrationState. The following policy tells Durable Task Scheduler to keep completed orchestration data for 1 day, after which this data is purged.

    {
  "retentionPeriodInDays": 1,
  "orchestrationState": "Completed"
}

Add specific policies to override the default policy applied to orchestrations. In the following example, the second and third policies override the default policy ("retentionPeriodInDays": 1).

  • Data associated with completed orchestrations is deleted as soon as possible.

  • Data associated with failed orchestrations is purged after 60 days.

    [
  {
    "retentionPeriodInDays": 1
  },
  {
    "retentionPeriodInDays": 0,
    "orchestrationState": "Completed"
  },
  {
    "retentionPeriodInDays": 60,
    "orchestrationState": "Failed"
  }
]

Since no specific policy is set for canceled or terminated orchestrations, the default policy still applies to them, purging their data after 1 day.

For more information, see the API reference spec for Durable Task Scheduler retention policies.

Enable autopurge

You can define retention policies using:

  • Durable Task CLI
  • Azure Resource Manager (ARM)
  • Bicep

Make sure you have the latest version of the Durable Task CLI extension.

az extension add --name durabletask
az extension update --name durabletask

Create or update the retention policy by running the following command.

az durabletask retention-policy create --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP --default-days 1 --completed-days 0 --failed-days 60

The following properties specify the retention duration for orchestration data of different statuses.

Property Description
--canceled-days or -x The number of days to retain canceled orchestrations.
--completed-days or -c The number of days to retain completed orchestrations.
--default-days or -d The number of days to retain orchestrations.
--failed-days or -f The number of days to retain failed orchestrations.
--terminated-days or -t The number of days to retain terminated orchestrations.

Example response

If creation is successful, you receive the following response.

{
  "id": "/subscriptions/SUBSCRIPTION_ID/resourceGroups/RESOURCE_GROUP/providers/Microsoft.DurableTask/schedulers/SCHEDULER_NAMER/retentionPolicies/default",
  "name": "default",
  "properties": {
    "provisioningState": "Succeeded",
    "retentionPolicies": [
      {
        "retentionPeriodInDays": 1
      },
      {
        "orchestrationState": "Completed",
        "retentionPeriodInDays": 0
      },
      {
        "orchestrationState": "Failed",
        "retentionPeriodInDays": 60
      }
    ]
  },
  "resourceGroup": "RESOURCE_GROUP",
  "systemData": {
    "createdAt": "2025-04-23T23:41:17.3165122Z",
    "createdBy": "someone@microsoft.com",
    "createdByType": "User",
    "lastModifiedAt": "2025-04-23T23:41:17.3165122Z",
    "lastModifiedBy": "someone@microsoft.com",
    "lastModifiedByType": "User"
  },
  "type": "microsoft.durabletask/schedulers/retentionpolicies"
}

Tip

Learn more about the retention policy command via the CLI reference.

Disable autopurge

Delete the retention policies using the following command. The Durable Task Scheduler stops cleaning orchestration data within 5 to 10 minutes.

az durabletask retention-policy delete --scheduler-name SCHEDULER_NAME --resource-group RESOURCE_GROUP

Next steps

Monitor and manage your orchestration status and history using the Durable Task Scheduler dashboard.