Configure data persistence for an Azure Cache for Redis instance
Redis persistence allows you to persist data stored in cache instance. If there's a hardware failure, the cache instance is rehydrated with data from the persistence file when it comes back online. The ability to persist data is an important way to boost the durability of a cache instance because all cache data is stored in memory. Data loss is possible if a failure occurs when cache nodes are down. Persistence should be a key part of your high availability and disaster recovery strategy with Azure Cache for Redis.
Warning
If you are using persistence on the Premium tier, check to see if your storage account has soft delete enabled before using the data persistence feature. Using data persistence with soft delete causes very high storage costs. For more information, see should I enable soft delete?.
Scope of availability
| Tier | Basic, Standard | Premium |
|---|---|---|
| Available | No | Yes |
Types of data persistence in Redis
You have two options for persistence with Azure Cache for Redis: the Redis database (RDB) format and Append only File (AOF) format:
- RDB persistence - When you use RDB persistence, Azure Cache for Redis persists a snapshot of your cache in a binary format. The snapshot is saved in an Azure Storage account. The configurable backup frequency determines how often to persist the snapshot. If a catastrophic event occurs that disables both the primary and replica cache, the cache is reconstructed automatically using the most recent snapshot. Learn more about the advantages and disadvantages of RDB persistence.
- AOF persistence - When you use AOF persistence, Azure Cache for Redis saves every write operation to a log. The log is saved at least once per second in an Azure Storage account. If a catastrophic event occurs that disables both the primary and replica caches, the cache is reconstructed automatically using the stored write operations. Learn more about the advantages and disadvantages of AOF persistence.
Azure Cache for Redis persistence features are intended to be used to restore data automatically to the same cache after data loss. The RDB/AOF persisted data files can't be imported to a new cache or the existing cache. To move data across caches, use the Import and Export feature. For more information, see Import and Export data in Azure Cache for Redis.
To generate any backups of data that can be added to a new cache, you can write automated scripts using PowerShell or CLI that export data periodically.
Prerequisites and limitations
Persistence features are intended to be used to restore data to the same cache after data loss.
- RDB/AOF persisted data files can't be imported to a new cache or the existing cache. Use the Import/Export feature instead.
- Persistence isn't supported with caches using passive geo-replication or active geo-replication.
- On the Premium tier, AOF persistence isn't supported with multiple replicas.
- On the Premium tier, data must be persisted to a storage account in the same region as the cache instance.
- On the Premium tier, storage accounts in different subscriptions can be used to persist data if managed identity is used to connect to the storage account.
How to set up data persistence using the Azure portal
To create a Premium cache, sign in to the Azure portal and select Create a resource. You can create caches in the Azure portal. You can also create them using Resource Manager templates, PowerShell, or Azure CLI. For more information about creating an Azure Cache for Redis, see Create a cache.
On the Create a resource page, select Databases and then select Azure Cache for Redis.
On the New Redis Cache page, configure the settings for your new premium cache.
Setting Suggested value Description DNS name Enter a globally unique name. The cache name must be a string between 1 and 63 characters that contain only numbers, letters, or hyphens. The name must start and end with a number or letter, and can't contain consecutive hyphens. The host name for your cache instance's is \<DNS name>.redis.cache.chinacloudapi.cn.Subscription Drop-down and select your subscription. The subscription under which to create this new Azure Cache for Redis instance. Resource group Drop-down and select a resource group, or select Create new and enter a new resource group name. Name for the resource group in which to create your cache and other resources. By putting all your app resources in one resource group, you can easily manage or delete them together. Location Drop-down and select a location. Select a region near other services that use your cache. Cache type Drop-down and select a premium cache to configure premium features. For details, see Azure Cache for Redis pricing. The pricing tier determines the size, performance, and features that are available for the cache. For more information, see Azure Cache for Redis Overview. Select the Networking tab or select the Networking button at the bottom of the page.
In the Networking tab, select your connectivity method. For premium cache instances, you connect either publicly, via Public IP addresses or service endpoints. You connect privately using a private endpoint.
Select the Next: Advanced tab or select the Next: Advanced button on the bottom of the page.
In the Advanced tab for a premium cache instance, configure the settings for non-TLS port, clustering, and data persistence. For data persistence, you can choose either RDB or AOF persistence.
To enable RDB persistence, select RDB and configure the settings.
Setting Suggested value Description Authentication Method Drop-down and select an authentication method. Choices are Managed Identity or Storage Key Choose your preferred authentication method. Using managed identity allows you to use a storage account in a different subscription than the one in which your cache is located. Subscription Drop-down and select a subscription. You can choose a storage account in a different subscription if you're using managed identity as the authentication method. Backup Frequency Drop-down and select a backup interval. Choices include 15 Minutes, 30 minutes, 60 minutes, 6 hours, 12 hours, and 24 hours. This interval starts counting down after the previous backup operation successfully completes. When it elapses, a new backup starts. Storage Account Drop-down and select your storage account. Choose a storage account in the same region and subscription as the cache. A Premium Storage account is recommended because it has higher throughput. Also, we strongly recommend that you disable the soft delete feature on the storage account as it leads to increased storage costs. For more information, see Pricing and billing. Storage Key Drop-down and choose either the Primary key or Secondary key to use. If the storage key for your persistence account is regenerated, you must reconfigure the key from the Storage Key drop-down. The first backup starts once the backup frequency interval elapses.
Note
When RDB files are backed up to storage, they are stored in the form of page blobs. If you're using a storage account with HNS enabled, persistence will tend to fail because page blobs aren't supported in storage accounts with HNS enabled (ADLS Gen2).
To enable AOF persistence, select AOF and configure the settings.
Setting Suggested value Description Authentication Method Drop-down and select an authentication method. Choices are Managed Identity or Storage Key Choose your preferred authentication method. Using managed identity allows you to use a storage account in a different subscription than the one in which your cache is located. Subscription Drop-down and select a subscription. You can choose a storage account in a different subscription if you're using managed identity as the authentication method. First Storage Account Drop-down and select your storage account. Choose a storage account in the same region and subscription as the cache. A Premium Storage account is recommended because it has higher throughput. Also, we strongly recommend that you disable the soft delete feature on the storage account as it leads to increased storage costs. For more information, see Pricing and billing. First Storage Key Drop-down and choose either the Primary key or Secondary key to use. If the storage key for your persistence account is regenerated, you must reconfigure the key from the Storage Key drop-down. Second Storage Account (Optional) Drop-down and select your secondary storage account. You can optionally configure another storage account. If a second storage account is configured, the writes to the replica cache are written to this second storage account. Second Storage Key (Optional) Drop-down and choose either the Primary key or Secondary key to use. If the storage key for your persistence account is regenerated, you must reconfigure the key from the Storage Key drop-down. With AOF persistence enabled, write operations to the cache are saved to the named storage account (or accounts if you've configured a second storage account). If there's a catastrophic failure that takes down both the primary and replica cache, the stored AOF log is used to rebuild the cache.
Select the Next: Tags tab or select the Next: Tags button at the bottom of the page.
Optionally, in the Tags tab, enter the name and value if you wish to categorize the resource.
Select Review + create. You're taken to the Review + create tab where Azure validates your configuration.
After the green Validation passed message appears, select Create.
It takes a while for the cache to create. You can monitor progress on the Azure Cache for Redis Overview page. When Status shows as Running, the cache is ready to use.
How to set up data persistence using PowerShell and Azure CLI
Managing data encryption
Because Redis persistence creates data at rest, encrypting this data is an important concern for many users. Encryption options vary based on the tier of Azure Cache for Redis being used.
With the Premium tier, data is streamed directly from the cache instance to Azure Storage when persistence is initiated. Various encryption methods can be used with Azure Storage, including Microsoft-managed keys, customer-managed keys, and customer-provided keys. For information on encryption methods, see Azure Storage encryption for data at rest.
Persistence FAQ
The following list contains answers to commonly asked questions about Azure Cache for Redis persistence.
- Can I enable persistence on a previously created cache?
- Can I enable AOF and RDB persistence at the same time?
- How does persistence work with geo-replication?
- Which persistence model should I choose?
- What happens if I've scaled to a different size and a backup is restored that was made before the scaling operation?
- Can I use the same storage account for persistence across two different caches?
- Will I be charged for the storage being used in Data Persistence
- How frequently does RDB and AOF persistence write to my blobs, and should I enable soft delete?
- Will having firewall exceptions on the storage account affect persistence
- How do I check if soft delete is enabled on my storage account?
RDB persistence
- Can I change the RDB backup frequency after I create the cache?
- Why is there more than 60 minutes between backups when I have an RDB backup frequency of 60 minutes?
- What happens to the old RDB backups when a new backup is made?
AOF persistence
- When should I use a second storage account?
- Does AOF persistence affect throughput, latency, or performance of my cache?
- How can I remove the second storage account?
- What is a rewrite and how does it affect my cache?
- What should I expect when scaling a cache with AOF enabled?
- How is my AOF data organized in storage?
- Can I have AOF persistence enabled if I have more than one replica?
Can I enable persistence on a previously created cache?
Yes, persistence can be configured both at cache creation and on existing Premium caches.
Can I enable AOF and RDB persistence at the same time?
No, you can enable RDB or AOF, but not both at the same time.
How does persistence work with geo-replication?
If you enable data persistence, geo-replication can't be enabled for your cache.
Which persistence model should I choose?
AOF persistence saves every write to a log, which has a significant effect on throughput. Compared AOF with RDB persistence, which saves backups based on the configured backup interval with minimal effect to performance. Choose AOF persistence if your primary goal is to minimize data loss, and you can handle a lower throughput for your cache. Choose RDB persistence if you wish to maintain optimal throughput on your cache, but still want a mechanism for data recovery.
- Learn more about the advantages and disadvantages of RDB persistence.
- Learn more about the advantages and disadvantages of AOF persistence.
For more information on performance when using AOF persistence, see Does AOF persistence affect throughput, latency, or performance of my cache?
Does AOF persistence affect throughput, latency, or performance of my cache?
AOF persistence does affect throughput. AOF runs on both the primary and replica process, therefore you see higher CPU and Server Load for a cache with AOF persistence than an identical cache without AOF persistence. AOF offers the best consistency with the data in memory because each write and delete is persisted with only a few seconds of delay. The trade-off is that AOF is more compute intensive.
As long as CPU and Server Load are both less than 90%, there's a penalty on throughput, but the cache operates normally, otherwise. Above 90% CPU and Server Load, the throughput penalty can get much higher, and the latency of all commands processed by the cache increases. Latency increases because AOF persistence runs on both the primary and replica process, increasing the load on the node in use, and putting persistence on the critical path of data.
What happens if I've scaled to a different size and a backup is restored that was made before the scaling operation?
For both RDB and AOF persistence:
- If you've scaled to a larger size, there's no effect.
- If you've scaled to a smaller size, and you have a custom databases setting that is greater than the databases limit for your new size, data in those databases isn't restored. For more information, see Is my custom databases setting affected during scaling?
- If you've scaled to a smaller size, and there isn't enough room in the smaller size to hold all of the data from the last backup, keys are evicted during the restore process. Typically, keys are evicted using the allkeys-lru eviction policy.
Can I use the same storage account for persistence across two different caches?
No, you must use different storage accounts for different caches. Each cache must have its own storage account to set up for persistence.
Important
Use separate storage accounts for persistence and performing periodic export operations on a cache.
Will I be charged for the storage being used in data persistence?
- For Premium caches, you're charged for the storage being used per the pricing model of the storage account being used.
How frequently does RDB and AOF persistence write to my blobs, and should I enable soft delete?
We recommend that you avoid enabling soft delete on storage accounts when used with Azure Cache for Redis data persistence with the Premium tier. RDB and AOF persistence can write to your blobs as frequently as every hour, every few minutes, or every second. Also, enabling soft delete on a storage account means Azure Cache for Redis can't minimize storage costs by deleting the old backup data.
Soft delete quickly becomes expensive with the typical data sizes of a cache that also performs write operations every second. For more information on soft delete costs, see Pricing and billing.
Can I change the RDB backup frequency after I create the cache?
Yes, you can change the backup frequency for RDB persistence using the Azure portal, CLI, or PowerShell.
Why is there more than 60 minutes between backups when I have an RDB backup frequency of 60 minutes?
The RDB persistence backup frequency interval doesn't start until the previous backup process has completed successfully. If the backup frequency is 60 minutes and it takes a backup process 15 minutes to complete, the next backup won't start until 75 minutes after the start time of the previous backup.
What happens to the old RDB backups when a new backup is made?
All RDB persistence backups, except for the most recent one, are automatically deleted. This deletion might not happen immediately, but older backups aren't persisted indefinitely. If you're using the Premium tier for persistence, and soft delete is turned on for your storage account, the soft delete setting applies, and existing backups continue to reside in the soft delete state.
When should I use a second storage account?
Use a second storage account for AOF persistence when you think you've higher than expected set operations on the cache. Setting up the secondary storage account helps ensure your cache doesn't reach storage bandwidth limits. This option is only available for Premium tier caches.
How can I remove the second storage account?
You can remove the AOF persistence secondary storage account by setting the second storage account to be the same as the first storage account. For existing caches, access Data persistence from the Resource menu for your cache. To disable AOF persistence, select Disabled.
What is a rewrite and how does it affect my cache?
When the AOF file becomes large enough, a rewrite is automatically queued on the cache. The rewrite resizes the AOF file with the minimal set of operations needed to create the current data set. During rewrites, you can expect to reach performance limits sooner, especially when dealing with large datasets. Rewrites occur less often as the AOF file becomes larger, but take a significant amount of time when it happens.
What should I expect when scaling a cache with AOF enabled?
If the AOF file at the time of scaling is large, then expect the scale operation to take longer than expected because it reloads the file after scaling has finished.
For more information on scaling, see What happens if I've scaled to a different size and a backup is restored that was made before the scaling operation?
How is my AOF data organized in storage?
When you use the Premium tier, data stored in AOF files is divided into multiple page blobs per shard. By default, half of the blobs are saved in the primary storage account and half are saved in the secondary storage account. Splitting the data across multiple page blobs and two different storage accounts increases the performance.
If the peak rate of writes to the cache isn't very high, then this extra performance might not be needed. In that case, the secondary storage account configuration can be removed. All of the AOF files are instead stored in just the single primary storage account. The following table displays how many total page blobs are used for each pricing tier:
| Premium tier | Blobs |
|---|---|
| P1 | 8 per shard |
| P2 | 16 per shard |
| P3 | 32 per shard |
| P4 | 40 per shard |
When clustering is enabled, each shard in the cache has its own set of page blobs, as indicated in the previous table. For example, a P2 cache with three shards distributes its AOF file across 48 page blobs: sixteen blobs per shard, with three shards.
After a rewrite, two sets of AOF files exist in storage. Rewrites occur in the background and append to the first set of files. Set operations, sent to the cache during the rewrite, append to the second set. A backup is temporarily stored during rewrites if there's a failure. The backup is promptly deleted after a rewrite finishes. If soft delete is turned on for your storage account, the soft delete setting applies and existing backups continue to stay in the soft delete state.
Will having firewall exceptions on the storage account affect persistence?
Yes. Using firewall settings on the storage account can prevent the persistence feature from working. You can see if there are errors in persisting data by viewing the Errors metric. This metric will indicate if the cache is unable to persist data due to firewall restrictions on the storage account or other problems.
In order to use data persistence with a storage account that has a firewall set up, use managed identity based authentication to connect to storage. Using managed identity adds the cache instance to the trusted services list, making firewall exceptions easier to carry out. If you aren't using managed identity and instead authorizing to a storage account using a key, then having firewall exceptions on the storage account tends to break the persistence process. This only applies to persistence in the Premium tier.
Can I have AOF persistence enabled if I have more than one replica?
With the Premium tier, you can't use Append-only File (AOF) persistence with multiple replicas.
How do I check if soft delete is enabled on my storage account?
Select the storage account that your cache is using for persistence. Select Data Protection from the Resource menu. In the working pane, check the state of Enable soft delete for blobs. For more information on soft delete in Azure storage accounts, see Enable soft delete for blobs.
Next steps
Learn more about Azure Cache for Redis features.