Change feed modes in Azure Cosmos DB

Latest version mode provides an easy way to process both real-time and historic changes to items in a container with the ability to go back to changes from the beginning of the container.

The following are scenarios well-suited to this mode:

• Migrations of an entire container to a secondary location.

• Ability to reprocess changes from the beginning of the container.

• Real-time processing of changes to items in a container resulting from create and update operations.

• Workloads that don't need to capture deletes or intermediate changes between reads.

The all versions and deletes change feed mode enables new scenarios for change feed, and simplifies others. You can read every change that occurred to items (even in cases in which multiple changes occurred between change feed reads), identify the operation type of changes being processed, and get changes that result from deletes.

A few common scenarios this mode enables and enhances are:

• Real-time transfer of data between two locations without implementing a soft delete.

• Triggering logic based on incremental changes if multiple change operations for a given item are expected between change feed polls.

• Triggering alerts on delete operations, like in auditing scenarios.

• The ability to process item creates, updates, and deletes differently based on operation type.

• The change feed includes insert and update operations that are made to items in the container.

• This mode of change feed doesn't log deletes. You can capture deletes by setting a "soft-delete" flag within your items instead of deleting them directly. For example, you can add an attribute in the item called deleted with the value true, and then set a Time to Live (TTL) on the item. The change feed captures it as an update and the item is automatically deleted when the TTL expires. Alternatively, you can set a finite expiration period for your items by using the TTL capability. With this solution, you have to process the changes within a shorter time interval than the TTL expiration period.

• Only the most recent change for a specific item is included in the change feed. Intermediate changes might not be available.

• When an item is deleted, it's no longer available in the change feed.

• Changes can be synchronized from any point in time, and there's no fixed data retention period for which changes are available.

• You can't filter the change feed for a specific type of operation. One possible alternative, is to add a "soft marker" on the item for updates and filter based on the marker when you process items in the change feed.

• The starting point to read change feed can be from the beginning of the container, from a point in time, from "now," or from a specific checkpoint. The precision of the start time is approximately five seconds.

• The change feed includes insert, update, and delete operations made to items within the container. Deletes from TTL expirations are also captured.

• Metadata is provided to determine the change type, including whether a delete was due to a TTL expiration.

• Change feed items come in the order of their modification time. Deletes from TTL expirations aren't guaranteed to appear in the feed immediately after the item expires. They appear when the item is purged from the container.

• All changes that occurred within the retention window for continuous backups on the account can be read. Attempting to read changes that occurred outside of the retention window results in an error. For example, if your container was created eight days ago and your continuous backup period retention period is seven days, then you can only read changes from the last seven days.

• The change feed starting point can be from "now" or from a specific checkpoint within your retention period. You can't read changes from the beginning of the container or from a specific point in time by using this mode.

You can use the following ways to consume changes from change feed in latest version mode:

Parse the response object

In latest version mode, the default response object is an array of items that changed. Each item contains the standard metadata for any Azure Cosmos DB item, including _etag and _ts, with the addition of a new property, _lsn.

The _etag format is internal and you shouldn't take dependency on it because it can change anytime. _ts is a modification or a creation time stamp. You can use _ts for chronological comparison. _lsn is a batch ID that is added for change feed only that represents the transaction ID. Many items can have same _lsn.

ETag on FeedResponse is different from the _etag you see on the item. _etag is an internal identifier, and it's used for concurrency control. The _etag property represents the version of the item, whereas the ETag property is used to sequence the feed.

During the preview, the following methods to read the change feed are available for each client SDK:

Get started

To get started using all versions and deletes change feed mode, enroll in the preview via the Preview Features page in your Azure Subscription overview page. Search for the AllVersionsAndDeletesChangeFeed feature and select Register.

Before you submit your request, ensure that you have at least one Azure Cosmos DB account in the subscription. This account can be an existing account or a new account that you created to try out the preview feature. If you have no accounts in the subscription when your request is received, the request is declined because there are no accounts to apply the feature to.

The Azure Cosmos DB team reviews your request and contacts you via email to confirm which Azure Cosmos DB accounts in the subscription you want to enroll in the preview. To use the preview, you must have continuous backups configured for your Azure Cosmos DB account. Continuous backups can be enabled either before or after being admitted to the preview, but continuous backups must be enabled before you attempt to read from the change feed in all versions and deletes mode.

Parse the response object

The response object is an array of items that represent each change. Different properties will be populated depending on the change type. There's currently no way to get the previous version of items for either replace or delete operations.

• Create operations

  {
    "current": {
      <The current version of the item that changed. All the properties of your item will appear here.>
    },
    "metadata": {
      "operationType": "create",
      "lsn": <A number that represents the batch ID. Many items can have the same lsn.>,
      "crts": <A number that represents the Conflict Resolved Timestamp. It has the same format as _ts.>
    }
  }
  

• Replace operations

  {
    "current": {
      <The current version of the item that changed. All the properties of your item will appear here.>
    },
    "metadata": {
      "operationType": "replace",
      "lsn": <A number that represents the batch ID. Many items can have the same lsn.>,
      "crts": <A number that represents the Conflict Resolved Timestamp. It has the same format as _ts.>,
      "previousImageLSN" : <A number that represents the batch ID of the change prior to this one.>,
    }
  }
  

• Delete operations

  {
    "metadata": {
      "operationType": "delete",
      "lsn": <A number that represents the batch ID. Many items can have the same lsn.>,
      "crts": <A number that represents the Conflict Resolved Timestamp. It has the same format as _ts.>,
      "previousImageLSN" : <A number that represents the batch ID of the change prior to this one.>,
      "id": "<Id of the deleted item.>",
      "partitionKey": {
        "<Partition key property name>": "<Partition key property value>"
      }
    }
  }
  

Limitations

• Supported for Azure Cosmos DB for NoSQL accounts. Other Azure Cosmos DB account types aren't supported.

• Continuous backups are required to use this change feed mode. The limitations of using continuous backup can be found in the documentation.

• Reading changes on a container that existed before continuous backups were enabled on the account isn't supported.

• The ability to start reading the change feed from the beginning or to select a start time based on a past time stamp isn't currently supported. You can either start from "now" or from a previous lease or continuation token.

• Receiving the previous version of items that were deleted or updated isn't currently available.

• Accounts that enabled merging partitions aren't supported.