Migrate Batch pool configuration from Cloud Services to Virtual Machine

  • Article

Currently, Batch pools can be created using either virtualMachineConfiguration or cloudServiceConfiguration. We recommend using Virtual Machine Configuration only, as this configuration supports all Batch capabilities.

Cloud Services Configuration pools don't support some of the current Batch features, and won't support any newly added features. You won't be able to create new 'CloudServiceConfiguration' pools or add new nodes to existing pools after February 29, 2024.

If your Batch solutions currently use 'cloudServiceConfiguration' pools, we recommend changing to 'virtualMachineConfiguration' as soon as possible. This will enable you to benefit from all Batch capabilities, such as an expanded selection of VM series, Linux VMs, containers, Azure Resource Manager virtual networks, and node disk encryption.

[!IMPORANT] Azure Batch account certificates are deprecated and will be retired after the same February 29, 2024 date as cloudServiceConfiguration pools. If you are using Batch account certificates, migrate your Batch account certificates to Azure Key Vault at the same time as migrating your pool configuration.

Create a pool using Virtual Machine Configuration

You can't switch an existing active pool that uses 'cloudServiceConfiguration' to use 'virtualMachineConfiguration'. Instead, you'll need to create new pools. Once you've created your new 'virtualMachineConfiguration' pools and replicated all of your jobs and tasks, you can delete the old 'cloudServiceConfiguration'pools that you're no longer using.

All Batch APIs, command-line tools, the Azure portal, and the Batch Explorer UI let you create pools using 'virtualMachineConfiguration'.

For a walkthrough of the process of creating pools that use 'virtualMachineConfiguration, see the .NET tutorial or the Python tutorial.

Pool configuration differences

Some of the key differences between the two configurations include:

  • 'cloudServiceConfiguration' pool nodes only use Windows OS. 'virtualMachineConfiguration' pools can use either Linux or Windows OS.

  • Compared to 'cloudServiceConfiguration' pools, 'virtualMachineConfiguration' pools have a richer set of capabilities, such as container support, data disks, and disk encryption.

  • Pool and node startup and delete times may differ slightly between 'cloudServiceConfiguration' pools and 'virtualMachineConfiguration' pools.

  • 'virtualMachineConfiguration' pool nodes utilize managed OS disks. The managed disk type that is used for each node depends on the VM size chosen for the pool. If a 's' VM size is specified for the pool, for example 'Standard_D2s_v3', then a premium SSD is used. If a 'non-s' VM size is specified, for example 'Standard_D2_v3', then a standard HDD is used.

    Important

    As with Virtual Machines and Virtual Machine Scale Sets, the OS managed disk used for each node incurs a cost, which is additional to the cost of the VMs. 'virtualMachineConfiguration' pools can use ephemeral OS disks, which create the OS disk on the VM cache or temporary disk, to avoid extra costs associated with managed disks.There is no OS disk cost for 'cloudServiceConfiguration' nodes, as the OS disk is created on the node's local disk.

Azure Data Factory custom activity pools

Azure Batch pools can be used to run Data Factory custom activities. Any 'cloudServiceConfiguration' pools used to run custom activities will need to be deleted and new 'virtualMachineConfiguration' pools created.

When creating your new pools to run Data Factory custom activities, follow these practices:

  • Pause all pipelines before creating the new pools and deleting the old ones to ensure no executions will be interrupted.
  • The same pool ID can be used to avoid linked service configuration changes.
  • Resume pipelines when new pools have been created.

For more information about using Azure Batch to run Data Factory custom activities, see Azure Batch linked service and Custom activities in a Data Factory pipeline

Next steps