Capacity planning and scaling for Azure Service Fabric

Before you create any Azure Service Fabric cluster or scale compute resources that host your cluster, it's important to plan for capacity. For more information about planning for capacity, see Planning the Service Fabric cluster capacity.

In addition to considering node type and cluster characteristics, you should expect scaling operations to take longer than an hour to complete for a production environment. This consideration is true regardless of the number of VMs you're adding.

Autoscaling

You should perform scaling operations via Azure Resource Manager templates, because it's the best practice to treat resource configurations as code.

Using automatic scaling through virtual machine scale sets will make your versioned Resource Manager template inaccurately define your instance counts for virtual machine scale sets. Inaccurate definition increases the risk that future deployments will cause unintended scaling operations. In general, you should use autoscaling if:

• Deploying your Resource Manager templates with appropriate capacity declared doesn't support your use case.

  In addition to manual scaling, you can configure a Continuous integration and delivery pipeline in Azure DevOps Services by using Azure resource group deployment projects. This pipeline is commonly triggered by a logic app that uses virtual machine performance metrics queried from the Azure Monitor REST API. The pipeline effectively autoscales based on whatever metrics you want, while optimizing for Resource Manager templates.

• You need to horizontally scale only one virtual machine scale set node at a time.

  To scale out by three or more nodes at a time, you should scale out a Service Fabric cluster by adding a virtual machine scale set. It's safest to scale in and scale out virtual machine scale sets horizontally, one node at a time.

• You have Silver reliability or higher for your Service Fabric cluster, and Silver durability or higher on any scale where you configure autoscaling rules.

  The minimum capacity for autoscaling rules must be equal to or greater than five virtual machine instances. It must also be equal to or greater than your Reliability Tier minimum for your primary node type.

Vertical scaling considerations

Vertical scaling a node type in Azure Service Fabric requires a number of steps and considerations. For example:

• The cluster must be healthy before scaling. Otherwise, you'll destabilize the cluster further.
• Silver durability level or greater is required for all Service Fabric cluster node types that host stateful services.

Vertical scaling a virtual machine scale set by simply changing its resource SKU is a destructive operation, because it re-images your hosts thus removing all locally persisted state. Instead, you'll want to horizontally scale your cluster by adding a new scale set with the desired SKU and then migrate your services to the new scale set to complete a safe vertical scaling operation.

Your cluster uses Service Fabric node properties and placement constraints to decide where to host your application's services. When vertically scaling a primary node type, you'll deploy a second primary node type and then set ("isPrimary": false) on the original primary node type and proceed to disable its nodes and remove its scale set and its related resources. For details, see Scale up a Service Fabric cluster primary node type.

With the node properties and placement constraints declared, do the following steps one VM instance at a time. This allows the system services (and your stateful services) to be shut down gracefully on the VM instance you're removing as new replicas are created elsewhere.

1. From PowerShell, run Disable-ServiceFabricNode with intent RemoveNode to disable the node you're going to remove. Remove the node type that has the highest number. For example, if you have a six-node cluster, remove the "MyNodeType_5" virtual machine instance.

2. Run Get-ServiceFabricNode to make sure that the node has transitioned to disabled. If not, wait until the node is disabled. This might take a couple hours for each node. Don't proceed until the node has transitioned to disabled.

3. Decrease the number of VMs by one in that node type. The highest VM instance will now be removed.

4. Repeat steps 1 through 3 as needed, but never scale in the number of instances in the primary node types less than what the reliability tier warrants. See Planning the Service Fabric cluster capacity for a list of recommended instances.

5. Once all VMs are gone (represented as "Down") the fabric:/System/InfrastructureService/[node name] will show an Error state. Then, you can update the cluster resource to remove the node type. You can either use the ARM template deployment. This will start a cluster upgrade which will remove the fabric:/System/InfrastructureService/[node type] service that is in error state.

6. After that you can optionally delete the VMScaleSet, you will still see the nodes as "Down" from Service Fabric Explorer view though. The last step would be to clean them up with Remove-ServiceFabricNodeState command.

Horizontal scaling

You can do horizontal scaling either manually or programmatically.

Scaling out

Scale out a Service Fabric cluster by increasing the instance count for a particular virtual machine scale set. You can scale out programmatically by using AzureClient and the ID for the desired scale set to increase the capacity.

var scaleSet = AzureClient.VirtualMachineScaleSets.GetById(ScaleSetId);
var newCapacity = (int)Math.Min(MaximumNodeCount, scaleSet.Capacity + 1);
scaleSet.Update().WithCapacity(newCapacity).Apply(); 

To scale out manually, update the capacity in the SKU property of the desired virtual machine scale set resource.

"sku": {
    "name": "[parameters('vmNodeType0Size')]",
    "capacity": "[parameters('nt0InstanceCount')]",
    "tier": "Standard"
}

Scaling in

Scaling in requires more consideration than scaling out. For example:

• Service Fabric system services run in the primary node type in your cluster. Never shut down or scale in the number of instances for that node type so that you have fewer instances than what the reliability tier warrants.
• For a stateful service, you need a certain number of nodes that are always up to maintain availability and preserve the state of your service. At a minimum, you need a number of nodes equal to the target replica set count of the partition or service.

To scale in manually, follow these steps:

1. From PowerShell, run Disable-ServiceFabricNode with intent RemoveNode to disable the node you're going to remove. Remove the node type that has the highest number. For example, if you have a six-node cluster, remove the "MyNodeType_5" virtual machine instance.
2. Run Get-ServiceFabricNode to make sure that the node has transitioned to disabled. If not, wait until the node is disabled. This might take a couple hours for each node. Don't proceed until the node has transitioned to disabled.
3. Decrease the number of VMs by one in that node type. The highest VM instance will now be removed.
4. Repeat steps 1 through 3 as needed until you provision the capacity you want. Don't scale in the number of instances in the primary node types to less than what the reliability tier warrants. See Planning the Service Fabric cluster capacity for a list of recommended instances.

To scale in manually, update the capacity in the SKU property of the desired virtual machine scale set resource.

"sku": {
    "name": "[parameters('vmNodeType0Size')]",
    "capacity": "[parameters('nt0InstanceCount')]",
    "tier": "Standard"
}

You must prepare the node for shutdown to scale in programmatically. Find the node to be removed (the highest-instance node). For example:

using (var client = new FabricClient())
{
    var mostRecentLiveNode = (await client.QueryManager.GetNodeListAsync())
        .Where(n => n.NodeType.Equals(NodeTypeToScale, StringComparison.OrdinalIgnoreCase))
        .Where(n => n.NodeStatus == System.Fabric.Query.NodeStatus.Up)
        .OrderByDescending(n =>
        {
            var instanceIdIndex = n.NodeName.LastIndexOf("_");
            var instanceIdString = n.NodeName.Substring(instanceIdIndex + 1);
            return int.Parse(instanceIdString);
        })
        .FirstOrDefault();

Deactivate and remove the node by using the same FabricClient instance (client in this case) and node instance (instanceIdString in this case) that you used in the previous code:

var scaleSet = AzureClient.VirtualMachineScaleSets.GetById(ScaleSetId);

// Remove the node from the Service Fabric cluster
ServiceEventSource.Current.ServiceMessage(Context, $"Disabling node {mostRecentLiveNode.NodeName}");
await client.ClusterManager.DeactivateNodeAsync(mostRecentLiveNode.NodeName, NodeDeactivationIntent.RemoveNode);

// Wait (up to a timeout) for the node to gracefully shut down
var timeout = TimeSpan.FromMinutes(5);
var waitStart = DateTime.Now;
while ((mostRecentLiveNode.NodeStatus == System.Fabric.Query.NodeStatus.Up || mostRecentLiveNode.NodeStatus == System.Fabric.Query.NodeStatus.Disabling) &&
        DateTime.Now - waitStart < timeout)
{
    mostRecentLiveNode = (await client.QueryManager.GetNodeListAsync()).FirstOrDefault(n => n.NodeName == mostRecentLiveNode.NodeName);
    await Task.Delay(10 * 1000);
}

// Decrement virtual machine scale set capacity
var newCapacity = (int)Math.Max(MinimumNodeCount, scaleSet.Capacity - 1); // Check min count 

scaleSet.Update().WithCapacity(newCapacity).Apply();

Reliability levels

The reliability level is a property of your Service Fabric cluster resource. It can't be configured differently for individual node types. It controls the replication factor of the system services for the cluster, and is a setting at the cluster resource level.

The reliability level will determine the minimum number of nodes that your primary node type must have. The reliability tier can take the following values:

• Platinum: Runs the system services with a target replica set count of seven and nine seed nodes.
• Gold: Runs the system services with a target replica set count of seven and seven seed nodes.
• Silver: Runs the system services with a target replica set count of five and five seed nodes.
• Bronze: Runs the system services with a target replica set count of three and three seed nodes.

The minimum recommended reliability level is Silver.

The reliability level is set in the properties section of the Microsoft.ServiceFabric/clusters resource, like this:

"properties":{
    "reliabilityLevel": "Silver"
}

Durability levels

The durability level must be set in two resources. One is the extension profile of the virtual machine scale set resource:

"extensionProfile": {
    "extensions":          {
        "name": "[concat('ServiceFabricNodeVmExt','_vmNodeType0Name')]",
        "properties": {
            "settings": {
                "durabilityLevel": "Bronze"
            }
        }
    }
}

The other resource is under nodeTypes in the Microsoft.ServiceFabric/clusters resource:

"nodeTypes": [
    {
        "name": "[variables('vmNodeType0Name')]",
        "durabilityLevel": "Bronze"
    }
]

Next steps

• Create a cluster on VMs or computers running Windows Server: Service Fabric cluster creation for Windows Server.
• Create a cluster on VMs or computers running Linux: Create a Linux cluster.
• Learn about Service Fabric support options.