缩小或扩大群集Scale a cluster in or out

警告

在缩放之前阅读此部分Read this section before you scale

要缩放计算资源来满足你的应用程序工作负载,需要实施有目的性的计划,此操作在生产环境中总是需要几乎一个多小时才能完成,而且你还要对你的工作负载和业务环境了如指掌。事实上,如果从未执行过此操作,建议在继续本文档其余部分之前,首先阅读并理解 Service Fabric 群集容量规划注意事项Scaling compute resources to source your application work load requires intentional planning, will nearly always take longer than an hour to complete for a production environment, and does require you to understand your workload and business context; in fact if you have never done this activity before, it's recommended you start by reading and understanding Service Fabric cluster capacity planning considerations, before continuing the remainder of this document. 该项建议的目的是避免出现意外的实时站点问题;此外,建议成功测试你决定针对非生产环境执行的操作。This recommendation is to avoid unintended LiveSite issues, and it's also recommended you successfully test the operations you decide to perform against a non-production environment. 可随时报告生产问题或请求 Azure 付费支持At any time you can report production issues or request paid support for Azure. 对于被分配根据适当情况执行这些操作的工程师,请参考本文了解缩放操作。但是,你必须决定并了解哪些操作适合你的情况,例如要缩放哪些资源(CPU、存储、内存)、按哪个方向进行缩放方向(垂直或水平),以及要执行哪些操作(资源模板部署、门户、PowerShell/CLI)。For engineers allocated to perform these operations that possess appropriate context, this article will describe scaling operations, but you must decide and understand which operations are appropriate for your use case; such as what resources to scale (CPU, Storage, Memory), what direction to scale (Vertically or Horizontally), and what operations to perform (Resource Template deployment, Portal, PowerShell/CLI).

备注

本文进行了更新,以便使用新的 Azure PowerShell Az 模块。This article has been updated to use the new Azure PowerShell Az module. 你仍然可以使用 AzureRM 模块,至少在 2020 年 12 月之前,它将继续接收 bug 修补程序。You can still use the AzureRM module, which will continue to receive bug fixes until at least December 2020. 若要详细了解新的 Az 模块和 AzureRM 兼容性,请参阅新 Azure Powershell Az 模块简介To learn more about the new Az module and AzureRM compatibility, see Introducing the new Azure PowerShell Az module. 有关 Az 模块安装说明,请参阅安装 Azure PowerShellFor Az module installation instructions, see Install Azure PowerShell.

使用自动缩放规则或者手动来扩展和缩减 Service Fabric 群集Scale a Service Fabric cluster in or out using auto-scale rules or manually

虚拟机规模集是一种 Azure 计算资源,可用于将一组虚拟机作为一个集进行部署和管理。Virtual machine scale sets are an Azure compute resource that you can use to deploy and manage a collection of virtual machines as a set. 在 Service Fabric 群集中定义的每个节点类型将设置为不同的虚拟机规模集。Every node type that is defined in a Service Fabric cluster is set up as a separate virtual machine scale set. 然后,每个节点类型可以独立缩减或扩展、打开不同的端口集,并可以有不同的容量指标。Each node type can then be scaled in or out independently, have different sets of ports open, and can have different capacity metrics. 可在 Service Fabric 节点类型文档中了解有关详细信息。Read more about it in the Service Fabric node types document. 由于群集中的 Service Fabric 节点类型由后端的虚拟机规模集构成,因此需要为每个节点类型/虚拟机规模集设置自动缩放规则。Since the Service Fabric node types in your cluster are made of virtual machine scale sets at the backend, you need to set up auto-scale rules for each node type/virtual machine scale set.

备注

订阅必须有足够的核心用于添加构成此群集的新 VM。Your subscription must have enough cores to add the new VMs that make up this cluster. 当前没有模型验证,因此如果达到任何配额限制,会遇到部署时故障。There is no model validation currently, so you get a deployment time failure, if any of the quota limits are hit. 此外,对于单个节点类型,每个 VMSS 不能超过 100 个节点。Also a single node type can not simply exceed 100 nodes per VMSS. 要实现目标缩放,可能需要添加 VMSS,且自动缩放不会自动添加 VMSS。You may need to add VMSS's to achieve the targeted scale, and auto-scaling can not automagically add VMSS's. 将 VMSS 就地添加到实时集群非常具有挑战性务,这通常会导致用户使用创建时预配的相应节点类型来预配新集群。请相应地计划群集容量Adding VMSS's in-place to a live cluster is a challenging task, and commonly this results in users provisioning new clusters with the appropriate node types provisioned at creation time; plan cluster capacity accordingly.

选择要缩放的节点类型/虚拟机规模集Choose the node type/Virtual Machine scale set to scale

当前无法使用门户为虚拟机规模集指定自动缩放规则,以创建 Service Fabric 群集,因此我们来使用 Azure PowerShell (1.0+) 列出节点类型,并向它们添加自动缩放规则。Currently, you are not able to specify the auto-scale rules for virtual machine scale sets using the portal to create a Service Fabric Cluster, so let us use Azure PowerShell (1.0+) to list the node types and then add auto-scale rules to them.

若要获取构成群集的虚拟机规模集的列表,请运行以下 cmdlet:To get the list of virtual machine scale set that make up your cluster, run the following cmdlets:

Get-AzResource -ResourceGroupName <RGname> -ResourceType Microsoft.Compute/VirtualMachineScaleSets

Get-AzVmss -ResourceGroupName <RGname> -VMScaleSetName <virtual machine scale set name>

为节点类型/虚拟机规模集设置自动缩放规则Set auto-scale rules for the node type/virtual machine scale set

如果群集具有多个节点类型,请为要缩放(扩展或缩减)的每个节点类型/虚拟机规模集重复此操作。If your cluster has multiple node types, then repeat this for each node types/virtual machine scale sets that you want to scale (in or out). 在设置自动缩放之前请考虑必须具有的节点数。Take into account the number of nodes that you must have before you set up auto-scaling. 对于主节点类型所必须具有的最小节点数受所选择的可靠性级别影响。The minimum number of nodes that you must have for the primary node type is driven by the reliability level you have chosen. 了解有关 可靠性级别的详细信息。Read more about reliability levels.

备注

将主节点类型减少到小于最小数量会使群集不稳定,甚至使其关闭。Scaling in the primary node type to less than the minimum number will make the cluster unstable or even bring it down. 这可能会对应用程序和系统服务造成数据丢失。This could result in data loss for your applications and for the system services.

当前,自动缩放功能不受应用程序可能向 Service Fabric 报告的负荷所影响。Currently the auto-scale feature is not driven by the loads that your applications may be reporting to Service Fabric. 因此当前获得的自动缩放纯粹受每个虚拟机规模集实例发出的性能计数器所影响。So at this time the auto-scale you get is purely driven by the performance counters that are emitted by each of the virtual machine scale set instances.

按照以下说明为每个虚拟机规模集设置自动缩放Follow these instructions to set up auto-scale for each virtual machine scale set.

备注

在横向缩减方案中,除非节点类型具有金级或银级持续性级别,否则需要使用相应节点名称来调用 Remove-ServiceFabricNodeState cmdletIn a scale in scenario, unless your node type has a durability level of Gold or Silver you need to call the Remove-ServiceFabricNodeState cmdlet with the appropriate node name. 对于铜级持续性,不建议一次横向缩减多个节点。For the Bronze durability, it's not recommended to scale in more than one node at a time.

手动将 VM 添加到节点类型/虚拟机规模集Manually add VMs to a node type/virtual machine scale set

横向扩展时,会添加更多虚拟机实例到规模集。When you scale out, you add more virtual machine instances to the scale set. 这些实例成为 Service Fabric 使用的节点。These instances become the nodes that Service Fabric uses. Service Fabric 知道规模集什么时候添加了更多实例(通过扩大实现)并自动做出反应。Service Fabric knows when the scale set has more instances added (by scaling out) and reacts automatically.

备注

添加 VM 需要时间,请不要期待马上就有结果。Adding VMs takes time, so do not expect the additions to be instantaneous. 请提前规划增加容量的时间,在副本/服务实例可以使用 VM 容量之前留出超过 10 分钟的时间让一切就绪。Plan to add capacity well in advance, allowing for over 10 minutes before the VM capacity is available for the replicas/service instances to get placed.

使用模板添加 VMAdd VMs using a template

根据快速入门模板库中的示例/说明更改每个节点类型的 VM 数目。Follow the sample/instructions in the quickstart template gallery to change the number of VMs in each node type.

使用 PowerShell 或 CLI 命令添加 VMAdd VMs using PowerShell or CLI commands

以下代码按名称获取规模集,并使规模集的容量增加 1。The following code gets a scale set by name and increases the capacity of the scale set by 1.

$scaleset = Get-AzVmss -ResourceGroupName SFCLUSTERTUTORIALGROUP -VMScaleSetName nt1vm
$scaleset.Sku.Capacity += 1

Update-AzVmss -ResourceGroupName $scaleset.ResourceGroupName -VMScaleSetName $scaleset.Name -VirtualMachineScaleSet $scaleset

此代码将容量设为 6。This code sets the capacity to 6.

# Get the name of the node with
az vmss list-instances -n nt1vm -g sfclustertutorialgroup --query [*].name

# Use the name to scale
az vmss scale -g sfclustertutorialgroup -n nt1vm --new-capacity 6

手动从节点类型/虚拟机规模集中删除 VMManually remove VMs from a node type/virtual machine scale set

按节点类型进行缩放时,将从规模集中删除 VM 实例。When you scale in a node type, you remove VM instances from the scale set. 如果节点类型为“铜”持续性级别,Service Fabric 将不知道发生了什么情况,因此会报告节点已丢失。If the node type is Bronze durability level, Service Fabric is unaware what has happened and reports that a node has gone missing. 然后 Service Fabric 将报告群集的不正常状态。Service Fabric then reports an unhealthy state for the cluster. 若要防止状态错误,必须从群集中显式删除节点并删除节点状态。To prevent that bad state, you must explicitly remove the node from the cluster and remove the node state.

Service Fabric 系统服务在群集中以主节点类型运行。The service fabric system services run in the primary node type in your cluster. 横向缩减主节点类型时,切勿在横向缩减实例数时使其少于可靠性层所需的数目。When scaling in the primary node type, never scale in the number of instances to less than what the reliability tier warrants.

对于有状态服务,需要一些始终启动的节点来保持可用性,以及保持服务的状态。For a stateful service, you need a certain number of nodes to be always up to maintain availability and preserve state of your service. 至少需要与分区/服务的目标副本集计数相等的节点数目。At the very minimum, you need the number of nodes equal to the target replica set count of the partition/service.

删除 Service Fabric 节点Remove the Service Fabric node

手动删除节点状态的步骤仅适用于具有“铜”持续性层的节点类型。The steps for manually removing node state apply only to node types with a Bronze durability tier. 对于“白银”和“黄金”持续性层,平台会自动完成上述步骤。 For Silver and Gold durability tier, these steps are done automatically by the platform. 有关持续性的详细信息,请参阅 Service Fabric 群集容量规划For more information about durability, see Service Fabric cluster capacity planning.

为了使群集节点均匀地分布在升级域和容错域中,从而使它们的利用均匀,应该首先删除最近创建的节点。To keep the nodes of the cluster evenly distributed across upgrade and fault domains, and hence enable their even utilization, the most recently created node should be removed first. 换句话说,应该按照创建节点的相反顺序删除节点。In other words, the nodes should be removed in the reverse order of their creation. 最近创建的节点是具有最大 virtual machine scale set InstanceId 属性值的节点。The most recently created node is the one with the greatest virtual machine scale set InstanceId property value. 下面的代码示例返回最近创建的节点。The code examples below return the most recently created node.

Get-ServiceFabricNode | Sort-Object NodeInstanceId -Descending | Select-Object -First 1
sfctl node list --query "sort_by(items[*], &name)[-1]"

Service Fabric 群集需要了解此节点将被删除。The Service Fabric cluster needs to know that this node is going to be removed. 需要执行以下三个步骤:There are three steps you need to take:

  1. 禁用节点,使其不再是数据复制。Disable the node so that it no longer is a replicate for data.
    PowerShell:Disable-ServiceFabricNodePowerShell: Disable-ServiceFabricNode
    sfctl:sfctl node disablesfctl: sfctl node disable

  2. 停止节点,使 Service Fabric 运行时完全关闭且应用获取终止请求。Stop the node so that the Service Fabric runtime shuts down cleanly, and your app gets a terminate request.
    PowerShell:Start-ServiceFabricNodeTransition -StopPowerShell: Start-ServiceFabricNodeTransition -Stop
    sfctl:sfctl node transition --node-transition-type Stopsfctl: sfctl node transition --node-transition-type Stop

  3. 从群集移除节点。Remove the node from the cluster.
    PowerShell:Remove-ServiceFabricNodeStatePowerShell: Remove-ServiceFabricNodeState
    sfctl:sfctl node remove-statesfctl: sfctl node remove-state

对节点执行这三个步骤后,即可将其从规模集中移除。Once these three steps have been applied to the node, it can be removed from the scale set. 如果使用除以外的任意持续性层,在移除规模集实例时会完成这些步骤。If you're using any durability tier besides bronze, these steps are done for you when the scale set instance is removed.

以下代码块获取最近创建的节点,禁用、停止并将节点从群集中移除。The following code block gets the last created node, disables, stops, and removes the node from the cluster.

#### After you've connected.....
# Get the node that was created last
$node = Get-ServiceFabricNode | Sort-Object { $_.NodeName.Substring($_.NodeName.LastIndexOf('_') + 1) } -Descending | Select-Object -First 1

# Node details for the disable/stop process
$nodename = $node.NodeName
$nodeid = $node.NodeInstanceId

$loopTimeout = 10

# Run disable logic
Disable-ServiceFabricNode -NodeName $nodename -Intent RemoveNode -TimeoutSec 300 -Force

$state = Get-ServiceFabricNode | Where-Object NodeName -eq $nodename | Select-Object -ExpandProperty NodeStatus

while (($state -ne [System.Fabric.Query.NodeStatus]::Disabled) -and ($loopTimeout -ne 0))
{
    Start-Sleep 5
    $loopTimeout -= 1
    $state = Get-ServiceFabricNode | Where-Object NodeName -eq $nodename | Select-Object -ExpandProperty NodeStatus
    Write-Host "Checking state... $state found"
}

# Exit if the node was unable to be disabled
if ($state -ne [System.Fabric.Query.NodeStatus]::Disabled)
{
    Write-Error "Disable failed with state $state"
}
else
{
    # Stop node
    $stopid = New-Guid
    Start-ServiceFabricNodeTransition -Stop -OperationId $stopid -NodeName $nodename -NodeInstanceId $nodeid -StopDurationInSeconds 300

    $state = (Get-ServiceFabricNodeTransitionProgress -OperationId $stopid).State
    $loopTimeout = 10

    # Watch the transaction
    while (($state -eq [System.Fabric.TestCommandProgressState]::Running) -and ($loopTimeout -ne 0))
    {
        Start-Sleep 5
        $state = (Get-ServiceFabricNodeTransitionProgress -OperationId $stopid).State
        Write-Host "Checking state... $state found"
    }

    if ($state -ne [System.Fabric.TestCommandProgressState]::Completed)
    {
        Write-Error "Stop transaction failed with $state"
    }
    else
    {
        # Remove the node from the cluster
        Remove-ServiceFabricNodeState -NodeName $nodename -TimeoutSec 300 -Force
    }
}

在下面的“sfctl”代码中,使用以下命令获取最近创建的节点 sfctl node list --query "sort_by(items[*], &name)[-1].name" 的 “node-name”值: In the sfctl code below, the following command is used to get the node-name value of the last-created node: sfctl node list --query "sort_by(items[*], &name)[-1].name"

# Inform the node that it is going to be removed
sfctl node disable --node-name _nt1vm_5 --deactivation-intent 4 -t 300

# Stop the node using a random guid as our operation id
sfctl node transition --node-instance-id 131541348482680775 --node-name _nt1vm_5 --node-transition-type Stop --operation-id c17bb4c5-9f6c-4eef-950f-3d03e1fef6fc --stop-duration-in-seconds 14400 -t 300

# Remove the node from the cluster
sfctl node remove-state --node-name _nt1vm_5

提示

使用以下“sfctl”查询检查每个步骤的状态。Use the following sfctl queries to check the status of each step

检查停用状态 sfctl node list --query "sort_by(items[*], &name)[-1].nodeDeactivationInfo"Check deactivation status sfctl node list --query "sort_by(items[*], &name)[-1].nodeDeactivationInfo"

检查停止状态 sfctl node list --query "sort_by(items[*], &name)[-1].isStopped"Check stop status sfctl node list --query "sort_by(items[*], &name)[-1].isStopped"

横向缩减规模集Scale in the scale set

将 Service Fabric 节点从群集中删除后,即可缩小虚拟机规模集。Now that the Service Fabric node has been removed from the cluster, the virtual machine scale set can be scaled in. 在下面的示例中,规模集容量缩小了 1。In the example below, the scale set capacity is reduced by 1.

$scaleset = Get-AzVmss -ResourceGroupName SFCLUSTERTUTORIALGROUP -VMScaleSetName nt1vm
$scaleset.Sku.Capacity -= 1

Update-AzVmss -ResourceGroupName SFCLUSTERTUTORIALGROUP -VMScaleSetName nt1vm -VirtualMachineScaleSet $scaleset

此代码将容量设为 5。This code sets the capacity to 5.

# Get the name of the node with
az vmss list-instances -n nt1vm -g sfclustertutorialgroup --query [*].name

# Use the name to scale
az vmss scale -g sfclustertutorialgroup -n nt1vm --new-capacity 5

可能会在 Service Fabric Explorer 中观察到的行为Behaviors you may observe in Service Fabric Explorer

横向扩展群集时,Service Fabric Explorer 会反映属于群集一部分的节点(虚拟机规模集实例)数。When you scale out a cluster the Service Fabric Explorer will reflect the number of nodes (virtual machine scale set instances) that are part of the cluster. 但是在缩减群集时,会看到已删除的节点/VM 实例显示为不正常状态,除非使用相应节点名称来调用 Remove-ServiceFabricNodeState cmdHowever, when you scale a cluster in you will see the removed node/VM instance displayed in an unhealthy state unless you call Remove-ServiceFabricNodeState cmd with the appropriate node name.

下面是此行为的说明。Here is the explanation for this behavior.

Service Fabric Explorer 中列出的节点是 Service Fabric 系统服务(特别是 FM)在群集具有的节点数方面所了解的信息的反映。The nodes listed in Service Fabric Explorer are a reflection of what the Service Fabric system services (FM specifically) knows about the number of nodes the cluster had/has. 横向缩减虚拟机规模集时,VM 已删除,但是 FM 系统服务仍然认为节点(映射到已删除的 VM)会恢复。When you scale the virtual machine scale set in, the VM was deleted but FM system service still thinks that the node (that was mapped to the VM that was deleted) will come back. 因此 Service Fabric Explorer 会继续显示该节点(尽管运行状况状态可能是错误或未知)。So Service Fabric Explorer continues to display that node (though the health state may be error or unknown).

若要确保在删除 VM 时删除节点,有两个选项:In order to make sure that a node is removed when a VM is removed, you have two options:

  1. 为群集中的节点类型选择金级或银级持续性级别,这会提供基础结构集成。Choose a durability level of Gold or Silver for the node types in your cluster, which gives you the infrastructure integration. 这随后会在横向缩减时自动从我们的系统服务 (FM) 状态中删除节点。Which will then automatically remove the nodes from our system services (FM) state when you scale in. 此处了解有关持续性级别的详细信息Refer to the details on durability levels here

  2. 横向缩减 VM 实例之后,需要调用 Remove-ServiceFabricNodeState cmdletOnce the VM instance has been scaled in, you need to call the Remove-ServiceFabricNodeState cmdlet.

    备注

    Service Fabric 群集需要有一定数量的节点可随时启动,以保持可用性和状态 - 称为“维持仲裁”。Service Fabric clusters require a certain number of nodes to be up at all the time in order to maintain availability and preserve state - referred to as "maintaining quorum." 因此,除非已事先执行状态的完整备份,否则关闭群集中的所有计算机通常是不安全的做法。So, it is typically unsafe to shut down all the machines in the cluster unless you have first performed a full backup of your state.

后续步骤Next steps

参阅以下文章以另外了解如何规划群集容量、升级群集以及对服务进行分区:Read the following to also learn about planning cluster capacity, upgrading a cluster, and partitioning services: