资源调控Resource governance

在同一节点或群集上运行多个服务时,其中一个服务可能会占用更多资源,导致相应流程中的其他服务缺少资源。When you're running multiple services on the same node or cluster, it's possible that one service might consume more resources, starving other services in the process. 这种问题称为“邻近干扰”问题。This problem is referred to as the "noisy neighbor" problem. 借助 Azure Service Fabric,开发者可以为每个服务指定资源预留和限制,从而保证并限制资源使用。Azure Service Fabric enables the developer to specify reservations and limits per service to guarantee resources and limit resource usage.


继续阅读本文之前,建议先熟悉 Service Fabric 应用程序模型Service Fabric 托管模型Before you proceed with this article, we recommend that you get familiar with the Service Fabric application model and the Service Fabric hosting model.

资源调控指标Resource governance metrics

根据服务包,Service Fabric 支持资源治理。Resource governance is supported in Service Fabric in accordance with the service package. 可以在代码包之间进一步划分分配到服务包的资源。The resources that are assigned to the service package can be further divided between code packages. 指定的资源限制也意味着资源预留。The resource limits that are specified also mean the reservation of the resources. Service Fabric 支持使用两个内置指标,为每个服务包指定 CPU 和内存:Service Fabric supports specifying CPU and memory per service package, with two built-in metrics:

  • CPU(指标名称 servicefabric:/_CpuCores):主机计算机上可用的逻辑核心。CPU (metric name servicefabric:/_CpuCores): A logical core that's available on the host machine. 所有节点上的全部内核都进行了相同的加权。All cores across all nodes are weighted the same.

  • 内存(指标名称 servicefabric:/_MemoryInMB):内存以 MB 表示,并映射到计算机上可用的物理内存。Memory (metric name servicefabric:/_MemoryInMB): Memory is expressed in megabytes, and it maps to physical memory that is available on the machine.

对于这两个指标,群集资源管理器跟踪总群集容量、群集中每个节点上的负载以及群集中剩余的资源。For these two metrics, Cluster Resource Manager tracks total cluster capacity, the load on each node in the cluster, and the remaining resources in the cluster. 这两个指标等同于其他任何用户指标或自定义指标。These two metrics are equivalent to any other user or custom metric. 现有全部功能都可以与它们结合使用:All existing features can be used with them:

  • 群集可根据这两个指标进行均衡(默认行为)。The cluster can be balanced according to these two metrics (default behavior).
  • 群集可根据这两个指标进行碎片整理The cluster can be defragmented according to these two metrics.
  • 描述群集时,可为这两个指标设置缓冲容量。When describing a cluster, buffered capacity can be set for these two metrics.


这些指标不支持动态负载报告;在创建时即定义了这些指标的负载。Dynamic load reporting is not supported for these metrics; loads for these metrics are defined at creation time.

资源治理机制Resource governance mechanism

Service Fabric 运行时当前不提供资源预留。The Service Fabric runtime currently does not provide reservation for resources. 当进程或容器打开时,运行时就会将资源限制设置为创建时定义的负载。When a process or a container is opened, the runtime sets the resource limits to the loads that were defined at creation time. 此外,如果资源超额,运行时还会拒绝打开新服务包。Furthermore, the runtime rejects the opening of new service packages that are available when resources are exceeded. 为了更好地理解此过程的工作原理,请以下面包含两个 CPU 内核的节点为例(等同于内存治理机制):To better understand how the process works, let's take an example of a node with two CPU cores (mechanism for memory governance is equivalent):

  1. 首先,在节点上放置一个需要一个 CPU 内核的容器。First, a container is placed on the node, requesting one CPU core. 运行时打开此容器,并将 CPU 限制设置为一个内核。The runtime opens the container and sets the CPU limit to one core. 此容器无法使用多个内核。The container won't be able to use more than one core.

  2. 然后,在节点上放置一个服务副本,相应的服务包指定一个 CPU 内核作为资源限制。Then, a replica of a service is placed on the node, and the corresponding service package specifies a limit of one CPU core. 运行时打开代码包,并将它的 CPU 限制设置为一个内核。The runtime opens the code package and sets its CPU limit to one core.

此时,限制之和等于节点容量。At this point, the sum of limits is equal to the capacity of the node. 进程和容器各使用一个内核在运行,互不干扰。A process and a container are running with one core each and not interfering with each other. Service Fabric 不再放置其他任何指定 CPU 限制的容器或副本。Service Fabric doesn't place any more containers or replicas when they are specifying the CPU limit.

不过,在两种情况下,其他进程可能会争用 CPU。However, there are two situations in which other processes might contend for CPU. 在这种情况下,示例中的进程和容器可能会遇到邻近干扰问题:In these situations, a process and a container from our example might experience the noisy neighbor problem:

  • 混用调控和非调控服务与容器:如果用户创建服务时没有指定任何资源治理,运行时将它视为不占用任何资源,能够将它放置在示例中的节点上。Mixing governed and non-governed services and containers: If a user creates a service without any resource governance specified, the runtime sees it as consuming no resources, and can place it on the node in our example. 在这种情况下,这一新进程实际上会占用部分 CPU,占用的是已在节点上运行的服务的份额。In this case, this new process effectively consumes some CPU at the expense of the services that are already running on the node. 此问题有两种解决方案。There are two solutions to this problem. 在同一群集中不混用治理和非治理服务,或使用放置约束,阻止这两种类型的服务最终位于同一组节点上。Either don't mix governed and non-governed services on the same cluster, or use placement constraints so that these two types of services don't end up on the same set of nodes.

  • 其他进程在 Service Fabric 外的节点上启动(例如 OS 服务) :在这种情况下,Service Fabric 外的进程也会与现有服务争用 CPU。When another process is started on the node, outside Service Fabric (for example, an OS service): In this situation, the process outside Service Fabric also contends for CPU with existing services. 此问题的解决方案是,考虑 OS 开销以正确设置节点容量,如下一部分中所示。The solution to this problem is to set up node capacities correctly to account for OS overhead, as shown in the next section.

启用资源治理所需的群集设置Cluster setup for enabling resource governance

当节点启动并加入群集时,Service Fabric 会先检测可用内存量和可用内核数,再设置这两个资源的节点容量。When a node starts and joins the cluster, Service Fabric detects the available amount of memory and the available number of cores, and then sets the node capacities for those two resources.

为了给操作系统以及可能在节点上运行的其他进程留出缓冲空间,Service Fabric 只使用节点上 80% 的可用资源。To leave buffer space for the operating system, and for other processes might be running on the node, Service Fabric uses only 80% of the available resources on the node. 此百分比可进行配置,并且可在群集清单中进行更改。This percentage is configurable, and can be changed in the cluster manifest.

以下示例介绍如何指示 Service Fabric 使用 50% 的可用 CPU 和 70% 的可用内存:Here is an example of how to instruct Service Fabric to use 50% of available CPU and 70% of available memory:

<Section Name="PlacementAndLoadBalancing">
    <!-- 0.0 means 0%, and 1.0 means 100%-->
    <Parameter Name="CpuPercentageNodeCapacity" Value="0.5" />
    <Parameter Name="MemoryPercentageNodeCapacity" Value="0.7" />

对于大多数客户和方案,建议配置为自动检测 CPU 和内存的节点容量(默认情况下自动检测已启用)。For most customers and scenarios, automatic detection of node capacities for the CPU and memory is the recommended configuration (automatic detection is turned on by default). 但是,如果需要完全手动设置节点容量,则可以使用用于描述群集中节点的机制按节点类型进行配置。However, if you need full manual setup of node capacities, you can configure those per node type using the mechanism for describing the nodes in the cluster. 下面的示例展示了如何设置具有四个核心和 2GB 内存的节点类型:Here is an example of how to set up the node type with four cores and 2 GB of memory:

    <NodeType Name="MyNodeType">
        <Capacity Name="servicefabric:/_CpuCores" Value="4"/>
        <Capacity Name="servicefabric:/_MemoryInMB" Value="2048"/>

如果已启用自动检测可用资源,并在群集清单中手动定义了节点容量,Service Fabric 会检查节点中的资源是否足以支持用户定义的容量:When auto-detection of available resources is enabled, and node capacities are manually defined in the cluster manifest, Service Fabric checks that the node has enough resources to support the capacity that the user has defined:

  • 如果清单中定义的节点容量小于或等于节点上的可用资源,Service Fabric 使用清单中指定的容量。If node capacities that are defined in the manifest are less than or equal to the available resources on the node, then Service Fabric uses the capacities that are specified in the manifest.

  • 如果清单中定义的节点容量大于可用资源,Service Fabric 使用可用资源作为节点容量。If node capacities that are defined in the manifest are greater than available resources, Service Fabric uses the available resources as node capacities.

如果不需要,可以禁用自动检测可用资源。Auto-detection of available resources can be turned off if it is not required. 若要禁用此功能,请更改以下设置:To turn it off, change the following setting:

<Section Name="PlacementAndLoadBalancing">
    <Parameter Name="AutoDetectAvailableResources" Value="false" />

为了获得最佳性能,还应在群集清单中打开以下设置:For optimal performance, the following setting should also be turned on in the cluster manifest:

<Section Name="PlacementAndLoadBalancing">
    <Parameter Name="PreventTransientOvercommit" Value="true" />
    <Parameter Name="AllowConstraintCheckFixesDuringApplicationUpgrade" Value="true" />


从 Service Fabric version 7.0 开始,我们更新了在用户手动提供节点资源容量值的情况下,节点资源容量的规则计算方法。Starting with Service Fabric version 7.0, we have updated the rule for how node resource capacities are calculated in the cases where user manually provides the values for node resource capacities. 让我们考虑以下这种情况:Let's consider the following scenario:

  • 节点上总共有 10 个 CPU 核心数There are 10 cpu cores total on the node
  • SF 配置为使用用户服务总资源的 80%(默认设置),这将为节点上运行的其他服务(包括 Service Fabric 系统服务)保留 20% 的缓冲区SF is configured to use 80% of the total resources for the user services (default setting), which leaves a buffer of 20% for the other services running on the node (incl. Service Fabric system services)
  • 用户决定手动覆盖 CPU 核心数指标的节点资源容量,并将其设置为 5 个核心User decides to manually override the node resource capacity for the cpu cores metric, and sets it to 5 cores

我们已更改了关于 Service Fabric 用户服务可用容量的规则计算方式,内容如下:We have changed the rule on how the available capacity for Service Fabric user services is calculated in the following way:

  • 在 Service Fabric 7.0 之前,用户服务的可用容量将计算为 5 个核心数(忽略 20% 的容量缓冲区)Before Service Fabric 7.0, available capacity for user services would be calculated to 5 cores (capacity buffer of 20% is ignored)
  • 从 Service Fabric 7.0 开始,用户服务的可用容量将计算为 4 个核心数(不忽略 20% 的容量缓冲区)Starting with Service Fabric 7.0, available capacity for user services would be calculated to 4 cores (capacity buffer of 20% is not ignored)

指定资源治理Specify resource governance

应用程序清单(ServiceManifestImport 部分)中指定了资源调控限制,如以下示例所示:Resource governance limits are specified in the application manifest (ServiceManifestImport section) as shown in the following example:

<?xml version='1.0' encoding='UTF-8'?>
<ApplicationManifest ApplicationTypeName='TestAppTC1' ApplicationTypeVersion='vTC1' xsi:schemaLocation='http://schemas.microsoft.com/2011/01/fabric ServiceFabricServiceModel.xsd' xmlns='http://schemas.microsoft.com/2011/01/fabric' xmlns:xsi='https://www.w3.org/2001/XMLSchema-instance'>

  ServicePackageA has the number of CPU cores defined, but doesn't have the MemoryInMB defined.
  In this case, Service Fabric sums the limits on code packages and uses the sum as 
  the overall ServicePackage limit.
    <ServiceManifestRef ServiceManifestName='ServicePackageA' ServiceManifestVersion='v1'/>
      <ServicePackageResourceGovernancePolicy CpuCores="1"/>
      <ResourceGovernancePolicy CodePackageRef="CodeA1" CpuShares="512" MemoryInMB="1000" />
      <ResourceGovernancePolicy CodePackageRef="CodeA2" CpuShares="256" MemoryInMB="1000" />

在此示例中,服务包 ServicePackageA 在驻留的节点上拥有一个内核的资源。In this example, the service package called ServicePackageA gets one core on the nodes where it is placed. 此服务包有两个代码包(CodeA1 和 CodeA2),并且都指定了 CpuShares 参数。This service package contains two code packages (CodeA1 and CodeA2), and both specify the CpuShares parameter. CpuShares 512:256 的比例将核心划分到两个代码包中。The proportion of CpuShares 512:256 divides the core across the two code packages.

因此,在此示例中,CodeA1 分得三分之二个内核,CodeA2 分得三分之一个内核(和相同的软保证预留)。Thus, in this example, CodeA1 gets two-thirds of a core, and CodeA2 gets one-third of a core (and a soft-guarantee reservation of the same). 如果没有为代码包指定 CpuShares,Service Fabric 会在这两个代码包之间平分内核。If CpuShares are not specified for code packages, Service Fabric divides the cores equally among them.

内存限制是绝对的,所以这两个代码包都限制为 1024 MB 内存(和相同的软保证预留)。Memory limits are absolute, so both code packages are limited to 1024 MB of memory (and a soft-guarantee reservation of the same). 代码包(容器或进程)无法分配到超出此限制的内存。如果尝试这样做,则会抛出内存不足异常。Code packages (containers or processes) can't allocate more memory than this limit, and attempting to do so results in an out-of-memory exception. 若要强制执行资源限制,服务包中的所有代码包均应指定内存限制。For resource limit enforcement to work, all code packages within a service package should have memory limits specified.

使用应用程序参数Using application parameters

指定资源调控设置时,可使用应用程序参数管理多个应用配置。When specifying resource governance settings, it is possible to use application parameters to manage multiple app configurations. 下例展示应用程序参数的用法:The following example shows the usage of application parameters:

<?xml version='1.0' encoding='UTF-8'?>
<ApplicationManifest ApplicationTypeName='TestAppTC1' ApplicationTypeVersion='vTC1' xsi:schemaLocation='http://schemas.microsoft.com/2011/01/fabric ServiceFabricServiceModel.xsd' xmlns='http://schemas.microsoft.com/2011/01/fabric' xmlns:xsi='https://www.w3.org/2001/XMLSchema-instance'>

    <Parameter Name="CpuCores" DefaultValue="4" />
    <Parameter Name="CpuSharesA" DefaultValue="512" />
    <Parameter Name="CpuSharesB" DefaultValue="512" />
    <Parameter Name="MemoryA" DefaultValue="2048" />
    <Parameter Name="MemoryB" DefaultValue="2048" />

    <ServiceManifestRef ServiceManifestName='ServicePackageA' ServiceManifestVersion='v1'/>
      <ServicePackageResourceGovernancePolicy CpuCores="[CpuCores]"/>
      <ResourceGovernancePolicy CodePackageRef="CodeA1" CpuShares="[CpuSharesA]" MemoryInMB="[MemoryA]" />
      <ResourceGovernancePolicy CodePackageRef="CodeA2" CpuShares="[CpuSharesB]" MemoryInMB="[MemoryB]" />

在此示例中,会为生产环境设置默认参数,其中每个服务包具有 4 核和 2 GB 内存。In this example, default parameter values are set for the production environment, where each Service Package would get 4 cores and 2 GB of memory. 可使用应用程序参数文件更改默认值。It is possible to change default values with application parameter files. 在此示例中,一个参数文件可以用来本地测试应用程序,其中它获得的资源将少于生产中所得:In this example, one parameter file can be used for testing the application locally, where it would get less resources than in production:

<!-- ApplicationParameters\Local.xml -->

<Application Name="fabric:/TestApplication1" xmlns="http://schemas.microsoft.com/2011/01/fabric">
    <Parameter Name="CpuCores" DefaultValue="2" />
    <Parameter Name="CpuSharesA" DefaultValue="512" />
    <Parameter Name="CpuSharesB" DefaultValue="512" />
    <Parameter Name="MemoryA" DefaultValue="1024" />
    <Parameter Name="MemoryB" DefaultValue="1024" />


从 Service Fabric 6.1 版开始,可使用应用程序参数指定资源调控。Specifying resource governance with application parameters is available starting with Service Fabric version 6.1.

使用应用程序参数指定资源调控时,Service Fabric 无法降级到 6.1 之前的版本。When application parameters are used to specify resource governance, Service Fabric cannot be downgraded to a version prior to version 6.1.

强制执行用户服务的资源限制Enforcing the resource limits for user services

虽然将资源治理应用于 Service Fabric 服务可以确保这些受治理资源服务不超过其资源配额,但许多用户仍然需要以非治理模式运行某些 Service Fabric 服务。While applying resource governance to your Service Fabric services guarantees that those resource-governed services cannot exceed their resources quota, many users still need to run some of their Service Fabric services in ungoverned mode. 使用非治理 Service Fabric 服务时,可能会遇到“失控”的非治理服务消耗 Service Fabric 节点上所有可用资源的情况,这可能会导致严重的问题,例如:When using ungoverned Service Fabric services, it is possible to run into situations where "runaway" ungoverned services consume all available resources on the Service Fabric nodes, which can lead to serious issues like:

  • 节点上运行的其他服务(包括 Service Fabric 系统服务)的资源不足Resource starvation of other services running on the nodes (including Service Fabric system services)
  • 节点以不正常状态结束Nodes ending up in an unhealthy state
  • 群集管理 API Service Fabric 无响应Unresponsive Service Fabric cluster management APIs

为了防止发生这些情况,Service Fabric 允许你对节点上运行的所有 Service Fabric 用户服务(受治理和未受治理)实施资源限制,以确保用户服务永远不会使用超过指定数量的资源 ** 。To prevent these situations from occurring, Service Fabric allows you to enforce the resource limits for all Service Fabric user services running on the node (both governed and ungoverned) to guarantee that user services will never use more than the specified amount of resources. 此限制可通过将 ClusterManifest 的 PlacementAndLoadBalancing 部分中的 EnforceUserServiceMetricCapacities 配置的值设置为 true 来实现。This is achieved by setting the value for the EnforceUserServiceMetricCapacities config in the PlacementAndLoadBalancing section of the ClusterManifest to true. 默认情况下,此设置处于关闭状态。This setting is turned off by default.

    <ParameterName="EnforceUserServiceMetricCapacities" Value="false"/>

其他备注:Additional remarks:

  • 资源限制强制仅适用于 servicefabric:/_CpuCoresservicefabric:/_MemoryInMB 资源指标Resource limit enforcement only applies to the servicefabric:/_CpuCores and servicefabric:/_MemoryInMB resource metrics
  • 仅当资源指标的节点容量可用于 Service Fabric 时,资源限制强制实施才会起作用,可以通过自动检测机制,也可以通过用户手动指定节点容量(如启用资源治理的群集设置部分所述)。Resource limit enforcement only works if node capacities for the resource metrics are available to Service Fabric, either via auto-detection mechanism, or via users manually specifying the node capacities (as explained in the Cluster setup for enabling resource governance section). 如果未配置节点容量,则无法使用资源限制强制实施功能,因为 Service Fabric 不知道要为用户服务保留多少资源。 If node capacities are not configured, the resource limit enforcement capability cannot be used since Service Fabric can't know how much resources to reserve for user services. 如果“EnforceUserServiceMetricCapacities”为 true 但未配置节点容量,则 Service Fabric 将发出运行状况警告。 Service Fabric will issue a health warning if "EnforceUserServiceMetricCapacities" is true but node capacities are not configured.

容器的其他资源Other resources for containers

除了 CPU 和内存之外,还可以为容器指定其他资源限制。Besides CPU and memory, it's possible to specify other resource limits for containers. 这些限制是在代码包一级指定,并在容器启动时应用。These limits are specified at the code-package level and are applied when the container is started. 这些资源与 CPU 和内存不同,群集资源管理器不会注意到它们,也不会针对它们进行任何容量检查或负载均衡。Unlike with CPU and memory, Cluster Resource Manager isn't aware of these resources, and won't do any capacity checks or load balancing for them.

  • MemorySwapInMB:容器可使用的交换内存量。MemorySwapInMB: The amount of swap memory that a container can use.
  • MemoryReservationInMB:内存调控软限制,仅当在节点上检测到内存争用时才强制执行此限制。MemoryReservationInMB: The soft limit for memory governance that is enforced only when memory contention is detected on the node.
  • CpuPercent:容器可使用的 CPU 百分比。CpuPercent: The percentage of CPU that the container can use. 如果为服务包指定了 CPU 限制,将有效忽略此参数。If CPU limits are specified for the service package, this parameter is effectively ignored.
  • MaximumIOps:容器可使用的最大 IOPS(读取和写入)。MaximumIOps: The maximum IOPS that a container can use (read and write).
  • MaximumIOBytesps:容器可使用(读取和写入)的最大 IO(字节/秒)。MaximumIOBytesps: The maximum IO (bytes per second) that a container can use (read and write).
  • BlockIOWeight:相对于其他容器的块 IO 权重。BlockIOWeight: The block IO weight for relative to other containers.

这些资源可与 CPU 和内存组合。These resources can be combined with CPU and memory. 以下示例显示如何为容器指定其他资源:Here is an example of how to specify additional resources for containers:

    <ServiceManifestRef ServiceManifestName="FrontendServicePackage" ServiceManifestVersion="1.0"/>
        <ResourceGovernancePolicy CodePackageRef="FrontendService.Code" CpuPercent="5"
        MemorySwapInMB="4084" MemoryReservationInMB="1024" MaximumIOPS="20" />

后续步骤Next steps