创建可高效列出 Batch 资源的查询Create queries to list Batch resources efficiently

几乎所有批处理应用程序都需执行某类监视操作或其他查询批处理服务的操作(通常按固定的时间间隔)。Nearly all Batch applications need to perform some type of monitoring or other operation that queries the Batch service, often at regular intervals. 例如,若要确定作业中是否还有排队的任务,必须获取作业中每个任务的相关数据。For example, to determine whether there are any queued tasks remaining in a job, you must get data on every task in the job. 若要确定池中节点的状态,必须获取池中每个节点的相关数据。To determine the status of nodes in your pool, you must get data on every node in the pool. 本文介绍如何以最有效方式执行此类查询。This article explains how to execute such queries in the most efficient way.

可通过减少使用 Batch .NET 库查询作业、任务、计算节点及其他资源时该服务返回的数据量,提高 Azure Batch 应用程序的性能。You can increase your Azure Batch application's performance by reducing the amount of data that is returned by the service when you query jobs, tasks, compute nodes, and other resources with the Batch .NET library.

备注

Batch 服务为作业中的任务计数以及 Batch 池中的计算节点计数这类常见方案提供 API 支持。The Batch service provides API support for the common scenarios of counting tasks in a job, and counting compute nodes in Batch pool. 对于这些方案可以调用获取任务计数列出池节点计数操作,而不是使用列表查询。Instead of using a list query for these, you can call the Get Task Counts and List Pool Node Counts operations. 这些操作比列表查询更高效,但返回的信息有限且并不总是最新信息。These operations are more efficient than a list query, but return more limited information that may not always be up to date. 有关详细信息,请参阅按状态对任务和计算节点计数For more information, see Count tasks and compute nodes by state.

指定详细信息级别Specify a detail level

在生产型批处理应用程序中,作业、任务和计算节点等实体的数目成千上万。In a production Batch application, entities like jobs, tasks, and compute nodes can number in the thousands. 请求这些资源的相关信息时,可能需要将大量的数据从 Batch 服务“跨网络”传输到执行每个查询的应用程序。When you request information on these resources, a potentially large amount of data must "cross the wire" from the Batch service to your application on each query. 通过限制查询时返回的项数和信息类型,可以提高查询速度,因此也会提高应用程序的性能。By limiting the number of items and type of information that is returned by a query, you can increase the speed of your queries, and therefore the performance of your application.

Batch .NET API 代码片段列出了与作业关联的每个任务,以及每个任务的所有属性 :This Batch .NET API code snippet lists every task that is associated with a job, along with all of the properties of each task:

// Get a collection of all of the tasks and all of their properties for job-001
IPagedEnumerable<CloudTask> allTasks =
    batchClient.JobOperations.ListTasks("job-001");

但是,用户可以通过向查询应用“详细信息级别”,执行效率高得多的列表查询。You can perform a much more efficient list query, however, by applying a "detail level" to your query. 可通过向 JobOperations.ListTasks 方法提供 ODATADetailLevel 对象实现此目的。You do this by supplying an ODATADetailLevel object to the JobOperations.ListTasks method. 此代码段仅返回已完成任务的 ID、命令行和计算节点信息属性:This snippet returns only the ID, command line, and compute node information properties of completed tasks:

// Configure an ODATADetailLevel specifying a subset of tasks and
// their properties to return
ODATADetailLevel detailLevel = new ODATADetailLevel();
detailLevel.FilterClause = "state eq 'completed'";
detailLevel.SelectClause = "id,commandLine,nodeInfo";

// Supply the ODATADetailLevel to the ListTasks method
IPagedEnumerable<CloudTask> completedTasks =
    batchClient.JobOperations.ListTasks("job-001", detailLevel);

在此示例方案中,如果作业中存在数以千计的任务,则通常情况下,第二个查询的结果的返回速度将远远快于第一个查询。In this example scenario, if there are thousands of tasks in the job, the results from the second query will typically be returned much quicker than the first. 下面提供了有关使用 Batch .NET API 列出项时使用 ODATADetailLevel 的详细信息。More information about using ODATADetailLevel when you list items with the Batch .NET API is included below.

重要

强烈建议始终 将 ODATADetailLevel 对象提供给 .NET API 列表调用,确保最大程度地提高应用程序的效率和性能。We highly recommend that you always supply an ODATADetailLevel object to your .NET API list calls to ensure maximum efficiency and performance of your application. 指定详细程度有助于缩短 Batch 服务响应时间、提高网络利用率,以及最大程度减少客户端应用程序的内存使用量。By specifying a detail level, you can help to lower Batch service response times, improve network utilization, and minimize memory usage by client applications.

Filter、select 和 expandFilter, select, and expand

Batch .NETBatch REST API 可以减少列表中返回的项数以及针对每个查询返回的信息量。The Batch .NET and Batch REST APIs provide the ability to reduce both the number of items that are returned in a list, as well as the amount of information that is returned for each. 在执行列表查询时可以通过指定 filterselectexpand 字符串来实现此目的。You do so by specifying filter, select, and expand strings when performing list queries.

筛选器Filter

filter 字符串是一个表达式,用于减少返回的项数。The filter string is an expression that reduces the number of items that are returned. 例如,可以只列出作业的正在运行的任务,或者只列出已准备好来运行任务的计算节点。For example, you can list only the running tasks for a job, or list only compute nodes that are ready to run tasks.

filter 字符串包含一个或多个表达式,其中一个表达式包含属性名称、运算符和值。The filter string consists of one or more expressions, with an expression that consists of a property name, operator, and value. 能够指定哪些属性取决于所查询的每个实体类型,每个属性所支持的运算符也是这样。The properties that can be specified are specific to each entity type that you query, as are the operators that are supported for each property. 可以使用逻辑运算符 andor 将多个表达式组合到一起。Multiple expressions can be combined by using the logical operators and and or.

此示例性 filter 字符串仅列出正在运行的“呈现”任务:(state eq 'running') and startswith(id, 'renderTask')This example filter string lists only the running "render" tasks: (state eq 'running') and startswith(id, 'renderTask').

SelectSelect

select 字符串用于限制为每个项返回的属性值。The select string limits the property values that are returned for each item. 指定以逗号分隔的属性名称列表,并且只为查询结果中的项返回这些属性值。可指定正在查询的实体类型的任何属性。You specify a list of comma-separated property names, and only those property values are returned for the items in the query results.You can specify any of the properties for the entity type you are querying.

此示例性的 select 字符串指定每个任务只应返回三项属性值:id, state, stateTransitionTimeThis example select string specifies that only three property values should be returned for each task: id, state, stateTransitionTime.

展开Expand

expand 字符串用于减少获取特定信息所需的 API 调用数。The expand string reduces the number of API calls that are required to obtain certain information. 使用 expand 字符串时,单次 API 调用可以获取每个项目的更多信息。When you use an expand string, more information about each item can be obtained with a single API call. 不必先获取实体的列表,然后请求列表中每项的信息。你可使用 expand 字符串通过单次 API 调用获取相同的信息,从而通过减少 API 调用来提高性能。Rather than first obtaining the list of entities, then requesting information for each item in the list, you use an expand string to obtain the same information in a single API call, helping to improve performance by reducing API calls.

与 select 字符串类似,expand 字符串用于控制是否允许某些数据包括在列表查询结果中。Similar to the select string, the expand string controls whether certain data is included in list query results. 当所有属性都是必需属性且没有指定 select 字符串时,必须 使用 expand 字符串来获取统计信息。When all properties are required and no select string is specified, the expand string must be used to get statistics information. 如果使用了 select 字符串来获取属性的子集,则可在 select 字符串中指定 stats,不需指定 expand 字符串。If a select string is used to obtain a subset of properties, then stats can be specified in the select string, and the expand string does not need to be specified.

expand 字符串只有在列出作业、作业计划、任务和池中使用时才受支持。The expand string is only supported when it is used in listing jobs, job schedules, tasks, and pools. 目前仅支持统计信息。Currently, it only supports statistics information.

此示例性 expand 字符串指定列表中的每个项都应返回统计信息:statsThis example expand string specifies that statistics information should be returned for each item in the list: stats.

备注

构造这三种查询字符串类型(filter、select 和 expand)中的任意一种类型时,必须确保属性名称和大小写与其 REST API 元素的对应项相匹配。When constructing any of the three query string types (filter, select, and expand), you must ensure that the property names and case match that of their REST API element counterparts. 例如,在使用 .NET CloudTask 类时,必须指定 state 而非 State,即使 .NET 属性为 CloudTask.StateFor example, when working with the .NET CloudTask class, you must specify state instead of State, even though the .NET property is CloudTask.State. 请参阅下表中 .NET 和 REST API 之间的属性映射。See the tables below for property mappings between the .NET and REST APIs.

filter、select 和 expand 字符串的规则Rules for filter, select, and expand strings

  • filter、select 和 expand 字符串中属性名称的显示方式应与其在 Batch REST API 中的显示方式相同,即使你使用 Batch .NET 或其他某个 Batch SDK 时也是如此。Properties names in filter, select, and expand strings should appear as they do in the Batch REST API, even when you use Batch .NET or one of the other Batch SDKs.

  • 所有属性名称均区分大小写,但属性值不区分大小写。All property names are case-sensitive, but property values are case insensitive.

  • 日期/时间字符串可以采用两种格式中的一种,并且必须在前面加上 DateTimeDate/time strings can be one of two formats, and must be preceded with DateTime.

    • W3C-DTF 格式示例:creationTime gt DateTime'2011-05-08T08:49:37Z'W3C-DTF format example: creationTime gt DateTime'2011-05-08T08:49:37Z'
    • RFC 1123 格式示例:creationTime gt DateTime'Sun, 08 May 2011 08:49:37 GMT'RFC 1123 format example: creationTime gt DateTime'Sun, 08 May 2011 08:49:37 GMT'
  • 布尔值字符串为 truefalseBoolean strings are either true or false.

  • 如果指定了无效的属性或运算符,则会导致 400 (Bad Request) 错误。If an invalid property or operator is specified, a 400 (Bad Request) error will result.

在 Batch .NET 中进行高效查询Efficient querying in Batch .NET

Batch .NET API 中,将通过 ODATADetailLevel 类来提供 filter、select 和 expand 字符串以列出相应操作。Within the Batch .NET API, the ODATADetailLevel class is used for supplying filter, select, and expand strings to list operations. ODataDetailLevel 类有三个公共字符串属性,这些属性可以在构造函数中指定,也可以直接在对象上设置。The ODataDetailLevel class has three public string properties that can be specified in the constructor, or set directly on the object. 然后即可将 ODataDetailLevel 对象作为参数传递给不同的列表操作,例如 ListPoolsListJobsListTasksYou then pass the ODataDetailLevel object as a parameter to the various list operations such as ListPools, ListJobs, and ListTasks.

以下代码段使用 Batch .NET API 对 Batch 服务进行有效的查询,查询其中是否存在特定池集的统计信息。The following code snippet uses the Batch .NET API to efficiently query the Batch service for the statistics of a specific set of pools. 在此方案中,Batch 用户既有测试池又有生产池。In this scenario, the Batch user has both test and production pools. 测试池 ID 具有“test”前缀,生产池 ID 具有“prod”前缀。The test pool IDs are prefixed with "test", and the production pool IDs are prefixed with "prod". 在代码片段中,myBatchClient 是正确初始化的 BatchClient 类实例。In the snippet, myBatchClient is a properly initialized instance of the BatchClient class.

// First we need an ODATADetailLevel instance on which to set the filter, select,
// and expand clause strings
ODATADetailLevel detailLevel = new ODATADetailLevel();

// We want to pull only the "test" pools, so we limit the number of items returned
// by using a FilterClause and specifying that the pool IDs must start with "test"
detailLevel.FilterClause = "startswith(id, 'test')";

// To further limit the data that crosses the wire, configure the SelectClause to
// limit the properties that are returned on each CloudPool object to only
// CloudPool.Id and CloudPool.Statistics
detailLevel.SelectClause = "id, stats";

// Specify the ExpandClause so that the .NET API pulls the statistics for the
// CloudPools in a single underlying REST API call. Note that we use the pool's
// REST API element name "stats" here as opposed to "Statistics" as it appears in
// the .NET API (CloudPool.Statistics)
detailLevel.ExpandClause = "stats";

// Now get our collection of pools, minimizing the amount of data that is returned
// by specifying the detail level that we configured above
List<CloudPool> testPools =
    await myBatchClient.PoolOperations.ListPools(detailLevel).ToListAsync();

提示

使用 Select 和 Expand 子句配置的 ODATADetailLevel 实例也可以传递给相应的 Get 方法(例如 PoolOperations.GetPool),以便限制返回的数据量。An instance of ODATADetailLevel that is configured with Select and Expand clauses can also be passed to appropriate Get methods, such as PoolOperations.GetPool, to limit the amount of data that is returned.

Batch REST 到 .NET API 映射Batch REST to .NET API mappings

filter、select 和 expand 字符串中的属性名称必须反映其 REST API 对应项,不管是名称本身还是大小写。Property names in filter, select, and expand strings must reflect their REST API counterparts, both in name and case. 下表提供了 .NET 和 REST API 的对应项之间的映射。The tables below provide mappings between the .NET and REST API counterparts.

filter 字符串的映射Mappings for filter strings

  • .NET 列表方法:此列中的每个 .NET API 方法都接受 ODATADetailLevel 对象作为参数。.NET list methods: Each of the .NET API methods in this column accepts an ODATADetailLevel object as a parameter.
  • REST 列表请求:此列中的每个 REST API 页面都包含一个表,该表指定了 filter 字符串中允许的属性和操作。REST list requests: Each REST API page linked to in this column contains a table that specifies the properties and operations that are allowed in filter strings. 在构造 ODATADetailLevel.FilterClause 字符串时,将使用这些属性名称和操作。You use these property names and operations when you construct an ODATADetailLevel.FilterClause string.
.NET 列表方法.NET list methods REST 列表请求REST list requests
CertificateOperations.ListCertificatesCertificateOperations.ListCertificates 列出帐户中的证书List the certificates in an account
CloudTask.ListNodeFilesCloudTask.ListNodeFiles 列出与任务关联的文件List the files associated with a task
JobOperations.ListJobPreparationAndReleaseTaskStatusJobOperations.ListJobPreparationAndReleaseTaskStatus 列出作业准备状态以及作业的作业版本任务List the status of the job preparation and job release tasks for a job
JobOperations.ListJobsJobOperations.ListJobs 列出帐户中的作业List the jobs in an account
JobOperations.ListNodeFilesJobOperations.ListNodeFiles 列出节点上的文件List the files on a node
JobOperations.ListTasksJobOperations.ListTasks 列出与作业关联的任务List the tasks associated with a job
JobScheduleOperations.ListJobSchedulesJobScheduleOperations.ListJobSchedules 列出帐户中的作业计划List the job schedules in an account
JobScheduleOperations.ListJobsJobScheduleOperations.ListJobs 列出与作业计划关联的作业List the jobs associated with a job schedule
PoolOperations.ListComputeNodesPoolOperations.ListComputeNodes 列出池中的计算节点List the compute nodes in a pool
PoolOperations.ListPoolsPoolOperations.ListPools 列出帐户中的池List the pools in an account

select 字符串的映射Mappings for select strings

  • Batch.NET 类型:Batch.NET API 类型。Batch .NET types: Batch .NET API types.
  • REST API 实体:此列中的每一页都包含一个或多个表,其中列出了类型的 REST API 属性名称。REST API entities: Each page in this column contains one or more tables that list the REST API property names for the type. 在构造 select 字符串时使用这些属性名称。These property names are used when you construct select strings. 在构造 ODATADetailLevel.SelectClause 字符串时,将使用这些相同的属性名称。You use these same property names when you construct an ODATADetailLevel.SelectClause string.
Batch .NET 类型Batch .NET types REST API 实体REST API entities
证书Certificate 获取有关证书的信息Get information about a certificate
CloudJobCloudJob 获取有关作业的信息Get information about a job
CloudJobScheduleCloudJobSchedule 获取有关作业计划的信息Get information about a job schedule
ComputeNodeComputeNode 获取有关节点的信息Get information about a node
CloudPoolCloudPool 获取有关池的信息Get information about a pool
CloudTaskCloudTask 获取有关任务的信息Get information about a task

示例:构造 filter 字符串Example: construct a filter string

针对 ODATADetailLevel.FilterClause 构造 filter 字符串时,请查阅上表,在“filter 字符串的映射”下找到与所希望执行的列表操作相对应的 REST API 文档页。When you construct a filter string for ODATADetailLevel.FilterClause, consult the table above under "Mappings for filter strings" to find the REST API documentation page that corresponds to the list operation that you wish to perform. 会在该页第一个多行表中找到可筛选属性及其支持的运算符。You will find the filterable properties and their supported operators in the first multirow table on that page. 例如,如果希望检索其退出代码不为零的所有任务,则可查看列出与作业相关联的任务上的此行,此行指定了相应的属性字符串以及允许的运算符:If you wish to retrieve all tasks whose exit code was nonzero, for example, this row on List the tasks associated with a job specifies the applicable property string and allowable operators:

属性Property 允许的操作Operations allowed 类型Type
executionInfo/exitCode eq, ge, gt, le , lt Int

因此,若要列出退出代码不为零的所有任务,可使用以下 filter 字符串:Thus, the filter string for listing all tasks with a nonzero exit code would be:

(executionInfo/exitCode lt 0) or (executionInfo/exitCode gt 0)

示例:构造 select 字符串Example: construct a select string

若要构造 ODATADetailLevel.SelectClause,请查阅上表,在“select 字符串的映射”下导航到与所列实体类型相对应的 REST API 页。To construct ODATADetailLevel.SelectClause, consult the table above under "Mappings for select strings" and navigate to the REST API page that corresponds to the type of entity that you are listing. 会在该页第一个多行表中找到可选择属性及其支持的运算符。You will find the selectable properties and their supported operators in the first multirow table on that page. 例如,如果希望仅检索列表中每个任务的 ID 和命令行,则可在获取有关任务的信息的相应表中找到这些行:If you wish to retrieve only the ID and command line for each task in a list, for example, you will find these rows in the applicable table on Get information about a task:

属性Property 类型Type 说明Notes
id String The ID of the task.
commandLine String The command line of the task.

因此,若要只包括每个所列任务的 ID 和命令行,可使用以下 select 字符串:The select string for including only the ID and command line with each listed task would then be:

id, commandLine

代码示例Code samples

高效列表查询代码示例Efficient list queries code sample

GitHub 上的 EfficientListQueries 示例项目显示了列表查询如何有效地影响应用程序的性能。The EfficientListQueries sample project on GitHub shows how efficient list querying can affect performance in an application. 此 C# 控制台应用程序创建大量的任务并将其添加到作业。This C# console application creates and adds a large number of tasks to a job. 然后,它对 JobOperations.ListTasks 方法进行多次调用,并传递配置了不同属性值的 ODATADetailLevel 对象,以改变要返回的数据量。Then, it makes multiple calls to the JobOperations.ListTasks method and passes ODATADetailLevel objects that are configured with different property values to vary the amount of data to be returned. 生成的输出如下所示:It produces output similar to the following:

Adding 5000 tasks to job jobEffQuery...
5000 tasks added in 00:00:47.3467587, hit ENTER to query tasks...

4943 tasks retrieved in 00:00:04.3408081 (ExpandClause:  | FilterClause: state eq 'active' | SelectClause: id,state)
0 tasks retrieved in 00:00:00.2662920 (ExpandClause:  | FilterClause: state eq 'running' | SelectClause: id,state)
59 tasks retrieved in 00:00:00.3337760 (ExpandClause:  | FilterClause: state eq 'completed' | SelectClause: id,state)
5000 tasks retrieved in 00:00:04.1429881 (ExpandClause:  | FilterClause:  | SelectClause: id,state)
5000 tasks retrieved in 00:00:15.1016127 (ExpandClause:  | FilterClause:  | SelectClause: id,state,environmentSettings)
5000 tasks retrieved in 00:00:17.0548145 (ExpandClause: stats | FilterClause:  | SelectClause: )

Sample complete, hit ENTER to continue...

如所用时间中所示,限制返回的属性和项数可以大大缩短查询响应时间。As shown in the elapsed times, you can greatly lower query response times by limiting the properties and the number of items that are returned. 可以在 GitHub 的 azure-batch-samples 存储库中查找此项目和其他示例项目。You can find this and other sample projects in the azure-batch-samples repository on GitHub.

BatchMetrics 库和代码示例BatchMetrics library and code sample

除了上述 EfficientListQueries 代码示例外,BatchMetrics 示例项目还演示了如何使用批处理 API 有效地监视 Azure Batch 作业进度。In addition to the EfficientListQueries code sample above, the BatchMetrics sample project demonstrates how to efficiently monitor Azure Batch job progress using the Batch API.

BatchMetrics 示例包括一个可以合并到用户自己的项目中的 .NET 类库项目,以及一个简单的命令行程序,可用于练习和演示库的使用。The BatchMetrics sample includes a .NET class library project which you can incorporate into your own projects, and a simple command-line program to exercise and demonstrate the use of the library.

项目中的该示例应用程序演示了以下操作:The sample application within the project demonstrates the following operations:

  1. 选择特定属性即可只下载所需属性Selecting specific attributes in order to download only the properties you need
  2. 若只下载自上次查询以来发生更改的内容,可针对状态转换时间进行筛选Filtering on state transition times in order to download only changes since the last query

例如,以下方法会显示在 BatchMetrics 库中。For example, the following method appears in the BatchMetrics library. 它会返回 ODATADetailLevel,指定仅针对所查询的实体获取 idstate 属性。It returns an ODATADetailLevel that specifies that only the id and state properties should be obtained for the entities that are queried. 它还指定只应返回其状态自指定 DateTime 参数以来发生更改的实体。It also specifies that only entities whose state has changed since the specified DateTime parameter should be returned.

internal static ODATADetailLevel OnlyChangedAfter(DateTime time)
{
    return new ODATADetailLevel(
        selectClause: "id, state",
        filterClause: string.Format("stateTransitionTime gt DateTime'{0:o}'", time)
    );
}

后续步骤Next steps