使用 Batch 服务 API 将任务数据保存到 Azure 存储Persist task data to Azure Storage with the Batch service API

在 Azure Batch 中运行的任务可能在其运行时产生输出数据。A task running in Azure Batch may produce output data when it runs. 通常需要存储任务输出数据,以便作业中的其他任务和/或执行该作业的客户端应用程序进行检索。Task output data often needs to be stored for retrieval by other tasks in the job, the client application that executed the job, or both. 任务可向 Batch 计算节点的文件系统写入输出数据,但当重置节点映像或节点离开池时,节点上的所有数据都会丢失。Tasks write output data to the file system of a Batch compute node, but all data on the node is lost when it is reimaged or when the node leaves the pool. 任务可能还具有文件保留期,超过此保留期后,系统将删除该任务创建的文件。Tasks may also have a file retention period, after which files created by the task are deleted. 出于这些原因,请务必将以后需要的任务输出保留到数据存储,如 Azure 存储For these reasons, it's important to persist task output that you'll need later to a data store such as Azure Storage.

有关 Batch 中的存储帐户选项,请参阅 Batch 功能概述For storage account options in Batch, see the Batch feature overview.

Batch 服务 API 支持将具有虚拟机配置的池上运行的任务和作业管理器任务的输出数据保存到 Azure 存储。The Batch service API supports persisting output data to Azure Storage for tasks and job manager tasks that run on pools with the virtual machine configuration. 添加任务时,可将 Azure 存储中的某个容器指定为该任务的输出目标。When you add a task, you can specify a container in Azure Storage as the destination for the task's output. 然后,在完成该任务时,Batch 服务会将所有输出数据写入该容器。The Batch service then writes any output data to that container when the task is complete.

使用 Batch 服务 API 保存任务输出的优势之一是不需要修改任务运行的应用程序。An advantage to using the Batch service API to persist task output is that you do not need to modify the application that the task is running. 只需对客户端应用程序进行几处修改,即可从创建任务的同一代码内部保存任务的输出。Instead, with a few modifications to your client application, you can persist the task's output from within the same code that creates the task.

重要

使用 Batch 服务 API 将任务数据持久保存到 Azure 存储的功能不适用于在2018 年 2 月 1 日之前创建的池。Persisting task data to Azure Storage with the Batch service API does not work with pools created before February 1, 2018.

何时使用 Batch 服务 API 保存任务输出?When do I use the Batch service API to persist task output?

Azure Batch 提供多种方式来保存任务输出。Azure Batch provides more than one way to persist task output. 使用 Batch 服务 API 是一种便捷的方法,最适合以下情况:Using the Batch service API is a convenient approach that's best suited to these scenarios:

  • 希望编写代码来从客户端应用程序内部保存任务输出,而无需修改任务运行的应用程序。You want to write code to persist task output from within your client application, without modifying the application that your task is running.
  • 希望保存使用虚拟机配置创建的池中的 Batch 任务和作业管理器任务的输出。You want to persist output from Batch tasks and job manager tasks in pools created with the virtual machine configuration.
  • 希望将输出保存到具有任意名称的 Azure 存储容器。You want to persist output to an Azure Storage container with an arbitrary name.
  • 希望将输出保存到按照 Batch 文件约定标准命名的 Azure 存储容器。You want to persist output to an Azure Storage container named according to the Batch File Conventions standard.

如果你的情况与上面不同,可能需要考虑不同的方法。If your scenario differs from those listed above, you may need to consider a different approach. 例如,在运行任务时,Batch 服务 API 目前不支持将输出流式传输到 Azure 存储。For example, the Batch service API does not currently support streaming output to Azure Storage while the task is running. 若要流式传输输出,请考虑使用适用于 .NET 的 Batch 文件约定库。To stream output, consider using the Batch File Conventions library, available for .NET. 对于其他语言,需要实现自己的解决方案。For other languages, you'll need to implement your own solution. 若要了解持久保存任务输出的其他选项,请参阅将作业和任务输出持久保存到 Azure 存储For information on other options for persisting task output, see Persist job and task output to Azure Storage.

在 Azure 存储中创建容器Create a container in Azure Storage

若要将任务输出保存到 Azure 存储,需要创建一个充当输出文件目标的容器。To persist task output to Azure Storage, you'll need to create a container that serves as the destination for your output files. 请在运行任务之前创建容器,最好是在提交作业之前创建。Create the container before you run your task, preferably before you submit your job. 若要创建容器,请使用适当的 Azure 存储客户端库或 SDK。To create the container, use the appropriate Azure Storage client library or SDK. 有关 Azure 存储 API 的详细信息,请参阅 Azure 存储文档For more information about Azure Storage APIs, see the Azure Storage documentation.

例如,在以 C# 编写应用程序时,可以使用适用于 .NET 的 Azure 存储客户端库For example, if you are writing your application in C#, use the Azure Storage client library for .NET. 以下示例演示如何创建容器:The following example shows how to create a container:

CloudBlobContainer container = storageAccount.CreateCloudBlobClient().GetContainerReference(containerName);
await container.CreateIfNotExists();

获取容器的共享访问签名Get a shared access signature for the container

创建容器后,请获取对容器拥有写入访问权限的共享访问签名 (SAS)。After you create the container, get a shared access signature (SAS) with write access to the container. SAS 提供对容器的委派访问权限。A SAS provides delegated access to the container. SAS 使用一组指定的权限授予在指定时间间隔内的访问权限。The SAS grants access with a specified set of permissions and over a specified time interval. Batch 服务需要一个拥有写入权限的 SAS 才能将任务输出写入到容器。The Batch service needs a SAS with write permissions to write task output to the container. 有关 SAS 的详细信息,请参阅在 Azure 存储中使用共享访问签名 (SAS)For more information about SAS, see Using shared access signatures (SAS) in Azure Storage.

使用 Azure 存储 API 获取 SAS 时,API 将返回 SAS 令牌字符串。When you get a SAS using the Azure Storage APIs, the API returns a SAS token string. 此令牌字符串包含 SAS 的所有参数,其中包括权限以及 SAS 的生效间隔。This token string includes all parameters of the SAS, including the permissions and the interval over which the SAS is valid. 若要使用 SAS 访问 Azure 存储中的容器,需要将 SAS 令牌字符串追加到资源 URI。To use the SAS to access a container in Azure Storage, you need to append the SAS token string to the resource URI. 资源 URI 连同追加的 SAS 令牌中提供对 Azure 存储的经过身份验证的访问权限。The resource URI, together with the appended SAS token, provides authenticated access to Azure Storage.

以下示例演示如何获取容器的只写 SAS 令牌字符串,然后将 SAS 追加到容器 URI:The following example shows how to get a write-only SAS token string for the container, then appends the SAS to the container URI:

string containerSasToken = container.GetSharedAccessSignature(new SharedAccessBlobPolicy()
{
    SharedAccessExpiryTime = DateTimeOffset.UtcNow.AddDays(1),
    Permissions = SharedAccessBlobPermissions.Write
});

string containerSasUrl = container.Uri.AbsoluteUri + containerSasToken;

指定任务输出的输出文件Specify output files for task output

若要指定任务的输出文件,请创建 OutputFile 对象的集合,并在创建任务时将该集合分配到 CloudTask.OutputFiles 属性。To specify output files for a task, create a collection of OutputFile objects and assign it to the CloudTask.OutputFiles property when you create the task.

以下 C# 代码示例创建可将随机数写入名为 output.txt 的文件的任务。The following C# code example creates a task that writes random numbers to a file named output.txt. 该示例为要写入容器的 output.txt 创建输出文件。The example creates an output file for output.txt to be written to the container. 该示例还为匹配文件模式 std*.txt(例如 stdout.txtstderr.txt)的所有日志文件创建输出文件。The example also creates output files for any log files that match the file pattern std*.txt (e.g., stdout.txt and stderr.txt). 容器 URL 需要先前为容器创建的 SAS。The container URL requires the SAS that was created previously for the container. Batch 服务使用 SAS 对容器的访问进行身份验证:The Batch service uses the SAS to authenticate access to the container:

new CloudTask(taskId, "cmd /v:ON /c \"echo off && set && (FOR /L %i IN (1,1,100000) DO (ECHO !RANDOM!)) > output.txt\"")
{
    OutputFiles = new List<OutputFile>
    {
        new OutputFile(
            filePattern: @"..\std*.txt",
            destination: new OutputFileDestination(
         new OutputFileBlobContainerDestination(
                    containerUrl: containerSasUrl,
                    path: taskId)),
            uploadOptions: new OutputFileUploadOptions(
            uploadCondition: OutputFileUploadCondition.TaskCompletion)),
        new OutputFile(
            filePattern: @"output.txt",
            destination: 
         new OutputFileDestination(new OutputFileBlobContainerDestination(
                    containerUrl: containerSasUrl,
                    path: taskId + @"\output.txt")),
            uploadOptions: new OutputFileUploadOptions(
            uploadCondition: OutputFileUploadCondition.TaskCompletion)),
}

备注

如果在 Linux 中使用此示例,请务必将反斜杠更改为正斜杠。If using this example with Linux, be sure to change the backslashes to forward slashes.

指定用于匹配的文件模式Specify a file pattern for matching

指定输出文件时,可以使用 OutputFile.FilePattern 属性指定用于匹配的文件模式。When you specify an output file, you can use the OutputFile.FilePattern property to specify a file pattern for matching. 文件模式可以匹配零个文件、一个文件或者任务创建的一组文件。The file pattern may match zero files, a single file, or a set of files that are created by the task.

FilePattern 属性支持标准文件系统通配符,例如 *(表示非递归匹配)和 **(表示递归匹配)。The FilePattern property supports standard filesystem wildcards such as * (for non-recursive matches) and ** (for recursive matches). 例如,上述代码示例指定要以非递归方式匹配 std*.txt 的文件模式:For example, the code sample above specifies the file pattern to match std*.txt non-recursively:

filePattern: @"..\std*.txt"

若要上传单个文件,请指定不带通配符的文件模式。To upload a single file, specify a file pattern with no wildcards. 例如,上述代码示例指定用于匹配 output.txt 的文件模式:For example, the code sample above specifies the file pattern to match output.txt:

filePattern: @"output.txt"

指定上传条件Specify an upload condition

OutputFileUploadOptions.UploadCondition 属性允许按条件上传输出文件。The OutputFileUploadOptions.UploadCondition property permits conditional uploading of output files. 一种常见方案是任务成功时上传一组文件,任务失败时上传另一组文件。A common scenario is to upload one set of files if the task succeeds, and a different set of files if it fails. 例如,你可能希望仅当任务失败并使用非零退出代码退出时,才上传详细日志文件。For example, you may want to upload verbose log files only when the task fails and exits with a nonzero exit code. 同样,你可能希望仅当任务成功时才上传结果文件,因为任务失败时这些文件可能会丢失或不完整。Similarly, you may want to upload result files only if the task succeeds, as those files may be missing or incomplete if the task fails.

上述代码示例将 UploadCondition 属性设置为 TaskCompletionThe code sample above sets the UploadCondition property to TaskCompletion. 此设置指定在任务完成后上传文件,不管退出代码的值是什么。This setting specifies that the file is to be uploaded after the tasks completes, regardless of the value of the exit code.

uploadCondition: OutputFileUploadCondition.TaskCompletion

有关其他设置,请参阅 OutputFileUploadCondition 枚举。For other settings, see the OutputFileUploadCondition enum.

区分同名的文件Disambiguate files with the same name

作业中的任务可能会生成同名的文件。The tasks in a job may produce files that have the same name. 例如,对于作业中运行的每个任务,都会创建 stdout.txtstderr.txtFor example, stdout.txt and stderr.txt are created for every task that runs in a job. 由于每个任务在其自身的上下文中运行,因此这些文件在节点的文件系统中不会冲突。Because each task runs in its own context, these files don't conflict on the node's file system. 但是,在将多个任务生成的文件上传到共享容器时,需要区分同名的文件。However, when you upload files from multiple tasks to a shared container, you'll need to disambiguate files with the same name.

OutputFileBlobContainerDestination.Path 属性指定输出文件的目标 Blob 或虚拟目录。The OutputFileBlobContainerDestination.Path property specifies the destination blob or virtual directory for output files. 可以使用 Path 属性来命名 Blob 或虚拟目录,使得同名的输出文件在 Azure 存储中具有唯一的名称。You can use the Path property to name the blob or virtual directory in such a way that output files with the same name are uniquely named in Azure Storage. 在路径中使用任务 ID 是一种很好的做法,可以确保名称唯一,同时轻松识别文件。Using the task ID in the path is a good way to ensure unique names and easily identify files.

如果 FilePattern 属性设置为通配符表达式,则与模式匹配的所有文件将上传到 Path 属性指定的虚拟目录。If the FilePattern property is set to a wildcard expression, then all files that match the pattern are uploaded to the virtual directory specified by the Path property. 例如,如果容器为 mycontainer,任务 ID 为 mytask,文件模式为 ..\std*.txt,则输出文件在 Azure 存储中的绝对 URI 将类似于:For example, if the container is mycontainer, the task ID is mytask, and the file pattern is ..\std*.txt, then the absolute URIs to the output files in Azure Storage will be similar to:

https://myaccount.blob.core.chinacloudapi.cn/mycontainer/mytask/stderr.txt
https://myaccount.blob.core.chinacloudapi.cn/mycontainer/mytask/stdout.txt

如果 FilePattern 属性设置为匹配单个文件名(意味着不包含任何通配符),则 Path 属性的值将指定完全限定的 Blob 名称。If the FilePattern property is set to match a single file name, meaning it does not contain any wildcard characters, then the value of the Path property specifies the fully qualified blob name. 如果预期命名会与多个任务生成的单个文件相冲突,在文件名中包含虚拟目录的名称即可区分这些文件。If you anticipate naming conflicts with a single file from multiple tasks, then include the name of the virtual directory as part of the file name to disambiguate those files. 例如,将 Path 属性设置为包含任务 ID、分隔符(通常是正斜杠)和文件名:For example, set the Path property to include the task ID, the delimiter character (typically a forward slash), and the file name:

path: taskId + @"/output.txt"

一组任务的输出文件的绝对 URI 类似于:The absolute URIs to the output files for a set of tasks will be similar to:

https://myaccount.blob.core.chinacloudapi.cn/mycontainer/task1/output.txt
https://myaccount.blob.core.chinacloudapi.cn/mycontainer/task2/output.txt

有关 Azure 存储中的虚拟目录的详细信息,请参阅列出容器中的 BlobFor more information about virtual directories in Azure Storage, see List the blobs in a container.

诊断文件上传错误Diagnose file upload errors

如果将输出文件上传到 Azure 存储失败,该任务将转为 Completed 状态,同时系统会设置 TaskExecutionInformation.FailureInformation 属性。If uploading output files to Azure Storage fails, then the task moves to the Completed state and the TaskExecutionInformation.FailureInformation property is set. 检查 FailureInformation 属性可确定发生了哪种错误。Examine the FailureInformation property to determine what error occurred. 例如,下面是在找不到容器的情况下,上传文件时发生的错误:For example, here is an error that occurs on file upload if the container cannot be found:

Category: UserError
Code: FileUploadContainerNotFound
Message: One of the specified Azure container(s) was not found while attempting to upload an output file

每次上传文件时,Batch 都会将以下两个日志文件写入计算节点:fileuploadout.txtfileuploaderr.txtOn every file upload, Batch writes two log files to the compute node, fileuploadout.txt and fileuploaderr.txt. 可以检查这些日志文件来详细了解具体的失败情况。You can examine these log files to learn more about a specific failure. 如果从未尝试上传文件(例如,因为任务本身无法运行),则这些日志文件不会存在。In cases where the file upload was never attempted, for example because the task itself couldn't run, then these log files will not exist.

诊断文件上传的性能Diagnose file upload performance

fileuploadout.txt 文件记录了上传进度。The fileuploadout.txt file logs upload progress. 可以检查此文件来详细了解上传文件所花费的时间。You can examine this file to learn more about how long your file uploads are taking. 请记住,有许多因素会影响上传性能,包括节点大小、上传时节点上的其他活动、目标容器是否在 Batch 池所在的同一区域中、将多少个节点同时上传到存储帐户,等等。Keep in mind that there are many contributing factors to upload performance, including the size of the node, other activity on the node at the time of the upload, whether the target container is in the same region as the Batch pool, how many nodes are uploading to the storage account at the same time, and so on.

结合 Batch 文件约定标准使用 Batch 服务 APIUse the Batch service API with the Batch File Conventions standard

使用 Batch 服务 API 保存任务输出时,可以根据喜好为目标容器和 Blob 命名。When you persist task output with the Batch service API, you can name your destination container and blobs however you like. 也可以选择根据 Batch 文件约定标准为其命名。You can also choose to name them according to the Batch File Conventions standard. 文件约定标准根据作业和任务的名称确定 Azure 存储中用于给定输出文件的目标容器和 Blob 的名称。The File Conventions standard determines the names of the destination container and blob in Azure Storage for a given output file based on the names of the job and task. 如果使用文件约定标准为输出文件命名,则可以在 Azure 门户中查看输出文件。If you do use the File Conventions standard for naming output files, then your output files are available for viewing in the Azure portal.

如果使用 C# 进行开发,则可以使用适用于 .NET 的 Batch 文件约定库中内置的方法。If you are developing in C#, you can use the methods built into the Batch File Conventions library for .NET. 此库将为你创建适当命名的容器和 Blob 路径。This library creates the properly named containers and blob paths for you. 例如,可以调用 API 根据作业名称来获取容器的正确名称:For example, you can call the API to get the correct name for the container, based on the job name:

string containerName = job.OutputStorageContainerName();

可以使用CloudJobExtensions.GetOutputStorageContainerUrl 方法返回用于写入容器的共享访问签名 (SAS) URL。You can use the CloudJobExtensions.GetOutputStorageContainerUrl method to return a shared access signature (SAS) URL that is used to write to the container. 然后,可将此 SAS 传递给 OutputFileBlobContainerDestination 构造函数。You can then pass this SAS to the OutputFileBlobContainerDestination constructor.

如果使用除 C# 以外的语言进行开发,则需要自行实现文件约定标准。If you are developing in a language other than C#, you will need to implement the File Conventions standard yourself.

代码示例Code sample

PersistOutputs 示例项目是 GitHub 上的 Azure Batch 代码示例之一。The PersistOutputs sample project is one of the Azure Batch code samples on GitHub. 此 Visual Studio 解决方案演示如何使用适用于 .NET 的 Batch 客户端库将任务输出保存到持久性存储。This Visual Studio solution demonstrates how to use the Batch client library for .NET to persist task output to durable storage. 若要运行该示例,请遵循以下步骤:To run the sample, follow these steps:

  1. 在 Visual Studio 2019 中打开项目。Open the project in Visual Studio 2019.
  2. 将 Batch 和存储帐户凭据添加到 Microsoft.Azure.Batch.Samples.Common 项目中的 AccountSettings.settingsAdd your Batch and Storage account credentials to AccountSettings.settings in the Microsoft.Azure.Batch.Samples.Common project.
  3. 生成(但不要运行)该解决方案。Build (but do not run) the solution. 根据提示还原所有 NuGet 包。Restore any NuGet packages if prompted.
  4. 使用 Azure 门户上传 PersistOutputsTask应用程序包Use the Azure portal to upload an application package for PersistOutputsTask. 在 .zip 包中包含 PersistOutputsTask.exe 及其依赖程序集,将应用程序 ID 设置为“PersistOutputsTask”,将应用程序包版本设置为“1.0”。Include the PersistOutputsTask.exe and its dependent assemblies in the .zip package, set the application ID to "PersistOutputsTask", and the application package version to "1.0".
  5. 启动(运行)PersistOutputs 项目。Start (run) the PersistOutputs project.
  6. 当系统提示选择用于运行示例的持久性技术时,请输入 2,以运行使用 Batch 服务 API 保存任务输出的示例。When prompted to choose the persistence technology to use for running the sample, enter 2 to run the sample using the Batch service API to persist task output.
  7. 如果需要,请输入 3 再次运行示例,以使用 Batch 服务 API 保存输出,并根据文件约定标准将目标容器和 Blob 路径命名。If desired, run the sample again, entering 3 to persist output with the Batch service API, and also to name the destination container and blob path according to the File Conventions standard.

后续步骤Next steps