用于渲染资产和输出文件的存储与数据移动选项Storage and data movement options for rendering asset and output files

有多个选项可以创建场景和资产文件用于渲染池 VM 上的应用程序:There are multiple options for making the scene and asset files available to the rendering applications on the pool VMs:

  • Azure Blob 存储Azure blob storage:

    • 将场景和资产文件从本地文件系统上传到 Blob 存储。Scene and asset files are uploaded to blob storage from a local file system. 当应用程序由某个任务运行时,会将所需的文件从 Blob 存储复制到 VM,使渲染应用程序能够访问它们。When the application is run by a task, then the required files are copied from blob storage onto the VM so they can be accessed by the rendering application. 输出文件由渲染应用程序写入 VM 磁盘,然后复制到 Blob 存储。The output files are written by the rendering application to the VM disk and then copied to blob storage. 如果需要,可将输出文件从 Blob 存储下载到本地文件系统。If necessary, the output files can be downloaded from blob storage to a local file system.
    • Azure blob 存储是简单且经济高效的选项,适合小型项目。Azure blob storage is a simple and cost-effective option for smaller projects. 由于每个池 VM 上需要所有资产文件,一旦资产文件的数量和大小增加,就需要谨慎确保文件传输尽量高效。As all asset files are required on each pool VM, then once the number and size of asset files increases care needs to be taken to ensure the file transfers are as efficient as possible.
  • 文件系统或文件共享:File system or file share:

    • 根据 VM 操作系统和性能/规模要求,选项包括 Azure 文件、使用附有 NFS 磁盘的 VM、使用附有分布式文件系统(例如 GlusterFS)磁盘的多个 VM,或使用第三方产品/服务。Depending on VM operating system and performance/scale requirements, then options include Azure Files, using a VM with attached disks for NFS, using multiple VMs with attached disks for a distributed file system like GlusterFS, or using a third-party offering.

    • Avere Systems 已由 Microsoft 收购,在不久的将来,将会推出理想的解决方案进行大规模的高性能渲染。Avere Systems is now part of Microsoft and will have solutions in the near future that are ideal for large-scale, high-performance rendering. 借助 Avere 解决方案能够创建可与 Blob 存储或本地 NAS 设备配合工作的基于 Azure 的 NFS 或 SMB 缓存。The Avere solution will enable an Azure-based NFS or SMB cache to be created that works in conjunction with blob storage or with on-premises NAS devices.

    • 使用文件系统可以直接在文件系统中读取或写入文件,或者在文件系统与池 VM 之间复制文件。With a file system, files can be read or written directly to the file system or can be copied between file system and the pool VMs.

    • 使用共享文件系统可以在要使用的项目与作业之间共享大量的资产,而渲染任务只需访问所需的内容。A shared file system allows a large number of assets shared between projects and jobs to be utilized, with rendering tasks only accessing what is required.

使用 Azure Blob 存储Using Azure blob storage

应使用 Blob 存储帐户或常规用途 v2 存储帐户。A blob storage account or a general-purpose v2 storage account should be used. 根据此博客文章中所述,为这两种类型的存储帐户配置的限制可以明显高于常规用途 v1 存储帐户。These two storage account types can be configured with significantly higher limits compared to a general-purpose v1 storage account, as detailed in this blog post. 配置后,更高的限制将会提供更好的性能和可伸缩性,尤其是有多个池 VM 访问存储帐户时。When configured, the higher limits will enable much better performance and scalability, especially when there are many pool VMs accessing the storage account.

在客户端与 Blob 存储之间复制文件Copying files between client and blob storage

若要向/从 Azure 存储复制文件,可以使用各种机制,包括存储 Blob API、Azure 存储数据移动库、适用于 WindowsLinux 的 azcopy 命令行工具、Azure 存储资源管理器Azure Batch ExplorerTo copy files to and from Azure storage, various mechanisms can be used including the storage blob API, the Azure Storage Data Movement Library, the azcopy command-line tool for Windows or Linux, Azure Storage Explorer, and Azure Batch Explorer.

例如,使用 azcopy 可按如下所示转移文件夹中的所有资产:For example, using azcopy, all assets in a folder can be transferred as follows:

azcopy /source:. /dest:https://account.blob.core.chinacloudapi.cn/rendering/project /destsas:"?st=2018-03-30T16%3A26%3A00Z&se=2020-03-31T16%3A26%3A00Z&sp=rwdl&sv=2017-04-17&sr=c&sig=sig" /Y

如果只想复制已修改的文件,可以使用 /XO 参数:To copy only modified files, the /XO parameter can be used:

azcopy /source:. /dest:https://account.blob.core.chinacloudapi.cn/rendering/project /destsas:"?st=2018-03-30T16%3A26%3A00Z&se=2020-03-31T16%3A26%3A00Z&sp=rwdl&sv=2017-04-17&sr=c&sig=sig" /XO /Y

将输入资产文件从 Blob 存储复制到 Batch 池 VMCopying input asset files from blob storage to Batch pool VMs

可通过多种不同的方法复制文件,最佳的方法取决于作业资产的大小。There are a couple of different approaches to copy files with the best approach determined by the size of the job assets. 最简单的方法是将每个作业的所有资产文件复制到池 VM:The simplest approach is to copy all the asset files to the pool VMs for each job:

  • 如果有些文件是某个作业独有的,但作业的所有任务都需要这些文件,则可以指定作业准备任务来复制所有文件。When there are files unique to a job, but are required for all the tasks of a job, then a job preparation task can be specified to copy all the files. 当第一个作业任务已在 VM 上执行,但未针对后续作业任务再次运行时,将运行作业准备任务一次。The job preparation task is run once when the first job task is executed on a VM but is not run again for subsequent job tasks.
  • 应指定一个作业释放任务,以便在完成作业后删除每个作业的文件;这可以避免所有作业资产文件占满 VM 磁盘。A job release task should be specified to remove the per-job files once the job has completed; this will avoid the VM disk getting filled by all the job asset files.
  • 如果有多个作业使用相同的资产,而每个作业的资产只是发生了增量更改,则仍会复制所有资产文件,即使只是更新了一部分文件。When there are multiple jobs using the same assets, with only incremental changes to the assets for each job, then all asset files are still copied, even if only a subset were updated. 存在大量的大型资产文件时,这种方法比较低效。This would be inefficient when there are lots of large asset files.

如果在作业之间重复使用资产文件,而作业之间只是发生了增量更改,则更有效但略微繁琐的方法是在 VM 上的共享文件夹中存储资产,并同步已更改的文件。When asset files are reused between jobs, with only incremental changes between jobs, then a more efficient but slightly more involved approach is to store assets in the shared folder on the VM and sync changed files.

  • 作业准备任务会结合 /XO 参数,使用 azcopy 向 AZ_BATCH_NODE_SHARED_DIR 环境变量指定的 VM 共享文件夹执行复制。The job preparation task would perform the copy using azcopy with the /XO parameter to the VM shared folder specified by AZ_BATCH_NODE_SHARED_DIR environment variable. 这只会将已更改的文件复制到每个 VM。This will only copy changed files to each VM.
  • 必须考虑所有资产的大小,以确保可将其放在池 VM 的临时驱动器上。Thought will have to be given to the size of all assets to ensure they will fit on the temporary drive of the pool VMs.

Azure Batch 内置支持在存储帐户与 Batch 池 VM 之间复制文件。Azure Batch has built-in support to copy files between a storage account and Batch pool VMs. 任务资源文件将文件从存储复制到池 VM。可为作业准备任务指定这些资源文件。Task resource files copy files from storage to pool VMs and could be specified for the job preparation task. 遗憾的是,如果存在数百个文件,有可能会达到限制,因而任务失败。Unfortunately, when there are hundreds of files it is possible to hit a limit and tasks to fail. 如果存在大量的资产,我们建议在作业准备任务中使用 azcopy 命令行,这样就可以使用通配符,且没有限制。When there are large numbers of assets it is recommended to use the azcopy command line in the job preparation task, which can use wildcards and has no limit.

将输出文件从 Batch 池 VM 复制到 Blob 存储Copying output files to blob storage from Batch pool VMs

可以使用输出文件将文件从池 VM 复制到存储。Output files can be used copy files from a pool VM to storage. 任务完成后,可将一个或多个文件从 VM 复制到指定的存储帐户。One or more files can be copied from the VM to a specified storage account once the task has completed. 应复制渲染的输出,但可能还需要存储日志文件。The rendered output should be copied, but it also may be desirable to store log files.

访问文件Accessing files

作业任务使用装载的文件系统指定输入文件和输出文件的路径。Job tasks specify paths for input files and output files using the mounted file system.

将输入资产文件从 Blob 存储复制到 Batch 池 VMCopying input asset files from blob storage to Batch pool VMs

由于文件只是 Azure 存储中的 Blob,因此可以使用标准的 Blob API、工具和 UI(例如,azcopy、存储资源管理器、Batch Explorer 等)在本地文件系统与 Blob 存储之间复制文件。As files are simply blobs in Azure Storage, then standard blob APIs, tools, and UIs can be used to copy files between an on-premises file system and blob storage; for example, azcopy, Storage Explorer, Batch Explorer, etc.

对 Windows VM 使用 Azure 文件Using Azure Files with Windows VMs

Azure 文件在云中提供可通过 SMB 协议访问的完全托管式文件共享。Azure Files offers fully managed file shares in the cloud that are accessible via the SMB protocol. Azure 文件基于 Azure Blob 存储;它非常经济高效,可以使用数据复制将其配置到另一个区域,以实现全局冗余。Azure Files is based on Azure blob storage; it is cost-efficient and can be configured with data replication to another region so globally redundant. 应评审缩放目标,以确定是否应该针对特定的预测池大小和资产文件数量使用 Azure 文件。Scale targets should be reviewed to determine if Azure Files should be used given the forecast pool size and number of asset files.

文档介绍了如何装载 Azure 文件共享。There is documentation covering how to mount an Azure File share.

装载 Azure 文件共享Mounting an Azure Files share

若要在 Batch 中使用,每次运行某个任务时,都需要执行装载操作,因为无法保持任务之间的连接。To use in Batch, a mount operation needs to be performed each time a task in run as it is not possible to persist the connection between tasks. 为实现此目的,最简单的方法是通过池配置中的启动任务,使用 cmdkey 来持久保存凭据,然后在每个任务之前装载共享。The easiest way to do this is to use cmdkey to persist credentials using the start task in the pool configuration, then mount the share before each task.

池模板中 cmdkey 的示例用法(经过转义,以便在 JSON 文件中使用)- 请注意,将 cmdkey 调用与 net use 调用隔开时,启动任务的用户上下文必须与用于运行任务的上下文相同:Example use of cmdkey in a pool template (escaped for use in JSON file) - note that when separating the cmdkey call from the net use call, the user context for the start task must be the same as that used for running the tasks:

"startTask": {
  "commandLine": "cmdkey /add:storageaccountname.file.core.chinacloudapi.cn
    /user:AZURE\\markscuscusbatch /pass:storage_account_key",
  "userIdentity":{
    "autoUser": {
      "elevationLevel": "nonadmin",
      "scope": "pool"
    }
}

示例作业任务命令行:Example job task command line:

"commandLine":"net use S:
  \\\\storageaccountname.file.core.chinacloudapi.cn\\rendering &
3dsmaxcmdio.exe -v:5 -rfw:0 -10 -end:10
  -bitmapPath:\"s:\\3dsMax\\Dragon\\Assets\"
  -outputName:\"s:\\3dsMax\\Dragon\\RenderOutput\\dragon.jpg\"
  -w:1280 -h:720
  \"s:\\3dsMax\\Dragon\\Assets\\Dragon_Character_Rig.max\""

访问文件Accessing files

作业任务使用装载的文件系统指定输入文件和输出文件的路径(使用映射驱动器或 UNC 路径)。Job tasks specify paths for input files and output files using the mounted file system, either using a mapped drive or a UNC path.

将输入资产文件从 Blob 存储复制到 Batch 池 VMCopying input asset files from blob storage to Batch pool VMs

支持 Azure 存储的所有主要 API 和工具(例如 azcopy、Azure CLI、存储资源管理器、Azure PowerShell、Batch Explorer 等)都支持 Azure 文件存储。Azure Files are supported by all the main APIs and tools that have Azure Storage support; for example, azcopy, Azure CLI, Storage Explorer, Azure PowerShell, Batch Explorer, etc.

后续步骤Next steps

有关存储选项的详细信息,请参阅深度文档:For more information about the storage options see the in-depth documentation: