在 Batch 池中装载虚拟文件系统Mount a virtual file system on a Batch pool

Azure Batch 现在支持在 Batch 池中的 Windows 或 Linux 计算节点上装载云存储或外部文件系统。Azure Batch now supports mounting cloud storage or an external file system on Windows or Linux compute nodes in your Batch pools. 当某个计算节点加入池时,将会装载虚拟文件系统并将其视为该节点上的本地驱动器。When a compute node joins a pool, the virtual file system is mounted and treated as a local drive on that node.

本文介绍如何使用适用于 .NET 的 Batch 管理库在计算节点池上装载虚拟文件系统。In this article, you'll learn how to mount a virtual file system on a pool of compute nodes using the Batch Management Library for .NET.

备注

支持在 2019 年 8 月 19 日或之后创建的 Batch 池中装载虚拟文件系统。Mounting a virtual file system is supported on Batch pools created on or after 2019-08-19. 在 2019 年 8 月 19 日之前创建的 Batch 池不支持此功能。Batch pools created prior to 2019-08-19 do not support this feature.

用于在计算节点上装载文件系统的 API 是 Batch .NET 库的一部分。The APIs for mounting file systems on a compute node are part of the Batch .NET library.

在池中装载可获得的优势Benefits of mounting on a pool

将文件系统安装到池中而不是让任务从大型数据集检索自身的数据,可使任务更方便、更高效地访问所需的数据。Mounting the file system to the pool, instead of letting tasks retrieve their own data from a large data set, makes it easier and more efficient for tasks to access the necessary data.

假设有多个任务(例如渲染电影)需要访问一组公用数据。Consider a scenario with multiple tasks requiring access to a common set of data, like rendering a movie. 每个任务每次渲染场景文件中的一个或多个帧。Each task renders one or more frames at a time from the scene files. 如果装载包含场景文件的驱动器,则计算节点可以更方便地访问共享的数据。By mounting a drive that contains the scene files, it's easier for compute nodes to access shared data. 此外,可以根据并发访问数据的计算节点数所需的性能和规模(吞吐量和 IOPS),单独选择和缩放基础文件系统。Additionally, the underlying file system can be chosen and scaled independently based on the performance and scale (throughput and IOPS) required by the number of compute nodes concurrently accessing the data. Blobfuse 仅在 Linux 节点上可用,但是,Azure 文件提供类似的工作流,并可在 Windows 和 Linux 上使用。Blobfuse is only available on Linux nodes, however, Azure Files provides a similar workflow and is available on both Windows and Linux.

在池中装载虚拟文件系统Mount a virtual file system on a pool

在池中装载虚拟文件系统可使池中的每个计算节点都能使用该文件系统。Mounting a virtual file system on a pool makes the file system available to every compute node in the pool. 当计算节点加入池,或者节点重启或重置映像时,将配置该文件系统。The file system is configured either when a compute node joins a pool, or when the node is restarted or reimaged.

若要在池中装载文件系统,请创建一个 MountConfiguration 对象。To mount a file system on a pool, create a MountConfiguration object. 选择适合虚拟文件系统的对象:AzureBlobFileSystemConfigurationAzureFileShareConfigurationNfsMountConfigurationCifsMountConfigurationChoose the object that fits your virtual file system: AzureBlobFileSystemConfiguration, AzureFileShareConfiguration, NfsMountConfiguration, or CifsMountConfiguration.

所有装载配置对象都需要以下基本参数。All mount configuration objects need the following base parameters. 某些装载配置具有特定于所用文件系统的参数,代码示例中将更详细地介绍这些参数。Some mount configurations have parameters specific to the file system being used, which are discussed in more detail in the code examples.

  • 帐户名或源:若要装载虚拟文件共享,需要获取存储帐户或其源的名称。Account name or source: To mount a virtual file share, you need the name of the storage account or its source.
  • 相对装载路径或源:在计算节点上装载的文件系统的位置,相对于节点上可通过 AZ_BATCH_NODE_MOUNTS_DIR 访问的标准 fsmounts 目录。Relative mount path or Source: The location of the file system mounted on the compute node, relative to standard fsmounts directory accessible on the node via AZ_BATCH_NODE_MOUNTS_DIR. 确切的位置根据节点上使用的操作系统而异。The exact location varies depending on the operating system used on the node. 例如,Ubuntu 节点上的物理位置将映射到 mnt\batch\tasks\fsmounts,而 CentOS 节点上的物理位置将映射到 mnt\resources\batch\tasks\fsmountsFor example, the physical location on an Ubuntu node is mapped to mnt\batch\tasks\fsmounts, and on a CentOS node it is mapped to mnt\resources\batch\tasks\fsmounts.
  • 装载选项或 blobfuse 选项:这些选项描述用于装载文件系统的特定参数。Mount options or blobfuse options: These options describe specific parameters for mounting a file system.

创建 MountConfiguration 对象后,在创建池时将该对象分配到 MountConfigurationList 属性。Once the MountConfiguration object is created, assign the object to the MountConfigurationList property when you create the pool. 当某个节点加入池,或者节点重启或重置映像时,将会装载该文件系统。The file system is mounted either when a node joins a pool or when the node is restarted or reimaged.

装载文件系统时,会创建一个指向已装载文件系统的位置的环境变量 AZ_BATCH_NODE_MOUNTS_DIR,并创建可用于故障排除和调试的日志文件。When the file system is mounted, an environment variable AZ_BATCH_NODE_MOUNTS_DIR is created which points to the location of the mounted file systems as well as log files, which are useful for troubleshooting and debugging. 诊断装载错误部分更详细地介绍了日志文件。Log files are explained in more detail in the Diagnose mount errors section.

重要

可在一个池中装载的最大文件系统数目为 10 个。The maximum number of mounted file systems on a pool is 10. 有关详细信息和其他限制,请参阅 Batch 服务配额和限制See Batch service quotas and limits for details and other limits.

示例Examples

以下代码示例演示如何将各种文件共享装载到计算节点池。The following code examples demonstrate mounting a variety of file shares to a pool of compute nodes.

Azure 文件共享Azure Files share

Azure 文件是标准的 Azure 云文件系统产品。Azure Files is the standard Azure cloud file system offering. 若要详细了解如何获取装载配置代码示例中的任何参数,请参阅使用 Azure 文件共享To learn more about how to get any of the parameters in the mount configuration code sample, see Use an Azure Files share.

new PoolAddParameter
{
    Id = poolId,
    MountConfiguration = new[]
    {
        new MountConfiguration
        {
            AzureFileShareConfiguration = new AzureFileShareConfiguration
            {
                AccountName = "AccountName",
                AzureFileUrl = "AzureFileShareUrl",
                AccountKey = "StorageAccountKey",
                RelativeMountPath = "RelativeMountPath",
                MountOptions = "-o vers=3.0,dir_mode=0777,file_mode=0777,sec=ntlmssp"
            },
        }
    }
}

Azure Blob 文件系统Azure Blob file system

装载 Blob 文件系统需要使用存储帐户的 AccountKeySasKeyMounting a blob file system requires an AccountKey or SasKey for your storage account. 有关获取这些密钥的信息,请参阅查看帐户密钥使用共享访问签名 (SAS)For information on getting these keys, see View account keys, or Using shared access signatures (SAS). 有关使用 blobfuse 的详细信息,请参阅 blobfuse 故障排除常见问题解答For more information on using blobfuse, see the blobfuse Troubleshoot FAQ. 若要获取对 blobfuse 装载目录的默认访问权限,请以管理员身份运行该任务。To get default access to the blobfuse mounted directory, run the task as an Administrator. blobfuse 在用户空间中装载目录;创建池时,将以 root 身份装载 blobfuse。Blobfuse mounts the directory at the user space, and at pool creation it is mounted as root. 在 Linux 中,所有管理员任务都以 root 身份运行。In Linux all Administrator tasks are root. FUSE 参考页中介绍了 FUSE 模块的所有选项。All options for the FUSE module is described in the FUSE reference page.

除了故障排除指南以外,还可以借助“blobfuse 中的问题”GitHub 存储库来检查当前的 blobfuse 问题并获取解决方法。In addition to the troubleshooting guide, GitHub issues in the blobfuse repository are a helpful way to check on current blobfuse issues and resolutions. 有关详细信息,请参阅 blobfuse 问题For more information, see blobfuse issues.

new PoolAddParameter
{
    Id = poolId,
    MountConfiguration = new[]
    {
        new MountConfiguration
        {
            AzureBlobFileSystemConfiguration = new AzureBlobFileSystemConfiguration
            {
                AccountName = "StorageAccountName",
                ContainerName = "containerName",
                AccountKey = "StorageAccountKey",
                SasKey = "",
                RelativeMountPath = "RelativeMountPath",
                BlobfuseOptions = "-o attr_timeout=240 -o entry_timeout=240 -o negative_timeout=120 "
            },
        }
    }
}

网络文件系统Network File System

还可以将网络文件系统 (NFS) 装载到池节点,使 Azure Batch 节点能够轻松访问传统的文件系统。Network File Systems (NFS) can also be mounted to pool nodes allowing traditional file systems to be easily accessed by Azure Batch nodes. 此类文件系统可能是部署在云中的单个 NFS 服务器,或通过虚拟网络访问的本地 NFS 服务器。This could be a single NFS server deployed in the cloud, or an on-premises NFS server accessed over a virtual network.

new PoolAddParameter
{
    Id = poolId,
    MountConfiguration = new[]
    {
        new MountConfiguration
        {
            NfsMountConfiguration = new NFSMountConfiguration
            {
                Source = "source",
                RelativeMountPath = "RelativeMountPath",
                MountOptions = "options ver=1.0"
            },
        }
    }
}

通用 Internet 文件系统Common Internet File System

还可以将通用 Internet 文件系统 (CIFS) 装载到池节点,使 Azure Batch 节点能够轻松访问传统的文件系统。Common Internet File Systems (CIFS) can also be mounted to pool nodes allowing traditional file systems to be easily accessed by Azure Batch nodes. CIFS 是一种文件共享协议,提供开放的跨平台机制用于请求网络服务器文件和服务。CIFS is a file-sharing protocol that provides an open and cross-platform mechanism for requesting network server files and services. CIFS 基于用于 Internet 和 Intranet 文件共享的 Microsoft 服务器消息块 (SMB) 协议增强版,用于在 Windows 节点上装载外部文件系统。CIFS is based on the enhanced version of Microsoft's Server Message Block (SMB) protocol for internet and intranet file sharing and is used to mount external file systems on Windows nodes. 若要详细了解 SMB,请参阅文件服务器和 SMBTo learn more about SMB, see File Server and SMB.

new PoolAddParameter
{
    Id = poolId,
    MountConfiguration = new[]
    {
        new MountConfiguration
        {
            CifsMountConfiguration = new CIFSMountConfiguration
            {
                Username = "StorageAccountName",
                RelativeMountPath = "cifsmountpoint",
                Source = "source",
                Password = "StorageAccountKey",
                MountOptions = "-o vers=3.0,dir_mode=0777,file_mode=0777,serverino"
            },
        }
    }
}

诊断装载错误Diagnose mount errors

如果某个装载配置失败,则池中的计算节点也会失败,并且节点状态将变为不可用。If a mount configuration fails, the compute node in the pool will fail and the node state becomes unusable. 若要诊断装载配置失败,请检查 ComputeNodeError 属性以了解有关错误的详细信息。To diagnose a mount configuration failure, inspect the ComputeNodeError property for details on the error.

若要获取日志文件进行调试,请使用 OutputFiles 上传 *.log 文件。To get the log files for debugging, use OutputFiles to upload the *.log files. *.log 文件包含有关位于 AZ_BATCH_NODE_MOUNTS_DIR 处的文件系统装入点的信息。The *.log files contain information about the file system mount at the AZ_BATCH_NODE_MOUNTS_DIR location. 每个装入点的装载日志文件格式为 <type>-<mountDirOrDrive>.logMount log files have the format: <type>-<mountDirOrDrive>.log for each mount. 例如,位于名为 test 的装载目录中的 cifs 装入点将包含名为 cifs-test.log 的装载日志文件。For example, a cifs mount at a mount directory named test will have a mount log file named: cifs-test.log.

支持的 SKUSupported SKUs

发布者Publisher 产品/服务Offer SKUSKU Azure 文件共享Azure Files Share BlobfuseBlobfuse NFS 装载NFS mount CIFS 装载CIFS mount
或批处理batch rendering-centos73rendering-centos73 呈现rendering ✔️
注意:与 CentOS 7.7 兼容Note: Compatible with CentOS 7.7
✔️ ✔️ ✔️
CanonicalCanonical UbuntuServerUbuntuServer 16.04-LTS、18.04-LTS16.04-LTS, 18.04-LTS ✔️ ✔️ ✔️ ✔️
CredativCredativ DebianDebian 88 ✔️ ✔️ ✔️
CredativCredativ DebianDebian 99 ✔️ ✔️ ✔️ ✔️
microsoft-adsmicrosoft-ads linux-data-science-vmlinux-data-science-vm linuxdsvmlinuxdsvm ✔️
注意:与 CentOS 7.4 兼容Note: Compatible with CentOS 7.4.
✔️ ✔️ ✔️
microsoft-azure-batchmicrosoft-azure-batch centos-containercentos-container 7.67.6 ✔️ ✔️ ✔️ ✔️
microsoft-azure-batchmicrosoft-azure-batch centos-container-rdmacentos-container-rdma 7.47.4 ✔️
注意:支持 A_8 或 A_9 存储Note: Supports A_8 or 9 storage
✔️ ✔️ ✔️
microsoft-azure-batchmicrosoft-azure-batch ubuntu-server-containerubuntu-server-container 16.04-LTS16.04-LTS ✔️ ✔️ ✔️ ✔️
microsoft-dsvmmicrosoft-dsvm linux-data-science-vm-ubuntulinux-data-science-vm-ubuntu linuxdsvmubuntulinuxdsvmubuntu ✔️ ✔️ ✔️ ✔️
OpenLogicOpenLogic CentOSCentOS 7.67.6 ✔️ ✔️ ✔️ ✔️
OpenLogicOpenLogic CentOS-HPCCentOS-HPC 7.4、7.3、7.17.4, 7.3, 7.1 ✔️ ✔️ ✔️ ✔️
OracleOracle Oracle-LinuxOracle-Linux 7.67.6
WindowsWindows WindowsServerWindowsServer 2012、2016、20192012, 2016, 2019 ✔️

后续步骤Next steps