Configure HPC storage on Azure

When migrating HPC environments to the cloud, it's essential to define and implement an effective storage strategy that meets your performance, scalability, and cost requirements. An effective storage strategy ensures that your HPC workloads can access and process data efficiently, securely, and reliably. This approach includes considering different types of storage solutions for various needs such as long-term data archiving, high-performance scratch space, and shared storage for collaborative work.

Proper data management practices, such as lifecycle policies and access controls, help maintain the integrity and security of your data. Additionally, efficient data movement techniques are necessary to handle large-scale data transfers and automate ETL processes to streamline workflows. Here are the key steps and considerations for setting up storage in the cloud:

Define storage needs

  • Storage types:

    • Long-term storage: Use Azure Blob Storage for data archiving. Azure Blob Storage provides a cost-effective solution for storing large volumes of data that are infrequently accessed but must be retained for compliance or historical purposes. It offers various access tiers (Hot, Cool, and Archive) to optimize costs based on how frequently the data is accessed.

    • Shared storage: Use Azure Files or NFS on Blob for user home directories and shared data. Azure Files offers fully managed file shares in the cloud that can be accessed via the industry-standard SMB protocol, making it easy for multiple users and applications to share data.

      NFS on Blob allows for POSIX-compliant shared access to Azure Blob Storage, enabling seamless integration with existing HPC workflows and applications.

  • Data management:

    • Implement data lifecycle policies: To manage data movement between hot, cool, and archive tiers, implement data lifecycle policies that automatically move data to the most appropriate storage tier based on usage patterns. This approach helps optimize storage costs by ensuring that frequently accessed data is kept in high-performance storage, while rarely accessed data is moved to more cost-effective archival storage.
    • Set up access controls: Use Azure Active Directory (AD) and role-based access control (RBAC) to set up granular access controls for your storage resources. Azure AD provides identity and access management capabilities, while RBAC allows you to assign specific permissions to users and groups based on their roles. This strategy ensures that only authorized users can access sensitive data, enhancing security and compliance.

  • Data movement:

    • Azure Data Box: Use Azure Data Box for large-scale offline data transfers. Azure Data Box is a secure, ruggedized appliance that allows you to transfer large amounts of data to Azure quickly and safely, minimizing the time and cost associated with network-based data transfer.
    • Azure Data Factory: Use Azure Data Factory for orchestrating and automating data movement and transformation. Azure Data Factory provides a fully managed ETL service that allows you to move data between on-premises and cloud storage solutions, schedule data workflows, and transform data as needed.
    • AzCopy: Use AzCopy for command-line data transfer. AzCopy is a command-line utility that provides high-performance, reliable data transfer between on-premises storage and Azure Blob Storage, Azure Files, and Azure Table Storage. It supports both synchronous and asynchronous transfer modes, making it suitable for various data movement scenarios.

Tools and services

  • Azure Blob Storage:

    • Use Azure Blob Storage for cost-effective long-term data archiving.
    • Implement data lifecycle policies to automatically move data between access tiers (Hot, Cool, Archive).
    • Set up access controls and integrate with data analytics services for efficient data management.

  • Azure Files:

    • Use Azure Files for fully managed file shares accessible via SMB protocol.
    • Configure Azure AD and RBAC for secure access management and compliance.
    • Ensure high availability with options for geo-redundant storage to protect against regional failures.

Best practices for HPC storage

  • Define clear storage requirements:

    • Identify the specific storage needs for different workloads, such as high-performance scratch space, long-term archiving, and shared storage.
    • Choose the appropriate storage solutions (for example, Azure Blob Storage) based on performance, scalability, and cost requirements.

  • Implement data lifecycle management:

    • Set up automated lifecycle policies to manage data movement between different storage tiers (Hot, Cool, Archive) to optimize costs and performance.
    • Regularly review and adjust lifecycle policies to ensure data is stored in the most cost-effective and performance-appropriate tier.

  • Ensure data security and compliance:

    • Use Azure Active Directory (AD) and role-based access control (RBAC) to enforce granular access controls on storage resources.
    • Implement encryption for data at rest and in transit to meet security and compliance requirements.

  • Optimize data movement:

    • Utilize tools like Azure Data Box for large-scale offline data transfers and AzCopy or rsync for efficient online data transfers.
    • Monitor and optimize data transfer processes to minimize downtime and ensure data integrity during migration.

  • Monitor and manage storage performance:

    • Continuously monitor storage performance and usage metrics to identify and address bottlenecks.
    • Use Azure Monitor and Azure Metrics to gain insights into storage performance and capacity utilization, and make necessary adjustments to meet workload demands.

These best practices ensure that your HPC storage strategy is effective, cost-efficient, and capable of meeting the performance and scalability requirements of your workloads.

Example steps for storage setup and deployment

This section provides detailed instructions for setting up various storage solutions for HPC in the cloud. It covers the deployment and configuration of NFS on Azure Blob, and Azure Files, including how to deploy these services and configure mount points on HPC nodes.

  1. Implementing NFS on Azure Blob:

    • Create an Azure Storage account:
      • Navigate to the Azure portal and create a new storage account.
      • Enable NFS v3 support during the creation process by selecting the appropriate options under "File shares."
    • Configure NFS client:

      • On each HPC node, install NFS client packages if not already present.

      • Configure the NFS client by adding entries to the /etc/fstab file or using the mount command to mount the Azure Blob storage.

      • Example:

        sudo mount -t nfs <STORAGE_ACCOUNT_URL>:/<FILE_SHARE_NAME> /mnt/blob

  2. Setting up Azure Files:

    • Deploy an Azure File Share:

      • Navigate to the Azure portal and search for "Azure Storage accounts."
      • Create a new storage account if not already existing by specifying parameters such as resource group, location, and performance tier (Standard or Premium).
      • Within the storage account, navigate to the "File shares" section and create a new file share by specifying the name and quota (size).

    • Configure mount points:

      • Once the file share is created, obtain the necessary mount information from the Azure portal.

      • On each HPC node, install the necessary client packages for the protocol used (SMB) if not already present.

      • Use the mount information to configure the mount points by adding entries to the /etc/fstab file or using the mount command directly.

      • Example for SMB:

        sudo mount -t cifs //<STORAGE_ACCOUNT_NAME>.file.core.chinacloudapi.cn/<FILE_SHARE_NAME> /mnt/azurefiles -o vers=3.0,username=<STORAGE_ACCOUNT_NAME>,password=<STORAGE_ACCOUNT_KEY>,dir_mode=0777,file_mode=0777,sec=ntlmssp

Resources

  • Last updated on