Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
This guide provides best practices and guidelines to a storage solution that is best suited to your high-performance computing (HPC) workload.
There's typically a trade-off between optimizing for costs and optimizing for performance. This workload best practices series is focused on getting the best storage solution for HPC workloads on Azure VMs. If your workload is less demanding, you might not require every recommended optimization. Consider your performance needs, costs, and workload patterns as you evaluate these recommendations.
Storage for HPC workloads consists of core storage and in some cases, an accelerator.
Core storage acts as the permanent home for your data. It contains rich data management features and is durable, available, scalable, elastic, and secure. An accelerator enhances core storage by providing high-performance data access. An accelerator can be provisioned on demand and gives your computational workload much faster access to data.
If you are starting from scratch, see Understand data store models to choose a data store and Choose an Azure storage service or Introduction to Azure Storage to get an idea of your storage service options.
Start with the amount of data that you plan to store. Then, consider the number of CPU cores used by your workload and the size of your files. These factors help you to narrow down which core storage service best suits your workload and whether to use an accelerator to enhance performance.
| Configuration | CPU cores | Sizes of files | Core Storage Recommendation | Accelerator Recommendation |
|---|---|---|---|---|
| Under 50 TiB | N/A | N/A | Azure Files. | No accelerator |
| 50 TiB - 5,000 TiB | Less than 500 | N/A | Azure Files. | No accelerator |
| 50 TiB - 5,000 TiB | Over 500 | 1 MiB and larger | Azure Standard Blob. It’s supported by all accelerators, supports many protocols, and is cost-effective. | |
| 50 TiB - 5,000 TiB | Over 500 | Smaller than 1 MiB | Azure Premium Blob or Azure Standard Blob. | |
| 50 TiB - 5,000 TiB | Over 500 | Smaller than 512 KiB | No accelerator | |
| Over 5,000 TiB | N/A | N/A | Talk to your field or account team. |
If you are still stuck between options after using the decision trees, here are more details for each solution:
| Solution | Optimal Performance & Scale | Data Access (Access Protocol) | Billing Model | Core Storage or Accelerator |
|---|---|---|---|---|
| Azure Standard Blob | Good for large file, bandwidth-intensive workloads. | Good for traditional (file) and cloud-native (REST) HPC apps. Easy to access, share, manage datasets. Works with all accelerators. |
Pay for what you use. | Core Storage. |
| Azure Premium Blob | IOPS and latency better than Standard Blob. Good for datasets with many medium-sized files and mixed file sizes. |
Good for traditional (file) and cloud-native (REST) HPC apps. Easy to access, share, manage datasets. Works with all accelerators. |
Pay for what you use. | Core Storage. |
| Azure Premium Files | Capacity and bandwidth suited for smaller scale (<1k cores). IOPS and latency good for medium sized files (>512 KiB). |
Easy integration with Linux (NFS) and Windows (SMB), but can't use both NFS+SMB to access the same data. | Pay for what you provision. | Core Storage. |
In order of most to least expensive, the core storage option prices are:
- Azure Premium Blob and Azure Premium Files
- Azure Standard Blob
For more info on the pricing, see Azure product pricing.