Submit HPC jobs from an on-premises computer to an HPC Pack cluster deployed in Azure

  • Article

Configure an on-premises client computer to submit jobs to a HPC Pack cluster in Azure. This article shows you how to set up a local computer with client tools to submit job over HTTPS to the cluster in Azure that has public endpoint exposed through internet. In this way, several cluster users can submit jobs to a cloud-based HPC Pack cluster, but without connecting directly to the head node VM or accessing an Azure subscription. If your local on-premises computer has direct connection to the head node in azure through Express Route or VNet, your client app shall connect directly to the cluster instead.

Submit a job to a cluster in Azure

Prerequisites

  • HPC Pack head node deployed in an Azure VM - We recommend that you use automated tools such as an Azure HPC Pack ARM template to deploy the head node and cluster. You need the DNS name of the head node and the credentials of a cluster administrator to complete the steps in this article.

  • Client computer - You need a Windows or Windows Server client computer that can run HPC Pack client utilities (see system requirements). If you only want to use the HPC Pack web portal or REST API to submit jobs, you can use any client computer of your choice.

  • HPC Pack installation media - To install the HPC Pack client utilities, the free installation package for the latest version of HPC Pack is available from the Microsoft Download Center. Make sure that you download the same version of HPC Pack that is installed on the head node VM.

Step 1: Install and configure the web components on the head node

HPC Pack 2016 has the REST interface enabled by default. But if you are still using previous version of HPC Pac, you need to ensure that the HPC Pack web components are configured on the HPC Pack head node to enable the REST interface to submit jobs to the cluster over HTTPS,. If they aren't already installed, first install the web components by running the HpcWebComponents.msi installation file. Then, configure the components by running the HPC PowerShell script Set-HPCWebComponents.ps1.

For detailed procedures, see Install the Microsoft HPC Pack WebComponents.

Tip

Certain Azure quickstart templates for HPC Pack clusters even with previous version install and configure the web components automatically.

Step 2: Install the HPC Pack client utilities on an on-premises computer

If you want to install the HPC Pack client utilities on your computer, download the HPC Pack setup files (full installation) from the Microsoft DownloadCenter. When you begin the installation, choose the setup option for the HPC Pack client utilities.

To use the HPC Pack client tools to submit jobs to the head node VM, you need to install the certificate used during HPC cluster setup on the client computer. Use standard Windows certificate management procedures to install the public certificate to the Certificates – Current user > Trusted Root Certification Authorities store. The certificate must be in .CER format.

Tip

During client set up, you may need to choose CN validation if your cert CN doesn't match your the scheduler url". You may also choose "skip both CA and CN check" so that you don't need to import the cert on your client machine. Skip CA and CN check

To import the certificate on the client computer

  1. Copy the certificate to a folder on the client computer.
  2. On the client computer, run certmgr.msc.
  3. In Certificate Manager, expand Certificates – Current user > Trusted Root Certification Authorities, right-click Certificates, and then click All Tasks > Import.
  4. In the Certificate Import Wizard, click Next and follow the steps to import the certificate that you exported from the head node to the Trusted Root Certification Authorities store.

Tip

You might see a security warning, because the certification authority on the head node isn't recognized by the client computer. For testing purposes, you can ignore this warning and complete the certificate import.

Step 3: Run test jobs on the cluster

To verify your configuration, try running jobs on the cluster in Azure from the on-premises computer. For example, you can use HPC Pack GUI tools or command-line commands to submit jobs to the cluster. You can also use a web-based portal to submit jobs.

To run job submission commands on the client computer

  1. On a client computer where the HPC Pack client utilities are installed, start a Command Prompt.

  2. Type a sample command. For example, to list all jobs on the cluster, type a command similar to one of the following, depending on the full DNS name of the head node or Azure load balancer DNS name for 3-head-node cluster:

    job list /scheduler:https://<HeadNodeDnsName>.<region>.cloudapp.azure.com|<Azure load balancer DNS name> /all

  3. When prompted, type the user name (in the form <DomainName>\<UserName>) and password of the HPC cluster administrator or another cluster user that you configured. You can choose to store the credentials locally for more job operations.

    A list of jobs appears.

To use HPC Job Manager on the client computer

  1. If you didn't previously store domain credentials for a cluster user when submitting a job, you can add the credentials in Credential Manager.

    a. In Control Panel on the client computer, start Credential Manager.

    b. Click Windows Credentials > Add a generic credential.

    c. Specify the Internet address (for example, https://<HeadNodeDnsName>.<region>.cloudapp.azure.com/HpcScheduler), and the user name (<DomainName>\<UserName>) and password of the cluster administrator or another cluster user that you configured.

  2. On the client computer, start HPC Job Manager.

  3. In the Select Head Node dialog box, type the URL to the head node in Azure (for example, https://<HeadNodeDnsName>.<region>.cloudapp.azure.com).

    HPC Job Manager opens and shows a list of jobs on the head node.

To use the web portal running on the head node

  1. Start a web browser on the client computer, and enter one of the following addresses, depending on the full DNS name of the head node:

    https://<HeadNodeDnsName>.<region>.cloudapp.azure.com/HpcPortal

  2. In the security dialog box that appears, type the domain credentials of the HPC cluster administrator. (You can also add other cluster users in different roles. See Managing Cluster Users.)

    The web portal opens to the job list view.

  3. To submit a sample job that returns the string “Hello World” from the cluster, click New job in the left-hand navigation.

  4. On the New Job page, under From submission pages, click HelloWorld. The job submission page appears.

  5. Click Submit. If prompted, provide the domain credentials of the HPC cluster administrator. The job is submitted, and the job ID appears on the My Jobs page.

  6. To view the results of the job that you submitted, click the job ID, and then click View Tasks to view the command output (under Output).

Next steps