Run MapReduce jobs with Apache Hadoop on HDInsight using REST

Learn how to use the Apache Hive WebHCat REST API to run MapReduce jobs on an Apache Hadoop on HDInsight cluster. Curl is used to demonstrate how you can interact with HDInsight by using raw HTTP requests to run MapReduce jobs.

Prerequisites

• An Apache Hadoop cluster on HDInsight. See Create Apache Hadoop clusters using the Azure portal.

Either:

• Windows PowerShell or,
• Curl with jq

Run a MapReduce job

Curl

1. For ease of use, set the variables below. This example is based on a Windows environment, revise as needed for your environment.

   set CLUSTERNAME=
   set PASSWORD=
   

2. From a command line, use the following command to verify that you can connect to your HDInsight cluster:

   curl -u admin:%PASSWORD% -G https://%CLUSTERNAME%.azurehdinsight.cn/templeton/v1/status
   

   The parameters used in this command are as follows:

   • -u: Indicates the user name and password used to authenticate the request
   • -G: Indicates that this operation is a GET request

   The beginning of the URI, https://CLUSTERNAME.azurehdinsight.cn/templeton/v1, is the same for all requests.

   You receive a response similar to the following JSON:

   {"version":"v1","status":"ok"}
   

3. To submit a MapReduce job, use the following command. Modify the path to jq as needed.

   curl -u admin:%PASSWORD% -d user.name=admin ^
   -d jar=/example/jars/hadoop-mapreduce-examples.jar ^
   -d class=wordcount -d arg=/example/data/gutenberg/davinci.txt -d arg=/example/data/output ^
   https://%CLUSTERNAME%.azurehdinsight.cn/templeton/v1/mapreduce/jar | ^
   C:\HDI\jq-win64.exe .id
   

   The end of the URI (/mapreduce/jar) tells WebHCat that this request starts a MapReduce job from a class in a jar file. The parameters used in this command are as follows:

   • -d: -G isn't used, so the request defaults to the POST method. -d specifies the data values that are sent with the request.
     • user.name: The user who is running the command
     • jar: The location of the jar file that contains class to be ran
     • class: The class that contains the MapReduce logic
     • arg: The arguments to be passed to the MapReduce job. In this case, the input text file and the directory that are used for the output

   This command should return a job ID that can be used to check the status of the job: job_1415651640909_0026.

4. To check the status of the job, use the following command. Replace the value for JOBID with the actual value returned in the previous step. Revise location of jq as needed.

   set JOBID=job_1415651640909_0026
   
   curl -G -u admin:%PASSWORD% -d user.name=admin https://%CLUSTERNAME%.azurehdinsight.cn/templeton/v1/jobs/%JOBID% | ^
   C:\HDI\jq-win64.exe .status.state
   

PowerShell

1. For ease of use, set the variables below. Replace CLUSTERNAME with your actual cluster name. Execute the command and enter the cluster login password when prompted.

   $clusterName="CLUSTERNAME"
   $creds = Get-Credential -UserName admin -Message "Enter the cluster login password"
   

2. Use the following command to verify that you can connect to your HDInsight cluster:

   $resp = Invoke-WebRequest -Uri "https://$clustername.azurehdinsight.cn/templeton/v1/status" `
       -Credential $creds `
       -UseBasicParsing
   $resp.Content
   

   You receive a response similar to the following JSON:

   {"version":"v1","status":"ok"}
   

3. To submit a MapReduce job, use the following command:

   $reqParams = @{}
   $reqParams."user.name" = "admin"
   $reqParams.jar = "/example/jars/hadoop-mapreduce-examples.jar"
   $reqParams.class = "wordcount"
   $reqParams.arg = @()
   $reqParams.arg += "/example/data/gutenberg/davinci.txt"
   $reqparams.arg += "/example/data/output"
   $resp = Invoke-WebRequest -Uri "https://$clusterName.azurehdinsight.cn/templeton/v1/mapreduce/jar" `
      -Credential $creds `
      -Body $reqParams `
      -Method POST `
      -UseBasicParsing
   $jobID = (ConvertFrom-Json $resp.Content).id
   $jobID
   

   The end of the URI (/mapreduce/jar) tells WebHCat that this request starts a MapReduce job from a class in a jar file. The parameters used in this command are as follows:

   • user.name: The user who is running the command
   • jar: The location of the jar file that contains class to be ran
   • class: The class that contains the MapReduce logic
   • arg: The arguments to be passed to the MapReduce job. In this case, the input text file and the directory that are used for the output

   This command should return a job ID that can be used to check the status of the job: job_1415651640909_0026.

4. To check the status of the job, use the following command:

   $reqParams=@{"user.name"="admin"}
   $resp = Invoke-WebRequest -Uri "https://$clusterName.azurehdinsight.cn/templeton/v1/jobs/$jobID" `
      -Credential $creds `
      -Body $reqParams `
      -UseBasicParsing
   
   # ConvertFrom-JSON can't handle duplicate names with different case
   # So change one to prevent the error
   $fixDup=$resp.Content.Replace("jobID","job_ID")
   (ConvertFrom-Json $fixDup).status.state
   

Both methods

1. If the job is complete, the state returned is SUCCEEDED.

2. When the state of the job has changed to SUCCEEDED, you can retrieve the results of the job from Azure Blob storage. The statusdir parameter that is passed with the query contains the location of the output file. In this example, the location is /example/curl. This address stores the output of the job in the clusters default storage at /example/curl.

You can list and download these files by using the Azure CLI. For more information on using the Azure CLI to work with Azure Blob storage, see Quickstart: Create, download, and list blobs with Azure CLI.

Next steps

For information about other ways you can work with Hadoop on HDInsight:

• Use MapReduce with Apache Hadoop on HDInsight
• Use Apache Hive with Apache Hadoop on HDInsight

For more information about the REST interface that is used in this article, see the WebHCat Reference.