用于 Go 的 HDInsight SDK(预览版)HDInsight SDK for Go (Preview)

概述Overview

用于 Go 的 HDInsight SDK 提供了用于管理 HDInsight 群集的类和函数。The HDInsight SDK for Go provides classes and functions that allow you to manage your HDInsight clusters. 该 SDK 包含用于创建、删除、更新、列出、调整大小、执行脚本操作,以及监视、获取 HDInsight 群集属性等操作。It includes operations to create, delete, update, list, resize, execute script actions, monitor, get properties of HDInsight clusters, and more.

备注

还可以从此处获得适用于此 SDK 的GoDoc 参考资料。GoDoc reference material for this SDK is also available here.

如果没有 Azure 订阅,请在开始前创建一个试用帐户If you don�t have an Azure subscription, create a trial account before you begin.

先决条件Prerequisites

SDK 安装SDK installation

从 GOPATH 位置中,运行 go get github.com/Azure/azure-sdk-for-go/tree/master/services/preview/hdinsight/mgmt/2018-06-01-preview/hdinsightFrom your GOPATH location, run go get github.com/Azure/azure-sdk-for-go/tree/master/services/preview/hdinsight/mgmt/2018-06-01-preview/hdinsight

身份验证Authentication

首先需要使用 Azure 订阅对该 SDK 进行身份验证。The SDK first needs to be authenticated with your Azure subscription. 请遵循以下示例创建服务主体,然后使用该服务主体进行身份验证。Follow the example below to create a service principal and use it to authenticate. 完成此操作后,将会获得 ClustersClient 的实例,其中包含可用于执行管理操作的许多函数(以下部分将概述这些函数)。After this is done, you will have an instance of a ClustersClient, which contains many functions (outlined in below sections) that can be used to perform management operations.

备注

除了以下示例中所示的方法以外,还有其他一些身份验证方法可能更符合你的需要。There are other ways to authenticate besides the below example that could potentially be better suited for your needs. 此处概述了所有函数:Azure SDK for Go 中的身份验证函数All functions are outlined here: Authentication functions in the Azure SDK for Go

使用服务主体的身份验证示例Authentication example using a service principal

验证当前使用的是要在其中创建服务主体的订阅。Verify you are currently using the subscription in which you want the service principal created.

az account show

订阅信息将显示为 JSON。Your subscription information is displayed as JSON.

{
  "environmentName": "AzureChinaCloud",
  "id": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "isDefault": true,
  "name": "XXXXXXX",
  "state": "Enabled",
  "tenantId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "user": {
    "cloudShellID": true,
    "name": "XXX@XXX.XXX",
    "type": "user"
  }
}

如果尚未登录到正确的订阅,请运行以下命令选择正确的订阅:If you're not logged into the correct subscription, select the correct one by running:

az account set -s <name or ID of subscription>

重要

如果尚未通过其他功能(例如,通过 Azure 门户创建 HDInsight 群集)注册 HDInsight 资源提供程序,则需要先执行一次此操作,然后才能进行身份验证。If you have not already registered the HDInsight Resource Provider by another function (such as by creating an HDInsight Cluster through the Azure Portal), you need to do this once before you can authenticate.

az provider register --namespace Microsoft.HDInsight

接下来,选择服务主体的名称,然后使用以下命令创建服务主体:Next, choose a name for your service principal and create it with the following command:

az ad sp create-for-rbac --name <Service Principal Name> --sdk-auth

服务主体信息将以 JSON 格式显示。The service principal information is displayed as JSON.

{
  "clientId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "clientSecret": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "subscriptionId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "tenantId": "XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX",
  "activeDirectoryEndpointUrl": "https://login.microsoftonline.cn",
  "resourceManagerEndpointUrl": "https://management.azure.cn/",
  "sqlManagementEndpointUrl": "https://management.core.chinacloudapi.cn:8443/",
  "galleryEndpointUrl": "https://gallery.azure.cn/",
  "managementEndpointUrl": "https://management.core.chinacloudapi.cn/"
}

复制以下代码片段,并在 TENANT_IDCLIENT_IDCLIENT_SECRETSUBSCRIPTION_ID 中填写运行创建服务主体的命令后返回的 JSON 中的字符串。Copy the below snippet and fill in TENANT_ID, CLIENT_ID, CLIENT_SECRET, and SUBSCRIPTION_ID with the strings from the JSON that was returned after running the command to create the service principal.

package main

import (
    "context"
    "github.com/Azure/go-autorest/autorest/azure/auth"
    hdi "github.com/Azure/azure-sdk-for-go/services/preview/hdinsight/mgmt/2018-06-01-preview/hdinsight"
    "github.com/Azure/go-autorest/autorest/to"    
)

func main() {
    var err error

    // Tenant ID for your Azure Subscription
    var TENANT_ID = ""
    // Your Service Principal App Client ID
    var CLIENT_ID = ""
    // Your Service Principal Client Secret
    var CLIENT_SECRET = ""
    // Azure Subscription ID
    var SUBSCRIPTION_ID = ""

    var credentials = auth.NewClientCredentialsConfig(CLIENT_ID, CLIENT_SECRET, TENANT_ID)
    var client = hdi.NewClustersClient(SUBSCRIPTION_ID)

    client.Authorizer, err = credentials.Authorizer()
    if (err != nil) {
        fmt.Println("Error: ", err)
    }

群集管理Cluster management

备注

本部分假设你已完成身份验证,已构造 ClusterClient 实例并已将其存储在名为 client 的变量中。This section assumes you have already authenticated and constructed a ClusterClient instance and store it in a variable called client. 在前面的“身份验证”部分中可以找到有关身份验证和获取 ClusterClient 的说明。Instructions for authenticating and obtaining a ClusterClient can be found in the Authentication section above.

创建群集Create a cluster

可以通过调用 client.Create() 来创建新群集。A new cluster can be created by calling client.Create().

示例Example

本示例演示如何创建包含 2 个头节点和 1 个工作节点的 Apache Spark 群集。This example demonstrates how to create an Apache Spark cluster with 2 head nodes and 1 worker node.

备注

首先需要创建一个资源组和存储帐户,下面将予以介绍。You first need to create a Resource Group and Storage Account, as explained below. 如果已创建资源组和存储帐户,则可以跳过这些步骤。If you have already created these, you can skip these steps.

创建资源组Creating a resource group

运行以下命令来创建资源组You can create a resource group by running

az group create -l <Region Name (i.e. chinaeast)> --n <Resource Group Name>
创建存储帐户Creating a storage account

运行以下命令来创建存储帐户You can create a storage account by running:

az storage account create -n <Storage Account Name> -g <Existing Resource Group Name> -l <Region Name (i.e. chinaeast)> --sku <SKU i.e. Standard_LRS>

现在,运行以下命令获取存储帐户的密钥(创建群集时需要用到):Now run the following command to get the key for your storage account (you will need this to create a cluster):

az storage account keys list -n <Storage Account Name>

以下 Go 代码片段创建包含 2 个头节点和 1 个工作节点的 Spark 群集。The below Go snippet creates a Spark cluster with 2 head nodes and 1 worker node. 按照注释中所述填写空白变量,并根据具体的需要任意更改其他参数。Fill in the blank variables as explained in the comments and feel free to change other parameters to suit your specific needs.

// The name for the cluster you are creating
var clusterName = "";
// The name of your existing Resource Group
var resourceGroupName = "";
// Choose a username
var username = "";
// Choose a password
var password = "";
// Replace <> with the name of your storage account
var storageAccount = "<>.blob.core.chinacloudapi.cn";
// Storage account key you obtained above
var storageAccountKey = "";
// Choose a region
var location = "";
var container = "default";

var parameters = hdi.ClusterCreateParametersExtended {
    Location: to.StringPtr(location),
    Tags: make(map[string]*string),
    Properties: &hdi.ClusterCreateProperties {
        ClusterVersion: to.StringPtr("3.6"),
        OsType: hdi.Linux,
        ClusterDefinition: &hdi.ClusterDefinition {
            Kind: to.StringPtr("spark"),
            Configurations: map[string]map[string]interface{}{
                "gateway": {
                    "restAuthCredential.isEnabled": "True",
                    "restAuthCredential.username":  username,
                    "restAuthCredential.password":  password,
                },
            },
        },
        Tier: hdi.Standard,
        ComputeProfile: &hdi.ComputeProfile {
            Roles: &[]hdi.Role {
                hdi.Role {
                    Name: to.StringPtr("headnode"),
                    TargetInstanceCount: to.Int32Ptr(2),
                    HardwareProfile: &hdi.HardwareProfile {
                        VMSize: to.StringPtr("Large"),
                    },
                    OsProfile: &hdi.OsProfile {
                        LinuxOperatingSystemProfile: &hdi.LinuxOperatingSystemProfile {
                            Username: to.StringPtr(username),
                            Password: to.StringPtr(password),
                        },
                    },
                },
                hdi.Role {
                    Name: to.StringPtr("workernode"),
                    TargetInstanceCount: to.Int32Ptr(1),
                    HardwareProfile: &hdi.HardwareProfile {
                        VMSize: to.StringPtr("Large"),
                    },
                    OsProfile: &hdi.OsProfile {
                        LinuxOperatingSystemProfile: &hdi.LinuxOperatingSystemProfile {
                            Username: to.StringPtr(username),
                            Password: to.StringPtr(password),
                        },
                    },
                },
            },
        },
        StorageProfile: &hdi.StorageProfile {
            Storageaccounts: &[]hdi.StorageAccount {
                hdi.StorageAccount {
                    Name: to.StringPtr(storageAccount),
                    Key: to.StringPtr(storageAccountKey),
                    Container: to.StringPtr(container),
                    IsDefault: to.BoolPtr(true),
                },
            },
        },
    },
}
client.Create(context.Background(), resourceGroupName, clusterName, parameters)

获取群集详细信息Get cluster details

获取给定群集的属性:To get properties for a given cluster:

client.Get(context.Background(), "<Resource Group Name>", "<Cluster Name>")

示例Example

可以使用 get 来确认已成功创建群集。You can use get to confirm that you have successfully created your cluster.

cluster, err := client.Get(context.Background(), resourceGroupName, clusterName)
if (err != nil) {
    fmt.Println("Error: ", err)
}
fmt.Println(*cluster.Name)
fmt.Println(*cluster.ID

输出应如下所示:The output should look like:

<Cluster Name>
/subscriptions/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX/resourceGroups/<Resource Group Name>/providers/Microsoft.HDInsight/clusters/<Cluster Name>

列出群集List clusters

列出订阅下的群集List clusters under the subscription

client.List()

按资源组列出群集List clusters by resource group

client.ListByResourceGroup("<Resource Group Name>")

备注

List()ListByResourceGroup() 都返回 ClusterListResultPage 结构。Both List() and ListByResourceGroup() return a ClusterListResultPage struct. 若要获取下一个页面,可以调用 Next()To get the next page, you can call Next(). 可以反复执行此调用,直到 ClusterListResultPage.NotDone() 返回 false,如以下示例中所示。This can be repeated until ClusterListResultPage.NotDone() returns false, as shown in the example below.

示例Example

以下示例列显当前订阅的所有群集的属性:The following example prints the properties of all clusters for the current subscription:

page, err := client.List(context.Background())
if (err != nil) {
    fmt.Println("Error: ", err)
}
for (page.NotDone()) {
    for _, cluster := range page.Values() {
        fmt.Println(*cluster.Name)
    }
    err = page.Next();
    if (err != nil) {
        fmt.Println("Error: ", err)
    }
}

删除群集Delete a cluster

删除群集:To delete a cluster:

client.Delete(context.Background(), "<Resource Group Name>", "<Cluster Name>")

更新群集标记Update cluster tags

可按如下所示更新给定群集的标记:You can update the tags of a given cluster like so:

client.Update(context.Background(), "<Resource Group Name>", "<Cluster Name>", hdi.ClusterPatchParameters{<map[string]*string} of Tags>)

示例Example

client.Update(context.Background(), "SDKTestRG", "SDKTest", hdi.ClusterPatchParameters{map[string]*string{"tag1Name" : to.StringPtr("tag1Value"), "tag2Name" : to.StringPtr("tag2Value")}})

调整群集大小Resize cluster

可以通过指定新大小来调整给定群集的工作节点数,如下所示:You can resize a given cluster's number of worker nodes by specifying a new size like so:

client.Resize(context.Background(), "<Resource Group Name>", "<Cluster Name>", hdi.ClusterResizeParameters{<Num of Worker Nodes (int)>})

脚本操作Script actions

HDInsight 提供了一个称为“脚本操作”的配置函数,用以调用自定义脚本来自定义群集。HDInsight provides a configuration function called script actions that invokes custom scripts to customize the cluster.

备注

有关如何使用脚本操作的详细信息见此处:使用脚本操作自定义基于 Linux 的 HDInsight 群集More information on how to use script actions can be found here: Customize Linux-based HDInsight clusters using script actions

执行脚本操作Execute script actions

可按如下所示在给定的群集上执行脚本操作:You can execute script actions on a given cluster like so:

var scriptAction1 = hdi.RuntimeScriptAction{Name: to.StringPtr("<Script Name>"), URI: to.StringPtr("<URL To Script>"), Roles: <&[]string of roles>} //valid roles are "headnode", "workernode", "zookeepernode", and "edgenode"
client.ExecuteScriptActions(context.Background(), "<Resource Group Name>", "<Cluster Name>", hdi.ExecuteScriptActionParameters{PersistOnSuccess: to.BoolPtr(true), ScriptActions: &[]hdi.RuntimeScriptAction{scriptAction1}}) //add more RuntimeScriptActions to the list to execute multiple scripts

对于“删除脚本操作”和“列出持久化脚本操作”操作,你需要创建一个 ScriptActionsClient,类似于创建 ClusterClient 以用于管理操作。For the 'Delete Script Action' and 'List Persisted Script Actions' operations, you need to create a ScriptActionsClient, similarly to how you created ClusterClient to use for management operations. 完成上面的“身份验证”部分后,可以创建一个 ScriptActionsClient,如下所示:Once you have completed the Authentication section above, you can create a ScriptActionsClient like so:

scriptActionsClient := hdi.NewScriptActionsClient(SUBSCRIPTION_ID)
scriptActionsClient.Authorizer, _ = credentials.Authorizer()

备注

下面的脚本操作示例假定你已如上所示初始化了一个名为 scriptActionsClientScriptActionsClient 并设置了其 AuthorizerThe below script actions examples assume you have already initialized a ScriptActionsClient called scriptActionsClient and set its Authorizer as shown above.

删除脚本操作Delete script action

删除给定群集上指定的持久化脚本操作:To delete a specified persisted script action on a given cluster:

scriptActionsClient.Delete(context.Background(), "<Resource Group Name>", "<Cluster Name>", "<Script Name>")

列出持久化脚本操作List persisted script actions

备注

ListByCluster() 返回一个 ScriptActionsListPage 结构。Both ListByCluster() returns a ScriptActionsListPage struct. 若要获取下一个页面,可以调用 Next()To get the next page, you can call Next(). 可以反复执行此调用,直到 ClusterListResultPage.NotDone() 返回 false,如以下示例中所示。This can be repeated until ClusterListResultPage.NotDone() returns false, as shown in the example below.

列出指定群集的所有持久化脚本操作:To list all persisted script actions for the specified cluster:

scriptActionsClient.ListByCluster(context.Background(), "<Resource Group Name>", "<Cluster Name>")

示例Example

page, err := scriptActionsClient.ListByCluster(context.Background(), resourceGroupName, clusterName)
if (err != nil) {
    fmt.Println("Error: ", err)
}
for (page.NotDone()) {
    for _, script := range page.Values() {          
        fmt.Println(*script.Name) //There are functions to get other properties of RuntimeScriptActionDetail besides Name, such as Status, Operation, StartTime, EndTime, etc. See reference documentation.
    }
    err = page.Next();
    if (err != nil) {
        fmt.Println("Error: ", err)
    }    
}

列出所有脚本的执行历史记录List all scripts' execution history

对于此操作,与创建 ScriptExecutionHistoryClient 以用于管理操作的方式类似,你需要创建一个 ClusterClientFor this operation, you need to create a ScriptExecutionHistoryClient, similarly to how you created ClusterClient to use for management operations. 完成上面的“身份验证”部分后,可以创建一个 ScriptActionsClient,如下所示:Once you have completed the Authentication section above, you can create a ScriptActionsClient like so:

scriptExecutionHistoryClient := hdi.NewScriptExecutionHistoryClient(SUBSCRIPTION_ID)
scriptExecutionHistoryClient.Authorizer, _ = credentials.Authorizer()

备注

以下示例假定你已如上所示初始化了一个名为 scriptExecutionHistoryClientScriptExecutionHistoryClient 并设置了其 AuthorizerThe below assumes you have already initialized a ScriptExecutionHistoryClient called scriptExecutionHistoryClient and set its Authorizer as shown above.

列出指定群集的所有脚本的执行历史记录:To list all scripts' execution history for the specified cluster:

scriptExecutionHistoryClient.ListByCluster(context.Background(), "<Resource Group Name>", "<Cluster Name>")

示例Example

此示例列显以往所有脚本执行活动的所有详细信息。This example prints all the details for all past script executions.

page, err := scriptExecutionHistoryClient.ListByCluster(context.Background(), resourceGroupName, clusterName)
if (err != nil) {
    fmt.Println("Error: ", err)
}
for (page.NotDone()) {
    for _, script := range page.Values() {          
        fmt.Println(*script.Name) //There are functions to get other properties of RuntimeScriptActionDetail besides Name, such as Status, Operation, StartTime, EndTime, etc. See reference documentation.
    }
    err = page.Next();
    if (err != nil) {
        fmt.Println("Error: ", err)
    }       
}

后续步骤Next steps