Create an Azure Data Explorer cluster and database using Go

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. To use Azure Data Explorer, you first create a cluster, and create one or more databases in that cluster. Then you ingest (load) data into a database so that you can run queries against it.

In this article, you create an Azure Data Explorer cluster and database using Go. You can then list and delete your new cluster and database and execute operations on your resources.

Prerequisites

  • An Azure subscription. Create a Azure account.
  • Install Git.
  • Install an appropriate version of Go. For more information regarding supported releases, see the Azure Go SDK.

Review the code

This section is optional. If you're interested to learn how the code works, you can review the following code snippets. Otherwise, you can skip ahead to Run the application.

Authentication

The program needs to authenticate to Azure Data Explorer before executing any operations. The Client credentials authentication type is used by auth.NewAuthorizerFromEnvironment that looks for the following pre-defined environment variables: AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, AZURE_TENANT_ID.

The following example shows how a kusto.ClustersClient is created using this technique:

func getClustersClient(subscription string) kusto.ClustersClient {
	client := kusto.NewClustersClient(subscription)
	authR, err := auth.NewAuthorizerFromEnvironment()
	if err != nil {
		log.Fatal(err)
	}
	client.Authorizer = authR

	return client
}

Tip

Use the auth.NewAuthorizerFromCLIWithResource function for local development if you have Azure CLI installed and configured for authentication.

Create cluster

Use the CreateOrUpdate function on kusto.ClustersClient to create a new Azure Data Explorer cluster. Wait for the process to complete before inspecting the results.

func createCluster(sub, name, location, rgName string) {
    ...
	result, err := client.CreateOrUpdate(ctx, rgName, name, kusto.Cluster{Location: &location, Sku: &kusto.AzureSku{Name: kusto.DevNoSLAStandardD11V2, Capacity: &numInstances, Tier: kusto.Basic}})
    ...
    err = result.WaitForCompletionRef(context.Background(), client.Client)
    ...
    r, err := result.Result(client)
}

List clusters

Use the ListByResourceGroup function on kusto.ClustersClient to get a kusto.ClusterListResult that is then iterated to show the output in a tabular format.

func listClusters(sub, rgName string) {
    ...
	result, err := getClustersClient(sub).ListByResourceGroup(ctx, rgName)
    ...
    for _, c := range *result.Value {
		// setup tabular representation
	}
    ...
}

Create database

Use CreateOrUpdate function on kusto.DatabasesClient to create a new Azure Data Explorer database in an existing cluster. Wait for the process to complete before inspecting the results.

func createDatabase(sub, rgName, clusterName, location, dbName string) {
	future, err := client.CreateOrUpdate(ctx, rgName, clusterName, dbName, kusto.ReadWriteDatabase{Kind: kusto.KindReadWrite, Location: &location})
    ...
    err = future.WaitForCompletionRef(context.Background(), client.Client)
    ...
    r, err := future.Result(client)
    ...
}

List databases

Use ListByCluster function on kusto.DatabasesClient to get kusto.DatabaseListResult that is then iterated to show the output in a tabular format.

func listDatabases(sub, rgName, clusterName string) {
	result, err := getDBClient(sub).ListByCluster(ctx, rgName, clusterName)
    ...
	for _, db := range *result.Value {
		// setup tabular representation
    }
    ...
}

Delete database

Use Delete function on a kusto.DatabasesClient to delete an existing database in a cluster. Wait for the process to complete before inspecting the results.

func deleteDatabase(sub, rgName, clusterName, dbName string) {
	...
    future, err := getDBClient(sub).Delete(ctx, rgName, clusterName, dbName)
    ...
    err = future.WaitForCompletionRef(context.Background(), client.Client)
    ...
    r, err := future.Result(client)
    if r.StatusCode == 200 {
        // determine success or failure
    }
    ...
}

Delete cluster

Use Delete function on a kusto.ClustersClient to delete a cluster. Wait for the process to complete before inspecting the results.

func deleteCluster(sub, clusterName, rgName string) {
	result, err := client.Delete(ctx, rgName, clusterName)
    ...
	err = result.WaitForCompletionRef(context.Background(), client.Client)
    ...
	r, err := result.Result(client)
    if r.StatusCode == 200 {
        // determine success or failure
    }
    ...
}

Run the application

When you run the sample code as is, the following actions are performed:

  1. An Azure Data Explorer cluster is created.

  2. All the Azure Data Explorer clusters in the specified resource group are listed.

  3. An Azure Data Explorer database is created in the existing cluster.

  4. All the databases in the specified cluster are listed.

  5. The database is deleted.

  6. The cluster is deleted.

    func main() {
        createCluster(subscription, clusterNamePrefix+clusterName, location, rgName)
        listClusters(subscription, rgName)
        createDatabase(subscription, rgName, clusterNamePrefix+clusterName, location, dbNamePrefix+databaseName)
        listDatabases(subscription, rgName, clusterNamePrefix+clusterName)
        deleteDatabase(subscription, rgName, clusterNamePrefix+clusterName, dbNamePrefix+databaseName)
        deleteCluster(subscription, clusterNamePrefix+clusterName, rgName)
    }
    

    Tip

    To try different combinations of operations, you can uncomment and comment the respective functions in main.go as needed.

  7. Clone the sample code from GitHub:

    git clone https://github.com/Azure-Samples/azure-data-explorer-go-cluster-management.git
    cd azure-data-explorer-go-cluster-management
    
  8. The program authenticates using client credentials. Use the Azure CLI az ad sp create-for-rbac command to create a service principal. Save the client ID, client secret, and tenant ID information for use in the next step.

  9. Export required environment variables including service principal information. Enter your subscription ID, resource group, and region where you want to create the cluster.

    export AZURE_CLIENT_ID="<enter service principal client ID>"
    export AZURE_CLIENT_SECRET="<enter service principal client secret>"
    export AZURE_TENANT_ID="<enter tenant ID>"
    
    export SUBSCRIPTION="<enter subscription ID>"
    export RESOURCE_GROUP="<enter resource group name>"
    export LOCATION="<enter azure location e.g. China East 2>"
    
    export CLUSTER_NAME_PREFIX="<enter prefix (cluster name will be [prefix]-ADXTestCluster)>"
    export DATABASE_NAME_PREFIX="<enter prefix (database name will be [prefix]-ADXTestDB)>"
    

    Tip

    You're likely to use environment variables in production scenarios to provide credentials to your application. As mentioned in Review the code, for local development, use auth.NewAuthorizerFromCLIWithResource if you have Azure CLI installed and configured for authentication. In that situation, you don't need to create a service principal.

  10. Run the program:

    go run main.go
    

    You'll get an output similar to the following:

    waiting for cluster creation to complete - fooADXTestCluster
    created cluster fooADXTestCluster
    listing clusters in resource group <your resource group>
    +-------------------+---------+----------------+-----------+-----------------------------------------------------------+
    |       NAME        |  STATE  |    LOCATION    | INSTANCES |                            URI                           |
    +-------------------+---------+----------------+-----------+-----------------------------------------------------------+
    | fooADXTestCluster | Running | China East 2   |         1 | https://fooADXTestCluster.chinaeast2.kusto.chinacloudapi.cn |
    +-------------------+---------+----------------+-----------+-----------------------------------------------------------+
    
    waiting for database creation to complete - barADXTestDB
    created DB fooADXTestCluster/barADXTestDB with ID /subscriptions/<your subscription ID>/resourceGroups/<your resource group>/providers/Microsoft.Kusto/Clusters/fooADXTestCluster/Databases/barADXTestDB and type Microsoft.Kusto/Clusters/Databases
    
    listing databases in cluster fooADXTestCluster
    +--------------------------------+-----------+----------------+------------------------------------+
    |              NAME              |   STATE   |    LOCATION    |                TYPE                |
    +--------------------------------+-----------+----------------+------------------------------------+
    | fooADXTestCluster/barADXTestDB | Succeeded |  China East 2  | Microsoft.Kusto/Clusters/Databases |
    +--------------------------------+-----------+----------------+------------------------------------+
    
    waiting for database deletion to complete - barADXTestDB
    deleted DB barADXTestDB from cluster fooADXTestCluster
    
    waiting for cluster deletion to complete - fooADXTestCluster
    deleted Azure Data Explorer cluster fooADXTestCluster from resource group <your resource group>
    

Clean up resources

If you didn't delete the cluster programmatically using the sample code in this article, delete them manually using Azure CLI.

Next steps

Ingest data using the Azure Data Explorer Go SDK