Create an Azure Data Explorer cluster and database

  • Article
  • 12/18/2024

Azure Data Explorer is a fast, fully managed data analytics service for real-time analysis on large volumes of data streaming from applications, websites, IoT devices, and more. To use Azure Data Explorer, you first create a cluster, and create one or more databases in that cluster. Then, you can ingest (load) data into a database and run queries against it.

In this article, you'll learn how to create a cluster and a database using either C#, Python, Go, the Azure CLI, PowerShell, or an Azure Resource Manager (ARM) template. To learn how to create a cluster and database using the Azure portal, see Quickstart: Create an Azure Data Explorer cluster and database.

For code samples based on previous SDK versions, see the archived article.

Prerequisites

Prerequisites by method of cluster and database creation:

Create an Azure Data Explorer cluster

This section guides you through the process of creating an Azure Data Explorer cluster. Choose the relevant tab for your preferred method to create the cluster.

The following code shows how to create a cluster.

  1. Set the required environment variables including service principal information from the prerequisites. Enter your subscription ID, resource group, and region where you want to create the cluster.

    export AZURE_CLIENT_ID="<enter service principal client ID>"
export AZURE_CLIENT_SECRET="<enter service principal client secret>"
export AZURE_TENANT_ID="<enter tenant ID>"

export SUBSCRIPTION="<enter subscription ID>"
export RESOURCE_GROUP="<enter resource group name>"
export LOCATION="<enter azure location e.g. China East 2>"

export CLUSTER_NAME_PREFIX="<enter prefix (cluster name will be [prefix]-ADXTestCluster)>"
export DATABASE_NAME_PREFIX="<enter prefix (database name will be [prefix]-ADXTestDB)>"

    Tip

    Use auth.NewAuthorizerFromCLIWithResource if you have Azure CLI installed and configured for authentication. In that situation, you don't need to create a service principal.

  2. Run the following code to create the cluster:

    import (
 "context"
 "log"
 "os"
 "strconv"

 "github.com/Azure/azure-sdk-for-go/sdk/azcore/to"
 "github.com/Azure/azure-sdk-for-go/sdk/azidentity"
 "github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/kusto/armkusto"
 "github.com/olekukonko/tablewriter"
)

const (
 subscriptionEnvVar      = "AZURE_SUBSCRIPTION_ID"
 resourceGroupEnvVar     = "AZURE_RESOURCE_GROUP"
 locationEnvVar          = "AZURE_LOCATION"
 clusterNamePrefixEnvVar = "CLUSTER_NAME_PREFIX"
 dbNamePrefixEnvVar      = "DATABASE_NAME_PREFIX"

 clusterName  = "ADXTestCluster"
 databaseName = "ADXTestDB"
)

func init() {
 subscription = os.Getenv(subscriptionEnvVar)
 if subscription == "" {
  log.Fatalf("missing environment variable %s", subscriptionEnvVar)
 }

 rgName = os.Getenv(resourceGroupEnvVar)
 if rgName == "" {
  log.Fatalf("missing environment variable %s", resourceGroupEnvVar)
 }

 location = os.Getenv(locationEnvVar)
 if location == "" {
  log.Fatalf("missing environment variable %s", locationEnvVar)
 }

 clusterNamePrefix = os.Getenv(clusterNamePrefixEnvVar)
 if clusterNamePrefix == "" {
  log.Fatalf("missing environment variable %s", clusterNamePrefixEnvVar)
 }

 dbNamePrefix = os.Getenv(dbNamePrefixEnvVar)
 if dbNamePrefix == "" {
  log.Fatalf("missing environment variable %s", dbNamePrefixEnvVar)
 }
}

func getClustersClient(subscription string) *armkusto.ClustersClient {
 cred, err := azidentity.NewDefaultAzureCredential(nil)
 if err != nil {
  log.Fatal(err)
 }

 client, err := armkusto.NewClustersClient(subscription, cred, nil)
 if err != nil {
  log.Fatal(err)
 }

 return client
}

// 1 instance, Basic tier with compute type Dev(No SLA)_Standard_D11_v2
func createCluster(sub, name, location, rgName string) {
 ctx := context.Background()

 numInstances := int32(1)
 client := getClustersClient(sub)
 result, err := client.BeginCreateOrUpdate(
  ctx,
  rgName,
  name,
  armkusto.Cluster{
   Location: &location,
   SKU: &armkusto.AzureSKU{
    Name:     to.Ptr(armkusto.AzureSKUNameDevNoSLAStandardD11V2),
    Capacity: &numInstances,
    Tier:     to.Ptr(armkusto.AzureSKUTierBasic),
   },
  },
  nil,
 )
 if err != nil {
  log.Fatal("failed to start cluster creation ", err)
 }

 log.Printf("waiting for cluster creation to complete - %s\n", name)
 r, err := result.PollUntilDone(ctx, nil)
 if err != nil {
  log.Fatal(err)
 }

 log.Printf("created cluster %s\n", *r.Name)
}

createCluster(subscription, clusterNamePrefix+clusterName, location, rgName)

  3. List the clusters to ensure successful creation:

    func listClusters(sub, rgName string) {
 log.Printf("listing clusters in resource group %s\n", rgName)
 ctx := context.Background()

 result := getClustersClient(sub).NewListByResourceGroupPager(rgName, nil)

 data := [][]string{}

 for result.More() {
  temp, err := result.NextPage(ctx)
  if err != nil {
   log.Fatal(err)
  }
  for _, c := range temp.Value {
   data = append(data, []string{*c.Name, string(*c.Properties.State), *c.Location, strconv.Itoa(int(*c.SKU.Capacity)), *c.Properties.URI})
  }
 }

 table := tablewriter.NewWriter(os.Stdout)
 table.SetHeader([]string{"Name", "State", "Location", "Instances", "URI"})

 for _, v := range data {
  table.Append(v)
 }
 table.Render()
}

listClusters(subscription, rgName)

Create an Azure Data Explorer database

In this section, you'll create a database within the cluster created in the previous section.

The following code shows how to create a database. The package imports and environment variable initiation is the same as in the previous section.

  1. Run the following code to create the database:

    func createDatabase(sub, rgName, clusterName, location, dbName string) {
 ctx := context.Background()

 client := getDBClient(sub)
 future, err := client.BeginCreateOrUpdate(ctx, rgName, clusterName, dbName, &armkusto.ReadWriteDatabase{Kind: to.Ptr(armkusto.KindReadWrite), Location: &location}, nil)

 if err != nil {
  log.Fatal("failed to start database creation ", err)
 }

 log.Printf("waiting for database creation to complete - %s\n", dbName)
 resp, err := future.PollUntilDone(ctx, nil)
 if err != nil {
  log.Fatal(err)
 }

 kdb := resp.GetDatabase()
 log.Printf("created DB %s with ID %s and type %s\n", *kdb.Name, *kdb.ID, *kdb.Type)
}

createDatabase(subscription, rgName, clusterNamePrefix+clusterName, location, dbNamePrefix+databaseName)

  2. List the databases to ensure successful creation:

    func listDatabases(sub, rgName, clusterName string) {
 log.Printf("listing databases in cluster %s\n", clusterName)

 ctx := context.Background()
 result := getDBClient(sub).NewListByClusterPager(rgName, clusterName, nil)

 data := [][]string{}

 for result.More() {
  temp, err := result.NextPage(ctx)
  if err != nil {
   log.Fatal(err)
  }
  for _, db := range temp.Value {
   if *db.GetDatabase().Kind == armkusto.KindReadWrite {
    data = append(data, []string{*db.GetDatabase().Name, string(*db.GetDatabase().Kind), *db.GetDatabase().Location, *db.GetDatabase().Type})
   }
  }
 }

 table := tablewriter.NewWriter(os.Stdout)
 table.SetHeader([]string{"Name", "State", "Location", "Type"})

 for _, v := range data {
  table.Append(v)
 }
 table.Render()
}

listDatabases(subscription, rgName, clusterNamePrefix+clusterName)

Next step

Create a table