Create a namespace with event hub and enable Capture using a template

Article
08/05/2024

This article shows how to use an Azure Resource Manager template that creates an Event Hubs namespace, with one event hub instance, and also enables the Capture feature on the event hub. The article describes how to define which resources are deployed, and how to define parameters that are specified when the deployment is executed. You can use this template for your own deployments, or customize it to meet your requirements.

This article also shows how to specify that events are captured into either Azure Storage Blobs or an Azure Data Lake Store, based on the destination you choose.

For more information about creating templates, see Authoring Azure Resource Manager templates. For the JSON syntax and properties to use in a template, see Microsoft.EventHub resource types.

For more information about patterns and practices for Azure Resources naming conventions, see Azure Resources naming conventions.

For the complete templates, select the following GitHub links:

Note

To check for the latest templates, visit the Azure Quickstart Templates gallery and search for Event Hubs.

What will you deploy?

With this template, you deploy an Event Hubs namespace with an event hub, and also enable Event Hubs Capture. Event Hubs Capture enables you to automatically deliver the streaming data in Event Hubs to Azure Blob storage or Azure Data Lake Store, within a specified time or size interval of your choosing. Select the following button to enable Event Hubs Capture into Azure Storage:

Select the following button to enable Event Hubs Capture into Azure Data Lake Store:

Parameters

With Azure Resource Manager, you define parameters for values you want to specify when the template is deployed. The template includes a section called Parameters that contains all the parameter values. You should define a parameter for those values that vary based on the project you're deploying or based on the environment you're deploying to. Don't define parameters for values that always stay the same. Each parameter value is used in the template to define the resources that are deployed.

The template defines the following parameters.

eventHubNamespaceName

The name of the Event Hubs namespace to create.

"eventHubNamespaceName":{
     "type":"string",
     "metadata":{
         "description":"Name of the EventHub namespace"
      }
}

eventHubName

The name of the event hub created in the Event Hubs namespace.

"eventHubName":{
    "type":"string",
    "metadata":{
        "description":"Name of the event hub"
    }
}

messageRetentionInDays

The number of days to retain the messages in the event hub.

"messageRetentionInDays":{
    "type":"int",
    "defaultValue": 1,
    "minValue":"1",
    "maxValue":"7",
    "metadata":{
       "description":"How long to retain the data in event hub"
     }
 }

partitionCount

The number of partitions to create in the event hub.

"partitionCount":{
    "type":"int",
    "defaultValue":2,
    "minValue":2,
    "maxValue":32,
    "metadata":{
        "description":"Number of partitions chosen"
    }
 }

captureEnabled

Enable Capture on the event hub.

"captureEnabled":{
    "type":"string",
    "defaultValue":"true",
    "allowedValues": [
    "false",
    "true"],
    "metadata":{
        "description":"Enable or disable the Capture for your event hub"
    }
 }

captureEncodingFormat

The encoding format you specify to serialize the event data.

"captureEncodingFormat":{
    "type":"string",
    "defaultValue":"Avro",
    "allowedValues":[
    "Avro"],
    "metadata":{
        "description":"The encoding format in which Capture serializes the EventData"
    }
}

captureTime

The time interval in which Event Hubs Capture starts capturing the data.

"captureTime":{
    "type":"int",
    "defaultValue":300,
    "minValue":60,
    "maxValue":900,
    "metadata":{
         "description":"The time window in seconds for the capture"
    }
}

captureSize

The size interval at which Capture starts capturing the data.

"captureSize":{
    "type":"int",
    "defaultValue":314572800,
    "minValue":10485760,
    "maxValue":524288000,
    "metadata":{
        "description":"The size window in bytes for capture"
    }
}

captureNameFormat

The name format used by Event Hubs Capture to write the Avro files. The capture name format must contain {Namespace}, {EventHub}, {PartitionId}, {Year}, {Month}, {Day}, {Hour}, {Minute}, and {Second} fields. These fields can be arranged in any order, with or without delimiters.

"captureNameFormat": {
      "type": "string",
      "defaultValue": "{Namespace}/{EventHub}/{PartitionId}/{Year}/{Month}/{Day}/{Hour}/{Minute}/{Second}",
      "metadata": {
        "description": "A Capture Name Format must contain {Namespace}, {EventHub}, {PartitionId}, {Year}, {Month}, {Day}, {Hour}, {Minute} and {Second} fields. These can be arranged in any order with or without delimeters. E.g.  Prod_{EventHub}/{Namespace}\\{PartitionId}_{Year}_{Month}/{Day}/{Hour}/{Minute}/{Second}"
      }
    }

apiVersion

The API version of the template.

 "apiVersion":{
    "type":"string",
    "defaultValue":"2017-04-01",
    "metadata":{
        "description":"ApiVersion used by the template"
    }
 }

Use the following parameters if you choose Azure Storage as your destination.

destinationStorageAccountResourceId

Capture requires an Azure Storage account resource ID to enable capturing to your desired Storage account.

 "destinationStorageAccountResourceId":{
    "type":"string",
    "metadata":{
        "description":"Your existing Storage account resource ID where you want the blobs be captured"
    }
 }

blobContainerName

The blob container in which to capture your event data.

 "blobContainerName":{
    "type":"string",
    "metadata":{
        "description":"Your existing storage container in which you want the blobs captured"
    }
}

Azure Storage as destination

Creates a namespace of type EventHub, with one event hub, and also enables Capture to Azure Blob Storage.

"resources":[
      {
         "apiVersion":"[variables('ehVersion')]",
         "name":"[parameters('eventHubNamespaceName')]",
         "type":"Microsoft.EventHub/Namespaces",
         "location":"[variables('location')]",
         "sku":{
            "name":"Standard",
            "tier":"Standard"
         },
         "resources": [
    {
      "apiVersion": "2017-04-01",
      "name": "[parameters('eventHubNamespaceName')]",
      "type": "Microsoft.EventHub/Namespaces",
      "location": "[resourceGroup().location]",
      "sku": {
        "name": "Standard"
      },
      "properties": {
        "isAutoInflateEnabled": "true",
        "maximumThroughputUnits": "7"
      },
      "resources": [
        {
          "apiVersion": "2017-04-01",
          "name": "[parameters('eventHubName')]",
          "type": "EventHubs",
          "dependsOn": [
            "[concat('Microsoft.EventHub/namespaces/', parameters('eventHubNamespaceName'))]"
          ],
          "properties": {
            "messageRetentionInDays": "[parameters('messageRetentionInDays')]",
            "partitionCount": "[parameters('partitionCount')]",
            "captureDescription": {
              "enabled": "true",
              "skipEmptyArchives": false,
              "encoding": "[parameters('captureEncodingFormat')]",
              "intervalInSeconds": "[parameters('captureTime')]",
              "sizeLimitInBytes": "[parameters('captureSize')]",
              "destination": {
                "name": "EventHubArchive.AzureBlockBlob",
                "properties": {
                  "storageAccountResourceId": "[parameters('destinationStorageAccountResourceId')]",
                  "blobContainer": "[parameters('blobContainerName')]",
                  "archiveNameFormat": "[parameters('captureNameFormat')]"
                }
              }
            }
          }

        }
      ]
    }
  ]

Commands to run deployment

To deploy the resources to Azure, you must be signed in to your Azure account and you must use the Azure Resource Manager module. To learn about using Azure Resource Manager with either Azure PowerShell or Azure CLI, see:

The following examples assume you already have a resource group in your account with the specified name.

PowerShell

Note

We recommend that you use the Azure Az PowerShell module to interact with Azure. See Install Azure PowerShell to get started. To learn how to migrate to the Az PowerShell module, see Migrate Azure PowerShell from AzureRM to Az.

Deploy your template to enable Event Hubs Capture into Azure Storage:

New-AzResourceGroupDeployment -ResourceGroupName \<resource-group-name\> -TemplateFile https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/quickstarts/microsoft.eventhub/eventhubs-create-namespace-and-enable-capture/azuredeploy.json

Azure CLI

Azure Blob Storage as destination:

az deployment group create \<my-resource-group\> \<my-deployment-name\> --template-uri [https://raw.githubusercontent.com/Azure/azure-quickstart-templates/master/quickstarts/microsoft.eventhub/eventhubs-create-namespace-and-enable-capture/azuredeploy.json][]

Next steps

You can also configure Event Hubs Capture via the Azure portal. For more information, see Enable Event Hubs Capture using the Azure portal.

You can learn more about Event Hubs by visiting the following links:

Create a namespace with event hub and enable Capture using a template

What will you deploy?

Parameters

eventHubNamespaceName

eventHubName

messageRetentionInDays

partitionCount

captureEnabled

captureEncodingFormat

captureTime

captureSize

captureNameFormat

apiVersion

destinationStorageAccountResourceId

blobContainerName

Azure Storage as destination

Commands to run deployment

PowerShell

Azure CLI

Next steps

Additional resources