Note
Access to this page requires authorization. You can try signing in or changing directories.
Access to this page requires authorization. You can try changing directories.
The Logs Ingestion API in Azure Monitor allows you to send custom data to a Log Analytics workspace. This tutorial uses Azure Resource Manager templates (ARM templates) to walk through configuration of the components required to support the API and then provides a sample application using both the REST API and client libraries for .NET, Go, Java, JavaScript, and Python.
Note
This tutorial uses ARM templates to configure the components required to support the Logs ingestion API. See Tutorial: Send data to Azure Monitor Logs with Logs ingestion API (Azure portal) for a similar tutorial that uses the Azure portal UI to configure these components.
The steps required to configure the Logs ingestion API are as follows:
- Create a Microsoft Entra application to authenticate against the API.
- Create a custom table in a Log Analytics workspace. This is the table you'll be sending data to.
- Create a data collection rule (DCR) to direct the data to the target table.
- Give the Microsoft Entra application access to the DCR.
- See Sample code to send data to Azure Monitor using Logs ingestion API for sample code to send data to using the Logs ingestion API.
Note
This article includes options for using a DCR ingestion endpoint or a data collection endpoint (DCE). Choose either one, but a DCE is required with Logs ingestion API if private link is used. See When is a DCE required?.
Prerequisites
To complete this tutorial, you need:
- A Log Analytics workspace where you have at least contributor rights.
- Permissions to create DCR objects in the workspace.
Collect workspace details
Start by gathering information that you'll need from your workspace.
Go to your workspace in the Log Analytics workspaces menu in the Azure portal. On the Properties page, copy the Resource ID and save it for later use. Record the subscription ID, resource group name, workspace name, and workspace region.
Note
The templates use those individual values to construct the workspace resource ID.
Create Microsoft Entra application
Start by registering a Microsoft Entra application to authenticate against the API.
Note
Any Resource Manager authentication scheme is supported, but this tutorial follows the Client Credential Grant Flow scheme.
On the Microsoft Entra ID menu in the Azure portal, select App registrations > New registration.
Give the application a name and change the tenancy scope if the default isn't appropriate for your environment. A Redirect URI isn't required.
After registration, you can view the details of the application. Note the Application (client) ID and the Directory (tenant) ID. You'll need these values later in the process.
Generate an application client secret, which is similar to creating a password to use with a username. Select Certificates & secrets > New client secret. Give the secret a name to identify its purpose and select an Expires duration.
Note
The option 12 months is selected here. For a production implementation, follow best practices for a secret rotation procedure or use a more secure authentication mode, such as a certificate.
Select Add to save the secret and then note the Value. Ensure that you record this value because you can't recover it after you leave this page. Use the same security measures as you would for safekeeping a password because it's the functional equivalent.
Create data collection endpoint
A DCE provides an ingestion endpoint for data sent to Azure Monitor. Skip this section if you use the DCR ingestion endpoint without private link. If you use a DCE, configure it, link it to the DCR, and use its ingestion endpoint in your application. The DCE must be in the same region as the DCR and the destination Log Analytics workspace.
In the Azure portal's search box, enter template and then select Deploy a custom template.
Select Build your own template in the editor.
For the portal procedure, use the ARM template tab, configure the template in the editor, and select Save. You can instead use the Bicep equivalent with the Bicep deployment commands.
Note
The following templates create a DCE for this tutorial:
- Use the same region as the workspace and DCR.
- The sample enables public network access. For private-link ingestion, follow Azure Monitor Private Link configuration and retain the required network settings.
Template deployments create or update resources; they aren't partial PATCH operations. If you reuse an existing DCE, preserve its network access, identity, tags, and other required configuration.
The following Bicep example uses the
Microsoft.Insights/dataCollectionEndpointsresource type.@description('Specifies the name of the data collection endpoint to create.') param dataCollectionEndpointName string = '<DataCollectionEndpointName>' @description('Specifies the region for the data collection endpoint.') param azureRegion string = '<AzureRegion>' resource dataCollectionEndpoint 'Microsoft.Insights/dataCollectionEndpoints@<ApiVersion>' = { name: dataCollectionEndpointName location: azureRegion properties: { networkAcls: { publicNetworkAccess: 'Enabled' } } } output dataCollectionEndpointId string = dataCollectionEndpoint.idOn the Custom deployment screen, specify a Subscription and Resource group to store the DCE, and provide the template parameters. Set
azureRegionto the workspace's region.Select Review + create and then select Create after you review the details.
Select JSON View to view other details for the DCE. Copy the Resource ID and the logsIngestion endpoint which you'll need in a later step.
Create new table in Log Analytics workspace
A custom table must be created before you can send data to it. Consider these constraints when you create the table:
- Each column requires the
name,type, anddescriptionproperties. - The properties
isHiddenandisDefaultDisplayboth default tofalseif you don't specify them. - Possible data types are
string,int,long,real,boolean,dateTime, anddynamic. The API and portal UI acceptguidas a column type, but GUIDs are stored and queried asstringtype. When you define the stream in your data collection rule, declare GUID columns asstring. For more information, see GUID columns and data type differences. - Custom tables must use a suffix of
_CL.
Note
This tutorial uses PowerShell to make REST API calls by using the Azure Monitor Tables API. You can use any other valid method to make these calls.
Use a new MyTable_CL table for this tutorial. Before applying the request to an existing table, preserve its other columns and its plan and retention settings.
Run the following code to create the tutorial's
MyTable_CLtable.Note
The table name matches the DCR output stream and the linked application samples.
The following Azure PowerShell example uses Invoke-AzRestMethod to call the
Tables - Create Or UpdateREST API operation.
Create new table in Log Analytics workspace
# Set variables $resourceGroupName = "<ResourceGroupName>" $workspaceName = "<WorkspaceName>" $tableName = "MyTable_CL" $apiVersion = "<ApiVersion>" # Get the subscription ID from the current Azure PowerShell context $subscriptionId = (Get-AzContext).Subscription.Id # Build request URL $apiEndpoint = "https://management.chinacloudapi.cn" $path = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName" $provider = "Microsoft.OperationalInsights/workspaces/$workspaceName" $queryString = "?api-version=$apiVersion" $url = "$apiEndpoint$path/providers/$provider/tables/$tableName$queryString" # Build request body $tableParams = @' { "properties": { "schema": { "name": "MyTable_CL", "columns": [ { "name": "TimeGenerated", "type": "dateTime", "description": "The time at which the data was generated" }, { "name": "Computer", "type": "string", "description": "The computer that generated the data" }, { "name": "AdditionalContext", "type": "dynamic", "description": "Additional message properties" }, { "name": "CounterName", "type": "string", "description": "Name of the counter" }, { "name": "CounterValue", "type": "real", "description": "Value collected for the counter" } ] } } } '@ # Define parameters for Invoke-AzRestMethod $invokeAzRestMethodParams = @{ Uri = $url Method = "PUT" Payload = $tableParams } # Create the table Invoke-AzRestMethod @invokeAzRestMethodParams
Create data collection rule
The DCR defines how the data will be handled once it's received. This includes:
- Schema of data that's sent to the endpoint.
- Transformation that's applied to the data before it's sent to the workspace.
- Destination workspace and table for the transformed data.
Note
Before you deploy either DCR template:
- Create the
MyTable_CLtable in the destination workspace with the schema from the preceding section. - Use the same Azure region for the DCR, workspace, and DCE when applicable.
- Choose the DCR ingestion endpoint variant, or use the DCE variant with an existing DCE and its subscription, resource group, and resource name.
- When you reuse an existing DCR, retain the streams, destinations, data flows, identity, and other settings that it still requires.
Template deployments create or update resources; they aren't partial PATCH operations. When reusing a DCR, preserve its existing streams, destinations, data flows, identity, and other required settings.
In the Azure portal's search box, enter template and then select Deploy a custom template.
Select Build your own template in the editor.
Choose the endpoint variant that you need. For the portal procedure, configure the template from its ARM template tab in the editor and then select Save. Alternatively, deploy its Bicep counterpart with the Bicep deployment commands.
DCR ingestion endpoint
Note
Notice the following details in the DCR defined in this template:
streamDeclarations: Column definitions of the incoming data.destinations: Destination workspace.dataFlows: Matches the stream with the destination workspace and specifies the transformation query and the destination table. The output of the destination query is what gets sent to the destination table.
The following Bicep example uses the
Microsoft.Insights/dataCollectionRulesresource type.
Create a DCR with a direct ingestion endpoint
@description('Specifies the name of the data collection rule to create.') param dataCollectionRuleName string = '<DataCollectionRuleName>' @description('Specifies the region for the data collection rule.') param azureRegion string = '<AzureRegion>' @description('Specifies the subscription that contains the workspace.') param workspaceSubscriptionId string = '<WorkspaceSubscriptionId>' @description('Specifies the resource group that contains the workspace.') param workspaceResourceGroupName string = '<WorkspaceResourceGroupName>' @description('Specifies the name of the existing Log Analytics workspace.') param workspaceName string = '<WorkspaceName>' var workspaceResourceId = resourceId( workspaceSubscriptionId, workspaceResourceGroupName, 'Microsoft.OperationalInsights/workspaces', workspaceName ) resource dataCollectionRule 'Microsoft.Insights/dataCollectionRules@<ApiVersion>' = { name: dataCollectionRuleName location: azureRegion kind: 'Direct' properties: { streamDeclarations: { 'Custom-MyTable': { columns: [ { name: 'Time' type: 'datetime' } { name: 'Computer' type: 'string' } { name: 'AdditionalContext' type: 'string' } { name: 'CounterName' type: 'string' } { name: 'CounterValue' type: 'real' } ] } } destinations: { logAnalytics: [ { workspaceResourceId: workspaceResourceId name: 'myworkspace' } ] } dataFlows: [ { streams: [ 'Custom-MyTable' ] destinations: [ 'myworkspace' ] transformKql: 'source | extend jsonContext = parse_json(AdditionalContext) | project TimeGenerated = Time, Computer, AdditionalContext = jsonContext, CounterName=tostring(jsonContext.CounterName), CounterValue=toreal(jsonContext.CounterValue)' outputStream: 'Custom-MyTable_CL' } ] } } output dataCollectionRuleId string = dataCollectionRule.idData collection endpoint
Note
Notice the following details in the DCR defined in this template:
dataCollectionEndpointId: Resource ID of the data collection endpoint.streamDeclarations: Column definitions of the incoming data.destinations: Destination workspace.dataFlows: Matches the stream with the destination workspace and specifies the transformation query and the destination table. The output of the destination query is what gets sent to the destination table.
The following Bicep example uses the
Microsoft.Insights/dataCollectionRulesresource type.
Create a DCR that uses a data collection endpoint
@description('Specifies the name of the data collection rule to create.') param dataCollectionRuleName string = '<DataCollectionRuleName>' @description('Specifies the region for the data collection rule.') param azureRegion string = '<AzureRegion>' @description('Specifies the subscription that contains the workspace.') param workspaceSubscriptionId string = '<WorkspaceSubscriptionId>' @description('Specifies the resource group that contains the workspace.') param workspaceResourceGroupName string = '<WorkspaceResourceGroupName>' @description('Specifies the name of the existing Log Analytics workspace.') param workspaceName string = '<WorkspaceName>' @description('Specifies the subscription that contains the data collection endpoint.') param endpointSubscriptionId string = '<EndpointSubscriptionId>' @description('Specifies the resource group that contains the data collection endpoint.') param endpointResourceGroupName string = '<EndpointResourceGroupName>' @description('Specifies the name of the existing data collection endpoint.') param dataCollectionEndpointName string = '<DataCollectionEndpointName>' var workspaceResourceId = resourceId( workspaceSubscriptionId, workspaceResourceGroupName, 'Microsoft.OperationalInsights/workspaces', workspaceName ) var endpointResourceId = resourceId( endpointSubscriptionId, endpointResourceGroupName, 'Microsoft.Insights/dataCollectionEndpoints', dataCollectionEndpointName ) resource dataCollectionRule 'Microsoft.Insights/dataCollectionRules@<ApiVersion>' = { name: dataCollectionRuleName location: azureRegion kind: 'Direct' properties: { dataCollectionEndpointId: endpointResourceId streamDeclarations: { 'Custom-MyTable': { columns: [ { name: 'Time' type: 'datetime' } { name: 'Computer' type: 'string' } { name: 'AdditionalContext' type: 'string' } { name: 'CounterName' type: 'string' } { name: 'CounterValue' type: 'real' } ] } } destinations: { logAnalytics: [ { workspaceResourceId: workspaceResourceId name: 'myworkspace' } ] } dataFlows: [ { streams: [ 'Custom-MyTable' ] destinations: [ 'myworkspace' ] transformKql: 'source | extend jsonContext = parse_json(AdditionalContext) | project TimeGenerated = Time, Computer, AdditionalContext = jsonContext, CounterName=tostring(jsonContext.CounterName), CounterValue=toreal(jsonContext.CounterValue)' outputStream: 'Custom-MyTable_CL' } ] } } output dataCollectionRuleId string = dataCollectionRule.idOn the Custom deployment screen, specify a Subscription and Resource group to store the DCR. Provide
dataCollectionRuleName,azureRegion,workspaceSubscriptionId,workspaceResourceGroupName, andworkspaceName. For the DCE variant, also provideendpointSubscriptionId,endpointResourceGroupName, anddataCollectionEndpointName. SetazureRegionto the workspace's region.Select Review + create and then select Create after you review the details.
When the deployment is complete, expand the Deployment details box and select your DCR to view its details. Select JSON View.
Copy the Immutable ID and Logs ingestion URI for the DCR. You'll use these when you send data to Azure Monitor using the API.
Assign permissions to a DCR
After the DCR has been created, the application needs to be given permission to it. Permission will allow any application using the correct application ID and application key to send data to the new DCR.
From the DCR in the Azure portal, select Access Control (IAM) > Add role assignment.
Select Monitoring Metrics Publisher and select Next. You could instead create a custom action with the
Microsoft.Insights/Telemetry/Writedata action.Select User, group, or service principal for Assign access to and choose Select members. Select the application that you created and choose Select.
Select Review + assign and verify the details before you save your role assignment.
Sample code
See Sample code to send data to Azure Monitor using Logs ingestion API for sample code using the components created in this tutorial.