Azure Stream Analytics monitoring data reference

  • Article

This article contains all the monitoring reference information for this service.

See Monitor Azure Stream Analytics for details on the data you can collect for Azure Stream Analytics and how to use it.

Metrics

This section lists all the automatically collected platform metrics for this service. These metrics are also part of the global list of all platform metrics supported in Azure Monitor.

For information on metric retention, see Azure Monitor Metrics overview.

Supported metrics for Microsoft.StreamAnalytics/streamingjobs

The following table lists the metrics available for the Microsoft.StreamAnalytics/streamingjobs resource type.

  • All columns may not be present in every table.
  • Some columns might be beyond the viewing area of the page. Select Expand table to view all available columns.

Table headings

  • Category - The metrics group or classification.
  • Metric - The metric display name as it appears in the Azure portal.
  • Name in REST API - The metric name as referred to in the REST API.
  • Unit - Unit of measure.
  • Aggregation - The default aggregation type. Valid values: Average (Avg), Minimum (Min), Maximum (Max), Total (Sum), Count.
  • Dimensions - Dimensions available for the metric.
  • Time Grains - Intervals at which the metric is sampled. For example, PT1M indicates that the metric is sampled every minute, PT30M every 30 minutes, PT1H every hour, and so on.
  • DS Export- Whether the metric is exportable to Azure Monitor Logs via diagnostic settings. For information on exporting metrics, see Create diagnostic settings in Azure Monitor.

Metrics descriptions

Azure Stream Analytics provides the following metrics for you to monitor your job's health.

Metric Definition
Backlogged Input Events Number of input events that are backlogged. A nonzero value for this metric implies that your job can't keep up with the number of incoming events. If this value is slowly increasing or is consistently nonzero, you should scale out your job. To learn more, see Understand and adjust streaming units.
Data Conversion Errors Number of output events that couldn't be converted to the expected output schema. To drop events that encounter this scenario, you can change the error policy to Drop.
CPU % Utilization (preview) Percentage of CPU that your job utilizes. Even if this value is very high (90 percent or more), you shouldn't increase the number of SUs based on this metric alone. If the number of backlogged input events or watermark delays increases, you can then use this metric to determine if the CPU is the bottleneck.

This metric might have intermittent spikes. We recommend that you do scale tests to determine the upper bound of your job after which inputs are backlogged or watermark delays increase because of a CPU bottleneck.
Early Input Events Events whose application time stamp is earlier than their arrival time by more than 5 minutes.
Failed Function Requests Number of failed Azure Machine Learning function calls (if present).
Function Events Number of events sent to the Azure Machine Learning function (if present).
Function Requests Number of calls to the Azure Machine Learning function (if present).
Input Deserialization Errors Number of input events that couldn't be deserialized.
Input Event Bytes Amount of data that the Stream Analytics job receives, in bytes. You can use this metric to validate that events are being sent to the input source.
Input Events Number of records deserialized from the input events. This count doesn't include incoming events that result in deserialization errors. Stream Analytics can ingest the same events multiple times in scenarios like internal recoveries and self-joins. Don't expect Input Events and Output Events metrics to match if your job has a simple pass-through query.
Input Sources Received Number of messages that the job receives. For Azure Event Hubs, a message is a single EventData item. For Azure Blob Storage, a message is a single blob.

Note that input sources are counted before deserialization. If there are deserialization errors, input sources can be greater than input events. Otherwise, input sources can be less than or equal to input events because each message can contain multiple events.
Late Input Events Events that arrived later than the configured tolerance window for late arrivals. Learn more about Azure Stream Analytics event order considerations.
Out-of-Order Events Number of events received out of order that were either dropped or given an adjusted time stamp, based on the event ordering policy. This metric can be affected by the configuration of the Out-of-Order Tolerance Window setting.
Output Events Amount of data that the Stream Analytics job sends to the output target, in number of events.
Runtime Errors Total number of errors related to query processing. It excludes errors found while ingesting events or outputting results.
SU (Memory) % Utilization Percentage of memory that your job utilizes. If this metric is consistently over 80 percent, the watermark delay is rising, and the number of backlogged events is rising, consider increasing streaming units (SUs). High utilization indicates that the job is using close to the maximum allocated resources.
Watermark Delay Maximum watermark delay across all partitions of all outputs in the job.

Metric dimensions

For information about what metric dimensions are, see Multi-dimensional metrics.

This service has the following dimensions associated with its metrics.

  • Logical Name: The input or output name for an Azure Stream Analytics job.
  • Partition ID: The ID of the input data partition from an input source.
  • Node Name: The identifier of a streaming node that's provisioned when a job runs.

For detailed information, see Dimensions for Azure Stream Analytics metrics.

Resource logs

This section lists the types of resource logs you can collect for this service. The section pulls from the list of all resource logs category types supported in Azure Monitor.

For the resource logs schema and properties for data errors and events, see Resource logs schema.

Azure Monitor Logs tables

This section refers to all of the Azure Monitor Logs tables relevant to this service, which are available for query by Log Analytics using Kusto queries.

This service uses the following tables to store resource log data.

Activity log

The linked table lists the operations that may be recorded in the activity log for this service. This is a subset of all the possible resource provider operations in the activity log.

For more information on the schema of activity log entries, see Activity Log schema.

Related content