IoT 中心数据连接IoT Hub data connection

Azure IoT 中心是一项托管服务,承载在云中,充当中央消息中心,用于 IoT 应用程序与其管理的设备之间的双向通信。Azure IoT Hub is a managed service, hosted in the cloud, that acts as a central message hub for bi-directional communication between your IoT application and the devices it manages. Azure 数据资源管理器使用其与事件中心兼容的内置终结点提供从客户托管 IoT 中心进行的连续引入。Azure Data Explorer offers continuous ingestion from customer managed IoT Hubs, using its Event Hub compatible built in endpoint.

IoT 引入管道需要完成几个步骤。The IoT ingestion pipeline goes through several steps. 首先,创建一个 IoT 中心,并将设备注册到此 IoT 中心。First, you create an IoT Hub, and register a device to it. 然后,创建 Azure 数据资源管理器目标表,使用给定的引入属性特定格式的数据引入到该表中。You then create a target table in Azure Data Explorer into which the data in a particular format, will be ingested using the given ingestion properties. IoT 中心连接需要知道事件路由才能连接到 Azure 数据资源管理器表。The Iot Hub connection needs to know events routing to connect to the Azure Data Explorer table. 根据事件系统属性映射,使用选定的属性嵌入数据。Data is embedded with selected properties according to the event system properties mapping. 可以通过 Azure 门户使用 C#Python 以编程方式管理此过程,也可以使用 Azure 资源管理器模板来这样做。This process can be managed through the Azure portal, programmatically with C# or Python, or with the Azure Resource Manager template.

有关 Azure 数据资源管理器中数据引入的常规信息,请参阅 Azure 数据资源管理器数据引入概述For general information about data ingestion in Azure Data Explorer, see Azure Data Explorer data ingestion overview.

数据格式Data format

  • 将以 EventData 对象的形式从事件中心终结点读取数据。Data is read from the Event Hub endpoint in form of EventData objects.
  • 请参阅支持的格式See supported formats.

    备注

    IoT 中心不支持 .raw 格式。IoT Hub doesn't support the .raw format.

  • 请参阅支持的压缩See supported compressions.
    • 原始的未压缩数据大小应该是 blob 元数据的一部分,否则 Azure 数据资源管理器会对其进行估算。The original uncompressed data size should be part of the blob metadata, or else Azure Data Explorer will estimate it. 每个文件的引入未压缩大小限制为 4 GB。The ingestion uncompressed size limit per file is 4 GB.

引入属性Ingestion properties

引入属性指示引入过程将数据路由到何处以及如何对其进行处理。Ingestion properties instruct the ingestion process where to route the data and how to process it. 可以使用 EventData.Properties 指定事件的引入属性You can specify Ingestion properties of the events using the EventData.Properties. 可以设置以下属性:You can set the following properties:

属性Property 说明Description
Table 现有目标表的名称(区分大小写)。Name (case sensitive) of the existing target table. 替代“Data Connection”窗格上设置的“Table”。Overrides the Table set on the Data Connection pane.
格式Format 数据格式。Data format. 替代“Data Connection”窗格上设置的“Data format”。Overrides the Data format set on the Data Connection pane.
IngestionMappingReferenceIngestionMappingReference 要使用的现有引入映射的名称。Name of the existing ingestion mapping to be used. 替代“Data Connection”窗格上设置的“Column mapping”。Overrides the Column mapping set on the Data Connection pane.
编码Encoding 数据编码,默认值为 UTF8。Data encoding, the default is UTF8. 可以是 .NET 支持的任何编码Can be any of .NET supported encodings.

备注

只有创建数据连接后进入队列的事件才会被引入。Only events enqueued after you create the data connection are ingested.

事件路由Events routing

设置到 Azure 数据资源管理器群集的 IoT 中心连接时,请指定目标表属性(表名、数据格式和映射)。When setting up an IoT Hub connection to Azure Data Explorer cluster, you specify target table properties (table name, data format, and mapping). 此设置是用于你的数据的默认路由,也称为“静态路由”。This setting is the default routing for your data, also referred to as static routing. 还可以使用事件属性指定每个事件的目标表属性。You can also specify target table properties for each event, using event properties. 连接将按照 EventData.Properties 中指定的要求动态路由数据,替代此事件的静态属性。The connection will dynamically route the data as specified in the EventData.Properties, overriding the static properties for this event.

备注

如果选择了“我的数据包括路由信息”,则必须提供必要的路由信息作为事件属性的一部分。If My data includes routing info selected, you must provide the necessary routing information as part of the events properties.

事件系统属性映射Event system properties mapping

系统属性是一个集合,用于存储收到事件时由 IoT 中心服务设置的属性。System properties are a collection used to store properties that are set by the IoT Hub service, on the time the event is received. Azure 数据资源管理器 IoT 中心连接会将所选属性嵌入置于表中的数据中。The Azure Data Explorer IoT Hub connection will embed the selected properties in the data landing in your table.

备注

对于 csv 映射,属性将按下表中列出的顺序添加到记录的开头。For csv mapping, properties are added at the beginning of the record in the order listed in the table below. 对于 json 映射,将根据下表中的属性名称添加属性。For json mapping, properties are added according to property names in the following table.

系统属性System properties

IoT 中心公开了以下系统属性:IoT Hub exposes the following system properties:

属性Property 说明Description
message-idmessage-id 用户可设置的消息标识符,用于请求-答复模式。A user-settable identifier for the message used for request-reply patterns.
sequence-numbersequence-number IoT 中心分配给每条云到设备消息的编号(对每个设备队列是唯一的)。A number (unique per device-queue) assigned by IoT Hub to each cloud-to-device message.
toto 在云到设备消息中指定的目标。A destination specified in Cloud-to-Device messages.
absolute-expiry-timeabsolute-expiry-time 消息过期的日期和时间。Date and time of message expiration.
iothub-enqueuedtimeiothub-enqueuedtime IoT 中心收到设备到云消息的日期和时间。Date and time the Device-to-Cloud message was received by IoT Hub.
correlation-idcorrelation-id 响应消息中的字符串属性,通常包含采用“请求-答复”模式的请求的 MessageId。A string property in a response message that typically contains the MessageId of the request, in request-reply patterns.
user-iduser-id 用于指定消息的源的 ID。An ID used to specify the origin of messages.
iothub-ackiothub-ack 反馈消息生成器。A feedback message generator.
iothub-connection-device-idiothub-connection-device-id IoT 中心对设备到云的消息设置的 ID。An ID set by IoT Hub on device-to-cloud messages. 它包含发送了消息的设备的 deviceId。It contains the deviceId of the device that sent the message.
iothub-connection-auth-generation-idiothub-connection-auth-generation-id IoT 中心对设备到云的消息设置的 ID。An ID set by IoT Hub on device-to-cloud messages. 它包含发送了消息的设备的 connectionDeviceGenerationId(取决于设备标识属性)。It contains the connectionDeviceGenerationId (as per Device identity properties) of the device that sent the message.
iothub-connection-auth-methodiothub-connection-auth-method 由 IoT 中心对设备到云的消息设置的身份验证方法。An authentication method set by IoT Hub on device-to-cloud messages. 此属性包含用于验证发送消息的设备的身份验证方法的相关信息。This property contains information about the authentication method used to authenticate the device sending the message.

如果在表的“数据源”部分中选择了“事件系统属性”,则必须在表架构和映射中包含这些属性。If you selected Event system properties in the Data Source section of the table, you must include the properties in the table schema and mapping.

架构映射示例Schema mapping examples

表架构映射示例Table schema mapping example

如果数据包含三列(TimespanMetricValue)并且包含的属性是 x-opt-enqueued-timex-opt-offset,请使用以下命令创建或更改表架构:If your data includes three columns (Timespan, Metric, and Value) and the properties you include are x-opt-enqueued-time and x-opt-offset, create or alter the table schema by using this command:

    .create-merge table TestTable (TimeStamp: datetime, Metric: string, Value: int, EventHubEnqueuedTime:datetime, EventHubOffset:string)

CSV 映射示例CSV mapping example

运行以下命令,将数据添加到记录的开头。Run the following commands to add data to the beginning of the record. 记下序号值。Note ordinal values.

    .create table TestTable ingestion csv mapping "CsvMapping1"
    '['
    '   { "column" : "Timespan", "Properties":{"Ordinal":"2"}},'
    '   { "column" : "Metric", "Properties":{"Ordinal":"3"}},'
    '   { "column" : "Value", "Properties":{"Ordinal":"4"}},'
    '   { "column" : "EventHubEnqueuedTime", "Properties":{"Ordinal":"0"}},'
    '   { "column" : "EventHubOffset", "Properties":{"Ordinal":"1"}}'
    ']'

JSON 映射示例JSON mapping example

使用系统属性映射添加数据。Data is added by using the system properties mapping. 运行以下命令:Run these commands:

    .create table TestTable ingestion json mapping "JsonMapping1"
    '['
    '    { "column" : "Timespan", "Properties":{"Path":"$.timestamp"}},'
    '    { "column" : "Metric", "Properties":{"Path":"$.metric"}},'
    '    { "column" : "Value", "Properties":{"Path":"$.metric_value"}},'
    '    { "column" : "EventHubEnqueuedTime", "Properties":{"Path":"$.x-opt-enqueued-time"}},'
    '    { "column" : "EventHubOffset", "Properties":{"Path":"$.x-opt-offset"}}'
    ']'

IoT 中心连接IoT Hub connection

备注

为了获得最佳性能,请在 Azure 数据资源管理器群集所在的区域中创建所有资源。For best performance, create all resources in the same region as the Azure Data Explorer cluster.

创建 IoT 中心Create an IoT Hub

创建 IoT 中心(如果还没有 IoT 中心)。If you don't already have one, Create an Iot Hub. 可以通过 Azure 门户使用 C#Python 以编程方式管理到 IoT 中心的连接,也可以使用 Azure 资源管理器模板来这样做。Connection to IoT Hub can be managed through the Azure portal, programmatically with C# or Python, or with the Azure Resource Manager template.

备注

  • device-to-cloud partitions 计数不可更改,因此在设置分区计数时应考虑长期规模。The device-to-cloud partitions count is not changeable, so you should consider long-term scale when setting partition count.
  • 使用者组对于每个使用者来说必须独一无二。Consumer group must be unique per consumer. 创建专用于 Azure 数据资源管理器连接的使用者组。Create a consumer group dedicated to Azure Data Explorer connection. 在 Azure 门户中找到你的资源,然后转到“Built-in endpoints”,以便添加新的使用者组。Find your resource in the Azure portal and go to Built-in endpoints to add a new consumer group.

发送事件Sending events

请参阅用于模拟设备并生成数据的示例项目See the sample project that simulates a device and generates data.

后续步骤Next steps

可以通过多种方法将数据引入 IoT 中心。There are various methods to ingest data to IoT Hub. 有关每种方法的演练,请参阅以下链接。See the following links for walkthroughs of each method.