gRPC 扩展协议gRPC extension protocol

在本文中,你将学习如何使用 gRPC 扩展协议在实时视频分析模块与 AI 或 CV 自定义扩展之间发送消息。In this article, you will learn about using gRPC extension protocol to send messages between Live Video Analytics module and your AI or CV custom extension.

gRPC 是一种新式的高性能开源 RPC 框架,可在任何环境中运行。gRPC is a modern, open-source, high-performance RPC framework that runs in any environment. gRPC 传输服务在以下两者之间使用 HTTP/2 双向流式传输:The gRPC transport service uses HTTP/2 bidirectional streaming between:

  • gRPC 客户端(IoT Edge 上的实时视频分析模块)与the gRPC client (Live Video Analytics on IoT Edge module) and
  • gRPC 服务器(自定义扩展)。the gRPC server (your custom extension).

gRPC 会话是通过 TCP/TLS 端口从 gRPC 客户端到 gRPC 服务器的单一连接。A gRPC session is a single connection from the gRPC client to the gRPC server over the TCP/TLS port.

在单个会话中:客户端通过 gRPC 流会话将媒体流描述符(后跟视频帧)作为 protobuf 消息发送到服务器。In a single session: The client sends a media stream descriptor followed by video frames to the server as a protobuf message over the gRPC stream session. 服务器验证流描述符、分析视频帧,并将推理结果作为 protobuf 消息返回。The server validates the stream descriptor, analyses the video frame, and returns inference results as a protobuf message.

gRPC 扩展协定

实现 gRPC 协议Implementing gRPC protocol

创建 gRPC 连接Creating a gRPC connection

自定义扩展插件必须实现以下 gRPC 服务:Custom extension must implement the following gRPC service:

service MediaGraphExtension {
  rpc ProcessMediaStream(stream MediaStreamMessage) returns (stream MediaStreamMessage);
}

调用时,将打开双向流,以便消息在 gRPC 扩展和实时视频分析图之间流动。When called, this will open a bi-directional stream for messages to flow between the gRPC extension and Live Video Analytics graph. 每个参与方在此流中发送的第一条消息都将包含一个 MediaStreamDescriptor,它定义将在后面的 MediaSample 中发送的具体信息。The first message sent in this stream by each party will contain a MediaStreamDescriptor, which defines what information will be sent in the following MediaSamples.

例如,图形扩展可发送该消息(此处以 JSON 表示),表示它会将 gRPC 消息中嵌入的 416x416 rgb24 编码的帧发送到自定义扩展。For example, the graph extension may send the message (expressed here in JSON) to indicate that it will send 416x416 rgb24-encoded frames embedded in the gRPC messages to the custom extension.

 {
    "sequence_number": 1,
    "ack_sequence_number": 0,
    "media_stream_descriptor": {
        "graph_identifier": {
            "media_services_arm_id": "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/resourceGroupName/providers/microsoft.media/mediaservices/mediaAccountName",
            "graph_instance_name": "mediaGraphName",
            "graph_node_name": "grpcExtension"
        },
        "media_descriptor": {
            "timescale": 90000,
            "video_frame_sample_format": {
                "encoding": "RAW",
                "pixel_format": "RGB24",
                "dimensions": {
                    "width": 416,
                    "height": 416
                },
                "stride_bytes": 1248
            }
        }
    }
}

自定义扩展将发送以下消息作为响应,该消息表示它仅返回推理。The custom extension would, in response, send the following message to indicate that it is returning inferences only.

{
    "sequence_number": 1,
    "ack_sequence_number": 1,
    "media_stream_descriptor": {
        "extension_identifier": "customExtensionName"    }
}

现在双方已交换媒体描述符,接下来实时视频分析将开始将帧传输到该扩展。Now that both sides have exchanged media descriptors, Live Video Analytics will start transmitting frames to the extension.

序列号Sequence numbers

gRPC 扩展节点和自定义扩展都将保留一组各自分配给其消息的序列号。Both the gRPC extension node and the custom extension maintain a separate set of sequence numbers, which are assigned to their messages. 这些序列号应从 1 开始单调递增。These sequence numbers should monotonically increase starting from 1. 如果未确认任何消息,则可忽略 ack_sequence_number,发送第一条消息时可能会发生此情况。ack_sequence_number can be ignored if no message is being acknowledged, which may occur when the first message sent.

完全处理相应的消息后,必须确认请求。A request must be acknowledged once the corresponding message has been fully processed. 如果是共享内存传输,则此确认表示发送方可释放共享内存,而接收方将不再访问它。In the case of a shared memory transfer, this acknowledgment indicates that sender may free the shared memory and that the receiver will not access it anymore. 在确认之前,发送方必须保留任何共享内存。The sender must hold any shared memory until it has been acknowledged.

读取嵌入式内容Reading embedded content

可通过 content_byte 字段直接从 MediaSample 消息中读取嵌入式内容。Embedded content may be read directly out of the MediaSample message through the content_bytes field. 将根据 MediaDescriptor 对数据进行编码。The data will be encoded according to the MediaDescriptor.

从共享内存读取内容Reading Content from Shared Memory

通过共享内存传输内容时,发送方会在首次建立会话时将 SharedMemoryBufferTransferProperties 包含在其 MediaStreamDescriptor 中。When transferring content through shared memory, the sender will include SharedMemoryBufferTransferProperties in their MediaStreamDescriptor when they first establish a session. 此内容如下所示(以 JSON 表示):This may look as follows (expressed in JSON):

{
  "handle_name": "inference_client_share_memory_2146989006636459346"
  "length_bytes": 20971520
}

然后,接收方打开文件 /dev/shm/inference_client_share_memory_2146989006636459346The receiver then opens the file /dev/shm/inference_client_share_memory_2146989006636459346. 发件人将发送 MediaSample 消息,其中包含引用此文件中特定位置的 ContentReferenceThe sender will send a MediaSample message, which contains a ContentReference referring to a specific place in this file.

{
    "timestamp": 143598615750000,
    "content_reference": {
        "address_offset": 519168,
        "length_bytes": 173056
    }
}

然后,接收方从文件中的此位置读取数据。The receiver then reads the data from this location in the file.

为了使实时视频分析容器能够通过共享内存通信,必须正确配置容器的 IPC 模式。For the Live Video Analytics container to communicate over shared memory, the IPC mode of the container must be configured correctly. 可采用多种方式完成此操作,但这里提供了一些建议的配置。This can be done in many ways, but here are some recommended configurations.

  • 与在主机设备上运行的 gRPC 推理引擎通信时,应将 IPC 模式设置为“主机”。When communicating with a gRPC inferencing engine running on the host device, the IPC mode should be set to host.
  • 与在另一 IoT Edge 模块中运行的 gRPC 服务器通信时,应将 IPC 模式设置为可对实时视频分析模块共享,并为自定义扩展设置 container:liveVideoAnalytics,其中 liveVideoAnalytics 是实时视频分析模块的名称。When communicating with a gRPC server running in another IoT Edge module, the IPC mode should be set to shareable for the Live Video Analytics module and container:liveVideoAnalytics for the custom extension, where liveVideoAnalytics is the name of the Live Video Analytics module.

下面是使用上述第一个选项时,设备孪生中可能呈现的内容。Here's what this might look like in the device twin using the first option from above.

"liveVideoAnalytics": {
  "version": "1.0",
  "type": "docker",
  "status": "running",
  "restartPolicy": "always",
  "settings": {
    "image": "mcr.microsoft.com/media/live-video-analytics:1",
    "createOptions": 
      "HostConfig": {
        "IpcMode": "host"
      }
    }
  }
}

有关 IPC 模式的详细信息,请参阅 https://docs.docker.com/engine/reference/run/#ipc-settings---ipcFor more information on IPC modes, see https://docs.docker.com/engine/reference/run/#ipc-settings---ipc.

媒体图 gRPC 扩展协定定义Media graph gRPC extension contract definitions

本部分定义可定义数据流的 gRPC 协定。This section defines the gRPC contract that defines the data flow.

协议消息Protocol messages

协议消息

客户端身份验证Client authentication

自定义扩展的实施者可验证传入的 gRPC 连接的真实性,以确保这些连接来自 gRPC 扩展节点。Implementers of custom extensions can validate the authenticity of incoming gRPC connections to be sure that they are coming from the gRPC extension node. 该节点将在请求头中提供要验证的条目。The node will provide an entry in the request headers to validate against.

可使用用户名/密码凭据来完成此操作。Username/password credentials can be used to accomplish this. 创建 gRPC 扩展节点时,将提供如下所示的凭据:When creating a gRPC extension node, the credentials are provided like below:

{
  "@type": "#Microsoft.Media.MediaGraphGrpcExtension",
  "name": "{moduleIdentifier}",
  "endpoint": {
    "@type": "#Microsoft.Media.MediaGraphUnsecuredEndpoint",
    "url": "tcp://customExtension:8081",
    "credentials": {
      "@type": "#Microsoft.Media.MediaGraphUsernamePasswordCredentials",
      "username": "username",
      "password": "password"
    }
  }
  // Other fields omitted
}

发送 gRPC 请求时,以下标头将包含在请求元数据中,模仿 HTTP 基本身份验证。When the gRPC request is sent, the following header will be included in the request metadata, mimicking HTTP Basic authentication.

x-ms-authentication: Basic (Base64 Encoded username:password)

通过 TLS 使用 gRPCUsing gRPC over TLS

可通过 TLS 为用于推理的 gRPC 连接提供保护。A gRPC connection used for inferencing may be secured over TLS. 无法保证实时视频分析和推理引擎之间的网络安全时,这非常有用。This is useful in situations where the security of the network between Live Video Analytics and the inferencing engine cannot be guaranteed. TLS 会对嵌入 gRPC 消息中的任何内容进行加密,导致在以高速率传输帧时产生额外的 CPU 开销。TLS will encrypt any content embedded into the gRPC messages, causing additional CPU overhead when transmitting frames at a high rate.

gRPC 不支持 IgnoreHostname 和 IgnoreSignature 验证选项,因此推理引擎提供的服务器证书必须包含与 gRPC 扩展节点的终结点 URL 中 IP 地址/主机名完全匹配的 CN。The IgnoreHostname and IgnoreSignature verification options are not supported by gRPC, so the server certificate, which the inferencing engine presents must contain a CN that matches exactly with the IP address/hostname in the gRPC extension node’s endpoint URL.

后续步骤Next steps

了解推理元数据架构Learn about the Inference Metadata Schema