媒体图Media graph

建议的读前准备Suggested pre-reading


通过使用媒体图,你可以定义应从何处捕获媒体、应如何处理媒体以及应将结果交付到何处。A media graph lets you define where media should be captured from, how it should be processed, and where the results should be delivered. 可以采用所需方式连接组件或节点来进行定义。You accomplish this by connecting components, or nodes, in the desired manner. 下图提供了媒体图的图形表示形式。The diagram below provides a graphical representation of a media graph.


IoT Edge 上的实时视频分析支持不同类型的源、处理器和接收器。Live Video Analytics on IoT Edge supports different types of sources, processors, and sinks.

  • 源节点 支持将媒体捕获到媒体图中。Source nodes enable capturing of media into the media graph. 从概念上讲,此上下文中的媒体可能是音频流、视频流、数据流或将音频、视频和/或数据组合在单个流中的流。Media in this context, conceptually, could be an audio stream, a video stream, a data stream, or a stream that has audio, video, and/or data combined together in a single stream.
  • 处理器节点 支持处理媒体图中的媒体。Processor nodes enable processing of media within the media graph.
  • 接收器节点 支持将处理结果交付到媒体图之外的服务和应用。Sink nodes enable delivering the processing results to services and apps outside the media graph.

媒体图拓扑和实例Media graph topologies and instances

通过 IoT Edge 上的实时视频分析,你可以通过两个概念来管理媒体图 -“图形拓扑”和“图形实例”。Live Video Analytics on IoT Edge enables you to manage media graphs via two concepts – "graph topology" and "graph instance". 通过图形拓扑,你可以定义图形的蓝图,其中参数作为值的占位符。A graph topology enables you to define the blueprint of a graph, with parameters as placeholders for values. 拓扑定义媒体图中使用的节点,以及它们在媒体图中的连接方式。The topology defines what nodes are used in the media graph, and how they are connected within the media graph. 例如,如果要从照相机录制源,则需要包含接收视频的源节点和写入视频的接收器节点的图形。For example, if you wanted to record the feed from a camera, then you would need a graph with a source node that receives video, and a sink node that writes the video.

拓扑中参数的值是在你创建引用拓扑的图形实例时指定的。The values for the parameters in the topology are specified when you create graph instances that reference the topology. 这使你能够创建多个引用相同拓扑的实例,但拓扑中指定的参数的值却不同。This enables you to create multiple instances referencing the same topology but with different values for the parameters specified in the topology. 在上面的示例中,可以使用参数来表示照相机的 IP 地址以及录制的视频的名称。In the above example, you could have used parameters to represent the camera's IP address, and the name for the recorded video. 可以使用该拓扑来创建许多图形实例 - 生成内容中每个照相机都对应一个实例,可能各自具有特定的 IP 地址和特定名称。You could create many graph instances with that topology - one instance for each camera in a building, perhaps, each with the specific IP address and specific name.

媒体图状态Media graph states

以下状态图显示了图形拓扑和图形实例的生命周期。The lifecycle of graph topologies and graph instances is shown in the following state diagram.


首先创建图形拓扑You begin with creating a graph topology. 然后对每个要使用此拓扑处理的实时视频源,创建图形实例Then, for each live video feed that you want to process with this topology, you create a graph instance.

图形实例将处于 Inactive(空闲)状态。The graph instance will be in the Inactive (idle) state.

准备好将实时视频源发送到图形实例时,激活该实例。When you are ready to send the live video feed into the graph instance, you activate it. 图形实例将短暂为过渡的 Activating 状态,如果成功,则进入 Active 状态。The graph instance will briefly go through a transitionary Activating state, and if successful, go into an Active state. Active 状态下,将会处理媒体(如果图形实例接收了输入数据)。In the Active state, media will be processed (if the graph instance receives input data).


图形实例可以在没有数据流通过它时处于活动状态(例如相机离线)。A graph instance can be active without data flowing through it (for example, the camera goes offline). 当图形实例处于活动状态时,将对你的 Azure 订阅进行计费。Your Azure subscription will be billed when the graph instance is in the active state.

如果有其他实时视频源要处理,则可以重复为同一拓扑创建和激活其他图形实例。You can repeat the process of creating and activating other graph instances for the same topology, if you have other live video feeds to process.

处理完实时视频源后,可以停用该图形实例。When you are done processing the live video feed, you can deactivate the graph instance. 图形实例将短暂为过渡的 Deactivating 状态,刷新其包含的所有数据,然后返回到 Inactive 状态。The graph instance will briefly go through a transitionary Deactivating state, flush any data that it has, and then return to the Inactive state.

仅当图形实例处于 Inactive 状态时,才能将其删除You can only delete a graph instance when it is in the Inactive state.

删除引用特定图形拓扑的所有图形实例后,可以删除图形拓扑After all graph instances that refer to a specific graph topology have been deleted, you can delete the graph topology.

源、处理器和接收器Sources, processors, and sinks

IoT Edge 上的实时视频分析支持媒体图中以下类型的节点:Live Video Analytics on IoT Edge supports the following types of nodes within a media graph:


RTSP 源RTSP source

使用 RTSP 源节点,你可以从 RTSP 服务器引入媒体。An RTSP source node enables you to ingest media from a RTSP server. 监控和基于 IP 的照相机使用名为 RTSP(实时流式处理协议)的协议传输它们的数据,这不同于其他类型的设备(例如手机和摄像机)。Surveillance and IP-based cameras transmit their data in a protocol called RTSP (real-time-streaming-protocol) which is different than other types of devices like phones and video cameras. 此协议用于建立和控制服务器(照相机)和客户端之间的媒体会话。This protocol is used for establishing and controlling the media sessions between a server (the camera) and a client. 媒体图中的 RTSP 源节点充当客户端,可以与 RTSP 服务器建立会话。The RTSP source node in a media graph acts as a client and can establish a session with an RTSP server. 许多设备(例如大多数 IP 照相机)有内置的 RTSP 服务器。Many devices such as most IP cameras have a built-in RTSP server. ONVIF 强制要求在 Profile G、S 和 T 兼容设备的定义中支持 RTSP。ONVIF mandates RTSP to be supported in its definition of Profiles G, S & T compliant devices. RTSP 源节点要求你指定 RTSP URL 以及凭据才能启用经过身份验证的连接。The RTSP source node requires you to specify an RTSP URL, along with credentials to enable an authenticated connection.

IoT 中心消息源IoT Hub message source

与其他 IoT Edge 模块一样,IoT Edge 上的实时视频分析模块可通过 IoT Edge 中心接收消息。Like other IoT Edge modules, Live Video Analytics on IoT Edge module can receive messages via the IoT Edge hub. 这些消息可能发送自其他模块或在 Edge 设备上运行的应用,或者发送自云。These messages can be sent from other modules, or apps running on the Edge device, or from the cloud. 此类消息会传递(路由)到模块上已命名的输出Such messages are delivered (routed) to a named input on the module. IoT 中心消息源节点使此类消息能够到达媒体图。An IoT Hub message source node allows such messages to reach a media graph. 这些消息或信号随后可以在媒体图内部使用,通常用于激活信号门(请参阅下面的信号门)。These messages or signals can then be used internally in the media graph, typically to activate signal gates (see signal gates below).

例如,你可以有一个在门打开时生成消息的 IoT Edge 模块。For example, you can have an IoT Edge module that generates a message when a door is opened. 来自该模块的消息可以路由到 IoT Edge 中心,然后可以从该中心路由到媒体图的 IoT 中心消息源。The message from that module can be routed to IoT Edge hub, from where it can be then routed to the IoT hub message source of a media graph. 在媒体图中,IoT 中心消息源可以将事件传递给信号门处理器,该处理器随后可以打开从 RTSP 源到文件的视频录制。Within the media graph, the IoT hub message source can pass the event to a signal gate processor, which can then turn on recording of the video from an RTSP source into a file.


运动检测处理器Motion detection processor

通过运动检测处理器节点,你可以在实时视频中检测运动。The motion detection processor node enables you to detect motion in live video. 它会检查传入的视频帧并确定视频中是否有移动。It examines incoming video frames and determines if there is movement in the video. 如果检测到运动,它会将视频帧传递到下游组件,并发出事件。If motion is detected, it passes on the video frames to the downstream component, and emits an event. 运动检测处理器节点(连同其他节点)可用于在检测到运动时触发传入视频的录制。The motion detection processor node (in conjunction with other nodes) can be used to trigger recording of the incoming video when there is motion detected.

帧速率筛选器处理器Frame rate filter processor

通过帧速率筛选器处理器节点,你可以采用指定的速率从传入的视频流中采样帧。The frame rate filter processor node enables you to sample frames from the incoming video stream at a specified rate. 这使你能够减少发送到下游组件(例如 HTTP 扩展处理器节点)以进行进一步处理的帧数。This enables you to reduce the number of frames sent to down-stream components (such as HTTP extension processor node) for further processing.


在最新发布的基于 IoT Edge 模块的实时视频分析中,此处理器已弃用。This processor is deprecated in the latest release of Live Video Analytics on IoT Edge module. 图形扩展处理器本身现在支持帧速率管理。Frame Rate management is now supported within the graph extension processors itself.

HTTP 扩展处理器HTTP extension processor

通过 HTTP 扩展处理器节点,你可以将自己的 IoT Edge 模块连接到媒体图。The HTTP extension processor node enables you to connect your own IoT Edge module to a media graph. 此节点以解码的视频帧作为输入,并将此类帧中继到模块公开的 HTTP REST 终结点。This node takes decoded video frames as the input, and relays such frames to a HTTP REST endpoint exposed by your module. 如果需要,此节点能够使用 REST 终结点进行身份验证。This node has the ability to authenticate with the REST endpoint if required. 此外,此节点具有内置的图像格式化程序,用于在视频帧中继到 REST 终结点之前对它们进行缩放和编码。Additionally, the node has a built-in image formatter for scaling and encoding of video frames before they are relayed to the REST endpoint. 缩放程序可以对图像纵横比进行保留、填充或拉伸。The scaler has options for the image aspect ratio to be preserved, padded or stretched. 图像编码器支持 JPEG、PNG 或 BMP 格式。The image encoder supports JPEG, PNG, or BMP formats. 请在此处详细了解处理器。Learn more about the processor here.

gRPC 扩展处理器gRPC extension processor

此 gRPC 扩展处理器节点以解码的视频帧作为输入,并将此类帧中继到模块公开的 gRPC 终结点。The gRPC extension processor node takes decoded video frames as the input, and relays such frames to a gRPC endpoint exposed by your module. 节点支持使用共享内存传输数据,或将内容直接嵌入 gRPC 消息的正文中。The node supports transferring of data using shared memory or directly embedding the content into the body of gRPC messages. 此外,此节点具有内置的图像格式化程序,用于在视频帧中继到 gRPC 终结点之前对它们进行缩放和编码。Additionally, the node has a built-in image formatter for scaling and encoding of video frames before they are relayed to the gRPC endpoint. 缩放程序可以对图像纵横比进行保留、填充或拉伸。The scaler has options for the image aspect ratio to be preserved, padded or stretched. 图像编码器支持 jpeg、png 或 bmp 格式。The image encoder supports jpeg, png, or bmp formats. 请在此处详细了解处理器。Learn more about the processor here.

信号门处理器Signal gate processor

通过信号门处理器节点,你可以有条件地将媒体从一个节点转发到另一个节点。The signal gate processor node enables you to conditionally forward media from one node to another. 它还充当缓冲区,支持媒体和事件的同步。It also acts as a buffer, allowing for synchronization of media and events. 典型的用例是将信号门处理器节点插入 RTSP 源节点和资产接收器节点之间,并使用运动检测器处理器节点的输出触发门。A typical use case is to insert a signal gate processor node between the RTSP source node and the asset sink node, and using the output of a motion detector processor node to trigger the gate. 使用这样的媒体图时,你只有在检测到运动时才录制视频。With such a media graph, you would be recording video only when motion is detected.


资产接收器Asset sink

通过资产接收器节点,你可以将媒体(视频和/或音频)数据写入 Azure 媒体服务资产。An asset sink node enables you to write media (video and/or audio) data to an Azure Media Services asset. 媒体图中只能有一个资产接收器节点。There can only be one asset sink node in a media graph. 若要详细了解资产以及它们在媒体录制和播放方面的作用,请参阅资产部分。See the asset section for more information about assets, and their role in recording and playback of media. 此外,还可以查看连续视频录制一文,了解如何使用此节点的属性的详细信息。You can also see the continuous video recording article for details on how the properties of this node are used.

文件接收器File sink

通过文件接收器节点,你可以将媒体(视频和/或音频)数据写入 IoT Edge 设备本地文件系统上的位置。The file sink node enables you to write media (video and/or audio) data to a location on the local file system of the IoT Edge device. 媒体图中只能有一个文件接收器节点,并且它必须是信号门处理器节点的下游。There can only be one file sink node in a media graph, and it must be downstream of a signal gate processor node. 这会将输出文件的持续时间限制为信号门处理器节点属性中指定的值。This limits the duration of the output files to values specified in the signal gate processor node properties. 为确保边缘设备不会耗尽磁盘空间,还可以设置一个最大大小,供 IoT Edge 模块上的实时视频分析用来存储数据。To ensure that that your edge device does not run out of disk space, you can also set the maximum size that the Live Video Analytics on IoT Edge module can use to store data.


如果文件接收器已满,则 IoT Edge 模块上的实时视频分析就会开始删除最旧的数据,将其替换为新数据。If the File Sink gets full, the Live Video Analytics on IoT Edge module will start deleting the oldest data and replace it with the new one.

IoT 中心消息接收器IoT Hub message sink

通过 IoT 中心消息接收器节点,你可以将事件发布到 IoT Edge 中心。An IoT Hub message sink node enables you to publish events to IoT Edge hub. IoT Edge 中心随后可以将数据路由到其他模块或边缘设备上的应用,或路由到云中的 IoT 中心(根据部署清单中指定的路由)。The IoT Edge hub can then route the data to other modules or apps on the edge device, or to IoT Hub in the cloud (per routes specified in the deployment manifest). IoT 中心消息接收器节点可以接受来自上游处理器(例如运动检测处理器节点)的事件,或通过 HTTP 扩展处理器节点接受来自外部推理服务的事件。The IoT Hub message sink node can accept events from upstream processors such as a motion detection processor node, or from an external inference service via a HTTP extension processor node.

使用节点的规则Rules on the use of nodes

有关如何在媒体图中使用不同节点的其他规则,请参阅图形拓扑的限制See limitations on graph topologies for additional rules on how different nodes can be used within a media graph.


使用上面定义的源、处理器和接收器组合,你可以为涉及实时视频分析的各种方案构建媒体图。Using a combination of the sources, processors, and sinks defined above, you can build media graphs for a variety of scenarios involving analysis of live video. 示例方案包括:Example scenarios are:

后续步骤Next steps

若要了解如何在实时视频源上运行运动检测,请参阅快速入门:使用自己的模型运行实时视频分析To see how you can run motion detection on a live video feed, see Quickstart: Run Live Video Analytics with your own model.