IoT Edge 上的实时视频分析常见问题解答Live Video Analytics on IoT Edge FAQ

本文解答了有关 Azure IoT Edge 上的实时视频分析的常见问题。This article answers commonly asked questions about Live Video Analytics on Azure IoT Edge.

常规General

我可以在图拓扑定义中使用哪些系统变量?What system variables can I use in the graph topology definition?

变量Variable 说明Description
System.DateTimeSystem.DateTime 表示某个 UTC 时刻,通常以日期和当天的时间表示,格式如下:Represents an instant in UTC time, typically expressed as a date and time of day in the following format:
yyyyMMddTHHmmssZyyyyMMddTHHmmssZ
System.PreciseDateTimeSystem.PreciseDateTime 表示 ISO8601 文件兼容格式的协调世界时 (UTC) 日期/时间实例(精确到毫秒),格式如下:Represents a Coordinated Universal Time (UTC) date-time instance in an ISO8601 file-compliant format with milliseconds, in the following format:
yyyyMMddTHHmmss.fffZyyyyMMddTHHmmss.fffZ
System.GraphTopologyNameSystem.GraphTopologyName 表示媒体图拓扑,保存图的蓝图。Represents a media graph topology, and holds the blueprint of a graph.
System.GraphInstanceNameSystem.GraphInstanceName 表示媒体图实例,保存参数值并引用拓扑。Represents a media graph instance, holds parameter values, and references the topology.

配置和部署Configuration and deployment

是否可以将媒体边缘模块部署到 Windows 10 设备?Can I deploy the media edge module to a Windows 10 device?

是的。Yes. 有关详细信息,请参阅 Windows 10 上的 Linux 容器For more information, see Linux containers on Windows 10.

从 IP 相机和 RTSP 设置捕获Capture from IP camera and RTSP settings

是否需要在设备上使用特殊的 SDK 来发送视频流?Do I need to use a special SDK on my device to send in a video stream?

否。IoT Edge 上的实时视频分析支持使用 RTSP(实时流式处理协议)进行视频流式处理来捕获媒体,大多数 IP 相机支持此协议。No, Live Video Analytics on IoT Edge supports capturing media by using RTSP (Real-Time Streaming Protocol) for video streaming, which is supported on most IP cameras.

是否可以使用实时消息传送协议 (RTMP) 或平滑流式处理协议(如媒体服务实时事件)将媒体推送到 IoT Edge 上的实时视频分析?Can I push media to Live Video Analytics on IoT Edge by using Real-Time Messaging Protocol (RTMP) or Smooth Streaming Protocol (such as a Media Services Live Event)?

否。实时视频分析仅支持使用 RTSP 从 IP 相机捕获视频。No, Live Video Analytics supports only RTSP for capturing video from IP cameras. 任何支持通过 TCP/HTTP 进行 RTSP 流式处理的相机都应正常工作。Any camera that supports RTSP streaming over TCP/HTTP should work.

是否可以在图实例中重置或更新 RTSP 源 URL?Can I reset or update the RTSP source URL in a graph instance?

可以。在图实例处于非活动状态时可以进行重置或更新。Yes, when the graph instance is in inactive state.

是否可以在测试和开发期间使用 RTSP 模拟器?Is an RTSP simulator available to use during testing and development?

可以。可以在快速入门和教程中使用 RTSP 模拟器边缘模块来支持学习过程。Yes, an RTSP simulator edge module is available for use in the quickstarts and tutorials to support the learning process. 我们会尽力提供此模块,但它可能并非始终都可用。This module is provided as best-effort and might not always be available. 强烈建议你不要让使用模拟器的时间超过数小时。We recommend strongly that you not use the simulator for more than a few hours. 在计划生产部署之前,应使用实际的 RTSP 源进行测试。You should invest in testing with your actual RTSP source before you plan a production deployment.

你们是否支持 ONVIF 在边缘发现 IP 照相机?Do you support ONVIF discovery of IP cameras at the edge?

否。我们不支持通过开放型网络视频接口论坛 (ONVIF) 在边缘发现设备。No, we don't support Open Network Video Interface Forum (ONVIF) discovery of devices on the edge.

流式处理和播放Streaming and playback

是否可以使用流式处理技术(例如 HLS 或 DASH)播放从边缘录制到 Azure 媒体服务的资产?Can I play back assets recorded to Azure Media Services from the edge by using streaming technologies such as HLS or DASH?

是的。Yes. 可以像处理 Azure 媒体服务中的任何其他资产一样,对录制的资产进行流式处理。You can stream recorded assets like any other asset in Azure Media Services. 若要流式处理内容,必须已创建流式处理终结点,并使其处于正在运行状态。To stream the content, you must have a streaming endpoint created and in the running state. 通过使用标准流式处理定位器创建过程,你可以访问 Apple HTTP Live Streaming (HLS) 或通过 HTTP 的动态自适应流式处理(DASH,也称 MPEG-DASH)清单,以便将其流式传输到任何支持的播放器框架。Using the standard Streaming Locator creation process will give you access to an Apple HTTP Live Streaming (HLS) or Dynamic Adaptive Streaming over HTTP (DASH, also known as MPEG-DASH) manifest for streaming to any capable player framework. 若要详细了解如何创建和发布 HLS 或 DASH 清单,请参阅动态打包For more information about creating and publishing HLS or DASH manifests, see dynamic packaging.

是否可以在存档资产上使用媒体服务的标准内容保护和 DRM 功能?Can I use the standard content protection and DRM features of Media Services on an archived asset?

是的。Yes. 所有标准动态加密内容保护和数字版权管理 (DRM) 功能都可用于从媒体图录制的资产。All the standard dynamic encryption content protection and digital rights management (DRM) features are available for use on assets that are recorded from a media graph.

我可以使用哪些播放器来查看录制的资产中的内容?What players can I use to view content from the recorded assets?

支持兼容的 HLS 版本 3 或版本 4 的所有标准播放机均受支持。All standard players that support compliant HLS version 3 or version 4 are supported. 此外,还支持能够进行兼容 MPEG-DASH 播放的任何播放机。In addition, any player that's capable of compliant MPEG-DASH playback is also supported.

建议用于测试的播放器包括:Recommended players for testing include:

对媒体图形资产进行流式处理存在哪些限制?What are the limits on streaming a media graph asset?

从媒体图流式处理实时资产或录制资产时会使用大规模基础结构和流式处理终结点,媒体服务支持相同的大规模基础结构和流式处理终结点,以便为媒体和娱乐客户、Over the Top (OTT) 客户以及广播客户进行按需和实时流式处理。Streaming a live or recorded asset from a media graph uses the same high-scale infrastructure and streaming endpoint that Media Services supports for on-demand and live streaming for Media & Entertainment, Over the Top (OTT), and broadcast customers. 这意味着你可快速轻松地启用 Verizon 或 Akamai,来将你的内容传递给受众,受众规模可以少至几名观众,也可多达数百万名观众,具体取决于你的场景。This means that you can quickly and easily enable Verizon, or Akamai to deliver your content to an audience as small as a few viewers or up to millions, depending on your scenario.

可以使用 Apple HLS 或 MPEG-DASH 来传送内容。You can deliver content by using either Apple HLS or MPEG-DASH.

设计 AI 模型Design your AI model

我有多个封装在 Docker 容器中的 AI 模型。如何在实时视频分析中使用它们?I have multiple AI models wrapped in a Docker container. How should I use them with Live Video Analytics?

解决方案因推理服务器用来与实时视频分析通信的通信协议而异。Solutions vary depending on the communication protocol that's used by the inferencing server to communicate with Live Video Analytics. 以下部分介绍每个协议的工作原理。The following sections describe how each protocol works.

使用 HTTP 协议:Use the HTTP protocol:

  • 单个容器(单个 lvaExtension):Single container (single lvaExtension):

    在推理服务器中,可以使用单个端口,但不同 AI 模型的终结点不同。In your inferencing server, you can use a single port but different endpoints for different AI models. 例如,对于 Python 示例,可以为每个模型使用不同的 route,如下所示:For example, for a Python sample you can use different routes per model, as shown here:

    @app.route('/score/face_detection', methods=['POST']) 
    ... 
    Your code specific to face detection model... 
    
    @app.route('/score/vehicle_detection', methods=['POST']) 
    ... 
    Your code specific to vehicle detection model 
    ... 
    

    然后,在实时视频分析部署中将图进行实例化时,为每个实例设置推理服务器 URL,如下所示:And then, in your Live Video Analytics deployment, when you instantiate graphs, set the inference server URL for each instance, as shown here:

    第一个实例:推理服务器 URL = http://lvaExtension:44000/score/face_detection1st instance: inference server URL=http://lvaExtension:44000/score/face_detection
    第二个实例:推理服务器 URL = http://lvaExtension:44000/score/vehicle_detection2nd instance: inference server URL=http://lvaExtension:44000/score/vehicle_detection

    备注

    也可在不同的端口上公开 AI 模型,并在将图进行实例化时调用它们。Alternatively, you can expose your AI models on different ports and call them when you instantiate graphs.

  • 多个容器:Multiple containers:

    使用不同的名称部署每个容器。Each container is deployed with a different name. 以前,我们在实时视频分析文档集中介绍了如何部署名为 lvaExtension 的扩展。Previously, in the Live Video Analytics documentation set, we showed you how to deploy an extension named lvaExtension. 现在,你可以开发两个不同的容器,每个容器使用相同的 HTTP 接口,这意味着它们具有相同的 /score 终结点。Now you can develop two different containers, each with the same HTTP interface, which means they have the same /score endpoint. 使用不同的名称部署这两个容器,并确保两个容器在不同的端口上进行侦听。Deploy these two containers with different names, and ensure that both are listening on different ports.

    例如,名为 lvaExtension1 的一个容器侦听端口 44000,另一个名为 lvaExtension2 的容器侦听端口 44001For example, one container named lvaExtension1 is listening for the port 44000, and a second container named lvaExtension2 is listening for the port 44001.

    在实时视频分析拓扑中,请使用不同的推理 URL 将两个图进行实例化,如下所示:In your Live Video Analytics topology, you instantiate two graphs with different inference URLs, as shown here:

    第一个实例:推理服务器 URL = http://lvaExtension1:44001/score First instance: inference server URL = http://lvaExtension1:44001/score
    第二个实例:推理服务器 URL = http://lvaExtension2:44001/scoreSecond instance: inference server URL = http://lvaExtension2:44001/score

使用 gRPC 协议:Use the gRPC protocol:

  • 在使用实时视频分析模块 1.0 的情况下,当使用常规用途远程过程调用 (gRPC) 协议时,那样做的唯一方法是让 gRPC 服务器通过不同的端口公开不同的 AI 模型。With Live Video Analytics module 1.0, when you use a general-purpose remote procedure call (gRPC) protocol, the only way to do so is if the gRPC server exposes different AI models via different ports. 此代码示例中,单个端口 (44000) 公开了所有 yolo 模型。In this code example, a single port, 44000, exposes all the yolo models. 理论上,可以重写 yolo gRPC 服务器,在端口 44000 上公开一些模型,在端口 45000 上公开另外一些模型。In theory, the yolo gRPC server could be rewritten to expose some models at port 44000 and others at port 45000.

  • 在使用实时视频分析模块 2.0 的情况下,新属性会添加到 gRPC 扩展节点。With Live Video Analytics module 2.0, a new property is added to the gRPC extension node. 此属性 (extensionConfiguration) 是可用作 gRPC 协定的一部分的可选字符串。This property, extensionConfiguration, is an optional string that can be used as a part of the gRPC contract. 在单个推理服务器中打包多个 AI 模型时,无需为每个 AI 模型公开一个节点。When you have multiple AI models packaged in a single inference server, you don't need to expose a node for every AI model. 对于图实例,你可以作为扩展提供者来定义如何通过使用 extensionConfiguration 属性来选择不同的 AI 模型。Instead, for a graph instance, you, as the extension provider, can define how to select the different AI models by using the extensionConfiguration property. 在执行期间,实时视频分析会将此字符串传递给推理服务器,而推理服务器则可以使用它来调用所需的 AI 模型。During execution, Live Video Analytics passes this string to the inferencing server, which can use it to invoke the desired AI model.

我在围绕 AI 模型构建 gRPC 服务器,并且我希望能够支持其可供多个相机或图实例使用。应如何构建服务器?I'm building a gRPC server around an AI model, and I want to be able to support its use by multiple cameras or graph instances. How should I build my server?

首先,请确保服务器可以一次处理多个请求,或者可以在并行线程中运行。First, be sure that your server can either handle more than one request at a time or work in parallel threads.

例如,以下实时视频分析 gRPC 示例中设置了默认的并行通道数:For example, a default number of parallel channels has been set in the following Live Video Analytics gRPC sample:

server = grpc.server(futures.ThreadPoolExecutor(max_workers=3)) 

在上述 gRPC 服务器实例化中,服务器一次只能为每个相机或每个图拓扑实例打开三个通道。In the preceding gRPC server instantiation, the server can open only three channels at a time per camera, or per graph topology instance. 不要尝试将三个以上的实例连接到服务器。Don't try to connect more than three instances to the server. 如果尝试打开三个以上的通道,请求会处于挂起状态,直到删除某个现有的通道。If you do try to open more than three channels, requests will be pending until an existing channel drops.

上面的 gRPC 服务器实现用在 Python 示例中。The preceding gRPC server implementation is used in our Python samples. 作为开发人员,你可以实现自己的服务器或使用前面的默认实现来增加辅助角色数目,你需要将该数目设置为用于视频源的相机数。As a developer, you can implement your own server or use the preceding default implementation to increase the worker number, which you set to the number of cameras to use for video feeds.

若要设置和使用多个相机,可以实例化多个图拓扑实例,每个实例指向相同的或不同的推理服务器(例如,上一段中提到的服务器)。To set up and use multiple cameras, you can instantiate multiple graph topology instances, each pointing to the same or a different inference server (for example, the server mentioned in the preceding paragraph).

我希望能在进行推理决策之前从上游接收多个帧。如何实现它?I want to be able to receive multiple frames from upstream before I make an inferencing decision. How can I enable that?

当前的默认示例在无状态模式下工作。Our current default samples work in a stateless mode. 它们不会保留前面的调用的状态,甚至不会保留调用方。They don't keep the state of the previous calls or even who called. 这意味着多个拓扑实例可能会调用同一推理服务器,但服务器无法区分正在进行调用的是哪个调用方,或者无法区分每个调用方的状态。This means that multiple topology instances might call the same inference server, but the server can't distinguish who is calling or the state per caller.

使用 HTTP 协议:Use the HTTP protocol:

若要保留状态、每个调用方或图拓扑实例,请使用调用方特有的 HTTP 查询参数来调用推理服务器。To keep the state, each caller, or graph topology instance, calls the inferencing server by using the HTTP query parameter that's unique to caller. 例如,每个实例的推理服务器 URL 地址如下所示:For example, the inference server URL addresses for each instance are shown here:

第一个拓扑实例 = http://lvaExtension:44000/score?id=11st topology instance= http://lvaExtension:44000/score?id=1
第二个拓扑实例 = http://lvaExtension:44000/score?id=22nd topology instance= http://lvaExtension:44000/score?id=2

......

在服务器端,分数路由知道谁正在进行调用。On the server side, the score route knows who is calling. 如果 ID = 1,则它可以单独为该调用方或图拓扑实例保留状态。If ID=1, then it can keep the state separately for that caller or graph topology instance. 然后,你可以将接收的视频帧保留在缓冲区中。You can then keep the received video frames in a buffer. 例如,可以使用数组,或者使用带有 DateTime 键的字典,并且值为帧。For example, use an array, or a dictionary with a DateTime key, and the value is the frame. 然后,可以定义要在接收 x 个帧后进行处理(推断)的服务器。You can then define the server to process (infer) after x number of frames are received.

使用 gRPC 协议:Use the gRPC protocol:

在使用 gRPC 扩展的情况下,一个会话对应于一个相机源,因此不需提供 ID。With a gRPC extension, each session is for a single camera feed, so there's no need to provide an ID. 现在,在使用 extensionConfiguration 属性的情况下,可以将视频帧存储在缓冲区中,并定义要在接收 x 个帧后进行处理(推断)的服务器。Now, with the extensionConfiguration property, you can store the video frames in a buffer and define the server to process (infer) after x number of frames are received.

特定容器上的所有 ProcessMediaStream 是否都运行同一 AI 模型?Do all ProcessMediaStreams on a particular container run the same AI model?

否。No. 启动或停止从图实例中的最终用户进行的调用会构成一个会话,也可能存在相机断开连接或重新连接的情况。Start or stop calls from the end user in a graph instance constitute a session, or perhaps there's a camera disconnect or reconnect. 如果相机是在流式处理视频,则目标是保留一个会话。The goal is to persist one session if the camera is streaming video.

  • 两个发送视频进行处理的相机会创建两个会话。Two cameras sending video for processing creates two sessions.
  • 如果将一个相机放入一个包含两个 gRPC 扩展节点的图中,该相机会创建两个会话。One camera going to a graph that has two gRPC extension nodes creates two sessions.

每个会话都是实时视频分析与 gRPC 服务器之间的全双工连接,可以有不同的模型或管道。Each session is a full duplex connection between Live Video Analytics and the gRPC server, and each session can have a different model or pipeline.

备注

如果相机断开连接或重新连接,并且相机脱机的时间超出容许限制,则实时视频分析会打开一个与 gRPC 服务器的新会话。In case of a camera disconnect or reconnect, with the camera going offline for a period beyond tolerance limits, Live Video Analytics will open a new session with the gRPC server. 不需服务器跟踪这些会话的状态。There's no requirement for the server to track the state across these sessions.

实时视频分析还为图实例中的单个相机添加了针对多个 gRPC 扩展的支持。Live Video Analytics also adds support for multiple gRPC extensions for a single camera in a graph instance. 你可以使用这些 gRPC 扩展以串行和/或并行的方式执行 AI 处理。You can use these gRPC extensions to carry out AI processing sequentially, in parallel, or as a combination of both.

备注

并行运行多个扩展会影响硬件资源。Having multiple extensions run in parallel will affect your hardware resources. 在选择符合计算需求的硬件时,请记住这一点。Keep this in mind as you're choosing the hardware that suits your computational needs.

同时出现的 ProcessMediaStream 的最大数量是多少?What is the maximum number of simultaneous ProcessMediaStreams?

实时视频分析不对此进行限制。Live Video Analytics applies no limits to this number.

如何确定我的推理服务器应使用 CPU、GPU 还是其他硬件加速器?How can I decide whether my inferencing server should use CPU or GPU or any other hardware accelerator?

你的决定取决于开发的 AI 模型的复杂性,以及你希望如何使用 CPU 和硬件加速器。Your decision depends on the complexity of the developed AI model and how you want to use the CPU and hardware accelerators. 在开发 AI 模型时,可以指定模型应使用的资源和应执行的操作。As you're developing the AI model, you can specify what resources the model should use and what actions it should perform.

如何在处理后存储带边界框的图像?How do I store images with bounding boxes post-processing?

目前,我们仅将边界框坐标作为推理消息提供。Today, we are providing bounding box coordinates as inference messages only. 可以构建一个可使用这些消息并覆盖视频帧上的边界框的自定义 MJPEG 流转化器。You can build a custom MJPEG streamer that can use these messages and overlay the bounding boxes on the video frames.

gRPC 兼容性gRPC compatibility

我如何知道媒体流描述符的必填字段有哪些?How will I know what the mandatory fields for the media stream descriptor are?

对于尚未为其提供值的任何字段,系统会为其提供一个由 gRPC 指定的默认值Any field that you don't supply a value to is given a default value, as specified by gRPC.

实时视频分析使用 proto3 版本的协议缓冲区语言。Live Video Analytics uses the proto3 version of the protocol buffer language. 实时视频分析协定使用的所有协议缓冲区数据都可以在协议缓冲区文件中找到。All the protocol buffer data that's used by Live Video Analytics contracts is available in the protocol buffer files.

如何确保我使用的是最新的协议缓冲区文件?How can I ensure that I'm using the latest protocol buffer files?

可以在协定文件站点上获取最新的协议缓冲区文件。You can obtain the latest protocol buffer files on the contract files site. 每当我们更新协定文件时,就会将其置于此位置。Whenever we update the contract files, they'll be in this location. 我们尚无更新协议文件的即时计划,因此,请在文件顶部查找包名称以了解版本。There's no immediate plan to update the protocol files, so look for the package name at the top of the files to know the version. 此内容应为:It should read:

microsoft.azure.media.live_video_analytics.extensibility.grpc.v1 

对这些文件进行更新时,会将名称末尾的“v-value”递增。Any updates to these files will increment the "v-value" at the end of the name.

备注

由于实时视频分析使用语言的 proto3 版本,因此字段为可选,版本后向和前后兼容。Because Live Video Analytics uses the proto3 version of the language, the fields are optional, and the version is backward and forward compatible.

可以通过实时视频分析使用哪些 gRPC 功能?哪些功能是必需的?哪些功能是可选的?What gRPC features are available for me to use with Live Video Analytics? Which features are mandatory and which are optional?

如果已履行协议缓冲区 (Protobuf) 协定,则可使用任何服务器端 gRPC 功能。You can use any server-side gRPC features, provided that the Protocol Buffers (Protobuf) contract is fulfilled.

监视和指标Monitoring and metrics

是否可以使用 Azure 事件网格监视边缘上的媒体图?Can I monitor the media graph on the edge by using Azure Event Grid?

是的。Yes. 可以使用 Prometheus 指标并将其发布到事件网格。You can consume Prometheus metrics and publish them to your event grid.

是否可以使用 Azure Monitor 查看云端或边缘的媒体图形的运行状况、指标和性能?Can I use Azure Monitor to view the health, metrics, and performance of my media graphs in the cloud or on the edge?

可以。我们支持此方法。Yes, we support this approach. 若要了解详细信息,请参阅 Azure Monitor 指标概述To learn more, see Azure Monitor Metrics overview.

是否有任何工具可用于简化监视媒体服务 IoT Edge 模块?Are there any tools to make it easier to monitor the Media Services IoT Edge module?

Visual Studio Code 支持 Azure IoT Tools 扩展,该扩展可用于轻松监视 LVAEdge 模块终结点。Visual Studio Code supports the Azure IoT Tools extension, with which you can easily monitor the LVAEdge module endpoints. 可以使用此工具快速地开始监视 IoT 中心内置终结点中的“事件”,并查看从边缘设备路由到云的推理消息。You can use this tool to quickly start monitoring your IoT hub built-in endpoint for "events" and view the inference messages that are routed from the edge device to the cloud.

此外,还可以使用此扩展编辑 LVAEdge 模块的模块孪生,以修改媒体图设置。In addition, you can use this extension to edit the module twin for the LVAEdge module to modify the media graph settings.

有关详细信息,请参阅监视和日志记录一文。For more information, see the monitoring and logging article.

计费和可用性Billing and availability

IoT Edge 上的实时视频分析如何计费?How is Live Video Analytics on IoT Edge billed?

如需计费详细信息,请参阅媒体服务定价For billing details, see Media Services pricing.

后续步骤Next steps

快速入门:IoT Edge 上的实时视频分析入门Quickstart: Get started with Live Video Analytics on IoT Edge