快速入门:使用自己的 HTTP 模型分析实时视频Quickstart: Analyze live video by using your own HTTP model

本快速入门介绍了如何在 IoT Edge 上使用实时视频分析来分析来自(模拟)IP 相机中的实时视频源。This quickstart shows you how to use Live Video Analytics on IoT Edge to analyze a live video feed from a (simulated) IP camera. 你将了解如何应用计算机视觉模型来检测对象。You'll see how to apply a computer vision model to detect objects. 实时视频源中的一部分帧被发送到推理服务。A subset of the frames in the live video feed is sent to an inference service. 结果将发送到 IoT Edge 中心。The results are sent to IoT Edge Hub.

此快速入门将 Azure VM 用作 IoT Edge 设备,并使用模拟的实时视频流。This quickstart uses an Azure VM as an IoT Edge device, and it uses a simulated live video stream. 它基于用 C# 编写的示例代码,并以检测运动并发出事件快速入门为基础。It's based on sample code written in C#, and it builds on the Detect motion and emit events quickstart.

此快速入门将 Azure VM 用作 IoT Edge 设备,并使用模拟的实时视频流。This quickstart uses an Azure VM as an IoT Edge device, and it uses a simulated live video stream. 它基于用 C# 编写的示例代码,并以检测运动并发出事件快速入门为基础。It's based on sample code written in C#, and it builds on the Detect motion and emit events quickstart.

先决条件Prerequisites

提示

在安装 Azure IoT Tools 时,系统可能会提示安装 Docker。When installing Azure IoT Tools, you might be prompted to install Docker. 可以忽略该提示。You can ignore the prompt.

观看示例视频Review the sample video

设置 Azure 资源时,一个高速公路车流量短视频被复制到 Azure 中用作 IoT Edge 设备的 Linux VM 上。When you set up the Azure resources, a short video of highway traffic is copied to the Linux VM in Azure that you're using as the IoT Edge device. 此快速入门使用视频文件来模拟实时流。This quickstart uses the video file to simulate a live stream.

打开一个应用程序,例如 VLC 媒体播放器Open an application such as VLC media player. 选择 Ctrl+N,然后粘贴指向高速公路交叉口示例视频的链接以开始播放。Select Ctrl+N and then paste a link to the highway intersection sample video to start playback. 可以看到许多车辆在高速公路上行驶的镜头。You see the footage of many vehicles moving in highway traffic.

在本快速入门中,你在 IoT Edge 上使用实时视频分析来检测车辆和人员等对象。In this quickstart, you'll use Live Video Analytics on IoT Edge to detect objects such as vehicles and persons. 将关联的推理事件发布到 IoT Edge 中心。You'll publish associated inference events to IoT Edge Hub.

概述Overview

概述

此图显示本快速入门中信号的流动方式。This diagram shows how the signals flow in this quickstart. Edge 模块模拟托管实时流式处理协议 (RTSP) 服务器的 IP 相机。An edge module simulates an IP camera hosting a Real-Time Streaming Protocol (RTSP) server. RTSP 源节点从该服务器拉取视频源,并将视频帧发送到帧速率筛选器处理器节点。An RTSP source node pulls the video feed from this server and sends video frames to the frame rate filter processor node. 该处理器会限制到达 HTTP 扩展处理器节点的视频流的帧速率。This processor limits the frame rate of the video stream that reaches the HTTP extension processor node.

HTTP 扩展节点扮演代理的角色。The HTTP extension node plays the role of a proxy. 它将视频帧转换为指定的图像类型。It converts the video frames to the specified image type. 然后,它将图像通过 REST 转发到另一个 Edge 模块,该模块在 HTTP 终结点后运行一个 AI 模型。Then it relays the image over REST to another edge module that runs an AI model behind an HTTP endpoint. 在此示例中,使用 YOLOv3 模型构建该 Edge 模块,该模型能够检测多种类型的对象。In this example, that edge module is built by using the YOLOv3 model, which can detect many types of objects. HTTP 扩展处理器节点收集检测结果并将事件发布到 IoT 中心接收器节点。The HTTP extension processor node gathers the detection results and publishes events to the IoT Hub sink node. 然后该节点将这些事件发送到 IoT Edge 中心The node then sends those events to IoT Edge Hub.

在本快速入门中,请执行以下操作:In this quickstart, you will:

  1. 创建并部署媒体图。Create and deploy the media graph.
  2. 解释结果。Interpret the results.
  3. 清理资源。Clean up resources.

创建和部署媒体图Create and deploy the media graph

检查和编辑示例文件Examine and edit the sample files

作为先决条件的一部分,请将示例代码下载到一个文件夹中。As part of the prerequisites, you downloaded the sample code to a folder. 按照以下步骤检查并编辑示例文件。Follow these steps to examine and edit the sample files.

  1. 在 Visual Studio Code 中,转到 src/edge。In Visual Studio Code, go to src/edge. 你可看到 .env 文件以及一些部署模板文件。You see your .env file and a few deployment template files.

    部署模板是指边缘设备的部署清单。The deployment template refers to the deployment manifest for the edge device. 它包含一些占位符值。It includes some placeholder values. 该 .env 文件包含这些变量的值。The .env file includes the values for those variables.

  2. 转到 src/cloud-to-device-console-app 文件夹。Go to the src/cloud-to-device-console-app folder. 你可在此处看到 appsettings.json 文件和一些其他文件:Here you see your appsettings.json file and a few other files:

    • c2d-console-app.csproj - Visual Studio Code 的项目文件。c2d-console-app.csproj - The project file for Visual Studio Code.

    • operations.json - 希望程序运行的操作的列表。operations.json - A list of the operations that you want the program to run.

    • Program.cs - 示例程序代码。Program.cs - The sample program code. 此代码:This code:

      • 加载应用设置。Loads the app settings.
      • 调用 IoT Edge 模块上的实时视频分析公开的直接方法。Invokes direct methods that the Live Video Analytics on IoT Edge module exposes. 可以通过调用模块的直接方法来使用该模块分析实时视频流。You can use the module to analyze live video streams by invoking its direct methods.
      • 暂停以检查“终端”窗口中程序的输出,并检查“输出”窗口中模块生成的事件 。Pauses so that you can examine the program's output in the TERMINAL window and examine the events that were generated by the module in the OUTPUT window.
      • 调用直接方法以清理资源。Invokes direct methods to clean up resources.
  3. 编辑 operations.json 文件:Edit the operations.json file:

    • 将链接更改为图拓扑:Change the link to the graph topology:

      "topologyUrl" : "https://raw.githubusercontent.com/Azure/live-video-analytics/master/MediaGraph/topologies/httpExtension/topology.json"

    • GraphInstanceSet 下,编辑图拓扑的名称,使其与上一个链接中的值匹配:Under GraphInstanceSet, edit the name of the graph topology to match the value in the preceding link:

      "topologyName" : "InferencingWithHttpExtension"

    • GraphTopologyDelete 下,编辑名称:Under GraphTopologyDelete, edit the name:

      "name": "InferencingWithHttpExtension"

生成并部署 IoT Edge 部署清单Generate and deploy the IoT Edge deployment manifest

  1. 右键单击“src/edge/ deployment.yolov3.template.json”文件,然后选择“生成 IoT Edge 部署清单”。Right-click the src/edge/ deployment.yolov3.template.json file and then select Generate IoT Edge Deployment Manifest.

    生成 IoT Edge 部署清单

    随即将在 src/edge/config 文件夹中创建一个清单文件 deployment.yolov3.amd64.json 。The deployment.yolov3.amd64.json manifest file is created in the src/edge/config folder.

  2. 如果已完成检测运动并发出事件快速入门,则跳过此步骤。If you completed the Detect motion and emit events quickstart, then skip this step.

    否则,请在左下角“AZURE IOT 中心”窗格附近选择“更多操作”图标,然后选择“设置 IoT 中心连接字符串” 。Otherwise, near the AZURE IOT HUB pane in the lower-left corner, select the More actions icon and then select Set IoT Hub Connection String. 可以从 appsettings.json 文件中复制字符串。You can copy the string from the appsettings.json file. 或者,为确保在 Visual Studio Code 中配置了正确的 IoT 中心,请使用选择 IoT 中心命令Or, to ensure you've configured the proper IoT hub within Visual Studio Code, use the Select IoT hub command.

    设置 IoT 中心连接字符串

  3. 右键单击“src/edge/config/ deployment.yolov3.amd64.json”,并选择“为单个设备创建部署”。Right-click src/edge/config/ deployment.yolov3.amd64.json and select Create Deployment for Single Device.

    为单个设备创建部署

  4. 如果系统提示你选择 IoT 中心设备,请选择“lva-sample-device”。When you're prompted to select an IoT Hub device, select lva-sample-device.

  5. 大约 30 秒后,在该窗口的左下角刷新 Azure IoT 中心。After about 30 seconds, in the lower-left corner of the window, refresh Azure IoT Hub. 边缘设备现在显示以下已部署的模块:The edge device now shows the following deployed modules:

    • 实时视频分析模块,名为 lvaEdgeThe Live Video Analytics module, named lvaEdge.

    • rtspsim 模块,可模拟 RTSP 服务器,充当实时视频源的源。The rtspsim module, which simulates an RTSP server and acts as the source of a live video feed.

      备注

      如果使用的是自己的边缘设备,而不是设置脚本预配的边缘设备,请转到你的边缘设备并使用管理员权限运行以下命令,以拉取并存储该快速入门所使用的示例视频文件:If you are using your own edge device instead of the one provisioned by our setup script, go to your edge device and run the following commands with admin rights, to pull and store the sample video file used for this quickstart:

      mkdir /home/lvaadmin/samples
      mkdir /home/lvaadmin/samples/input    
      curl https://lvamedia.blob.core.windows.net/public/camera-300s.mkv > /home/lvaadmin/samples/input/camera-300s.mkv  
      chown -R lvaadmin /home/lvaadmin/samples/  
      
      • yolov3 模块,是 YoloV3 对象检测模型,该模型将计算机视觉应用于图像并返回对象类型的多个类The yolov3 module, which is the YoloV3 object detection model that applies computer vision to the images and returns multiple classes of object types

      部署在边缘设备中的模块

准备监视事件Prepare to monitor events

右键单击实时视频分析设备,并选择“开始监视内置事件终结点”。Right-click the Live Video Analytics device and select Start Monitoring Built-in Event Endpoint. 需要执行此步骤,以在 Visual Studio Code 的“输出”窗口中监视 IoT 中心事件。You need this step to monitor the IoT Hub events in the OUTPUT window of Visual Studio Code.

开始监视

运行示例程序Run the sample program

  1. 若要启动调试会话,请选择 F5 键。To start a debugging session, select the F5 key. 你可在“终端”窗口中看到打印的消息。You see messages printed in the TERMINAL window.

  2. operations.json 代码首先调用直接方法 GraphTopologyListGraphInstanceListThe operations.json code starts off with calls to the direct methods GraphTopologyList and GraphInstanceList. 如果你在完成先前的快速入门后清理了资源,则该过程将返回空列表,然后暂停。If you cleaned up resources after you completed previous quickstarts, then this process will return empty lists and then pause. 若要继续,请选择 Enter 键。To continue, select the Enter key.

    --------------------------------------------------------------------------
    Executing operation GraphTopologyList
    -----------------------  Request: GraphTopologyList  --------------------------------------------------
    {
    "@apiVersion": "1.0"
    }
    ---------------  Response: GraphTopologyList - Status: 200  ---------------
    {
    "value": []
    }
    --------------------------------------------------------------------------
    Executing operation WaitForInput
    Press Enter to continue
    

    “终端”窗口将显示下一组直接方法调用:The TERMINAL window shows the next set of direct method calls:

    • GraphTopologySet 的调用,该调用使用前面的 topologyUrlA call to GraphTopologySet that uses the preceding topologyUrl

    • GraphInstanceSet 的调用,该调用使用以下正文:A call to GraphInstanceSet that uses the following body:

      {
        "@apiVersion": "1.0",
        "name": "Sample-Graph-1",
        "properties": {
          "topologyName": "InferencingWithHttpExtension",
          "description": "Sample graph description",
          "parameters": [
            {
              "name": "rtspUrl",
              "value": "rtsp://rtspsim:554/media/camera-300s.mkv"
            },
            {
              "name": "rtspUserName",
              "value": "testuser"
            },
            {
              "name": "rtspPassword",
              "value": "testpassword"
            }
          ]
        }
      }
      
    • GraphInstanceActivate 的调用,用于启动图形实例和视频流A call to GraphInstanceActivate that starts the graph instance and the flow of video

    • GraphInstanceList 的第二次调用,显示图形实例处于运行状态A second call to GraphInstanceList that shows that the graph instance is in the running state

  3. “终端”窗口中的输出会在出现 Press Enter to continue 提示时暂停。The output in the TERMINAL window pauses at a Press Enter to continue prompt. 暂时不要选择 Enter。Don't select Enter yet. 向上滚动,查看调用的直接方法的 JSON 响应有效负载。Scroll up to see the JSON response payloads for the direct methods you invoked.

  4. 切换到 Visual Studio Code 中的“输出”窗口。Switch to the OUTPUT window in Visual Studio Code. 可看到 IoT Edge 模块上的实时视频分析正发送到 IoT 中心的消息。You see messages that the Live Video Analytics on IoT Edge module is sending to the IoT hub. 本快速入门中的以下部分将讨论这些消息。The following section of this quickstart discusses these messages.

  5. 媒体图将继续运行并打印结果。The media graph continues to run and print results. RTSP 模拟器不断循环源视频。The RTSP simulator keeps looping the source video. 若要停止媒体图,请返回“终端”窗口,并选择 Enter。To stop the media graph, return to the TERMINAL window and select Enter.

    接下来会执行一系列调用,以清理资源:The next series of calls cleans up resources:

    • 调用 GraphInstanceDeactivate 停用图形实例。A call to GraphInstanceDeactivate deactivates the graph instance.
    • 调用 GraphInstanceDelete 删除该实例。A call to GraphInstanceDelete deletes the instance.
    • 调用 GraphTopologyDelete 删除拓扑。A call to GraphTopologyDelete deletes the topology.
    • GraphTopologyList 的最后一次调用显示该列表为空。A final call to GraphTopologyList shows that the list is empty.

解释结果Interpret results

运行媒体图时,来自 HTTP 扩展处理器节点的结果将通过 IoT 中心接收器节点传递到 IoT 中心。When you run the media graph, the results from the HTTP extension processor node pass through the IoT Hub sink node to the IoT hub. 在“输出”窗口中看到的消息包含 bodyapplicationProperties 部分。The messages you see in the OUTPUT window contain a body section and an applicationProperties section. 有关详细信息,请参阅创建和读取 IoT 中心消息For more information, see Create and read IoT Hub messages.

在下面的消息中,实时视频分析模块定义了应用程序属性和正文内容。In the following messages, the Live Video Analytics module defines the application properties and the content of the body.

MediaSessionEstablished 事件MediaSessionEstablished event

对媒体图进行实例化后,RTSP 源节点尝试连接到在 rtspsim-live555 容器上运行的 RTSP 服务器。When a media graph is instantiated, the RTSP source node attempts to connect to the RTSP server that runs on the rtspsim-live555 container. 如果连接成功,则打印以下事件。If the connection succeeds, then the following event is printed. 事件类型为 Microsoft.Media.MediaGraph.Diagnostics.MediaSessionEstablishedThe event type is Microsoft.Media.MediaGraph.Diagnostics.MediaSessionEstablished.

[IoTHubMonitor] [9:42:18 AM] Message received from [lvaedgesample/lvaEdge]:
{
  "body": {
    "sdp": "SDP:\nv=0\r\no=- 1586450538111534 1 IN IP4 nnn.nn.0.6\r\ns=Matroska video+audio+(optional)subtitles, streamed by the LIVE555 Media Server\r\ni=media/camera-300s.mkv\r\nt=0 0\r\na=tool:LIVE555 Streaming Media v2020.03.06\r\na=type:broadcast\r\na=control:*\r\na=range:npt=0-300.000\r\na=x-qt-text-nam:Matroska video+audio+(optional)subtitles, streamed by the LIVE555 Media Server\r\na=x-qt-text-inf:media/camera-300s.mkv\r\nm=video 0 RTP/AVP 96\r\nc=IN IP4 0.0.0.0\r\nb=AS:500\r\na=rtpmap:96 H264/90000\r\na=fmtp:96 packetization-mode=1;profile-level-id=4D0029;sprop-parameter-sets=Z00AKeKQCgC3YC3AQEBpB4kRUA==,aO48gA==\r\na=control:track1\r\n"
  },
  "applicationProperties": {
    "dataVersion": "1.0",
    "topic": "/subscriptions/{subscriptionID}/resourceGroups/{name}/providers/microsoft.media/mediaservices/hubname",
    "subject": "/graphInstances/GRAPHINSTANCENAMEHERE/sources/rtspSource",
    "eventType": "Microsoft.Media.MediaGraph.Diagnostics.MediaSessionEstablished",
    "eventTime": "2020-04-09T16:42:18.1280000Z"
  }
}

在此消息中,请注意以下详细信息:In this message, notice these details:

  • 消息为诊断事件。The message is a diagnostics event. MediaSessionEstablished 指示 RTSP 源节点(使用者)与 RTSP 模拟器连接,并已开始接收(模拟的)实时馈送。MediaSessionEstablished indicates that the RTSP source node (the subject) connected with the RTSP simulator and has begun to receive a (simulated) live feed.
  • applicationProperties 中的 subject 指示消息是从媒体图中的 RTSP 源节点生成的。In applicationProperties, subject indicates that the message was generated from the RTSP source node in the media graph.
  • applicationProperties 中的 eventType 指示此事件是诊断事件。In applicationProperties, eventType indicates that this event is a diagnostics event.
  • eventTime 指示事件发生的时间。The eventTime indicates the time when the event occurred.
  • body 包含有关诊断事件的数据。The body contains data about the diagnostics event. 在本例中,数据包含会话描述协议 (SDP) 详细信息。In this case, the data comprises the Session Description Protocol (SDP) details.

推理事件Inference event

HTTP 扩展处理器节点从 yolov3 模块接收推理结果。The HTTP extension processor node receives inference results from the yolov3 module. 然后它通过 IoT 中心接收器节点将结果作为推理事件发出。It then emits the results through the IoT Hub sink node as inference events.

在这些事件中,类型设置为 entity,用于指示它是实体,如汽车或卡车等。In these events, the type is set to entity to indicate it's an entity, such as a car or truck. eventTime 值为检测到对象时的 UTC 时间。The eventTime value is the UTC time when the object was detected.

在以下示例中,在同一视频帧检测到两辆汽车,置信度各不相同。In the following example, two cars were detected in the same video frame, with varying levels of confidence.

[IoTHubMonitor] [11:37:17 PM] Message received from [lva-sample-device/lvaEdge]:
{
  "body": {
    "inferences": [
      {
        "entity": {
          "box": {
            "h": 0.0344108157687717,
            "l": 0.5756940841674805,
            "t": 0.5929375966389974,
            "w": 0.04484643936157227
          },
          "tag": {
            "confidence": 0.8714089393615723,
            "value": "car"
          }
        },
        "type": "entity"
      },
      {
        "entity": {
          "box": {
            "h": 0.03960910373263889,
            "l": 0.2750667095184326,
            "t": 0.6102327558729383,
            "w": 0.031027007102966308
          },
          "tag": {
            "confidence": 0.7042660713195801,
            "value": "car"
          }
        },
        "type": "entity"
      }
    ]
  },
  "applicationProperties": {
    "topic": "/subscriptions/{subscriptionID}/resourceGroups/{name}/providers/microsoft.media/mediaservices/hubname",
    "subject": "/graphInstances/GRAPHINSTANCENAMEHERE/processors/inferenceClient",
    "eventType": "Microsoft.Media.Graph.Analytics.Inference",
    "eventTime": "2020-04-23T06:37:16.097Z"
  }
}

在消息中,请注意以下详细信息:In the messages, notice the following details:

  • applicationProperties 中的 subject 引用生成消息的图形拓扑中的节点。In applicationProperties, subject references the node in the graph topology from which the message was generated.
  • applicationProperties 中的 eventType 指示此事件是分析事件。In applicationProperties, eventType indicates that this event is an analytics event.
  • eventTime 值为事件发生的时间。The eventTime value is the time when the event occurred.
  • body 部分包含有关分析事件的数据。The body section contains data about the analytics event. 在本例中,该事件是推理事件,因此正文包含 inferences 数据。In this case, the event is an inference event, so the body contains inferences data.
  • inferences 部分指示 typeentityThe inferences section indicates that the type is entity. 本部分包含有关实体的其他数据。This section includes additional data about the entity.

清理资源Clean up resources

如果计划学习其他快速入门,请保留创建的资源。If you intend to try other quickstarts, keep the resources you created. 否则,请转到 Azure 门户,再转到资源组,选择运行本快速入门所用的资源组,并删除所有资源。Otherwise, go to the Azure portal, go to your resource groups, select the resource group where you ran this quickstart, and delete all the resources.

后续步骤Next steps

查看高级用户面临的其他挑战:Review additional challenges for advanced users: