将 Microsoft 认知工具包深入学习模型与 Azure HDInsight Spark 群集配合使用Use Microsoft Cognitive Toolkit deep learning model with Azure HDInsight Spark cluster

• 03/23/2020
• • 

本文涉及以下步骤。In this article, you do the following steps.

1. 运行自定义脚本，以在 Azure HDInsight Spark 群集上安装 Microsoft Cognitive Toolkit。Run a custom script to install Microsoft Cognitive Toolkit on an Azure HDInsight Spark cluster.

2. 将 Jupyter Notebook 上传到 Apache Spark 群集，以了解如何使用 Spark Python API (PySpark) 将定型的 Microsoft Cognitive Toolkit 深度学习模型应用于 Azure Blob 存储帐户中的文件Upload a Jupyter Notebook to the Apache Spark cluster to see how to apply a trained Microsoft Cognitive Toolkit deep learning model to files in an Azure Blob Storage Account using the Spark Python API (PySpark)

先决条件Prerequisites

• HDInsight 上的 Apache Spark 群集。An Apache Spark cluster on HDInsight. 请参阅创建 Apache Spark 群集。See Create an Apache Spark cluster.

• 熟悉 Jupyter Notebook 和 Spark on HDInsight 的结合使用。Familiarity with using Jupyter Notebooks with Spark on HDInsight. 有关详细信息，请参阅使用 Apache Spark on HDInsight 加载数据并运行查询。For more information, see Load data and run queries with Apache Spark on HDInsight.

此解决方案的流程如何？How does this solution flow?

此解决方案分为两部分，即本文和在本文中上传的 Jupyter 笔记本。This solution is divided between this article and a Jupyter notebook that you upload as part of this article. 在本文中，完成以下步骤：In this article, you complete the following steps:

• 在 HDInsight Spark 群集上运行脚本操作，安装 Microsoft 认知工具包和 Python 包。Run a script action on an HDInsight Spark cluster to install Microsoft Cognitive Toolkit and Python packages.
• 将运行解决方案的 Jupyter 笔记本上传到 HDInsight Spark 群集中。Upload the Jupyter notebook that runs the solution to the HDInsight Spark cluster.

以下其余步骤涵盖在 Jupyter 笔记本中。The following remaining steps are covered in the Jupyter notebook.

• 将示例图像加载到 Spark 弹性分布式数据集或 RDD 中。Load sample images into a Spark Resilient Distributed Dataset or RDD.
  • 加载模块并定义预设。Load modules and define presets.
  • 将数据集下载到本地 Spark 群集上。Download the dataset locally on the Spark cluster.
  • 将数据集格式转换为 RDD。Convert the dataset into an RDD.
• 使用定型的 Cognitive Toolkit 模型对图像评分。Score the images using a trained Cognitive Toolkit model.
  • 将定型的 Cognitive Toolkit 模型下载到 Spark 群集。Download the trained Cognitive Toolkit model to the Spark cluster.
  • 定义由辅助角色节点使用的函数。Define functions to be used by worker nodes.
  • 对辅助角色节点上的图像评分。Score the images on worker nodes.
  • 评估模型准确性。Evaluate model accuracy.

安装 Microsoft 认知工具包Install Microsoft Cognitive Toolkit

可以使用脚本操作在 Spark 群集上安装 Microsoft 认知工具包。You can install Microsoft Cognitive Toolkit on a Spark cluster using script action. 脚本操作使用自定义脚本在群集上安装默认情况下未提供的组件。Script action uses custom scripts to install components on the cluster that are not available by default. 可以从 Azure 门户、通过 HDInsight .NET SDK 或 Azure PowerShell 使用自定义脚本。You can use the custom script from the Azure Portal, by using HDInsight .NET SDK, or by using Azure PowerShell. 还可以在创建群集过程中或者在群集已启动并运行之后使用脚本安装工具包。You can also use the script to install the toolkit either as part of cluster creation, or after the cluster is up and running.

在本文中，我们在群集创建完成后使用门户安装该工具包。In this article, we use the portal to install the toolkit, after the cluster has been created. 有关运行自定义脚本的其他方式，请参阅使用脚本操作自定义 HDInsight 群集。For other ways to run the custom script, see Customize HDInsight clusters using Script Action.

使用 Azure 门户Using the Azure Portal

有关如何使用 Azure 门户运行脚本操作的说明，请参阅使用脚本操作自定义 HDInsight 群集。For instructions on how to use the Azure portal to run script action, see Customize HDInsight clusters using Script Action. 确保提供以下输入，以便安装 Microsoft 认知工具包。Make sure you provide the following inputs to install Microsoft Cognitive Toolkit. 对于脚本操作，请使用以下值：Use the following values for your script action:

将 Jupyter 笔记本上传到 Azure HDInsight Spark 群集Upload the Jupyter notebook to Azure HDInsight Spark cluster

要将 Microsoft 认知工具包与 Azure HDInsight Spark 群集配合使用，必须将 Jupyter 笔记本 CNTK_model_scoring_on_Spark_walkthrough.ipynb 加载到 Azure HDInsight Spark 群集中。To use the Microsoft Cognitive Toolkit with the Azure HDInsight Spark cluster, you must load the Jupyter notebook CNTK_model_scoring_on_Spark_walkthrough.ipynb to the Azure HDInsight Spark cluster. GitHub https://github.com/Azure-Samples/hdinsight-pyspark-cntk-integration 中提供了此笔记本。This notebook is available on GitHub at https://github.com/Azure-Samples/hdinsight-pyspark-cntk-integration.

1. 下载并解压缩 https://github.com/Azure-Samples/hdinsight-pyspark-cntk-integration。Download and unzip https://github.com/Azure-Samples/hdinsight-pyspark-cntk-integration.

2. 在 Web 浏览器中，导航到 https://CLUSTERNAME.azurehdinsight.cn/jupyter，其中 CLUSTERNAME 是群集的名称。From a web browser, navigate to https://CLUSTERNAME.azurehdinsight.cn/jupyter, where CLUSTERNAME is the name of your cluster.

3. 在 Jupyter 笔记本中，选择右上角的“上传” ，然后导航到“下载”并选择文件 CNTK_model_scoring_on_Spark_walkthrough.ipynb。From the Jupyter notebook, select Upload in the top-right corner and then navigate to the download and select file CNTK_model_scoring_on_Spark_walkthrough.ipynb.

   

4. 再次选择“上传” 。Select Upload again.

5. 笔记本上传后，单击笔记本的名称，并按照笔记本本身中有关如何加载数据集和执行本文的说明进行操作。After the notebook is uploaded, click the name of the notebook and then follow the instructions in the notebook itself on how to load the data set and perform the article.

另请参阅See also

• 概述：Azure HDInsight 上的 Apache SparkOverview: Apache Spark on Azure HDInsight

方案Scenarios

• Apache Spark 与 BI：使用 HDInsight 中的 Spark 和 BI 工具执行交互式数据分析Apache Spark with BI: Perform interactive data analysis using Spark in HDInsight with BI tools
• Apache Spark 与机器学习：使用 HDInsight 中的 Spark 结合 HVAC 数据分析建筑物温度Apache Spark with Machine Learning: Use Spark in HDInsight for analyzing building temperature using HVAC data
• Apache Spark 与机器学习：使用 HDInsight 中的 Spark 预测食品检查结果Apache Spark with Machine Learning: Use Spark in HDInsight to predict food inspection results
• 使用 HDInsight 中的 Apache Spark 分析网站日志Website log analysis using Apache Spark in HDInsight

创建和运行应用程序Create and run applications

• 使用 Scala 创建独立的应用程序Create a standalone application using Scala
• 使用 Apache Livy 在 Apache Spark 群集中远程运行作业Run jobs remotely on an Apache Spark cluster using Apache Livy

工具和扩展Tools and extensions

• 使用适用于 IntelliJ IDEA 的 HDInsight 工具插件创建和提交 Spark Scala 应用程序Use HDInsight Tools Plugin for IntelliJ IDEA to create and submit Spark Scala applications
• 使用适用于 IntelliJ IDEA 的 HDInsight 工具插件远程调试 Apache Spark 应用程序Use HDInsight Tools Plugin for IntelliJ IDEA to debug Apache Spark applications remotely
• 在 HDInsight 上的 Apache Spark 群集中使用 Apache Zeppelin 笔记本Use Apache Zeppelin notebooks with an Apache Spark cluster on HDInsight
• 在 HDInsight 的 Apache Spark 群集中可用于 Jupyter Notebook 的内核Kernels available for Jupyter notebook in Apache Spark cluster for HDInsight
• Use external packages with Jupyter notebooks（将外部包与 Jupyter 笔记本配合使用）Use external packages with Jupyter notebooks
• Install Jupyter on your computer and connect to an HDInsight Spark cluster（在计算机上安装 Jupyter 并连接到 HDInsight Spark 群集）Install Jupyter on your computer and connect to an HDInsight Spark cluster

管理资源Manage resources

• 管理 Azure HDInsight 中 Apache Spark 群集的资源Manage resources for the Apache Spark cluster in Azure HDInsight
• Track and debug jobs running on an Apache Spark cluster in HDInsight（跟踪和调试 HDInsight 中的 Apache Spark 群集上运行的作业）Track and debug jobs running on an Apache Spark cluster in HDInsight