在 Azure HDInsight 上的 Apache Spark 群集中使用 Apache Zeppelin 笔记本Use Apache Zeppelin notebooks with Apache Spark cluster on Azure HDInsight

03/23/2020

HDInsight Spark 群集包括可用于运行 Apache Spark 作业的 Apache Zeppelin 笔记本。HDInsight Spark clusters include Apache Zeppelin notebooks that you can use to run Apache Spark jobs. 本文介绍如何在 HDInsight 群集中使用 Zeppelin 笔记本。In this article, you learn how to use the Zeppelin notebook on an HDInsight cluster.

先决条件Prerequisites

HDInsight 上的 Apache Spark 群集。An Apache Spark cluster on HDInsight. 有关说明，请参阅在 Azure HDInsight 中创建 Apache Spark 群集。For instructions, see Create Apache Spark clusters in Azure HDInsight.
群集主存储的 URI 方案。The URI scheme for your clusters primary storage. 对于 Azure Blob 存储，此值为 wasb://；对于 Azure Data Lake Storage Gen2，此值为 abfs://。This would be wasb:// for Azure Blob Storage, abfs:// for Azure Data Lake Storage Gen2 . 如果为 Blob 存储启用了安全传输，则 URI 将为 wasbs://。If secure transfer is enabled for Blob Storage, the URI would be wasbs://. 有关详细信息，请参阅在 Azure 存储中要求安全传输。For more information, see Require secure transfer in Azure Storage .

启动 Apache Zeppelin 笔记本Launch an Apache Zeppelin notebook

在 Spark 群集的“概述”中，从群集仪表板选择“Zeppelin 笔记本”。 From the Spark cluster Overview, select Zeppelin notebook from Cluster dashboards. 输入群集的管理员凭据。Enter the admin credentials for the cluster.

备注

也可以在浏览器中打开以下 URL 来访问群集的 Zeppelin 笔记本。You may also reach the Zeppelin Notebook for your cluster by opening the following URL in your browser. 将 CLUSTERNAME 替换为群集的名称：Replace CLUSTERNAME with the name of your cluster:

https://CLUSTERNAME.azurehdinsight.cn/zeppelin
创建新的笔记本。Create a new notebook. 在标题窗格中，导航到“笔记本” > “创建新笔记”。 From the header pane, navigate to Notebook > Create new note.

输入笔记本的名称，然后选择“创建笔记” 。Enter a name for the notebook, then select Create Note.
确保笔记本标题显示“已连接”状态。Ensure the notebook header shows a connected status. 该状态由右上角的一个绿点表示。It is denoted by a green dot in the top-right corner.
将示例数据载入临时表。Load sample data into a temporary table. 在 HDInsight 中创建 Spark 群集时，系统会将示例数据文件 hvac.csv 复制到 \HdiSamples\SensorSampleData\hvac 下的关联存储帐户。When you create a Spark cluster in HDInsight, the sample data file, hvac.csv, is copied to the associated storage account under \HdiSamples\SensorSampleData\hvac.

将以下代码段粘贴到新笔记本中默认创建的空白段落处。In the empty paragraph that is created by default in the new notebook, paste the following snippet.
```
%livy2.spark
//The above magic instructs Zeppelin to use the Livy Scala interpreter

// Create an RDD using the default Spark context, sc
val hvacText = sc.textFile("wasbs:///HdiSamples/HdiSamples/SensorSampleData/hvac/HVAC.csv")

// Define a schema
case class Hvac(date: String, time: String, targettemp: Integer, actualtemp: Integer, buildingID: String)

// Map the values in the .csv file to the schema
val hvac = hvacText.map(s => s.split(",")).filter(s => s(0) != "Date").map(
    s => Hvac(s(0), 
            s(1),
            s(2).toInt,
            s(3).toInt,
            s(6)
    )
).toDF()

// Register as a temporary table called "hvac"
hvac.registerTempTable("hvac")
```
按 SHIFT + ENTER 或为段落选择“播放” 按钮以运行代码片段。Press SHIFT + ENTER or select the Play button for the paragraph to run the snippet. 段落右上角的状态应从“就绪”逐渐变成“挂起”、“正在运行”和“已完成”。The status on the right-corner of the paragraph should progress from READY, PENDING, RUNNING to FINISHED. 输出会显示在同一段落的底部。The output shows up at the bottom of the same paragraph. 屏幕截图如下所示：The screenshot looks like the following:

也可以为每个段落提供标题。You can also provide a title to each paragraph. 在段落的右侧一角，选择“设置”图标（齿轮图标），然后选择“显示标题”。 From the right-hand corner of the paragraph, select the Settings icon (sprocket), and then select Show title.

备注

所有 HDInsight 版本中的 Zeppelin 笔记本均不支持 %spark2 解释器，HDInsight 4.0 及更高版本不支持 %sh 解释器。%spark2 interpreter is not supported in Zeppelin notebooks across all HDInsight versions, and %sh interpreter will not be supported from HDInsight 4.0 onwards.

现在可以针对 hvac 表运行 Spark SQL 语句。You can now run Spark SQL statements on the hvac table. 将以下查询粘贴到新段落中。Paste the following query in a new paragraph. 该查询将检索建筑物 ID，以及每栋建筑物在指定日期的目标温度与实际温度之间的差异。The query retrieves the building ID and the difference between the target and actual temperatures for each building on a given date. 按 SHIFT + ENTER。Press SHIFT + ENTER.
```
%sql
select buildingID, (targettemp - actualtemp) as temp_diff, date from hvac where date = "6/1/13" 
```
开头的 %Sql 语句告诉笔记本要使用 Livy Scala 解释器。The %sql statement at the beginning tells the notebook to use the Livy Scala interpreter.
选择“条形图”图标以更改显示内容。 Select the Bar Chart icon to change the display. 使用选择“条形图”后显示的“设置”可以选择“键”和“值”。 settings, which appears after you have selected Bar Chart, allows you to choose Keys, and Values. 以下屏幕快照显示了输出。The following screenshot shows the output.
还可以在查询中使用变量来运行 Spark SQL 语句。You can also run Spark SQL statements using variables in the query. 以下代码片段演示如何在查询中使用可以用来查询的值定义 Temp 变量。The next snippet shows how to define a variable, Temp, in the query with the possible values you want to query with. 首次运行查询时，下拉列表中会自动填充你指定的变量值。When you first run the query, a drop-down is automatically populated with the values you specified for the variable.
```
%sql  
select buildingID, date, targettemp, (targettemp - actualtemp) as temp_diff from hvac where targettemp > "${Temp = 65,65|75|85}"
```
将此代码段粘贴到新段落，并按 SHIFT + ENTER。Paste this snippet in a new paragraph and press SHIFT + ENTER. 然后，从“温度”下拉列表中选择“65”。 Then select 65 from the Temp drop-down list.
选择“条形图”图标以更改显示内容。 Select the Bar Chart icon to change the display. 然后选择“设置”并进行以下更改： Then select settings and make the following changes:
- 组：添加 targettemp。Groups: Add targettemp.
- 值： 1.Values: 1. 删除 date。Remove date. 2.2. 添加 temp_diff。Add temp_diff. 3.3. 将聚合器从“SUM”更改为“AVG”。 Change the aggregator from SUM to AVG.
  
  以下屏幕快照显示了输出。The following screenshot shows the output.

如何在笔记本中使用外部包？How do I use external packages with the notebook?

可在 HDInsight 上的 Apache Spark 群集中配置 Zeppelin 笔记本，以使用未现成包含在群集中的、由社区贡献的外部包。You can configure the Zeppelin notebook in Apache Spark cluster on HDInsight to use external, community-contributed packages that are not included out-of-the-box in the cluster. 可以在 Maven 存储库中搜索可用包的完整列表。You can search the Maven repository for the complete list of packages that are available. 也可以从其他源获取可用包的列表。You can also get a list of available packages from other sources. 例如， Spark 包中提供了社区贡献包的完整列表。For example, a complete list of community-contributed packages is available at Spark Packages.

本文将介绍如何在 Jupyter 笔记本中使用 spark-csv 包。In this article, you'll see how to use the spark-csv package with the Jupyter notebook.

打开解释器设置。Open interpreter settings. 选择右上角的登录用户名，然后选择“解释器” 。From the top-right corner, select the logged in user name, then select Interpreter.
滚动到“livy2”，然后选择“编辑”。 Scroll to livy2, then select edit.
导航到键 livy.spark.jars.packages，并以 group:id:version 格式设置其值。Navigate to key livy.spark.jars.packages, and set its value in the format group:id:version. 因此，如果要使用 spark-csv 包，必须将键的值设置为 com.databricks:spark-csv_2.10:1.4.0。So, if you want to use the spark-csv package, you must set the value of the key to com.databricks:spark-csv_2.10:1.4.0.

依次选择“保存”、“确定”，以重启 Livy 解释器。 Select Save and then OK to restart the Livy interpreter.
要了解如何访问上面输入的键的值，请查看以下内容。If you want to understand how to arrive at the value of the key entered above, here's how.

a.a. 在 Maven 存储库中找出该包。Locate the package in the Maven Repository. 在本文中，使用了 spark-csv。For this article, we used spark-csv.

b.b. 从存储库中收集 GroupId、ArtifactId 和 Version 的值。From the repository, gather the values for GroupId, ArtifactId, and Version.

c.c. 串连这三个值并以冒号分隔 ( : )。Concatenate the three values, separated by a colon (:).
```
 com.databricks:spark-csv_2.10:1.4.0
```

Zeppelin 笔记本保存在何处？Where are the Zeppelin notebooks saved?

Zeppelin 笔记本保存在群集头节点。The Zeppelin notebooks are saved to the cluster headnodes. 因此，如果删除群集，笔记本也会被删除。So, if you delete the cluster, the notebooks will be deleted as well. 如果想要保留笔记本以供将来在其他群集中使用，那么必须在运行完作业之后，将笔记本导出。If you want to preserve your notebooks for later use on other clusters, you must export them after you have finished running the jobs. 若要导出笔记本，请选择下图所示的“导出”图标。 To export a notebook, select the Export icon as shown in the image below.

下载笔记本 Download notebook

此操作可在下载位置将笔记本保存为 JSON 文件。This saves the notebook as a JSON file in your download location.

Livy 会话管理Livy session management

在 Zeppelin 笔记本中运行第一个代码段时，在 HDInsight Spark 群集中创建了新的 Livy 会话。When you run the first code paragraph in your Zeppelin notebook, a new Livy session is created in your HDInsight Spark cluster. 此会话在随后创建的所有 Zeppelin 笔记本中共享。This session is shared across all Zeppelin notebooks that you subsequently create. 如果由于某种原因（群集重新启动等）导致 Livy 会话终止，则将无法从 Zeppelin notebook 运行作业。If for some reason the Livy session is killed (cluster reboot, and so on), you won't be able to run jobs from the Zeppelin notebook.

在这种情况下，必须首先执行以下步骤，才能开始在 Zeppelin 笔记本中运行作业。In such a case, you must perform the following steps before you can start running jobs from a Zeppelin notebook.

在 Zeppelin 笔记本中重启 Livy 解释器。Restart the Livy interpreter from the Zeppelin notebook. 为此，请选择右上角的登录用户名打开解释器设置，然后选择“解释器” 。To do so, open interpreter settings by selecting the logged in user name from the top-right corner, then select Interpreter.
滚动到“livy2”，然后选择“重启”。 Scroll to livy2, then select restart.
在现有的 Zeppelin 笔记本中运行代码单元。Run a code cell from an existing Zeppelin notebook. 此操作可在 HDInsight 群集中创建新的 Livy 会话。This creates a new Livy session in the HDInsight cluster.

常规信息General information

验证服务Validate service

若要从 Ambari 验证服务，请导航到 https://CLUSTERNAME.azurehdinsight.cn/#/main/services/ZEPPELIN/summary，其中 CLUSTERNAME 是群集的名称。To validate the service from Ambari, navigate to https://CLUSTERNAME.azurehdinsight.cn/#/main/services/ZEPPELIN/summary where CLUSTERNAME is the name of your cluster.

若要从命令行验证服务，请通过 SSH 连接到头节点。To validate the service from a command line, SSH to the head node. 使用命令 sudo su zeppelin 将用户切换到 zeppelin。Switch user to zeppelin using command sudo su zeppelin. 状态命令：Status commands:

命令Command	说明Description
`/usr/hdp/current/zeppelin-server/bin/zeppelin-daemon.sh status`	服务状态。Service status.
`/usr/hdp/current/zeppelin-server/bin/zeppelin-daemon.sh --version`	服务版本。Service version.
`ps -aux \| grep zeppelin`	标识 PID。Identify PID.

日志位置Log locations

服务Service	`Path`Path
zeppelin-serverzeppelin-server	/usr/hdp/current/zeppelin-server//usr/hdp/current/zeppelin-server/
服务器日志Server Logs	/var/log/zeppelin/var/log/zeppelin
配置解释器、Shiro、site.xml、log4jConfiguration Interpreter, Shiro, site.xml, log4j	/usr/hdp/current/zeppelin-server/conf or /etc/zeppelin/conf/usr/hdp/current/zeppelin-server/conf or /etc/zeppelin/conf
PID 目录PID directory	/var/run/zeppelin/var/run/zeppelin

启用调试日志记录Enable debug logging

导航到 https://CLUSTERNAME.azurehdinsight.cn/#/main/services/ZEPPELIN/summary，其中 CLUSTERNAME 是群集的名称。Navigate to https://CLUSTERNAME.azurehdinsight.cn/#/main/services/ZEPPELIN/summary where CLUSTERNAME is the name of your cluster.
导航到“CONFIGS” > “Advanced zeppelin-log4j-properties” > “log4j_properties_content” 。Navigate to CONFIGS > Advanced zeppelin-log4j-properties > log4j_properties_content.
将 log4j.appender.dailyfile.Threshold = INFO 修改为 log4j.appender.dailyfile.Threshold = DEBUG。Modify log4j.appender.dailyfile.Threshold = INFO to log4j.appender.dailyfile.Threshold = DEBUG.
添加 log4j.logger.org.apache.zeppelin.realm=DEBUG。Add log4j.logger.org.apache.zeppelin.realm=DEBUG.
保存更改并重启服务。Save changes and restart service.

后续步骤Next steps

概述：Azure HDInsight 上的 Apache SparkOverview: Apache Spark on Azure HDInsight