Run Apache Spark from the Spark Shell

An interactive Apache Spark Shell provides a REPL (read-execute-print loop) environment for running Spark commands one at a time and seeing the results. This process is useful for development and debugging. Spark provides one shell for each of its supported languages: Scala, Python, and R.

Run an Apache Spark Shell

1. Use ssh command to connect to your cluster. Edit the command below by replacing CLUSTERNAME with the name of your cluster, and then enter the command:

   ssh sshuser@CLUSTERNAME-ssh.azurehdinsight.cn
   

2. Spark provides shells for Scala (spark-shell), and Python (pyspark). In your SSH session, enter one of the following commands:

   spark-shell
   
   # Optional configurations
   # spark-shell --num-executors 4 --executor-memory 4g --executor-cores 2 --driver-memory 8g --driver-cores 4
   

   pyspark
   
   # Optional configurations
   # pyspark --num-executors 4 --executor-memory 4g --executor-cores 2 --driver-memory 8g --driver-cores 4
   

   If you intend to use any optional configuration, ensure you first review OutOfMemoryError exception for Apache Spark.

3. A few basic example commands. Choose the relevant language:

   val textFile = spark.read.textFile("/example/data/fruits.txt")
   textFile.first()
   textFile.filter(line => line.contains("apple")).show()
   

   textFile = spark.read.text("/example/data/fruits.txt")
   textFile.first()
   textFile.filter(textFile.value.contains("apple")).show()
   

4. Query a CSV file. Note the language below works for spark-shell and pyspark.

   spark.read.csv("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv").show()
   

5. Query a CSV file and store results in variable:

   var data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv")
   

   data = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/HdiSamples/HdiSamples/SensorSampleData/building/building.csv")
   

6. Display results:

   data.show()
   data.select($"BuildingID", $"Country").show(10)
   

   data.show()
   data.select("BuildingID", "Country").show(10)
   

7. Exit

   :q
   

   exit()
   

SparkSession and SparkContext instances

By default when you run the Spark Shell, instances of SparkSession and SparkContext are automatically instantiated for you.

To access the SparkSession instance, enter spark. To access the SparkContext instance, enter sc.

Important shell parameters

The Spark Shell command (spark-shell, or pyspark) supports many command-line parameters. To see a full list of parameters, start the Spark Shell with the switch --help. Some of these parameters may only apply to spark-submit, which the Spark Shell wraps.

Next steps

• See Introduction to Apache Spark on Azure HDInsight for an overview.
• See Create an Apache Spark cluster in Azure HDInsight to work with Spark clusters and SparkSQL.
• See What is Apache Spark Structured Streaming? to write applications that process streaming data with Spark.