Quickstart: Create a serverless Apache Spark pool using Synapse Studio

  • Article

Azure Synapse Analytics offers various analytics engines to help you ingest, transform, model, analyze, and serve your data. Apache Spark pool offers open-source big data compute capabilities. After you create an Apache Spark pool in your Synapse workspace, data can be loaded, modeled, processed, and served to obtain insights.

This quickstart describes the steps to create an Apache Spark pool in a Synapse workspace by using Synapse Studio.

Important

Billing for Spark instances is prorated per minute, whether you are using them or not. Be sure to shutdown your Spark instance after you have finished using it, or set a short timeout. For more information, see the Clean up resources section of this article.

If you don't have an Azure subscription, create a trial account before you begin.

Prerequisites

Sign in to the Azure portal

Sign in to the Azure portal

  1. Navigate to the Synapse workspace where the Apache Spark pool will be created by typing the service name (or resource name directly) into the search bar. Screenshot from the Azure portal of the search bar with Synapse workspaces typed in.

  2. From the list of workspaces, type the name (or part of the name) of the workspace to open. For this example, we use a workspace named contosoanalytics. Screenshot from the Azure portal of the list of Synapse workspaces filtered to show those containing the name Contoso.

Launch Synapse Studio

From the workspace overview, select the Workspace web URL to open Synapse Studio.

Screenshot from the Azure portal of a Synapse workspace overview with Launch Synapse Studio highlighted.

Create the Apache Spark pool in Synapse Studio

Important

Azure Synapse Runtime for Apache Spark 2.4 has been deprecated and officially not supported since September 2023. Given Spark 3.1 and Spark 3.2 are also End of Support announced, we recommend customers migrate to Spark 3.3.

  1. On the Synapse Studio home page, navigate to the Management Hub in the left navigation by selecting the Manage icon. Screenshot from the Azure portal of the Synapse Studio home page with Management Hub section highlighted.

  2. Once in the Management Hub, navigate to the Apache Spark pools section to see the current list of Apache Spark pools that are available in the workspace. Screenshot from the Azure portal of the Synapse Studio management hub with Apache Spark pools navigation selected.

  3. Select + New and the new Apache Spark pool create wizard will appear.

  4. Enter the following details in the Basics tab:

    Setting Suggested value Description
    Apache Spark pool name A valid pool name, like contosospark This is the name that the Apache Spark pool will have.
    Node size Small (4 vCPU / 32 GB) Set this to the smallest size to reduce costs for this quickstart
    Autoscale Disabled We won't need autoscale in this quickstart
    Number of nodes 8 Use a small size to limit costs in this quickstart
    Dynamically allocate executors Disabled This setting maps to the dynamic allocation property in Spark configuration for Spark Application executors allocation. We won't need autoscale in this quickstart.

    Screenshot from the Azure portal of the Basics for Synapse Studio new Apache Spark pool.

    Important

    There are specific limitations for the names that Apache Spark pools can use. Names must contain letters or numbers only, must be 15 or less characters, must start with a letter, not contain reserved words, and be unique in the workspace.

  5. In the next tab, Additional settings, leave all settings as defaults.

  6. Select Tags. Consider using Azure tags. For example, the "Owner" or "CreatedBy" tag to identify who created the resource, and the "Environment" tag to identify whether this resource is in Production, Development, etc. For more information, see Develop your naming and tagging strategy for Azure resources. When ready, select Review + create.

  7. In the Review + create tab, make sure that the details look correct based on what was previously entered, and press Create.

    Screenshot from the Azure portal of the Create Synapse Studio new Apache Spark pool.

  8. The Apache Spark pool will start the provisioning process.

  9. Once the provisioning is complete, the new Apache Spark pool will appear in the list.

    Screenshot from the Azure portal of the Synapse Studio new Apache Spark pool list.

Clean up Apache Spark pool resources using Synapse Studio

The following steps delete the Apache Spark pool from the workspace using Synapse Studio.

Warning

Deleting a Spark pool will remove the analytics engine from the workspace. It will no longer be possible to connect to the pool, and all queries, pipelines, and notebooks that use this Spark pool will no longer work.

If you want to delete the Apache Spark pool, do the following steps:

  1. Navigate to the Apache Spark pools in the Management Hub in Synapse Studio.

  2. Select the ellipsis next to the Apache pool to be deleted (in this case, contosospark) to show the commands for the Apache Spark pool.

    Screenshot from the Azure portal of a list of Apache Spark pools, with the recently created pool selected.

  3. Select Delete.

  4. Confirm the deletion, and press Delete button.

  5. When the process completes successfully, the Apache Spark pool will no longer be listed in the workspace resources.

Related content