Connect to HiveServer2 using Beeline or install Beeline locally to connect from your local

  • Article

Apache Beeline is a Hive client that is included on the head nodes of your HDInsight cluster. This article describes how to connect to HiveServer2 using the Beeline client installed on your HDInsight cluster across different types of connections. It also discusses how to Install the Beeline client locally.

Types of connections

From an SSH session

When you connect from an SSH session to a cluster headnode, you can then connect to the headnodehost address on port 10001:

beeline -u 'jdbc:hive2://headnodehost:10001/;transportMode=http'

Over an Azure Virtual Network

When you connect from a client to HDInsight over an Azure Virtual Network, you must provide the fully qualified domain name (FQDN) of a cluster head node. Since this connection is made directly to the cluster nodes, the connection uses port 10001:

beeline -u 'jdbc:hive2://<headnode-FQDN>:10001/;transportMode=http'

Replace <headnode-FQDN> with the fully qualified domain name of a cluster headnode. To find the fully qualified domain name of a headnode, use the information in the Managed HDInsight using the Apache Ambari REST API document.

To HDInsight Enterprise Security Package (ESP) cluster using Kerberos

When you connect from a client to an Enterprise Security Package (ESP) cluster joined to Microsoft Entra Domain Services on a machine in same realm of the cluster, you must also specify the domain name <AAD-Domain> and the name of a domain user account with permissions to access the cluster <username>:

kinit <username>
beeline -u 'jdbc:hive2://<headnode-FQDN>:10001/default;principal=hive/_HOST@<AAD-Domain>;auth-kerberos;transportMode=http' -n <username>

Replace <username> with the name of an account on the domain with permissions to access the cluster. Replace <AAD-DOMAIN> with the name of the Microsoft Entra ID that the cluster is joined to. Use an uppercase string for the <AAD-DOMAIN> value, otherwise the credential won't be found. Check /etc/krb5.conf for the realm names if needed.

To find the JDBC URL from Ambari:

  1. From a web browser, navigate to https://CLUSTERNAME.azurehdinsight.cn/#/main/services/HIVE/summary, where CLUSTERNAME is the name of your cluster. Ensure that HiveServer2 is running.

  2. Use clipboard to copy the HiveServer2 JDBC URL.

Over public or private endpoints

When you connect to a cluster using the public or private endpoints, you must provide the cluster login account name (default admin) and password. For example, using Beeline from a client system to connect to the clustername.azurehdinsight.cn address. This connection is made over port 443, and is encrypted using TLS/SSL.

Replace clustername with the name of your HDInsight cluster. Replace admin with the cluster login account for your cluster. For ESP clusters, use the full UPN (for example, user@domain.com). Replace password with the password for the cluster login account.

beeline -u 'jdbc:hive2://clustername.azurehdinsight.cn:443/;ssl=true;transportMode=http;httpPath=/hive2' -n admin -p 'password'

Or for private endpoint:

beeline -u 'jdbc:hive2://clustername-int.azurehdinsight.cn:443/;ssl=true;transportMode=http;httpPath=/hive2' -n admin -p 'password'

Private endpoints point to a basic load balancer, which can only be accessed from the VNETs peered in the same region. See constraints on global VNet peering and load balancers for more info. You can use the curl command with -v option to troubleshoot any connectivity problems with public or private endpoints before using beeline.

Use Beeline with Apache Spark

Apache Spark provides its own implementation of HiveServer2, which is sometimes referred to as the Spark Thrift server. This service uses Spark SQL to resolve queries instead of Hive. And may provide better performance depending on your query.

Through public or private endpoints

The connection string used is slightly different. Instead of containing httpPath=/hive2 it uses httpPath/sparkhive2. Replace clustername with the name of your HDInsight cluster. Replace admin with the cluster login account for your cluster. Replace password with the password for the cluster login account.

Note

For ESP clusters, replace admin with full UPN (for example, user@domain.com).

beeline -u 'jdbc:hive2://clustername.azurehdinsight.cn:443/;ssl=true;transportMode=http;httpPath=/sparkhive2' -n admin -p 'password'

Or for private endpoint:

beeline -u 'jdbc:hive2://clustername-int.azurehdinsight.cn:443/;ssl=true;transportMode=http;httpPath=/sparkhive2' -n admin -p 'password'

Private endpoints point to a basic load balancer, which can only be accessed from the VNETs peered in the same region. See constraints on global VNet peering and load balancers for more info. You can use the curl command with -v option to troubleshoot any connectivity problems with public or private endpoints before using beeline.

From cluster head node or inside Azure Virtual Network with Apache Spark

When you connect directly from the cluster head node, or from a resource inside the same Azure Virtual Network as the HDInsight cluster, port 10002 should be used for Spark Thrift server instead of 10001. The following example shows how to connect directly to the head node:

/usr/hdp/current/spark2-client/bin/beeline -u 'jdbc:hive2://headnodehost:10002/;transportMode=http'

Install Beeline client

Although Beeline is included on the head nodes, you may want to install it locally. The install steps for a local machine are based on a Windows Subsystem for Linux.

  1. Update package lists. Enter the following command in your bash shell:

    sudo apt-get update

  2. Install Java if not installed. You can check with the which java command.

    1. If no java package is installed, enter the following command:

      sudo apt install openjdk-11-jre-headless

    2. Open the bashrc file (often found in ~/.bashrc): nano ~/.bashrc.

    3. Amend the bashrc file. Add the following line at the end of the file:

      export JAVA_HOME=/usr/lib/jvm/java-1.11.0-openjdk-amd64

      Then press Ctrl+X, then Y, then enter.

  3. Download Hadoop and Beeline archives, enter the following commands:

    wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz
wget https://archive.apache.org/dist/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz

  4. Unpack the archives, enter the following commands:

    tar -xvzf hadoop-2.7.3.tar.gz
tar -xvzf apache-hive-1.2.1-bin.tar.gz

  5. Further amend the bashrc file. You'll need to identify the path to where the archives were unpacked. If using the Windows Subsystem for Linux, and you followed the steps exactly, your path would be /mnt/c/Users/user/, where user is your user name.

    1. Open the file: nano ~/.bashrc

    2. Modify the commands below with the appropriate path and then enter them at the end of the bashrc file:

      export HADOOP_HOME=/path_where_the_archives_were_unpacked/hadoop-2.7.3
export HIVE_HOME=/path_where_the_archives_were_unpacked/apache-hive-1.2.1-bin
PATH=$PATH:$HIVE_HOME/bin

    3. Then press Ctrl+X, then Y, then enter.

  6. Close and then reopen your bash session.

  7. Test your connection. Use the connection format from Over public or private endpoints, above.

Next steps