Hive Component Usage

Connection Methods

Connecting to Hive Server2 via Beeline

Environment Preparation

Prepare a pod for the Hive client. Create a new YAML file locally, for example, hive-client.yaml, and fill in the following content.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hive-client
  namespace: default
  labels:
    app: hive-client
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hive-client
  template:
    metadata:
      labels:
        app: hive-client
    spec:
      volumes:
        - name: hive-config
          configMap:
            name: hive-server2-context
            defaultMode: 420
        - name: hdfs-config
          configMap:
            name: hdfs-config
            defaultMode: 420
        - name: kerberos-config
          configMap:
            name: krb5-config
            defaultMode: 420
        - name: user-keytab
          persistentVolumeClaim:
            claimName: home-keytab-data-pvc
      containers:
        - name: hive-client
          image: od-registry.linktimecloud.com/ltc-hms:3.1.3-1.17
          command:
            - tail
            - '-f'
            - /dev/null
          env: []
          resources:
            limits:
              cpu: '2'
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 512Mi
          volumeMounts:
            - name: hive-config
              readOnly: true
              mountPath: /opt/hive/conf/hive-site.xml
              subPath: hive-site.xml
            - name: hdfs-config
              readOnly: true
              mountPath: /opt/hive/conf/core-site.xml
              subPath: core-site.xml
            - name: hdfs-config
              readOnly: true
              mountPath: /opt/hive/conf/hdfs-site.xml
              subPath: hdfs-site.xml
            - name: kerberos-config
              readOnly: true
              mountPath: /etc/krb5.conf
              subPath: krb5.conf
            - name: user-keytab
              readOnly: true
              mountPath: /keytab
          imagePullPolicy: IfNotPresent
      restartPolicy: Always
      terminationGracePeriodSeconds: 30
      dnsPolicy: ClusterFirst
      securityContext: {}
      imagePullSecrets:
        - name: devregistry
      affinity:
        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - hive-client
              topologyKey: kubernetes.io/hostname

Notes:

Set the namespace to where the hdfs/hive-metastore/hive-server2 cluster is located.
image use hive-metastore image
configmap hive-server2-context contains the Hive Server2 configuration including hive-site.xml.
configmap hdfs-config contains the HDFS configuration including core-site.xml and hdfs-site.xml.
configmap krb5-config contains the KDC configuration including krb5.conf.
pvc home-keytab-data-pvc contains the user's keytab.

If Kerberos is not enabled in the cluster, you can remove the volumes/volumeMounts for KDC/keytab.

Execute kubectl apply -f hive-client.yaml to create a Hive client pod.

For example, in the default namespace, execute the kubectl command to enter the Hive client pod.

kubectl exec -it hive-client -n default -- bash

Connecting in High Availability Mode

Multiple instances of hive-server2 will register their service addresses to ZooKeeper, and clients can obtain a random address from ZooKeeper.

If Kerberos is enabled, first execute kinit, then connect with Beeline.

kinit -kt /keytab/user1/user1.keytab user1
beeline -u 'jdbc:hive2://zookeeper:2181/;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=default_hiveserver2/server;principal=hive/_HOST@BDOS.CLOUD'

If Kerberos is not enabled, you can connect directly as the root user.

beeline -u 'jdbc:hive2://zookeeper:2181/;serviceDiscoveryMode=zookeeper;zooKeeperNamespace=default_hiveserver2/server' -n root

Note：The value to fill in for zooKeeperNamespace in the above Beeline connection string is _hiveserver2/server. If the namespace is admin, then it should be replaced with zooKeeperNamespace = admin_hiveserver2/server.

Connecting to a Specific Hive Server2 Instance

You can also connect to a specific instance of hive-server2.

With Kerberos enabled:
beeline -u 'jdbc:hive2://hive-server2-0.hive-server2:10000/;principal=hive/_HOST@BDOS.CLOUD'
# Without Kerberos enabled:
beeline -u 'jdbc:hive2://hive-server2-0.hive-server2:10000/' -n root

Connecting to Hive Server2 via Java

Configure project dependencies (hadoop-common and hive-jdbc) in the pom.xml file. The additional project dependencies for this example are as follows.

<dependencies>
  <dependency>
    <groupId>org.apache.hive</groupId>
    <artifactId>hive-jdbc</artifactId>
    <version>3.1.3</version>
  </dependency>
  <dependency>
    <groupId>org.apache.hadoop</groupId>
    <artifactId>hadoop-common</artifactId>
    <version>3.1.1</version>
  </dependency>
</dependencies>

Write code to connect to HiveServer2 and manipulate Hive table data. The example code is as follows.

import java.sql.*;

public class App
{
    private static String driverName = "org.apache.hive.jdbc.HiveDriver";

    public static void main(String[] args) throws SQLException {

        try {
            Class.forName(driverName);
        } catch (ClassNotFoundException e) {
            e.printStackTrace();
        }

        Connection con = DriverManager.getConnection(
            "jdbc:hive2://hive-server2-0.hive-server2:10000", "root", "");

        Statement stmt = con.createStatement();

        String sql = "select * from sample_tbl limit 10";
        ResultSet res = stmt.executeQuery(sql);

        while (res.next()) {
            System.out.println(res.getString(1) + "\t" + res.getString(2));
        }

    }
}